Why We See What We Do
A probabilistic strategy based on past experience explains the remarkable difference between what we see and physical reality
Visual illusions fascinate people. What we see—whether considered in terms of the brightness of objects, their colors or their arrangement in space—is often at odds with the underlying reality measured with photometers, spectrophotometers or rulers. In the 18th century, the Irish philosopher George Berkeley provided some insight into these discrepancies. In his "Essay Towards a New Theory of Vision," Berkeley pointed out that the judgment of distance, for example, cannot be derived directly from the geometrical information in the retinal image. Thus, a given line in the retinal image could have been generated by the edge of a physically small object nearby, or equally well by an edge associated with a larger object farther away.
Indeed, all retinal information suffers from this inherent ambiguity. The illumination of objects and the physical properties that determine the amount and quality of light that objects return to the eye are also conflated in the retinal stimulus; thus Berkeley's general argument applies to sensations of brightness and color, as well as to the perception of space. In each of these basic aspects of vision, the information in the retinal image cannot directly reveal the true sources of the stimulus in the physical world. As a result, the relation between the world and our perception of it is, by its nature, an uncertain one.
In addition to providing some indication of why what we see might not always tally with reality, this fundamental fact about vision presents a biological dilemma. Survival in a complex and potentially hostile environment clearly depends on responding appropriately to the physical reality that underlies the images on the retina. For example, mistaking a smaller object nearby for a larger but more distant one could obviously be disastrous for a deluded observer. If, however, the image on the retina cannot uniquely define the underlying reality that the observer must respond to, how then does the visual system generate behavior that usually deals successfully with a world that it cannot directly apprehend?
As we show here, a growing body of evidence indicates that the visual system of humans—and presumably many other visual animals—solves Berkeley's dilemma by generating perceptions on a wholly empirical basis. Rather than analyzing the components of the retinal image as such, percepts are determined probabilistically, using feedback from the outcome of visually guided behavior in the past to progressively improve performance in the face of the inevitable uncertainty of retinal information. The result of this process, and indeed the evidence for it, is that what we perceive accords not with the features of the retinal stimulus or the properties of the underlying objects, but with what the same or similar stimuli have typically signified in both the experience of the species over the eons and the experience of individuals over their lifetimes.