Why We See What We Do
A probabilistic strategy based on past experience explains the remarkable difference between what we see and physical reality
Perceiving Scene Geometry
Vision scientists noted long ago that the perception of lines does not always accord with the real-world geometry of the underlying objects. For instance, the angles formed by lines making, or implying, an acute angle are seen as being a few degrees larger than they really are, whereas obtuse angles are seen as being a few degrees smaller. Despite a great deal of speculation about this anomaly dating from the latter part of the 19th century, there has been no consensus about its origin. Thus we asked whether these and other geometrical misperceptions might be explained in much the same empirical terms as brightness and color.
Much like luminance or spectral power, the stimulus that gives rise to a perceived angle is profoundly ambiguous. An angle projected onto a surface—the retina, for example—can arise from objects having a variety of measured angles and arm lengths, arranged in infinitely many three-dimensional orientations. In interacting with the objects that give rise to particular angle projections on the retina, observers through the ages would have experienced great variation between a given angle in retinal projection and the angles of its real-world sources. Moreover, the variations are systematic. In consequence, the perceptions elicited by different angles projected onto the retina would be expected to correspond to these frequency distributions.
To test this interpretation, we first needed to determine the probability distribution of all the possible three-dimensional sources of a projected angle. When these distributions for all possible angles were computed using the principles of projective geometry, we found that acute-angle projections usually come from sources with larger angles than the projections. Conversely, the sources of obtuse-angle projections are typically generated by sources that are somewhat smaller than the projected angle. Right-angle projections and straight lines are generated by sources that, on average, have the angular subtense of the object itself. If percepts are determined empirically, then the visual system should generate perceptions of angles that incorporate and reflect these statistical facts of projective geometry.
We assessed this prediction by asking subjects to report their perceptions of different angular stimuli in a series of tests in which the adjustment of a test line indicated the angular subtense they were actually seeing. For example, if the subject perceived the angle to be bigger than it actually is, then the test line would be set in a position that revealed this discrepancy—being not quite parallel with the angle's arm. The results derived from such tests tallied quite well with the probability distribution of the possible sources of the corresponding stimuli, indicating that the spatial arrangement observers see is neither the retinal projection nor its real-world source, but its empirical, or past, significance.
Taken together, this evidence drawn from the perception of brightness, color and geometry supports the idea that the problem first emphasized by Berkeley is resolved by generating visual percepts according to the probability distribution of the possible sources of the visual stimulus, whatever it may be. As a result, observers see what a visual scene typically signified in the past, rather than what it actually is in the present. We see what we do, therefore, because the statistics of past experience is the basis on which the visual system contends with the dilemma posed by the inherent ambiguity of visual stimuli.