The False Promise of View-invariance

Any complete theory of visual object recognition must explain how humans are able to reliably identify specific objects from a near-infinity of different orientations. For example, can you identify the image at the start of this article?

Many theorists, although perhaps most emphatically Jeff Hawkins, have claimed that this 'view invariance' is the hardest problem in computational vision. Accordingly, many explorations of possible object recognition mechanisms have posited the existence of view-invariant geometric primitives, such as geons, somewhere in visual cortex. This theory says that objects are recognized by extracting these geons from 2-d retinal images, and the relationships between these geons form the basis for basic-level object categorization.

The major strengths of view-invariant theories (relative to 'template-based' approaches) are their viewpoint-invariance (well, obviously!), their resistance to visual noise, a sufficient combinatorial power to describe the object space of human visual experience, correspondence with some experimental and anecdotal data (complementary-part priming and contour deletion), and a superficial complementarity with what is known about receptive field structure in inferotemporal cortex. Additionally, given that geons are simple combinations of nonaccidental features, it seems to be a tractable way of implementing basic three-dimensionality in object recognition.

But in a seminal paper from Cognition in 1995, Michael Tarr took a step back from the intuitive appeal of invariant object recognition, and forced theorists to face up to the facts: human object recognition is not invariant - we are clearly faster at recognizing objects from specific, characteristic views. Could it be that we actually store a unique representation for every possible view of every possible object?

It sounds implausible, but painstaking experiments show that this is a more parsimonious explanation of human object recognition than view-invariant theories. For example, the cases in which object recognition seems view-invariant can be explained by floor effects in reaction time measures, such that in these cases all the stimuli are discriminable on the basis of features arranged along a single dimension. As soon as these features are arranged with an additional degree of freedom, reaction times are found that are consistent with view-dependence. Results from Shepherd's classic mental rotation experiments clearly show that recognition is not completely invariant: reaction times increase linearly with degree of rotation, as though we require some 3-D mental rotation in order to match images with their stored representations.

Secondly, view-dependence also seems to be more compatible with neurophysiological data. Single cell recordings from monkey IT show view-dependent response patterns. Although there are some neurons that have been identified that are entirely view-invariant ('bill clinton neuron', 'halle berry neuron') these appear to be nearly "everything-invariant": because those neurons fire reliably for both caricatures and photos (both young and old), they appear to be more conceptual than visual.

Further, certain algorithms have been discovered which can recognize objects from novel perspectives by interpolating between (or extrapolating beyond) characteristic views of those objects. It appears that just a few orthogonal images of each object (perhaps as few as five or six, but depedent on expertise and geometric complexity) in combination with some sophisticated processing of these images, are sufficient to recognize an enormous array of perspectives on an enormous number objects.

Related Posts:
Active Maintenance and the Visual Refresh Rate
Kosslyn's Cognitive Architecture
Language Colors Vision