Pictures in the Brain
The new issue of Scientific American Mind has a nice recap on the mental imagery debate, which has been going strong for over 10 years now. The debate proceeds roughly as follows: when we imagine something, are we merely activating its abstract, propositional representation in long term memory, or is imagination actually a process of neurally reinstantiating this distributed information into a visual form available for inspection and spatial manipulation?Though this latter "depictive" theory may seem far-fetched, much experimental evidence by Kosslyn and others supports the idea that we use the same neural machinery in mental imagery as we do in direct visual experience with the world. For example, when subjects are asked to imagine a rabbit next to a flea, they are much quicker to answer questions pertaining to small visual deatils on the rabbit (such as whiskers) than other subjects who had imagined the rabbit next to an elephant; this result suggests that the inspection of mental images requires a "zooming" process that is scale sensitive, and not simply propositional or code-like.
A similar result occurs when subjects are asked to approach an elephant or a smaller object until that image begins to "overflow" from their frame of reference; the imagined distance at which subjects stop is remarkably consistent with the real distance at which such an image would eclipse an observer's field of view. Likewise, when subjects are asked to study a figure consisting of multiple features separated in space, and then to identify specific features on that figure from memory alone; reaction times are consistent with the distance between features, as though some "attentional spotlight" must physically shift from one location to another in the mind's representation.
Neuroimaging evidence also supports the claim that mental imagery actually projects to many of the same anatomical areas we use for sensory perception. Patients with hemispheric neglect also show this same pattern of results; those with right parietal lesions are impaired both at orienting to visible stimuli in the left hemifield, as well as in orienting to imagined stimuli in that hemifield.
Yet the conclusions drawn from this research is hotly debated by researchers such as Zenon Pylyshyn, who believe that the behavioral evidence reflects both "tacit knowledge" on the part of the subjects as well as experimenter/task bias. Pylyshyn argues that subjects may understand the task as being to actively simulate visual perception, and hence they produce reaction times that are consistent with visual perception though these are merely artifacts of propositional encodings of features and relations between them. Pylyshyn also points to experiments, described in the SciAmMind piece, which seem to support this artifactual view of imagery scanning times. Finally, several visual illusions (such as the necker cube) and other visual phenomena are not seen in mental imagery; this poses a problem for theories which invoke the same neural machinery in imagery as in perception.
Who is right? Get a copy of Pylyshyn and Kosslyn's heated exchange (in which Pylyshyn calls Kosslyn's theory "grotesque," and Kosslyn calls Pylyshyn "nihilistic") in March 2003's Trends in Cognitive Sciences, and see for yourself. Like most theoretical schisms in cognitive psychology, the answer is likely to begin with the phrase "it depends" - but the precise nature of this dependency is both a far deeper and much less explored question.
Related Posts:
An Informal Integration of Object Recognition Models
The Attentional Spotlight
Mind's Eye: Models of the Attentional Blink
False Promise of View Invariance
posted by Chris Chatham at 2:24:00 PM. Now go to the current Developing Intelligence blog!
3 Comments:
The debate about the cognitive mechanisms underlying mental imagery has been going strong for a lot more than 10 years. It dates back at least to Pylyshyn's first paper on the subject in 1973 (Pylyshyn, 1973), and Kosslyn got involved very soon after that.
Anyway, I agree that the Scientific American Mind article does a useful service in introducing the debate about mental imagery to a wider audience. However, it also does a disservice, by further propagating the very persistent meme that there are only two possible (or known) theoretical approaches to mental imagery: either they are in some sense pictures in the mind/brain (as Kosslyn holds), or they are descriptions in some sort of broadly language-like symbolic representation system (as Pylyshyn holds). I know this is what virtually all the textbooks say (I know of one honorable exception - Morris & Hampson, 1983), but it is simply not true. There is at least one other type of theory of imagery, what I call Perceptual Activity theory, that has been in the literature at least as long as description theory has (proposed, in various forms, by some very well respected theorists, such as Hebb (1968), Hochberg (1968), Sarbin (1972), and Neisser (1976, 1978a)). More recently, quite a bit of evidence has emerged that supports it directly (e.g., Ruggieri & Alfieri, 1992; Brandt & Stark, 1997; Demarais & Cohen, 1998; Spivey & Geng, 2001; Laeng & Teodorescu, 2002; Bartolomeo & Chokron, 2002; Bensafi et al., 2003; de'Sperati, 2003; Johansson et al., 2005 - see Thomas, 2006, for further references and discussion), but has still never been given anything like the sort of attention that the picture and description theory have received. Most cognitive scientists seem to be simply unaware of it, and, if they have actually read any of the relevant papers they force them into the Procrustean framework of the so called analog-propositional (picture vs. description) debate, misconstruing them as being weird and confused versions of either picture or description theory. Indeed, both Kosslyn (Kosslyn & Thompson, 2003) and Pylyshyn (2002) have recently attempted to assimilate my version of Perceptual Activity theory (Thomas, 1999, 2002) to their own positions, even though I quite explicitly reject, and argue strongly against, both of their theories.
In fact, as I see it, Kosslyn's and Pylyshyn's theories have lot more in common with each other than either does with Perceptual Activity theory. Both their theories depend on understanding perception as, fundamentally, a bottom-up driven process whereby energies impinging on the sense receptors are transformed into inner representations, and (notwithstanding the neuroscientific turn in Kosslyn's more recent work), they both come out of the tradition that understands cognition as being almost entirely a matter of computational symbol manipulation. By contrast, Perceptual Activity theory depends upon an understanding of perception as a mainly top-down driven process of active exploration of the environment (as urged, for example, by O'Regan & Noë, 2001; Noë, 2004), and belongs in the theoretical camp of situated or embodied cognition (e.g. Clancey, 1997), autonomous (Brooks, 1991) and "active/animate vision" robotics (e.g. Bajcsy, 1988; Ballard, 1991; Blake & Yuille, 1992; Fitzpatrick, 2003; Triesch et al., 2003)
To the best of my knowledge, there have been virtually no attempts made to criticize or refute Perceptual Activity theory (again, the honorable exception is Hampson & Morris (1978), but see Neisser's (1978b) reply). It has not been discussed on its merits, but simply ignored. So far as I can tell, this is not because it is perceived to suffer from any obvious conceptual or empirical defects (at least, no worse than its better known rivals) but because of historical circumstances. It emerged at just about the time (in the late1960s and the 70s) that the symbolic computation paradigm was taking the world of cognitive theory by storm. The Perceptual Activity theory of imagery was at odds with that paradigm, whereas Kosslyn's and Pylyshyn's theories were explicitly designed to fit it. Thus, the heated "analog-propositional debate" of the 70s (the continuation of which is the subject of the Sci Am Mind article) was not really a debate about mental imagery as such, but about what account of imagery would fit best into the symbolic computation framework. Since the 1980s, symbolic computationalism no longer dominates cognitive theory the way it once did, but the fame of the "analog-propositional debate" seems to have got the idea firmly stuck in people's heads that only these two sorts of theory (or, as is occasionally suggested, some sort of hybrid of them) are available to be considered. The idea is really very hard to shift. I have had students of mine, whom I have explicitly told that, despite what they will read in many places, there are not two but three contending theories of imagery (and for whom I have set readings to back up the point), still write papers for me saying that there are just two theories (i.e., analog or propositional).
It is, of course, in the interests of Kosslyn and Pylyshyn to keep the "two-and-only-two-imagery-theories" meme alive. Their careers have been built upon this unending battle between the two allegedly exclusive alternatives. As I see it, however, the battle keeps going only because, when the alternatives are restricted to pictures vs. descriptions, neither side can be proved right because both are wrong! Certainly both face severe empirical and conceptual challenges, not only those that the other side of the long debate have pointed out, but also some others that both sides prefer to keep quiet about (Thomas, 1999, 2002). I am not saying, of course, that Perceptual Activity theory cannot possibly be wrong too (perhaps some fourth type of theory, that no-one has thought of yet, is the true one), but the case against it has yet to be made, and as things stand it seems to be broadly compatible with the evidence from both cognitive psychology and recent neuroscience (Bartolomeo, 2002; Thomas, 1999; Thomas, 2006; Kosslyn & Thompson, 2003). [I say "broadly" because many of the fine details of how Perceptual Activity theory would deal with all the empirical facts remain to be worked out. That sort of thing can only really emerge when a theory is developed in response to detailed criticism, which is still lacking. What I am saying is that I am not aware of any difficulties that look insurmountable, and no-one else, as yet, is pointing any out.]
References
Bajcsy, R. (1988). Active Perception. Proceedings of the IEEE (76) 996-1005.
Ballard, D.H. (1991). Animate Vision. Artificial Intelligence (48) 57-86.
Bartolomeo P. (2002) the Relationship Between Visual Perception and Visual Mental Imagery: a Reappraisal of the Neuropsychological Evidence. Cortex (38) 357-78. [Available from the archive at the Cortex site ]
Bartolomeo, P. & Chokron, S. (2002). Can We Change our Vantage Point to Explore Imaginal Neglect? Behavioral and Brain Sciences (25) 184-185.
Bensafi, M., Porter, J., Pouliot, S., Mainland, J., Johnson, B., Zelano, C., Young, N., Bremner, E., Aframian. D., Khan, R., & Sobel, N. (2003). Olfactomotor Activity During Imagery Mimics That During Perception. Nature Neuroscience (6) 1142 - 1144.
Brandt, S.A. & Stark, L.W. (1997). Spontaneous Eye Movements During Visual Imagery Reflect the Content of the Visual Scene. Journal of Cognitive Neuroscience (9) 27-38.
Brooks. R.A. (1991). Intelligence Without Representation. Artificial Intelligence (47) 1991 139-159.
Clancey, W. J. (1997). Situated Cognition: on Human Knowledge and Computer Representations. Cambridge: Cambridge University Press.
Demarais, A.M & Cohen, B.H. (1998). Evidence for Image-Scanning Eye Movements during Transitive Inference. Biological Psychology (49) 229-247.
de’Sperati, C. (2003). Precise Oculomotor Correlates of Visuospatial Mental Rotation and Circular Motion Imagery. Journal of Cognitive Neuroscience (15) 1244–1259.
Fitzpatrick, P. (2003). First Contact: An Active Vision Approach to Segmentation. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vagas, Nevada, October 27 - 31, 2003. [Online: http://www.ai.mit.edu/projects/humanoid-robotics-group/publications.html]
Hampson, P.J. & Morris, P.E. (1978). Unfulfilled Expectations: a critique of Neisser's theory of imagery. Cognition (6) 79-85.
Hebb, D. O. (1968). Concerning Imagery. Psychological Review, 75, 466-477.
Hochberg, J. (1968). In the Mind's Eye. In R. N. Haber (Ed.), Contemporary Theory and Research in Visual Perception (pp. 309-331). New York: Holt Rinehart & Winston.
Johansson, R., Holsanova, J., & Holmqvist, K. (2005). What Do Eye Movements Reveal About Mental Imagery? Evidence From Visual and Verbal Elicitations. In Bara, B. G., Barsalou, L., Bucciarelli, M. (Eds.), Proceedings of the 27th Annual Conference of the Cognitive Science Society [Stressa, Italy, July 21-23, 2005], pp. 1054-1059. Mahwah, NJ: Erlbaum. [See also this poster.]
Kosslyn, S.M. & Thompson, W.L. ( 2003). When is Early Visual Cortex Activated During Visual Imagery? Psychological Bulletin, 129, 723-746.
Laeng, B. & Teodorescu, D.-S. (2002). Eye Scanpaths During Visual Imagery Reenact those of Perception of the Same Visual Scene. Cognitive Science (26) 207-231. [Online at http://www.sv.uit.no/seksjon/psyk/pdf/laeng/Laeng&Teodorescu.pdf]
Morris, P. E., & Hampson, P. J. (1983). Imagery and Consciousness. London: Academic Press.
Neisser, U. (1976). Cognition and Reality. San Francisco: Freeman.
Neisser, U. ( 1978a). Perceiving, Anticipating and Imagining. Minnesota Studies in the Philosophy of Science (9) 89-106.
Neisser, U. (1978b). Anticipations, Images and Introspection. Cognition (6) 167-174.
Noë, A. (2004). Action in Perception. Cambridge, MA: MIT Press.
O'Regan, J. K., & Noë, A. (2001). A Sensorimotor Account of Vision and Visual Consciousness. Behavoral and Brain Sciences (24) 939-973.
Pylyshyn, Z. W. (1973). What the mind's eye tells the mind's brain: A critique of mental imagery. Psychological Bulletin, 80, 1-25.
Pylyshyn, Z.W. (2002). Stalking the Elusive Mental Image Screen. Behavioral and Brain Sciences (25) 216-237.
Ruggieri, V. & Alfieri, G. (1992). The Eyes in Imagery and Perceptual Processes: First Remarks. Perceptual and Motor Skills (75) 287-290.
Sarbin, T.R. ( 1972). Imagination as Muted Role Taking. In P.W. Sheehan (ed.) The Function and Nature of Imagery (pp. 333-354). New York: Academic Press.
Spivey, M.J. & Geng J.J. (2001). Oculomotor Mechanisms Activated by Imagery and Memory: Eye Movements to Absent Objects. Psychological Research (65) 235-241.
Thomas, N.J.T. ( 1999). Are Theories of Imagery Theories of Imagination? An Active Perception Approach to Conscious Mental Content. Cognitive Science (23) 207-245.
Thomas, N.J.T. (2002). The False Dichotomy of Imagery. Behavioral and Brain Sciences, 25, 211.
Thomas, N.J.T. (2006). New Support for the Perceptual Activity Theory of Mental Imagery. Unpublished. Online at http://www.calstatela.edu/faculty/nthomas/newsupa.htm.
Triesch, J., Ballard, D.H., Hayhoe, M.M., & Sullivan, B.T. (2003). What you see is what you need. Journal of Vision, 3, 86-94. Online serial, URL: http://journalofvision.org/3/1/9/.
What actually is the Perceptual Activity theory of imagery?
After making that huge long comment (sorry it was so long), I realize that I failed, in the end to explain just what the Perceptual Activity theory of imagery is actually saying. In a nutshell the idea is that perception (as O'Regan & Noë hold, together with various "active vision" roboticists, and, I believe, J.J. Gibson properly understood) is, at root, a matter of active, purposeful exploration of the environment, and imagery occurs when we (largely covertly) re-enact or go through the motions of such exploratory activity when the relevant object or scene is not actually present. For instance, if we are visually imagining a cat we are enacting the exploratory behaviors that would be brought into play in recognizing and visually examining a real cat if one were actually present. For both perception and imagery, the idea is that the experience comes from the actual enaction of the exploratory behavior, and not from the establishment of some inner representational end-product of perception. Some sort of internal representational structure is, no doubt, needed to guide the exploratory behavior appropriately, but it is not this that is experienced either as image or percept. For more detail, justification, and, I hope, clarity, see Thomas (1999), and other material on my web site: http://www.members.leeds.ac.uk/n.j.thomas70.
In fact, the re-enactment during imagery is not all so covert as I thought back in 1999, as the eye tracking etc. evidence cited above shows (Brandt & Stark, 1997; Demarais & Cohen, 1998; Spivey & Geng, 2001; Laeng & Teodorescu, 2002; Bensafi et al., 2003; de'Sperati, 2003; Johansson et al., 2005). Nevertheless, I am inclined to think that a lot of the relevant exploratory behavior, in both perception and imagery, may be entirely internal. During perceiving, executive structures in the brain are probably directing the exploration not only of the external world itself (via eye movements etc.) but also of what might be called "passive" (bottom-up driven) internal representations such as the current momentary state of the retinotopic maps in the early visual areas. Such exploration presumably consists in the "probing" of the detailed state of these maps via the many neurons that backproject to the early visual areas (Felleman & Van Essen, 1991). It is this sort of exploratory probing, I believe, that accounts for the activity sometimes (not always) observed in the early visual areas during imagery (Kosslyn et al., 2001; Sparing et al., 2002; Kosslyn & Thompson, 2003). However, the point remains that the visual or quasi-visual experience arises not from the mere presence of excitation patterns in early visual cortex, nor, indeed, from the mere presence of the higher level representations that guide the exploration, but from the actual exploratory behavior itself, from the "recurrent" rather than the "feedforward" processing (Lamme & Roelfsema, 2000; Pascual-Leone & Walsh, 2001).
Additional references (see previous comment for the rest):
Felleman, D. J. and Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex (1) 1-47.
Kosslyn, S.M., Ganis, G., & Thompson W.L. (2001). Neural Foundations of Imagery. Nature Reviews: Neuroscience (2) 635-642.
Lamme, V.A.F. & Roelfsema, P.R. (2000). The Distinct Modes of Vision Offered by Feedforward and Recurrent Processing. Trends in Neurosciences (23) 571-579.
Pascual-Leone, A. & Walsh, V. (2001). Fast Backprojections from the Motion to the Primary Visual Area Necessary for Visual Awareness. Science (292) 10-12.
Sparing, R., Mottaghy, F.M., Ganis, G., Thompson, W.L., Töpper, R., Kosslyn, S.M., & Pascual-Leone, A. (2002). Visual cortex excitability increases during visual imagery - a TMS study in healthy human subjects. Brain Research (938) 92-97.
This is fascinating stuff - thank you for taking so much time to explain the alternate theory of imagery.
I really like your approach, and although I'm not (yet) familiar with the eyetracking data you mention, I am familiar with other proposals for a more embodied view of vision/visual processing. Goodale & Humphreys 1998 cognition paper (The objects of action and perception) in which they advocate "a ‘duplex’ approach to high-level
vision suggests that Marrian or ‘reconstructive’ approaches and Gibsonian or ‘purposiveanimate-
behaviorist’ approaches need not be seen as mutually exclusive, but rather as complementary
in their emphases on different aspects of visual function."
Likewise, it seems related to Hommel et al.'s "Theory of Event Coding" in which they argue that "On the perceptual side, the dominant cognitive view largely underestimates,
and thus fails to account for, the impact of action-related processes on both the processing of perceptual information and
on perceptual learning."
Obviously there are differences - some very interesting ones - and I'll likely do a post on your 1999 paper pretty soon, in the context of new approaches to "embodied cognition."
Post a Comment
<< Home