Current Research 2011

William P. Seeley

Department of Philosophy

Bates College

Current research in the cognitive neuroscience of art lies at the confluence of two broad research strategies: experimental aesthetics and aesthetic experimentalism. Experimental aesthetics is a field of research in psychology that traces its roots to the founder of psychophysics, Gustav Fechner (1876). The central claim of this field of research is that we can learn about the nature of art and associated aesthetic experiences by using the methods of psychology and related fields to examine behavioral responses to artworks. The central claim of aesthetic experimentalism is that we can learn about the operations of perceptual systems by examining the productive strategies of artists (e.g. the way visual artists develop and use formal techniques to convey information in their works). A general model for the cognitive neuroscience of visual art emerges from the rapprochement between these two research strategies. Cognitive science, in its broadest sense, is the study of the way organisms acquire, recognize, manipulate, and use information for the production of behavior. Cognition can, in this context, be understood in terms of representational structures that encode information about the environment and computational processes that interpret and transform those structures. Artworks are abstract stimuli intentionally designed to trigger ordinary perceptual, affective, and cognitive responses in spectators (e.g. realistic paintings are 2-D representations of 3-D scenes and objects). Questions about the production, understanding, and appreciation of art are, therefore, in part questions about the way spectators, listeners, and readers acquire, represent, and transform information encoded in artworks in order to recognize, categorize, and evaluate their content (e.g., how does a painting convey its content; what components of its formal structure are critical to its performing this function; how does an artist determine what these components are; what do the answers to these questions teach researchers about the structure of the visual system, the nature of object recognition, the relative roles played by memory and attention in perception, or the processes underlying our affective responses to natural stimuli). Therefore, artworks can be used in the study of a broad range of cognitive phenomena and the results of this research can contribute to debates within cognitive science and the philosophy of mind about the structure of cognition and the nature of mental states. Results from these studies can, in turn, contribute data to help clarify difficult concepts and adjudicate between competing theories in philosophy of art. Therefore, the relationship between cognitive science and aesthetics is a two way street. Finally, this research model is not reductive in the sense the term is ordinarily used in the philosophy. Rather it is explanatory. It is an attempt to use research from one scientific domain to clarify assumptions in another.

1. Hearing how smooth it looks: selective attention and cross-modal perception in the arts.

A broad range of behavior is associated with cross-modal perception in the arts: we hear the sounds of a movie emanating from the location of the visual event it depicts; the perceived duration of a note we hear is influenced by the gesture with which we see it performed; motor skill and perceived task difficulty can influence the way we hear music and see the spatial layout of events depicted in pictures; and we sometimes report that the music we hear feels sad or exuberant. Philosophical explanations of cross-modal perception often make reference to neuroscientific discussions of multisensory integration in selective attention (Spence & Driver, 2004). This research demonstrates that superior colliculus plays a regulative role in attention, integrating unique modality specific visual, auditory, and somatosensory spatial maps into a common spatial framework for action, and that motor skill, emotional salience, and the semantic salience contribute to the integration of auditory, visual, and somatosensory information in ordinary perceptual contexts. I present a model for multisensory integration in our engagement with artworks derived from a diagnostic recognition framework for object recognition (Schyns, 1998) and a biased competition model for selective attention (Desimone and Duncan, 1995). The proposed model attributes a role to superior colliculus in a broader fronto-parietal attentional network that integrates sensory information, primes perceptual systems to the expectation of stimulus features salient to particular tasks (e.g. visual search, object identification, or reaching and grasping) at particular locations, and inhibits the perception of task irrelevant distracters. I argue that this model demonstrates cross-modal effects are the rule not the exception in perception and discuss ways in which it explains a range of cross-modal effects in our perceptual engagement with visual artworks and musical compositions.

2. Art, meaning, and perception: a question of methods for a cognitive neuroscience of art.

Neuroscience of art might give us traction with aesthetic issues. However it can be seen to have trouble modeling the artistically salient semantic properties of artworks. This is a particularly trenchant problem for perceptual media, e.g. dance, music, and painting. So if meaning really matters…and it does…even in aesthetic contexts…the prospects for this nascent field are dim. The issue boils down to a question of whether or not we can get a grip on the kinds of constraints present and available to guide interpretive behavior in our engagement with artworks. I argue that biased competition models of selective attention can be used to solve this problem, generalize to the affective content of our responses to artworks, and so show that research in cognitive neuroscience is germane to the types of problems of interest within the philosophy of art. Along the way I will use this case study to discuss a range of more general methodological issues that confront any attempt for a rapprochement between neuroscience and philosophy of art.

3. Looking at Mona Lisa: a psychophysiological study.

It is often asserted that the aesthetic qualities of the Mona Lisa include the dynamic qualities of her expression. Margaret Livingston has argued that this is, in a sense, true (Livingstone, 2002). Leonardo used a technique called sfumato to render the expression of the figure in the painting, the corners of her mouth and eyes. Sfumato is a formal technique in which artists blur the sharp edges that define object features in a painting so that these boundaries disappear into the broad contours of soft, “smoky” shadow. Livingstone filtered a reproduction of Leonardo’s painting in order to separate out the low, middle, and high spatial frequency information used to depict Mona Lisa’s face. The sfumato contours that define Mona Lisa's smile were more apparent in the images representing low and middle spatial frequency information in the painting than the sharp lines of the high spatial frequency image. Therefore, critical formal features defining Mona Lisa's smile are depicted only in low and middle spatial frequency information. The spatial resolution of human vision decreases dramatically as one moves from the center of the visual field towards the periphery. This difference in spatial resolution between central, or foveal, and peripheral vision is explained by the fact that the receptive fields of peripheral retinal neurons are dramatically larger than those of their foveal counterparts. The result is that foveal neurons are sensitive to sharp, narrow luminance boundaries that carry high spatial frequency information, but are unable to register coarse, broad luminance gradients, like contours rendered in sfumato, that carry low and medium spatial frequency information. Conversely, the wider receptive fields of neurons in the peripheral field are well suited to record the latter category of contours, but are nearly blind to high spatial frequency information. Livingstone argues, as a result, that when one foveates on, or directs one’s attention to, Mona Lisa’s smile it disappears. However, the smile reappears in a viewer’s peripheral field when he or she looks away. Livingstone argues that Mona Lisa’s expression varies systematically with the eccentricity of a viewer’s gaze from the center of the painting. This suggests a way to test her theory about our engagement with the painting. Facial EMG studies using the International Affective Picture System have demonstrated that we mimic the expressions depicted in photographs of faces (Land, Greenwald, Bradley, and Ham. 1993). These studies have demonstrated that the perception of pictures depicting angry, fearful, or happy faces produces a similar facial pattern in participants, e.g. enhanced corrugator (frown) and zygomatic (smile) muscle activity for angry/fearful versus happy faces respectively. We hypothesize as a result that, if Livingstone’s theory is sound, one ought to find measurable changes in zygomatic activity a viewer’s gaze varies in eccentricity from Mona Lisa’s smile.

4. Capoeira & Clint Eastwood: two case studies for the methodological utility of art in neuroscience.

Current research in the cognitive neuroscience of visual art lies at the confluence of two broad research strategies: empirical aesthetics and aesthetic experimentalism (Carroll, Moore, and Seeley, forthcoming; Rollins, 2004). The central claim of empirical aesthetics is that the methods of psychology and related fields can be used to enhance our understanding of the nature of art and associated behavior. Aesthetic experimentalism is derived from the observation that artists’ productive strategies emerge from systematic explorations of the perceptual effects of different sets of medium specific formal cues (Gombrich, 1960). The central claim of aesthetic experimentalism is that we can learn about the operations of perceptual systems by examining the productive strategies of artists (e.g. the way visual artists develop and use formal techniques to convey information in their works). The conjunction of these two research strategies suggest that the relationship between the philosophy of art and cognitive neuroscience is a two way street. This claim is most often made in regards to the visual arts, and canonically for the works of naturalist painters who aim for realistic depictions of landscapes, events, and people. However, in these cases artworks are most often used to illustrate the contributions of underlying neurophysiological mechanisms to perception, not as novel stimuli in experiments (Cavanaugh, 2005; Zeki, 1999). In this paper I examine two domains in which artworks are actively used as stimuli for experimental investigations in cognitive neuroscience: the use of dance in studies of the relationship between perception and action (Calvo-Merino et al, 2005; Calvo-Merino et al, 2006; Cross et al, 2006; Cross et al, 2009; Urgesi et al, 2007) and the use of Hollywood films to study the degree to which perceptual responses to ordinary events are synchronized across agents in natural contexts (Hasson et al 2008; Hasson et al, 2004). I argue that these case studies demonstrate the potential for a genuine rapprochement between philosophy and neuroscience in the study of art and cognition.

5. Imagination, picture perception, & narrative understanding.

a. Imagining Crawling Home. Philosophical accounts of narrative fiction can be loosely divided into two types. Participant accounts argue that some sort of simulation, or first-person perspective taking plays a critical role in our engagement with narratives. Observer accounts argue to the contrary that we primarily engage narrative fictions from a third-person point of view, as either outside observers or side participants to the depicted events. Recent psychological research suggests a means to evaluate this debate. The perception of distance and slope is influenced by the energetic (e.g., task difficulty) and emotional (e.g., anxiety) costs of actions (Proffitt, 2006). These effects are limited to increases in the costs of actions agents intend to perform themselves, generalize to cases where participants imagine acting, and demonstrate a role for tacit motor simulation in action planning. If the participant account is sound, one should, therefore, find similar effects across changes in the interpretation of the costs of actions depicted in static images. We asked people to copy the rough spatial layout of two paintings, Andrew Wyeth’s Christina’s World and Winter, 1946, across different interpretations of the costs of the actions they depict. We predicted that increasing costs would cause participants to draw distances as longer and hills as steeper. Our results confirm this prediction for the energetic, but not the emotional, costs of actions (although participants did expand the extent of the landscape in both contexts). In this paper we present the results of our study and argue that they demonstrate a significant role for simulation and first-person perspective taking in narrative understanding. Future directions involve using a larger range of images and stories to more carefully control for the effects of variance in the energetic costs of depicted actions. These results can also contribute to discussions of the role motor simulation plays in our general cognitive economy and debates about mental representation and embodied cognition in the philosophy of mind. This study was published as “Imagining Crawling Home: A Case Study in Cognitive Science and Aesthetics,” in Review of Philosophy and Psychology 1(3), 2010: 407-426

b. Whose Effectors, Your or Mine? Identification, Interpretation and Motor Simulation in Narrative Fiction. Energetic costs effects in slope and distance perception are effector specific, or limited to energetic costs for the particular muscle groups associated with the joints involved in anticipated actions. For instance, energetic costs associated with walking do not affect distance assessments in throwing contexts, energetic costs associated with throwing do not affect distance assessments in walking contexts, and energetic cost effects associated with anticipating reaching to a target with one’s right hand are not affected by the concurrent task of manipulating a keyboard with one’s left hand (Witt & Proffitt, 2008; Witt, Proffitt, & Epstein, 2004). These results are consistent with evidence that motor simulation selectively activates areas of the premotor cortex associated with the muscle groups that would be used to perform the target action and can contribute to the debate between participant and observer accounts of narrative understanding. There is a discrepancy between the global perspectives of spectators/readers and the local perspectives of characters within a narrative. Characters lack the global perspective of spectators and readers. Characters, unlike spectators and readers are as a result neither apprehensive nor hopeful about the probabilities of future events in the story. These phenomenal differences between the depicted experiences of characters and the real experiences of spectators and readers challenge the claim that we simulate the behavior of characters from a first person perspective (Carroll, 1997; Kieran, 2003). However, the effector specificity of motor simulation suggests that we need not project ourselves into a character in order to simulate their perspective. Rather, we hypothesize that we use our own cognitive processes to model discrete aspects of a characters’ depicted behavior when this information would contribute our understanding of their behavior. This would enable us to continue to maintain a strong sense of self while simulating the psychological perspectives of characters, as we do in ordinary contexts while simulating the behavior of others. Future directions involve generating a range of image sets and energetic costs stories to enable us to use the copying protocol described above to test this hypothesis.

c. Motor skill as a mediating variable in the effects of energetic costs on apparent egocentric distance. Jessica We predicted that, if effector specific motor simulation is the controlling mechanism, variations in motor skill, which influence task difficulty, should also affect apparent egocentric distance. We are currently conducting two experiments to test this hypothesis. In the first experiment (Ofeldt & Seeley) participants use perceptual matching measures and verbal assessments to make distance judgments about a target in three conditions: a) prior to reaching with scissors to make a small precision cut across the target with their dominant or non-dominant hands while holding the scissors, b) prior to reaching with scissors to make a small precision cut across the target with their dominant and non-dominant hands with the scissors beside them on the table, and c) while reaching and imagining using scissors to make a small precision cut across the target with their dominant and non-dominant hands (with the scissors on the table beside them). The procedures for the second experiment (Kahan & Seeley) are the same as the first except that participants anticipate or imagine reaching with a long pencil, held at the top near the eraser, to write a short word across the target with either their dominant or non-dominant hands. We predict that distances will appear longer when participants imagine or anticipate reaching to perform the task with their non-dominant hands.

d. What happens in a book stays in a book? Motor Simulation and Narrative Understanding in Literature.

It has been argued that, whereas motor simulation might play a role in the perception and understanding of visual narratives, it is unnecessary in literary contexts. Why? In these cases dialogue and narration are used as formal devices to explicitly tell the reader what he or she needs to know about a character’s motivations and dispositions. However, recent research in cognitive neuroscience has demonstrated that the premotor areas involved in motor preparation, action observation, and motor simulation play a role in the comprehension of action sentences (Fisher & Zwaan, 2008), and that these processes generalize to our understanding of actions depicted in literary contexts (Speer, Reynolds, Swallow, & Zacks, 2009). In this paper we review this research and argue it entails that our participant account model for energetic and emotional costs effects in picture perception generalizes to narrative contexts in literature.

6. Movement, gesture, and meaning: a sensiromotor model for audience engagement with dance.

The neuroscience of dance is a growing field of genuine interdisciplinary collaboration. The goal of this paper is to evaluate differences in the approaches to dance taken by neuroscientists and psychologically oriented philosophers. Studies of dance in neuroscience of art have focused on correlations between affective responses and preference ratings among viewers (Calvo-Merino et al 2009). Philosophers of art draw a distinction between these types of subjective aesthetic responses and questions about interpretation, including questions about the ways artists' use formal strategies to express emotions and ideas (Carrol and Banes, 1982). Barbara Montero (2006) and Jonathan Cole (Cole & Montero, 2006) have argued that dance recruits proprioception (i.e. awareness of the relative orientation of our limbs within our body space) as an aesthetic sense. This model for aesthetic responses to dance suggests that the formal strategies used by choregraphers to express emotions and ideas harness our natural tendency to simulate the observed actions of others (e.g. Kate Weare). However, expertise effects in action understanding (Calvo-Merino, 2005, 2006) suggest that the phrases employed in dance may be a difficult to understand when they are interpreted simply as abstract choreographed movements. I argue that narrative devices (e.g. Bill T. Jones), conventional formal /compositional devices (e.g. George Balanchine or Merce Cunningham), and a general knowledge of artistic conventions (e.g. Yvonne Rainer) play a critical role in dance, enabling audiences to contextualize abstract choreographed movements as actions with intentional (cognitive) and expressive (emotional) content.

7. What's driving the bus: embodied appraisals, direct perception, and organismic control.

Jesse Prinz, in his discussion of embodied appraisals, marks a distinction between direct perception and perception that is under cognitive control. This distinction is spelled out as a distinction between perceptual events that are under either direct environmental control or the control of top down neurophysiological processes ordinarily associated with executive function. Prinz argues that embodied appraisals are an example of direct perception. I argue that recent research demonstrates that, in ordinary contexts, the perception of affectively salient stimuli requires endogenously directed attention (Pessoa & Adolphs, 2010; Pessoa, Kastner, & Ungerleider, 2002). Endogenous control of attention is a canonical example of executive function, cognitive process mediated by top down processes neurophysiological processes. Therefore the distinction driving Prinz's articulation of embodied appraisals and the more general distinction between cognitivist and non-cognitivist theories of emotion needs to be re-visited.

Bibliography:

Adolphs, R. (2004). Emotional vision. Nature Reviews Neuroscience, 7(11), 1167-1168.

Adolphs, R. and L. Pessoa (2010). Emotion Processing and the Amygdala: from a low road to many roads of evaluating biological significance. Nature Reviews Neuroscience 11: 773-783.

Beardsley, M. (1982). What is going on in dance? Dance Research Journal, 15(1), 31-36.

Bonnar, L., Gosselin, F., and Schyns, P G. (2002). Understanding Dali’s Slave Market with Disappearing Bust of Voltaire: a case study in the scale information driving perception. Perception, 31, 683- 691.

Calvo-Merino B. Jola C, Glaser D, Haggard P (2008) Towards a sensorimotor aesthetics of performing art. Consciousness and Cognition, 17(3), 911-922.

Calvo-Merino, B., Grèzes, J., Glaser, D. E., Passingham, R. E., Haggard, P. (2006). Seeing or doing? Influence of visual and motor familiarity in action observation. Current Biology, 16(19), 1905-1910.

Calvo-Merino, B., Glaser, D. E., Grèzes, J, Passingham R. E, & Haggard, P. (2005). Action observation and acquired motor skills: an fMRI study with expert dancers. Cerebral Cortex,15, 1245-1249.

Carr, L., Iacoboni, M., Dubeau, M. C., Mazziotta, J. C., & Lenzi, G. L. (2003). Neural mechanisms of empathy in humans: a relay from neural systems for imitation to limbic areas. Proceedings of the National Academy of Science, 100, 5497-5502.

Carroll, N. (1999). The Philosophy of Art (new York: Routledge).

Carroll, N. (1997). Simulation, emotions, and morality. Beyond Aesthetics (pp. 306-317). New York: Cambridge University Press, 2001.

Carroll, N., Moore, M. & Seeley, W. P. (2010). The philosophy of art and aesthetics, psychology, and neuroscience: studies in literature, visual arts, and music. In Arthur P. Shimamura & Stephen E. Palmer (Eds.), Aesthetic Science: Connecting Minds, Brains, and Experience. New York: Oxford University Press.

Chun, M. & Marois, R. (2002). The dark side of visual attention. Current Opinion in Neurobiology, 12(22), 1-6.

Cross, E. S., Hamilton, A. F. de C., Kraemer, D. J. M, Kelley, W. M. & Grafton, S. T. (2009). Dissociable substrates for body motion and physical experience in the human action observation network. European Journal of Neuroscience, 30, 1383-1392.

Cross, E. S., Hamilton, A. F. de C., & Grafton, S. T. (2006). Building a motor simulation de novo: observation of dance by dancers. NeuroImage, 31(3), 1257-1267.

Fisher, M. H. & Zwaan, R. A. (2008). Embodied language: a review of the role of the motor system in language comprehension. The Quarterly Journal of Experimental Psychology, 61(6), 825-850.

Gallese, V., Goldman, A. (1998). Mirror neurons and the simulation theory of mind reading. Trends in Cognition Science, 2, 493–501.

Hamilton, A.F. & Grafton, S.T. (2006) Goal representation in human anterior intraparietal sulcus. The Journal of Neuroscience, 26, 1133–1137.

Hasson, U. Landsman, O., Knappmeyer, B., Vallines, I., Rubin, N.,  Heeger. D. J. (2008). Neurocinematics: The neuroscience of making movies. Projections: The Journal for Movies and Mind, 1(2): 2-26.

Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., and Malach, R. (2004). Intersubjective synchronization of cortical activity during natural vision. Science 303, 1634-1640.

Iacoboni, M. & Dapretto, M. (2006) The mirror neuron system and the consequences of its dysfunction. Nature Reviews Neuroscience, 7, 942–951.

Kastner, S. (2004). Attentional response modulation in the human visual system. In ed. Michael Posner, The Cognitive Neuroscience of Attention (New York: The Guilford Press), 144-156.

Kieran, M. (2003). In search of a narrative. Kieran, M, and Lopes, D. M. (eds.) Imagination, Philosophy, and the Arts (pp. 69-87). New York: Routledge.

Koivisto, M. & Revonsuo, A. (2007). How meaning shapes seeing. Psychological Science 18(10), 845-849.

Land, P. J., Greenwald, M. K., Bradley, M. M., and Hamm, A. O. (1993). Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30: 261-273.

Langer, S. (1953). Feeling and Form. New York: Charles Scribner’s Sons.

Livingston, M. S. (2002) Vision and Art: The Biology of Seeing. New York, New York: Harry N. Abrams.

Montero, B. (2006). Proprioception as an aesthetic sense. Journal of Aesthetics and Art Criticism, 64(2), 231-142.

Pessoa, L. S. Kastner, and L. G. Ungerleider (2002). Attentional Control of the Processing of Neutral and Emotional Stimuli. Cognitive Brain Research 15: 31-45

Prinz, J. (2004). Gut Reactions (New York: Oxford University Press).

Proffitt, D. (2006). Embodied perception and the economy of action. Perspectives in Psychological Science, 1(2), 110-122.

Rollins, R. (2004). What Monet meant: intention and attention in understanding art. Journal of Aesthetics and Art Criticism, 62(2), 2004, 175-188.

Schyns, P. (1998). Diagnostic recognition. Cognition, 67(1-2), 147-179
Seeley, W. P. (2010). Imagining crawling home: a case study in cognitive science and aesthetics. Review of Philosophy: Psychology and Experimental Philosophy.

Seeley, W. P. & Kozbelt, A. (2008). Art, artists, & perception: a model for premotor contributions to perceptual analysis and form recognition. Philosophical Psychology, 21(2), 149-171.

Shiffrar, M. & Pinto, J. (2002). The visual analysis of bodily motion. In eds. W. Prinz & B. Hommel, Common Mechanisms in Perception & Action (New York: Oxford University Press), 2002.

Speer, N. K., Reynolds, J. R., Swallow, K. M., & Zacks, J. M. (2209). Reading stories activates neural representations of visual and motor experiences. Psychological Science, 20(8), 989-999.

Witt, J. K. & Proffitt, D. R. (2008).Action-specific influences on distance perception: a role for motor simulation. Journal of Experimental Psychology: Human Perception and Performance, 34(6), 1479-1492.

Witt, J. K., Proffitt, D. R., & Epstein, W. (2004). Perceiving distance: a role of effort and intent. Perception 33(5), 570-590.

Urgesi, C., Calvo-Merino, B., Haggard, P., Aglioti, S. M. (2007). Transcranial magnetic stimulation reveals two cortical pathways for visual body processing. Journal of Neuroscience, 27(30), 8023-8030.

Vuilleumier, P. (2005). How brains beware: neural mechanisms of emotional attention. TRENDS in Cognitive Sciences, 9(12), 585-594.

Zatorre, R. J., Evans, A. C. & Meyer, E. (1994). Neural mechanisms underlying melody perception and memory for pitch. The Journal of Neuroscience, 14(4), 1908-1919.