Metaphor processing is supramodal semantic processing: The role of the bilateral lateral temporal regions in multimodal communication

Gina Joue, Linda Boven, Klaus Willmes, Vito Evola, Liliana R. Demenescu, Julius Hassemer, Irene Mittelberg, Klaus Mathiak, Frank Schneider, Ute Habel

Research output: Contribution to journalArticle

Abstract

This paper presents an fMRI study on healthy adult understanding of metaphors in multimodal communication. We investigated metaphors expressed either only in coverbal gestures (“monomodal metaphors”) or in speech with accompanying gestures (“multimodal metaphors”). Monomodal metaphoric gestures convey metaphoric information not expressed in the accompanying speech (e.g. saying the non-metaphoric utterance, “She felt bad” while dropping down the hand with palm facing up; here, the gesture alone indicates metaphoricity), whereas coverbal gestures in multimodal metaphors indicate metaphoricity redundant to the speech (e.g. saying the metaphoric utterance, “Her spirits fell” while dropping the hand with palm facing up). In other words, in monomodal metaphors, gestures add information not spoken, whereas the gestures in multimodal metaphors can be redundant to the spoken content. Understanding and integrating the information in each modality, here spoken and visual, is important in multimodal communication, but most prior studies have only considered multimodal metaphors where the gesture is redundant to what is spoken. Our participants watched audiovisual clips of an actor speaking while gesturing. We found that abstract metaphor comprehension recruited the lateral superior/middle temporal cortices, regardless of the modality in which the conceptual metaphor is expressed. These results suggest that abstract metaphors, regardless of modality, involve resources implicated in general semantic processing and are consistent with the role of these areas in supramodal semantic processing as well as the theory of embodied cognition.
Original languageEnglish
Article number104772
JournalBrain and Language
Volume205
DOIs
Publication statusPublished - 1 Jun 2020

Keywords

  • Cognitive linguistics
  • Coverbal gestures
  • fMRI
  • Metaphor
  • Monomodal metaphors
  • Multimodal metaphors

Fingerprint Dive into the research topics of 'Metaphor processing is supramodal semantic processing: The role of the bilateral lateral temporal regions in multimodal communication'. Together they form a unique fingerprint.

  • Cite this