Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension Using Distributional Semantic Models (2020)

Abstract: Recent years have seen a growing interest within the natural language processing (NLP) community in evaluating the ability of semantic models to capture human meaning representation in the brain. Existing research has mainly focused on applying semantic models to decode brain activity patterns associated with the meaning of individual words, and, more recently, this approach has been extended to sentences and larger text fragments. Our work is the first to investigate metaphor processing in the brain in this context. We evaluate a range of semantic models (word embeddings, compositional, and visual models) in their ability to decode brain activity associated with reading of both literal and metaphoric sentences. Our results suggest that compositional models and word embeddings are able to capture differences in the processing of literal and metaphoric sentences, providing support for the idea that the literal meaning is not fully accessible during familiar metaphor comprehension.

## introduction distributional semantics aims to represent the meaning of linguistic fragments as high-dimensional dense vectors. it has been successfully used to model the meaning of individual words in semantic similarity and analogy tasks @xcite ; as well as the meaning of larger linguistic units in a variety of tasks, such as translation @xcite and natural language inference @xcite . recent research has also demonstrated the ability of distributional models to predict patterns of brain activity associated with the meaning of words, obtained via functional magnetic resonance imaging (fmri) @xcite @xcite . following in their steps, @xcite have investigated visually grounded semantic models in this context. they found that while both visual and text-based models can equally decode concrete words, textbased models show an overall advantage over visual models when decoding more abstract words. other research has shown that data-driven semantic models can also successfully predict patterns of brain activity associated with the processing of sentences @xcite and larger narrative text passages @xcite . recently, @xcite investigated long short-term memory (lstm) recurrent neural networks and showed that semantic models that incorporate larger-sized context windows outperform those with smaller-sized context windows, as well as the baseline bagof-words model, in predicting brain activity associated with narrative listening. this suggests that compositional semantic models are sufficiently advanced to study the impact of linguistic context on semantic representation in the brain. in this paper, we investigate the extent to which lexical and compositional semantic models are able to capture differences in human meaning representations, resulting from meaning disambiguation of literal and metaphoric uses of words in context. metaphoric uses of words involve a transfer of meaning, arising through semantic composition @xcite . for instance, the meaning of the verb push is not intrinsically metaphorical; yet it receives a metaphorical interpretation when we talk about pushing agendas, pushing drugs, or pushing ourselves. theories of metaphor comprehension differ in terms of the kinds of processes (and stages) involved in arriving at the metaphorical interpretation, mainly whether or not the abstract meaning is indirectly accessed via processing the literal meaning first or directly accessible largely bypassing the literal meaning @xcite . to this extent, the role that access to and retrieval of the literal meaning plays during metaphor processing is often debated. on the one hand, metaphor comprehension involves juxtaposing two unlike things and this may invite a search for common relational structure through a process of direct comparison. inferences flow from the vehicle to the topic giving rise to the metaphoric interpretation @xcite . in a slightly different vein, @xcite suggest that metaphor comprehension involves systematic mappings (between a concrete domain onto another typically more abstract domain) that become established through co-occurrences over the course of experience. this draws on mental imagery or the re-activation of neural representations involved during primary experience (i.e., sensorimotor simulation) allowing appropriate inferences to be made. other theories, however, suggest that the literal meaning in metaphor comprehension may be largely bypassed if the abstract meaning is directly or immediately accessible involving more categorical processing @xcite . for example, the word used metaphorically could be immediately recognized as belonging to an abstract superordinate category of which both the vehicle and topic belong. alternatively, it has been suggested that more familiar metaphors involve categorical processing, while comparatively novel metaphor will involve initially greater processing of the literal meaning @xcite . to contribute to our understanding of metaphor comprehension, including the accessibility of the literal meaning, we investigate whether semantic models are able to decode patterns of brain activity associated with literal and metaphoric sentence comprehension, using the fmri dataset of @xcite . this dataset contains neural activity associated with the processing of both literal and familiar metaphorical uses of handaction verbs (such as push, grasp, squeeze, etc.) in the context of their nominal object. we experiment with several kinds of semantic models: (1) word-based models, namely, word embeddings of the verb and the nominal object; (2) compositional models, namely, vector addition and an lstm recurrent neural network; and (3) visual models, learning visual representations of the verb and its nominal object. this choice of models allows us to investigate: (1) the role of the verb and its nominal object (captured by their respective word embeddings) in the interpretation of literal and metaphoric sentences; (2) the extent to which compositional models capture the patterns of human meaning representation in case of literal and metaphoric use; and (3) the role of visual information in literal and metaphor interpretation. we test these models in their ability to decode brain activity associated with literal and metaphoric sentence comprehension, using the similarity decoding method of @xcite . we perform decoding at the whole brain level, as well as within specific regions implicated in linguistic, motor and visual processing. our results demonstrate that several of our semantic models are able to predict patterns of brain activity associated with the meaning of literal and metaphorical sentences. we find that (1) compositional semantic models are superior in decoding both literal and metaphorical sentences as compared to the lexical (i.e., word-based) models; (2) semantic representations of the verb are superior compared to that of the nominal object in decoding literal phrases, whereas semantic representations of the object are superior to that of the verb in decoding metaphorical phrases; and (3) linguistic models capture both language-related and sensorimotor representations for literal sentencesin contrast, for metaphoric sentences, linguistic models capture language-related representations and the visual models captured sensorimotor representations in the brain. although the results do not offer straightforward conclusions regarding the role of the literal meaning in metaphor comprehension, they provide some support to the idea that lexical-semantic relations associated with the literal meaning are not fully accessible during familiar metaphor comprehension, particularly within action-related brain regions. ## brain imaging data stimuli consisted of sentences divided into five main conditions: 40 affirmative literal, 40 negated literal, 40 affirmative metaphor, 40 negated metaphor, and 40 affirmative literal paraphrases of the metaphor. stimuli stimuli consisted of sentences divided into five main conditions: 40 affirmative literal, 40 negated literal, 40 affirmative metaphor, 40 negated metaphor, and 40 affirmative literal paraphrases of the metaphor (used as control). a total of 31 unique hand-action verbs were used (9 verbs were re-used twice per condition). for each verb, the authors created four conditions: affirmative literal, affirmative metaphoric, negated literal, and negated metaphoric. all sentences were in the third person singular, present tense, progressive, see figure 1 . stimuli were created by the authors and normed for psycholinguistic variables (i.e., length, familiarity, concreteness) by an independent set of participants in a behavioral experiment. participants fifteen adults (8 women, ages 18 to 35) were involved in the fmri study. all participants were right-handed, native english speakers. experimental paradigm participants were instructed to passively read the object of the sentence (e.g., ''the yellow lemon''), briefly shown on screen first, followed by the sentence (e.g., ''she's squeezing the lemon''). the object was shown on screen for 2 s, followed by a 0.5 s interval, then the sentence was presented for 4 s followed by a rest of 8 s. a total of 5 runs were completed, each lasting 10.15 minutes (3 participants only completed 4 runs). stimulus presentation was randomized across participants. fmri data acquisition fmri images were acquired with a siemens magnetom trio 3t system with a 32-channel head matrix coil. highresolution anatomical scans were acquired with a structural t1-weighted magnetization prepared rapid gradient echo (mprage) with @xmath0, @xmath1.26 ms, flip angle 10%, 256 × 256 mm matrix, 1 mm resolution, and 208 coronal slices. whole brain functional images were obtained with a t2* weighted single-shot gradientrecalled echoplanar imaging, echo-planar sequence (epi) using blood oxygenation-level-dependent contrast with @xmath2, @xmath3, flip angle 90 degrees, 64×64 mm matrix, 3.5 mm resolution. each functional image consisted of 37 contiguous axial slices, acquired in interleaved mode. ## experiments and results we use group-level similarity-decoding to decode brain activity associated with literal and metaphoric sentences using each of our semantic models. we perform decoding at the sentence level for literal and metaphor conditions (affirmative only), separately. decoding was performed at the wholebrain level and across the brain's lobes, as well as within a priori defined rois implicated in visual, action and language-related processing. ## conclusion and future directions we presented the first study evaluating a range of semantic models in their ability to decode brain activity when reading literal and metaphoric sentences. we found evidence to suggest that compositional models can decode sentences irrespective of figurativeness in the brain and that at least for the linguistic models the verb model may be more closely associated with the literal (concrete) meaning and the object model more closely associated with the metaphoric (abstract) meaning. this includes a closer relationship between the verb model and action-related brain regions in the brain during literal sentence processing, in line with neuroimaging work showing that literal versus familiar metaphoric actions more reliably activate sensorimotor areas. this adds support to the idea that the literal meaning may not be as accessible for familiar metaphors. taken together, the linguistic model results are in line with prior neuroscientific studies suggesting that differences between literal and metaphoric sentence processing align with concrete versus abstract concept processing in the brain, mainly with a greater reliance of concrete concepts on sensorimotor areas, while abstract concepts rely more heavily on language-related brain regions. interestingly, however, the results with the visual models point to the need to also consider how metaphor (abstract language) may be grounded in more abstract knowledge about actions or social-interaction. future studies will need to further investigate the accessibility of the literal meaning (and abstract meaning) in metaphor comprehension using a larger dataset. for example, by considering a wider range of metaphors (e.g., metaphoric uses of objects) representing different semantic domains and different degrees of ambiguity. also, it may be useful to consider event embeddings optimized towards learning representations of events and their thematic roles that may be better able to deal with different verb senses by learning non-linear compositions of predicates and their arguments @xcite .