Characterizing Dynamic Word Meaning Representations in the Brain (2020)

Abstract: During sentence comprehension, humans adjust word meanings according to the combination of the concepts that occur in the sentence. This paper presents a neural network model called CEREBRA (Context-dEpendent meaning REpresentation in the BRAin) that demonstrates this process based on fMRI sentence patterns and the Concept Attribute Rep-resentation (CAR) theory. In several experiments, CEREBRA is used to quantify conceptual combination effect and demonstrate that it matters to humans. Such context-based representations could be used in future natural language processing systems allowing them to mirror human performance more accurately.

## introduction a word meaning is more than an entry in a dictionary. it involves a vast amount of knowledge relating the scenes and experiences people encounter (i.e., a rich encyclopedic knowledge), a set of referents to which the word properly applies (i.e., the boy was angry vs. the chair was angry), combination of other words, and grammatical constructions in which the word occurs. the meaning of the word varies from situation to situation and across contexts of use. for example, the word small means something different when used to describe a mosquito, a whale, or a planet. the properties associated with small vary in context-dependent ways: it is necessary to know what the word means, but also the context in which is used, and how the words combine in order to construct the word meaning @xcite . while humans have a remarkable ability to form new word meanings by combining existing concepts, modeling this process is challenging @xcite @xcite @xcite . the same concept can be combined to produce different meanings: corn oil means oil made of corn, baby oil means oil rubbed on babies, and lamp oil means oil for lighting lamps @xcite . since lamp is an object, oil is likely to be a member of the inanimate category. however, corn and baby are living things, which suggest otherwise. how do language users determine the membership structure of such combinations of concepts, and how do they deduce the interpretation? as this example illustrates, there is no simple rule on how to combine concepts @xcite . computational models of such phenomena could potentially shed light into human cognition and advance ai applications that interact with humans via natural language. such applications need to be able to understand and to form by themselves novel combinations of concepts. consider for example virtual assistants such as siri, ok google, or alexa. these applications are built to answer questions posed by humans in natural language. all of them have natural language processing software to recognize speech and to give a response. however, whereas humans process language at many levels, machines process linguistic data with no inherent meaning. given the ambiguity and flexibility of human language, modeling human conceptual representations is essential in building ai systems that interact effectively with humans. today's experimental methods allow studying neural mechanisms underlying the semantic memory system. neuroimaging (fmri) technology, for instance, provides a way to measure brain activity during word and sentence comprehension. when humans listen or read sentences they use different brain systems to simulate seeing the scenes and performing the actions that are described. as a result, parts of the brain that control these actions light up in the fmri. hence, semantic models have become a popular tool for prediction and interpretation of brain activity. recently, machine learning systems in vision and language processing have been proposed based on single-word vector spaces @xcite . they are able to extract low-level features in order to recognize concepts (e.g. cat), but such representations are shallow and fall short from symbol grounding (meaning). in general, these models build semantic representations from text corpora, where words that appear in the same context are likely to have similar meanings @xcite @xcite @xcite @xcite . this problem has driven researchers to develop new componential approaches where concepts are represented by a set of basic features, integrating different modalities like textual and visual inputs. @xcite @xcite . however, even with these multimodal embedding spaces, such vector representations lack intrinsic meaning, and therefore sometimes different concepts may appear similar. a truly multimodal representations should account for the full array of human senses @xcite . embodiment theories of concept representation provide such an array @xcite @xcite . they allow for a direct analysis in terms of sensory, motor, spatial, temporal, affective, and social experience. further, these theories can be mapped to brain systems. recent fmri studies helped identify a distributed large-scale brain network of multimodal sensory systems linked to the storage and retrieval of conceptual knowledge @xcite . this network was then used as a basis for concept attribute representation (car) theory (a.k.a. the experiential attribute representation model). this theory is a semantic approach that represents concepts as a set of features that are the basic components of meaning, and grounds them in brain systems @xcite @xcite ). an intriguing challenge to semantic modeling is that concepts are dynamic, i.e. word meaning depends on context and recent experiences @xcite @xcite . for example, a pianist would invoke different aspects of the word piano depending on whether he will be playing in a concert or moving the piano. when thinking about a coming performance, the emphasis will be on the piano's function, including sound and fine hand movements. when moving the piano, the emphasis will be on shape, size, weight and other larger limb movements @xcite . ## modeling framework to understand how word meanings change under the context of a sentence, three issues are addressed: ## computational model cerebra model was developed to investigate how words change under the context of a sentence using imaging data (figure 2 ). it is based on the car semantic feature model and the fgrep neural network architecture (forming global representations with extended backpropagation; @xcite . the model is trained to predict fmri patterns of subjects reading everyday sentences. the fgrep mechanism is used to determine how the cars would have to change to predict the fmri patterns more accurately. these changes represent the effect of context; it is thus possible to track the brain dynamic meanings of words by tracking how the cars feature-weightings change across contexts. more specifically, the model is first trained to map carwords (word attribute ratings) to synthwords (fmri synthetic words). once it has learned this task, it is used to modify car words in context. synthwords are combined to form synthsent for the predicted sentence by averaging all words in the sentence. the synthsent is then compared to the actual fmrisent (original fmri data), to form a new error signal. that is, for each sentence, the carwords are propagated and the error is formed as before, but during backpropagation, the network is no longer changed. instead, the error is used to change the carwords themselves (which is the fgrep method; @xcite . this modification can be carried out until the error goes to zero, or no additional change is possible (because the car attributes are already at their max or min limits). eventually, the revised carword represents the word meaning in the current sentence. the cerebra model was trained 20 times for each of the eight fmri subjects with different random seeds. a total of 20 different sets of 786 context word representations (one word representation for each sentence where the word appears) were thus produced for each subject. afterwards, the mean of the 20 representations was used as the final representation for each word (per subject). it is important to emphasize that the goal of the cerebra model is not to predict the fmri patterns as accurately and generally as possible, instead, it is used as a framework to identify and measure context-dependent changes in the car words @xcite @xcite . ## experiments and results to evaluate the performance of cerebra as well as the context-based representations, two computational experiments and a behavioral analysis were conducted. the first two experiments measure how the car representation of a word changes in different sentences, and correlates these changes to the car representations of the other words in the sentence (ows). the behavioral study evaluates the cerebra context-based representations against human judgements. next, an individual example of the conceptual combination effect is first presented, followed by the aggregate analysis and the behavioral study. ## discussion and future work an interesting future work direction would be to replicate the study on a more extensive data set with a fully balanced stimuli and with fmri images of individual words. the differences should be even stronger and it should be possible to uncover more refined effects. such data should also improve the survey, since it would be possible to identify questions where the effects can be expected to be more reliable. compared to other approaches, such as distributional semantic models (dsms), car theory enables a mapping between conceptual content and neural representations. in cars conceptual knowledge is distributed across a small set of modality-specific neural systems that are engaged when instances of the concept are experienced. in contrast, dsms reflect conceptual knowledge acquired through a lifetime of linguistic experience, and they are not grounded on perception and action. experiential data specify the perceived physical attributes or properties associated with the referents of words (e.g., a carrot refers to an object whose attributes describes it as orange, conical/cylindrical, juicy, crispy, sweet). in contrast, linguistic data specify how a given word is statistically distributed across different texts (e.g., a carrot is a root vegetable, usually orange, dutch invented the orange carrots, it contains high carotene, human body turns carotene into vitamin a). a lot of experiential data is usually unstated in such texts. thus, experiential data provide a foundation that support both perceptual data (e.g., answering "orange" to "what color are carrots?), as well as associative/encyclopedic data (e.g., answering "rabbit" to "what animal likes to eat carrots?"; @xcite @xcite . in the future, multimodal cerebra representations could be used to make natural language processing systems more robust. for instance, it may be possible to train a neural network to represent context simultaneously from both dsms and cerebra representations as part of a natural language understanding system for service robot applications. for instance, service robots with such representations would have the capability to understand natural language commands (e.g., watering plants), to have encyclopedic knowledge (i.e., to make decisions), to ground language by adapting to the environment (i.e., object recognition, location) and by understanding novel concepts (i.e., "rain water"). thus, the cerebra representations provide the experiential-based data (i.e., concrete words) and the dsms provide the association-based data (i.e., abstract words), leading to a more robust performance. ## conclusion the cerebra model was constructed to test the hypothesis that word meanings change dynamically based on context. the results suggest three significant findings: (1) context-dependent meaning representations are embedded in the fmri sentences, (2) they can be characterized using brain-based semantic representations (cars) together with the cerebra model, and (3) the attribute weighting changes are real and meaningful to the subjects. cerebra thus takes a step towards understanding how the brain constructs sentence-level meanings dynamically from word-level features.