Understanding the Semantic Space: How Word Meanings Dynamically Adapt in the Context of a Sentence (2021)

Abstract: How do people understand the meaning of the word “small” when used to describe a mosquito, a church, or a planet? While humans have a remarkable ability to form meanings by combining existing concepts, modeling this process is challenging. This paper addresses that challenge through CEREBRA (Context-dEpendent meaning REpresentations in the BRAin) neural network model. CEREBRA characterizes how word meanings dynamically adapt in the context of a sentence by decomposing sentence fMRI into words and words into embodied brain-based semantic features. It demonstrates that words in different contexts have different representations and the word meaning changes in a way that is meaningful to human subjects. CEREBRA’s context-based representations can potentially be used to make NLP applications more human-like.

## introduction the properties associated with a word such as small vary in context-dependent ways: it is necessary to know what the word means, but also the context in which is used, and how the words combine in order to construct the word meaning. humans have a remarkable ability to form meanings by combining existing concepts. modeling this process is difficult @xcite @xcite @xcite @xcite @xcite . how are concepts represented in the brain? how do word meanings change during concept combination or under the context of a sentence? what tools and approaches serve to quantify such changes? significant progress has been made in understanding how concepts and word meanings are represented in the brain. in particular, the first two issues are addressed by the concept attribute representation theory (car; @xcite @xcite . car theory represents concepts as a set of features that constitute the basic components of meaning in terms of known brain systems. it relates semantic content to systematic modulation in neuroimaging activity (fmri patterns). it suggests that word meanings are instantiated by the weights given to different feature dimensions according to the context. the third issue is addressed by the cerebra or context-dependent meaning representation in the brain neural network model @xcite @xcite @xcite . it is based on the car theory to characterize how the attribute weighting changes across contexts. in this paper the car theory is first reviewed. then, the cerebra model is introduced, followed by the data that provides the basis for the model. later, experimental results are presented, showing an individual example on the concept combination effect on word meanings, how this effect applies to the entire corpus, and a behavioral analysis to evaluate the neural network model. @xcite @xcite . it is supported by substantial evidence on how humans acquire and learn concepts @xcite @xcite . the central axiom of this theory is that concept knowledge is built from experience, as a result, knowledge representation in the brain is dynamic. the features are weighted according to statistical regularities. the semantic content of a given concept is estimated from ratings provided by human participants. for example, concepts referring to things that make sounds (e.g., explosion, thunder) receive high ratings on a feature representing auditory experience, relative to things that do not make a sound (e.g., milk, flower). each word is modeled as a collection of 66 features that captures the strength of association between each neural attribute and word meaning. specifically, the degree of activation of each attribute associated with the concept can be modified depending on the linguistic context, or combination of words in which the concept occurs. more detailed account of the attribute selection and definition is given by @xcite @xcite . figure 1 , shows an example of the weighted cars for the concept church. the weight values represent average human ratings for each feature. given that church is an object, it has low weightings on animate attributes such as face, body, and speech, and high weighting on attributes like vision, size, shape, and weight. however, since it is a building and a place for worship, it does include strong weightings for sound and music, spatial attributes such as landmark and scene, event attributes like social, time and duration, as well as others such as communication and benefit. ## the cerebra model building on the idea of grounded word representation in car theory, this work aims to understand how word meanings change depending on context. the following sections describe the computational model that characterizes such representations. the specific terms to the cerebra model are denoted by abbreviations throughout the paper (e.g., carword, fmrisent, synthword). for reference, they are described in figure 2 . ## data collection and processing the cerebra model is based on the following sets of data: a sentence collection prepared by @xcite , the semantic vectors (car ratings) for the words obtained via mechanical turk, and the fmri images for the sentences, collected by the medical college of wisconsin @xcite @xcite @xcite . additionally, fmri representations for individual words (called synthword) were synthesized by averaging the sentence fmri. ## experiments cerebra's context-based representations were evaluated through several computational experiments as well as through a behavioral analysis. the computational experiments quantify how the car representation of a word changes in different sentences for individual cases by correlating these changes to the car representations of the other words in the sentence (ows). the behavioral study evaluates the cerebra context-based representations against human judgements. ## discussion and future work this paper described how the car theory, the fmri images, and the cerebra model form the groundwork to characterize dynamic word meanings. the cerebra model generates good interpretations of word meanings considering that the dataset was limited and was not originally designed to address the dynamic effects in meaning. in future work, it would be interesting to replicate the studies on a more extensive data set. a fully balanced stimuli including sentences with identical contexts (e.g., the yellow bird flew over the field vs. the yellow plane flew over the field) and contrasting contexts (e.g., the aggressive dog chased the boy vs. the friendly dog chased the boy), could help characterize the effects in more detail. the context-based changes should be even stronger, and it should be possible to uncover more refined effects. such data should also improve the survey design, since it would be possible to identify questions where the effects can be expected to be more reliable. similarly, it would be desirable to extend the fmri data with images for individual words. the cerebra process of mapping semantic cars to synthwords and further to sentence fmri refines the synthetic representations by removing noise. however, such representations blend together the meanings of many words in many sentences. thus, by acquiring actual word fmri, the observed effects should become even more clear. one disadvantage on cerebra is that it is expensive to collect fmri patterns and human ratings at a massive scale compared to running a statistical algorithm on a data repository. furthermore, any changes to the model (e.g., adding features) would require new data to be collected. on the other hand, such data provides a grounding to neural processes and behavior that does not exist with statistical approaches. concept representation in the car approach can be compared to other methods such as conceptual spaces (cs; @xcite , and distributional semantic models (dsms; @xcite @xcite @xcite @xcite . the car theory and cs characterize concepts with a list of features or dimensions as the building blocks. the car theory provides a set of primitive features for the analysis of conceptual content in terms of neural processes (grounded in perception and action). the cs framework suggests a set of "quality" dimensions as relations that represent cognitive similarities between stimuli (observations or instances of concepts). cs is also considered a grounding mechanism that connects abstract symbols to the real world. the car and cs approaches include similar dimensions (i.e., weight, temperature, brightness) and some of those dimensions are part of a larger domain (e.g., color) or a process (e.g., visual system). whereas car theory is a brain-based semantic representation where people weigh concept dimensions differently based in context, dsms are not grounded on perception and motor mechanisms. instead, dsm representations reflect semantic knowledge acquired through a lifetime of linguistic experience based on statistical cooccurrence. dsms do not provide precise information about the experienced features of the concept itself @xcite . in cerebra, this grounding provides a multimodal approach where features directly relate semantic content to neural activity. ## conclusions the cerebra model was constructed to test the hypothesis that word meanings change dynamically based on context. the results suggest three significant conclusions: (1) contextdependent meaning representations are embedded in the fmri sentences, (2) they can be characterized using cars together with the cerebra model, and (3) the attribute weighting changes are real and meaningful to human subjects. thus, cerebra opens the door for cognitive scientists to achieve better understanding and form new hypotheses about how semantic knowledge is represented in the brain. additionally, the context-based representations produced by the model could be used for a broad range of artificial natural language processing systems, where grounding concepts as well as understanding novel combinations of concepts is critical.