Non-Complementarity of Information in Word-Embedding and Brain Representations in Distinguishing between Concrete and Abstract Words (2021)

Abstract: Word concreteness and imageability have proven crucial in understanding how humans process and represent language in the brain. While word-embeddings do not explicitly incorporate the concreteness of words into their computations, they have been shown to accurately predict human judgments of concreteness and imageability. Inspired by the recent interest in using neural activity patterns to analyze distributed meaning representations, we first show that brain responses acquired while human subjects passively comprehend natural stories can significantly distinguish the concreteness levels of the words encountered. We then examine for the same task whether the additional perceptual information in the brain representations can complement the contextual information in the word-embeddings. However, the results of our predictive models and residual analyses indicate the contrary. We find that the relevant information in the brain representations is a subset of the relevant information in the contextualized word-embeddings, providing new insight into the existing state of natural language processing models.

## introduction language comprises concrete and abstract words that are distinctively used in everyday conversations. concrete words refer to entities that can be easily perceived with the senses (e.g., "house", "blink", "red"). on the other hand, abstract words refer to concepts that one cannot directly perceive with the senses (e.g., "luck", "justify", "risky"), but relies on the use of language to understand them @xcite . this categorization of words based on their concreteness is rooted in theoretical accounts in cognitive science. one such account is the dual coding theory @xcite , according to which two separate but interconnected cognitive systems represent word meanings, i.e., a non-verbal system that encodes perceptual properties of words and a verbal system that encodes linguistic properties of words. concrete concepts can be easily imagined and are represented in the brain with both verbal and non-verbal codes. abstract concepts are less imaginable and are represented with only verbal codes. for example, one can readily picture as well as describe the word bicycle (e.g., "has a chain", "has wheels"), but relies more on a verbal description for the word bravery. the concreteness of words has since been used as a differentiating property of word meaning representations. previous studies in natural language processing (nlp) have examined the wordembedding spaces of concrete and abstract words and showed: (i) distinct vector representations of the two categories within and across languages @xcite , and (ii) high predictability of concreteness scores from pre-trained wordembeddings @xcite . neurolinguistic studies have shown an extensive, distributed network of brain regions representing the conceptual meaning of words @xcite @xcite . among these, regions more closely involved in sensory processing have been shown to respond favorably to concrete words @xcite over abstract words. @xcite argued that concrete and abstract concepts must be represented differently in the human brain by showing through a statistical analysis that concrete concepts have fewer but stronger associations in the mind with other concepts, while abstract concepts have weak associations with several other concepts. @xcite showed that functional magnetic resonance imaging (fmri) signals of brain activity recorded as subjects attempted to decide which two out of a triplet of words were most similar contained sufficient information to classify the concreteness level of the word triplet, providing further evidence of the dissimilar representations of the two categories in the brain. however, it remains an open question whether the brain responses within the semantic system can directly predict concreteness levels in the more challenging setting of naturalistic word stimuli (e.g., words encountered while reading a story). moreover, given the human brain's expertise in generating and processing perceptual as well as linguistic information, one could expect the brain representations to provide information that complements the word-embeddings purely learned from linguistic contexts, improving their predictive capability. we address both these questions in this paper. while several related works exist, the following limitations prompted a new study: (i) @xcite indirectly decoded the brain representations for concrete and abstract nouns with the help of word-embeddings and convolutional neural network image representations. instead of building a predictive model, the authors used a similarity metric to determine which signal in a pair of fmri signals corresponds to which word in a pair of words. however, a direct, supervised decoding approach (as adopted here) would provide more substantial evidence about the strengths and weaknesses of the different information modalities. (ii) @xcite found word concreteness scores to be highly correlated with both visual and haptic perceptual strength. however, multi-modal methods @xcite have incorporated only visual features (as the second source of information) instead of general perceptual features into their predictions. by incorporating brain representations in our models, we do not miss out on such perceptual information (e.g., the adjectives "silky", "crispy", and "salty" are concrete but not as imagery-inducing as the adjective "blue"). (iii) in contrast to previous studies that have required participants to actively imagine a randomly presented word stimulus 1 (before being given a few seconds to "reset" their thoughts) during the brain data acquisition task @xcite @xcite , we adopt a task where participants would read highly engaging natural stories (without unnatural pauses), enabling them to process the word stimuli in a more realistic context. to summarize, our objectives with this paper are twofold. first, we investigate how well human 1 e.g., one word would be presented every 10s. ## related work a few studies have shown that the concreteness (and imageability) of words can be directly predicted with high accuracy from precomputed wordembeddings using supervised learning algorithms. recently, @xcite used a combination of word-embeddings and morphological features to predict the word concreteness and imageability values provided in seven publicly available datasets. @xcite extended the idea to perform a cross-lingual transfer of concreteness and imageability scores by exploiting pretrained bilingual aligned word-embeddings @xcite . multi-modal models that use both linguistic and perceptual information have been shown to outperform language models at various nlp tasks, such as learning concrete or abstract word embeddings @xcite , concept categorization @xcite , and compositionality prediction @xcite . @xcite found that the concreteness of nouns could be predicted equally well from the textual, visual, and combined modalities. this suggests that the textual and visual modalities independently provided reliable, non-complementary information to represent both concrete and abstract nouns. several studies have addressed the idea of decoding neural activity patterns recorded in subjects when presented with certain textual or visual stimuli. @xcite applied linguistic and visually-grounded computational models to decode the fmri representations of a set of concrete and abstract nouns. they, too, reported no decoding advantage for multi-modal combinations over the linguistic model. @xcite demonstrated that fmri signals contained sufficient information to perform a 7-way classification of a set of words into wordnet-based @xcite taxonomic categories. lately, there has been an increasing research interest at the intersection of neuroimaging and language models @xcite @xcite @xcite @xcite . in an interesting study, @xcite finetuned the bert language model to predict the fmri responses of text-reading participants to obtain representations that encode brain-activityrelevant semantic information. while the modified representations could better predict neural activity and even generalize to new participants, this inclusion of brain-relevant bias did not improve or degrade the model's performance on downstream nlp tasks. ## data preparation rating and vectorizing using the wordrepresentation for each story and a list of the fmri acquisition-times (identical for all subjects), we partitioned the words into disjoint chunks so that all words in a chunk correspond to the same acquisition. therefore, all words read by the subjects within a duration of 1 t r from the start of the acquisition pulse were included in the same chunk. we used gpt-2 to vectorize each word in a story by supplying all words in the story leading up to it 3 as context and extracting the network's hidden layer representation corresponding to the last input position. to rate the words in the story, we first lowercased and lemmatized them and then used the @xcite concreteness dataset to assign a rating to each word in a chunk. only around 7% of all words in the stories were not covered by the dataset and were dropped before subsequent analysis. we stored the ith preprocessed functional image of each subject as an n b -dimensional voxelresponse vector b i , where n b denotes the number of voxels for that subject's brain. typical values for n b were found to lie in the 70k-90k range (with a mean of 80976 and a standard deviation of 6173, across subjects). downsampling since the rate at which the text stimulus was presented to the subjects (the narration rate) is higher than the rate of fmri data acquisition (2.0045s per acquisition), several words may occur within the tr corresponding to a single acquisition and will all fall under the same chunk. therefore, we downsampled the stimulus to match the acquisition rate before further analysis by averaging out the concreteness ratings (r w ) and word-embeddings ( e w ) within each tr. thus, the chunk-rating and chunk-embedding for chunk c i are given by: stacking we temporally stacked the voxelresponse vectors, chunk-embeddings and chunkratings, first within each story and then across all 11 stories to obtain (i) a per-subject voxel-response matrix @xmath0, (ii) an embedding matrix @xmath1, and (iii) a rating vector @xmath2, where t denotes the total number of fmri acquisitions across all stories per subject, and d denotes the dimensionality of the word-embedding space. @xmath3, and 11 stories with an average duration close to 12.5 min per story gives @xmath4. ## results we use the abbreviations e for the wordembedding based model, b for the voxel-response based (brain) model, e+b for the combinedrepresentation model, e-b for the word-embedding model with voxel-response information removed, and b-e for the voxel-response model with wordembedding information removed. figure 1 shows the classification accuracies of all models across the six subjects. ## conclusion this paper has three key findings. first, we showed that words encountered in natural stories could be classified based on concreteness purely from the neural activity elicited as subjects passively comprehended the stories, using a direct, supervised approach. second, we showed that in making the concrete/abstract distinction, contextualized wordembeddings (i.e., gpt-2) do not benefit from the inclusion of information from the accompanying fmri signals, despite evidence from several neurolinguistic studies of the human brain exhibiting fundamentally different representations over the two categories. finally, we found that while the residual information remaining in fmri signals after regressing out word-embedding information can no longer distinguish concrete from abstract words, the residual information in word-embeddings beyond the fmri signals performs significantly at this task. this shows that the information in the voxel-responses important to our prediction task is a subset of the corresponding information in the contextualized word-embeddings. our results should be interpreted in light of the following observations: a limitation of our work is that while the voxelresponses and word-embeddings (from gpt-2) considered provide contextualized stimulus representations, the @xcite dataset provides non-contextualized ratings for each word. we partially addressed this discrepancy by formulating the prediction task as a classification problem since the available labels are now much more likely to match ground-truth. i.e., it is reasonable to assume that the broad binary concreteness class of a word will rarely be modified by context as much as the continuous scores would. future work could overcome this limitation by developing the ideas from the recently introduced concretext task foot_3 another observation is that while b (69 ± 2.5%) significantly distinguishes concrete from abstract words, it still does not perform as well as e (87%) at this task. there could be two reasons for this difference. first, b does not handle abstract stimuli as well as e does. quantitatively, while b achieves a recall of 77 ± 2.6% on concrete chunks, its recall on abstract chunks is significantly lower at 63 ± 3.6%. on the other hand, e shows nearly identical performances over the categories. to conclude, we believe that this paper will inspire future work to take up the following exciting directions: which natural language processing tasks may benefit from incorporating human language processing information into the existing frameworks? are there ways of including such information to expose avenues for improvement in these models?