A Psycholinguistic Analysis of BERT ’s Representations of Compounds (2023)

Abstract: This work studies the semantic representations learned by BERT for compounds, that is, expressions such as sunlight or bodyguard. We build on recent studies that explore semantic information in Transformers at the word level and test whether BERT aligns with human semantic intuitions when dealing with expressions (e.g., sunlight) whose overall meaning depends—to a various extent—on the semantics of the constituent words (sun, light). We leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD; quantifying the weight of each constituent toward the compound meaning) and semantic transparency (ST; evaluating the extent to which the compound meaning is recoverable from the constituents’ semantics). We show that BERT-based measures moderately align with human intuitions, especially when using contextualized representations, and that LMD is overall more predictable than ST. Contrary to the results reported for ‘standard’ words, higher, more contextualized layers are the best at representing compound meaning. These findings shed new light on the abilities of BERT in dealing with fine-grained semantic phenomena. Moreover, they can provide insights into how speakers represent compounds.

## introduction compounds such as sunlight or bodyguard are an interesting benchmark to probe the semantic representations learned by any nlp models. on the one hand, compounds that are part of a language lexicon (i.e., lexicalized compounds; @xcite have their own (sets of) established meaning(s). as such, they are lexical items just like any other word. on the other hand, the semantic status of compounds is special since their meaning is the result of the combination of the meaning of two words (hence, the constituents). according to psycholinguistic evidence, this semantic relation does not disappear with lexicalization. indeed, speakers actively combine constituent meanings when processing both novel and lexicalized compounds @xcite @xcite . in this work, we argue that nlp systems capable of faithfully representing word meanings should account for these aspects. for example, to acknowledge that the meaning of handgun relies more on the semantics of gun than of hand (indeed, a handgun is a type of gun). or, that the meaning of sunlight is more directly recoverable from the semantics of its constituents (it is more transparent) than is the meaning of muskrat (which is very opaque). transformer-based encoders such as bert @xcite are shown to produce word representations that align well with human semantic intuitions, particularly at their lower layers @xcite . this suggests that these models are effective in encoding the meaning of a word, without any additional taskspecific fine-tuning. however, these conclusions are based on evaluations that explore semantic relations between words, such as pairwise similarity patterns-not between words and their parts. in parallel, bert's contextualized embeddings have been leveraged for tasks that involve lexical composition. for example, to learn compound representations that are effective in predicting the literality of a compound or its semantic interpretation @xcite . in this case, mixed results were reported. while bert-based models are effective to judge, e.g., that market (but not flea) has a literal meaning in flea market, they are far behind humans in predicting, e.g., that body part stands for part that makes up a body. crucially, these results were obtained by training a binary classifier on the top of bert's embeddings. since the encoder parameters were updated during learning, no conclusions can be drawn on the ef-fectiveness of bert's embeddings in dealing with these and similar fine-grained semantic aspects. in this work, we leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (lmd) and semantic transparency (st). the former quantifies the semantic weight of each constituent toward the compound meaning. for example, that gun has more semantic weight in handgun than hand does. the second evaluates the extent to which the compound meaning is recoverable from the semantics of the constituents. for example, that handgun is very transparent, while muskrat is much less so. we test whether, and to what extent, the measures of lmd and st that we obtain from bert's representations of compounds and compound constituents align with human judgments. we carry out comprehensive experiments on model versions, contexts, pooling methods, layers. 1 we show that: ## data we use a psycholinguistic dataset of human judgments on compound lmd and st @xcite . the dataset includes 629 lexicalized english compounds annotated by 189 participants for various variables. foot_2 lmd is a score that captures which of the two constituents of a compound is semantically dominant for the compound meaning. it ranges in [0,10], where 0 means totally dependent on the left constituent and 10 means totally dependent on the right constituent. in st is defined as a score that quantifies the degree to which the meaning of a compound can be inferred or recovered from the meaning of the constituents: the higher the st, the more transparent the compound. the compounds handgun and wartime in since only the compounds, but not the constituents, are provided in the dataset, we manually annotate each compound (e.g., handgun) with its left (hand) and right (gun) constituents, so to obtain a dataset of ⟨compound, left, right⟩ triplets. while doing so, we decided to discard the pseudocompound mushroom. we were left with 628 triplets, that we use in our experiments. ## method we test whether, and to what extent, bert's representations of compounds and compound constituents approximate human judgments on lmd and st. to do so, we obtain word-level representations using two versions of bert. we experiment with representations obtained by feeding the word either in isolation or in the context of a sentence. moreover, building on previous work, we explore various pooling methods over bert outputs. ## analysis 6.1 lmd: reversed compounds from the results reported in section 5.1, it appears that bert is capable of obtaining sensible representations of compounds that encode the relation between the constituents and their respective semantic 'weight'. however, it might still be that the reported moderate correlations result from the model assigning a default high/low similarity to the constituents while being no or little aware of the semantic and syntactic (i.e., the modifier/head) relation which ties them. if that is the case, the model would consider, e.g., the contribution of war in wartime and timewar to be identical-though the meaning of the reversed compound would be intuitively very different. as such, we might expect a similar/same lmd value assigned by the model to these two compounds, and therefore a similar correlation with human judgments. otherwise, if bert represents a compound by genuinely accounting for the relationship between its constituents, the predicted lmd for the reversed compound (timewar) is likely to be different. as such, we might expect a much lower correlation with human intuitions. in this analysis, we test this issue by re-running the lmd experiment on the reversed version of the compounds in our dataset, i.e., @xmath0, @xmath1, etc. lmd is computed exactly as above, except that we replace the representation of the compound with that of its reversed version. since (most of) these reversed compounds are unlikely to occur in standard corpora of texts, we experiment with nc representations. moreover, we experiment with bert base since it outperformed bert large on this setting. as can be seen in figure 5 , the correlation for reversed compounds is much lower than the original one (0.11 vs 0.39). this is a good sign, which confirms that bert does not rely on shortcuts when representing compound meanings and the relative weight of each constituent. instead, it appears that the model can account for the semantic and syntactic relation 1 2 3 4 5 6 7 8 9 10 11 12 bert layer 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 spearman correlation lmd vs reversed lmd normal lmd nc-all normal lmd nc-cls normal lmd nc-nospec reversed lmd nc-all reversed lmd nc-cls reversed lmd nc-nospec between the constituents, even when information from the surrounding context is not available. ## conclusion in this paper, we study how bert represents the meaning of lexicalized compounds. we take a psycholinguistic angle and show that the model does a reasonably good job of making semantic judgments in line with those of human speakers. since higher, more contextualized layers are shown to correlate best with human intuitions, we propose that speakers may access a specific, context-dependent representation when making these judgments. in future work, recent approaches to multimodal word representation in transformers @xcite could be leveraged to test the role of visual grounding in compound semantics @xcite .