A Systematic Search for Compound Semantics in Pretrained BERT Architectures (2023)

Abstract: To date, transformer-based models such as BERT have been less successful in predicting compositionality of noun compounds than static word embeddings. This is likely related to a suboptimal use of the encoded information, reflecting an incomplete grasp of how the models represent the meanings of complex linguistic structures. This paper investigates variants of semantic knowledge derived from pretrained BERT when predicting the degrees of compositionality for 280 English noun compounds associated with human compositionality ratings. Our performance strongly improves on earlier unsupervised implementations of pretrained BERT and highlights beneficial decisions in data preprocessing, embedding computation, and compositionality estimation. The distinct linguistic roles of heads and modifiers are reflected by differences in BERT-derived representations, with empirical properties such as frequency, productivity, and ambiguity affecting model performance. The most relevant representational information is concentrated in the initial layers of the model architecture.

## introduction the meaning of multiword expressions such as noun compounds is notoriously difficult to model, particularly because of variability in their degree of compositionality, i.e. the relatedness of the meaning of a compound (e.g. flea market) to that of the individual constituents (flea and market). the degree of compositionality has been successfully predicted using static word embeddings, but transformer-based models such as bert @xcite have so far been less successful. this might be related to a suboptimal use of the information encoded by the models, reflecting our incomplete grasp of how they represent the meanings of complex linguistic structures. in this paper, we aim to improve this understanding, as well as produce actionable methodological recommendations. we predict the degrees of compositionality of 280 english noun compounds associated with human compositionality ratings. we extract their occurrences from a web corpus and model them using pretrained bert. like previous work, we assume that the contextualized nature of these representations may capture key aspects of compound semantics. but we do not expect this information to be equally accessible across the model or independent from underlying linguistic properties. we therefore experiment with variants of bert-derived semantic knowledge (comprising over 40,000 ways of computing the degree of compositionality), and analyze the linguistic roles of compound constituents and their empirical properties (frequency, ambiguity, and productivity). we provide the following contributions: the remainder of this paper is organized as follows. we first review related studies ( §2), and then introduce our data ( §3) and experimental setup ( §4). we then analyze and discuss the results ( §5) and provide a conclusion ( §6). ## related work the meaning of noun compounds is modeled from a broad range of perspectives. in psycholinguistics, for example, there is a long tradition of research on human processing of compound semantics. its focus is usually on semantic transparency, which is operationalized using measures including the semantic relatedness of the constituents and the retention of their meaning in the compound (e.g. @xcite @xcite . computational studies have examined a similar range of compound properties, aiming to predict the meaning of the whole compound @xcite , the semantic relations between a compound's constituents @xcite , or the compound's degree of compositionality. the latter issue is also the focus of our work. the ability of computational models of compound semantics to predict the degree of compositionality is usually evaluated on a ranking task with gold standard data in form of human compositionality ratings, which exist for languages including english @xcite , german (von der @xcite , french and portuguese @xcite . strong results have been obtained using static word embeddings, generally by learning dedicated representations for the whole compound and comparing them against the representations of the individual constituents, often combined using composition functions @xcite @xcite @xcite @xcite . most studies predict the compositionality of the whole compound, but differences between heads and modifiers have been underscored. reported effects on model performance are related to their distinct linguistic roles and empirical properties such as frequency, productivity, and ambiguity (schulte im @xcite @xcite . more recently, compositionality prediction has been addressed using pretrained transformer-based models. while bert-derived representations used in a subsequently trained classifier performed well on binary classification @xcite , their unsupervised use in the standard ranking task formulation has been less successful. for instance, an implementation based on comparisons between contextualized and non-contextualized compound representations obtained significantly poorer results compared to static word embeddings, leading to a suggestion that these models do not capture compositionality in a way similar to human annotators @xcite . stronger correlations with human judgment were obtained in a probing study, but using external linguistic knowledgegold standard synonyms of noun compounds @xcite . we are unaware of unsupervised implementations of bert-derived representations that are competitive with static word embeddings on this task. this might be related to a suboptimal use of the information encoded by bert, as the layers in its architecture capture different aspects of linguistic structure @xcite . for instance, it has been suggested that semantic knowledge in general @xcite and word sense information in particular @xcite is encoded in higher layers, but also that type-level information is encoded in lower layers @xcite . more generally, these and other types of implementation decisions impact bert performance on other linguistically oriented tasks (e.g. @xcite . to the best of our knowledge, these patterns have not been investigated in detail for compound semantics, with the cited studies generally relying on widely used solutions (e.g. computing a token-level embedding by averaging over the last four layers). ## data this section introduces the data resources we used. for details on licenses, see appendix a. ## results and discussion we evaluate each constellation of parameters by calculating spearman's rank correlation coefficient between the compositionality estimates it produces and human judgments from the test set. all implementations are evaluated on three prediction targets: compositionality scores for the compound as a whole, the head, and the modifier. ## conclusion we have presented an experiment on predicting the degree of compositionality of 280 english noun compounds using a wide range of variants of semantic knowledge derived from pretrained bert. we have identified a competitive best implementation achieving ρ = 0.706 with human judgement and highlighted clear takeaways. in terms of preprocessing, stronger results were obtained when modeling a larger number of occurrences per compound, whereas controlling for sequence length did not have a clear effect. on embedding computation, different pooling functions led to comparable performance, but there was a clear advantage for layers in the low-to-mid range, which strongly improved on layer combinations used in earlier studies. as for compositionality estimates, it was clearly beneficial to compute them on the token rather than type level, as well as to use (i) representations targeting the constituent of interest; (ii) representations of the surrounding context; (iii) comparisons across complementary -rather than redundant -representational information. looking at empirical properties of compounds, low-frequency, low-productivity, and lowambiguity heads obtain better compositionality predictions. this trend confirms that more limited semantic dispersion makes it easier to model compound meaning. the fact that it holds across compound, head, and modifier compositionality scores highlights the importance of the head in the linguistic structure -and computational modeling -of compounds. taken together, our results point to important practical decisions when running similar implementations and contribute to our understanding of the way in which bert represents lexical meaning, supporting the view that the pretrained model encodes at least some aspects of compound semantics.