Are representations built from the ground up? An empirical examination of local composition in language models (2022)

Abstract: Compositionality, the phenomenon where the meaning of a phrase can be derived from its constituent parts, is a hallmark of human language. At the same time, many phrases are non-compositional, carrying a meaning beyond that of each part in isolation. Representing both of these types of phrases is critical for language understanding, but it is an open question whether modern language models (LMs) learn to do so; in this work we examine this question. We first formulate a problem of predicting the LM-internal representations of longer phrases given those of their constituents. We find that the representation of a parent phrase can be predicted with some accuracy given an affine transformation of its children. While we would expect the predictive accuracy to correlate with human judgments of semantic compositionality, we find this is largely not the case, indicating that LMs may not accurately distinguish between compositional and non-compositional phrases. We perform a variety of analyses, shedding light on when different varieties of LMs do and do not generate compositional representations, and discuss implications for future modeling work.

## introduction compositionality is argued to be a hallmark of linguistic generalization (szabó, 2020). however, some phrases are non-compositional, and cannot be reconstructed from individual constituents @xcite . intuitively, a phrase like "i own cats and dogs" is locally compositional, whereas "it's raining cats and dogs" is not. therefore, any representation of language must be easily composable, but it must also correctly handle cases that deviate from compositional rules. both lack @xcite and excess @xcite of compositionality have been cited as common sources of 1 code and data available at https://github.com/ nightingal3/lm-compositionality x a b in general form, the compositionality principle is simply "the meaning of an expression is a function of the meanings of its parts and of the way they are syntactically combined" @xcite . however, this definition is underspecified @xcite . recent efforts to evaluate the compositional abilities of neural networks have resulted in several testable definitions of compositionality @xcite . previous work on compositionality in natural language focuses largely on the definition of substitutivity, by focusing on changes to the constituents of a complex phrase and how they change its representation @xcite @xcite . the definition we examine is localism: whether or not the representation of a complex phrase is derivable only from its local structure and the representations of its immediate "children" @xcite . a similar concept has been proposed separately to measure the compositionality of learned representations, which we use in this work @xcite . we focus on localism because it is a more direct definition and does not rely on the collection of contrastive pairs of phrases. this allows us to examine a wider range of phrases of different types and lengths. in this paper, we ask whether reasonable compo-sitional probes can predict an lm's representation of a phrase from its children in a syntax tree, and if so, which kinds of phrase are more or less compositional. we also ask whether this corresponds to human judgements of compositionality. we first establish a method to examine local compositionality on phrases through probes that try to predict the representation of a parent given its children (section 2). we create two englishlanguage datasets upon which to experiment: a large-scale dataset of 823k phrases mined from the penn treebank, and a new dataset of idioms and paired non-idiomatic phrases for which we elicit human compositionality judgements, which we call the compositionality of human-annotated idiomatic phrases dataset (chip) (section 3). for multiple models and phrase types, we find that phrase embeddings across models and representation types have a fairly predictable affine compositional structure based on embeddings of their constituents (section 4). we find that there are significant differences in compositionality across phrase types, and analyze these trends in detail, contributing to understanding how lms represent phrases (section 5). interestingly, we find that human judgments do not generally align well with the compositionality level of model representations (section 6). this implies there is still work to be done at the language modelling level to capture a proper level of compositionality in representations. ## conclusion we analyze the compositionality of representations from several language models and find that there is an effective affine approximation in terms of a phrase's syntactic children for many phrases. although lm representations may be surprisingly predictable, we find that human compositionality judgments do not align well with how lm representations are structured. in this work, we study the representations produced after extensive training. however, the consistency of several trends we observed suggests that there may be theoretical reasons why lm representations are structured in certain ways. future work could investigate the evolution of compositionality through training, or motivate methods that would allow lms to achieve improved compositional generalization while representing non-compositionality.