How do Transformer Embeddings Represent Compositions? A Functional Analysis (2025)

Abstract: Compositionality is a key aspect of human intelligence, essential for reasoning and generalization. While transformer-based models have become the de facto standard for many language modeling tasks, little is known about how they represent compound words, and whether these representations are compositional. In this study, we test compositionality in Mistral, OpenAI Large, and Google embedding models, and compare them with BERT. First, we evaluate compositionality in the representations by examining six diverse models of compositionality (addition, multiplication, dilation, regression, etc.). We find that ridge regression, albeit linear, best accounts for compositionality. Surprisingly, we find that the classic vector addition model performs almost as well as any other model. Next, we verify that most embedding models are highly compositional, while BERT shows much poorer compositionality. We verify and visualize our findings with a synthetic dataset consisting of fully transparent adjective-noun compositions. Overall, we present a thorough investigation of compositionality.

## introduction large language models (llms) have emerged as the de facto standard for language generation and modeling tasks @xcite , largely due to their superior generalization abilities compared to earlier models like bert @xcite . one key factor contributing to this generalizability is compositionality -the ability to form new concepts by combining known ones @xcite . however, testing the compositionality of these models is challenging. previous works on transformer models have mostly relied on performance on benchmarks as proxies for compositional generalization @xcite , but such numbers can be often misleading @xcite . thus, representation * correspondence to: aishiknagar@gmail.com figure 1 : linear concept transformation functions (female and royal) used to transform the representations of nouns @xcite . while a substantial body of literature exists on the functional compositionality of vector space models (vsms) @xcite , such understanding of transformer models, which also rely on the distributional hypothesis @xcite , remains limited. additionally, exploring differences in how current models comprehend compositionality is crucial. to address these, we adopt a functional approach to investigate the compositionality of transformer-based embedding models. we focus on embedding models, due to their widespread use in applications like retrieval-augmented generation @xcite and vector databases @xcite . the primary goal of this work is specifically to analyze how transformer models represent constitutuents of compound words, and how these token-level representations relate to the embedding generated for the entire compound word, taken as context. transformer models excel at generating contextual embeddings, which differ fundamentally from static embeddings traditionally used in vector space models (vsms). this context-dependent embedding generation allows transformer models to represent higher-order se-mantic features, such as semantic dominance and transparency, which emerge naturally in language usage. for example, the model might have learned that the term "red cross," although literally composed of "red" and "cross," frequently carries an organization-like meaning distinct from a purely additive combination of its constituents, resulting in a lower similarity between their consituent representations and the compound, regardless of the method of tokenization. while it may seem intuitive that transformer models represent compound words as weighted averages of their constituent parts, the non-linear and multi-layered nature of these models makes composition far less predictable @xcite . tokenized compounds (e.g., "blue" + "tooth") are not always interpreted in a compositional way, and may instead be mapped to distinct, non-compositional meanings based on learned context @xcite . this ambiguity-especially for polysemous or rare compounds-motivates a closer examination of whether transformer embeddings respect compositionality in practice. compound words are a significant aspect of natural language. in english, compound words account for approximately 4% of the total vocabulary in large corpora like the british national corpus @xcite . language models often fail to generalize to novel compounds or properly encode existing ones @xcite and if a model struggles with word-level composition, it may face challenges with more complex structures @xcite . understanding how models handle lexical composition is therefore foundational for scaling up to phrases and sentences. adjectivenoun compositions and compound words have been widely studied as tests of compositionality @xcite . they allow studying semantic composition in a controlled setting, minimizing syntactic variability. although seemingly simple, two-word compounds and compositions can present significant challenges. for example, ladybird is a beetle unrelated to female birds. red ball refers to a red-colored ball, whereas chocolate ball refers to a ball made of chocolate. moreover, humans often create novel compositions, such as deepfake, which first appeared in 2018. as such, we focus on two-word compounds and adjective-noun compositions. we extend the study of functional compositionality in vsms to embedding models, by testing six compositionality models for combining constituent word embeddings to generate the embedding of their composition. we find that (i) ridge regression can effectively explain compositionality in these models, and (ii) current models are significantly more compositional than bert. we also test the generalizability of our method on novel compound words created through data augmentation. finally, we test the properties of the models on the ladec dataset and on our simple adjective-noun compositions (sanc) dataset to (iii) investigate and visualize the underlying mechanics. another study, by @xcite , utilized human judgement to evaluate two psycholinguistic measures of compound semantic analysis, namely, lexeme meaning dominance and semantic transparency. this work demonstrated that bert resembled human intuition in terms of semantic measures, especially when provided sufficient context to the data inputted. @xcite and bullinaria and levy (2012)'s work on distributional semantic models (dsms) is discussed in appendix a.1. ## testing compositionality models we test 6 compositionality models (both simple mathematical models and ones with learned parameters). these are tested on 4 common embeddings. ## generalization to systematic compositions the ladec dataset contains compound words like 'bookstore', 'dumbbell' and 'waterfall', which are all single-word concepts composed from two constituent words. these compounds have varying levels of semantic transparency, and have arbitrary semantic relationships between the constituents. for example, 'bookstore' is a "store for books", but 'dumbbell' is not a "bell for dumbs". 'dumbbell' originates from it looking like a church bell, but is dumb or silent (i.e. the noun is the second constituent), whereas 'waterfall' relates to water that falls (i.e. noun is first constituent). do the results in the previous section generalize to two-word compositions that are composed more systematically? to investigate this, we turn to adjectives and nouns, because in adjective-noun compositions, there is a clear semantic relationship between the constituents. ## deeper performance analysis our findings suggest some insights for new models: • the strong performance of simple linear models suggests that transformer embeddings support linear additive compositionality. this aligns with previous findings and indicates that embedding spaces could be optimized for linear operations. a complete discussion on this can be found in appendix b. ## conclusion in this study, we explored the compositionality of various transformer-based embedding models. for the first time, we comprehensively analyse modern embedding models and evaluate their compositionality over several metrics and factors. our results showed that modern transformer models exhibit strong compositionality compared to bert. functional analysis revealed that ridge regression, a linear model, best explained compositionality in these embeddings. we also found that these embeddings are highly compositional regardless of whether the compound words were seen during training, suggesting that compositionality arises from the embedding models' architectures, rather than training process or data. our work significantly advances our understanding of compositionality in modern language models.