Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025)

Abstract: For language models to generalize correctly to novel expressions, it is critical that they exploit access compositional meanings when this is justified. Even if we don’t know what a “pelp” is, we can use our knowledge of numbers to understand that “ten pelps” makes more pelps than “two pelps”. Static word embeddings such as Word2vec made strong, indeed excessive, claims about compositionality. The SOTA generative, transformer models and graph models, however, go too far in the other direction by providing no real limits on shifts in meaning due to context. To quantify the additive compositionality, we formalize a two-step, generalized evaluation that (i) measures the linearity between known entity attributes and their embeddings via canonical correlation analysis, and (ii) evaluates additive generalization by reconstructing embeddings for unseen attribute combinations and checking reconstruction metrics such as L2 loss, cosine similarity, and retrieval accuracy. These metrics also capture failure cases where linear composition breaks down. Sentences, knowledge graphs, and word embeddings are evaluated and tracked the compositionality across all layers and training stages. Stronger compositional signals are observed in later training stages across data modalities, and in deeper layers of the transformer-based model before a decline at the top layer. Code will be publicly available on GitHub upon acceptance.

## introduction the representation of entities, concepts, and relations as vector embeddings is a foundational technique in machine learning @xcite . despite their broad success, embeddings often lack interpretability. one approach to improving interpretability involves examining the compositional-* joint first author. compositionality also enables models to understand and generate novel combinations of familiar components @xcite . for instance, in natural language, we can infer the meaning of a phrase like eco-friendly transport even if we have never seen it before, simply by combining the meanings of eco-friendly and transport. in the context of embeddings, if a model can construct representations for unseen combinations by composing the embeddings of known parts, it suggests that the learned representation space is structured and generalizable. recent work has explored compositionality across word embeddings @xcite , sentence embeddings @xcite , and graph embeddings @xcite , often highlighting examples like king -man + woman ≈ queen as evidence of linear semantic structure. however, these claims, particularly for static embeddings such as word2vec, have faced strong criticism. notably, @xcite and others have argued that such findings are often cherry-picked and fail to generalize, thereby overstating the role of compositionality. in contrast, transformer-based models such as bert @xcite , @xcite , and llama @xcite offer more expressive and flexible architectures. these models typically rely on lastlayer representations to encode sentence meaning, but often in ways that are less constrained by additive composition. in addition, many linguistic phenomena such as compound nouns (hot dog), phrasal verbs (give up), and sarcastic expressions (great job, you broke it again) cannot be accurately captured through additive mechanisms alone, as their meanings often depend on context rather than the meanings of individual components. this contrast between early enthusiasm for additive structure and the complexity of meaning in modern models motivates a more rigorous investigation. it is essential to evaluate how much linear additive composition is actually retained across different architectures, modalities, and stages of training. our work formalizes such an evaluation framework and applies it systematically to word, sentence, and graph embeddings. specifically, we formalize a two-step, modality-agnostic evaluation: (i) measuring the linear alignment between known attributes and their embeddings using canonical correlation analysis (cca), and (ii) assessing additive generalization by reconstructing embeddings for unseen attribute combinations and checking distance loss, cosine similarity, and retrieval accuracy. our study makes several novel contributions: the classical principle that "the meaning of a whole is built from its parts" underlies linguistic theory and modern distributional semantics, motivating decades of work on whether and how vector embeddings encode such structure. moving beyond intuition, @xcite define representational compositionality information-theoretically: a representation is compositional if it can be recoded as discrete symbol strings whose semantics is delivered by a low-complexity mapping. in this paper we instantiate, and quantitatively test, the linear additive case of that criterion across words, sentences and knowledge-graph entities. when compositionality fails. not all language is built compositionally: idiomatic compounds (hot dog), phrasal verbs (take off ), sentiment of expressions (yeah right), and sarcasm invert or ignore the literal sum of their parts @xcite @xcite . early computational work detects this breakdown by treating non-compositional expressions as a statistical outlier: unusual association or semantic-asymmetry scores flag verb-object or noun compounds that resisted substitution tests @xcite . a parallel strand relied on syntactic and selectional-preference cues @xcite , while neural phraseembedding models merge frequent idioms into single atomic tokens @xcite . noncompositionality also manifests in vector spaces: analogy tests misrank many compounds that contradict the conceptual priors of compositionality @xcite , and work detects systematic noncompositionality in multiword expressions @xcite . even if compositionality exists, the mapping of conceptual components could be nonlinear, as quadratic or polynomial projections, complexity-theoretic definitions, or dynamic, roledependent nonlinear combinations @xcite @xcite . the set of conceptual components could also be incomplete or defined inaccurately @xcite . these realities frame our scope: we analyze to what extent the conceptual field where compositionality presents and, we only test its linear-additive form defined over a fixed set of conceptual components. linear structure across modalities. early wordanalogy work established that semantic relations align with linear directions in skip-gram vectors @xcite @xcite ; similar structures appear in word @xcite @xcite , sentence embeddings @xcite @xcite , and representations of llms, vision-language models, and other deep learning models @xcite @xcite . @xcite formalise this "linear representation hypothesis", tying linear probes to steerable directions via a causal inner product. measuring compositionality. empirical work measures compositional structure in three main ways. (i) geometry-based measures: vector arithmetic in skip-gram embeddings @xcite , formal additive-offset analyses @xcite , linear decomposition for subject, verb and object in a sentence @xcite , graded tre approximation scores @xcite , and higher-order ica correlations @xcite all ask whether representations decompose as (near)linear sums of parts. (ii) probing and perturbation: simple classifiers or representational-similarity scores reveal which linguistic factors a layer encodes @xcite @xcite @xcite . (iii) behavioral generalization: model outputs on nonce or zero-shot combinations expose the underlying mechanism, e.g. analogy vs. rules in derivational morphology @xcite . our contribution is a unified and statistically robust two-step diagnostic consisting of linearity quantification and additive generalization. it applies across word, sentence, and knowledge graph embeddings, and supports layer-and time-resolved analysis of compositional structure. ## tasks and formulations previous work has explored methods to detect compositionality in embeddings by examining the relationships between entity attributes and their representations, including linear or syntactic probing, linear representation hypothesis, correlation-based linearity detection, embedding of labels and additive compositionality detection @xcite @xcite @xcite . building on the geometric and behavioral strands surveyed in section 2, we formalize a two-step, modalityagnostic diagnostic to quantify compositional generalization. first assesses linearity between the attributes matrix and their embedding spaces. second, tests whether those linear parts combine to predict embeddings for unseen attribute mixes. the pipeline is shown in figure 1 . for each entity, we consider two representations: ## experiments before presenting results, we first illustrate how our framework is applied in practice. for sentences, attributes correspond to concepts such as genre, theater name, and location. for knowledge graphs, they correspond to demographic attributes such as gender and age groups. for words, they correspond to morphological components including roots and suffixes. the experiments then test whether embeddings align linearly with these attributes and whether additive combinations can reconstruct unseen entities. ## discussion and conclusion linearity and compositionality our paper adopted a two-step approach to explore the inherent structure of embeddings. in the first step, quantifying linearity confirmed a robust linear correlation between embeddings and candidate semantic features. this step established the necessary foundation for the second step, quantifying generalizability, where we rigorously demonstrated that final embeddings can be reconstructed as an additive combination of concept embeddings. our findings address the broader question of whether relations between structured entities can be captured through simple vector operations. eval-uating a range of word, sentence, and graph embeddings showed a common thread of additive compositionality and therefore shows the generalizability. for sentence embeddings, conceptual components can often be combined additively, indicating that even large, non-linear models (e.g., gpt or llama) retain interpretable vector structures. word embeddings effectively encode both morphological and semantic relationships, as illustrated by transformations like weight + y + @xmath0. quantifying compositionality the degree of compositionality increases across models regardless of training strategy. in multiberts, compositionality steadily increase with training (see cosine real and l2 norm in