Disentangling lexical and grammatical information in word embeddings (2025)

Abstract: To enable finer-grained linguistic analysis, we propose a method for the separation of lexical and grammatical information within contextualized word embeddings. Using CamemBERT embeddings for French, we apply our method to 14,472 inflected word forms extracted from the Lexical Network of French ( LN-fr ), covering 1,468 nouns, 202 adjectives and 299 verbs inflected via 14 distinct grammatical feature values. Our iterative distillation alternates two steps until convergence: (i) estimating lexical or grammatical vectors by averaging the embeddings of words that share the same lexeme or grammatical feature value, and (ii) isolating the complementary component of each word embedding by subtracting the estimated vector. To assess the quality of the decomposition, we measure whether the resulting lexical and grammatical vectors form more compact clusters within their respective groups and whether their sum better reconstructs the original word embeddings. All evaluations rely on L2 distance. The observed improvements in both clustering and reconstruction accuracy demonstrate the effectiveness of our approach.

## introduction static word embeddings, such as those generated by word2vec @xcite , assign a single, fixed vector to each word form based on its general contextual usage. this approach conflates distinct meanings of polysemous words or homonyms and fails to capture morphological compositionality, as it does not model how word forms may share a common core lexical meaning or how affixes encode grammatical features. for morphologically rich languages, this entanglement can hinder finegrained linguistic analysis. previous work, such as @xcite , addressed this issue by proposing a method to decompose static embeddings into lexical and inflectional components, aiming to obtain semantically purer representations. contextualized embeddings from pretrained language models such as bert @xcite produce dynamic, context-sensitive vectors that implicitly encode a range of linguistic information, including morphology and syntax. this has substantially improved the modeling of polysemy, homonymy, and morphosyntactic variation compared to static embeddings. however, it remains unclear how lexical and grammatical features are represented within these embeddings and whether they can be meaningfully disentangled. in this paper, we revisit the problem of separating lexical and grammatical information in word embeddings, focusing on embeddings produced by camembert @xcite for french, a language with a relatively rich morphology. this work was originally motivated by a separate study where we aimed to measure the semantic idiomaticity of french idioms. semantic idiomaticity refers to the extent to which the meaning of an idiom cannot be inferred from its component words. while camembert is able to distinguish free simple lexemes from words within idioms, it struggles with component words within idioms of different levels of semantic idiomaticity @xcite . this suggests that the model captures idiomaticity at a superficial lexical level, but is not sensitive to the internal semantic structure of idioms. we hypothesized that this limitation is due to the entanglement of multiple types of idiomaticity, not only semantic, but also morphological and syntactic. in order to isolate purely semantic meaning from grammatical interference, we turned to the problem of disentangling lexical and grammatical components in contextual embeddings. the current study develops and evaluates a method for this task, inspired by the methodology proposed by @xcite . we assume that a word embedding can be modeled as the linear combination of two components, a lexical vector capturing its core lexical meaning, and a grammatical vector encoding morphosyntactic information. therefore, we should be able to isolate one component by subtracting the other. our method relies on two assumptions: under this framework, the lexical vector of a lexeme can be estimated either directly, by averaging the embeddings of its inflected forms, or indirectly, by subtracting a shared grammatical vector from each word embedding. likewise, grammatical vectors associated with specific feature values can be derived by averaging over relevant word embeddings or by removing lexical content. to obtain more accurate and disentangled representations, we develop an iterative distillation process that integrates both estimation strategies. at each step, one component is isolated by subtracting the current estimate of the other, then refined by averaging over the pertinent group of words (e.g., all inflected words of a lexeme, or all words sharing a grammatical feature value). this process incrementally improves both components over successive iterations. we hypothesize that, after distillation, lexical vectors belonging to the same lexeme on the one hand, and grammatical vectors sharing the same feature value on the other, get closer in the vector space. we evaluate this by comparing the average pairwise l2 distances within each group before and after distillation. we also assess the reconstruction accuracy of the original embeddings by measuring the difference between each embedding and the sum of its distilled lexical and grammatical components. in this study, we focus specifically on inflection and leave aside derivation, as it is often noncompositional. we worked on french because it has a sufficiently rich morphology for it to be nontrivial, and we had access to the data we needed. however, our method is language-agnostic, and such data is relatively easy to come by for a variety of languages. ## related work recent studies have highlighted that contextualized word embeddings encode various types of linguis-tic information in a high entangled form @xcite . this has sparked growing interest in disentangling grammatical information. however, most existing work addresses this challenge in the context of downstream tasks or model performance, rather than focusing on extracting grammatically meaningful representations for linguistic analysis @xcite @xcite @xcite . to our knowledge, few studies have explicitly addressed this question from the perspective of linguistic analysis. the work most closely related to ours that we know of is by @xcite , who developed a method applied to decompose static word2vec embeddings in english. their approach, based on averaging and subtraction, was tested on a small-scale dataset of around 20 verbs, with a primary focus on lexical vectors. their method struggled with homonyms due to the static nature of word2vec embeddings. in contrast, our approach leverages contextualized embeddings, which mitigate this issue. it is also applied to a much larger and more diverse natural corpus. while inspired by their methodology, we extend it with an iterative refinement process and expand the analysis to include grammatical vectors as well. in addition, we introduce a broader set of evaluation metrics. ## results and discussion in this section, we evaluate whether the lexical and grammatical components extracted from word embeddings display greater internal consistency after distillation. in each comparison, we measure the target evaluation metric before and after the procedure. in all result tables, the before column reports results calculated using the original embeddings, while the after column shows results based on distilled vectors. the relative change (∆) is calculated as af ter-bef ore bef ore , reflecting the proportion of the resulting change. all reported results are rounded to two decimals. ## conclusion we aimed to disentangle contextualized word embeddings in camembert into lexical and grammatical parts. we proposed an iterative distillation method based on the complementarity of averaging and subtraction. a word's lexical vector can be approximated either by averaging the vectors of all words that share the same lexical meaning, or by subtracting the vectors corresponding to its grammatical features. similarly, its grammatical vector can be obtained either by subtracting the lexical part from the original embedding, or by averaging the embeddings of all words that share the same grammatical feature. if effectively separated, the lexical and grammatical vectors should be more distinct, with minimal overlap between their respective contents compared to their initial estimates. each vector should convey more clearly the structural regularities shared with similar words, resulting in tighter alignment within their lexical or grammatical groups. as such, they are better suited to jointly approximate the original word embedding. as expected, in our evaluation, the final lexical and grammatical vectors that we extracted are more clearly clustered with their structurally similar counterparts, when combined, reproduce original word embeddings with minimal loss. notably, the reduction in distance is much more pronounced for grammatical vectors than for lexical vectors-around 40% versus 10% on average. since the initial grammatical vectors are averaged over a large set of lexemes and contain great lexical noise, which is removed during distillation, leading to tighter alignment. the extent of this convergence is relatively stable across different feature values, but varies across parts of speech. verbs, especially, show a stronger reduction in distance and a larger drop in reconstruction error compared to nouns and adjectives. this may be due to the richness of verbal morphology in french, where suffixes often encode several grammatical features at once, making the grammatical signal more prominent and its removal more effective. nouns show only minor reconstruction gains, likely due to limited grammatical variation from number inflection alone. also, large number of nouns in our dataset may stabilize their initial estimates, leaving less room for improvement. adjectives fall in between, showing moderate gains. another important factor behind these observations concerns tokenization and the model's sensitivity to morphological markers. in camembert, frequent, short, and morphologically informative tokens are more likely to be consistently encoded or even assigned special status during training @xcite @xcite . in contrast, lexical roots often span longer or rarer sub-word tokens and are more prone to being split or distorted, especially in low-frequency contexts. as a result, grammatical information is already more cleanly separated and clearly encoded in the model's internal representations, making it easier to distill effectively. this also explains why verbs, whose suffixes encode multiple features in compact forms, benefit the most from the process. future work will further explore how factors such as word frequency and tokenization affect the separation of lexical and grammatical vectors. our method assumes a linear relationship between lexical and grammatical vectors; in a follow-up study, we plan to explore non-linear relationships. given that our method does not rely on language-specific morphological rules, we will apply and evaluate it across languages. in addition, we are interested in extending the approach using learning-based methods, and in incorporating morphology-aware tokenizers to improve grammatical representation. finally, we aim to assess the practical value of our decomposition through downstream tasks.