Raffinage des représentations des tokens dans les modèles de langue pré-entraînés avec l’apprentissage contrastif : une étude entre modèles et entre langues (2025)

Abstract: Les modèles de langue pré-entraînés ont apporté des avancées significatives dans les représentations contextuelles des phrases et des mots. Cependant, les tâches lexicales restent un défi pour ces représentations en raison des problèmes tels que la faible similarité des representations d’un même mot dans des contextes similaires. Mosolova et al. (2024) ont montré que l’apprentissage contrastif supervisé au niveau des tokens permettait d’améliorer les performances sur les tâches lexicales. Dans cet article, nous étudions la généralisabilité de leurs résultats obtenus en anglais au français, à d’autres modèles de langue et à plusieurs parties du discours. Nous démontrons que cette méthode d’apprentissage contrastif améliore systématiquement la performance sur les tâches de Word-in-Context et surpasse celle des modèles de langage pré-entraînés standards. L’analyse de l’espace des plongements lexicaux montre que l’affinage des modèles rapproche les exemples ayant le même sens et éloigne ceux avec des sens différents, ce qui indique une meilleure discrimination des sens dans l’espace vectoriel final.

## introduction and related work with the advent of large language models (llms), many natural language processing tasks (nlp) have come closer to being resolved @xcite . yet, concerning lexical semantics tasks, llms do not offer as astonishing improvements as for text generation tasks. having an automatic system to disambiguate the word occurrences of a large corpus remains a challenging task for most languages, although highly useful for linguistic corpus analysis. anonymous (2025) report that in zero-shot setting, llms do not surpass supervised systems for word sense disambiguation (wsd), indicating that high wsd performance still requires annotated data. state-of-the-art methods for wsd @xcite rely both on labeled data (babelnet @xcite ), and on pre-trained language models (plms) providing contextualized embeddings. such embeddings are also a core part of current work in the unsupervised version of wsd, namely word sense induction (wsi) @xcite . despite their widespread use for wsd and wsi, contextualized embeddings have also been proven to be inappropriate per se for lexical semantics. in particular, studies on the word-in-context task (wic), consisting in predicting whether two context sentences of a lemma share the same sense or not @xcite , have shown that original plm representations fail to capture senses effectively and perform poorly (accuracy below 66% when using bert-large). this inadequacy may stem from the known limitations of contextualized embeddings when it comes to represent lexical semantics : they show low similarity between the representations of the same word in different contexts and various biases such as frequency of the word or presence of punctuation in a sentence, which affect plms' ability to accurately represent word meaning @xcite @xcite . for these reasons, approaches have been proposed to make contextualized embeddings more suitable for lexical tasks. a common technique is to fine-tune plms using contrastive learning, consisting in using specific loss functions to teach models to bring similar examples (positive examples) closer in the embeddings space while pushing away the dissimilar ones (negative examples) @xcite . importantly, any wsd or wsi system that uses contextualized embeddings can benefit from such fine-tuned plms. in nlp, contrastive learning was initially applied to enhance sentence representations @xcite @xcite , and was later adapted for token-level representations. previous works differ in how positive and negative examples are constructed : "self-supervised" approaches generate positive examples by automatically modifying the original instance, while negative examples are randomly selected from the rest of the dataset. in this vein, @xcite propose the mirrorwic model, which uses random masking of words in the original context surrounding the target word to create self-supervised positive examples. @xcite use the denoising autoencoder @xcite to generate perturbed versions of the original example, serving as positive examples. supervised approaches, in contrast, create positive examples from instances with the same label, extracted from some classification dataset. @xcite use supervised contrastive learning with the aim of performing semantic frame induction. the positive examples are extracted from the english framenet @xcite , as pairs of instances evoking the same semantic frame. @xcite propose to use examplar sentences of word senses from an electronic dictionary (the english wiktionary) : positive examples are sets of exemplars of the same sense, while negative ones are exemplars of different senses of the same lemma. the authors emphasize that this approach is portable to the many languages for which a large wiktionary exists. on top of the wic task, these works have been evaluated on a range of lexical semantics tasks, including wsi @xcite , wsd @xcite , usim @xcite , cosimlex @xcite , and frame induction. all have shown that contrastive learning provides substantial improvements over the original plms. in this paper, we focus on the approach proposed by @xcite and test its generalizability across different model architectures (roberta and modernbert on top of bert), parts of speech (english nouns and adjectives, on top of verbs), and languages (french on top of english). additionally, we analyze changes in the embedding space after fine-tuning, an aspect that was not explored in the original study. in the following sections, we present the approach of @xcite (section 2), the methodology (section 3), the experimental results (section 4), and the qualitative analysis of the embedding space (section 5), followed by our conclusions (section 6). ## summary of mosolova et al. (2024) supervised fine-tuning approach the goal of study by @xcite was to evaluate whether contrastive fine-tuning produces more semantically meaningful token embeddings, better suited for lexical-level tasks than standard plm embeddings. they expected that, during fine-tuning, representations of the same word used in the same sense will be pulled together, while those having different senses will be pushed further apart. for this purpose, they employed the multiple-positives contrastive loss for fine-tuning foot_0 . the loss for one lemma l is defined as : here e(l) is the set of example sentences for lemma l, j is an example sentence within e(l), and s(j) is the subset of these examples, where the lemma l has the same sense as in example j, except for j itself. e(l) \ j is the set of all examples of lemma l except j. s(m, n) is the similarity measure between the embeddings of the target tokens in examples m and n (they use cosine similarity) and τ is a scalar temperature hyperparameter. notice that, since the denominator sums over all examples of lemma l (not only those sharing the same sense), this will mechanically reduce the similarity between examples of different senses, while increasing the similarity between same-sense examples j and j ′ . the term -1 |s(j)| is added to normalize the loss, ensuring that instances with many same-sense pairs do not contribute disproportionately. this prevents lemmas with a large number of examples from dominating the overall training objective. for example, given the target sentence tu devrais frapper avant d'entrer with the target lemma entrer and the following supplementary examples : (1) entrez votre nom d'utilisateur et votre mot de passe, (2) il ne peut pas entrer dans le royaume de dieu, (3) elle avait prévu d'entrer dans la profession juridique, (4) des inconnus pourraient entrer dans la salle, and (5) il est entré immédiatement, the loss function is expected to bring the representation of the target lemma entrer in the target sentence closer to examples 2, 4, and 5, while pushing it away from examples 1 and 3. dataset for their experiments, the authors extracted examples from english dbnary @xcite . the dataset includes all examples of verbal lemmas with at least 1 and at most 10 senses, excluding those with only a single sense and a single example, as well as multiword verbs. the dataset contains 13,118 verbs with a total of 68,271 examples having in total 26,398 senses. mean number of examples per sense in the dataset is 2.59(±5.41), mean number of senses per verb is 2.01(±1.54) and mean number of examples per verb is 5.21(±12.68). the dataset is divided into 3 parts : 80% for fine-tuning, 10% for development and 10% for testing. we reuse their methodology to create datasets for english nouns and adjectives, as well as french verbs.