Injecting W iktionary to improve token-level contextual representations using contrastive learning (2024)

Abstract: While static word embeddings are blind to context, for lexical semantics tasks context is rather too present in contextual word embeddings, vectors of same-meaning occurrences being too different (Ethayarajh, 2019). Fine-tuning pre-trained language models (PLMs) using contrastive learning was proposed, leveraging automatically self-augmented examples (Liu et al., 2021b). In this paper, we investigate how to inject a lexicon as an alternative source of supervision, using the English Wiktionary. We also test how dimensionality reduction impacts the resulting contextual word embeddings. We evaluate our approach on the Word-In-Context (WiC) task, in the unsupervised setting (not using the training set). We achieve new SoTA result on the original WiC test set. We also propose two new WiC test sets for which we show that our fine-tuning method achieves substantial improvements. We also observe improvements, although modest, for the semantic frame induction task. Although we experimented on English to allow comparison with related work, our method is adaptable to the many languages for which large Wiktionaries exist.

## introduction pretrained language models (plms) have brought great advances in most nlp tasks. as far as word embeddings are concerned, though, we have moved from one extreme to the other, namely from static word embeddings providing a single representation for a given form, no matter how ambiguous it is, to contextual token embeddings providing one representation per occurrence. for lexical level tasks, while it is desirable that token-level vectors of the same word sense are close in the semantic space, this is not the case for the majority of plms @xcite . in this paper, we address the tuning of tokenlevel contextual representations to better target the lexical sense instantiated by a given token. we use the contrastive learning (cl), which proved efficient for getting sentence embeddings that better capture sentence-level similarity @xcite @xcite and for getting better tokenlevel embeddings @xcite . these approaches use self-supervised cl, with positive examples created by pairing an original sentence and an automatically modified version of it. in this paper, we rather investigate how to leverage hand-crafted lexicons. although these are not always perfectly tailored to nlp tasks, due to coverage and granularity mismatches with the task or domain at hand, they do contain an enormous amount of lexical information that is a pity not to make use of. to do so, we use cl on the example sentences of the english wiktionary, a crowdsourced lexicon. we will show the approach is beneficial for both the word-in-context (wic) task (intrinsic evaluation), and for the frame induction task (extrinsic evaluation). crucially, although we experiment on english to allow comparison with related work, our method is adaptable to a large number of languages for which large wiktionaries exist. we also investigate whether reducing dimensions can provide better-suited token-level contextual embeddings. in the following, we describe related work ( § 2), and how we adapted the cl loss to wiktionary examples ( § 3). we present our language model fine-tuning experiments, along with an evaluation on the word-in-context task ( § 4). we test whether our fine-tuned token embeddings can help cluster verbal occurrences into semantic frames ( § 5). ## related work within the deep metric learning paradigm, contrastive learning (cl) became increasingly popular in computer vision and in nlp @xcite . it consists in modifying the representation space so that similar objects (positive examples) are brought closer while dissimilar objects are pushed away from each other. @xcite proposed one of the first contrastive loss functions, for binary positive examples. cl methods are either supervised or self-supervised. while the former rely on labeled data, the latter employ automatic modifications of objects to produce binary positive pairs (self-augmentation). since there can be more than two examples of the same class, @xcite adapt the contrastive loss to handle "multiple-positive" examples for computer vision. in nlp, cl is primarily used to improve sentence representations, better capturing sentence similarity, mainly in the self-supervised paradigm. self-augmentation techniques include back translation @xcite ), text corruption @xcite , or plm's dropout to produce slightly different embeddings per encoding run @xcite . @xcite combine whitening and cl to fine-tune sentence representations by plms. supervised cl is much less common. we can only cite @xcite who use it for fine-tuning a plm while learning a downstream sentiment-analysis classifier. in contrast to sentence embeddings, fewer works focus on token-level plm representations. @xcite fine-tune contextual embeddings using self-supervised cl, creating positive pairs with dropout and random masking of context tokens. @xcite use cl to favor more isotropic tokenlevel representations. they train a student bert model on the masked language modeling task with a help of a frozen teacher bert model: cl aims at increasing the similarity of student and teacher token representations. apart from cl, there was also work in enhancing bert with senses during pre-training. for example, @xcite add supersense prediction for every masked word as pre-training objective. finally, since we heavily rely on similarities of contextual embeddings, we mention studies reporting the particularities of such spaces. @xcite show that very few dimensions dominate the cosine similarity and propose postprocessing methods to smooth this effect. @xcite tried to solve the problem of underestimated cosine similarity for high-frequency words. our goal is to obtain token-level contextual representations more aware of lexical semantics, by injecting lexicon-based information using cl. we show that this injection is beneficial for the closely related wic task, and, to some extent, for the more downstream task of frame induction. ## cl for lexical sense examples our method fine-tunes the token-level contextual representations of a plm using supervised cl, taking the examples of a lexicon as supervision. more precisely, each example sentence in the lexicon is associated with a word sense and contains a target word occurrence used in this particular sense. we adapt the multiple-positive contrastive loss of @xcite to the use of a lexicon as labeled data. 1 let e(l) be the set of example sentences for lemma l. for an example @xmath0, let s(j) be the subset of e(l) of examples concerning the same word sense as j, except for j itself. for every lemma l, we create a single batch, and we define a loss summing over the set e(l) of all examples of l: with e(l) \ j being e(l) except j. we write s(m, n) for the similarity between the embeddings of the target tokens in examples m and n (s can be any vector similarity function), and τ is a scalar temperature hyperparameter. in order to cope with known flaws of cosine similarity for high-dimensional spaces, we also experiment with a simple pca reduction of the plm embeddings, with or without whitening. ## plm fine-tuning experiments training dataset more precisely, our training data includes the examples for all verbs having from 1 to 10 senses, except verbs having a single sense with a single example, and multiword verbs. in total, we obtained a dataset of 13,118 verbs having in total 26,398 senses, with a total of 68,271 examples. mean number of examples per sense is 2.59 (std. dev. is 5.41). mean number of senses per verb is 2.01 (std. dev. is 1.54). mean number of examples per verb is 5.21 (std. dev. is 12.68). each example concerns a target verb occurrence. for hyperparameter tuning and evaluation, we split the dataset into 95/5/5% for training, development and test sets, ensuring that verb lemmas do not overlap between the three sets. ## extrinsic evaluation : frame induction we now turn to evaluating our fine-tuning approach on semantic frame induction. compared to word sense induction, frame induction seeks to identify semantic classes (or frames) that may group senses of different lemmas. it is thus a challenging task for token embeddings. we reuse the dataset of yamada et al. (2021) (hereafter y21), extracted from the lexicographic part of framenet 1.7. we reproduce the approach of y21 with minor modifications. it takes as input a set of words, each in the context of a sentence. occurrences of the same lemma are clustered first, and the resulting clusters (called pseudo-lexical units) are then averaged and further clustered to form frames. to represent the target words to cluster, y21 use a weighted average of two token embeddings obtained after applying a plm on the original sentence, with and without masking the target word. we describe our minor modifications and hyperparameter tuning on y21's dev set in appendix a.3. we select the best hyperparameter combination (using the f-b-cubed metric of the second clustering step) for each of the four types of embeddings: with and without cl fine-tuning, and with and without pca. results on the test set are provided in for frame induction (second step), while for the dev set our cl fine-tuning is clearly beneficial (+5.1 points for bcf), the increment on the test set is more modest and is only obtained with pca (62.1 compared to 61.3). the utility of cl finetuning for this task is thus limited, but with pca it provides shorter embeddings, reducing computational cost for downstream tasks. we also notice that the best layers are high layers for the first step, but low layers for the second step. moreover, after fine-tuning, the tuned α 2 is close to 0, suggesting that flaws of the original unmasked token representations that were fixed when combining with the masked embeddings, were smoothed away during the fine-tuning step. ## conclusions we presented a new approach for fine-tuning tokenlevel representations of plms, using contrastive learning with examples from the english wiktionary, a crowd-sourced lexicon. we show its effectiveness on the word-in-context task: we establish the new sota on the wic test set, in the unsupervised setting (not using the wic training set), and we also obtain substantial gains on two new wic test sets, with different sense inventories. we also report improvements, though more modest, on the downstream task of semantic frame induction. although we experimented on english, our method is adaptable to the many languages for which large wiktionaries exist and provides a simple way to obtain token-level embeddings more adapted for lexical semantic tasks. a promising continuation of this work is to create positive examples using wiktionary example sentences for distinct lemmas.