G loss R eader at S em E val-2021 Task 2: Reading Definitions Improves Contextualized Word Embeddings (2021)

Abstract: Consulting a dictionary or a glossary is a familiar way for many humans to figure out what does a word in a particular context mean. We hypothesize that a system that can select a proper definition for a particular word occurrence can also naturally solve tasks related to word senses. To verify this hypothesis we developed a solution for the Multilingual and Cross-lingual Word-in-Context (MCL-WiC) task, that does not use any of the shared task data or other WiC data for training. Instead, it is trained to embed word definitions from English WordNet and word occurrences in English texts into the same vector space following an approach previously proposed for Word Sense Disambiguation (WSD). To estimate the similarity in meaning of two word occurrences, we compared different metrics in this shared vector space and found that L1-distance between normalized contextualized word embeddings outperforms traditionally employed cosine similarity and several other metrics. To solve the task for languages other than English, we rely on zero-shot cross-lingual transfer capabilities of the multilingual XLM-R masked language model. Despite not using MCL-WiC training data, in the shared task our approach achieves an accuracy of 89.5% on the English test set, which is only 4% less than the best system. In the multilingual subtask zero-shot cross-lingual transfer shows competitive results, that are within 2% from the best systems for Russian, French, and Arabic. In the cross-lingual subtask are within 2-4% from the best systems.

## introduction semeval-2021 task 2 is a multilingual and crosslingual word-in-context disambiguation task (mcl-wic) for five different languages @xcite . 1 each example in the multilingual subtask consists of two sentences in english, russian, french, arabic, or chinese language containing occurrences of the same target word. in the cross-lingual subtask each example consists of two sentences in different languages containing occurrences of two different target words. the participants were asked to detect whether those occurrences corresponded to the same or different meanings. these tasks are formalized as binary classification tasks. the datasets contain the same number of examples for each class. accuracy is utilized as the main performance metric. recent sota approaches to the wic task mainly include fine-tuning large universally pre-trained masked language models @xcite on labeled wic datasets. instead, we decided to train a system that selects the most appropriate definition for each word occurrence following an approach proposed by @xcite for word sense disambiguation (wsd) and experimented with different ways of adapting such system to the mcl-wic task. during the evaluation period, we experimented with distances between probability distributions over word definitions but did not manage to achieve good results. but through the post-evaluation period, we switched to distances between the contextualized word embeddings of our gloss-informed language model and improved our results significantly achieving comparable results with the 2nd best system for french and 6th best system for arabic in the multilingual subtask. our main interest was whether the word-incontext systems can benefit from using gloss information, provided for each possible sense of the word. ## background here we summarize prior work linking word occurrences and word definitions. one of the first approaches in this field @xcite calculated the lexical overlap between the context of a particular word occurrence and all possible definitions of this word. this approach did not take into account word synonymy or other lexical relations. the following work tried to combine state-of-the-art language models with glosses from some dictionaries. one of such methods has been proposed by @xcite . their ewise system used a pre-training procedure for a gloss encoder, that learned knowledge graph embeddings from word-net @xcite . after this pre-training, the authors froze the gloss encoder and started to train a context encoder with labeled wsd data. while the method of @xcite requires relational information from a knowledge graph, the method proposed by @xcite relies fully on gloss information. the developed system jointly encodes the context with all possible glosses of the target word. the authors used a pre-trained bert @xcite model as initialization for their encoder. a similar approach has been proposed by @xcite , who trained two separate transformer-based encoders for word occurrences (context encoder) and word definitions (gloss encoder), both initialized with bert weights @xcite . to represent a word occurrence, the outputs of the context encoder for all of its subwords were averaged. to represent a definition, the output of the gloss encoder from [cls] token was taken. finally, for a word occurrence and all of its definitions, the dot products between those outputs were calculated and the softmax function was applied to them, resulting in a probability distribution over possible word senses. the whole model was trained using cross-entropy loss to select the correct word sense on wsd data. ## system overview in order to learn sense-dependent representations of words, we pre-train our system on the word sense disambiguation task. following the bem model @xcite , our system consists of two separate encoders: context encoder and gloss encoder. context encoder (t c ) takes a sentence @xmath0, . . . , c i-1 , w c , c i+1 , . . . , c n containing a target word w c to be disambiguated, where w c is the i th word in the sentence. the encoder then produces the target word representation: r @xmath1, we average representations of these subwords. gloss encoder (t g ) takes as input a gloss g s that defines a word sense s and encodes it as: taking the output from the first input token, which should be [cls] for bert @xmath2. we can score each of the possible senses @xmath3, for a target word w c by taking the dot product of r wc against every r s for @xmath4: both encoders were initialized with bert or xlm-r weights. then the whole system was pretrained on english wsd data @xcite with cross-entropy loss. we denote this pre-training procedure as gloss language modeling (glm) and compare it with pure masked language modeling (mlm) pre-training. in both cases, the models were not fine-tuned on any mcl-wic data. ## experimental setup besides choosing the best final predictor for mcl-wic, we also experimented on encoder initialization. in the result section, we compare the performance of the models, initialized with bert base @xcite , xlm-r base, and xlm-r large @xcite . we trained our models on the english semcor dataset @xcite with glosses from the wordnet 3.0 @xcite . systems based on the xlm-r base and xlm-r large @xcite were trained 20 and 10 epochs respectively. following standard practices, we used @xcite as our development set for early stopping. for the system with bert-base, we used the originally provided checkpoint by @xcite . ## conclusion in this paper, we presented gloss language modeling (glm) procedure as a pre-training strategy for mcl-wic systems. we have shown that this procedure improves multilingual wic performance on all languages for both xlm-r and bert backbones. apart from that, we proposed an interpretable zero-shot multilingual wic algorithm which does not require any labeled data for the multilingual wic task except for the threshold selection, which can be performed using only english development data without loss of accuracy for other languages. we also found that l1-distance between normalized contextualized word embeddings outperforms traditionally employed cosine distance.