MCL @ IITK at S em E val-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation using Augmented Data, Signals, and Transformers (2021)

Abstract: In this work, we present our approach for solving the SemEval 2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC). The task is a sentence pair classification problem where the goal is to detect whether a given word common to both the sentences evokes the same meaning. We submit systems for both the settings - Multilingual (the pair’s sentences belong to the same language) and Cross-Lingual (the pair’s sentences belong to different languages). The training data is provided only in English. Consequently, we employ cross-lingual transfer techniques. Our approach employs fine-tuning pre-trained transformer-based language models, like ELECTRA and ALBERT, for the English task and XLM-R for all other tasks. To improve these systems’ performance, we propose adding a signal to the word to be disambiguated and augmenting our data by sentence pair reversal. We further augment the dataset provided to us with WiC, XL-WiC and SemCor 3.0. Using ensembles, we achieve strong performance in the Multilingual task, placing first in the EN-EN and FR-FR sub-tasks. For the Cross-Lingual setting, we employed translate-test methods and a zero-shot method, using our multilingual models, with the latter performing slightly better.

## introduction a key challenge in lexical semantics is to identify or to encode the different senses of an ambiguous word. the word sense disambiguation task (wsd) @xcite is a framework used to evaluate systems in their ability to identify different senses of the word. the task involves selecting the correct sense (meaning) of a target word from a list of senses listed in a sense inventory like wordnet @xcite . @xcite proposed a novel benchmark (wic -word in context disambiguation) for the task casting the problem as a binary classification task, wherein it has to be identified whether a word common to a sentence pair is used in the same sense or not. the wic task frees up the word sense disambiguation task from being tied to any sense inventory. the semeval 2021 task 2: multilingual and cross-lingual word-in-context disambiguation @xcite extends the wic framework proposed by @xcite to more languages. the task is divided into two subtasks -the multilingual task and the cross-lingual task. the sentence pair, with a word in common, which is to be disambiguated, is drawn from the same language in the multilingual task, whereas the pair is drawn from two different languages in the cross-lingual task. the task is posed as binary classification task over a pair of sentence wide contexts sent1 and sent2, containing word sequences w 1 and w 2 respectively. the word sequences, w 1 and w 2 , have a common word in lemmatized form lemma. when w 1 and w 2 invoke the same sense of the lemma in their respective contexts, it is to be labeled as 't' (true) class, else it labeled 'f' (false). as mentioned before, for the multilingual setting, sent1 and sent2 are from the same language; for the cross-lingual setting, sent1 is from english, and sent2 is from a non-english language. the languages considered for the task are arabic (ar), english (en), french (fr), russian (ru), and chinese (zh). therefore for the multilingual evaluation, we have ar-ar, en-en, fr-fr, ru-ru, and zh-zh settings, and for the cross-lingual evaluation, we have en-ar, en-fr, en-ru, and en-zh settings. an example of cross-lingual sentence pair is given in figure 1 . this task provides an evaluation benchmark for word sense disambiguation systems in languages other than english, a direction that has been less explored. for this task, the training data is only provided for the en-en setting, and the development sets are provided only for the multilingual task. as we are proposing supervised systems, it is essential to consider if we have training data or not. therefore, splitting the multilingual task, we propose systems for three components -(i) en-en (train and dev data available), (ii) non-english multilingual (only dev data available), and (iii) cross-lingual (neither train nor dev data available). our models and implementations are available here 1 . 2 related work word sense disambiguation: the techniques for the wsd task are broadly divided into knowledge-based and supervised approaches. the supervised approaches include fine-tuning bert for sequence classification @xcite , ewise @xcite , and bilstm with attention @xcite . the knowledgebased methods use the information present in sense inventories such as wordnet @xcite , ba-belnet @xcite and wikipedia to derive semantic knowledge, assisting in the task of wsd. these include building sense embeddings for each sense of the word and disambiguate the target word using the nearest neighbour sense embedding: sensembert @xcite , @xcite , or augmenting the pretraining objecting of bert to take into account the sense information available in the wordnet: sensebert @xcite . there are two existing benchmarks to evaluate the performance of wsd systems. one method is linked to the sense inventories, and the task is framed as a multi-class classification among the senses of a word listed in the inventory @xcite . the other is the wic framework, not tied to any sense inventory, and asks if a target word has the same sense or not in the two given sentences. recently, transformer-based architectures @xcite (e.g., t5 @xcite ) have outperformed all existing approaches when finetuned on the wic task. cross-lingual nlp: there are many nlp tasks for which the data is present in only some high resource language (often english), but the task needs to be solved for other languages. some existing methods involve -(i) using multi-lingual language models (like mbert @xcite , xlm-r @xcite ) to train on the high resource language and transfer the learning to other languages, or (ii) the translation approaches where we can translate train or test data to a target language and train a language-specific model. these methods have shown good performance on benchmarks like xglue @xcite and xtreme @xcite , which have been designed to test this cross-lingual transfer performance of systems, with train data being present majorly in english, while the testing is to be done in other languages. ## corpus description and data augmentation a brief summary of the available data is shown in wic: we use the data provided by the wic task (pilehvar and camacho-collados, 2018), which proposes the same problem as this task but only in english. they collected their data semiautomatically from wordnet @xcite , and covered only nouns and verbs. xl-wic: it is a dataset that is an extension of the wic dataset to multiple languages (equivalent to our multilingual setting) @xcite . again, the data was collected automatically from wordnet and wiktionaries of various languages. we are only interested in a few languages, from the wordnet development sets, the ones with good human performance, because we often note that this data does not accurately represent human distinguishable senses, which our manually collected task data set does. specifically, we use chinese(zh), danish(da), croatian(hr), and dutch(nl). farsi(fa) also has good human performance but we could not include it due to a pre-processing error. ausemcor: we created our own augmented dataset ausemcor from the semcor @xcite dataset, which is a sense annotated corpora in english, with senses tagged using wordnet as its sense inventory. to generate data points for the same sense (t class), we pair up sentences containing a common lemma, whose wordnet senses are identical. for the other class (f class), we pair up sentences with a common lemma, but this lemma has different wordnet sense across the sentence pair. in addition, for the f class, we make sure that the wordnet supersense is also different for creating coarser sense distinctions as suggested by @xcite . we obtain 4986 datapoints with 2520 unique words. it is approximately balanced (2495 'f' and 2491 't' pairs). like wic, it covers only nouns and verbs. ## proposed approach we shall now describe our proposed approaches for the two subtasks, multilingual and cross-lingual. we deal with english separately because training data is available in english and not in any other language. since the data across the two tasks differ only in the language pairs, some general approaches apply to both settings. we finally submit an ensemble of models in all the tasks. the ensembling was done by taking average of the probability scores of the models (probability sum ensemble). ## experiments and results we ran various experiments to test out the efficacy of the different approaches. all experiments have been carried out with a learning rate set at 10 -5 , using adamw @xcite optimizer, with batch sizes varying in {8, 16, 32}. we noticed that using a batch size of 32 was ideal. however, limited compute availability prevented us from trying it out. we trained our models for 10 epochs. to choose our best model, we performed validation multiple times during an epoch -5 times an epoch for the en-en sub-task and 4 times an epoch for all other tasks. in all tables, we report accuracy, which the task's official evaluation metric. the results and experimental set up of various models on the english set are summarized in the results and experimental set up of various models on the non-english multilingual set are summarized in the performance of various translate-test models is shown in ## ablation study we perform an analysis of the various proposed approaches ( and probably make better internal representations in the process. we do not observe a significant change in model performance on using the augmented data. in fact, for bert base , it decays a little. running models trained exclusively on wic and ausemcor, we note that the ausemcor model performs better, but both models lag behind the model trained on mcl-en. we note strong improvements in performance by ensembling various models ( for the multilingual setting, we note that there is not much difference between the performance of oov and rand models ( we note that some pre-trained language models perform better for this task, especially electra and albert, which improve upon the scores of roberta @xcite , and bert. we also note that their 'large' variants are always better performers than the 'base' variants.