C ambridge at S em E val-2021 Task 2: Neural W i C -Model with Data Augmentation and Exploration of Representation (2021)

Abstract: This paper describes the system of the Cambridge team submitted to the SemEval-2021 shared task on Multilingual and Cross-lingual Word-in-Context Disambiguation. Building on top of a pre-trained masked language model, our system is first pre-trained on out-of-domain data, and then fine-tuned on in-domain data. We demonstrate the effectiveness of the proposed two-step training strategy and the benefits of data augmentation from both existing examples and new resources. We further investigate different representations and show that the addition of distance-based features is helpful in the word-in-context disambiguation task. Our system yields highly competitive results in the cross-lingual track without training on any cross-lingual data; and achieves state-of-the-art results in the multilingual track, ranking first in two languages (Arabic and Russian) and second in French out of 171 submitted systems.

## introduction polysemy still poses a great challenge to natural language processing (nlp) applications. depending on its context, an ambiguous word can refer to multiple, potentially unrelated, meanings. recently, as an application of word sense disambiguation (wsd) @xcite , word-in-context (wic) disambiguation has been framed as a binary classification task to identify if the occurrences of a target word in two contexts correspond to the same meaning or not. the release of the wic dataset (pilehvar and camacho-collados, 2019), followed by the multilingual word-in-context (xl-wic) dataset @xcite , has helped provide a common ground for evaluating and comparing systems while encouraging research in wsd and context-sensitive word embeddings. in this paper, we describe our submission to the semeval-2021 shared task on multilingual and cross-lingual word-in-context (mcl-wic) disambiguation @xcite , which involves determining whether a word shared by two sentences in the same language (multilingual track) or across different languages (cross-lingual track) has the same meaning in both contexts. compared to previous wic and xl-wic benchmarks, two new languages are introduced as well as a crosslingual track where systems are evaluated under a 'zero-shot' setting. the mcl-wic task directly classifies pairs of sentences with regard to the meaning of the shared word. by turning wsd into a binary comparison task, mcl-wic avoids the need for sense tags of previous wsd shared tasks @xcite @xcite . it also resembles the word sense alignment (wsa) task @xcite ) more closely, in which definitions from different dictionaries have to be aligned. contextualised word embeddings and pre-trained transformer-based @xcite language models have been increasingly applied to these tasks and state-of-the-art results have been reported @xcite @xcite @xcite @xcite . in line with previous research, we develop a neural system based on pre-trained multilingual masked language model xlm-r @xcite . additionally, we introduce three distancebased features to be used together with the widely used sequence and token representations for mcl-wic disambiguation. to further improve system performance, we apply automatic data augmentation and extract examples from multiple external resources. a two-step training strategy is then employed to make use of both in-domain and out-of- ## task description the mcl-wic dataset used in the shared task consists of sentence pairs sharing the same target word in the same language or across different languages. the task considers five languages: arabic (ar), chinese (zh), english (en), french (fr) and russian (ru); and contains five multilingual (en-en, ar-ar, fr-fr, ru-ru, zh-zh) foot_1 and four crosslingual (en-ar, en-fr, en-ru, en-zh) subtracks. training data is available for the multilingual en-en sub-track only, and development data is available for all five multilingual sub-tracks. no cross-lingual training or development data is provided. statistics of the mcl-wic dataset are presented in results are computed using the accuracy measure, i.e. the ratio of correctly predicted instances (true positives or true negatives) to the total number of instances. ## official evaluation results we submit our systems to all multilingual and cross-lingual tracks. the official results of our best submission for each track are reported in ## conclusion in this paper, we presented the contribution of the cambridge university team to the semeval 2021 shared task on mcl-wic disambiguation. using xlm-r, a pre-trained multilingual transformerbased language model, as a starting point, we investigated automatic data augmentation, the use of multiple external datasets, multi-stage training strategies, and the representation of tokens and their distance. our detailed analysis demonstrated the effectiveness of the two-step training strategy for making use of both in-domain and out-of-domain data, as well as the benefits of adding distancebased features to the representation for wic disambiguation. our best system yields highly competitive results in the cross-lingual track and achieves state-of-the-art results in the multilingual track, ranking first in two languages (ar and ru) and second in fr out of 171 total submissions.