S koltech NLP at S em E val-2021 Task 2: Generating Cross-Lingual Training Data for the Word-in-Context Task (2021)

Abstract: In this paper, we present a system for the solution of the cross-lingual and multilingual word-in-context disambiguation task. Task organizers provided monolingual data in several languages, but no cross-lingual training data were available. To address the lack of the officially provided cross-lingual training data, we decided to generate such data ourselves. We describe a simple yet effective approach based on machine translation and back translation of the lexical units to the original language used in the context of this shared task. In our experiments, we used a neural system based on the XLM-R, a pre-trained transformer-based masked language model, as a baseline. We show the effectiveness of the proposed approach as it allows to substantially improve the performance of this strong neural baseline model. In addition, in this study, we present multiple types of the XLM-R based classifier, experimenting with various ways of mixing information from the first and second occurrences of the target word in two samples.

## introduction the goal of the second task of semeval-2021 @xcite is to perform multilingual and cross-lingual word-in-context disambiguation. more specifically, participants are asked to distinguish whether the meanings of a target word in two provided contexts are the same or not. organizers provided a training set of 8 000 english language (en-en) context pairs and validation sets of 1 000 context pairs for english-english (en-en), french-french (fr-fr), russian-russian (ru-ru), arabic-arabic (ar-ar), and chinese-chinese (zh-zh) languages. since no cross-lingual training data were provided, except for a very small trial set barely usable for training, we decided to venture into generating such data automatically. essentially, the given task is a binary classification problem. the first question was which supervised model to use for the classification of context pairs. recently, pre-trained masked language models such as bert @xcite and xlm-r @xcite have been used to reach promising results in a variety of similar nlu classification tasks. thus, we decided to make use of contextualized embeddings from xlm-r, which provides multilingual-lingual embeddings for more than 100 languages, covering all language pairs of interest in the shared task. in all our experiments, this model is used as the backbone. a straightforward way of solving tasks where two contexts are to be compared, as the word-incontext tasks, is to use deep contextualized embeddings and train a classifier over these embeddings as has been explored in the original monolingual word-in-context task (pilehvar and camacho-collados, 2019). note that commonly embeddings of two contexts are simply concatenated @xcite and this operation is asymmetric. in our work, we explored various symmetric ways of aggregating embeddings from two contexts. the contributions of our work are two-fold. first, we present a simple yet effective method for the generation of cross-lingual training data, showing that it can substantially improve the performance compared to the model trained using monolingual data. second, we test various ways of encoding two input target word occurrences contexts using the xlm-r model. ## baseline supervised wic system massively multilingual transformers pretrained with language modeling objectives xlm-r were shown to be useful for zero-shot cross-lingual transfer in nlp @xcite . as a baseline, we rely on a supervised system that takes as an input two sentences and spans corresponding to occurrences of the target word. pre-trained multilingual encoders are used to represent sentences in different languages in the same space. ## generation of cross-lingual training data using machine translation in this section, we describe a machine-translationbased method for the generation of synthetic training data for the cross-lingual word-in-context task to address the lack of cross-lingual training data usable for the supervised model described above. ## experiments and results below we report the results of the two setups of this shared task: multi-and cross-lingual settings. we train the model six times; reporting mean and standard deviation of accuracy on the test dataset. ## conclusion in this paper, we presented a solution to the crosslingual word-in-context task. the main challenge of this task, as formulated by the organizers, is the lack of explicit training data. to address it, we developed a way of generating synthetic crosslingual data for the word-in-context disambiguating task; we demonstrate the positive influence of such synthetic data on the performance of a model on test datasets. as the baseline model in our experiments, a supervised model based on xlm-r pre-trained language model @xcite was used. we performed tests of various settings based on this model and demonstrated that symmetric aggregation of embeddings for context comparison tasks outperforms asymmetric ways on zero-shot and supervised settings. the code and the produced data, enabling reproducing our experiment, are available online. 3