PALI at S em E val-2021 Task 2: Fine-Tune XLM - R o BERT a for Word in Context Disambiguation (2021)

Abstract: This paper presents the PALI team’s winning system for SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation. We fine-tune XLM-RoBERTa model to solve the task of word in context disambiguation, i.e., to determine whether the target word in the two contexts contains the same meaning or not. In implementation, we first specifically design an input tag to emphasize the target word in the contexts. Second, we construct a new vector on the fine-tuned embeddings from XLM-RoBERTa and feed it to a fully-connected network to output the probability of whether the target word in the context has the same meaning or not. The new vector is attained by concatenating the embedding of the [CLS] token and the embeddings of the target word in the contexts. In training, we explore several tricks, such as the Ranger optimizer, data augmentation, and adversarial training, to improve the model prediction. Consequently, we attain the first place in all four cross-lingual tasks.

## introduction this year, the semeval-2021 task 2, multilingual and cross-lingual word-in-context (wic) disambiguation @xcite , defines the task of identifying the polysemous nature of words without relying on a fixed sense inventory in a multilingual and cross-lingual setting. the task aims to perform a binary classification task to determine whether the target word contains the same meaning or not in two given contexts under both the same language (multilingual) setting and the different languages (cross-lingual) setting. in the multilingual setting, the tasks consist of english-english (en-en), arabic-arabic (ar-ar), french-french (fr-fr), chinese-chinese (zh-zh) and russian-russian (ru-ru) while in the cross-lingual setting, the tasks consist of english-chinese (en-zh), english-french (en-fr), english-russian (en-ru), and english-arabic (en-ar). the tasks contain the following challenges: the rest of this paper is organized as follows: in sec. 2, we briefly introduce related work. in sec. 3, we detail our proposed system. in sec. 4, we present the experimental setup, procedure, and the results. finally, we conclude our work in sec. 5. ## related work the semeval-2021 task 2 aims to handling the tasks of multilingual and cross-lingual word-incontext disambiguation @xcite , i.e., to determine whether the target word contains the same meaning in both given contexts. in the following, we will elaborate several related work. some recent effort, e.g., (pilehvar and camacho-collados, 2018), has been conducted to curate and release datasets to solve the task of wic disambiguation. though it can be narrowed down to binary classification, some techniques have to be implemented to enhance the model performance. for example, the trick of input highlighting mechanism @xcite ) can be facilitated to promote the importance of the target word. the idea of unifying entity linking and word sense disambiguation @xcite can be borrowed to solve the task. the idea of freezing the trained model for other languages @xcite can be explored to relieve the issue of no training data in the cross-lingual tasks. recently, due to the superior performance in tackling nlp tasks @xcite @xcite , pre-trained language models, such as bert @xcite and roberta @xcite , start to dominate the way of word representations than static word embedding methods, e.g., word2vec @xcite and fasttext @xcite . especially, the xlm-roberta @xcite model is a newly released large cross-lingual language model based on roberta and is trained on 2.5tb filtered commoncrawl data in 100 languages. different from other xlm models, xlm-roberta does not require the language token to understand which language is used and can determine the correct language from the input ids. it is a powerful tool for understanding multilingual languages and is very helpful for solving the wic disambiguation task under the cross-lingual setting. hence, we choose xlm-roberta in our system. a critical issue of the task is lack of training data. though existing methods, e.g., lexical substitution @xcite , back translation @xcite , and data augmentation @xcite , can be applied to enrich the data, we mainly explore the usage of wordnet @xcite and the technique of pseudo labelling @xcite because wordnet contains rich synonyms while pseudo labelling is effective to utilize the abundant unlabeled data via their pseudo labels. adversarial training @xcite is an effective method to regularize parameters by introducing noise and to improve model robustness and generalization. we also explore its possibility in fine-tuning xlm-roberta to increase the robustness of the learned the word embeddings. ## overview in the following, we present the task definition, data preprocessing, and our proposed system design. ## experiments in the following, we detail our experimental setup and present the results with analysis. ## conclusion in this paper, we present our system to tackle the word-in-context disambiguation task. we fine-tune the xlm-roberta model to solve both multilingual and cross-lingual word-in-context disambiguation tasks. we specifically design the input format to emphasize the target word in two contexts and promote the importance of the target word by concatenating the embeddings in the corresponding context with the [cls] token to output the classification probability. we apply several training tricks to improve the robustness of model and attain improvement during this procedure. the competition results demonstrate the effectiveness of our implementation. in the future, we plan to explore more model architecture to boost the performance for multilingual tasks.