T rans W i C at S em E val-2021 Task 2: Transformer-based Multilingual and Cross-lingual Word-in-Context Disambiguation (2021)

Abstract: Identifying whether a word carries the same meaning or different meaning in two contexts is an important research area in natural language processing which plays a significant role in many applications such as question answering, document summarisation, information retrieval and information extraction. Most of the previous work in this area rely on language-specific resources making it difficult to generalise across languages. Considering this limitation, our approach to SemEval-2021 Task 2 is based only on pretrained transformer models and does not use any language-specific processing and resources. Despite that, our best model achieves 0.90 accuracy for English-English subtask which is very compatible compared to the best result of the subtask; 0.93 accuracy. Our approach also achieves satisfactory results in other monolingual and cross-lingual language pairs as well.

## introduction words' semantics have a dynamic nature which depends on the surrounding context @xcite . therefore, the majority of words tends to be polysemous (i.e. have multiple senses). for few examples, words such as "cell", "bank" and "report" can be mentioned. due to this nature in natural language, it is important to focus on word-in-context sense while extracting the meaning of a word which appeared in a text segment. also, this is a critical requirement to many applications such as question answering, document summarisation, information retrieval and information extraction. word sense disambiguation (wsd)-based approaches were widely used by previous research to tackle this problem @xcite . wsd associates the word in a text with its correct meaning from a predefined sense inventory @xcite . as such inventories, wordnet @xcite and babel-net @xcite were commonly used. however, these approaches fail to generalise into different languages as these inventories are often limited to high resource languages. targeting this gap, semeval-2021 task 2: multilingual and cross-lingual word-in-context disambiguation is designed to capture the word sense without relying on fixed sense inventories in both monolingual and cross-lingual setting. in summary, this task is designed as a binary classification problem which predicts whether the target word has the same meaning or different meaning in different contexts of the same language (monolingual setting) or different languages (cross-lingual setting). this paper describes our submission to semeval-2021 task 2 @xcite . our approach is mainly focused on transformer-based models with different text pair classification architectures. we remodel the default text pair classification architecture and introduce several strategies that outperform the default text pair classification architecture for this task. for effortless generalisation across the languages, we do not use any language-specific processing and resources. in the subtasks where only a few training instances were available, we use few-shot learning and in the subtasks where there were no training instances were available, we use zero-shot learning taking advantage of the cross-lingual nature of the multilingual transformer models. the remainder of this paper is organised as follows. section 2 describes the related work done in the field of word-in-context disambiguation. details of the task data sets are provided in section 3. section 4 describes the proposed architecture and section 5 provides the experimental setup details. following them, section 6 demonstrates the obtained results and section 7 concludes the paper with final remarks and future research directions. supervised systems supervised systems rely on semantically-annotated corpora for training @xcite . early approaches were based on traditional machine learning algorithms like support vector machines @xcite . with the word embedding-based approaches getting popular in natural language processing tasks, more recent approaches on wsd were based on neural network architectures @xcite . however, they rely on large manuallycurated training data to train the machine learning models which in turn hinders the ability of these approaches to scale over unseen words and new languages. more recently, contextual representations of words have been used in wsd where the contextual representations have been employed for the creation of sense embeddings @xcite . however, they also rely on sense-annotated corpora to gather contextual information for each sense, and hence are limited to languages for which gold annotations are available. a very recent approach sensembert @xcite provide wsd by leveraging the mapping between senses and wikipedia pages, the relations among babelnet synsets and the expressiveness of contextualised embeddings, getting rid of manual annotations. however, sensembert @xcite only supports five languages making it difficult to use with other languages. considering the limitations of the above meth-ods, in this paper we propose an approach which is based on general purpose transformer models and does not rely on external knowledge bases. also, our approach shows strong few-shot/zeroshot learning performance removing the hurdle of having manually-curated training data for each language pair. ## data the data set used for semeval-2021 task 2 is designed targeting a binary classification problem following @xcite . to preserve the multilinguality and cross-linguality of the task, five different languages: english, arabic, french, russian and chinese have been considered for data set preparation. in the monolingual setting, per instance, a sentence pair written in the same language is provided with a targeted lemma to predict whether it has the same meaning (true) or different meanings (false) in both sentences. in the cross-lingual setting, each sentence pair is written in two different languages with the same prediction requirement. few samples from the monolingual and cross-lingual data sets are shown in the monolingual data set covers the language pairs: en-en, ar-ar, fr-fr, ru-ru and zh-zh. for each language, 8-instance trial data sets with labels were provided to give an insight into the task. as training data, 8,000 labelled instances were provided only for the english language and as dev data, 1,000 labelled instances were provided per each language. to use with final evaluation, for each language, 1,000-instance test data sets were provided. the cross-lingual data set covers the language pairs: en-ar, en-fr, en-ru and en-zh. similar to the monolingual data set, 8-instance trial data sets with labels were provided for each language pair. however, no training or dev data sets were provided for the cross-lingual setting. to use with the final evaluation, 1,000-instance test data sets were provided per each language pair. ## transwic architecture the main motivation behind the transwic architecture is the success transformer-based architectures had in various natural language processing tasks like offensive language identification @xcite @xcite , offensive spans identification @xcite , language detection @xcite lang. sentence 1 sentence 2 label ml fr-fr la souris mange le fromage le chat court après la souris t en-en in the private sector , activities are guided by the motive to earn money. the volume v of the sector is related to the area a of the cap. f cl en-fr click the right mouse button le chat court après la souris f en-fr any alterations which it is proposed to make as a result of this review are to be reported to the interdepartmental committee on charter repertory for its approval. il a aussi été indiqué que, selon les dossiers médicaux, justiniano hurtado torre était mort de maladie. t preprocessing as a preprocessing step we add two tokens to the transformer model's vocabulary: <@xmath0. we place them around the target word in both sentences. for example, the sentence "la souris mange le fromage" with the target word "souris" will be changed to "@xmath1. ## experimental setup this section describes the training data and hyperparameter configurations used during the experiments. ## results and evaluation organisers used the accuracy as the evaluation metric as shown in equation 1 where tp is true positive, tn is true negative, fp is false positive and fn is false negative. @xmath2, we trained and evaluated models for each of our strategies using english-english training and dev sets. then the best models are picked to use with few-shot and zero-shot learning approaches. we report the results obtained by english-english evaluation in as shown in since multilingual models provided the best results for the english-english dataset, it provided an additional advantage as they can be used directly in other language pairs too as mentioned in section 5. for other language pairs, we did not perform any evaluation due to the lack of data availability. we trusted the cross-lingual performance of xlm-r and used the best three models of the english-english experiment. for the rest of the monolingual pairs, we used the few-shot learning approach using the given dev sets and for the cross-lingual pairs, we used the zero-shot learning approach mentioned in section 5. we report the results we got for the test set in ## conclusions in this paper, we presented our approach for tackling the semeval-2021 task 2: multilingual and cross-lingual word-in-context disambiguation. we use the pretrained transformer models and remodel the sentence pair classification architecture for this task with several strategies. our best strategies outperform the default sentence pair classification setting for english-english. for other monolingual language pairs, we use the few-shot learning approach while for cross-lingual language pairs we use the zero-shot approach. our results are compatible with the best systems submitted for each language pair and are at a satisfactory level given the fact that we did not use any language-specific processing nor resources. as future work, we would be looking to improve our results more with new strategies. we would like to experiment with whether adding languagespecific processing and resources would improve the results. we are keen to add different neural network architectures like siamese transformer networks @xcite that perform well in sentence pair classification tasks @xcite to the transwic framework. furthermore, we are hoping to work in a multi-task environment and experiment whether transfer learning from a similar task like semantic textual similarity @xcite would improve the results for this task.