U o B _ UK at S em E val 2021 Task 2: Zero-Shot and Few-Shot Learning for Multi-lingual and Cross-lingual Word Sense Disambiguation. (2021)

Abstract: This paper describes our submission to SemEval 2021 Task 2. We compare XLM-RoBERTa Base and Large in the few-shot and zero-shot settings and additionally test the effectiveness of using a k-nearest neighbors classifier in the few-shot setting instead of the more traditional multi-layered perceptron. Our experiments on both the multi-lingual and cross-lingual data show that XLM-RoBERTa Large, unlike the Base version, seems to be able to more effectively transfer learning in a few-shot setting and that the k-nearest neighbors classifier is indeed a more powerful classifier than a multi-layered perceptron when used in few-shot learning.

## introduction and motivation word sense disambiguation (wsd) is the task of disambiguating semantic meaning at the word level and is an important part of natural language processing (nlp) with applications in several downstream tasks @xcite . in recent years, few shot learning (fsl) has been successful in several domains @xcite including in nlp @xcite . modern deep neural networks require a significant amount of training data that might not always be available. fsl is a solution to this problem, wherein training data from a related domain or language can be used to augment training on the target domain/language with significantly less data. fsl can be characterised as n-way k-shot classification. we participate in semeval task 2 multilingual and cross-lingual word in context disambiguation @xcite , which provides 8,000 training examples in english but only 32 in the other target languages. in addition the tasks requires disambiguation of word pairs in the cross lingual setting between en-ar, en-fr, en-ru, and en-zh, where only 8 examples each are available. this sparsity in training data led us to explore the use of fsl for this task. we hypothesise that: our experiments on both the multi-lingual and cross-lingual data show that these hypotheses are in fact correct. we show that xlm-roberta large, unlike the base version, seems to be able to more effectively transfer knowledge in a few-shot setting, and that the knn classifier is indeed a more powerful classifier than a multi-layered perceptron (mlp) when used in few-shot learning. to ensure reproducibility and so other researchers can build on this work, we release the program code, hyperparameters and experiments associated with this work foot_0 . ## related work the first effective method that implemented few shot learning in nlp was that by @xcite , who introduced the application of the siamese network in one-shot learning. the siamese network is typically used to calculate semantic similarity between sentences and was shown to be powerful in the fsl setting. a recent use of zero-shot learning in wsd was work by @xcite , who proposed the extension of wsd systems by incorporating sense embeddings (ewise). these sense embeddings are derived from a knowledge graph, namely word-net @xcite , and graph embeddings. ewise predicts over an embedding space instead of the discrete label space and allow generalized zero-shot learning capability. instead of using annotated data, it uses definitions from wordnet. @xcite proposed a simple and effective method which uses clustering in egonetworks. egocentric networks are local networks with one central node, known as the "ego". this method allows labeling of words in the context with learned sense vectors thus providing a new approach to unsupervised wsd. our use of the k nearest neighbours (knn) classifier is motivated by the work by @xcite , who proposed a very straightforward network, similar to the nearest class mean approach @xcite , that makes use of the classes of "prototypical" examples to classify new examples based on their distances. we extend this method by use of pre-trained models and a knn classifier (section 3). ## methodology in this section, we describe the different models used for the task. our models are built on top of xlm-roberta, but because cross-lingual data is limited, we used fsl during training. we also perform data preprocessing, especially in the case of chinese and arabic where tokenisation is inexact. ## experiments design we design several experiments to find the best performing model. all the experiments are performed on the same platform foot_2 . we also use warmup in all the experiments and further hyperparameters are detailed in the documentation associated with the program code released with this work. for each experiment, five different random seed are tested and we choose the best performing model. we use the english training data to train our baseline and zero-shot models. the development data provided consisted of 1000 examples for each language in the multi-lingual setting (en-en, ar-ar, zh-zh, fr-fr, and ru-ru). we divide this into a dev-train subset consisting of 600 examples, a dev-validation and dev-test subsets consisting of 200 examples each. in the cross-lingual setting, where we have only 8 examples from each language pair, we use the trial data for few shot training and do not use a validation or test set due to data sparsity. finally, due to data sparsity in the target languages we freeze the transformer layers of xlm-roberta during the few shot training phase. based on the results of these experiments, we select the best five models to submit to the official websites. these models were: ## results and analysis the final results in the cross-lingual and multilingual settings are displayed in tables 1 and 2 respectively. each table displays the accuracy on the corresponding test set with the rank achieved by that submission in brackets (out of a total of 87 teams). in each of the two tables, rows 1, 3 and 2, 4 provide a comparison between the zero-shot and few shot settings. it is interesting to note that few shot learning is not effective when using the base version of xlm-roberta. across both the crosslingual and multi-lingual settings, the minimal additional data does little to boost performance and in some cases actually reduces performance. xlm-roberta large on the other hand, seems to be able to transfer knowledge extracted from training in english to the other languages and gains the most when those languages are significantly different from english, the language in which the majority of the training data is available in. the impact of few shot learning is the largest when the difference between the original training data (in this case english) and the target languages is largest. as can be seen from the use of a knn classifier, in place of an mlp, improves performance across the board providing the best results in a lot of the cases and comparable results in the rest. these results seem to validate the results obtained by @xcite , who show that the use of a knn to classify examples related to prototypical examples is an effective method in few-shot learning (section 2). ## conclusions and future work this paper described our submission to semeval 2021 task2, multi-lingual and cross-lingual word in context disambiguation. given the nature of the task, wherein we are provided with training data in english and very limited training data in the other target languages, we use zero-shot and few-shot learning. we hypothesised (section 1) that two methods will significantly boost performance in the fewshot learning setting: a) the use of "large" pretrained models which have been shown to have access to a lot more linguistic information and so generalisability, and b) the use of a knn classifier instead of a multi-layered perceptron. our experiments, described in section 5, confirm that this is indeed the case. we find that xml-roberta large is able to significantly increase performance in the few-shot setting, especially when the target languages are dissimilar to english, which is the language the majority of the training data is available in. we additionally find that the use of a knn classifier boosts performance in the few-shot setting. we additionally show that when using pretrained models, tokenisers might split words in ways that are not conducive to the task at hand, especially in languages such as chinese, where word delimitation is inexact. we handle this limitation by using a comma to delimit words in ways that are specific to the problem, which is both effective and easy to implement. in future, we intend to explore the use of these methods on other datasets and problems beyond word sense disambiguation.