U ppsala NLP at S em E val-2021 Task 2: Multilingual Language Models for Fine-tuning and Feature Extraction in Word-in-Context Disambiguation (2021)

Abstract: We describe the Uppsala NLP submission to SemEval-2021 Task 2 on multilingual and cross-lingual word-in-context disambiguation. We explore the usefulness of three pre-trained multilingual language models, XLM-RoBERTa (XLMR), Multilingual BERT (mBERT) and multilingual distilled BERT (mDistilBERT). We compare these three models in two setups, fine-tuning and as feature extractors. In the second case we also experiment with using dependency-based information. We find that fine-tuning is better than feature extraction. XLMR performs better than mBERT in the cross-lingual setting both with fine-tuning and feature extraction, whereas these two models give a similar performance in the multilingual setting. mDistilBERT performs poorly with fine-tuning but gives similar results to the other models when used as a feature extractor. We submitted our two best systems, fine-tuned with XLMR and mBERT.

## introduction semeval-2021 task 2: multilingual and crosslingual word-in-context disambiguation (mcl-wic) @xcite is an extension from wic (pilehvar and camacho-collados, 2019), a shared task at the ijcai-19 semdeep workshop (semdeep-5). wic was proposed as a benchmark to evaluate context-sensitive word representations. the wic dataset foot_0 consists of a list of english sentence-pairs. each sentence-pair has a target word, and the task is to determine whether the target word is used in the same meaning or different meanings in the two sentences, thus as a binary classification task. mcl-wic extends wic to multilingual and cross-lingual datasets, foot_1 and covers 5 example label the cat chases after the mouse. f click the right mouse button. the cat chases after the mouse. t la souris mange le fromage. ('the mouse eats the cheese') our main interest is to investigate the usefulness of pre-trained multilingual language models (lms) in this mcl-wic task, without resorting to sense inventories, dictionaries, or other resources. as our main method, we fine-tune the language models with a span classification head. we also experiment with using the multilingual language models as feature extractors, extracting contextual embeddings for the target word. in this setting, we also add information about syntactical dependency (i.e. head words and dependent words), with the intuition that it can contain relevant contextual information for disambiguation, as in figure 1 , where the head words chases and button could help in disambiguating mouse. we compare three different lms: xlm-roberta (xlmr), multilingual bert (mbert) and multilingual distilled bert (mdistilbert). we show that the fine-tuned models are stronger than any of the models based on feature extraction, by a large margin. xlmr is stronger than mbert in the cross-lingual setting both with finetuning and feature extraction. mdistilbert gives poor results with fine-tuning, but is competitive to the other lms when used for feature extraction. adding dependency syntax to our feature extraction method led to mixed results. we submitted our two strongest systems to the shared task, those fine-tuned with xlmr and mbert. ## related work in wic at semdeep-5, many participating systems capitalized on contextualized word representations. the lmms (language modelling makes sense) system by @xcite used word embeddings from bert, together with sense embeddings from wordnet 3.0 @xcite . @xcite used the contextualized representations from elmo @xcite and trained a separate classification model. @xcite experimented with several contextualized representations and used cosine similarity to measure word similarities. @xcite included wic as one of the tasks in the proposed superglue benchmark, with the approach of fine-tuning bert. at the end of the wic evaluation period, the best result was achieved by @xcite with an accuracy of 68.36%, while human-level performance is 80%, as provided by the dataset curators. @xcite recently proposed sensem-bert 3 , a knowledge-based approach to sense embeddings for multiple languages. an important source for building senseembert is the contextualized representations from a pretrained language model. they experimented with sensembert on both english and multilingual word sense disambiguation (wsd) tasks, and showed that sensem-bert is able to achieve state-of-the-art result on both english and multilingual wsd datasets. ## system description we use the pre-trained language models in two different ways: for fine-tuning (section 4.1) and as feature extractors (section 4.2 -4.3). depending on whether feature transformation is involved, the features extracted can be further categorized into target word embeddings (section 4.2) and dependencybased syntax-incorporated word embeddings (section 4.3). in the following sub-sections, we describe the three systems respectively. due to time constraints we did not use xlmr in the systems with feature extraction. ## results and analysis the evaluation results on the test sets are shown in among the systems with feature extraction, the relative performance of the three sets of contextual embeddings differ from the fine-tuning. here, mdistilbert are competitive to the other two embeddings. we only use xlmr with lr, and again, we see that it gives the best performance in the cross-lingual setting among all systems with lr, just as with fine-tuning. in the multilingual setting, xlmr is also strong, having the best result for three out of five languages. compared to finetuning, mdistilbert performs surprisingly well here. it is on par or better than mbert in most cases across all settings. comparing the different architectures used with the feature extraction strategy, we see that using an mlp is preferable to lr, leading to large improvements in most cases. an exception is english-chinese, where the mlp without syntax performs worse than lr. for english-french on the other hand, the mlp outperforms lr by around 10 percentage points, whereas we see small improvements for english-russian. finally, the addition of syntax leads to mixed results. for the english-chinese system, we see large improvements, whereas we see the opposite for english-french. for english-russian as well as for all multilingual systems, the differences are overall smaller. we also note that the performance is stronger for english-english than for the other languages in most settings. this is expected, since we only have english-english training data. a notable exception is for lr, where english-english performs considerably worse than in all other settings and is on par with the other languages in the same setting. with fine-tuning we overall see stronger results in the multilingual setting, than in the cross-lingual setting, where we mix language pairs. we do not see this difference for our feature extraction systems, however. ## conclusion and future work we have investigated the use of three large language models for multilingual and cross-lingual word-in-context disambiguation. we found that fine-tuning the language models is preferable to using them as feature extractors either for an mlp or for logistic regression. trying to add dependencybased syntax information in the mlp gave mixed results. we also found that xlmr performed better than mbert in the cross-lingual setting, both with fine-tuning and feature extraction, whereas the two models had a more similar performance in the multilingual setting. mdistilbert did not perform well with fine-tuning, but was competitive to the other models in the feature extraction setting. we submitted our two best systems, fine-tuning with