LIORI at S em E val-2021 Task 2: Span Prediction and Binary Classification approaches to Word-in-Context Disambiguation (2021)

Abstract: This paper presents our approaches to SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation task. The first approach attempted to reformulate the task as a question answering problem, while the second one framed it as a binary classification problem. Our best system, which is an ensemble of XLM-R based binary classifiers trained with data augmentation, is among the 3 best-performing systems for Russian, French and Arabic in the multilingual subtask. In the post-evaluation period, we experimented with batch normalization, subword pooling and target word occurrence aggregation methods, resulting in further performance improvements.

## introduction in the semeval-2021 task 2: multilingual and cross-lingual word-in-context disambiguation task, the participants were asked to classify whether the target word, occurring in two sentences (sen-tence1 and sentence2), is used in the same or in a different meaning. the two sentences could be in the same language or different languages. there were two subtasks: more detailed information regarding the task is provided by @xcite . we participated in both tracks and experimented with two approaches foot_0 . the first approach fine-tunes xlm-r @xcite as a question answering system searching in the second sentence for a word with the same meaning as the target word in the first sentence. the second approach fine-tunes xlm-r as a binary classifier, and ensembles several such classifiers. also, we used data augmentation to double the number of training examples. this second approach took the 2nd place for the monolingual subtask in arabic and the 3rd place for the monolingual subtask in french and russian. in the cross-lingual subtask, the system ranked 6th for french and arabic. the same system was applied to all subtasks and languages. during the post-evaluation period, we performed thorough experiments with our system. we compared different subword pooling methods, including mean, max, first pooling and their combinations, and found that combinations do not help and mean pooling is overall the best choice. unlike pooling, instead of a simple concatenation of contextualized embeddings for the target word occurrences, it is helpful to combine their difference and normalized component-wise product. finally, we found it beneficial to add a batch normalization layer before feeding those vectors into the classification head. ## related work word sense disambiguation (wsd) is the task of associating the occurrence of a word in a text with its correct meaning from a predefined inventory of senses @xcite . wordin-context disambiguation is a new declination of wsd aiming to evaluate the ability of modern language models to accurately represent contextsensitive words @xcite . its advantage is that it does not rely on pre-defined sense inventories. because word sense disambiguation relies on world knowledge for successful solving @xcite , modern large pre-trained models show promising results in solving this task. among such works, we can mention ares @xcite . ares is a semi-supervised approach for creating sense embeddings. the authors use bert and ukb @xcite to find contexts that are similar to each other and link them to meanings in wordnet @xcite . then, they enrich synset contexts with collocational information from syntagnet @xcite . finally, they enrich semcor @xcite contexts and wordnet glosses to create senselevel representations. ares performs better than models with a comparable number of parameters such as bert or roberta @xcite . however, there has been substantial progress in the field of language modelling since bert first appeared. many researchers have noticed that bert is undertrained and that training it longer and on more data, increases the model performance. among such new models, we may name xlm-roberta (xlm-r) @xcite . xlm-r, as well as bert, is based on a transformer model @xcite . xlm-r uses masked language modelling objective @xcite for model training, where some tokens are replaced with a special "[mask]" token and the model is to restore the masked tokens. xlm-r was trained on a cleaned two-terabyte commoncrawl corpus in 100 languages. a new promising approach to language task modelling is treating any natural language task as a question answering problem. among such works, we can mention @xcite where the au- ## submitted systems description our systems are based on xlm-roberta (xlm-r) model @xcite . we used xlm-r large model as a backbone in all our submissions but switched to xlm-r base for some of the postevaluation experiments. two model training scenarios have been tested. in the first case (ag), due to the symmetric nature of the dataset, we decided to augment the dataset and flip the first and the second sentences. in the second case (mtl), multi-task learning was applied. more detailed descriptions are provided in the following sections. we used transformers library @xcite . ## experiments and results as our submissions showed us that bc models perform much better than qa models, in our postfigure 4: comparison of target aggregation methods for xlmr.base (mean pooling, no batchnorm) figure 5: comparison of subword pooling methods for xlmr.base (dmn agg. with batchnorm) in the first experiment, we trained the submitted models with and without data augmentation. figure 3 shows that the augmentation never decreases performance, at least if the learning rate is selected properly. thus, we always performed data augmentation in the following experiments. figure 4 compares target aggregation techniques. the concatenation of the difference of nonnormalized vectors and component-wise product of normalized vectors (dmn) proved to outperform all other methods by a large margin, especially when the learning rate is properly selected. thus, we used this technique for the following experiments. a simple concatenation of target embeddings, which was used in our submissions, is more than 3% worse and is often outperformed by the difference or component-wise product. since concatenating normalized and nonnormalized vectors can make training difficult, we decided to apply batch normalization before feeding those vectors into the classification head. figure 6 shows that batch normalization does not improve results for english much, but considerably improves the average performance on other languages. this is probably due to the fact, that the overwhelming majority of training examples are in english. also, from figure 6 we notice that for english different poolings give similar performance. for other languages, first pooling is a bad option. we hypothesise that this results from non-english words being split into sub-word tokens more frequently. mean pooling consistently outperforms other poolings. figure 5 additionally compares combinations of different subword poolings. however, those combinations did not improve results compared to single mean pooling. finally, we estimated how much the additional multilingual training data help compared to using only english training examples and counting on cross-lingual zero-shot transfer. in as we figured out that dmn target aggregation and mean subword pooling performs significantly better compared to other variants for xlmr.base model, we trained xlmr.large version with hyperparameters from the best xlmr.base model. the results of the models validated either by score on english dev set (en), or by the average score for non-english dev sets (nen), or by scores on each dev set individually (xx), are shown in the third group of results in the table3. we see that these models outperform any single model from the evaluation phase for all multilingual subtask's test sets and test.en-ar set from cross-lingual subtask. and lastly, we report the results for an ensemble of three xlmr.large models: two mean-dmn models trained with learning rates 1e -5 and 2e -5 and one mean-cnmn model trained with learning rate equal to 1e -5. we see that using ensemble of models with new subword pooling and target aggregation techniques helps us to improve our official results from competition. we improved our results for test.ru-ru (3 → 2), test.fr-fr (2 → 1), test.enen (15 → 12), test.zh-zh (21 → 17), test.en-ar (6 → 4) and test.en-zh (17 → 12) sets. ## conclusion in this paper, we have described our approach to semeval-2021 task 2. we tried treating word-in-context disambiguation as question answering and binary classification problems. in our case, binary classification turned out to be a more promising approach. also, we found that mean pooling over subwords is the best option, batch normalization helps when added before classification head, and concate-