PAW at S em E val-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation : Exploring Cross Lingual Transfer, Augmentations and Adversarial Training (2021)

Abstract: We experiment with XLM RoBERTa for Word in Context Disambiguation in the Multi Lingual and Cross Lingual setting so as to develop a single model having knowledge about both settings. We solve the problem as a binary classification problem and also experiment with data augmentation and adversarial training techniques. In addition, we also experiment with a 2-stage training technique. Our approaches prove to be beneficial for better performance and robustness.

## introduction language is complex even for human beings, let alone for computers. the same word serves different purposes in different scenarios, thus increasing the complexity of the word sense disambiguation (wsd). for example in english, the word "bank" can refer to a financial institution or the land alongside a river. many works revolving around wsd have been done with the help of explicit word sense inventories like wordnet 1 and babelnet 2 . with the advent of advanced deep learning models, it is desirous to develop systems that have a good understanding of languages without such gold standards of word sense. this unsupervised learning can help the model learn better latent representations of words in different contexts. in this paper, we aim to develop a single system that has knowledge of both multilingual and cross-lingual word sense disambiguation by training models with the combined data for both settings. we present our approaches for wsd in the multilingual and cross-lingual domain. the task is treated as a binary-classification problem: whether words have the same sense in the two given pairs of sentences. we experiment with xlm-roberta 1 https://wordnet.princeton.edu/ 2 https://babelnet.org/ @xcite , which is based on the transformer architecture @xcite , as the backbone of our architectures in both the settings. in addition, we also leverage external data and different training techniques and data augmentation. the rest of the paper is organized as follows : various related works have been discussed in section 2, followed by a brief description of the shared task dataset in section 3. the system overview and experimental settings are covered in sections 4 and 5. sections 6 contain the results. section 7 concludes the paper and also includes scope of future work. ## related work silberer and ponzetto (2010) make use of graph algorithms for the word sense disambiguation task. they build a multilingual co-occurence graph in which the multilingual nodes are connected with translation edges and labelled with the target word's translations as obtained from the corresponding contexts. authors in @xcite , use multilingual vector space which is obtained by expanding monolingual features engineered from more than one language, in order to generate a more effective, robust and utilitarian vector representation. these engineered features are then used for wsd. languages like arabic do not have as many resources in the available dataset as compared to more common languages like english. to tackle this issue for the persian language, @xcite follow a two phase approach -in the first phase, they utilize an english word sense disambiguation system to assign "sense tags" to words appearing in english sentences and then in the following phase, they transfer the senses obtained in the previous phase to corresponding persian words. in the semeval-2013 wsd task @xcite , @xcite take a classificationbased approach to the cross-lingual wsd task. they build the hltdi system in which they perform word alignment on the europarl corpus. this helps them find samples in the training data which have ambiguous focus words. the paper describes three variants of the classifier -one is trained over local features, the second is trained over the data with translation of the focus word in the four target languages added to the feature vector and the final variant builds a markov network of the first classifier in order to find the best translation. a few works have also been submitted as a part of semdeep-5 workshop @xcite . @xcite make use of contextualised elmo word embeddings. a bidirectional long short term memory(lstm) cell is used to extract better representation of the given sentences. to disambiguate the words, they optimise the cosine distance between the concatenated hidden representations of words, preceding and following the target word. @xcite augment the dataset by automatically substituting target words using contextual similarity. they then experiment with different contextual word embeddings and train a logistic regression classifier on top of that. ## dataset the dataset @xcite foot_0 provided by the shared task organizers consists of both multilingual and cross-lingual data in english (en), arabic (ar), french (fr), russian (ru) and chinese (zh). the dataset consists of two sentences and the words in corresponding sentences (which need disambiguation) and the corresponding label. ## system overview our experiments revolve around facebook's xlm roberta model, which was an update to their xlm-100 language model. xlm roberta is based on the transformer architecture consisting of multi-attention heads which apply a sequence-tosequence transformation on the input text sequence. the training procedure is inspired from roberta @xcite i.e. only the masked language model objective is used. xlm roberta is scaled up to 100 languages, thus becoming a good choice for multi-lingual datasets. another motivation to experiment with xlm roberta comes from the facility of "cross lin-gual transfer", which can help with unbalanced data of different languages. knowledge is transferred for all languages if the model is trained for a particular task using data of only one language. thus, this feature saves effort of gathering more data to make the data distribution balanced. ## experimental setup we make use of combined training and validation data provided by the shared task organizers. we perform a stratified 5 fold cross validation using the combined data. in all our experiments, we fine tune the entire model. each fold is trained for 20 epochs using early stopping with patience of 6 and tolerance of 1e-3. the models are optimised using adamw @xcite with a learning rate of 5e-6 and a batch size of 16 foot_1 . inputs of maximum sequence length 172 are used in the model. the models have been implemented using pytorch @xcite ) and huggingface's transformers @xcite library. ## results accuracy score is the official evaluation metric for the shared task. the test predictions are obtained by combining the predictions of all the 5 fold models (by averaging the predictions from all models). a few observations can be made by looking at the results: 2. in the cross lingual setting, models trained with and without adversarial training are competent to the same extent. on the other hand, in the multilingual setting, models trained without adversarial training seem to have the upper hand. ## conclusion and future work we explore the performance of xlm roberta at word in context disambiguation both in the multilingual and cross lingual setting. we also explore different training techniques such as twostage training and adversarial training along with some simple augmentations to make our models robust and more generalized. test time augmentations, based on training augmentation turn out to useful. for future work, we can explore the performance of ensembling different kinds of models trained with and without adversarial training together, so as to produce more robust results. it will also be interesting to experiment with larger backbone models in the current architecture.