Data Augmentation Method for Boosting Multilingual Word Sense Disambiguation (2023)

Abstract: Recent advances in Word Sense Disambiguation suggest neural language models can be successfully improved by incorporating knowledge base structure. Such class of models are called hybrid solutions. We propose a method of improving hybrid WSD models by harnessing data augmentation techniques and bilingual training. The data augmentation consist of structure augmentation using interlingual connections between wordnets and text data augmentation based on multilingual glosses and usage examples. We utilise language-agnostic neural model trained both with SemCor and Princeton WordNet gloss and example corpora, as well as with Polish WordNet glosses and usage examples. This augmentation technique proves to make well-known hybrid WSD architecture to be competitive, when compared to current State-of-the-Art models, even more complex.

## introduction word sense disambiguation is well recognised issue in natural language processing. due to word ambiguity it is impossible to give a priori a proper semantic interpretation of a text, so senses ought to be disamiguated. in recent years a great improvement has been achieved in the field with the use of deep neural networks (dnn). for low-resourced languages, however, wsd is still an open problem because of the lack of large-scale sense annotated corpora required by modern neural models. large number of categories (which are senses themselves) makes the task very hard for dnn classifiers, because of the bottleneck of sense annotation sparseness. constructing a large sense annotated corpus is a very laborious task, so this problem affects nlp for most world languages (the estimated number of which exceeds 6,000). on the other hand, even nlp for languages that possess vast wsd corpora (i.e. semcors and extensive wordnet-based corpora) has to cope with a huge number of senses that are rarely occurring in texts (for such senses the available dnn representation might not be sufficient). two main solutions have been proposed to these problems: first, the usage of knowledge bases facilitates wsd algorithm through propagating information within a semantic network. second, the use of pre-trained language models, especially multilingual (or language agnostic) allows to train a model on existing resources (especially english ones) and apply it to a new language context. we present a slight but successful modification of the ewiser model @xcite ) in which we merge both approaches. the novelty lies in special data augmentation technique focused on structural properties of knowledge bases in other than english language, namely polish. starting from ewiser language-agnostic architecture pre-trained on english and polish sense annotated datasets, we then propagate dnn vector representations through combined structures of princeton wordnet and polish wordnet, two largest nowadays wordnets in the world. this modification boost the wsd multilingual performance above current state-of-the-art solutions based on multilingual language models e.g. xl-wsd framework @xcite , and gives comparable behaviour to earlier sota model of consec, despite the fact that ewiser architecture -even with our modifications -is much simpler. ## related work the supervised approaches have proved to be the most effective solution to wsd when a representative training sample is available. with recent progress in neural language modeling the supervised solutions have been improved even more and outperformed earlier models on almost every single benchmark. however, the existing wsd data yet has its flaws, including a non-representative training sample for verb, adverb and adjective senses, most frequent sense bias, and limited sense coverage. although very successful, the supervised models are overfitting easily to training samples which harms their generalisation abilities and reduces sense coverage when non-representative samples are used for training @xcite . the knowledge-based solutions were designed to increase the coverage of underrepresented word senses when a limited training sample is available. however, the performance gap between supervised and knowledge-based solutions encouraged the researchers to focus more on former approaches. the prior work on supervised models considered wsd task as token classification problem where the model learns to generate discrete labels representing predicted meanings @xcite @xcite . a typical architecture consisted of neural context encoder and sense discrimination layer e.g. lstm with attention and softmax layer trained on semcor data to disambiguate tokens in a fully supervised manner. recent studies in the area of word sense disambiguation show that the most successful solutions are based on hybrid architectures with a strong emphasis on zero-shot supervision. a zero-shot component was introduced to replace full supervision and improve the ability of generalising to unseen senses @xcite . subsequent approaches utilised the benefits of transformer architectures @xcite and representation learning using external knowledge sources, such as sense definitions @xcite @xcite and sense usage examples. on the other hand, structural properties of lexico-semantic networks used to be ignored in neural architectures. recent studies show that hybrid solutions utilising textual descriptions of senses together with their structural properties can also improve wsd performance. most related to our work is xl-wsd framework with a crosslingual benchmark built on the basis of open multilingual wordnet data and ba-belnet resources. the benchmark has been introduced as a platform to evaluate zero-shot wsd methods and crosslingual transfer with multilingual language models. other multilingual solutions include mulan @xcite , ewiser @xcite , consec @xcite . however, only few of them were actually evaluated against all of datasets available in xl-wsd framework. the usual crosslingual evaluation setting consists of english, spanish, french, german and italian datasets proposed at se-meval competition. xl-wsd was a step towards preparing a crosslingual evaluation at scale including more languages. as far as we know, none of the previous solutions evaluated within xl-wsd framework were hybrid models joining neural text encoders with structural knowledge base features. regarding the negative transfer phenomenon, several studies were focused on identification of troublesome nlp tasks where simultaneous finetuning of multilingual language models to downstream tasks has a harmful impact on model performance @xcite . however, none of them were focused strictly on wsd task. it is an open issue whether negative transfer occurs when fine tuning multilingual language models to wsd task. ## models as a baseline architecture we decided to use ewiser @xcite ) as its codebase is extensible and freely available. ewiser is a supervised hybrid architecture utilising sense annotated corpora and knowledge base structure simultaneously. the model is based on transformer architecture with additional sense discrimination layer and structured logit mechanism injecting structural information into model during training. the key idea is to utilise existing wordnet links between senses to reinforce training procedure and incorporate logit scores of neighboring senses into scoring function of word's candidate meanings. ## experiments in this section we present the results of our experimental part. we decided to split evaluation into two different settings. first, we would like to investigate the impact of underlying language model on wsd performance. the second setting is focused on data augmentation using plwordnet data (the network structure, as well as glosses and examples). ## results and discussion in tests on 15 languages our technique turned out to be successful in beating the xl-wsd and the ewiser model and comparable to some extent text (e.g. a gloss, a definition, a sentence) xlmroberta-large language-agnostic model trainable neural network [𝛔 1 , 𝛔 2 , …, 𝛔 117697 ] 117697d synset probability space wngt semcor the ranking of synsets s 1,w , …, s m,w for the m-sense word w wordnet structural properties t t t 1024d space [a 1 , a 2 , …, a 1023 , a 1024 ] with the consec model. since testing data sets were constructed independently, we decided to compare average model f1 performances. u-mann-whitney paired test was applied to the task, separately for consec and for xl-wsd with ewiser) and p-values were corrected for false discovery ratio through benjamini-hochberg procedure @xcite . our two models performed better on average than xl-wsd (xlmr-l) and ewiser baseline models (for 15 languages) and not worse than consec model (for 6 languages). presented in this paper experiments proved that augmenting english training data sets with glosses and examples from other than english wordnet can lead to the improvement of a multilingual wsd algorithm. the proposed novel technique of augmenting princeton wordnet structure also resulted in better than or equal to sota scores. surprisingly, used here ewiser architecture is simpler than current sota dnn models. this suggests the validity of training data enlargement and curation techniques. the step that could not be fully superseded by constructing new, even more sophisticated 2 the evaluation of consec model was limited to the results provided by the authors in @xcite . at the time of publication, the training procedure was not fully reproducible and the codebase was incompatible with xl-wsd sense indices.