Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations (2022)

Abstract: In this paper, we advocate for using large pre-trained monolingual language models in cross lingual zero-shot word sense disambiguation (WSD) coupled with a contextualized mapping mechanism. We also report rigorous experiments that illustrate the effectiveness of employing sparse contextualized word representations obtained via a dictionary learning procedure. Our experimental results demonstrate that the above modifications yield a significant improvement of nearly 6.5 points of increase in the average F-score (from 62.0 to 68.5) over a collection of 17 typologically diverse set of target languages. We release our source code for replicating our experiments at https://github.com/begab/sparsity_makes_sense .

## introduction word sense disambiguation (wsd) is a longstanding and fundamental problem of natural language processing, known to be affected by the knowledge acquisition bottleneck @xcite . large pre-trained neural language models are known to effectively mitigate the problems related to the paucity of high quality, large-coverage sense annotated training data for wsd @xcite @xcite . most recently, the knowledge acquisition bottleneck has been identified as an immense problem in the cross-lingual setting as well @xcite . a straightforward solution for handling this problem is to apply large multilingual pre-trained language models in a zero-shot setting, however, this approach has a potential limitation owing to the curse of multilinguality @xcite , i.e., the inability of such models to handle the large number of languages involved during training such models to an equally good quality. the research community replied to the limitations of large massively multilingual models by developing language-specific monolingual language iso huggingface model identifier bg deeppavlov/bert-base-bg-cs-pl-ru-cased @xcite ca plantl-gob-es/roberta-base-ca @xcite da maltehb/danish-bert-botxo de bert-base-german-cased es dccuchile/bert-base-spanish-wwm-cased @xcite et embeddia/finest-bert (ulčar and robnik-šikonja, 2020) eu ixa-ehu/berteus-base-cased @xcite fr camembert-base @xcite gl dvilares/bertinho-gl-base-cased @xcite hr embeddia/crosloengual-bert (ulčar and robnik-šikonja, 2020) hu sztaki-hlt/hubert-base-cc @xcite it musixmatch/umberto-commoncrawl-cased-v1 ja cl-tohoku/bert-base-japanese-whole-word-masking ko snunlp/kr-bert-char16424 nl gronlp/bert-base-dutch-cased @xcite sl embeddia/sloberta zh bert-base-chinese with the prevalence of large monolingual pretrained models, the important research question arises if their language-specific nature can be successfully exploited during zero-shot learning. our research provides a thorough comparison of the application of large multilingual and monolingual pre-trained language models for zero-shot wsd. another crucial aspect that we carefully investigate in this paper is the integration of sparse contextualized word representations into cross-lingual zero-shot wsd. sparse word representations have a demonstrated ability to align with word senses @xcite . while the benefits of employing sparsity has been shown for wsd in english @xcite , its viability in the cross-lingual setting has not yet been verified. in order to conduct such an analysis, we propose an algorithm for obtaining cross-lingual sparse contextualized word representations from independently trained monolingual language models. ## related work the analysis and the investigation of the transfer capabilities of large pre-trained language models (such as mbert or xlm) across languages has spurred significant research interest @xcite @xcite . in contrast to the availability of multilingual neural language models, a series of recent papers have argued for the creation of dedicated neural language models for different languages (see e.g. wsd has been a fundamental and challenging problem in nlp for many decades, dating back to @xcite . the utilization of contextualized word representations was first advocated by @xcite , later popularized by @xcite . @xcite offers a survey of the recent approaches. most recently, @xcite have explored the usage of multilingual language models (xlm) in zero-shot wsd. while the experiments in @xcite cover four related target languages (german, spanish, french and italian), our investigation involves a typologically diverse set of 17 target languages (beyond english) from @xcite . our work also extends that line of research in important aspects, as we show that the application of monolingual neural language models can vastly improve the performance of crosslingual zero-shot wsd. additionally, we also provide a careful evaluation of sparse contextualized word representations in zero-shot wsd. berend (2020a) introduced sparse contextualized word representations via the application of dictionary learning, and showed that sense representations that are obtained from the co-occurrence statistics of the sparsity structure of the contextualized word representations and their sense annotations can provide significant improvement in monolingual wsd. our work relates to that line of research by providing a mapping-based procedure, which enables the usage of such sense representations created in some source language to be applied in other target languages as well. the kind of mapping we employ can be viewed as a generalization of the approach introduced in @xcite with the notable exception that in this work, we obtain sparse word representations for contextualized models as opposed to static word embeddings. ## methodology in order to allow for zero-shot transfer between monolingual language models pre-trained in isolation from each other, we need to determine a mapping between their hidden representations. we first introduce our methodology for doing so, then we integrate this to the creation of sparse contextualized word representations. ## experimental results all the neural language models that we relied on during our experiments were obtained from the transformers library @xcite . we used four nvidia titan 2080 gpus for our experiments. as the multilingual language model, we used the 24-layer transformer architecture, xlm-roberta (xlm-r for short) @xcite . we chose the cased bert @xcite large model as the monolingual model for encoding english text. as for the rest of the monolingual language models involved in our experiments, we relied on the models listed in for evaluation purposes, we used the extra-large cross-lingual evaluation benchmark xl-wsd, recently proposed in @xcite . the database contains a high-quality sense annotated corpus for english as the concatenation of the sem-cor dataset @xcite and the sense definitions and example sentences from wordnet @xcite . xl-wsd uses the unified crosslingual sense inventory of babelnet @xcite . the dataset contains 17 additional typologically diverse languages besides english (that we listed in throughout the application of sparse contextualized representations, we employ the same set of hyperparameters that were used in @xcite , i.e., we set the number of the regularization coefficient to λ = 0.05 and the number of (sparse) coordinates to @xmath0. there made one optional change, i.e., we decided whether to use the normalization of pmi values @xcite during the calculation of the sense representation matrix φ on a per language basis based on development set performances. an ablation study related to the (optional) normalization of pmi scores is reported in when we do not employ the sparsification of the contextualized word representations for determining the sense representations, we follow the approach introduced in @xcite . that is, we take the centroid of word vectors belonging to a particular sense as the representation of that sense, and perform a nearest neighbor search during inference. ## conclusions in this paper we provided a systematic investigation of the benefits of using large monolingual pretrained language models in place of multilingual language models, such as xlm-r. we have shown that since monolingual neural language models are specifically tailored for a single (or at most a few related) languages, they can effectively mitigate the curse of multilinguality typical of multilingual models, and their application can significantly improve the f-scores in zero-shot wsd. we additionally showed that the benefits of sparse con-textualized word representations, obtained via a dictionary learning procedure, also convey to the cross-lingual setting, and that it provides complementary improvements to the usage of monolingual neural language models. zeyu yun, yubei chen, bruno olshausen, and yann le-cun. 2021. transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. in proceedings of deep learning inside out (deelio): the 2nd workshop on knowledge extraction and integration for deep learning architectures, pages 1-10, online. association for computational linguistics.