On the Cross-lingual Transferability of Contextualized Sense Embeddings (2021)

Abstract: In this paper we analyze the extent to which contextualized sense embeddings, i.e., sense embeddings that are computed based on contextualized word embeddings, are transferable across languages. To this end, we compiled a unified cross-lingual benchmark for Word Sense Disambiguation. We then propose two simple strategies to transfer sense-specific knowledge across languages and test them on the benchmark. Experimental results show that this contextualized knowledge can be effectively transferred to similar languages through pre-trained multilingual language models, to the extent that they can out-perform monolingual representations learnednfrom existing language-specific data.

## introduction word sense disambiguation (wsd) is an indispensable component of language understanding @xcite ; hence, it has been one of the most studied long-standing problems in lexical semantics. currently, the dominant wsd paradigm is the supervised approach @xcite , which highly relies on sense-annotated data. similarly to many other supervised tasks, the amount of labeled (sense-annotated) data for wsd highly determines downstream performance. one of the factors that make wsd a challenging problem is that creating sense-annotated data is an expensive and arduous process, i.e., the so-called knowledge-acquisition bottleneck @xcite . moreover, wsd research often focuses on the english language. while datasets for other languages exist @xcite , these are generally automatically generated @xcite @xcite or not large enough for training supervised wsd models @xcite authors marked with a star ( ) contributed equally. the contributions are threefold: (1) we adapt existing datasets to build a unified benchmark for cross-lingual wsd based on wordnet; (2) we test the effectiveness of contextualized embeddings for cross-lingual transfer in the context of wsd; and ## related work this works lies at the intersection of two areas of nlp research: word sense disambiguation and cross-lingual semantic representation. hence, we cover the recent relevant work in the corresponding literature. ## cross-lingual wsd benchmark in order to develop a unified benchmark for crosslingual word sense disambiguation, we opted for princeton wordnet @xcite as our reference sense inventory. thanks to its completeness (covering different parts of speech) and open nature, pwn is regarded as the de facto sense inventory for wsd in english. moreover, the multilingual efforts from open multilingual wordnet @xcite , linked to the english pwn, make this resource prompt to extensions for sense-annotated corpora in other languages. we use pwn v3.0 for all our experiments, including the unified cross-lingual benchmark, converting all datasets to both the same xml schema of @xcite and a practical json format. in the following we describe the datasets used to build the cross-lingual wsd benchmark, with ## methodology we describe two wsd strategies based on contextualized embeddings ( §4.1) and propose adaptations to the cross-lingual setting ( §4.2). figure 1 provides an overview of the overarching methodology. ## evaluation we evaluate the proposed strategies in two different settings, using wsd as our test bed: monolingual (section 5.1) and cross-lingual (section 5.2). experimental setting. we use roberta @xcite for monolingual experiments, and xlm-r @xcite for cross-lingual experiments. 4 these two models are state-of-theart for english and multilingual tasks while sharing a very similar architecture. the most important difference between these models is that while roberta is pre-trained only on english texts, xlm-r is pre-trained on 100 languages (unevenly distributed). we use the large variants of these models (355m parameters for each). 5 all results are measured according to the f-measure. ## conclusions in this paper, we analyzed to what extent contextualized embeddings can be transferred across languages, using wsd as our test bed. to this end, we developed a unified framework that can be used for evaluating cross-lingual models. the first results are encouraging, as they show that multilingual language models can learn contextualized sense embeddings that can be effectively transferred from one language to another, attaining competitive results in wsd with no access to annotated data in the target language or external resources. one limitation of this work is in the nature of languages evaluated, which are all indo-european for which test data was available. as future work it will be interesting to extend this benchmark to languages