Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models (2024)

Abstract: Pretrained Language Models (PLMs) learn rich cross-lingual knowledge and perform well on diverse tasks such as translation and multilingual word sense disambiguation (WSD) when finetuned. However, they often struggle at disambiguating word sense in a zero-shot setting. To better understand this contrast, we present a new study investigating how well PLMs capture cross-lingual word sense with Contextual Word-Level Translation (C-WLT), an extension of word-level translation that prompts the model to translate a given word in context. We find that as the model size increases, PLMs encode more cross-lingual word sense knowledge and better use context to improve WLT performance. Building on C-WLT, we introduce a zero-shot prompting approach for WSD, tested on 18 languages from the XL-WSD dataset. Our method outperforms fully supervised baselines on recall for many evaluation languages without additional training or finetuning. This study presents a first step towards understanding how to best leverage the cross-lingual knowledge inside PLMs for robust zero-shot reasoning in any language.

## introduction pretrained language models (plms) perform many cross-lingual tasks without explicit crosslingual training signal, including word-level translation (wlt) across languages @xcite . these models also demonstrate crosslingual knowledge when finetuned for the word sense disambiguation (wsd) @xcite . however, the extent to which word sense knowledge comes from pretraining rather than finetuning is unclear: many plms struggle to disambiguate word sense when formulated as a binary classification task, the most common word sense setup for prompting language models @xcite . to investigate this, we measure the ability of multilingual autoregressive language models to understand the cross-lingual meaning of words in a given context. specifically, we extend the wlt task setup to include a specific context in the prompt, which we call contextual word-level translation (c-wlt). we empirically show that pretrained language models leverage contextual information in the prompt to improve wlt performance. in addition, both english and multilingual plms perform better on the contextual wlt tasks as model size increases, demonstrating improved cross-lingual knowledge at scale. translations of a word that change based on context are frequently due to differing word senses not shared by an analogous word in the target language @xcite . inspired by this, we apply c-wlt to the task of wsd by translating the ambiguous word w in context with wlt and then assigning w with the senses in the overlap of the translated word's sense set with w's senses (figure 4 , left). we test this zero-shot approach for wsd on 18 languages from the xl-wsd dataset @xcite . in our best setting, zero-shot wsd via c-wlt prompting outperforms prior supervised works on recall for many evaluation languages, even though our method requires no additional training on labeled wsd data. we also observe that ensembling diverse target languages with this method narrows down the predicted set of senses, as demonstrated by the improvements in jaccard similarity with the reference set. finally, we analyze our design choices and the types of errors made by this approach to better understand the behavior of wsd via c-wlt and how it relates to supervised wsd classification. the overall findings of this work are as follows: in sum, we evaluate the lexical translation skills of plms in context, and we present a first step towards applying that skill to the downstream task of wsd. given that most wsd training data outside of english are automatically created (e.g., @xcite , and that annotating gold data incurs significant costs for each new language, zero-shot approaches such as our proposed wsd via c-wlt approach are crucial for improving wsd in low-resource languages. ## contextual word-level translation a standard method of evaluating the cross-lingual capabilities of plms is the task of a word-level translation (wlt), where the model is prompted to translate a word w s from a source language l s into another target language l t (gonen et al., 2020). however, this setup does not consider variations in the translation of w s into l t that occur when the surface form of w s represents multiple meanings (i.e., senses) in different contexts. we propose an extension of the word-level translation task, contextual word-level translation (c-wlt), which requires translating words correctly based on how they are used in a given context (figure 4 , right panel). specifically, we prompt the plm to translate w s from l s into l t when conditioned on a specific context c s where w @xmath0; we then measure whether it produced the correct translation(s) w t in context of w s . for example, if we want to translate "plant" into chinese based on the context sentence "the plant sprouted a new leaf", we prompt the plm with in the sentence "the plant sprouted a new leaf", the word "plant" is translated into chinese as __. this evaluation allows us to quantify a plm's ability to align meaning across languages in a contextspecific manner. ## zero-shot word sense disambiguation via c-wlt building on the intuition from the previous section that contextual word-level translation can differentiate between different meanings of a word in the source language, we apply c-wlt to the task of multilingual word sense disambiguation (figure 4 ). specifically, we propose a two-step process wherein we (1) prompt the plm for c-wlt to translate the ambiguous target word, w, in the relevant context and (2) disambiguate w based on the senses of its translation. for instance, to disambiguate the word "plant" as it is used in the context "the plant sprouted a new leaf", we first prompt the plm to translate "plant" into the chosen target language (e.g., chinese) with the c-wlt setup from the previous section. we then take the plm's top translation (in this case, "植物") and obtain its senses from a multilingual word sense ontology. we then label the example with the senses shared by "plant" and "植物". ## multilingual wsd results and analysis we first present the performance of our method for multilingual wsd on the two automatic metrics, recall and jaccard index, and compare this approach to prior work on this task (section 4.1). we then consider the effect of ablating different modeling choices on our method (such as the choice of target language for c-wlt and prompt language; section 4.2), and we analyze the types of errors the approach produces more closely (section 4.3). ## discussion and related work we first analyze the performance of plms in the new contextual word-level translation (c-wlt) setting to evaluate how well these models produce context-sensitive lexical translations. other related work has instead tested the efficacy of prompting multilingual plms for sentence-level translation, such as @xcite and @xcite . notably, @xcite observe incorrect language prediction with multilingual plms, similar to our findings in section 4.2. we then apply the c-wlt setup to zero-shot multilingual wsd. this approach builds on @xcite , which highlights the role of multilingual language models in addressing the knowledge acquisition bottleneck problem in wsd. other works have proposed different finetuning improvements to perform wsd better cross-lingually @xcite . unlike these approaches, our method does not require annotated training data, allowing it to generalize easily. our proposed method is, to be best of our knowledge, the first attempt to apply large-scale autoregressive plms to word sense classification via in-context learning. prior work on word sense prompting frames wsd as a binary classification task comparing a word's meaning in two contexts @xcite . more generally, wsd is closely related to and motivated by machine translation; @xcite outlines the relationship between lexical translation and wsd. a commonly proposed use case of wsd systems is to improve the translation of ambiguous words in mt; as such, multiple methods to incorporate word sense information (such as sense embeddings) into nmt systems have been proposed (e.g., @xcite . furthermore, word sense knowledge has been used to evaluate nmt systems @xcite . prior work has also leveraged mt systems and data to improve an underlying wsd classifier @xcite and automatically annotate wsd data @xcite @xcite @xcite . we build on this latter line of work's intuition to extrapolate word senses from the translations of ambiguous words in context. ## conclusion in this work, we examine the ability of pretrained language models to utilize contextual information in cross-lingual settings. specifically, we propose contextual word-level translation (c-wlt) and test different plms' ability to improve lexical translations in context. we then propose a zero-shot prompting technique for multilingual wsd, using c-wlt as a component. our experiments show the method's effectiveness on 18 languages, including those not included in the plm's pretraining. the performance of wsd via c-wlt relies on the relationship between pretraining languages, model size, and the choice of the target language: smaller multilingual plms are more effective for languages on which they have been pretrained but are more sensitive to design choices, lacking the broad applicability of their larger english counterparts. future research examining these interactions and their tradeoffs more closely is vital for improving zero-shot wsd approaches and building better cross-lingual applications of plms in general.