What do Language Models know about word senses? Zero-Shot WSD with Language Models and Domain Inventories (2023)

Abstract: Language Models are the core for almost any Natural Language Processing system nowadays. One of their particularities is their contextualized representations, a game changer feature when a disambiguation between word senses is necessary. In this paper we aim to explore to what extent language models are capable of discerning among senses at inference time. We performed this analysis by prompting commonly used Languages Models such as BERT or RoBERTa to perform the task of Word Sense Disambiguation (WSD). We leverage the relation between word senses and domains, and cast WSD as a textual entailment problem, where the different hypothesis refer to the domains of the word senses. Our results show that this approach is indeed effective, close to supervised systems.

## introduction it is undeniable that language models (lm) have drastically changed the natural language processing (nlp) field @xcite . more recently, those lm have also shown to be capable of performing nlp tasks with just few examples given in the context @xcite , using the so called prompting. one of their particularities, and the key difference with previous approaches, is their contextualized token representation. allowing the model to adopt different representations for words (tokens) depending on the context has supposed a huge advantage when sense disambiguation is required for a given inference. but, to what extent do lm actually know about word senses? in this work, we tried to answer that question by evaluating lms directly on the word sense disambiguation (wsd) task via prompting. word sense disambiguation is the task of identifying the correct sense of a word in a given context. current state-of-the-art on wsd involves finetuning a lm on semcor @xcite to predict the correct among all possible sense glosses of the word in the given context. other methods leverage the contextual representations of lm to perform wsd with a simple k-nn algorithm on the embedding space. lately, the use of domain inventories was proposed to alleviate the high granularity of knowledge-bases @xcite . recent studies that worked on zero-shot wsd refer to the task of predicting the senses of new lemmas not seeing during training as zero-shot @xcite ) wsd, however we aim for a completely zero-shot evaluation, where no annotated data is available for any lemma. despite the knowledge already encoded in the lm, training data is used in one way or another to introduce knowledge about the task. to avoid drawing noisy conclusions, we evaluated the lm as they are, without further fine-tuning on or using any kind of wsd training data. to that end, we prompted lms like bert @xcite and roberta @xcite to perform a task that requires wsd knowledge to be successfully solved. figure 1 shows an example of how a model can be prompted to solve wsd using textual entailment as a proxy. on this example we consider that the word bank has senses from three different domains: geography and places, business, economics and finance and geology and geophysics. the three possible domains are converted to hypothesis using predefined prompts. finally, a supervised textual entailment model is used to perform the inference. more details on of the approach are discussed in section 2. in this work we first evaluated commonly used lms as a zero-shot domain labelers with 3 different domain inventories. then, following @xcite we addressed the wsd using domain inventories and evaluated the lms on them. we showed that lms have some notion of senses as they perform zero-shot wsd significantly better than a random baseline and sometimes close to the supervised state-of-the-art. we also provided different analysis comparing different prompts and performing an error analysis over the two evaluated tasks. ## prompting language models since the past few years, prompting has become the de facto approach to probe language models @xcite . @xcite defined prompting as the practice of adding natural language text, often short phrases, to the input or output to encourage pre-trained models to perform specific tasks. however, due to its wide definition, several different ways of prompting exists, such as instruction based, template-based or proxy-task based. for more information about prompting we encourage the reader to read the @xcite survey. in this work we focused on the proxy-task based approach, more precisely, we made use of the next sentence prediction (nsp) and textual entailment (te) tasks as a proxy. the te is also known as natural language inference (nli), we will use both terms interchangeably. the choice of this approach was made based on previous works on zero-shot domain labelling @xcite . both, nsp and te are sentence-pair classification tasks: the first attempts to predict whether a sentence is followed by another and the second aims to predict if an entailment relation exists between both sentences (premise and hypothesis). figure 2 shows an example of how to perform wsd using nsp or te models. the process can be briefly summarized as follows: (1) for each possible sense s of the target word w we obtain their corresponding domain d using a domain inventory d (domain inventories are discussed in more detail in section 3). (2) predefined prompts are used to generate verbalizations that will serve as possible continuations (on nsp) or hypothesis (on te) h. (3) a pretrained nsp or te model is used to obtain a probability for each sentence/hypothesis and therefore, to each domain. formally, for a te model we defined the probability of word w being from domain d @xmath0: ## domain inventories a domain inventory is a set of domain labels such as health and medicine, culture or business and economics that aims to cover the wider spectrum of domains as possible with a specific granularity level. actually, these domain inventories are used to label synsets from knowledge-bases like wordnet @xcite and babelnet @xcite . examples of wordnet synset annotations from different domain inventories are shown in the babeldomains @xcite is a unified resource that includes domain information for wikipedia, wordnet and babelnet. it inherits the domains from wikipedia domains of knowledge, a total of 34 coarse labels. although it is semi-automatically annotated, two gold standard datasets (for wordnet and wikipedia) are provided for evaluation. coarse sense inventory (csi) @xcite was created to reduce the level of granularity of wordnet synsets while maintaining their expressiveness. it contains a total of 45 labels shared across the lexicon. compared to previous alternatives, csi provided a higher agreement among annotators. also it was already proven to be useful for the wsd task. wordnet domains @xcite ) is a fine-grained domain inventory containing about 160 labels. it is organised in a hierarchical way, from global concepts such as pure_science to specific concepts as oceanography. this inventory provides a domain label to each synset in wordnet. due to the hierarchical nature and fine granularity, in our experiments we kept only the domain labels until the third level, mapping all the labels below to the closest available domain. we end up with 60 domain labels. ## experimental setup in this section we describe the models we evaluated, and the domain labelling and word sense disambiguation tasks we used for evaluation. ## results in this section we discuss the results obtained on each experiment. first we discuss the results obtained on the domain labelling task. then, we show the results from word sense disambiguation. and finally we analyze the correlation between both tasks as they share the label space. are language models able to discriminate domains in sense glosses? figure 4 shows the results obtained for the domain labelling task. as a general overview, the three models obtain decent results considering no data for training was provided. comparing nli models vs the nsp model, we can conclude that nli based models perform better in all cases, in concordance with previous works @xcite . however, additional te data (nli vs nli*) does not seem to be very useful for the task. finally, the results shows that the domain hints in the gloss affects significantly to the performance, specially in wordnet domains, where the labels are very fine-grained. ## related work word sense disambiguation approaches to wsd range from supervised that only use annotated data @xcite @xcite to knowledgebased @xcite @xcite , as well as approaches that combine supervised and knowledge-based approaches @xcite @xcite @xcite . knowledge-based approaches employ graph algorithms on a semantic network @xcite @xcite , in which senses are connected through semantic relations and are described with definitions and usage examples. unfortunately, their independence from annotated data comes at the expense of performing worse than supervised models (pilehvar and navigli, 2014). supervised approaches frame the task as a classification problem and use available annotated data to learn mapping the words in context to senses. before supervised neural models emerged as state of the art in nlp, the task of supervised wsd was performed based on a variety of lexico-syntantic and semantic feature representations that are fed to a supervised machine learning classifier @xcite . instead, current state-of-the-art supervised models rely on the use of pretrained transformers as core architecture of the model. among these models we can find approaches that exclusively use annotated data to learn effective representations of the target word in context and feed it to some classification head @xcite @xcite . some approaches have shown that an effective way to improve sense representation is to exploit the glosses provided by the sense inventories. gloss representation are then incorporated to the sense embedding @xcite , in which the most probable sense is retrieve according to the similarity with the given context. multiple works have been shown effective in wsd such as lmss @xcite , sensembert @xcite , ares @xcite , sref @xcite , ewise @xcite and ewiser @xcite , among many others. glosses have also been exploited in sequence-tagging approaches @xcite , where the task is framed as sequence classification problem @xcite . in a similar manner, @xcite propose a generative approach to cast wsd as sequence classification problem. in adition to glosses, other approaches presented ways to make use of the knowledge encoded in kbs such as wordnet. for instance, @xcite propagate sense embeddings using wordnet as a graph. please refer to @xcite to obtain further details of the recent trends in wsd. prompting language models has changed the paradigm of how language models can be used to extract even more potential from them. initially with very large lm like @xcite and later with smaller ones @xcite prompts allowed the models to perform zero or few-shot classifications with simple natural language. this ability also allowed models to improve performance on data-scarce problems by large margin @xcite @xcite . these prompts can be discrete @xcite close to natural language or continuous @xcite close to other efficient deep learning methods like adapters @xcite . closer to our work, textual entailment @xcite has been used as a source of external supervision to solve several text classification tasks @xcite @xcite @xcite , named entity recognition @xcite @xcite , relation extraction @xcite , event extraction @xcite , event argument extraction @xcite , intent classification @xcite , aspect-based sentiment analysis @xcite and many more. domain inventories. domain information was added to princeton wordnet @xcite since version 3.0. in total 440 topics were represented as a synsets in the graph. the topic label assignment was achieved through pointers from source synsets to target synsets. being the most frequent topic is law, jurisprudence. however, the manual assignment of topic labels to synsets in wordnet is very costly. as a consequence, semi-automatic methods were developed. for instance, wordnet domains @xcite is a semi-automatically annotated domain inventory that labels wordnet synsets with 165 hierarchically organised domains. the use of domain inventories such as wordnet domains, allowed to reduce polysemy degree of wordnet synsets by grouping those that belong to the same domain @xcite . however, far from being perfect, many synsets were labelled as factotum, meaning that the synset cannot be labelled with a particular domain. several works were proposed to improve wordnet domains, such as extended wordnet domains @xcite , that applied graph-based methods to propagate the labels through the word-net structure. domain information is not only available in wordnet, for example iate foot_0 is a european union inter-institutional terminology database. the domain labels of iate are based on the eurovoc thesaurus foot_1 and were introduced manually. more recently, several new domain inventories appeared, such as babeldomains @xcite or coarse sense inventory @xcite . ## conclusions in this work we present an evaluation approach to test language models on the tasks of domain labelling and word sense disambiguation without annotated data requirements. for the wsd task we followed @xcite to reduce the granularity level. our results showed that the language models we tested here have some notion of word senses. they easily outperformed the baseline, and sometimes almost reached to supervised systems performance. in addition, our further analysis shows that there is very low error propagation from domain labelling to wsd as their errors are poorly correlated. for the future, we plan to evaluate larger language models on the task to try to understand to what extent scaling these lms affects to sense recognition.