With More Contexts Comes Better Performance: Contextualized Sense Embeddings for All-Round Word Sense Disambiguation (2020)

Abstract: Contextualized word embeddings have been employed effectively across several tasks in Natural Language Processing, as they have proved to carry useful semantic information. However, it is still hard to link them to structured sources of knowledge. In this paper we present ARES (context-AwaRe Embeddings of Senses), a semi-supervised approach to producing sense embeddings for the lexical meanings within a lexical knowledge base that lie in a space that is comparable to that of contextualized word vectors. ARES representations enable a simple 1 Nearest-Neighbour algorithm to outperform state-of-the-art models, not only in the English Word Sense Disambiguation task, but also in the multilingual one, whilst training on sense-annotated data in English only. We further assess the quality of our embeddings in the Word-in-Context task, where, when used as an external source of knowledge, they consistently improve the performance of a neural model, leading it to compete with other more complex architectures. ARES embeddings for all WordNet concepts and the automatically-extracted contexts used for creating the sense representations are freely available at http://sensembert.org/ares .

## introduction contextualized word embeddings have proved to be highly beneficial to the majority of natural language processing tasks @xcite . indeed, language models like bert @xcite , roberta @xcite , xlnet @xcite , etc., enable architectures built on top of them to attain performances that were previously out of reach @xcite . the main reason behind this great success is the fact that contextualized embeddings of words encode the semantics defined by their input context @xcite . indeed, when tested in the word-in-context (wic) task (pilehvar and camacho-collados, 2019), i.e., a binary classification problem where a model has to classify whether a target word is used with the same meaning in two different sentences, contextualized word embeddings placed themselves as the best approaches across the board. nevertheless, these latent representations do not provide any explicit information regarding the meaning expressed by the word in context, hence making it difficult to link texts to structured sources of knowledge such as lexical knowledge bases (lkb). the task of associating a word in context with the most suitable meaning from a predefined sense inventory is better known as word sense disambiguation @xcite , and is usually tackled by two kinds of approach: knowledge-based and supervised ones. on the one hand, knowledgebased approaches @xcite are able to scale across languages since they do not need sense-annotated corpora and rely only on the information within their underlying lkb. on the other hand, supervised models @xcite ) have proved to achieve state-of-the-art results on the english benchmarks by taking advantage of manually-annotated data for the task and machine learning algorithms. however, supervised approaches are mostly focused on english @xcite and have only recently been applied to lower-resourced languages thanks to automatically-produced datasets @xcite @xcite . another effective approach in this direction has been presented by @xcite , who introduced sensembert, a knowledge-based approach to building sense embeddings without relying on sense-annotated data. since it is not tied to semantic annotations, sensembert scales over different languages. however, it is limited to nominal concepts only and provides different representations for the same concepts across different languages, which hinders its applicability to cross-lingual tasks. in this paper we present ares (context-aware embeddings of senses), a semi-supervised approach to producing sense embeddings for all the word senses in a language vocabulary. ares makes up for the paucity of manually-annotated examples for a large portion of words' meanings by coupling the information within a knowledge base with the representational power of a pre-trained language model. this enables reliable representations to be built for those senses not appearing in manually-curated resources, while at the same time enriching the vectors for all the other concepts. we tested our embeddings on the two tasks that measure a model's capabilities to encode word meanings, i.e., wsd and wic. in both tasks, ares representations prove to be of great benefit. in wsd, while employing a simple 1-nearest-neighbour (1-nn) algorithm, they attain state-ofthe-art results on english, even beating dedicated architectures with long and expensive fine-tuning procedures. in wic they lead a simple bertbased model to perform in the same ballpark as other state-of-the-art alternatives which rely on more complex architectures. furthermore, by taking advantage of pre-trained multilingual models we provide unified representations of meanings across languages, which, while using english data only, outperform their competitors and achieve the state of the art on all the languages available in the all-words multilingual wsd tasks, i.e., french, german, italian and spanish. ## related work word sense disambiguation (wsd) is a core task in lexical semantics and has mainly been tackled by two kinds of approach: knowledge-based and supervised ones. knowledge-based methods build upon lexical knowledge bases, such as wordnet @xcite and babelnet @xcite , and employ algorithms on graphs to address the word ambiguity in texts @xcite @xcite . these approaches do not rely on semantically-tagged training data and are hence able to scale over all the languages supported by their underlying knowledge base. nevertheless, they lag behind their supervised counterparts on english in terms of performance. supervised ap-proaches, by framing wsd as a classification task, have acquitted themselves as the state of the art in english @xcite @xcite @xcite , outperforming their knowledge-based competitors by several points. recently, pilehvar and camacho-collados (2019) provided a new declination of wsd, formulating it as a binary classification problem where, given a target word and two contexts, a model has to predict whether the target word is used with the same meaning. this setting has the advantage of not drawing on sense inventories and provides an effective testbed for context-based word embeddings @xcite . contextualized sense representations have recently been employed to compute sense representations that can be applied directly to disambiguation. some of the first approaches of this kind were proposed by @xcite and @xcite , who exploited the semantically-tagged sentences of semcor @xcite and neural language models to create embeddings for the senses in wordnet. similarly, @xcite computed sense embeddings using bert @xcite and the relations in a lexical knowledge base in order to also provide vectors for those meanings that do not appear in semcor. the most recent effort in this direction is sensembert @xcite , which drops the need for sense-annotated corpora by exploiting the babelnet mapping between wordnet senses and wikipedia pages so as to collect contextual information for the senses in wordnet. since it does not rely on manually-annotated data sensembert can scale over different languages, being limited, however, to nominal senses only. in this work we continue along this latter line of research and propose a novel method for producing sense embeddings which, by relying on english data only, also proves to be able to model meanings across languages. rather than leveraging word-net relations as lmms does, ares creates vector representations for all senses by automatically providing usage examples for the synsets within a knowledge base. in contrast to sensembert, instead, ares covers all the four wordnet pos tags, and, at the same time, disposes of the resources required by sensembert, such as nasari and the wikipedia category graph. ## preliminaries we now describe the resources that we use to build ares embeddings. wordnet @xcite is the most used lexical knowledge base for english. it can be viewed as a graph where nodes are concepts, i.e., synsets, and edges are semantic relations between them. each synset contains a set of synonyms, e.g., the synset defined as a natural flow of ground water comprises the lemmas spring, fountain and natural spring. we use the notation {l 1 , . . . , l n } (g) to refer to the concept with gloss g and expressed by the lemmas l 1 , . . . , l n . we define a sense as a lemma-gloss pair, i.e., a meaning that is specific to a given lemma, e.g., fountain-(a natural flow of ground water) is a sense of {spring, fountain, natural spring} (a natural flow of ground water). syntagnet @xcite is a repository containing approximately 88k lexical-semantic collocations, i.e., pairs of wordnet synsets that co-occur more frequently than would be expected. 1 for example, the concepts {coach, bus, autobus} (a vehicle carrying many passengers) and {driver, motorist} (the operator of a motor vehicle) appear in syntagnet as they form a collocation. ukb @xcite is a knowledge-based approach to wsd based on the personalized pagerank algorithm @xcite . we set wordnet as underlying knowledge base, disable the most frequent sense backoff strategy and set the parameters according to @xcite . semcor @xcite is the standard manually-curated corpus for wsd including more than 220k words tagged with 25k distinct word-net meanings, hence providing annotated contexts for roughly 15% of the synsets in wordnet. bert @xcite is a deep neural architecture trained with the masked language model objective. given a text, it provides contextual embeddings for the subtokens therein. we choose bert because it has proven to capture the semantics of a word in context @xcite , while also being able to effectively generalize cross-lingually thanks to its multilingual representations @xcite . 2 ## ares we now introduce ares, a semi-supervised approach for creating sense embeddings that cover all the senses in a language vocabulary. given as input a corpus c of raw sentences and a synset @xmath0, ares operates the following three steps: ## statistics in ## wsd experimental setup we now report the setup of the evaluation we conducted on the english and multilingual wsd tasks. evaluation datasets we carried out the evaluation on the english all-words wsd framework by @xcite , foot_5 comprising five standard test sets, namely, senseval-2 @xcite , senseval-3 @xcite , semeval-07 @xcite , semeval-13 @xcite , semeval-15 @xcite along with all, i.e., the concatenation of all the test sets. as concerns the multilingual evaluation, we considered the latest versions of the two multilingual all-words wsd datasets of semeval-13 @xcite and semeval-15 @xcite , containing test sets for french, german, italian and spanish. 7 we report all results in terms of the f1 score, i.e., the harmonic mean of the precision and recall. 8 ## wsd results we now report the results of the evaluation we carried out on the english and multilingual wsd tasks, along with an ablation study of ares components. ## wic experimental setup in this section we further inspect the properties of our embeddings by measuring the improvements they bring to the word-in-context (wic) task. 15 evaluation dataset we tested on the word-in-context task (pilehvar and camacho-collados, 2019, wic), foot_14 i.e., a binary classification problem where, given a target word w and two contexts c 1 and c 2 , the task is to determine if w occurs with the embedding and the two representations of the target word w in c 1 and c 2 . as additional features, we considered the senses s 1 and s 2 of w in c 1 and c 2 , respectively, that we predicted by means of ares as in section 6. then, we applied a dense layer -which we trained during finetuning -to the ares embeddings of s 1 and s 2 and reduced their dimensionality to 1024. finally, we concatenated the input of the classifier with these two new representations. 17 comparison systems we compared ares against the best performing models on the wic task. we considered three pre-trained language models fine-tuned on wic, i.e., bert large (devlin et al., 2019), roberta @xcite and t5 @xcite , and two language models which leverage external knowledge while pretraining, i.e., knowbert @xcite and sensebert large @xcite . ## wic results in a straightforward manner at finetuning. moreover, bert ares performs better than or on a par with its closest competitors, i.e., knowbert, sensebert large and t5 (large, and 3b), which, instead, rely on more complex architectures, specific pre-training phases and between 3000 m and 40 m more parameters. t5-11b is the only model achieving better results than bert ares , mainly due to the large difference in terms of trainable weights (with t5-11b being 30 times bigger.) ## conclusion in this paper we presented ares, a semisupervised approach for producing embeddings of senses in english and across different languages. ares can couple the information within senseannotated corpora with that automatically created by means of a cluster-based algorithm so as to produce high-quality latent representations for the concepts within a lexical knowledge base. our experiments showed that despite relying on english data only ares outperforms all its alternatives. it achieves state-of-the-art results on both english and multilingual wsd benchmarks, leveraging bert large and mbert, respectively, as underlying pre-trained language models. we further tested our embeddings in the wic task where they lead a baseline neural model to outperform its closest competitors that rely on larger architectures or dedicated pre-training routines. our embeddings computed with bert large and mbert and the automatically-extracted contexts are available at http://sensembert.org/ares . as future work, we plan to exploit the information brought by our embeddings to other downstream tasks, such as multilingual semantic role labeling @xcite and cross-lingual semantic parsing @xcite .