C lu BERT : A Cluster-Based Approach for Learning Sense Distributions in Multiple Languages (2020)

Abstract: Knowing the Most Frequent Sense (MFS) of a word has been proved to help Word Sense Disambiguation (WSD) models significantly. However, the scarcity of sense-annotated data makes it difficult to induce a reliable and high-coverage distribution of the meanings in a language vocabulary. To address this issue, in this paper we present CluBERT, an automatic and multilingual approach for inducing the distributions of word senses from a corpus of raw sentences. Our experiments show that CluBERT learns distributions over English senses that are of higher quality than those extracted by alternative approaches. When used to induce the MFS of a lemma, CluBERT attains state-of-the-art results on the English Word Sense Disambiguation tasks and helps to improve the disambiguation performance of two off-the-shelf WSD models. Moreover, our distributions also prove to be effective in other languages, beating all their alternatives for computing the MFS on the multilingual WSD tasks. We release our sense distributions in five different languages at https://github.com/SapienzaNLP/clubert .

## introduction word sense disambiguation (wsd) is the task of associating a word in context with a meaning from a given inventory of senses @xcite . it resides at the core of natural language processing and has been proved to be beneficial to different downstream tasks, e.g., information extraction @xcite and machine translation @xcite . current approaches to wsd can mainly be divided into supervised and knowledge-based methods. while the former leverage manually-annotated data to train statistical models, the latter exploit the knowledge enclosed within a semantic network to identify the most appropriate meaning of a word in context. both kinds of approach, however, suffer from the knowledge acquisition bottleneck problem @xcite . in fact, since words and senses follow a zipfian distribution @xcite , information on rare words and meanings is scarce in both semantically-annotated data and knowledge bases. this undermines the ability of supervised and knowledge-based approaches to deal with words unseen at training time, or that have only a few connections within a semantic network. to overcome this limitation, the most frequent sense (mfs) backoff strategy, i.e., tagging a word with its meaning that has been manually annotated as the most frequent one, is employed by both approaches. nevertheless, while the mfs proved to be a strong baseline in the general-domain setting of wsd, it does not scale over specific domains @xcite and its applicability is limited to languages where annotated data are available, i.e., english. furthermore, the way words and meanings are used changes over time, hence making old annotations unreliable. this is the case with wordnet @xcite , i.e., the most used electronic english dictionary in wsd. word-net provides information about sense frequency that is either manually-annotated or derived from semcor @xcite , i.e., a corpus where words are manually tagged with wordnet meanings. however, neither wordnet nor semcor have been updated in the past 10 years, thus making their information about sense frequency outdated. for example, the wordnet most frequent sense for the noun pipe is its smoking device meaning, although, nowadays, one would expect the metal pipe sense to appear more often in general. to overcome some of the aforementioned limitations, different approaches to automatically extracting the distribution of senses have been proposed @xcite . however, these fail to match the wordnet mfs performance and are either dependent on bilingual corpora @xcite , or limited to nouns only @xcite . in this paper, we present clubert, a multilingual cluster-based approach that automatically induces the distribution of word senses from a corpus of raw sentences without relying on manuallyannotated data. by exploiting the assumption that similar meanings appear in similar contexts @xcite and the representational power of bert @xcite , clubert can learn distributions that are of better quality -according to both intrinsic and extrinsic evaluation -than those extracted either by its competitors, or from manually-curated resources. furthermore, our approach outperforms its alternatives in all multilingual and most domain-specific wsd test sets. finally, when used as backoff strategy of a wsd architecture, our automatically-induced distributions are shown to lead the underlying model to higher results than when using the standard manuallycurated distributions of wordnet, hence placing themselves as a better and more flexible alternative. ## related work word sense disambiguation (wsd) is a longstanding problem in natural language processing which was first formulated to address the ambiguity of words in the context of machine translation @xcite . nowadays, wsd models can be mainly divided in two groups: knowledgebased and supervised. knowledge-based methods @xcite @xcite rely on the information enclosed within a semantic network such as wordnet @xcite , a manually-curated resource organised in a graph structure where nodes are concepts and edges are semantic relations between them, or babelnet @xcite , a large multilingual knowledge base where synsets are lexicalised in more than 250 languages. since knowledge-based approaches do not rely on semantically-annotated corpora, they can easily scale over different languages as long as their underlying semantic network supports them @xcite @xcite . nevertheless, these approaches struggle to remain competitive on english when compared to supervised methods. supervised approaches, instead, take advantage of sense-annotated data and frame the wsd task as a classification problem, where each word has its own set of labels, i.e., its possible meanings according to a given sense inventory. ranging from word-based approaches, where a single svm classifier is specialised in disambiguating only one word in a sentence @xcite @xcite , to more general neural architectures that classify all the words together @xcite @xcite , supervised methods have proved to outperform their knowledge-based counterparts whenever annotated data are available @xcite . despite the progress and the increment in the overall performance, both kinds of approach still rely, most of the time, on the most frequent sense heuristic whenever a word does not appear tagged in the training set, or the confidence score of its disambiguation is lower than a threshold. the mfs baseline, in fact, has proved to be very competitive @xcite ), yet, it is limited to words and senses comprised in a manually-annotated corpus such as semcor @xcite . to cope with this limitation, several works have been proposed over the years to automatically learn the most frequent sense of a word. a seminal work in this direction was that of @xcite , where a thesaurus and the distributional similarity between words were used to find the predominant meaning of a given lemma. more recent works, instead, have focused on inducing the full distribution over the senses of a given word. @xcite exploited topic modelling techniques, whereas @xcite presented two multilingual approaches that provided full distributions over nominal senses, not only for english, but also for words in other languages. the work we propose in this paper stands out from previous approaches, exploiting for the first time, to the best of our knowledge, bert contextualized embeddings together with a knowledgebased wsd model to compute the distribution of word meanings. our approach is not tied to any specific language and can potentially be applied to all languages supported by both bert (104) and babelnet (more than 280). ## clubert in this section, we present clubert, a multilingual approach for computing the distribution of cluster 1 the working of glass requires lower temperatures. vitrinite has a shiny appearance resembling glass. most of the roof and walls are made out of glass. ## experimental setup we now present a battery of experiments to assess the quality of our induced sense distributions on both intrinsic and extrinsic evaluation tasks. first, we set the parameters of the model, namely, the sense inventory, the corpus, the number of words to retain in each bag of words, and the number of clusters to create for each lemma. then, we evaluate our automatically-induced distributions intrinsically, by computing their distance in comparison to a manually-annotated distribution, and extrinsically, on the standard english and multilingual word sense disambiguation tasks. system parameters as sense inventory, we use all the synsets in babelnet that also contain a sense from wordnet. concerning the corpus, we use wikipedia 7 since it is freely available and covers more than 300 languages and most of the semantic domains. as regards the number of clusters for a given lemma l, we set the parameter k of the kmeans algorithm to the number of l's meanings in babelnet. finally, we tune the number of words n to retain within each cluster's bag of words by manually evaluating the quality of the disambiguation step (see section 3.2) when varying n between 5 and 20 with a 5 step and set @xmath0. we compute the distributions for all the lemmas in english, italian, spanish, french and german which have at least one corresponding synset within the sense inventory. comparison systems we compare clubert with the most recent and best-performing automatic and manual approaches for sense-distribution learning and mfs detection. as regards the automatic methods for inducing sense distributions, we consider the two knowledge-based and multilingual approaches proposed by @xcite , i.e., endi and dad, and the topic modelling-based approach proposed by @xcite , i.e., lexsemtm. we also compare against three other approaches specialised in identifying the mfs of a word, namely, comp2sense @xcite , which exploits the distance between a word and a sense in a knowledge base, and wct-vec @xcite and umfs-we @xcite , which, instead, leverage the distance between words and sense embeddings. as for the manually-annotated competitors, we compare against the sense distributions and the mfs of wordnet @xcite . these are both determined by the frequency of the senses in semcor @xcite , when possible, and by manual annotations of the synsets' ranks, otherwise. concerning the multilingual evaluation, instead, we compare clubert with endi, dad and the babelnet mfs, which computes the mfs for a given lemma by taking its highest ranked sense according to babelnet. ## intrinsic evaluation in this section we estimate the quality of our automatically-induced sense distributions by comparing them to gold standard ones. we use the dataset proposed by @xcite which, 7 we used the june 2019 dump. contains 50 distinct lemmas annotated with a gold distribution over their senses. hence, we compare the distributions for the target lemmas induced by clubert and its competitors with the manuallyannotated ones. ## extrinsic evaluation in this section we evaluate clubert's distributions on the english, domain-specific and multilingual all-words wsd tasks. to this end, we leverage the sense distributions to extract a lemma's most frequent sense (mfs), which is then used to annotate each occurrence of the lemma in the test sets. in addition, we also integrate clubert mfs into two off-the-shelf wsd models and measure its impact. ## conclusions in this paper we presented clubert, an automatic multilingual approach which induces the distribution of word senses in an arbitrary input corpus by exploiting the contextual information coming from bert and the lexical-semantic knowledge available in babelnet. clubert attains state-of-the-art results on both intrinsic and extrinsic evaluations, also beating the widely-used and manually-curated wordnet mfs. when considering input corpora that come from specific domains, clubert showed an unmatched nimbleness in shaping the distributions accordingly, hence outperforming its manual and automatic competitors on most domains. similarly, our approach demonstrated its ability to scale well on different languages, attaining state-of-the-art results on the multilingual wsd tasks. finally, when injecting clubert mfs into off-the-shelf wsd models, we showed that it brings greater benefits than the wordnet mfs. we release the sense distributions in five different languages at https://github.com/sapienzanlp/clubert . as future work, we plan to refine our approach by exploiting other strategies for weighting the words in the clusters and to leverage them for automatically building multilingual sense-tagged corpora.