Large Scale Substitution-based Word Sense Induction (2022)

Abstract: We present a word-sense induction method based on pre-trained masked language models (MLMs), which can cheaply scale to large vocabularies and large corpora. The result is a corpus which is sense-tagged according to a corpus-derived sense inventory and where each sense is associated with indicative words. Evaluation on English Wikipedia that was sense-tagged using our method shows that both the induced senses, and the per-instance sense assignment, are of high quality even compared to WSD methods, such as Babelfy. Furthermore, by training a static word embeddings algorithm on the sense-tagged corpus, we obtain high-quality static senseful embeddings. These outperform existing senseful embeddings methods on the WiC dataset and on a new outlier detection dataset we developed. The data driven nature of the algorithm allows to induce corpora-specific senses, which may not appear in standard sense inventories, as we demonstrate using a case study on the scientific domain.

## introduction word forms are ambiguous, and derive meaning from the context in which they appear. for example, the form "bass" can refer to a musical instrument, a low-frequency sound, a type of voice, or a kind of fish. the correct reference is determined by the surrounding linguistic context. traditionally, this kind of ambiguity was dealt via word sense disambiguation (wsd), a task that disambiguates word forms in context between symbolic sense-ids from a sense inventory such as wordnet @xcite or, more recently, babelnet @xcite . such sense inventories rely heavily on manual curation, are labor intensive to produce, are not available in specialized domains and inherently unsuitable for words with emerging senses. 1 this can be remedied by word sense induction (wsi), a task where the input is a given word-type and a corpus, and the output is a derived sense inventory for that word. then, sense disambiguation can be performed over the wsi-derived senses. the introduction of large-scale pre-trained lms and masked lms (mlm) seemingly made wsi/wsd tasks obsolete: instead of representing tokens with symbols that encode sense information, each token is associated with a contextualized vector embeddings that captures various aspects of its in-context semantics, including the word-sense. these contextualized vectors proved to be very effective as features for downstream nlp tasks. however, contextualized embeddings also have some major shortcomings: most notably for our case, they are expensive to store (e.g. bert embeddings are 768 or 1024 floating point numbers for each token), and are hard to index and query at scale. even if we do manage to store and query them, they are not interpretable, making it impossible for a user to query for a particular sense of a word without providing a full disambiguating context for that word. for example, consider a user wishing to query a dataset for sentences discussing oracle in the mythology-prophet sense, rather than the tech company sense. it is not clear how to formulate such a query to an index of contextualized word vectors. however, it is trivial to do for an index that annotates each token with its derived sense-id (in terms of ui, after a user issues a query such as "oracle", the system may show a prompt such as "did you mean oracle related to ibm; sun; microsoft, or to prophet; temple; queen", allowing to narrow the search in the right direction). @xcite show how contextualized embeddings can be used for achieving state-of-the-art wsi results. the core idea of their wsi algorithm is based on the intuition, first proposed by @xcite , that occurrences of a word that share a sense, also share in-context our main contribution in this work is proposing a method that scales up amrami and goldberg (2018)'s work to efficiently annotate all tokens in a large corpus (e.g. wikipedia) with automatically derived word-senses. this combines the high-accuracy of the mlm-based approach, with the symbolic representation provided by discrete sense annotations. the discrete annotations are interpretable (each sense is represented as a set of words), editable, indexable and searchable using standard ir techniques. we show two applications of the discrete annotations, the first one is senseaware information retrieval ( §7), and the second is high-quality senseful static word embeddings we can derive by training a static embeddings model on the large sense annotated corpus ( §8). we first show how the method proposed by amrami and goldberg (2018) can be adapted from deriving senses of individual lemmas to efficiently and cheaply annotating all the corpus occurrences of all the words in a large vocabulary ( §3). deriving word-sense clusters for all of english wikipedia words that appear as single-token words in bert-large's @xcite vocabulary, and assigning a sense to each occurrence in the corpus, required 100 hours of cheap p100 gpus (5 hours of wall-clock time on 20 single gpu machines) followed by roughly 4 hours on a single 96-cores cpu machines. the whole process requires less than 50gb of disk space, and costs less than 150$ on google cloud platform. after describing the clustering algorithm ( §4), we evaluate the quality of our system and of the automatic sense tagging using semeval datasets and a new manually annotated dataset we created ( §5). we show that with the produced annotated corpora it is easy to serve sense-aware information retrieval applications ( §7). another immediate application is feeding the sense-annotated corpora to a static embedding algorithm such as word2vec @xcite , for deriving sense-aware static embeddings ( §8). this results in state-of-theart sense-aware embeddings, which we evaluate both on an existing wic benchmark (pilehvar and camacho-collados, 2019) and on a new challenging benchmark which we create ( §9). in contrast to wsd which relies on curated sense inventories, our method is data-driven, therefore resulting senses are corpus dependent. the method can be applied to any domain for which a bertlike model is available, as we demonstrate by applying it to the pubmed abstracts of scientific papers, using scibert @xcite . the resulting senses cover scientific terms which are not typically found in standard sense inventories ( §6). figure 1 shows examples of induced senses for selected words from the english wikipedia corpus. for each sense we list 5 communitybased representatives ( §3), as well as the 5 closest neighbours in the sense-aware embedding space ( §8). additional examples are available in appendix a. code and resources are available in github.com/allenai/wsiatscale. an alternative approach for sense tagging is based on word sense disambiguation (wsd). the two main wsd methods are supervised wsd and knowledge-based wsd. supervised wsd suffers from the difficulty of obtaining an adequate amount of annotated data. indeed, even semcor, the largest manually annotated tagged corpus, consists of only 226,036 annotated tokens. among different supervisied wsd methods, @xcite suggested a svm based approach and @xcite ; @xcite suggested lstms paired with nearest neighbours classification. knowledgebase wsd @xcite , on the other hand, avoids the reliance on large annotated word-to-sense corpus and instead maps words to senses from a closed sense inventory (e.g. wordnet @xcite , babelnet @xcite ). as such, the quality of knowledge-based wsd heavily depends on the availability, quality and coverage of the associated annotated resources. sense embeddings in §8 we exploit the sense-induced corpus to train sense embeddings. @xcite were the first to suggest creating multiple representations for ambiguous words. numerous recent papers @xcite @xcite @xcite aim to produce similar embeddings, all of which use either wordnet or babelnet as semantic network. our method is similar to @xcite , with the difference being that they rely on semantic networks (via babelfy @xcite ). in contrast and similarly to us, @xcite does not rely on lexical resources such as wordnet. the authors proposed splitting pretrained embeddings (such as word2vec) to a number of prototype senseembeddings. yet in our work, we directly learn the multi-prototype sense-embeddings which is only possible due to the large-scale corpus annotation. when comparing both methods in §9.1 we infer it is better to directly learn multi-prototype senseembeddings. ## sense clustering algorithm we replace the hierarchical clustering algorithm used by @xcite with a community-detection, graph-based clustering algorithm. one major benefit of the community detection algorithms is that they naturally produces a dynamic number of clusters, and provide a list of interpretable discrete representative lemmas for each cluster. we additionally found this method to be more stable. graph-based clustering for word-sense induction typically constructs a graph from word occurrences in the final step of the algorithm. or collocations, where the goal is to identify sensespecific sub-graphs within the graph that best induce different senses @xcite . we instead construct the graph based on word substitutes. following jurgens (2011), we pose identifying sense-specific clusters as a community detection problem, where a community is defined as a group of connected nodes that are more connected to each other than to the rest of the graph. graph construction for each word w in the vocabulary, we construct a graph g @xmath0, e w ) where each vertex @xmath1, and an edge (u, v) ∈ e w connects substitutes that are predicted for the same instance. the edge is weighted by the number of instances in which both u and v were predicted. more formally, let @xmath2, and community detection a community in a subgraph corresponds to a set of tokens that tend to co-occur in top-k substitutes of many instances, and not co-occur with top-k substitutes of other instances. this corresponds well to senses and we take community's nodes as sense's representatives. we identify communities using the fast "louvain" method @xcite . briefly, louvain searches for an assignment of nodes to clusters such that the modularity score q-which measures the density of edges inside communities compared to edges between communities-is maximized: ## intrinsic evaluation of clustering algorithm we start by intrinsically evaluating the wsi clustering method on: (a) semeval 2010 and semeval 2013; and (b) a new test set we develop for largescale wsi. in section 9, we additionally extrinsically evaluate the accuracy of static embeddings derived from a sense-induced wikipedia dataset. when collecting word-substitutes, we lemmatize the top-k list, join equivalent lemmas, remove stopwords and the target word from the list, and keep the top-5 remaining lemmas. ## application to scientific corpora a benefit of a wsi approach compared to wsd methods is that it does not rely on a pre-specified sense inventory, and can be applied to any corpus for which a bert-like model is available. thus, in addition to the wikipedia dataset that has been presented throughout the paper, we also automatically induce senses over a corpus of 31 million pubmed abstracts, foot_7 using scibert @xcite . as this dataset is larger than the wikipedia dump, the process required roughly 145 gpu hours and resulting in 14, 225 sense-annotated lemmas, with an average number of 2.89 senses per lemma. this dataset highlights the data-driven advantages of sense-induction: the algorithm recovers many senses that are science specific and are not represented in the wikipedia corpora. while performing a wide-scale evaluation of the scientific wsi is beyond our scope in this work, we do show a few examples to qualitatively demonstrate the kinds of induced senses we get for scientific texts. for each of the words mosaic, race and swine we show the induced clusters and the top-5 cluster representatives for each cluster. while senses mosaic 0 (the common mosaic virus of plants) and mosaic 2 ("something resembling a mosaic", "mosaic of..") are represented in wikipedia, senses mosaic 1 (the mosaic genetic disorder) and mosaic 3 (mosaic is a quality, e.g., "mosaic border", "mosaic pattern") are specific to the scientific corpora (the wikipedia corpora, on the other hand, includes a sense of mosaic as a decorative art-form, which is not represented in pubmed). ## sense-aware information retrieval an immediate application of a high quality sensetagged corpus is sense-aware retrieval. we incorporate the sense information in the spike extractive search system (shlain et al., 2020) 9 for wikipedia and pubmed datasets. when entering a search term, suffixing it with @ triggers sense selection allowing to narrow the search for the specific sense. consider a scientist looking for pubmed occurrences of the word "swine" in its influenza meaning. as shown in figure 3 , this can be easily done by writing "swine@" and choosing the second item in the resulting popup window. the outputs are sentences with the word "swine" in the matching sense. as far as we know, spike is the first system with such wsi capabilities for ir. similarly, @xcite suggested to enhance ir with sense information, but differently from us, this is done by automatically tagging words with senses from a predefined inventory. ## sense-aware static embeddings learning static word embeddings of senseambiguous words is a long standing research goal @xcite . there are numerous real-world tasks where context is not available, precluding the use of contextualized-embeddings. these include outlier detection @xcite , term set expansion @xcite the hypernymy task @xcite , etc. additionally, static embeddings are substantially more efficient to use, can accommodate larger vocabulary sizes, and can accommodate efficient indexing and retrieval. yet, despite their flexibility and success, common word embedding methods still represent ambiguous words as a single vector, and suffer from the inability to distinguish between different meanings of a word @xcite . using our sense-tagged corpus we suggest a simple and effective method for deriving sense-aware static embeddings: we run an off-the-shelf embedding algorithm, foot_9 on the corpus where single-token words are replaced with a concatenation of the word and its induced sense (e.g. "i caught a bass." becomes "i caught@0 a bass@2."). this makes the embedding algorithm learn embeddings for all senses of each word out-of-the-box. 11 an integral property of the embedding algorithm is that it represents both the sense-annotated tokens and the other vocabulary items in the same embedding space - this helps sense inferring about words that are represented in the mlm as multi-tokens words (even though these correspond to less-frequent and often less ambiguous words @xcite @xcite ). for example, in the top-5 nearest neighbours for the different bass senses as shown below, smallmouth and pumpkinseed, multi-token words in bertlarge's vocabulary, are close neighbours the bass instances that correspond to the fish sense. note that some neighbours are sense annotated (single-token words that were tagged by our system), while others are not (multi-token words). for english wikipedia, we obtain a total vocabulary of 1.4m forms, 90, 023 of which are senseannotated. compared to the community-based representative words, the top neighbours in the embedding space tend to capture members of the same semantic class rather than direct potential replacements. ## conclusion we show that substitution-based word-sense induction algorithms based on word-substitutions derived from mlms are easily scalable to large corpora and vocabulary sizes, allowing to efficiently obtain high-quality sense annotated corpora. we demonstrate the utility of such large-scale sense annotation, both in the context of a scientific search application, and for deriving high-quality senseaware static word embeddings. as a secondary contribution, we also develop a new variant of the outlier detection evaluation task, which explicitly targets ambiguous words.