Word Sense Induction with Attentive Context Clustering (2021)

Abstract: In this paper, we present ACCWSI (Attentive Context Clustering WSI), a method for Word Sense Induction, suitable for languages with limited resources. Pretrained on a small corpus and given an ambiguous word (query word) and a set of excerpts that contain it, ACCWSI uses an attention mechanism for generating context-aware embeddings, distinguishing between the different senses assigned to the query word. These embeddings are then clustered to provide groups of main common uses of the query word. This method demonstrates practical applicability for shedding light on the meanings of ambiguous words in ancient languages, such as Classical Hebrew.

## introduction natural language expresses human concepts, thoughts, emotions and insights. that is, natural language represents a model of extremely high complexity-the human mind (at least, its communication-driven layers). some researchers believe that natural language is an environment in which compromise is inevitable when projecting the infinite number of dimensions of human thinking onto the much smaller number of dimensions of human speech @xcite . multiplicity of meaning of a single word, such as polysemy (similarity obtained from a common source) or homonyny (accidental similarity), is therefore an expected product of this compromise. below are two common examples of word sense ambiguity: humans are able to disambiguate the polysemy/homonyny or understand contextual nuances by using clues that come from the context of the ambiguous word. one of the fundamental tasks of natural language processing is word sense induction (wsi), a task of automatic discrimination of different senses of words by finding these contextual clues. it is difficult to overestimate the importance of accurate word sense induction when dealing with common natural language processing (nlp) tasks, such as information retrieval or search clustering. furthermore, historical research seeks to correctly induce the meaning of words in order to resolve doubts about many historical issues. as a good example we can refer to the akkadian lemma "galû", the meaning of which ranges between the negative shade of "exile" or "deportation", the neutral shade of "relocation" and the positive one of "appointment". another example is the hebrew lemma "zakar", which takes on both the meanings of "memory" and "male". accurate word sense induction is essential for correct understanding of ancient documents. in this paper, we present an attentive context clustering wsi (accwsi). accwsi first creates a word-embedding for each word, which is identical for any context that it appears in. accwsi uses the cosine similarity between the words in the context and the word in focus to determine the attention that each word should achieve to form a context aware vector representation for each appearance of the word in focus. accwsi then clusters the resulting vectors, such that each cluster represents a different meaning of the word. accwsi has demonstrated high practical applicability in languages with limited resources and obtained a very high score by the evaluation framework of semeval-2 2010 task 14 @xcite . accwsi achieved a high score not only with the original training dataset, but also with a training dataset reduced to a fraction of 2.6% of the original dataset, which is comparable to the size of the hebrew bible. ## related work word sense induction and word sense disambiguation provide fertile ground for researchers, starting from very early attempts to tackle these non-trivial tasks, such as "simulated annealing" according to human-edited dictionary @xcite and employing the "conceptual distance" between contexts @xcite , going through later unsupervised methods, that use patterns of word co-occurrence @xcite or bigrams of web search results @xcite , continuing with "hidden concepts" of the contextual words, that not necessarily overlap with the sense of the ambiguous word @xcite , and ending with the most recent solutions like @xcite , that uses word substitutions of modern masked language models, such as google bert mlm. our research was inspired by two main works: the context-group discrimination algorithm @xcite from the context clustering category and the google bert language model @xcite . @xcite utilize google bert for their wsi method. however, their method does not meet our requirement of being able to induce word senses in languages with limited resources, as training google bert on small corpora does not provide sufficient accuracy @xcite . the high scores achieved by the bertwsi model in the semeval-2 2010 task 14 @xcite metrics are credited to the fact that the underlying model was pre-trained by google on a huge corpus of text. our solution takes advantage of the basic mechanism of attention @xcite underlying bert without applying the complex process of learning attention weights and thus achieves good results when applied to small datasets. the only weight learning process we use is the word2vec @xcite model training that requires far fewer resources than attention-based learning. thus, we provide a practical tool in the study of the meanings of words in resource-limited languages, such as ancient dead languages. the clustering by committee work @xcite gave us the idea to use a threshold of 0.5 as an acceptable proportion of orphan instances when measuring the quality of a clustering solution (see section 3.4.3). we also explored lin's algorithm @xcite , which uses the word clustering approach by combining words with similar semantics into sense representations, but it was found less effective when it came to discriminating senses of words in resource-constrained languages. ## experimental evaluation we ran an experiment to evaluate the algorithm on sem-eval 2010 task 14 @xcite , which aims to objectively measure and compare the quality of wsi systems. both training and test data are english sentences containing polysemous or homonymous nouns and verbs. the goal of the task is to split the instances of each ambiguous word and their contexts into clusters representing different meanings. the result is assessed by comparison with the "gold standard" clustering performed by human experts. in section 4.1 we present the unsupervised v-measure and f-score metrics of this assessment as well as the supervised recall metric. ## application examples in this section we present examples of applying our method to a relatively small hebrew corpusthe hebrew bible. we used the text-fabric version of the bhsa project to generate the appropriate dataset and run the accwsi algorithm on it. figure 3 shows the operation of the accwsi algorithm used to obtain two different meanings of "bank" in english. figure 4 and figure 5 present the induced classes for two ambiguous hebrew biblical lemmas: khalal (dead body/desecrate) and zakar (male/memory). the instances of the first lemma were split into 2 sense clusters while the instances of the second lemma were split into 5 sense clusters. accwsi seems to perform well and provide satisfactory clusters despite the small training corpus. ## future work iterating the process of generating context embeddings may improve the accuracy of the clustering. in our future work we plan to develop a method for determining the "center of mass" (or "centroid" for convex clusters) of every cluster. these centers will be treated as new "query" embeddings and the accwsi attention-weighted technique will be reapplied within each cluster using its new query (its center). this should provide finer discrimination of meanings. this iterative process can be repeated many times until maximum accuracy is achieved. another effort we lead these days is word sense induction in ancient akkadian texts. between the 9th to the late 7th centuries bce, the assyrian empire deported millions of people across the near east. by even the most humble estimates, around 1.3 million people were moved around as a result of conquest, labour recruitment or as punishment, just to name the central reasons for this dire process (sano 2020). however, the records for these deportations are numerous and came down to us in different genres that deal with the act of deportation, or forced migration, from different points of view: contemporaneous assyrian royal inscriptions, letters and administrative texts, as well as babylonian historical chronicles, written many years after the events in question. all were written in assyrian and babylonian, two close dialects of akkadian, the oldest known (east-)semitic language in the world. in all, 19 different verbs deal with various stages of the forced migration, like the capture of people or forced recruitment, their change of location, and resettlement. even then, there are differences across meanings for specific verbs, sometimes minute ones, but also quite substantial in terms of semantics. a good example of such a complicated verb is galû which the chicago assyrian dictionary (cad), the most comprehensive dictionary of akkadian, translates as "1. to go into exile, 2. to deport, to exile ( š-stem, causative)" (cad š/3, 201). its usage is limited to a babylonian context, either in assyrian letters dealing with babylonia or babylonian chronicles @xcite . as text 1 shows, the usage, much like that of biblical hebrew gly/h, is used in consequence of a military takes on the sense of either dead body(as a noun) or desecrate(as a verb). this figure presents the appropriate clusters generated by accwsi. the "attention highlight" column shows the most relevant context words. in the context of desecrate (cluster 0), the attention is paid to words like god, sacred, nation etc. while in the context of dead body (cluster 1), the attention is paid to sword, stab, fall etc. conflict. however, a single instance in a letter from the time of tiglath-pileser iii (c. 731-730 bce), here text 2, shows that under certain political circumstances people could ask for someone to deport them to assyria, perhaps referring to the safety of being a protected refugee under the direct responsibility of the assyrian king. this might also be the meaning of certain cases in aramaic, where gly in g-stem active participle means "exile, refugee", or in d-stem means "to emigrate", (comprehensive aramaic lexicon, s.v. gly d and c). text 1: saa 19, 27 rev. 4'-8a' (online edition, luukko 2012) 4' lugal? lu? ú-di nim.ma.ki -a -[a] 5' l ú.erim-me š-šú-nu ta dumu m gin-numun la? 6' i-du-ku ù ša-daa-ni 7' ú-sag-li-šú-nu šú-nu-ú-ma 8' ig-da-al-ú the "attention highlight" column shows the most relevant context words. the first cluster represents the sense of male human, the second one -god's memory, the third one -male animal sacrifice, the fourth -the role of scribe and the fifth -chronological memory (rev. 4'-5') the elamites killed their soldiers with the son of mukin-zeri and (6'-8') deported them by force. they too went into exile. text 2: saa 19, 87 obv. 8b'-13a' (online edition, luukko 2012) 8' . . . e-gir-tum ša ina ugu 9' [ md ]amar.utu-a-sum-na na-u-ni-ni ittab-lu-ni 10' [ina] pa-ni-ni i-si-si-ú : ù m ba-la-su 11' [ip]-ta-la-a a-da-niš ma-a an-nu-rig x+[x x] 12' [at]-tu-nu tal-la-ka ma-a ša-ga-la-ni [o] 13' [i]-si-ku-nu la-al-lik . . . (obv. 8'-9') they intercepted the letter which was brought to merodach-baladan (10') and read it [in] our [pr]esence. but balassu (11') [g]ot very scared, saying: (12') "you (pl.) must come this moment and deport me! (13') i will go [wit]h you (pl.)." ## conclusion in this paper we propose accwsi, an algorithm to automatically induce various senses of ambiguous words by automatically focusing on the most relevant words from their contexts. after learning generic word embeddings into a word2vec model, accwsi uses the basic attention technique for determining the most relevant context members and generating context-aware embeddings, each with a semantic direction that aggregates the directions of its context members. distant meanings imply distant context embeddings and vice versa, and thus standard clustering techniques can be easily applied for grouping the context embeddings by their common semantic directions. accwsi has shown excellent performance even when trained on a small subset of the training data in the semeval-2 2010 task 14. furtheremore, accwsi demonstrated high applicability in disambiguation of word senses in ancient semitic languages, such as classical hebrew and akkadian.