Word Sense Disambiguation: Towards Interactive Context Exploitation from Both Word and Sense Perspectives (2021)

Abstract: Lately proposed Word Sense Disambiguation (WSD) systems have approached the estimated upper bound of the task on standard evaluation benchmarks. However, these systems typically implement the disambiguation of words in a document almost independently, underutilizing sense and word dependency in context. In this paper, we convert the nearly isolated decisions into interrelated ones by exposing senses in context when learning sense embeddings in a similarity-based Sense Aware Context Exploitation (SACE) architecture. Meanwhile, we enhance the context embedding learning with selected sentences from the same document, rather than utilizing only the sentence where each ambiguous word appears. Experiments on both English and multilingual WSD datasets have shown the effectiveness of our approach, surpassing previous state-of-the-art by large margins (3.7% and 1.2% respectively), especially on few-shot (14.3%) and zero-shot (35.9%) scenarios.

## introduction word sense disambiguation (wsd) is the task of determining a word's sense given its context. recently, contextualized representation learning @xcite have accelerated the advancement of wsd, raising the performance on a standard evaluation framework @xcite from slightly higher than 70% @xcite @xcite to about 80% @xcite ; bevilacqua and * corresponding author @xcite . this is an estimated upper bound of the task, which is from the inter-annotator agreement: the percentage of words that are annotated with the same meaning by two or more annotators @xcite . there is a clear trend that supervised systems tend to incorporate sense knowledge into their architecture, ranging from sense definition, usage examples to sense relation. however, the disambiguation of words in a document is almost independent of each other, especially from the perspective of senses in context. the connection of each word's disambiguation is limited to the utilization of a sentence @xcite @xcite or a small window of text @xcite because of computation cost or model restriction. more severely, the interaction of senses in context is barely explored. similar to word cooccurrence, the appearance of one sense can sometimes dominate the choice of another sense in the same sentence @xcite . in this paper, we introduce sace, a similaritybased wsd approach. precisely, we transform the previously almost isolated disambiguation of words in a document into interrelated ones to maximize the contribution of context from both word and sense perspectives. we summarize our contributions as follows: ## related work there are mainly two alternatives for solving wsd, namely knowledge-based and supervised approaches. while the former mainly relies on a sense inventory for disambiguation, the latter is dependent on sense-annotated corpora to train a sense classifier, either for each word or the whole vocabulary. however, many recently proposed systems combine the above two strategies, injecting sense knowledge into their supervised models while somehow inadequately modeling the provided context in a document from both word and sense perspectives. ## preliminary wsd is to select the correct sense 𝑠𝑠̃𝑗𝑗 of a word 𝑤𝑤 𝑖𝑖𝑗𝑗 given its context. 𝑤𝑤 𝑖𝑖𝑗𝑗 is the 𝑗𝑗 𝑡𝑡ℎ word in the 𝑖𝑖 𝑡𝑡ℎ sentence 𝑆𝑆 𝑖𝑖 = {𝑤𝑤 𝑖𝑖1 , 𝑤𝑤 𝑖𝑖2 , … , 𝑤𝑤 𝑖𝑖𝑗𝑗 , … , 𝑤𝑤 𝑖𝑖𝑖𝑖 } of a document 𝐷𝐷 = {𝑆𝑆 1 , 𝑆𝑆 2 , … , 𝑆𝑆 𝑖𝑖 , … , 𝑆𝑆 𝑚𝑚 }. the candidate senses 𝑠𝑠�𝑤𝑤 𝑖𝑖𝑗𝑗 � = {𝑠𝑠 𝑗𝑗1 , 𝑠𝑠 𝑗𝑗2 , … , 𝑠𝑠 𝑗𝑗𝑗𝑗 , … , 𝑠𝑠 𝑗𝑗𝑗𝑗 } are from a sense inventory such as wordnet. here, 𝑖𝑖, 𝑗𝑗, and 𝑗𝑗 denote the index of sentence, word, and sense respectively. in a similarity-based wsd approach, the disambiguation of a word is determined by the similarity between its context representation 𝑣𝑣 𝑤𝑤 𝑖𝑖𝑗𝑗 and each candidate sense representation 𝑣𝑣 𝑠𝑠 𝑗𝑗𝑗𝑗 . in many cases, both representations are vectors and the similarity is measured by their dot product after normalization. then, the sense with the highest similarity is selected as the correct sense. typically, a word's context representation is learned using the sentence 𝑆𝑆 𝑖𝑖 where the word appears @xcite @xcite . the representation of a candidate sense is obtained using its gloss/definition 𝐺𝐺 𝑠𝑠 𝑗𝑗𝑗𝑗 defined in wordnet @xcite . a common approach of encoding these two sequences in recent research is to utilize pre-trained models such as bert, roberta @xcite , and so on, taking the sum of the outputs of the last four layers as encoded features @xcite , as in (1) and (2). before feeding 𝑆𝑆 𝑖𝑖 and 𝐺𝐺 𝑠𝑠 𝑗𝑗𝑗𝑗 to the models, a special token [cls]/[sep] is added to the beginning/end of the sequence, modifying them into 𝑆𝑆 ̅ 𝑖𝑖 and 𝐺𝐺 ̅ 𝑠𝑠 𝑗𝑗𝑗𝑗 , respectively. for each 𝑤𝑤 𝑖𝑖𝑗𝑗 's context representation, a normal choice is to utilize the model's output at the position of the word (𝑗𝑗), using 𝑆𝑆 ̅ 𝑖𝑖 as input, shown in equation ( 1 ). if the word is tokenized into several pieces, their mean is taken. in contrast, for each sense representation, when it is fine-tuning a pretrained model, the sense embedding is the output at the position of [cls] @xcite , with the modified gloss as input, as in (2). to utilize the supervision from a training corpus, a cross-entropy loss is implemented against the similarity distribution of candidate senses (the softmax product without index 𝑗𝑗 in (3)) and the one-hot ground-truth distribution, shown in equation ( 4 ). 𝑉𝑉 𝑠𝑠�𝑤𝑤 𝑖𝑖𝑗𝑗 � ∈ ℝ �𝑠𝑠�𝑤𝑤 𝑖𝑖𝑗𝑗 ��×ℎ is a matrix of concatenated sense embeddings arranged in rows. ℎ is the dimension of the pre-trained model's hidden states (768 or 1024 of bert). 𝑦𝑦 𝑗𝑗𝑗𝑗 is equal to 1 when 𝑠𝑠 𝑗𝑗𝑗𝑗 (the 𝑗𝑗 𝑡𝑡ℎ sense of 𝑤𝑤 𝑖𝑖𝑗𝑗 ) is the correct sense, otherwise 0, representing each element in the ground-truth one-hot vector. for prediction, the model selects the sense with the largest dot product for each word. in the above approach (from bem, blevins and zettlemoyer, 2020), the embedding learning process of different senses is independent of each other, relying merely on sense gloss. besides, the 𝑺 𝒊 𝑺 𝒊-𝟏 𝑺 𝒊-𝟏𝟐 𝑺 𝒊+𝟏 𝑺 𝒊+𝟕 … 𝒈𝒐𝒆𝒔 𝑻𝒉𝒆𝒓𝒆 𝒃𝒆𝒍𝒍 𝒗 � 𝒃𝒆𝒍𝒍 … 𝒕𝒉𝒆𝒓𝒆 𝒃𝒆𝒍𝒍 … 𝒈𝒐 … gloss(𝒃𝒆𝒍𝒍. 𝒏. 𝟎𝟑): roberta roberta 𝒗 𝒃𝒆𝒍𝒍.𝒏.𝟎𝟑 𝒗 𝒃𝒆𝒍𝒍.𝒏.𝟎𝟐 𝒗 𝒃𝒆𝒍𝒍.𝒏.𝟎𝟏 𝒗 � 𝒃𝒆𝒍𝒍.𝒏.𝟎𝟏 𝒗 � 𝒃𝒆𝒍𝒍.𝒏.𝟎𝟐 … 𝒗 � 𝒃𝒆𝒍𝒍.𝒏.𝟎𝟑 the sound of a bell being struck. ## conclusion in this paper, we propose an interactive context exploitation method from both word and sense perspectives in a supervised similarity-based wsd architecture. experiments on english and crosslingual all-words wsd datasets verify the effectiveness of our approach, surpassing previous state-of-the-art by large margins. it also shows that the proposed method has an overwhelming advantage of learning few-shot and zero-shot wsd ability. for future work, we intend to utilize reinforcement learning to enhance current interactive wsd by customizing the context exploitation for different instances. the source code is available at: https://github.com/lwmlyy/sace .