BOS at LSCD iscovery: Lexical Substitution for Interpretable Lexical Semantic Change Detection (2022)

Abstract: We propose a solution for the LSCDiscovery shared task on Lexical Semantic Change Detection in Spanish. Our approach is based on generating lexical substitutes that describe old and new senses of a given word. This approach achieves the second best result in sense loss and sense gain detection subtasks. By observing those substitutes that are specific for only one time period, one can understand which senses were obtained or lost. This allows providing more detailed information about semantic change to the user and makes our method interpretable.

## introduction lscdiscovery is a shared task on lexical semantic change detection (lscd) in spanish (d. @xcite . the participants were provided with two corpora in spanish, corresponding to @xcite respectively, and were asked to solve two subtasks. in the first subtask the participants were asked to rank the given list of about 4k words according to the degree of their semantic change. the second subtask required to determine for each given word if its senses occurring in two corpora are different (and optionally, if it has acquired some new senses, and if it has lost any old ones). ## background our approach is based on the bag-of-substitutes (bos) representation of word meaning in context @xcite . lexical substitutes are those words that can replace a given target word in a given text fragment without making this fragment ungrammatical or substantially changing the meaning of the target word. for ambiguous words, lexical substitutes depend on their meaning expressed in a particular context. for instance, some reasonable substitutes for the word fly in the sentence a noisy fly sat on my shoulder are bug, beetle, butterfly, firefly, insect, etc. but in the sentence we will fly to london they are different: walk, run, bike, etc. in order to generate lexical substitutes, we employ the xlm-r foot_0 masked language model @xcite . this model was pre-trained on 2.5t of data in 100 languages as a masked language model, i.e. it received text fragments with some tokens hidden (replaced with the @xmath0. this kind of pre-training is partially aligned with the lexical substitution task because the model can predict words compatible with the given context. however, there are no guarantees that these words are similar or related by meaning to the target word. suitable types of lexical substitutes (e.g., synonyms, hypernyms, cohyponyms) and suitable degree of their similarity to the target word depend on the target task and can be controlled with various techniques explored in @xcite . in our solution, we employ the dynamic patterns proposed by @xcite and explained in 3.2. unlike the traditional bag-of-words representation, which contains those words that occur in a text fragment, the bos representation is built from lexical substitutes. thus, it better represents the meaning of some specific target word in a given text fragment rather than the whole fragment in general. clustering of the bos vectors is a successful approach to solve the word sense induction (wsi) task, i.e. to discover senses of ambiguous words. this approach was explored in many papers, including @xcite @xcite @xcite among others. also, a substitution-based wsi model was employed to solve the lscd task in @xcite . however, in our solution we avoid solving the more ## model description for each target word we sample some examples of its usage from both corpora and generate lexical substitutes for them. then we build two sets of bos vectors for old and new examples, describing old and new senses of the word respectively. finally, the distances from old to new examples are calculated, and their average is returned as the predicted score of graded change. following previous works on lscd @xcite , we will denote this average as the average pairwise distance (apd). notice that our vector representation is very different from those works. for the second subtask, if apd is greater than a certain threshold, we predict that this word has changed its meaning. to determine whether it has acquired new senses and whether any old senses were lost, we propose three different methods based on pairwise distances. ## discriminative substitutes the main advantage of ls-based models is their interpretability. we can roughly understand word meanings looking at the discriminative substitutes, i.e. the substitutes specific for a particular subset of examples. from the ## efficiency the set of the target words proposed in phase 1 was supposed to be a challenge for participants due to its size. for 4385 words given we have collected about 777k examples. generation of substitutes for all examples took 13 gpu-hours and 310 gpuhours for each one-mask and two-mask pattern respectively on v100 gpus. all other steps took incomparably less time. ## conclusion we have proposed an interpretable approach to lexical semantic change detection. this approach shows the 2nd best result for sense loss and sense gain detection subtasks. it provides techniques to understand which senses were obtained or lost by a word.