Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution (2020)

Abstract: Lexical substitution, i.e. generation of plausible words that can replace a particular target word in a given context, is an extremely powerful technology that can be used as a backbone of various NLP applications, including word sense induction and disambiguation, lexical relation extraction, data augmentation, etc. In this paper, we present a large-scale comparative study of lexical substitution methods employing both rather old and most recent language and masked language models (LMs and MLMs), such as context2vec, ELMo, BERT, RoBERTa, XLNet. We show that already competitive results achieved by SOTA LMs/MLMs can be further substantially improved if information about the target word is injected properly. Several existing and new target word injection methods are compared for each LM/MLM using both intrinsic evaluation on lexical substitution datasets and extrinsic evaluation on word sense induction (WSI) datasets. On two WSI datasets we obtain new SOTA results. Besides, we analyze the types of semantic relations between target words and their substitutes generated by different models or given by annotators.

## introduction lexical substitution is the task of generating words that can replace a given word in a given textual context. for instance, in the sentence "my daughter purchased a new car" the word car can be substituted by its synonym automobile, but also with co-hyponym bike, or even hypernym motor vehicle while keeping the original sentence grammatical. lexical substitution has been proven effective in various applications, such as word sense induction @xcite , lexical relation extraction @xcite , paraphrase generation, text simplification, textual data augmentation, etc. note that the preferable type (e.g., synonym, hypernym, co-hyponym, etc.) of generated substitutes depends on the task at hand. the new generation of language and masked language models (lms/mlms) based on deep neural networks enabled a profound breakthrough in almost all nlp tasks. these models are commonly used to perform pre-training of deep neural networks, which are then fine-tuned to some final task different from language modeling. however, in this paper we study how the progress in unsupervised pre-training over the last five years affected the quality of lexical substitution. we adapt context2vec @xcite , elmo @xcite , bert @xcite , roberta @xcite and xlnet @xcite to solve lexical substitution task without any fine-tuning, but using additional techniques to ensure similarity of substitutes to the target word, which we call target word injection techniques. we provide the first large-scale comparison of various neural lms/mlms with several target word injection methods on lexical substitution and wsi tasks. our research questions are the following (i) which pre-trained models are the best for substitution in context, (ii) additionally to pre-training larger models on more data, what are the other ways to improve lexical substitution, and (iii) what are the generated substitutes semantically. more specifically, the main contributions of the paper are as follows foot_0 : ## related work solving the lexical substitution task requires finding words that are both appropriate in the given context and related to the target word in some sense (which may vary depending on the application of generated substitutes). to achieve this, unsupervised substitution models heavily rely on distributional similarity models of words (dsms) and language models (lms). probably, the most commonly used dsm is word2vec model @xcite . it learns word embeddings and context embeddings to be similar when they tend to occur together, resulting in similar embeddings for distributionally similar words. contexts are either nearby words or syntactically related words @xcite . in @xcite several metrics for lexical substitution were proposed based on embedding similarity of substitutes both to the target word and to the words in the given context. later @xcite improved this approach by switching to dot-product instead of cosine similarity and applying an additional trainable transformation to context word embeddings. a more sophisticated context2vec model producing embeddings for a word in a particular context (contextualized word embeddings) was proposed in @xcite and was shown to outperform previous models in a ranking scenario when candidate substitutes are given. the training objective is similar to word2vec, but context representation is produced by two lstms (a forward and a backward for the left and the right context), in which final outputs are combined by feed-forward layers. for lexical substitution, candidate word embeddings are ranked by their similarity to the given context representation. a similar architecture consisting of a forward and a backward lstm is employed in elmo @xcite . however, in elmo each lstm was trained with the lm objective instead. to rank candidate substitutes using elmo @xcite proposed calculating cosine similarity between contextualized elmo embeddings of the target word and all candidate substitutes. this requires feeding the original example with the target word replaced by one of the candidate substitutes at a time. the average of outputs at the target timestep from all elmo layers performed best. however, they found context2vec performing even better and explained this by the negative sampling training objective, which is more related to the task. recently, transformer-based models pre-trained on huge corpora with lm or similar objectives have shown sota results in various nlp tasks. bert @xcite and roberta @xcite were trained to restore a word replaced with a special [mask] token given its full left and right contexts (masked lm objective), while xlnet @xcite predicted a word at a specified position given only some randomly selected words from its context (permutation lm objective). in @xcite , bert was reported to perform poorly for lexical substitution (which is contrary to our experiments), and two improvements were proposed to achieve sota results using it. firstly, dropout is applied to the target word embedding before showing it to the model. secondly, the similarity between the original contextualized representations of the context words and their representations after replacing the target by one of the possible substitutes are integrated into the ranking metric to ensure minimal changes in the sentence meaning. this approach is very computationally expensive, requiring calculation of several forward passes of bert for each input example, depending on the number of possible substitutes. we are not aware of any work applying xlnet for lexical substitution, but our experiments show that it outperforms bert by a large margin. supervised approaches to lexical substitution include @xcite @xcite . these approaches rely on manually curated lexical resources like wordnet, so they are not easily transferable to different languages, unlike those described above. also, the latest unsupervised methods were shown to perform better @xcite . ## neural language models for lexical substitution with target word injection to generate substitutes, we introduce several substitute probability estimators, which are models taking a text fragment and a target word position in it as input and producing a list of substitutes with their probabilities. to build our substitute probability estimators we employ the following lms/mlms: context2vec @xcite , elmo @xcite , bert @xcite , roberta @xcite and xlnet @xcite . these models were selected to represent the progress in unsupervised pre-training with language modeling and similar tasks over the last five years. given a target word occurrence, the basic approach for models like context2vec and elmo is to encode its context and predict the probability distribution over possible center words in this particular context. this way, the model does not see the target word. for mlms, the same result can be achieved by masking the target word. this basic approach employs the core ability of lms/mlms of predicting words that fit a particular context. however, these words are often not related to the target. the information about the target word can improve generated substitutes, but what is the best method of injecting this information is an open question. ## baseline lexical substitution models lexical substitution models described above are compared to the best previously published results, as well as our re-implementations of the following baseline models proposed in @xcite . ooc out of context model ranks possible substitutes by their cosine similarity to the target word and completely ignores given context. following @xcite we use dependency-based embeddings foot_4 released by @xcite . ## intrinsic evaluation we perform an intrinsic evaluation of the proposed models on two lexical substitution datasets. ## extrinsic evaluation in this section, we perform the extrinsic evaluation of our models applied to the word sense induction (wsi) task. the data for this task commonly consists of a list of ambiguous target words and a corpus of sentences containing these words. models are required to cluster all occurrences of each target word according to their meaning. thus, the senses of all target words are discovered in an unsupervised fashion. for example, suppose that we have the following sentences with the target word bank: ## conclusion we presented the first comparison of a wide range of lms/mlms with different target word injection methods on the tasks of lexical substitution and word sense induction. our results are the following: (i) large pre-trained language models yield better results than previous unsupervised and supervised methods of lexical substitution; (ii) if properly done, the integration of information about the target word substantially improves the quality of lexical substitution models. the proposed target injection method based on a fusion of a context-based distribution p (s|c) with a target similarity distribution p (s|t ) proved to be the best one. when applied to the xlnet model, it yields new sota results on two wsi datasets. finally, we study the semantics of the produced substitutes. this information can be valuable for practitioners selecting the most appropriate lexical substitution method for a particular nlp application.