Unsupervised Lexical Substitution with Decontextualised Embeddings (2022)

Abstract: We propose a new unsupervised method for lexical substitution using pre-trained language models. Compared to previous approaches that use the generative capability of language models to predict substitutes, our method retrieves substitutes based on the similarity of contextualised and decontextualised word embeddings, i.e. the average contextual representation of a word in multiple contexts. We conduct experiments in English and Italian, and show that our method substantially outperforms strong baselines and establishes a new state-of-the-art without any explicit supervision or fine-tuning. We further show that our method performs particularly well at predicting low-frequency substitutes, and also generates a diverse list of substitute candidates, reducing morphophonetic or morphosyntactic biases induced by article-noun agreement.

## introduction there has been growing interest in developing automatic writing support systems to assist humans to write documents. one relevant task to this research goal is lexical substitution, where given a target word and its surrounding context, a system suggests a list of word substitutions that can replace the target word without changing its core meaning. for instance, given the target word great and the context he is a great artist, the model might suggest alternative words such as outstanding, terrific, or distinguished. writers can use such suggestions to improve the fluency of their writing, reduce lexical repetition, or search for better expressions that represent their ideas more creatively. as with other nlp tasks, recent studies have shown that masked language models such as bert (devlin et al., 2019) perform very well on lexical substitution, even without any task-specific supervision. a common approach is to employ language models as generative models and predict substitutes based on their generative capability. however, this approach has some limitations. first, it is extremely difficult for language models to predict rare words -especially those that are segmented into multiple subword tokens -since the models inevitably assign them very low probabilities. second, word prediction is highly affected by morphosyntactic constraints from the surrounding context, which overshadows the (arguably more important) question of semantic fit. for instance, if the target word is increase in the context ... with an increase in ..., language models tend to suggest words that also start with a vowel sound due to the presence of an before the target word, missing other possible substitutes such as hike or boost. in fact, this problem is even more pronounced in languages where words have grammatical gender (e.g. italian nouns) or a high degree of inflection (e.g. japanese verbs). in this paper, we propose a new approach that explicitly deals with these limitations. instead of generating words based on language model prediction, we propose to find synonymous words based on the similarity of contextualised and decontextualised word embeddings, where the latter refers to the "average" contextual representation of a word in multiple contexts. experiments on english and italian lexical substitution show that our fully unsupervised method outperforms previous models by a large margin. furthermore, we show that our model performs particularly well at predicting low-frequency words, and also generates more diverse substitutes with less morphophonetic or morphosyntactic bias, e.g. as a result of article-noun agreement in english and italian. ## related work in the pre-bert era, most lexical substitution methods employed linguistic resources such as wordnet (fellbaum, 1998) to obtain substitute candidates (szarvas et al., 2013; hintz and biemann, 2016). however, recent studies have shown that pre-trained language models such as bert outperform these models without any external lexical resources. for instance, zhou et al. (2019) feed a target sentence into bert while partially masking the target word using dropout (srivastava et al., 2014), and generate substitutes based on the probability distribution at the target word position. the masking strategy was shown to be effective on semeval-07 but not on swords. similarly, yang et al. (2022) feed two sentences into bert, concatenating the target sentence with itself but with the target word replaced by [mask], and predict words based on the mask-filling probability. michalopoulos et al. (2022) augment pre-trained language models with wordnet and outperform zhou et al. (2019). lacerra et al. (2021a) finetune bert on lexical substitution data sets that are automatically generated using bert. they show that this approach is effective at ranking, and that adding manually-annotated data further boosts performance. lacerra et al. (2021b) fine-tune bart on human-annotated data, and make it generate a list of substitutes given a target sentence in an end-to-end manner. they show that this generative approach rivals zhou et al. (2019). note that all of these recent models are evaluated on english only. ## conclusion we present a new unsupervised approach to lexical substitution using pre-trained language models. we showed that our method substantially outperforms previous methods on english and italian data sets, establishing a new state-of-the-art. by comparing performance on lexical substitution using different layers, we found that middle layers perform better than first or last layers. we also compared the substitutes predicted by the previous generative approach and our method, and showed that our approach works better at predicting low-frequency substitutes and reduces morphophonetic or morphosyntactic biases induced by article-noun agreement in english and italian.