P ara LS : Lexical Substitution via Pretrained Paraphraser (2023)

Abstract: Lexical substitution (LS) aims at finding appropriate substitutes for a target word in a sentence. Recently, LS methods based on pretrained language models have made remarkable progress, generating potential substitutes for a target word through analysis of its contextual surroundings. However, these methods tend to overlook the preservation of the sentence’s meaning when generating the substitutes. This study explores how to generate the substitute candidates from a paraphraser, as the generated paraphrases from a paraphraser contain variations in word choice and preserve the sentence’s meaning. Since we cannot directly generate the substitutes via commonly used decoding strategies, we propose two simple decoding strategies that focus on the variations of the target word during decoding. Experimental results show that our methods outperform state-of-the-art LS methods based on pre-trained language models on three benchmarks.

## introduction lexical substitution (ls) in context @xcite @xcite is an extremely powerful technology that can be used as a backbone of various nlp applications such as writing assistance @xcite , word sense disambiguation @xcite , and lexical simplification @xcite . compared with traditional ls methods based on linguistic databases (e.g., wordnet) @xcite or word embedding models @xcite , ls methods based on pretrained language models have made remarkable progress in generating substitutes by considering the context @xcite @xcite . these methods feed the sentence into bert @xcite or xlnet @xcite to obtain the top probability words corresponding to the target word as the substitute candidates. however, they have the following two limitations. to address the limitations mentioned above, we study how to generate substitutes via paraphrase modeling. recent neural paraphrasers based on encoder-decoder framework @xcite produce fluent, meaningpreserving english paraphrases but contain variations in word choice. therefore, our idea is whether we can decode the substitute candidates from the hidden representation of the target word. in this way, the substitutes are not only semantically consistent with the target word and fit in the context, but also can preserve the sentence's meaning. the meaning-preserving properties of a paraphraser can aid in addressing the first limitation, while autoregressive paraphrasers can address the second limitation. to the best of our knowledge, paraphraser for ls task has not yet been explored, as current decoding methods focus on lexical variations within the entire sentence rather than the target word, resulting in a scarcity of appropriate substitutes for the target word. to specifically focus on lexical variations of the sent1 surprisingly in such a dry continent as australia , salt becomes a • • • labels arid, waterless bert wet,arid,moist,humid,damp xlnet wet, flat, moist, desert, cold ours desiccated,drought,arid,dead, parched sent2 remember that the delegates ' life is not always easy. labels simple, trouble free, undemanding, uncomplicated, straightforward bert simple, hard,complicated, difficult, exciting xlnet cheap, simple, quick, hard, fast ours simple, light, good, ease, facilitated to the best of our knowledge, parals is the first ls method that can produce substitute candidates by considering preserving the sentence's meaning. on three benchmarks, parals and parals⋆ achieve state-of-the-art performance across various evaluation metrics. moreover, parals⋆ without the step of substitute ranking outperforms all existing methods with the step of substitute ranking. additionally, we propose a novel strategy for the step of substitute ranking by text generation evaluation metrics bartscore @xcite and bleurt @xcite . our method embeds each substitute into the original sentence to create an updated version. by using bartscore and bleurt to compute the relationship between the original and updated sentences, they can quantify the extent to which the meaning of the original sentence has been preserved by each substitute. experimental results show that substitute ranking using only bartscore outperforms the previous state-of-the-art ranking methods when the same substitution candidate lists are provided for two popular ls benchmarks. the code and the experimental results are source-opened in github foot_0 . recent ls methods based on pretrained language models have attracted much attention @xcite @xcite , in which the pretrained bert is the most widely used one. zhou et al. @xcite apply dropout to the embeddings of target words, michalopoulos et al. @xcite propose a new mix-up embedding strategy by incorporating the knowledge of wordnet into the prediction process of bert, and lin et al. @xcite proposed an auxiliary gloss regularizer module to bert pre-training. lacerra et al. @xcite tried to train pretrained language models by merging the development set of two ls datasets (coinco and twsi). the current work @xcite sought to evaluate all existing pretrained language models, and found that combining the prediction of pretrained language models xlnet and word2vec achieved the best results. overall, pretrained language modeling-based ls methods consider contextual information of tar-get words when generating substitute candidates, but do not concern with the impact of applying substitutes on sentence meaning. in contrast to the aforementioned methods, we try to utilize the knowledge of a pretrained paraphraser to generate substitute candidates. ## method given a given sentence @xmath0, x 2 , ..., x t , ..., x n } and the target word x t , we need a pretrained paraphraser based on an autoregressive model, instead of a pretrained language modeling like existing ls methods @xcite @xcite , e.g., bert or xl-net. ls method consists of two steps: substitute generation and substitute ranking. after feeding sentence x into the paraphraser, we aim to extract substitute candidates for the target word x t by two novel decoding strategies (section 2.2). then, we rank the candidates to choose the most appropriate substitution without modifying the meaning of x (section 2.3). ## conclusions we introduce two novel paraphraser-based ls methods named parals and parals⋆, which generate substitute candidates by considering the context and preserving the sentence's meaning. specifically, we design two decoding strategies that center on lexical variations of the target word during decoding and propose a substitute candidate ranking strategy by utilizing the newest text generation evaluation metrics. experimental results show that par-als and parals⋆ significantly outperform the stateof-the-art ls methods. in the future, we will apply the methods to different languages, and verify our method on many downstream tasks to investigate further the method's general availability.