Unsupervised Lexical Simplification with Context Augmentation (2023)

Abstract: We propose a new unsupervised lexical simplification method that uses only monolingual data and pre-trained language models. Given a target word and its context, our method generates substitutes based on the target context and also additional contexts sampled from monolingual data. We conduct experiments in English, Portuguese, and Spanish on the TSAR-2022 shared task, and show that our model substantially outperforms other unsupervised systems across all languages. We also establish a new state-of-the-art by ensembling our model with GPT-3.5. Lastly, we evaluate our model on the SWORDS lexical substitution data set, achieving a state-of-the-art result.

## introduction lexical simplification is the task of replacing a word in context with an easier term without changing its core meaning, to make text easier to read for non-technical audiences, non-native speakers, or people with cognitive disabilities (e.g. dyslexia). one common approach @xcite is to use a masked language model (mlm) such as bert @xcite and predict substitutes via word prediction over the masked target word. however, one limitation is that it critically relies on the target context being discriminative of the semantics of the target word, which is not always the case. given this, we propose a new unsupervised method that performs context augmentation. specifically, we sample sentences that contain the target word from monolingual data, and identify substitutes that can replace the word in the target and sampled sentences. based on our experiments in english, portuguese, and spanish over the tsar-2022 shared task @xcite , we show that our model comfortably outperforms other unsupervised models. we also es1 code is available at: https://github.com/twadada/ lexsub_decontextualised . ## method we propose a fully unsupervised model using pretrained language models (without fine-tuning) and monolingual data. given the target word x and context c x , our model generates substitutes for x based on not only c x but also augmented contexts sampled from monolingual data. ## ablation study we perform ablation studies on the effect of clustering in eqn. (2), and present the results in the table shows that our proposed method performs the best overall, albeit with a small margin over "no clustering". in fact, this is more or less expected since the majority of target words are used in their predominant senses (as evidenced by the strong performance of gpt-3.5 w/o context in ta-ble 1), in which case, retrieving the most-generated words across all sampled sentences would suffice to produce good substitutes. ## related work recent lexical simplification models are based on generating substitute candidates using mlm prediction and reranking, using features such as fast-text embedding similarities and word frequency @xcite . some also use external tools or resources such as pos taggers or paraphrase databases @xcite . however, @xcite show that gpt-3.5 substantially outperforms previous models on the tsar-2022 shared task. similar to this work, our prior work @xcite samples sentences from monolingual corpora and use them to paraphrase multiword expressions with literal expressions (composed of 1 or 2 words). ## conclusion we propose a new unsupervised lexical simplification method with context augmentation. we show that our model outperforms previous unsupervised methods, and by combining our model with in-structgpt, we achieve a new state-of-the-art for lexical simplification and substitution.