CILS at TSAR -2022 Shared Task: Investigating the Applicability of Lexical Substitution Methods for Lexical Simplification (2022)

Abstract: Lexical simplification — which aims to simplify complex text through the replacement of difficult words using simpler alternatives while maintaining the meaning of the given text — is popular as a way of improving text accessibility for both people and computers. First, lexical simplification through substitution can improve the understandability of complex text for, for example, non-native speakers, second language learners, and people with low literacy. Second, its usefulness has been demonstrated in many natural language processing problems like data augmentation, paraphrase generation, or word sense induction. In this paper, we investigated the applicability of existing unsupervised lexical substitution methods based on pre-trained contextual embedding models and WordNet, which incorporate Context Information, for Lexical Simplification (CILS). Although the performance of this CILS approach has been outstanding in lexical substitution tasks, its usefulness was limited at the TSAR-2022 shared task on lexical simplification. Consequently, a minimally supervised approach with careful tuning to a given simplification task may work better than unsupervised methods. Our investigation also encouraged further work on evaluating the simplicity of potential candidates and incorporating them into the lexical simplification methods.

## introduction lexical simplification -which aims to simplify complex words and phrases in text while maintaining the meaning of the original text -is an important natural language processing (nlp) problem to improve the understandability of text for, for example, non-native speakers, second language learners, and people with low literacy skills @xcite . due to its importance in achieving complete text simplification with simpler and easier-to-read content, lexical simplification has received rising interest over the years. @xcite has introduced a lexical simplification pipeline, which consists of several subproblems, including, for instance, complex word identification, simpler substitution generation, substitution selection, and substitution ranking. out of these four sub-problems, the latter three entirely focus on the generation of relevant and simpler substitutes for better understandability. underpinned by @xcite among others, over the years researchers have introduced a wide range of methods for simpler substitution generation for the complex words identified in text. earlier approaches to substitution generation have relied on rule-based methods and lexical resources like wordnet @xcite or paraphrase databases @xcite . lexical substitution research has advanced to the use of word embedding models (e.g., word2vec @xcite , glove @xcite , embeddings from language models (elmo) @xcite ) to remove the requirement of lexical resources and obtain potential candidates through the cosine similarity of word embeddings. the introduction of transformers @xcite has resulted in advanced contextual word and sentence embedding models like bidirectional encoder representations from transformers (bert) @xcite , robustly optimised bert (roberta) @xcite , and xlnet @xcite which have been extensively used for nlp, including, but not limited to, lexical simplification and lexical substitution. these models have been useful in generating potential candidates for simplification given a target word and the context, taking the meaning preservation aspect into account; researchers have introduced methods and frameworks for lexical simplification, some of which rely entirely on contextual embeddings @xcite whereas some others incorporate lexical resources alongside contextual embedding models @xcite . lexical substitution can be identified as a broader problem, which aims to generate alternative substitutes for a target word @xcite whereas lexical simplification specifically focuses on generating simpler substitutes @xcite . although not identical, the two problems are coupled; both aim to generate substitutes for an identified target word. hence, also the proposed, studied, and adopted solution techniques have similarities. in our work, we investigated the applicability of lexical substitution methods for simpler substitution generation in lexical simplification. we applied the cilex solution proposed in our previous work @xcite on context information for lexical substitution to lexical simplification foot_0 . the objective of the research was to evaluate the usefulness of existing substitution methods in a given text simplification task and to identify how these methods can be improved. ## related work researchers have used different techniques to achieve lexical simplification; this problem aims to simplify complex content in text while maintaining the meaning, for better understandability. earliest lexical simplification approaches relied on rule-based methods and lexical resources @xcite where a set of rules was defined to extract simpler substitutes from lexical resources like wordnet @xcite and rank them based on a simplicity metric. extending beyond these linguistic databases, researchers also used parallel corpora that consisted of complex and simpler sentences to identify simpler substitutes for a target word @xcite . however, given both these approaches were dependent on linguistic databases and parallel corpora, they had limitations with respect to the availability and coverage of simpler alternatives. to address the limitations of lexical resources, researchers adopted word embedding models for lexical simplification @xcite . further improving on the word embedding models, researchers introduced context-aware lexical simplification methods @xcite which also incorporated linguistic features and information from lexical resources. the introduction of transformer-based language models like bert, roberta, and xlnet resulted in widely adopting them for downstream nlp problems. to illustrate, @xcite introduced a recursive simplification method called lsbert based on bert. similar techniques have been used for lexical substitution, which is a broader problem aiming to generate alternative words for a given target word. early methods, which relied on rule-based systems and lexical resources, have evolved to the methods that use word embeddings @xcite , contextual embeddings @xcite and methods that incorporate additional information from lexical resources @xcite . given the similarities in lexical substitution and simplification problems, we investigated the applicability of the cilex lexical substitution solution also for lexical simplification. this investigation was part of the text simplification, accessibility, and readability (tsar) shared task on lexical simplification in 2022 @xcite . ## discussion in this paper, we have adopted our previous work on lexical substitution for the tsar-2022 shared task on lexical simplification and experimented with the use of different methods that provide context information. the three methods used for our experiments have not performed as well as the ls-bert baseline. however, they have outperformed tuner-baseline at tsar-2022 @xcite and our approach has excelled in lexical substitution as evidenced by its evaluation using two annotated lexical substitution datasets that are widely used for the broader problem @xcite . the observed performance difference for lexical simplification and lexical substitution can be explained by the problem differences. the methods used in our experiments were developed to target lexical substitution, which can be identified as a substantially broader problem than lexical simplification. lexical substitution generally focuses on generating similar rather than simpler substitutes -the narrower focus of lexical simplification. in order to tackle this issue of our lexical substitution approach for lexical simplification, identifying metrics, which can evaluate the simplicity of the potential candidates and using them to rank the potential candidates can be done. ## conclusion we have applied our previous work on lexical substitution for lexical simplification, focusing on the added value of context information for the lexical simplification problem. the results from our methods indicate, that even though the proposed approach has performed well in lexical substitution more broadly, their usefulness in the narrower lexical simplification problem at the tsar-2022 shared task was limited; a minimally supervised approach with some careful tuning to a given simplification task may have worked better at tsar-2022 than unsupervised lexical substitution methods with pre-trained models. our investigation encourages further work on evaluating the simplicity of potential substitution candidates and incorporating them into lexical substitution methods. this approach should extend these broader methods to lexical simplification by targeting the more specific constraints for substitutes in the narrower text simplification problem.