ALEXSIS +: Improving Substitute Generation and Selection for Lexical Simplification with Information Retrieval (2023)

Abstract: Lexical simplification (LS) automatically replaces words that are deemed difficult to understand for a given target population with simpler alternatives, whilst preserving the meaning of the original sentence. The TSAR-2022 shared task on LS provided participants with a multilingual lexical simplification test set. It contained nearly 1,200 complex words in English, Portuguese, and Spanish and presented multiple candidate substitutions for each complex word. The competition did not make training data available; therefore, teams had to use either off-the-shelf pre-trained large language models (LLMs) or out-domain data to develop their LS systems. As such, participants were unable to fully explore the capabilities of LLMs by re-training and/or fine-tuning them on in-domain data. To address this important limitation, we present ALEXSIS+, a multilingual dataset in the aforementioned three languages, and ALEXSIS++, an English monolingual dataset that together contains more than 50,000 unique sentences retrieved from news corpora and annotated with cosine similarities to the original complex word and sentence. Using these additional contexts, we are able to generate new high-quality candidate substitutions that improve LS performance on the TSAR-2022 test set regardless of the language or model.

## introduction text simplification (ts) is utilized in educational technologies to automatically reduce the complexity of texts making them more accessible for various target populations, including children, second language learners, individuals with low-literacy, or those suffering from a reading disability, such as dyslexia or aphasia @xcite @xcite . with an increase in online learning, there has emerged a greater need for personalized learning platforms @xcite . these educational technology platforms need to be accessible to users. ts systems provide a solution by adapting content specifically for a user's level of literacy in a given target language (figure 1 ). complex sentence bombardment by regime forces complex word identification cwi: bombardment substitute generation sg: assault, raid, attack simplified sentence attack by regime forces substitute ranking sr: #1. attack , #2. assault substitute selection ss: assault, raid, attack lexical simplification (ls) is a precursor to ts @xcite . ls replaces challenging words, known as complex words, with simpler alternatives, hereby referred to as candidate substitutions. the generation of these candidate substitutions is known as substitute generation (sg) @xcite @xcite . sg attempts to predict viable candidate substitutions for an identified complex word. these candidate substitutions need to be easier to read and comprehend as well as be semantically similar to the identified complex word in its given context. an ls system would identify a complex word, for instance, "bombardment", as being in need of simplification. it would then suggest such words as "attack","assault" or "raid" as being valid candidate substitutions since they are shorter, more familiar to a set of annotators, or are found to be more frequent within a reference corpus. these candidate substitutions would then be passed to a ts system that would, in turn, simplify any unnecessary syntax resulting in an easier to read sentence. various methods have been applied to the task of sg for ls. the use of pre-trained llms trained with a masked language modeling (mlm) objective is the most favored approach to this task and has been shown to outperform other methods @xcite . however, the performance of mlm for sg is largely dependent on the model and the dataset it has been pre-trained on @xcite . this hinders sg for ls, since many ls datasets contain a small number of instances or a low number of gold candidate substitutions @xcite . as such, participants in the tsar-2022 shared-task on ls @xcite were forced to conduct zero-shot predictions for sg due to insufficient training data. this paper presents alexsis+ and alex-sis++ 1 , two new datasets for ls. we propose an information retrieval (ir) approach that utilizes collected data from news sources. these two datasets contain 50,000 additional contexts for the original 1,500 complex words of the alexsis dataset @xcite , and can be used to generate accurate candidate substitutions for sg in a zeroshot condition identical to that at tsar-2022 @xcite . we demonstrate how these new datasets can be applied to any language or model without re-training or fine-tuning to increase ls performance on the tsar-2022 test set. alex-sis+ and alexsis++ were also constructed using only the data available to the participants of the tsar-2022 shared-task, making our ir approach to sg, and later substitute selection (ss), highly adaptable. furthermore, unlike alexsis, which only features candidate substitutions, alexsis+ and alexsis++ feature multiple sentences per complex word providing new contexts that serve as useful data for mlm. finally, as the approach doesn't require manually annotating data as in the original alexsis, it can be used to improve the same unsupervised ls approaches purposed for the tsar-2022 shared-task. the main contributions of this paper are: pipeline the ls pipeline contains three subtasks (figure 1 ). the first of these is sg which produces @xmath0, 3, 5, or 10] @xcite . the top candidate (k@1) is then chosen to replace the complex word. this candidate is selected through two additional sub-tasks: ss, and substitute ranking (sr). ss filters inappropriate candidate substitutions by removing candidates that are equal to or semantically dissimilar to the complex word along with those that are inappropriate in that context. sr orders a list of candidate substitutions based on their appropriateness. techniques for ss and sr include sorting or filtering on frequency @xcite , word length @xcite , cosine similarity between word embeddings @xcite . more recent approaches have used regression @xcite , referred to as lexical complexity prediction (lcp) designed to replace binary cwi @xcite , as well as prompt learning @xcite . the tsar-2022 shared-task @xcite challenged participating teams with generating a list of @xmath1, spanish, and portuguese. one of tsar's key findings is that sr is less impactful on overall ls performance compared to sg, regardless of language. systems that relied solely on sg with minimal sr outperformed those that employed various sr methods. ls systems that relied purely on sg were often found to have used a pre-trained llm trained with an mlm objective to generate their top-k candidate substitutions. we therefore focus on an ir approach that only improves the performance of ls through the generation and selection of additional candidate substitutions for the tsar test set. masked language modeling mlm for ls involves feeding two concatenated sentences into an llm separated by the [sep] special token. the first sentence is the unaltered original sentence. the second sentence is the same as the original sentence, however, the target complex word is converted into the [mask] special token. the llm then examines both the first unaltered sentence and the words left and right of the [mask] special token in the altered second sentence. it uses this information to predict a candidate substitution for the masked complex word. from this, an llm is able to predict a candidate substitution that is suitable for both the provided context and for replacing the complex word. @xcite was the first to apply mlm for spanish sg. their ls-bert model surpassed all prior state-of-the-art approaches @xcite , including the use of lexicon, rule-based, statistical, n-gram, and word embedding models @xcite . lsbert was used as the baseline model at the tsar-2022 shared-task @xcite . inspired by the performance of lsbert, other studies have subsequently used mlm for sg @xcite @xcite . available resources a number of ls datasets containing complex words in context with gold candidate substitutions are available @xcite . for english, there are lexmturk @xcite with 500 complex words, benchls @xcite with 929 complex words, and nnseval @xcite with 239 complex words. for other languages, there is easier @xcite with 5,310 spanish complex words, simplex-pb (hartmann and aluísio, 2020) with 730 portuguese complex words, and hanls @xcite with 534 chinese complex words. there are also datasets that contain a large number of complex words in context without gold candidate substitutions @xcite @xcite . the largest ls dataset that contains both context and gold candidate substitutions is alexsis, referring to the combined english, spanish (alexsis-es) @xcite , and portuguese (alexsis-pt) @xcite dataset used at the tsar-2022 shared-task. ## alexsis+ as detailed in section 2, mlm requires context in order to predict a suitable candidate substitution for a given complex word. furthermore, mlm also requires a set of gold candidate substitutions to evaluate the quality of those it produces. with this in mind, we expand the alexsis dataset by including a large number of unique additional contexts ( lang. complex word data type sentences with same complex word generated candidate subs. (word.sim) sent.sim. en replica . a original the statue was moved to the academia, duplicate (0.398), replacement (0.333), 0.308 gallery and later replaced... by a replica. restoration (0.286), statue (0.426), ... a+ additional his project that he chose an exact day copy (0.503), version (0.322) and time for the replica he created.... prototype (0.518), clone (0.456), ... es municipio a original cobisa es un municipio español de la.. pueblo (0.5143), ayuntamiento (0.750), 0.490 [cobisa is a spanish municipality in the]... localidad (0.691), barrio (0.561), .. a+ additional el tortuga reapareció en el municipio... pedanía (0.542), pedanías (0.542), [the turtle reappeared in the municipality]... barriada (0.501), huerta (0.276), ... pt incremento a original coronel reconheceu incremento roubos... crescimento (0.856), aumento (0.878), 0.585 [colonel acknowledged increased robberies]... incre (0.835), avanço (0.680), ... a+ additional projetos inscritos devem... incremento... ativos ( 0.505), relevantes (0.541), [submitted projects must... increase]... diversos (0.407), essenciais (0.507), ... the extracted contexts. alexsis+ has a total of 12,831, 13,353, and 13,541 matched complex words in unique contexts for english, spanish, and portuguese, respectively. the larger alexsis++ dataset contains matched complex words in 33,149 unique contexts only for english, including those contexts already provided by alexsis+. both datasets provide embedding similarity scores between their additional sentences and the original context (sent.sim), as well as between their additional candidate substitutions and the original complex word (word.sim). sentence embeddings were generated using sentence-bert (sbert) @xcite . sbert is a state-of-the-art sentence-encoder. it employs siamese and triplet network structures to produce sentence embeddings that can be used to compare the semantic similarity between sentences by calculating the cosine similarity between sentence embeddings. english word embeddings were obtained using the en-vectors-web-lg model that provides ∼500k word vectors. spanish and portuguese word embeddings were taken from the pt-core-news-lg, and es-core-news-lg models trained on crawled news articles. dataset format alexis+ and alexsis++ are divided into three sub-corpora corresponding to the three languages, english (en), spanish (es), and portuguese (pt). each dataset contains: original cw, context, and candidate substitutions from the tsar-2022 shared-task, new contexts and new candidate substitutes generated on each new context, cosine similarities between new and old contexts and word similarities between word embeddings of the new candidate substitutions and the target complex word. alexsis+ and alexsis++ have the following nine headers separated by tab (\t): ## evaluation this section evaluates the performance of electra, roberta-large-bne, and bertimbau on the tsar-2022 test set using our ir approach to sg and ss and the alexsis+ and alexsis++ datasets (section 5.1). we also provide the performance of our various ss pipelines (section 5.3). for the evaluation, we removed duplicate gold labels within the tsar-2022 test set. performances are reported in terms of accuracy (acc), mean absolute precision, and potential following the tsar-2022 shared-task @xcite . the performances reported at the tsar-2022 shared-task (section 2) show that ls is still chal-lenging. even small improvements in performances can lead to greater gains down-stream for ts. for this reason, ls is often primarily evaluated on the quality of the top candidate substitution produced (k@1). the accuracy of the top k@1 candidate substitution (acc@1) is the ratio of instances whereby the best candidate generated is also the most appropriate candidate substitution among the gold labels. acc@1 is often used to determine the overall performance of a ls system, since it is this candidate substitution which replaces the complex word. in addition, ls is also evaluated on its f1score, potential (pot) and mean average precision (map). pot is the ratio of the candidate substitutions that are within all of the gold labels. map provides a score of the number of the returned candidate substitutions which match a gold label and its index. ## conclusion and future work this paper presents alexsis+ and alexsis++, two new version of the alexsis dataset used at the tsar-2022 shared-task @xcite . these datasets contain more than 50,000 unique sentences covering three languages retrieved from news corpora and annotated with cosine similarities to the original complex word and sentence. we have demonstrated that the use of these datasets, alongside an effective method of ss, can be used to generate and then select a more appropriate candidate substitution which, in turn, improves ls performance without the need for re-training or fine-tuning. in other words, results showed that the use of additional unique contexts can result in increases in ls performance, despite these contexts being dissimilar from the original context of the complex word. this increase in performance may appear small. however, even a small improvement in ls can have wider downstream implications that enhance the performance of a ts system substantially. we hypothesize that through further experimentation with alternative sg methods and ss filters, the performance gained by using alex-sis+ and alexsis++ will increase. we provide these two new ls datasets and make them publicly available to the wider research community. to the best of our knowledge, this is the first ir approach to ls opening exciting new avenues for research in this field. we show that the approach increases overall performance and that it can be applied to any ls model or language. in the future, we would like to incorporate this ir-based approach in a real-world personalized ts system that can be used in educational technology applications and online learning @xcite .