Controllable Lexical Simplification for E nglish (2022)

Abstract: Fine-tuning Transformer-based approaches have recently shown exciting results on sentence simplification task. However, so far, no research has applied similar approaches to the Lexical Simplification (LS) task. In this paper, we present ConLS, a Controllable Lexical Simplification system fine-tuned with T5 (a Transformer-based model pre-trained with a BERT-style approach and several other tasks). The evaluation results on three datasets (LexMTurk, BenchLS, and NNSeval) have shown that our model performs comparable to LSBert (the current state-of-the-art) and even outperforms it in some cases. We also conducted a detailed comparison on the effectiveness of control tokens to give a clear view of how each token contributes to the model.

## introduction lexical simplification (ls) is a natural language processing task that modifies texts by substituting difficult words with easier words (or phrases) while keeping the original information and meaning @xcite . ## related work early lexical simplification approaches with unsupervised models used: latent words language models @xcite , wikipediabased models/rules @xcite @xcite and distributional lexical semantics @xcite . (paetzold and specia, 2017b) started the use of neural networks for the task combined with a retrofitted context-aware word embedding model. @xcite presented lsbert, a lexical simplification system that uses a pretrained bert @xcite model for english to generate substitution candidates. lsbert has two main phases: 1) substitution generation with the bert masked language model, and 2) substitution filtering and ranking with several features: bert prediction order, a bert language model, ppdb database, corpus-based word frequency, and fasttext similarity. @xcite presented access a controllable text simplification system based on sequence-to-sequence models. this system allows explicit control of simplification conditions such as length, amount of paraphrasing, lexical complexity, and syntactic complexity. access achieved sota results in text simplification benchmarks on the wikilarge test set. later on, @xcite introduced muss (an extended version) by fine-tuning bart @xcite with ac-cess, and the results were improved. in addition, sheang and saggion (2021) took a similar approach, adding another control token (number of words) and fine-tuning it with t5 @xcite . ## system description following recent works of @xcite , @xcite , sheang and saggion (2021), and @xcite , we are inspired to apply a similar approach in lexical simplification task. specifically, our model is based on sheang and @xcite , a model originally developed for sentence simplification foot_1 . we propose a controllable mechanism for ls because we believe that the embedded token values extracted from training data could give additional information to the model about the relations between the source and the target word; so that at inference, we could define different token values that fulfill our objectives, which in this case is to find the best candidates. in the following paragraphs, we describe all the details about each token and the reason why they are chosen. word length (wl) is the character length ratio between the complex word and the target word. it is the number of characters of the target word divided by the number of characters of the complex word. based on our analysis of the training dataset (tsar-en), 65.71% of the time complex word is longer than the best candidate, 21.30% the complex word is shorter than the best candidate, and 12.99% both are the same length. word rank (wr) is the inverse frequency of the target word divided by that of the complex word. the inverse frequency order is extracted from the fasttext pre-trained model. based on our analysis of the tsar-en dataset, 85.45% of the time, the complex word has a lower frequency than the best candidate. therefore, this token is a good indicator to help guide the model to predict simpler candidates. candidate ranking (cr) is the ranking order extracted from the training data. the values are given to candidates by the ranking order. e.g., the best-ranking candidate is given the value 1.00, the second 0.75, the third 0.50, the fourth 0.25, and starting from the fifth, it is given 0.00. we used only five different values to avoid overloading the model, as the training data is relatively small. in addition, the rationale behind using these values is that we want the model to learn candidates ranking from data through the training process rather than injecting additional information or doing postprocessing. ## experiments in our experiments, we compare our model with the current state-of-the-art model lsbert @xcite . we used the original lsbert configurations and resources, and we made the following changes to have a detailed comparison with our model. by default, lsbert returns only a single best candidate for each complex word, so we made the changes to return the 10 best-ranked candidates. we changed the number of bert mask selections from 10 to 15 so that after removing duplicate candidates, we still have around 10 candidates. moreover, we filtered out all the candidates that were equal to the complex word. due to the fact that all the used datasets have gold annotated simpler substitutions in all instances, we could assume that returning the complex word would be incorrect. ## results and discussion in in our experiments show that the modified lsbert had improved its accuracy@1 metric results with respect to the ones seen in the original lsbert paper @xcite : accuracy@1 has improved from 79.20 to 84.80 for lexmturk, from 61.60 to 67.59 for benchls, and from 43.60 to 44.76 for nnseval. on the other hand, for the accuracy@1 metric the conls system does not improve the results of the modified lsbert system but improves the results of the original lsbert for the lexmturk and benchls datasets. the results of the accuracy@k@top1 metric show that the modified lsbert achieves better results at @xmath0,2} and the conls achieves better results at @xmath1,4,5} for all datasets. this indicates that with more candidates allowed (3, 4, and 5 candidates) the conls is able to generate more instances with candidates within the top-1(s) gold annotated substitution(s) with respect to lsbert. the results of the potential@k metric show these facts: 1) in lexmturk and benchls, the conls is outperforming lsbert gradually and increasingly from @xmath2; 2) in nnseval, conls improves the potential of ls-bert only at @xmath3. for the map@k metric, we show that conls is able to improve the results of the metric at @xmath4,5,10} in all the datasets with respect to the modified lsbert. finally, the results of the precision@k and recall@k metrics show the same pattern: 1) for lexmturk, conls outperforms the lsbert in all @xmath5,5,10}; 2) for benchls and nnseval, conls outperforms the lsbert only in @xmath6,10}. we also conducted a comparison on the effect of different t5 models trained with tsar-en and evaluated with lexmturk. to evaluate the effectiveness of the control tokens, we conducted further experiments with different set of combinations. we trained and evaluated each set of tokens using t5-large with tsar-en for training and lexmturk for evaluation. the results on for future work, we plan to build a custom model to predict the best control token values from a given input instance. having instance-customized control token values seems more adequate, as humans usually select the best candidate based on context.