Findings of the TSAR -2022 Shared Task on Multilingual Lexical Simplification (2022)

Abstract: We report findings of the TSAR-2022 shared task on multilingual lexical simplification, organized as part of the Workshop on Text Simplification, Accessibility, and Readability TSAR-2022 held in conjunction with EMNLP 2022. The task called the Natural Language Processing research community to contribute with methods to advance the state of the art in multilingual lexical simplification for English, Portuguese, and Spanish. A total of 14 teams submitted the results of their lexical simplification systems for the provided test data. Results of the shared task indicate new benchmarks in Lexical Simplification with English lexical simplification quantitative results noticeably higher than those obtained for Spanish and (Brazilian) Portuguese.

## introduction lexical simplification @xcite ) is a sub-task of automatic text simplification @xcite that aims at replacing difficult words with easier to read (or understand) synonyms while preserving the information and meaning of the original text. this is a key task to facilitate reading comprehension to different target readerships such as foreign language learners, native speakers with low literacy levels or people with different reading impairments (e.g. dyslexic individuals). as such, it has gained considerable attention in the past few years @xcite . although lexical simplification systems can be developed following different architectural precepts, several studies have suggested the following pipe-lined approach: there exists a considerable body of research in lexical simplification for english @xcite @xcite . however, and in spite of several lexical simplification studies for languages other than english notably @xcite @xcite for spanish, @xcite for portuguese, @xcite for french, @xcite for chinese, @xcite for japanese and @xcite for swedish, there is a clear need to broaden the scope of lexical simplification in terms of language coverage. moreover, given its social relevance in making information accessible to broader audiences, we believe it is important to understand how far automatic systems can go in this task. we therefore established this first shared task on multilingual lexical simplification calling the nlp research community to contribute with methods to advance the state of the art. the task called for systems able to simplify words in context in (one or more of) three languages, namely english, portuguese, and spanish. systems have to deal with steps 2-5 above to generate, select, rank, and adapt to context substitutes for a given complex word in a sentence. as the result of our call for systems, of the 22 teams registered to the task, 14 sent their system outputs for evaluation. there were 31 different runs for english, 15 for spanish, and 14 for portuguese. this paper overviews the first shared task on multilingual lexical simplification. we describe in detail the task, the trial and test data used, the evaluation metrics, and the results. we also provide an analysis of the results and consider possible ways to expand the current scope of the task. ## task description the tsar-2022 shared task featured three tracks: in all tracks, the task was the same. given a sentence/context and one target (complex) word in it, provide substitutes for the target word that would make the sentence easier to understand. it was allowed to submit up to 10 substitutes, ordered from the best to the least fitting/simple one. ties were not allowed. participants were provided with several trial examples in each language. training datasets were not provided. however, participants were allowed to use any external resources for building their lexical simplification systems. participating systems were evaluated on test sets using several metrics. foot_0 ## participating systems we received the outputs of 13 teams for english, 6 for spanish, and 5 for (brazilian) portuguese. each team was allowed to submit outputs of up to 3 systems. this totaled to 31 submitted outputs for english, 15 for spanish, and 14 for portuguese. cils @xcite submitted three systems for the english track. all systems use the model prediction score and embedding similarity score for candidate generation. a model prediction score is computed using the xlnet model (yang 8 https://github.com/lastus-taln-upf/ tsar-2022-shared-task et al., 2019) given the context and the target word with any word in the vocabulary of xlnet. the embedding similarity score is the inner product of the embedding of the target word and the embedding of the respective word. the three systems differ in the ranking module. they rank the candidates based on different combinations of scores such as 1) the score from the candidate generation; 2) sentence similarity score (cosine similarity between the source and target sentence); 3) gloss sentence similarity score (the cosine similarity between the target word and the candidate); 4) wordnet score (a cosine similarity between the target word and the candidate extracted from wordnet); and 5) validation score (a cosine similarity of the bert-base between the source and target sentence). presiuniv (whistely et al., 2022) uses masked language model (follows lsbert) for candidate generation, ranks candidates by cosine similarity (extracted from fasttext), and then filter them out by checking part-of-speech. the systems for the three languages are the same, except that the language model is specific for each language. it is interesting that this approach works the best on spanish dataset, but not so well on the portuguese and english datasets (lower than the baseline). uom&mmu @xcite uses an approach that consists of three steps: 1) candidate generation based on different prompt templates (e.g., <easier, @xmath0, @xmath1; 2) fine-tuning of a language model (bert-based model) to select and rank can-didates; and 3) post-processing to filter out noise and antonyms. this approach achieves the second rank on the spanish dataset and the third rank on the english dataset, but to our surprise, the model ranks the lowest on the portuguese dataset. polyu-cbs @xcite proposes three approaches for the candidate ranking. in all three approaches, the candidates are generated using a masked language model. then, the first approach ranks candidates based on the probability received from the candidate generation (base probability) and sentence probability extracted from gpt-2 pre-trained model by replacing the target word with its candidate. the second approach ranks candidates by base probability and masked language model scoring @xcite . the third model ranks candidates by base probability and contextualized embedding similarity (cosine similarity between the target word and its candidate in the context of the original sentence). based on the official results, the third approach performs better than the other two in all languages. cental @xcite explored the use of masked language model for candidate generation with three strategies for context expansion: copy, query expansion, and paraphrase. the copy strategy is a copy of the sentence itself (follows that of lsbert). the query expansion strategy extracts alternative words for the target word from fasttext and then replaces the original sentence with each alternative word. the paraphrase strategy (english only) extracts paraphrases from pegasus @xcite . the authors propose three ranking approaches: 1) using the frequency of words generated by the three strategies; 2) training a binary classifier (english only) for the ranking; 3) the english ranking module (the binary classifier) performs cross-lingual ranking for spanish and portuguese. mantis @xcite adapts masked language model (roberta) for candidate generation and performs the candidate ranking with three different approaches. the first ranking approach uses three features with different weights to rank the candidates: 1) pre-trained language model feature (the probability of the candidate extracted during the candidate generation), 2) word frequency, and 3) semantic similarity (cosine similarity between the fasttext vector of the target word and the candidate). the second and third approaches rank candidates by word prevalence and equivalence score. the second approach uses crowd-sourcing word prevalence, which is a proportion of the population that knows a given word based on a crowd-sourcing study involving 220,000 people @xcite . the third approach uses corpus-derived word prevalence, which is an estimate of the number of books that a word appears in @xcite . the equivalence score is the entailment score of the original sentence and the sentence replaced with the candidate. the experimental results have shown that the first approach performs better than the other two. unihd @xcite submitted two systems. the first system was a zero-shot prompted gpt-3 with a prompt asking for simplified synonyms given a particular context. simplifications are then ranked. the second system was an ensemble over six different gpt-3 prompts/configurations with average rank aggregation. the second system attained the highest score for english on all metrics. the approach is simplistic in nature, relying heavily on the underlying language model which is only available for research through a paid interface. rcml (aleksandrova and brochu dufour, 2022) proposes a system (english only) by applying the lexical substitution framework lexsubgen (based on xlnet) for candidate generation and ranks the candidates based on grammaticality (pos + morphological features), meaning preservation (bertscore of the source and target sentences), and simplicity (predicted by an svm classifier trained on cefr level data). gmu-wlv @xcite submitted two models for each of the three languages. these two models follow the approach of lsbert, except the second model uses an additional zipf frequency in the candidate ranking module. the first model performs the best on the portuguese dataset. team language approach cils en sg: language model (lm) probability and similarity score, sr: sg score, cosine similarity scores presiuniv en, es, pt sg: masked language model (mlm), sr: cosine similarity, pos check uom&mmu en, es, pt sg: lm with prompt, sr: fined-tuned bert model as classifier polyu-cbs en, es, pt sg: mlm, sr: mlm probability, gpt-2 probability, sentence probability, cosine similarity cental en, es, pt sg: mlm, sr: word frequency, binary classifier teampn en sg: mlm, verbnet, ppdb, knowledge graph, sr: mlm probability mantis en sg: mlm, sr: mlm probability, word frequency, cosine similarity unihd en gtp-3 prompts: zero-shot, few-shot rcml en sg: lexical substitution, sr: pos, bertscore, svm classifier gmu-wlv en, es, pt sg: mlm, sr: mlm probability, word frequency a comparative of system approaches is provided in ## results and discussions in the following subsections we describe the results obtained by the participant teams for each of the tracks in the tsar 2022 multilingual lexical simplification shared task. 9 note that we will base our description on the ranking obtained by sorting submissions according to the acc@1 metric as well as summarizing methods for which a paper has been submitted and accepted for the shared task (see section 3). we also provided an extended version of the results, 10 which included acc@1, potential@k, map@k, macro-averaged precision@k, macroaveraged recall@k, and accuracy@k@top1, for @xmath2, 2, ..., 10}. precision@k and recall @k were defined as follows: overall, we observe that several systems achieved new state-of-the-art results in the different tracks overtaking a previous competitive neural language model for lexical simplification (ls-bert). 9 please note that official results can also be queried at https://taln.upf.edu/pages/tsar2022-st/#results 10 https://github.com/lastus-taln-upf/ tsar-2022-shared-task/tree/main/results/extended ## conclusions and further work lexical simplification, the task of replacing difficult words in a sentence by easier to read or understand synonyms preserving the meaning of the orig-inal sentence is an important problem which has gained considerable attention in the past few years. in spite of its popularity for english, the task has attracted less research for other languages. considering its social relevance in today's digital world, we put forward the first shared task on multilingual lexical simplification addressing three languages: english, (brazilian) portuguese, and spanish, and called the research community to challenge the state of the art. to carry out the task, we have prepared three datasets, one per each language, following similar data collection and data annotation approaches, leading the way to the development of future datasets for additional languages. the datasets are composed of sentences each containing a single complex word which needs to be simplified. although the datasets were intended only for testing the participating systems, a small (between 10 and 12 instances) portion was released as trial data. this was particularly useful for several teams to fine-tune their computational methods or prompts. the task also featured two baselines: one based on a competitive neural approach, and another one on a traditional (dictionary-based) pipe-lined architecture. the shared task attracted a considerable number of participants with a total of 60 systems' runs submitted across the three tracks. several systems outperformed the competition, setting a new benchmark in lexical simplification. it is observed that pre-trained masked language models when fine-tuned to the lexical simplification task produce very competitive approaches in combination with additional syntactic/lexical resources or corpora. the submitted systems relied heavily on pretrained language models, which are known to hallucinate (i.e., generate non-factual statements based on previously seen contexts). in the context of substitution generation, hallucination may indicate that incorrect simplifications are returned when the context is under-specified or unfamiliar. further work to ensure that the simplifications generated by such systems are faithful to the original text and are factual in nature will help to engender a culture of security and trust in simplification research. in our dataset, we have not considered a key aspect of simplification, which is the user. our datasets assume that there is one correct simplification that is the best simplification for all users. in fact, our 'best' simplification is collected from many users and is based on the most frequently returned simplification. it is interesting to note that when asked to simplify the same word in the same context, users will answer differently. it is logical to conclude then, that a simplification system must return a term which is appropriate to a user. concerning the selected evaluation metrics, although the map@k metric takes into account the order of returned items and it is very useful for cases when multiple relevant items are expected, it has the disadvantage that the relevance of the returned items is binary. so, in further work, it could be included a metric that could take into account the possibility of graded or weighted relevance allowing the participants to submit a weight associated to each prediction and allow ties. finally, we note that our evaluation methodology is entirely automated, due to the constraints of a shared task environment. whilst this is very useful for developing systems for lexical simplification, we strongly encourage those working on in-production systems to directly evaluate the resulting systems with the user bases that they are intended for. automated evaluation is secondary to human evaluation, and this is especially true in simplification where the goal is to enable the user to better understand the original information.