MANTIS at TSAR -2022 Shared Task: Improved Unsupervised Lexical Simplification with Pretrained Encoders (2022)

Abstract: In this paper we present our contribution to the TSAR-2022 Shared Task on Lexical Simplification of the EMNLP 2022 Workshop on Text Simplification, Accessibility, and Readability. Our approach builds on and extends the unsupervised lexical simplification system with pretrained encoders (LSBert) system introduced in Qiang et al. (2020) in the following ways: For the subtask of simplification candidate selection, it utilizes a RoBERTa transformer language model and expands the size of the generated candidate list. For subsequent substitution ranking, it introduces a new feature weighting scheme and adopts a candidate filtering method based on textual entailment to maximize semantic similarity between the target word and its simplification. Our best-performing system improves LSBert by 5.9% accuracy and achieves second place out of 33 ranked solutions.

## introduction lexical simplification (ls) is a natural language processing (nlp) task that involves automatically reducing the lexical complexity of a given text, while retaining its original meaning @xcite . since ls has a high potential for social benefit and improving social inclusion for many people, it has attracted increasing attention in the nlp community @xcite . ls systems are commonly framed as a pipeline of three main steps @xcite : (1) complex word identification (cwi), (2) substitute generation (sg), and (3) substitute ranking (sr), with cwi often being treated as an independent task. in this paper, we present our contributions to the english track of the tsar-2022 shared task on ls @xcite . focusing on steps (2) and (3) in the pipeline above, the task was defined as follows: given a sentence containing a complex word, systems should return an ordered list of "simpler" valid substitutes for the complex word in its original context. the list of simpler words (up to a maximum of 10) returned by the system should be ordered by the confidence the system has in its prediction (best predictions first). the ordered list must not contain ties. the task employed a new benchmark dataset for lexical simplification in english, spanish, and (brazilian) portuguese. the gold annotations consists of all simpler substitutes suggested by crowdsourced workers and checked for quality by at least one computational linguist who is native speaker of the respective language (for details, see @xcite ). contributing teams were provided with a small sample with gold standard annotations as a trial dataset. for english, this trial dataset consists of 10 instances of a sentence, a target complex word and a list of substitution candidates. the english test dataset consisted of 373 instances of sentence/complex word pairs. submission were evaluated in terms of ten performance metrics that fall into three groups: (1) map@k (mean average precision@k) for @xmath0, 3, 5, 10 candidate words. this metric evaluates a ranked list of predicted substitutes that is matched (relevant) and not matched (irrelevant) terms against the set of the gold-standard annotations for evaluation. (2) potential@k: @xmath1, 3, 5, 10. potential scores quantify the percentage of instances for which at least one of the substitutions predicted is present in the set of gold annotations and (3) accuracy@k@top1: @xmath2, 2, 3. accuracy scores represent the ratio of instances where at least one of the k top predicted candidates matches the most frequently suggested synonym/s in the gold list of annotated candidates. ## system description our contributions to the tsar shared task builds on and extends the approach to unsupervised lexical simplification with pretrained encoders -ls-bert -described in @xcite and @xcite . this approach leverages a pretrained transformer language models to generate contextaware simplifications for complex words. the lsbert simplification algorithm addresses two of three principal subtasks of ls: simplification candidate generation and substitution ranking. our approach extends lsbert in the following ways: (1) it utilizes a roberta transformer language model for simplification candidate generation and expands the size of the generated candidate list. (2) it introduces new substitution ranking methods that involve (i) a re-weighting of the ranking features used by lsbert and (ii) the adoption of equivalence scores based on textual entailment to maximize semantic similarity between the target word and its simplification. in submissions (runs) 2 and 3, we further explore the utility of crowdsourcing-and corpus-based measure of word prevalence for substitution ranking. the simplification algorithm underlying the three submissions described in this paper is shown in algorithm 1. in the following we describe the details of simplification candidate generation (2.1), substitution ranking (2.2) and obtaining equivalence scores (2.3). algorithm 1 lexical simplification input: sentence s, complex word w output: sorted suggestion list word_list 1: replace word w of s @xmath3: concatenate s and s ′ @xmath4: p(•|s, s ′ \ {w}) ← robert a(s, s ′ ) 4: scs ← top_probability(p(•|s, s ′ \ {w})) 5: all_ranks ← ∅ 6: for each feature f and its weight c f do 7: scores ← ∅ 8: for each @xmath5: scores ← scores ∪ f (sc) 10: end for 11: rank ← c f × rank_numbers(scores) 12: all_ranks ← all_ranks ∪ rank 13: end for 14: tot_rank ← sum(all_ranks) 15: word_list ′ ← sort_ascending(tot_rank) 16: word_list ← postproc(word_list ′ ) 17: return word_list