CENTAL at TSAR -2022 Shared Task: How Does Context Impact BERT -Generated Substitutions for Lexical Simplification? (2022)

Abstract: Lexical simplification is the task of substituting a difficult word with a simpler equivalent for a target audience. This is currently commonly done by modeling lexical complexity on a continuous scale to identify simpler alternatives to difficult words. In the TSAR shared task, the organizers call for systems capable of generating substitutions in a zero-shot-task context, for English, Spanish and Portuguese. In this paper, we present the solution we (the cental team) proposed for the task. We explore the ability of BERT-like models to generate substitution words by masking the difficult word. To do so, we investigate various context enhancement strategies, that we combined into an ensemble method. We also explore different substitution ranking methods. We report on a post-submission analysis of the results and present our insights for potential improvements. The code for all our experiments is available at https://gitlab.com/Cental-FR/cental-tsar2022 .

## introduction lexical simplification (ls) aims at identifying words that are considered too difficult for a given audience and replacing them with simpler substitutes. @xcite following housen and simoens (2016, 166), we distinguish the notion of absolute complexity that refers to "inherent linguistic properties of a language feature" from the notion of difficulty, which depends on "how costly, demanding, or difficult a given language feature is for a given language learner in a given learning context, particularly in terms of the mental resources allocated and cognitive mechanisms." the tsar shared task (saggion et al., 2022) asks for solutions generating and ranking substitutes for predefined difficult words in sentences in english, spanish and portuguese. this paper @xcite for a recent description of text simplification and lexical complexity, see north et al. (2022). the remainder of this paper is organized as follows: section 2 describes the task proposed in the tsar shared task, their corpora and the additional corpora that we use. section 3 details the proposed solution for generating and ranking substitutions while their results are shown in section 4. finally, in section 5, we present the error analysis and possible solutions for improving the performance of the proposed methods. ## task and corpora the tsar shared task proposes a zero-shot task, where a trial set composed of only 10 trial sentences with difficult words and their substitutions and later assessed the systems on a test corpus for english, spanish and portuguese. the corpus consists of sentences with one difficult word per sentence to be substituted. the tsar corpus is consti-tuted of 1,115 sentences with target words (373 for english, 368 for spanish and 374 for portuguese) annotated by 25 crowdsourced workers, whose sociodemographics are not provided. they proposed simpler substitutions for the difficult words, taking the sentence as context. an expert later selected the proposals and only non-multiword expressions were kept (saggion et al., 2022). @xcite we used additional corpora for parameter optimization and hyperparameter tuning of the classification algorithm used in our ranking approach, given the zero-shot nature of the task. for english, we used a monolingual lexical simplification corpus (specia et al., 2012) constituted of 2,010 english sentences annotated with difficult words and their ranked substitute words or phrases. for spanish, we selected a cross-lingual lexical substitution corpus (mihalcea et al., 2010) constituted of 1,300 english sentences, which are a subset of the monolingual corpus, in which the substitutes are in spanish. to obtain both sentences and substitutions in spanish and portuguese, we used the google vision translation api to translate the english sentences from the cross-lingual corpus to spanish and the sentences and substitutions from the monolingual corpus to portuguese. after translating the corpora, we automatically marked the difficult words using the list of substitutions (i.e., simpler words). @xcite we divided this corpus into 80% for training and hyperparameter tuning (using crossvalidation) and 20% for testing. the testing part is used for internal comparison of the methods described in sections 3.1 and 3.2 and the training part is used in the ranking method (section 3.2). ## our approach we detail here the runs submitted (2 for english and 3 for spanish and portuguese each). figure 1 illustrates our pipeline, and ## evaluation our evaluation of the 8 runs submitted (one for each ranking method 8 ) focuses on the map/potential@1 metric (@1 in our tables). all official metrics adopted by the shared task are in appendix a. lang method @1 rank en qe bert l 1 .4155 21 qe bert l 2 .5281 8 es qe roberta l 1 .3109 10 qe roberta l 2 .4477 1 pt qe bert 1 .4090 5 qe bert 2 .4759 2 en copy b u .4959 12 copy l u .5040 11 copy roberta b .4772 14 copy roberta l .3994 23 es copy c .4211 2 copy u .2989 12 pt copy b .4331 4 copy l .4705 3 en paraphrase .2171 36 ## error analysis to better understand our results, we evaluated the substitution generation and the ranking strategies. we also measure the gap between our best ranking model (vote) and a perfect substitution generation step (i.e., an oracle). for the sg step, our methods rely on providing bert models with a single mask, but they cannot produce multiword expressions. to identify the impact of this limitation, we calculated their proportion in the gold standard: 3.35% for english, 6.27% for spanish, and 2.97% for portuguese. we also studied the extent to which substitutions generated by our methods were grammatically correct regarding the context. to do so, we compared the morpho-syntactic information of each candidate against its respective difficult word, after analyzing the sentences with stanza (qi et al., 2020), assuming the parser output is correct. out of all the candidates present in our submitted runs, there was a mismatch in 10.68% of the cases for english, 6.09% for spanish, and 12.28% for portuguese. we corrected those mismatches by using dela dictionaries (courtois, 1990). @xcite whenever exceptions such as repeatb u for english. a mismatch was detected, we converted the stanza information to the dela format. using the candidate's lemma, we checked whether an inflected form with the same morpho-syntactic information existed. if it did, we replaced the candidate with the correct form, otherwise, we deleted the candidate from the list. we can see that there is a slight improvement (up to .03 on map/potential@1), indicating that while it solves issues, inflection is not the main shortcoming of the submitted lists. @xcite in future work, we would like to apply this correction phase to each individual model's output in order to apply the ranking to morpho-syntactically correct candidates. in as for the ranking methods, we see that for spanish and portuguese, voting produces better results than ranking, while for english, ranking produces better results than voting. we hypothesize that voting prioritizes frequent and contextually suitable words that are generated by multiple methods, while ranking performs better on the tail end of the distribution. to test it, we used the ranking system exclusively to break ties created by the vote. this produces slightly better results than a full ranking in all cases, indicating that the ranking does indeed learn about simple words, yet does not have enough information on its own to rank a full list in the order given by the gold standard. we also explored the importance of a substitution selection method, instead of a simple filter. to do so, we analyze the best possible results using all the generated substitutions for the voting method. so, we drop all generated words that are not in the gold standard and apply the same voting method. this substitution selection is exemplified in the line "oracle+ss" in it is interesting to note that the results of the substitution generation methods outperform our ranking methods, including vote, which only counts the unitex-lingua/tree/master/ 11 ## conclusion this paper presented the solution proposed by the cental team in the tsar shared task on lexical simplification. we proposed three substitution generation strategies, where we saw that query expansion is superior. moreover, generation strategies can produce and sort suggestions with good performance. the query expansion strategy could achieve 8 th , 1 st and 2 nd positions for english, spanish and portuguese respectively by itself. we also identified that the voting method might produce promising results, but a good substitution selection step is required. this step would improve morphologically incorrect substitutions and remove semantically/contextually inappropriate substitutions. in addition, the ranking methods can be useful for breaking ties in voting. @xcite