GMU - WLV at TSAR -2022 Shared Task: Evaluating Lexical Simplification Models (2022)

Abstract: This paper describes team GMU-WLV submission to the TSAR shared-task on multilingual lexical simplification. The goal of the task is to automatically provide a set of candidate substitutions for complex words in context. The organizers provided participants with ALEXSIS a manually annotated dataset with instances split between a small trial set with a dozen instances in each of the three languages of the competition (English, Portuguese, Spanish) and a test set with over 300 instances in the three aforementioned languages. To cope with the lack of training data, participants had to either use alternative data sources or pre-trained language models. We experimented with monolingual models: BERTimbau, ELECTRA, and RoBERTA-largeBNE. Our best system achieved 1st place out of sixteen systems for Portuguese, 8th out of thirty-three systems for English, and 6th out of twelve systems for Spanish.

## introduction text simplification (ts) is an important nlp application that consists of applying automatic methods to make texts more accessible to various target populations, such as children @xcite , second language learners @xcite , individuals with low-literacy levels @xcite , and individuals with reading disabilities @xcite @xcite . the core component of ts systems is lexical simplification (ls) which addresses the simplification of single complex words, complex multi-word expressions or both. ls is a multi-stage process. in the first step, systems need to recognize words that are likely considered to be hard to understand by a given target population. this step is known as complex word identification (cwi) @xcite or lexical complexity prediction (lcp) @xcite @xcite . the second step in ls systems is to provide suitable candidate substitutions for complex words also known as substitute generation (sg) @xcite @xcite . these candidate substitutions are then filtered in regards to their suitability, known as substitute selection (ss) @xcite , and then ranked in accordance to their simplicity, referred to as substitute ranking (sr) @xcite @xcite . the most appropriate candidate is then selected to replace the complex word. while most of the work in ls deals with english, recent advances in multilingual and cross-lingual nlp models have motivated the study of multilingual models and datasets for ls with the goal of improving performance for languages other than english @xcite @xcite . the text simplification, accessibility, and readability (tsar-2022) shared-task @xcite follows this trend by providing participants with a multilingual ls dataset containing annotated data in english, portuguese, and spanish following the alexis protocol @xcite . in this paper, we present team gmu-wlv's submissions to tsar-2022 where we evaluate multiple models for this task. we describe prior methods of sg (section 2), the task and data (section 3), our model architecture (section 4), and results (section 5). ## related work as discussed by @xcite , various approaches have been used for ls. early approaches relied on predefined lists of complex words with candidate substitutions @xcite . wordnet @xcite is another widely used resource. numerous sg systems take the synonyms provided by wordnet as valid simplifications of a complex word @xcite @xcite while others use wordnet's list of hyponyms and hypernyms to identify and rank suitable replacements @xcite . finally, some combine wordnet with other datasets consisting of linguistic features indicative of a word's complexity, such as the psycholinguistic database @xcite . more recent approaches have used transformerbased models that are able to more effectively capture and utilize contextual information as described by @xcite . @xcite used a pretrained bert model to generate candidate substitutions using masked language modelling (mlm) @xcite . @xcite experimented with multiple pre-trained multilingual and monolingual transformers for mlm to generate spanish candidate substitutions, including beto @xcite , mbert @xcite , bertin @xcite , roberta-base-bne, and roberta-large-bne @xcite . ## task and data the tsar-2022 shared task was hosted at the empirical methods in natural language processing (emnlp) conference. participants were tasked with creating an ls system that returns an ordered list of a maximum of 10 potential candidate substitutions for a given complex word. tsar-2022 supplied participants with datasets in english, portuguese @xcite , and spanish @xcite ) each having their own track within the competition @xcite . the task received 33, 17 and 16 entries in the english, spanish, and portuguese tracks respectively. the datasets contained excerpts from journalistic texts and wikipedia articles. the english and spanish datasets contained extracts from wikinews and wikipedia articles, whereas the portuguese dataset contained extracts from locally sourced brazilian newspapers. the portuguese dataset is the only variety-specific dataset of the three containing only brazilian portuguese texts. the three datasets are comparable in terms of size. the english dataset consisted of 386 instances, the spanish dataset contained 381 instances, and the portuguese dataset had 386 instances. each dataset was split into trial and test sets and were provided to the participants with the trial set being released approximately 2 months prior. the trial set had only 10-12 instances per language, whereas the test set contained 369-376 instances per language. the test set did not contain the candidate substitution for each instance's complex word. the datasets were formatted as follows: <@xmath0.word>, providing the original context for each complex word. ## gmu-wlv: system description we approached this task with two model architectures inspired by the performance of large pre-trained monolingual transformers @xcite . we submitted two unsupervised models for each track due to the limited size of the development and train sets. the first model consisted of a pre-trained monolingual transformer with substitute ranking of the probabilities produced by mlm, which we name gmu-wlvvanilla. the second model consisted of the same transformer model but with zipf frequency for additional substitute ranking, which we name gmu-wlv-zipf. both gmu-wlv-vanilla and gmu-wlv-zipf models conducted mlm similar to that described in @xcite . we masked the complex word of the original sentence and fed both the original sentence and the masked sentence separated by a [sep] token to predict the masked token or in this case, the candidate substitution. roberta-large-bne foot_0 was seen to perform well for spanish by @xcite . as such, we selected several large pre-trained monolingual models for each track. for english, we used electra foot_1 @xcite , for spanish we used roberta-large-bne @xcite , and for portuguese we used the bertimbau model foot_2 @xcite . roberta-large-bne was pre-trained on the national library of spain (biblioteca nacional de españa) corpus @xcite containing 135 billion spanish tokens extracted from crawling all .es domains. electra was pre-trained on english wikipedia data with a vocabulary size of 30522 tokens (clark in regards to zipf frequency ranking, we used the wordfreq python library @xcite to rank candidate substitutions. inspired by previous work in cwi and lcp @xcite @xcite , we pose that those candidate substitutions with a higher zipf frequency would be considered more familiar to the user and therefore would be considered less complex than compared to those with a lower zipf frequency. ## results the results obtained by gmu-wlv-vanilla are presented in ## discussion we believe that our gmu-wlv-vanilla model's performance on the portuguese track was a result of it being a large pre-trained model trained only on brazilian portuguese data @xcite . gmu-wlv-vanilla model's competitive performance on the english and spanish tracks was also likely due to the use of large monolingual models. we were hoping that multilingual models may be able to transfer useful information learned from the vector representations of multiple or similar languages, such as spanish, to the target language, for instance, portuguese. however, during our experimentation, multilingual models, such as mbert @xcite or xlm-r @xcite , were found to produce candidate substitutions in languages other than the target language. removing these words still resulted in a list of candidate substitutions that appeared to be less suitable than those produced by the monolingual models. this was also found to be the case after having applied zipf frequency ranking. we had previously theorised that ranking candidate substitutions per their zipf-frequency would produce a list of candidate substitutions ordered from most to least familiar for a specific or general target audience. nevertheless, given that the performance of our gmu-wlv-zipf model was worst than that of our gmu-wlv-vanilla model, we concluded that zipf-frequency ranking was not in alignment with the annotators' notion of simplicity, regardless of language. given the complex word "colapsó" (collapsed), our gmu-wlv-zipf model generated several candidate substitutions, including "hizo" (made), "puso" (put), "detuvo" (stopped), and "acabara" (finished). without taking the meaning of the complex word or its context into consideration, "hizo" (made) or "puso" (put) would be the most logical candidate substitutions as they are shorter and more common in comparison to the other candidates. however, they do not have the desired meaning in this context. on the other hand, "detuvo" (stopped) or "acabara" (finished) are more semantically similar to the complex word despite being longer and less common. for this reason, zipf-frequency is not always a useful feature for substitute ranking. ## conclusion and future work this paper presents gmu-wlv's submission to the tsar shared-task on multilingual lexical simplification. our gmu-wlv-vanilla model came first place at generating candidate substitutions for portuguese, eighth for english, and sixth for spanish. we demonstrate the importance of relying upon monolingual models for sg with pretrain models, such as bertimbau and roberta-large-bne, performing exceptionally well. we also show that the use of zipf-frequency ranking for substitute ranking may result in inferior candidate substitutions being selected for simplification. transfer learning allows for the utilization of large pre-existing datasets to under-resourced nlp-related tasks, such as ls of portuguese or spanish. we hope to experiment with transfer learning on a number of datasets related to ls but are not formatted in such a way as to allow for the direct training of sg models, including datasets such as the complex dataset @xcite , a large pre-exsisting dataset containing continuous lexical complexity values, or the binary comparative complex dataset @xcite , a somewhat smaller dataset consisting of comparative judgements between lexical complexities. we hypothesize that transfer learning will substantially increase the performance of our models.