ALEXSIS - PT : A New Resource for P ortuguese Lexical Simplification (2022)

Abstract: Lexical simplification (LS) is the task of automatically replacing complex words for easier ones making texts more accessible to various target populations (e.g. individuals with low literacy, individuals with learning disabilities, second language learners). To train and test models, LS systems usually require corpora that feature complex words in context along with their potential substitutions. To continue improving the performance of LS systems we introduce ALEXSIS-PT, a novel multi-candidate dataset for Brazilian Portuguese LS containing 9,605 candidate substitutions for 387 complex words. ALEXSIS-PT has been compiled following the ALEXSIS-ES protocol for Spanish opening exciting new avenues for cross-lingual models. ALEXSIS-PT is the first LS multi-candidate dataset that contains Brazilian newspaper articles. We evaluated three models for substitute generation on this dataset, namely mBERT, XLM-R, and BERTimbau. The latter achieved the highest performance across all evaluation metrics.

## introduction the development of lexical simplification (ls) systems provides a cost-effective means of making texts accessible to individuals with reading disabilities or low-literacy who are at an economic and social disadvantage @xcite . ls systems aim to replace difficult to understand (complex) words or phrases with simpler alternatives @xcite . consider the examples in various studies have been published on brazilian portuguese (pt-br) ls @xcite @xcite . brazil is a country with important literacy challenges where only 1% of its population working in the agricultural sector are proficient readers @xcite . these educational challenges motivate the development of technology to assist readers and the creation of resources for pt-br. one of the most popular pt-br ls datasets is simplex-pb 2.0 @xcite featuring substitutions for over 1,500 complex words. however, in simplex-pb 2.0 only 5 ranked candidate substitutions are available for 730 of its complex words making the dataset less useful as a benchmark for state-of-the-art large pre-trained language models. moreover, it contains texts from children's books which makes it domain-specific and therefore not a great fit for general ls. to address these gaps we introduce alexsis-pt, a pt-br ls dataset featuring excerpts of newspaper articles containing a larger number of candidate substitutions per target word (up to 25). to the best of our knowledge, alexsis-pt has the highest average number of pt-br ranked candidate substitutions per complex word. alexsis-pt has been compiled according to the alexsis protocol for spanish @xcite , henceforth alexsis-es, opening the possibility of using cross-lingual models for these languages. alexsis-pt is one of the official datasets of the tsar shared task @xcite . the main contributions of this paper are: 1. alexsis-pt, the first multi-candidate dataset for the development and evaluation of ls systems for pt-br newspaper articles. trained a bert-based model to generate top-k candidate substitutions for their english dataset using masked language modelling (mlm). others have used various transformer-based models for substitute ranking as well as precursor tasks, such as complex word identification or lexical complexity prediction @xcite . ## alexsis-pt we created a new dataset for pt-br ls containing newspaper articles, referred to as alexsis-pt. we did this since previous pt-br ts datasets are either not of the newspaper genre (simplex-pb 2.0 and 3.0) or do not contain pre-identified candidate substitutions for ls (portsimple or porsimplessent). alexsis-pt contains a total of 387 instances with 348 of these instances having unique complex words. each instance is taken from the porsimplessent dataset @xcite and is retrieved from newspapers. porsimplessent is essentially a collection of original and simplified sentences thus not containing individual complex word annotations. therefore, we had to carry out word alignment between the original and complex instances to identify complex words. this alignment was manually checked by a linguist and only the instances containing complex words that were deemed to be correctly identified were later included in the crowdsourcing platform, mturk, for annotators to provide candidate substitutions. as show in context suggestions os sedimentos são arrastados para a parte baixa do rio. resíduos [waste] (9), detritos [debris] (6), lixos [garbage] (2), ... en: sediments are carried to the lower part of the river. fragmentos [fragments] (2), camadas [layers] (1), ... simpatizantes foram arregimentados. agrupados [grouped] (10), reunidos [gathered] (7), ... en: supporters were enlisted. convocados [summoned] (2), arrebanhados [herded] (1), ... neste ano ocorrerão quatro ações simultâneas conjuntas [joint] (7), ao mesmo tempo [at the same time] (7), ... en: this year four simultaneous actions will occur. juntas [together] (4), paralelas [parallel] (3), ... os partidos estão mais cautelosos cuidadosos [careful] (12), prudentes [prudent] (3), ... en: the parties are more cautious and careful. precavidos [cautious] (3), comedidos [restrained] (1), ... as testemunhas contrariam esta versão negam [deny] (15), desmentem [deny] (2), ... en: the witnesses contradict this version. discordam [disagree] (2), desdizem [unsay] (1), ... ## substitute generation we developed and evaluated four transformer models for substitute generation, the first step in ls pipelines. the four models are available at hugging face. three of them are multilingual, being multilingual mdistilbert (mdistilbert) @xcite , multilingual bert (mbert) @xcite and xlm-r @xcite , while one model is pre-trained solely on pt-br, bertimbau @xcite . we followed a similar mlm strategy to @xcite where we masked the complex word of the original sentence and fed both the original sentence and the masked sentence separated by a [sep] token to predict the masked token. the parameters of these models are displayed in on all of our evaluation metrics. this is due to bertimbau being pretrained on a single large pt-br dataset rather than on multiple languages like mdistilbert, mbert and xlm-r which were found to produce candidate substitutions that were either in european portuguese or another language entirely. generating top-@xmath2. the bertimbau model achieved a precision of 0.212 when tasked with supplying top-@xmath3, yet attained an inferior precision when set to return top-@xmath4, 10, or 50 (figure 1 ). this showed that our models were successful at predicting ground truth labels when producing a small number of candidate substitutions. as we increase the number of top-k candidate substitutions generated we saw an increase in all models' potential and recall scores (figure 1). unsurprisingly, this indicates that with a greater pool of candidate substitutions, it was more likely that our models' would successfully predict multiple ground truth labels. our bertimbau model achieved the highest potential and recall scores of 0.888 and 0.346 respectively ( comparing models performances across datasets, we can see that all models achieved better performances on alexsis-pt in comparison to simplex-pb 3.0, with the exception of mdistilbert when set to generate top-@xmath5, 3] candidate substitutions. there are two likely explanations: (1). simplex-pb 3.0 is domain specific and therefore pretrained language models may be unable to simplify vocabulary or jargon related to its genre of children's texts, and/or (2). simplex-pb 3.0 contains only 5 ranked candidate substitutions, thus the prediction of 10 or more candidate substitutions is less rewarding in regards to improving overall performance. @xcite compares the performances of lsbert @xcite on alexsis-pt, alexsis-es @xcite , as well as a third english dataset akin to the other two datasets. all three datasets consist of a similar number of instances (complex words in context) being 386 instances, and candidate substitutions per complex word. it was found that lsbert achieved the greatest accuracy of 30.8 on the english dataset, with alexsis-pt achieving the second greatest accuracy of 15.5. in comparison, alexsis-es produced an inferior accuracy of 9.7.