I rekia LF es: a New Open Benchmark and Baseline Systems for S panish Automatic Text Simplification (2022)

Abstract: Automatic Text simplification (ATS) seeks to reduce the complexity of a text for a general public or a target audience. In the last years, deep learning methods have become the most used systems in ATS research, but these systems need large and good quality datasets to be evaluated. Moreover, these data are available on a large scale only for English and in some cases with restrictive licenses. In this paper, we present IrekiaLF_es, an open-license benchmark for Spanish text simplification. It consists of a document-level corpus and a sentence-level test set that has been manually aligned. We also conduct a neurolinguistically-based evaluation of the corpus in order to reveal its suitability for text simplification. This evaluation follows the Lexicon-Unification-Linearity (LeULi) model of neurolinguistic complexity assessment. Finally, we present a set of experiments and baselines of ATS systems in a zero-shot scenario.

## introduction according to the un, illiteracy affects 16 per cent of the world population (759 million adults) 1 and the number of children experiencing reading difficulties has increased from 460 million to 584 million after the covid-19 pandemic. 2 moreover, in the oecd countries, between 4.9 % and 27.7 % of adults only has proficiency at the lowest levels in literacy @xcite and regarding the young people 10 % of new graduates have low literacy skills @xcite . due to these facts, plain language 3 and easy-toread initiatives 4 give some guidelines to make texts more accessible. basically, their recommendations 1 https://www.un.org/en/chronicle/ article/education-all-rising-challenge 2 https://news.un.org/en/story/2021/03/ 1088392 3 https://www.plainlanguage.gov/ 4 https://www.inclusion-europe.eu/ easy-to-read/ can be summarised in writing for the audience, organising the information, using short and positive sentences, using active instead of passive voice, choosing words carefully, being concise, employing an appropriate design for smooth reading and, in the case of online communication, following the web standards. in this line, text simplification seeks to reduce the complexity of texts (at the lexical, syntactic and discourse levels) for a general public or a target audience. as adapting the texts manually is a hard-working task, researchers in natural language processing (nlp) have tried to automatise it since the mid 90s. the pioneers were @xcite and their main motivation was related to the problems that long and complex sentences caused in advanced nlp applications. nowadays, however, most of the research on automatic text simplification (ats) focuses on human target audience @xcite . as other nlp tasks, ats has evolved from rulebased systems, statistic systems, hybrid systems mixing hand-crafted rules and machine learning, to the present deep learning paradigm. the interested reader is referred to the following state-of-the-art reports for more information about the evolution of ats @xcite @xcite @xcite . current deep learning techniques, however, need extensive data and these data are mainly available for english. moreover, some corpora do not have open licenses. there are nlp techniques to alleviate this problem such as transfer learning and cross-lingual learning, but, even in these cases, high-quality evaluation benchmarks are needed. in this paper, we present irekialf_es, an open corpus and benchmark for spanish ats systems. irekialf_es compiles texts published by the basque government in both original and easy-toread format. the corpus is divided in a document-level version containing 288 documents, and a sentence-level version, where 35 of them have been manually aligned. the corpus is available with an open license. 5 furthermore, in order to reveal its quality and suitability for ats, we have evaluated neurolinguistic complexity of the corpus following the lexicon-unification-linearity (leuli) model @xcite . this model of neurolinguistic complexity assessment is entirely inspired by @xcite @xcite @xcite )'s memory, unification, control (muc) model of language neurobiology. finally, we have evaluated three different systems that will serve as baselines for future research in this corpus. this paper is structured as follows: right after this introduction (section 2), we present the related work on simplified corpora; in section 3 we describe the methodology to build the irekialf_es corpus; in section 4 we summarise the leuli model and provide the rationale for its use; in section 5 we carry out a leuli-based complexity evaluation of the corpus; in section 6 we show the experiments with the three baseline systems; and we conclude with the take-home messages and outline the future work in section 7. ## related work corpora or datasets for ats are built with two main objectives: on the one hand, to study the process and operations carried out when simplifying texts e.g. by annotating the operations @xcite @xcite , and, on the other hand, to use them as resources to build and evaluate machine learning systems (see next paragraph for references). when creating and compiling corpora of simplified texts, the strategies (intuitive or structural) and the target audiences can be different @xcite . hence, there is no unique answer or simplified correct sentence for a given complex or original sentence. a recent overview on the creation of ats corpora can be found in @xcite . the main research in text simplification, both from an educational perspective and from an nlp perspective, has focused on english and, therefore, the majority of corpora as well as the largest ones are in such language @xcite @xcite . the most used datasets for nlp are i) those derived from wikipedia and simple wikipedia (and therefore with open licences): wik-ismall, originally created by @xcite and adapted by @xcite , wikilarge, compiled by @xcite and usually used for training, turkcorpus @xcite and asset (alva-manchego et al., 2020a) used for evaluation and ii) newsela @xcite , which has proprietary licence but can be obtained for research. in order to study the document level simplification, d-wikipedia dataset has been proposed @xcite . however, there are also corpora for other languages such as brazilian portuguese @xcite @xcite , danish @xcite , german @xcite @xcite , italian @xcite @xcite , basque @xcite and french @xcite . recently, a multilingual corpus of news has been compiled that includes finnish, french, italian, swedish, english and german @xcite . regarding spanish, the first ats corpus was developed in @xcite 's project, which aimed to build an ats system guided by the socalled easy-to-read principles. the corpus, named simplext, was created manually by trained experts and the target audience were students with down syndrome. an analysis of the operations needed to simplify the original text revealed that the most frequent operations in simplext were change (transformation), delete, insert and split @xcite . this corpus is available upon request from the authors. the newsela corpus @xcite , which is also available upon request, contains a portion in spanish, but there is no information about the particulars of the spanish subset. there are three resources for spanish focusing on lexical simplification: the lexsis corpus @xcite (obtained upon request), and the easier corpus @xcite (available at github, but without explicit license), and alexsis @xcite ), which will be available after what the authors call embargo period (but so far there is no explicit license). finally, a bilingual (en/es) dataset about covid-19 texts (simple tico-19) has been released @xcite and a corpus for spanish medical text simplification, the clara-med comparable corpus, is made up of 24 298 pairs of professional and simplified texts @xcite . ## building irekialf_es irekia is the open-government communication channel of the basque government. this web site contains, among others, news about the government, written in a non-administrative language, both in spanish and basque. some of the news are adapted to the easy-to-read format, thereby making the site very valuable as a source to compile complex-simple parallel texts, which, moreover, can be bilingual. the portal has cc-by license, so that its content can be used to derive research datasets. 6 based on this resource, the aim of this work is to create a good-quality corpus of spanish, irekialf_es, with original and adapted/simplified texts. we release the corpus under an open license, thus expanding the options for ats researchers to train and test systems for spanish. irekialf_es is built by crawling all the spanish news that have an easy-to-read counterpart until 17/11/2021 (unfortunately the last adapted version was published in 28/12/2021). the first document's date is 1/04/2017. after removing the duplicates, we have compiled a document-level corpus comprising 288 parallel documents. the dataset is publicly available foot_2 under cc by-sa 4.0 license. in addition to the document-level corpus, we have created a subcorpus, a part of the documentlevel corpus, that is manually aligned at a sentence level, foot_5 and which comprises 705 aligned sentences from 35 documents. 11 we have followed this methodology to align the sentences: we considered this as a substantial agreement and, therefore, as a good basis to align the rest of the corpus. 3. general alignment: one of the annotators has aligned the rest of the documents in the sentence-aligned subcorpus (20 documents). in in ## leuli model of complexity assessment evaluation of simplified corpora and ats systems is more often than not largely based on formal, purely linguistic complexity metrics. however, determining the so-called readability can only be achieved in terms of neurolinguistic complexity: text comprehension takes place in the brain, and as such it requires a cognitive assessment. on the basis of @xcite @xcite @xcite )'s memory, unification, control (muc) model of language neurobiology, gutiérrez-fandiño (2022) 13 proposes a model of three cate-gories of neurolinguistic complexity assessment: lexicon-unification-linearity (leuli). complexity assessment of irekialf_es has therefore been conducted according to the leuli model, which is synthesised in regarding lexical complexity, it has been shown that the less frequent a word is, the more effortful it is retrieve from long-term memory (ltm), resulting in higher levels of neural activation @xcite . infrequent words are not just more difficult to access, they are also more likely to be unknown to the reader, in which case they do not even exist in the mental lexicon and are hence impossible to retrieve from ltm. the unification of information from different language modules of the brain is a costly operation in language processing. accordingly, to lighten complexity of the unification category, linguistic phenomena involving several language modules should be strongly avoided in ats: for instance, coreference (principally measured by pronoun incidence) demands the integration of information from the syntactic and semantic language modules. 14 similarly, the large presence of elements that help avoid the presence of coreference is also an indicator of the lack of unification complexity: the ratio of proper nouns for all nouns and content word overlap, for example, are metrics showing how often referents are repeated, instead of being pronominalised, coreferenced. lastly, linearity affects sentence comprehension as it determines both the temporal separation of chunks provisionally stored in working memory (wm) and the number of chunks stored in such temporary buffer during the processing of a sentence. it is primarily measured by sentence length, but also by non-selected constituents (adjunction and coordination), since selected constituents such as complements are effortlessly retrieved from ltm as part of the syntactic template of any lexical item. en la reunión han estudiado el trabajo de estas empresas y se han propuesto nuevas maneras de colaborar. to base our complexity evaluation on recent evidence on sentence and text processing complexity, and thus we have decided to follow the metrics provided by the leuli model (gutiérrez-fandiño, 2022). for the automatic measurement of complexity metrics, we have employed multiaztertest @xcite , an opensource multilingual text analysis tool which examines more than 130 features at various linguistic levels. to calculate word frequencies we have used python's wordfreq package @xcite . specifically, we have grouped words in eight bins according to the logarithm of their frequencies. 15 next, we present histograms and violin plots @xcite comparing the scores of orig and e2r texts according to the neurolinguistic complexity metrics of the leuli model. regarding the lexicon category, there are substantially more infrequent words (0-4 levels) in orig 15 which corresponds to the zipf_frequency of the wordfreq package. than in e2r texts (figure 2 ), where words around frequency level 6 are more regularly distributed, as a result of the conscious, purposeful use of frequent words. when it comes to the unification category, in figure 3 it can be observed that there is a positive but too slight difference in the incidence of pronouns: e2r texts should have a markedly lower score than orig in this metric. the higher the pronoun incidence, the higher the unification complexity. figure 3 : incidence of pronouns. as previously explained, the ratio of proper nouns for all nouns and content word overlap are metrics that show the extent to which coreference category constrainer constrainee metrics lexicon ltm infrequent words: hard to access or not stored word frequency unification wm the integration of information from different modules multi-module phenomena (mainly coreference in syntacticsemantic unification) linearity wm time and volume of temporary storage sentence length mainly, but also adjunction and coordination in figure 8 , we see that there is a bigger difference in np decendents (adjuncts+complements) than in np modifiers (adjuncts) between orig and e2r. in terms of neurolinguistic simplicity, however, such difference should be bigger in adjuncts, which are non-selected constituents incurring an extra processing cost. in sum, taking into account the leuli framework ( ## experiments in this section we report the results of three neuralbased ats systems when evaluated in the sentencelevel irekialf_es dataset. given its small size, we followed a zero-shot scenario where the neural ats systems are trained using simplext @xcite different domains in spanish. we have decided to train the baseline systems on simplext because the text genre and the spanish variety (peninsular spanish) are similar. we followed @xcite and split the corpus in 574 sentences for training, 143 for development, and 693 for testing. we used the training set for finetuning, and the development set for model selection (the test split of simplext was not used). regarding irekialf_es, we discarded the 1:0 alignments as well as the merge operations (29.3 % of the sentence level corpus), which results in a test set with 498 sentence pairs. the ats systems are the following: edit+synt is an edit-based seq2seq system that adds syntactic information at the word level @xcite . in the preprocessing stage, the training dataset was lowercased and the sentences were tokenised and parsed with spacy, 16 using the large model. the model was trained for 50 epochs, with a batch size of 64, a learning rate of 10 -3 , a hidden dimension of 200, a decay factor of 10 -6 . we used default values for the hyperparameters, and did not perform any hyperparameter tuning. regarding model selection, we selected the checkpoints that obtained the best sari @xcite score in the simplext development test. to evaluate the models, we followed usual practice foot_12 and computed the blue @xcite and sari metrics, using easse @xcite . in ## conclusion and future work in this paper, we have presented irekialf_es, a new open corpus for automatic text simplification in spanish. the corpus compiles a document-level version with 288 parallel original and easy-to-read texts and a sentence-level version, where 35 of the documents have been manually aligned to create a test set of 705 sentences. the aim of this test set is to serve as a benchmark to evaluate ats systems at the sentence level. we have evaluated the neurolinguistic complexity of the corpus by following the lexicon-unification-linearity (leuli) model. the evaluation yields positive results regarding the lexicon category of complexity, mostly negative regarding the unification category and mostly positive regarding the linearity category. therefore, we can conclude that this corpus is suitable for ats training and evaluation regarding lexical simplification and sentence simplification, but it may hinder end users' comprehension when it comes to discourse simplification due to the significant presence of multi-module phenomena (unification category). this important drawback should be considered for future work. a specific quantitative benchmark for establishing the boundaries of the qualitative assessment (positive/halfway/negative) of the leulibased evaluation results should also be addressed in future work. we have also evaluated three different systems that will serve as baselines for future research with this corpus. results show good sari values for all systems, but very low blue scores, which we attribute to the used zero-shot approach. such results suggest the need of simplification datasets in spanish where ats systems can be trained or finetuned on. in the future, we plan to align automatically the rest of the corpus by developing/adapting specific tools. from a linguistic point of view, we also foresee to study the operations carried out to simplify the texts. from an experimental point of view, we would like to carry out crosslingual experiments and test them in this corpus. finally, we plan to create the basque version of the corpus.