Revisiting non- E nglish Text Simplification: A Unified Multilingual Benchmark (2023)

Abstract: Recent advancements in high-quality, large-scale English resources have pushed the frontier of English Automatic Text Simplification (ATS) research. However, less work has been done on multilingual text simplification due to the lack of a diverse evaluation benchmark that covers complex-simple sentence pairs in many languages. This paper introduces the MultiSim benchmark, a collection of 27 resources in 12 distinct languages containing over 1.7 million complex-simple sentence pairs. This benchmark will encourage research in developing more effective multilingual text simplification models and evaluation metrics. Our experiments using MultiSim with pre-trained multilingual language models reveal exciting performance improvements from multilingual training in non-English settings. We observe strong performance from Russian in zero-shot cross-lingual transfer to low-resource languages. We further show that few-shot prompting with BLOOM-176b achieves comparable quality to reference simplifications outperforming fine-tuned models in most languages. We validate these findings through human evaluation.

## introduction automatic text simplification (ats) is the task of reducing the complexity of a text without changing its original content and meaning @xcite . ats has many applications, from making a text easier to read for people with reading and cognitive disabilities @xcite and second language learners @xcite to reducing the complexity of medical texts for easier understanding by the general public @xcite . for better accessibility to diverse communities, this technology should be available without language barriers. much of the recent success in english text simplification comes from large parallel corpora of texts 1 code and data available at https://github.com/ xenonmolecule/multisim 2006 2008 2010 2012 2014 2016 2018 with the same content written using both complicated and simple sentences @xcite @xcite . these resources enable the training of large language models for ats in english @xcite @xcite . ats research in other languages has received much less attention @xcite . figure 1 shows that the growth of english text simplification research outpaces progress in other languages. a diverse multilingual benchmark is essential for a more comprehensive evaluation of multilingual simplification methods, pre-trained models, and evaluation metrics. the lack of a multilingual benchmark that covers a set of high, medium, and low-resource languages belonging to different scripts and language families hinders advancement in multilingual ats. in this paper, we address this gap in the field by introducing the mul-tisim benchmark that covers 27 text simplification datasets (complex-simple pairs) in 12 different languages. multisim consists of a collection of datasets from the literature that we unify into a single format for easier accessibility to the research community. in summary, our main contributions are as follows: ## parallel simplification corpora so far, 31 parallel simplification corpora exist in non-english languages. we organize the discussion of these corpora by their creation strategy. figure 2 shows the amount of data in each language divided by collection strategy. corpus source(s) simplification author collection strategy alignment level sentence aligned complex sentences simple sentences access arabic corpora saaq al-bambuu (khallaf and sharoff, 2022) writer ⋆ sentence auto 2,980 2,980 private basque corpora cbst (gonzalez-dios et al., 2018) translator, teacher document manual 458 591 on request brazilian portuguese corpora porsimples (aluísio and gasperin, 2010) linguist document manual 7,902 10,174 on request danish corpora dsim (klerke and søgaard, 2012) journalists ⋆ sentence auto 47,887 60,528 on request english corpora † asset (alva-manchego et al., 2020) crowdsource sentence manual 2,359 23,590 open source newsela en (xu et al., 2015) experts ⋆ document auto 393,798 402,222 on request wiki-auto (jiang et al., 2020) crowdsource document auto 10,144,476 1,241,671 open source french corpora alector (gala et al., 2020) experts document na 1,230 1,192 open source clear (grabar and cardon, 2018) crowdsource, experts sentence auto 4,596 4,596 open source wikilarge fr (cardon and grabar, 2020) crowdsource sentence auto 307,067 308,409 open source german corpora geolinotest (mallinson et al., 2020) linguist sentence manual 1,198 1,198 open source german news (säuberli et al., 2020) news agency ⋆ document auto 15,239 14,344 on request klexikon (aumiller and gertz, 2022) crowdsource document na 771,059 96,870 open source simple patho (trienes et al., 2023) medical students paragraph manual 22,191 26,551 private simple german (battisti et al., 2020) government ⋆ document auto 12,806 8,400 on request* textcomplexityde (naderi et al., 2019) native speaker document manual 250 250 open source italian corpora adminit (miliani et al., 2022) researchers sentence manual 777 763 open source simpitiki wiki (tonelli et al., 2016) crowdsource sentence manual 575 575 open source paccss-it (brunato et al., 2016) crowdsource sentence auto 63,006 63,006 open source teacher (brunato et al., 2015) teachers document manual 204 195 open source terence (brunato et al., 2015) experts document manual 1,035 1,060 open source japanese corpora easyjapanese (maruyama and yamamoto, 2018) students sentence manual 50,000 50,000 open source easyjapaneseextended (katsuta and yamamoto, 2018) crowdsource sentence manual 34,400 35,000 open source japanese news (goto et al., 2015) journalists, teachers ⋆ document auto 13,356 13,356 private russian corpora ruadapt encyclopedia (dmitrieva et al., 2021) researchers document auto 9,729 10,230 open source ruadapt fairytale (dmitrieva et al., 2021) researchers document auto 310 404 open source ruadapt lit (dmitrieva and tiedemann, 2021) writers document auto 24,152 28,259 on request rsse (sakhovskiy et al., 2021) crowdsource sentence manual 2,000 6,804 open source ruwikilarge (sakhovskiy et al., 2021) crowdsource sentence auto 278,499 289,788 on request slovene corpora slots (gorenc and robnik-šikonja, 2022) experts ⋆ sentence manual 1,181 1,287 open source spanish corpora first (orasan et al., 2013) experts document manual 320 332 private newsela es (xu et al., 2015) experts ⋆ document auto 46,256 45,519 on request simplext (saggion et al., 2015) researchers document manual 1,108 1,742 on request urdu corpora simplifyureval (qasmi et al., 2020) expert sentence manual 500 736 open source ## the multisim benchmark we release the multisim benchmark, a collection of 27 parallel simplification corpora in 12 languages and 4 scripts. 18 of these corpora are opensourced and are available online. for 9 corpora, permission must be obtained from the original authors. we provide data loaders for these resources. ## human evaluation for manual evaluation, we enlisted eight volunteers to annotate system outputs in english, russian, italian, and urdu (two in each language) for three properties: adequacy (is the meaning preserved?), fluency (is the simplification eloquent/grammatical?), and simplicity (is the output simpler?). this follows the standard annotation methodology in text simplification research @xcite . we asked annotators to rate 20 sentences for each model using a likert scale from 1-5. we report the manual evaluation key in ## conclusion we release multisim, the first multilingual text simplification benchmark, a collection of 27 sentence-aligned parallel corpora in 12 diverse languages. we collected these resources by surveying the literature for all existing text simplification resources in non-english languages, which were created via distinct methodologies that we categorize into four main approaches. using multisim, we perform fine-tuning, few-shot, and zero-shot crosslingual transfer experiments with generative multilingual language models (mt5, bloom), which revealed new insights in multilingual text simplification. our results demonstrate the value of domain and script match for zero-shot cross-lingual transfer. we show that russian is a good candidate pivot language, outperforming transfer from english in two of our case studies on low-resource and out-ofscript languages. further, we show that few-shot prompting bloom with examples obtained via semantic similarity outperforms fine-tuned models for low-resource languages. by releasing this benchmark, we hope to encourage and enable the development and evaluation of multilingual models and evaluation metrics for text simplification.