R ead M e++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment (2024)

Abstract: We present a comprehensive evaluation of large language models for multilingual readability assessment. Existing evaluation resources lack domain and language diversity, limiting the ability for cross-domain and cross-lingual analyses. This paper introduces ReadMe++, a multilingual multi-domain dataset with human annotations of 9757 sentences in Arabic, English, French, Hindi, and Russian, collected from 112 different data sources. This benchmark will encourage research on developing robust multilingual readability assessment methods. Using ReadMe++, we benchmark multilingual and monolingual language models in the supervised, unsupervised, and few-shot prompting settings. The domain and language diversity in ReadMe++ enable us to test more effective few-shot prompting, and identify shortcomings in state-of-the-art unsupervised methods. Our experiments also reveal exciting results of superior domain generalization and enhanced cross-lingual transfer capabilities by models trained on ReadMe++. We will make our data publicly available and release a python package tool for multilingual sentence readability prediction using our trained models at: https://github.com/tareknaous/readme

## introduction readability assessment is the task of determining how difficult it is for a specific audience to read and comprehend a piece of text @xcite . developing methods for automatically predicting the readability of a sentence is beneficial for many applications such as controllable text simplification @xcite , ranking search engine results by their level of difficulty @xcite , and selecting appropriate reading material for language learners @xcite . making such technologies robust to textual variations and accessible to a global community with diverse languages requires readability prediction methods that generalize across different text domains and language families. recent advancements in language models (lms) @xcite have enabled the development of neural-based readability assessment methods @xcite . despite the progress made, the absence of a diverse benchmark limits the ability to effectively evaluate how well lm-based methods, whether supervised, unsupervised, or prompting-based, perform across domains and languages. current evaluation resources for sentence readability assessment suffer from a few crucial shortcomings. first, existing datasets are primarily composed of sentences collected from wikipedia @xcite @xcite or news articles @xcite . however, lms have been shown to struggle when handling data from a differ- ## constructing readme++ corpus we present the detailed procedure for constructing the readme++ corpus. to maximize the diversity of domains, we identified 112 data sources that are either with open licenses or shareable for non-commercial purposes (see ## benchmarking experiments as shown in figures 2 and 3 , the readme++ corpus offers a diverse coverage of domains, readability levels, and sentence lengths, making it an ideal testbed for evaluating readability assessment methods. we benchmark supervised, unsupervised, and few-shot approaches using recently developed lms. we use the same random train/valid/test split (detailed statistics in appendix d.2) based on a 60/10/30% ratio per domain for all experiments, except the domain generalization study in §5. 4.1 supervised & prompting methods supervised. we fine-tune lms to classify sentence readability. we compare multilingual models, mbert (devlin et al., 2019) and xlm-r (conneau et al., 2020), to monolingual models that include bert (devlin et al., 2019) for english, arabert (antoun et al., 2020) and arbert (abdul-mageed et al., 2021) for arabic, camem-bert for french (martin et al., 2020), and ru-bert (kuratov and arkhipov, 2019) for russian. for hindi, we use muril (khanuja et al., 2021) and indicbertv2 (kakwani et al., 2020), both pretrained on 12 indian languages. we also consider encoder-decoder lms, mt5 (xue et al., 2021), aya101 (üstün et al., 2024), and arat5 (elmadany @xcite . we fine-tune for 20 epochs using the cross-entropy loss and the adam optimizer and tune the learning rate in the set {1e -5 , 1e -6 , 1e -7 }. we select checkpoints based on the best performance on the validation set. we report the average of 5 runs with different random initialization seeds. prompting. we perform in-context learning using gpt3.5, gpt4 (apr 2024), llama2-7b (touvron et al., 2023), llama3.1-8b (dubey et al., 2024), and aya23-8b @xcite . we provide lms with a definition of readability and the descriptors of the six cefr levels. we show the model five randomly sampled in-context examples from the train set and their corresponding cefr levels, then ask the model to assess the readability of a new sentence based on the cefr scale. prompt details can be found in appendix d.3. ## cross-domain cross-lingual analyses we test the ability of lms trained on readme++ to generalize to unseen domains (5.1) and transfer to other languages (5.2) compared with models trained on previous datasets. ## conclusion we introduced readme++, a multi-domain dataset for multilingual sentence readability assessment. readme++ provides 9757 sentences in arabic, english, french, hindi, and russian that are collected from 112 different data sources and annotated by humans based on the cefr scale. we showed that lms trained using readme++ achieve strong performance across different textual domains and perform well in cross-lingual transfer from english to 6 target languages, outperforming models trained on previous datasets. by releasing readme++, we hope to encourage and enable the development and evaluation of more effective and robust methods for multilingual sentence readability assessment.