A Benchmark for Neural Readability Assessment of Texts in S panish (2022)

Abstract: We release a new benchmark for Automated Readability Assessment (ARA) of texts in Spanish. We combined existing corpora with suitable texts collected from the Web, thus creating the largest available dataset for ARA of Spanish texts. All data was pre-processed and categorised to allow experimenting with ARA models that make predictions at two (simple and complex) or three (basic, intermediate, and advanced) readability levels, and at two text granularities (paragraphs and sentences). An analysis based on readability indices shows that our proposed datasets groupings are suitable for their designated readability level. We use our benchmark to train neural ARA models based on BERT in zero-shot, few-shot, and cross-lingual settings. Results show that either a monolingual or multilingual pre-trained model can achieve good results when fine-tuned in language-specific data. In addition, all mod- els decrease their performance when predicting three classes instead of two, showing opportunities for the development of better ARA models for Spanish with existing resources.

## introduction the readability of a text refers to the aggregation of all its elements that affect the reader's understanding, reading speed, and interest in the content @xcite . some of these elements are the words in the text, the grammatical structure of its sentences, and its writing style @xcite . for example, a newspaper article may be more readable than a scientific paper or a novel. automated readability assessment (ara) aims to exploit these textual elements to predict how "difficult" or comprehensible a text is @xcite . for texts in english, several techniques have been developed, ranging from formulae that relies on surface characteristics such as average word and sentence lengths @xcite to machine learning approaches based on feature engineering @xcite @xcite and, more recently, neural networks and deep learning models @xcite @xcite . 1 similar to english, some work on ara for texts in spanish has developed methods that rely on surface features @xcite . others have implemented tools that extract readability indices (e.g. lexical diversity, word information, syntactic complexity) and used them as features to train standard machine learning classifiers to estimate a text's readability @xcite @xcite . however, these studies were performed on small corpora of at most 300 texts (for both training and testing), limiting its generalisability. in addition, it is unknown to what extend modern neural models are able perform the task for texts in spanish. to mitigate the aforementioned issues, we introduce a new benchmark for training and evaluating models for ara of texts in spanish. our benchmark includes the following contributions: 2 we expect to contribute to the development of ara models that can help tailor relevant content for wider populations, and even benefit downstream nlp tasks, such as text simplification. ## related work earlier readability studies focused on readers' background since audiences may have specific needs, and hence individual difficulties when reading a text. these audiences included people with dyslexia who struggle with long and uncommon words @xcite ; or second-language learners, who are more affected by grammatical aspects than the content itself @xcite . other studies focused on methods for readability assessment that relied on surface features, such as character and sentence counts @xcite . in recent years, researchers have explored alternative methods based on user-oriented studies where scroll interactions are captured to determine a document's easiness to read @xcite . ara has also been used in the evaluation of downstream nlp tasks, such as text simplification @xcite and word complexity analysis @xcite . readability assessment itself has been approached in multiple ways, including through supervised and unsupervised methodologies @xcite . the simplest approach is to use traditional metrics such as gunning fog index (gfi, @xcite , flesch reading ease (fre) and flesch-kincaid grade level @xcite , which evaluated the readability of a document based on its characters, words, syllables, and sentences. while most ara work is done for texts in english, there is research for other languages such as portuguese @xcite @xcite , german @xcite , french @xcite , italian @xcite , russian (reynolds, 2016), vietnamise @xcite and swedish @xcite . however, these studies tend to be language and/or domain specific and thus, sparse without benchmarking multiple models. recent readability studies in spanish are focused on specific audiences. these applications include the evaluation of the readability of e-government websites @xcite , the evaluation of the suitability of hearing aid user guides in spanish @xcite or in more specialised domain such as medical @xcite . these studies mostly use traditional metrics (e.g., number of syllables, words, sentences), rather than neural approaches. finally, there are limited resources for readability in spanish and most of them are shared within the domain of text simplification @xcite @xcite and text complexity analysis @xcite . we contribute with the collection of the proposed readability datasets (section 3) and a benchmark of neural models (section 4) to this growing field of research in spanish. ## dataset collection we describe the data sources and characteristics of each dataset (section 3.1) in our benchmark, as well as the standardisation process applied to each that allows for ara experimentation (section 3.2). we also analyse the readability of the documents and text groupings in our benchmark using readability metrics (section 3.3), and comment on the datasets limitations (section 3.4). ## neural ara models considering the characteristics of our dataset, we treated ara as a classification task, and used the datasets in our benchmark to implement neural supervised models. following previous work @xcite , we used bert @xcite models with fully connected layers and softmax outputs. as base bert models, we experimented with: we used the bertin foot_8 and mbert foot_9 checkpoints available in huggingface @xcite to implement six models in the following settings: zero-shot. models based on bertin and mbert are not trained in any task-specific data and are used directly to make predictions in the test set. this aims to explore if pre-trained models (monolingual or multilingual) are by default capable of performing ara without any fine-tuning. we refer to these models as bertin (zero) and mbert (zero). ## experimental setting baseline. we implemented a simple approach based on logistic regression and tf-idf. 12 we extracted the features for each text using tf-idf algorithm. 13 . then, we trained a linear regression classifier 14 using these features in splits (train/dev). ## results most bert-based models are consistently better than the tf-idf baseline, for all text granularities and readability labels. an exception are mbert (zero) and bertin (zero) who had the lowest performance in all cases. this implies that pretrained models by themselves are unable to perform ara for spanish texts in our benchmark. all models dropped their performance when trained in the 3-class setting compared to the 2-class one, including in the zero-shot models. as shown in figure 1a , figure 1b and figure 1c , an "intermediate" class makes it more difficult to classify the samples, especially in the boundaries ("basic"/"intermediate" and "intermediate"/"advanced"). we can observe that it is easier to distinguish between "basic" and "advanced", evidenced by just having a few misclassified samples (see the matrices corner values). in contrast, there is a significant number of incorrect samples between the "intermediate"-"advanced" boundaries in the 3 models. the cross-lingual mbert (en) models performed comparably to or worse than the baseline in the 3-class group for text granularities. adding language-specific data, makes the model perform better, making mbert (en+es) comparable to the mbert (es) model in all cases. while bertin (es) is still the best in the 3-class paragraph setting, these results suggest that a multilingual pre-trained model can be leveraged for ara in spanish is no language-specific model is available. overall, we can observe that fine-tuning either bertin or mbert with language-specific data would result in a good performing model for the task, for all the settings we considered. as such, these serve as strong baselines for future research. ## discussion our zero-shot (cross-lingual) experiments demonstrate that the readability task is not trivial to learn to perform ara, and that it is directly transferable between languages in the settings we studied. for bertin (zero) and mbert (zero), the models were not previously trained for the readability classification task resulting in poor performance. the decrease in performance between models in the paragraph-based and sentence-based datasets could be attributed to multiple reasons. first, not all texts in the datasets could be mapped to three classes, resulting in fewer instances for training and evaluation. in addition, it may be easier for a model to distinguish between extremes ("simple" or "complex") than to also consider an "intermediate" class. this effect is clearer in the analysis of the confusion matrices in figure 1 . regarding our best models, we benefited from the fact that the bertin model was trained on spanish texts, which contributes for a better "understanding" of readability in this specific language. the multilingual model (mbert) was trained in multiple languages beside spanish, which could have contributed to the improvement of its results. while our results may be encouraging, we state the limitations of our experiments. when short and simple texts are used for training, readability results can easily be related to short sentences and words. however, texts can also be readable in other scenarios, such as using active voice, instead of passive voice, being consistent in the narrative (e.g. following on the same topic) and the use of simpler words, which are not necessarily shorter. these features are harder to learn, as shown in our 3-class experiments, but with the use of more corpora from multiple domains, it may be possible to obtain more robust ara models. regarding the models, we could consider that bertin model is not uncased, whereas the multilingual model is; this could also be a limitation and a variability factor in the models performance. overall, current datasets are scarce, and it is advisable to train in wider corpora for the generalisation in multiple domains. ## conclusion and future work in this paper, we have introduced a new benchmark for ara of texts in spanish. we combined existing datasets for research in ara and text simplification, with other resources scraped from the web. with these data, we trained neural ara models based on bert to classify texts into "simple" and "complex" (2-class), or "basic", "intermediate" and "advanced" (3-class), at two levels of text granularities (paragraph and sentence). the neural models proved to be better than simple baselines. in the future, we plan to include more datasets to the benchmark, such the one used in @xcite . in addition, we plan on training feature-based models for a more comprehensive evaluation of our neural models. finally, it would be interesting to study the effect that larger multilingual pre-trained models, like xlm-r @xcite , could have on the performance of neural models. all of our models are publicly available, as well as demo that showcases their performances. we expect that research communities in spanish speaking countries will benefit from this effort towards the further development of the field.