Automatic Readability Assessment for Closely Related Languages (2023)

Abstract: In recent years, the main focus of research on automatic readability assessment (ARA) has shifted towards using expensive deep learning-based methods with the primary goal of increasing models’ accuracy. This, however, is rarely applicable for low-resource languages where traditional handcrafted features are still widely used due to the lack of existing NLP tools to extract deeper linguistic representations. In this work, we take a step back from the technical component and focus on how linguistic aspects such as mutual intelligibility or degree of language relatedness can improve ARA in a low-resource setting. We collect short stories written in three languages in the Philippines—Tagalog, Bikol, and Cebuano—to train readability assessment models and explore the interaction of data and features in various cross-lingual setups. Our results show that the inclusion of CrossNGO, a novel specialized feature exploiting n-gram overlap applied to languages with high mutual intelligibility, significantly improves the performance of ARA models compared to the use of off-the-shelf large multilingual language models alone. Consequently, when both linguistic representations are combined, we achieve state-of-the-art results for Tagalog and Cebuano, and baseline scores for ARA in Bikol.

## introduction automatic readability assessment (ara) is the task that aims to approximate the difficulty level of a piece of literary material using computer-aided tools. the need for such application arises from challenges related to the misalignment of difficulty labels when humans with various domain expertise provide annotations, as well as to the difficulty of manual extraction of complex text-based features @xcite . at the same time, readability assessment tools often use different definitions of complexity levels based on (a) age level @xcite , (b) grade level @xcite , or on established frameworks such as (c) the common european framework of reference for languages (cefr) 1 @xcite @xcite @xcite . in recent years, deep learning methods and large language models (llms) have gained popularity in the research community. often studies using these methodologies focus primarily on improving the performance across various metrics. this is particularly manifest in ara research in languages with a high number of accessible and publicly-available readability corpora such as english @xcite @xcite and german @xcite @xcite to name a few. at the same time, existing studies focusing on low-resource languages such as cebuano @xcite and bengala @xcite are still at the stage of primarily using traditional features such as word and sentence lengths to train predictive models. we identify two problems that are related to the use of complex neural-based approaches: the success of such models depends on (a) whether there is enough available data to train a model using a customized deep neural network, and (b) in the case of llms, whether there exists an available off-the-shelf pre-trained model for a low-resource language of interest. @xcite have recently shown that merely integrating extracted embeddings from a multilingual bert model as features for cebuano, a low-resource philippine language, does not outperform models trained with orthographic features such as syllable patterns customized for the language. these challenges provide motivation for researchers to further explore methods that do not rely on the availability of large amounts of data or complex pre-trained models and investigate simpler, more interpretable models instead of black box architectures. in this paper, we take a step back and focus on the data available for low-resource philippine languages and the features extracted from them rather than on the algorithmic aspects. specifically, we explore a scenario where small readability corpora are available for languages that are closely related or belong to one major language family tree. to the best of our knowledge, incorporating the degree of language closeness or relatedness has not been explored before in any cross-lingual ara setup. in this study, we make the following contributions: ## readability corpora in philippine languages we have compiled open source readability datasets for tagalog, cebuano, and bikol from online library websites and repositories. each data instance in this study is a fictional short story. tagalog and cebuano. datasets in these languages have already been used in previous research, including imperial et al. (2019); imperial and ong (2020); imperial (2021); imperial and ong (2021a); imperial et al. (2022). we use the same datasets as in previous research and incorporate them into this study for comparison. for tagalog, we have assembled 265 instances of children's fictional stories from adarna house foot_2 and the department of education (deped) foot_3 . for cebuano, we use the dataset collected by imperial et al. (2022) from let's read asia foot_4 and bloom library foot_5 , which were funded by the summer institute of linguistics (sil international) and booklabs to make literary materials in multiple languages available to the public. ## results and discussion ## conclusion in this work, we took a step back from the trend of exploring various technical components of the complex, deep learning models and, instead, focused on studying the potential effectiveness of linguistic characteristics such as mutual intelligibility for ara in closely related philippine languages -tagalog, bikol, and cebuano. we implemented three cross-lingual setups to closely study the effects of interaction between the three languages and proposed a new feature utilizing n-gram overlap, crossngo, which is specially developed for cross-lingual ara using closely related languages. our results show that: (a) using crossngo combined with handcrafted features achieves significantly higher performance than using mbert embeddings, (b) the inclusion of another closely related philippine language reduces model confusion, and (c) using the conventional fine-tuning for llms like mbert in this setup still does not outperform models with traditional features. consequently, we come to the conclusion that using languages with high intelligibility is more suited for cross-lingual ara. this is demonstrated in experiments with english added as an example of a non-related language, in which we do not achieve a substantial increase in performances for tagalog, cebuano, and bikol. our results agree with the findings of previous studies in cross-lingual ara such as those of madrazo azpiazu and pera (2020) using english, spanish, basque, italian, french, catalan, and @xcite using english and german, that also showed that the inclusion of additional language data can improve ara results on other languages. however, our work is primarily motivated by the degree of language relatedness: we show that better results can be achieved for ara in low-resource languages if we use closely related languages rather than any language, including nonrelated ones like english. our study also provides an encouragement for researchers to consider approaches grounded in linguistic theories which can potentially be used to improve the performance in nlp tasks rather than always resorting to models that are expensive to train and hard to interpret.