A Baseline Readability Model for C ebuano (2022)

Abstract: In this study, we developed the first baseline readability model for the Cebuano language. Cebuano is the second most-used native language in the Philippines with about 27.5 million speakers. As the baseline, we extracted traditional or surface-based features, syllable patterns based from Cebuano’s documented orthography, and neural embeddings from the multilingual BERT model. Results show that the use of the first two handcrafted linguistic features obtained the best performance trained on an optimized Random Forest model with approximately 87% across all metrics. The feature sets and algorithm used also is similar to previous results in readability assessment for the Filipino language—showing potential of crosslingual application. To encourage more work for readability assessment in Philippine languages such as Cebuano, we open-sourced both code and data.

## introduction the proper identification of the difficulty levels of reading materials is a vital aspect of the language learning process. it enables teachers and educators alike to assign appropriate materials to young learners in which they can fully comprehend, preventing boredom and disinterest @xcite . however, assessing readability presents challenges, particularly when you have a large corpus of text to sift through. manually extracting and calculating a wide range of linguistic features can be time-consuming and expensive and can lead to subjectivity of labels due to human errors @xcite . to tackle this problem, more and more research in the field have focused on experimenting with automated methods for extracting possible linguistic predictors to train models for readability assessment. while automating readability assessment is a challenge itself, one of the original problem in the field starts with data. in the philippines, the mother-tongue based multilingual education (mtb-mle) scheme was introduced by the department of education (deped) in 2013. with this initiative, there were little to no available tool for automatically assessing readability of reading resources, instructional materials, and grammatical materials in mother tongue languages aside from filipino such as cebuano, hiligaynon, and bikol @xcite ). to answer this challenge, in this paper, we investigate various linguistic features ranging from traditional or surface-based predictors, orthography-based features from syllable patterns, and neural representations to develop a baseline readability assessment model for the cebuano language. we use an array of traditional machine learning algorithms to train the assessment models with hyperparameter optimization. our results show that using non-neural features are enough to produce a competitive model for identifying the readability levels of children's books in cebuano. ## previous work readability assessment has been the subject of research of linguistic experts and book publishers as a method of measuring comprehensibility of a given text or document. villamin and de guzman (1979) pioneered a readability assessment for the filipino language in 1979. hand-crafted indices and surface information from texts, such as hand counts of words, phrases, and sentences, are used in these formula-based techniques. an equivalent technique of traditional formula was applied on to waray language @xcite to complement the deped's mtb-mle program in certain regions of the philippines such as in samar and leyte. while traditional featured formulas relied on linear models, recent studies on readability research assessment have shifted their focus on ex-panding the traditional method to more fine-grained features. @xcite and macahilig (2014) introduced the use of a logistic regression model trained with unique word counts, total word and sentence counts, and mean log of word frequency. a few years later, lexical, syllable patterns, morphology, and syntactic features were eventually explored for readability of filipino text by works of @xcite @xcite . ## the cebuano language cebuano (ceb) is an austronesian language mostly spoken in the southern parts of the philippines such as in major regions of visayas and mindanao. it is the language with the second highest speaker count foot_1 in the country with 27.5 million, just after tagalog, where the national language is derived from, with 82 million speakers. both cebuano and tagalog languages observe linguistic similarities such as in derivation, prefixing, disyllabic roots, and reduplication @xcite . on the other hand, differences are seen in syntax such as use of particles (ay, y), phonetic changes, and morphological changes on verbs. figure 1 illustrates a portion of the philippine language family tree emphasizing on where cebuano originated. cebuano is part of the central philippine subtree along with tagalog and bikol which can be attributed to their similarities and differences as mentioned. the full image can be viewed at @xcite . ## linguistic features in this study, we extracted three linguistic feature groups from our cebuano text corpus: traditional or surface-based features, orthography-based features, and neural embeddings. to the best of our knowledge, no study has ever been conducted to assess and explore the readability assessment of cebuano text using these features. ## experiment setup the task at hand is a multiclass classification problem with three classes being the aforementioned grade levels. we specifically chose traditional learning algorithms such as logistic regression, support vector machines, and random forest for building the baseline models for post-training interpretation techniques described in the succeeding sections. to reduce bias, a k-fold cross validation where @xmath0. for the intrinsic evaluation, we used standard metrics such as accuracy, precision, recall and macro f1-score. in addition, we also used grid search to optimize the following model-specific hyperparamters: solver and regularization penalties for logistic regression, kernel type, maximum iterations, and regularization penalties for support vector machines, and number of estimators, maximum features, and maximum depth for random forest. ## results to assess the effectiveness of the proposed framework in the experimentation, we examined model performances on three different ablation studies: (a) linguistic features only, (b) neural embeddings only, and (c) combination of the two via concatenation. the results of each fine-tuned model utilizing the given evaluation metric are showed in tables 2 , 3 , and 4. across the board, the best performing model and feature combination for cebuano achieved approximately 87.3% for all metrics using the combination of trad and syll features with random forest. this top performing model makes used of 100 tree estimators, automatically adjusted maximum features, and a max depth of 20. interestingly, the feature combination and the algorithm of choice is also the same for filipino readability assessment as seen in the work ofimperial and ong (2021b). this may suggest that, despite language differences and similarities, the use of surface-based features such as counts and syllable patterns are accepted for both filipino and cebuano languages in the readability assessment task. referring again to figure 1 for emphasis, both languages are part of the central philippine subtree which opens the possibility of a cross-lingual application of linguistic features for future research. this effectiveness of surface-based features is also seen for the optimized logistic regression model where using trad features obtained the best performance. in the case of the optimized support vector machine model, the use of neural embeddings alone obtained better scores than the combination of traditional and syllable pattern features. this result affirms the observation in imperial (2021) where the extracted neural embeddings can serve as substitute features and can relatively be at par with handcrafted features. ## outlook we developed the first ever baseline machine learning model for readability assessment in cebuano. among the three linguistic feature groups extracted to build the model, the combination of traditional or surface-based features (trad) with syllable pattern based features (syll) produced the highest performance using an optimized random forest model. one of the main challenges in the field is the limited amount of resource for tools and data especially for low-resource languages @xcite ). to answer this call and encourage growth of research in this direction, we open-sourced the compiled dataset of annotated cebuano reading materials and the code for model development.