LXPER Index 2.0: Improving Text Readability Assessment Model for L 2 E nglish Students in K orea (2020)

Abstract: Developing a text readability assessment model specifically for texts in a foreign English Language Training (ELT) curriculum has never had much attention in the field of Natural Language Processing. Hence, most developed models show extremely low accuracy for L2 English texts, up to the point where not many even serve as a fair comparison. In this paper, we investigate a text readability assessment model for L2 English learners in Korea. In accordance, we improve and expand the Text Corpus of the Korean ELT curriculum (CoKEC-text). Each text is labeled with its target grade level. We train our model with CoKEC-text and significantly improve the accuracy of readability assessment for texts in the Korean ELT curriculum.

## related work research efforts in developing automated text readability assessment models for l2 students only emerged in the 2000s @xcite . @xcite showed that grammatical features and lexical features play particularly important roles in l2 text readability prediction. meanwhile, vajjala and meurers (2014) showed that the additional use of lexical features could significantly improve l2 readability assessment. @xcite also reported the importance of lexical features in general (for l1 speakers of english) text readability assessment. however, the common limitation of the previous research in l2 readability assessment was the training corpus annotated with the grade levels for l1 readers of english. our results, which we obtain from training our model using cokec-text, introduces the possibility that lexical features are not as important as the previous researchers reported. in addition, we also show that a considerably accurate text readability model can built even with a small data set if the model is optimized and the corpus is well-labeled. ## corpus since our goal is to improve the accuracy in text readability assessment of l2 texts, our ideal corpus has to fully consist of l2 texts from a non-native elt curriculum. the base corpus that we use is cokec-text @xcite , which is a collection of 2760 unique grade-labeled texts that are officially administered by the korean ministry of education (moe). similar texts are also used in the national assessment of educational achievement, college scholastic ability test, and moeapproved middle school textbooks in korea. however, as shown in in addition, we found some k7 ∼ k9 texts that are only partially english. they contained ascii korean characters (often explanations of difficult english words by the author). these count as a token (or possibly, even a sentence from case to case) in the nltk parsing process but provide no meaningful linguistic properties. this can produce miscalculations of the "average of x" (e.g., the average number of words per sentence) features that we discuss in section 4. we manually went through every text to make sure that clean data is used for model training. our final training corpus consists of 3700 original l2 texts from k7 ∼ k12.5. k12.5 grade texts are from csat, which is a college entrance exam for korean universities. in general, the korean grades k7 to k12 are for middle and high school students of ages 13 to 19. ## selecting features we now describe the 35 linguistic features we studied. ## conclusion in this research, we introduced lxper index 2.0, a readability assessment tool that incorporates traditional, pos, entity density, lexical chain, and word difficulty features. then, we trained the model on our own expanded version of cokec-text. we obtained a continuously increasing output for l2 texts from k7 to k12. in addition, we achieved our initial target average accuracy error of less than 0.5 grade levels, which is more accurate than any l2 text readability prediction model we are aware of. the improvements we report in this paper are largely due to two changes: 1. the cokec expansion and 2. the use of a logistic regression model. the contribution from the corpus is quite obvious in that our model could now learn more about the lower grades (k7 ∼ k9). however, the contribution from the change of the regression model is something that we should put more thought into. but it seems evident that the "grades" classification task is better modeled with a logistic regression model. a possible explanation could be that the difficulty of a text does not linearly correlate with the target grades. even though we wanted to test our model on other east asia l2 elt curricula, like japan and china, we could not implement due to the lack of openly-available corpus in the countries. the novelty of the lxper index model is that it focuses on in-curriculum text readability analysis, possibly even with a small data set of less than 4000 texts. thus, applying the model will fail to give meaningful outcomes without a pre-processed and labeled corpus like cokec. thus, the application of a similar model on those countries would first require foundation research on constructing corpora.