Assessing sentence readability for G erman language learners with broad linguistic modeling or readability formulas: When do linguistic insights make a difference? (2022)

Abstract: We present a new state-of-the-art sentence-wise readability assessment model for German L2 readers. We build a linguistically broadly informed machine learning model and compare its performance against four commonly used readability formulas. To understand when the linguistic insights used to inform our model make a difference for readability assessment and when simple readability formulas suffice, we compare their performance based on two common automatic readability assessment tasks: predictive regression and sentence pair ranking. We find that leveraging linguistic insights yields top performances across tasks, but that for the identification of simplified sentences also readability formulas – which are easier to compute and more accessible – can be sufficiently precise. Linguistically informed modeling, however, is the only viable option for high quality outcomes in fine-grained prediction tasks. We then explore the sentence-wise readability profile of leveled texts written for language learners at a beginning, intermediate, and advanced level of German to showcase the valuable insights that sentence-wise readability assessment can have for the adaptation of learning materials and better understand how sentences’ individual readability contributes to larger texts’ overall readability.

## introduction comprehensible input is key to foster language learning @xcite , especially when it challenges learners by falling slightly above their individual level of language competence @xcite . also in content-matter education, input comprehensibility has been linked to learning success (e.g., @xcite . thus, automatic readability assessment (ara) is a crucial tool to support education. ara seeks to align language input with readers' comprehension skills @xcite . it can not only identify suitable reading materials, but can also ensure learner-input alignment in applications such as tutoring systems or educational conversational agents or as a validation tool for publishers of educational materials. yet, most work on ara focuses on english native speakers, leaving much potential for other languages and approaches specifically tailored to the needs of second or foreign language (l2) learners who experience language barriers differently than native speakers @xcite . although most work on ara has focused on estimating the readability of entire documents, there are many application scenarios in which sentencelevel readability assessment is more suitable. beyond the identification of text simplification targets @xcite , they are also more suitable for very short text types including social media language (e.g., tweets and chats), questionnaire or test items used in assessment and empirical education research, or shorter text units in traditional learning materials (e.g., captions or tasks in schoolbooks). furthermore, there has been little research on the link between sentence and document readability (but see @xcite which is immediately relevant for the targeted design and adaptation of educational materials. there is a startling gap between the methods proposed in ara research and those used in practice. while for the last two decades, research on ara has favored machine learning approaches over traditional readability formulas @xcite due to their generally better performance (e.g., @xcite , simple formulas continue to be used extensively in practice due to their ease of use and low computation demands @xcite . this discrepancy raises the practical question when simple approximations of readability through formulas suffice, and when the use of more elaborate systems is necessary. this paper addresses these issues with four major contributions: first, we present a new state-ofthe-art (sota) sentence-level readability model for l2 german readers which is based on broad linguistic complexity assessment. its performance on a 7-point likert scale is comparable to human raters when it comes to estimating the readability of sentences for german l2 readers. second, we make this model accessible online to enhance the impact of our work outside academic discourse. users can extract features from their texts using the publicly available web platform ctap @xcite and use the results as input for a pre-written r script that applies the model to users' input files in the format that is returned by ctap. 1 third, we compare our sota machine learning-based approach with commonly used readability formulas for the two common ara tasks predictive regression and ranking to answer the question when using linguistic insights indeed makes a difference and for which tasks simple readability formulas suffice. finally, we leverage our sota model to explore sentence profiles of leveled l2 articles to provide new insights into the role of sentence readability for document difficulty that can help inform input adaptation strategies for educational materials. the remainder of this paper is structured as follows: after a brief literature review (section 2), we introduce the data (section 3) and linguistic features (section 4) used for our studies. we then report on the model training and evaluation for predictive regression and sentence ranking (section 5). finally, we explore the readability profile of german l2 articles on a document level (section 6) and discuss our overall findings (section 7). we conclude with finals remarks on the impact of our findings and an outlook on future work (section 8). ## related work early approaches to ara date back to the last century when traditional readability formulas (e.g., @xcite were developed, see @xcite for a comprehensive overview. readability formulas estimate text readability solely based on surface level proxies of text characteristics (e.g., sentence and word 1 both, the complexity-based model and the r script can be accessed at https://osf.io/jg6kc/?@xmath0. they have been heavily criticized for their lack of linguistic insight and robustness, and have been shown to yield inferior results to statistical approaches to ara on authentic data @xcite @xcite . yet, they are still the most widely distributed form of ara in practice due to their low computational demands, ease of use, and availability for a variety of languages @xcite . common use cases include work on health literacy @xcite @xcite and as evaluation metrics in computational linguistic work on machine translation @xcite @xcite or conversational agents @xcite @xcite . since the early 2000s (cf. @xcite , statistical approaches became dominant in research on ara. this includes feature-based approaches leveraging rich linguistic information for their predictions as well as neural approaches without prior feature engineering. while either method has been shown to yield sota performances (e.g., @xcite @xcite on the on-estopenglish corpus by @xcite , neural approaches have been argued to be more easily applicable for cross-linguistic readability assessment @xcite , but see @xcite . feature-based approaches, instead, are more appropriate when little data is available or when users need an explanation for the obtained readability score, as is commonly the case in education contexts and for publishers of leveled reading materials who might want to revise their texts after obtaining a readability score @xcite . established features measure aspects of syntax and lexicon @xcite , morphology @xcite @xcite , and discourse features. they intersect with common features from automatic writing quality assessment (crossley, 2020) and second language acquisition research @xcite . only limited progress has been made on ara for german, after early work on readability formulas (e.g., @xcite @xcite . the now unavailable delite system has been used to predict readability for german municipal texts @xcite . @xcite and @xcite focused on the binary distinction of texts for adult versus young native speaking readers. however, binary ara is of limited use in practice. @xcite present to our knowledge the first and only multi-level classification approach for german documents after introducing the first multilevel readability corpus for german, which is part of a larger multi-lingual readability corpus for language learners. for sentence-wise readability assessment, @xcite al. (2019a) little research has investigated the relationship between sentence and document readability, even though there has been some work testing the reliability of readability assessment for very short texts @xcite and sentences @xcite @xcite . @xcite inspect readability differences between sentences from wikipedia and simple wikipedia to investigate the poor performance of documentlevel ara models for the identification of sentences from simple and regular texts. they find that sentences from wikipedia are not systematically more complex than sentences from simple wikipedia. this raises several questions for further inquiry. the lack of observable differences might be caused by an insufficient sensitivity of the document-level model for sentence-level readability differences. also, simple wikipedia has criticized as not systematically simpler than wikipedia (e.g., @xcite @xcite . more research is needed to confirm or refute their finding that harder texts are not simply characterized by containing generally less readable sentences which would have direct implications for work on targeted document adaptation seeking to identify language barriers in educational materials. ## feature extraction and selection we extracted 543 features of linguistic complexity from the linguistic domains of syntax, lexicon, and morphology as well as psycho-linguistic features of text cohesion, language use, and human language processing and surface level text features inspired by traditional readability formulas. all features have a long standing tradition in ara research @xcite or in related work on automatic text scoring (crossley, 2020) and second language acquisition complexity research @xcite . for feature extraction, we used the ctap system (chen and meurers, 2016, http://ctapweb.com ) which has been extended to facilitate the analysis of german by @xcite . we chose this system, because it is to our knowledge the most extensive available analysis system for german. the underlying feature extraction engine for german has proven highly successful and robust in a variety of education-related tasks including readability assessment @xcite @xcite and work linked to writing quality assessment @xcite @xcite . also, using a publicly available web-based system increases the re-usability of any model using these features in practice. ## discussion study 1 investigated the performance of linguistically informed readability models and readability formulas for sentence-wise readability assessment for two common ara tasks: precise predictive regression (section 5.2) and ranking to identify simplified sentences in sentence simplification pairs (section 5.3). the results showcase the versatile performance of linguistically informed readability models: only our complexity-based model achieved top performance for both tasks. for the more difficult and authentic task of precise predictive regression, we showed that our linguistically informed complexity-based model clearly outperforms simplistic formulas and set a new sota performance (@xmath1.685) on the data set. the better performance cannot be exclusively attributed to the statistically stronger method, because on both tasks, the complexity-based model clearly outperformed the length-based model. this shows that broad linguistic modeling adds valuable insights beyond the powerful statistical training method. for ranking, all ara models achieved an accuracy well above 90% and two readability formulas performed at par with our complexity-based model. this shows that even simple ara approaches can successfully distinguish relative differences in readability between content-wise equivalent sentences that are being introduced by text simplification. despite being a rather artificial task, this has some limited applications, e.g., when evaluating machine translation and text simplification systems. in study 2, we used our complexity-based model to inspect the sentence-wise readability profiles of leveled texts for l2 readers. our findings clearly show that while there is a tendency for easier texts to contain more sentence with lower difficulty scores, also medium and advanced texts contain mostly accessible sentences. it is really the presence of difficult sentences within documents that dictates an articles' overall readability. this has clear implications for the design and simplification of educational materials: to efficiently adjust the overall readability level of a text, we need to identify specific sentences that form language barriers rather than simplifying the entire text. ## conclusion we have presented a new sota sentence-wise ara model for german l2 readers which is pub-licly available and accessible for users with minimal background in r. leveraging broad linguistic insights, it predicts readability with a margin of error even below the acceptable disagreement range for humans raters. we showed that to flag simplified sentences also traditional readability formulas suffice, but that broad linguistic modeling is needed to obtain the precise predictive readability estimates that are often required in practice (e.g., to adapting learning and teaching materials). we further explored leveled articles for german l2 readers to illustrate the practical benefits of sentence-level ara and gain insights into text profiles of leveled documents. our findings highlight that the readability of texts is driven by the maximum rather than the overall readability of sentences. this has direct implications for the adaptation of teaching materials, which should focus on identifying specific sentences posing language barriers rather than the simplification of all or any sentence in a text. to our knowledge, this is the first time detailed analysis of sentence profiles of leveled reading materials for german. future work should further explore the implications of this for text simplification, for example using eye-tracking studies. our work lays the foundation for further research on ara for german and opens up numerous opportunities for educational applications, such as ara for captions and task descriptions in school books or the analysis of social media and chat conversations with l2 learners.