Automatic Readability Assessment of G erman Sentences with Transformer Ensembles (2022)

Abstract: Reliable methods for automatic readability assessment have the potential to impact a variety of fields, ranging from machine translation to self-informed learning. Recently, large language models for the German language (such as GBERT and GPT-2-Wechsel) have become available, allowing to develop Deep Learning based approaches that promise to further improve automatic readability assessment. In this contribution, we studied the ability of ensembles of fine-tuned GBERT and GPT-2-Wechsel models to reliably predict the readability of German sentences. We combined these models with linguistic features and investigated the dependence of prediction performance on ensemble size and composition. Mixed ensembles of GBERT and GPT-2-Wechsel performed better than ensembles of the same size consisting of only GBERT or GPT-2-Wechsel models. Our models were evaluated in the GermEval 2022 Shared Task on Text Complexity Assessment on data of German sentences. On out-of-sample data, our best ensemble achieved a root mean squared error of 0:435.

## introduction automatic readability assessment (ara) is a well-known challenge in natural language processing (nlp) research @xcite @xcite . systems for reliable readability assessment have the potential to support readers with learning disabilities, inform self-directed learning, or help control the reading level of automatically generated text translations @xcite . the development of methods for text readability assessment may be described in three phases. (i) traditional text readability formulas were based on statistical measures of lexical and syntactic features (such as word difficulty and length). techniques from nlp further improved upon traditional formulas by incorporating high-level textual features such as semantic and discursive text characteristics @xcite . (ii) in the early 21st century, engineered linguistic features were used to train shallow classifiers and regressors from machine learning (such as support vector machines and decision trees) which further improved prediction accuracy @xcite . (iii) the latest phase has been characterized by the advent of large language models (llms) developed in the deep learning community. such neural networks learn features (vector representations of text) automatically from large text corpora during self-supervised pretraining. successful network architectures such as bert @xcite or gpt @xcite @xcite closely follow the influential transformer model @xcite that allows for efficient modeling of long-range correlations in texts. by combining representations derived from bert with linguistic features, recent studies observed increased accuracy in assessing the readability of english texts @xcite . training large language models requires large text corpora, a prerequisite that is difficult to meet in languages with fewer resources (compared to english) such as german. thus, most approaches to assess the readability of german texts have been based on linguistic features and traditional models from statistical learning such as polynomial regression, support vector machines, or random forests @xcite @xcite . only recently, large language models have become available for german, most notably gbert @xcite , which is based on bert, and gpt-2-wechsel @xcite which was derived from the english gpt-2 model @xcite . it is largely unknown to which extent these german language models can improve the automatic readability assessment of german texts. in this contribution, we investigate the ability of ensembles of gbert and gpt-2-wechsel models to assess the readability of german sentences. we combine these models with traditional linguistic features and evaluate our approach on a recently published dataset of german sentences @xcite . inspired by previous work on ensembling large language models @xcite , we studied the dependence of model accuracy on the number of ensemble members and ensemble composition. finally, we describe the models that were evaluated in the germeval 2022 shared task on text complexity assessment @xcite . the implementation details of our experiments (team "acomplexity") are available online 1 . ## data and tasks the dataset consisted of 1000 labeled sentences @xcite and was provided by the organizers of the germeval 2022 shared task on text complexity assessment @xcite . the sentences were drawn from 23 wikipedia articles. 250 of these sentences were manually simplified by native german speakers @xcite . the scores (labels) were obtained via an online survey system. participants were asked to rate the complexity, understandability, and lexical difficulty of the sentences on a 7-point likert scale. on this scale, 1 denotes the lowest and 7 the highest possible value @xcite . in total, 10650 valid sentence ratings were collected, distributed among the 1000 sentences. following a data screening procedure, 5 to 18 ratings per sentence were deemed valid and then used to calculate the arithmetic mean, called the mean opinion score (mos), of each metric @xcite . the shared task was to predict the mos of the text complexity of german sentences. since the mos was defined as a decimal value (see figure 1 https://github.com/dslaborg/tcc2022 bei der tour de france liegt die höchste durchschnittsgeschwindigkeit eines fahrers bei 41 km/h. (mos: 1.5) für die union resultiert daraus sowohl ein akzeptanzproblem bei den eu-bürgern, denen "brüssel" immer undurchsichtiger erscheint, als auch die mit dem mitgliederwachstum verbundene schwierigkeit, im bestehenden institutionengefüge die arbeitsund handlungsfähigkeit der einzelnen organe zu gewährleisten. (mos: 6.33) figure 2: samples (german sentences) from the dataset of the germeval 2022 shared task on text complexity assessment. numbers in parentheses denote text complexity scores. ## conclusion we studied the ability of ensembles of fine-tuned german language models to reliably predict the readability of german sentences. all proposed models also used traditional linguistic features that slightly increased prediction performance (data not shown), consistent with previous reports on text readability assessment of english texts @xcite . we observed mixed ensembles of gbert and gpt-2-wechsel to better predict readability scores than ensembles of the same size consisting of only gbert or gpt-2-wechsel models. furthermore, prediction accuracy as quantified by the root mean squared error decreased with increasing ensemble size, which resembled findings for hate speech classification reported previously @xcite .