HIIG at G erm E val 2022: Best of Both Worlds Ensemble for Automatic Text Complexity Assessment (2022)

Abstract: In this paper we explain HIIG’s contribution to the shared task Text Complexity DE Challenge 2022. Our best-performing model for the task of automatically determining the complexity level of a German-language sentence is a combination of a transformer model and a classic feature-based model, which achieves a mapped root square mean error of 0.446 on the test data.

## introduction text complexity is not only a highly interesting topic from a linguistic perspective; it also has several implications on a societal level. a text that has the appropriate complexity level for a specific reader not only ensures that the reader can fully understand the information presented in the text, but it also keeps the reader engaged and can help the reader to learn new structures and expand their vocabulary. this last point is particularly relevant for language learners and readers who are reading text in a language that is not their native language. the text complexity de challenge focuses on this specific target group as the task involves predicting the complexity of a sentence in german, which have been annotated on a scale of 1 to 7 by german learners whose language proficiency is at b level (on the cefr scale). an overview of the shared task and the results from all the teams can be found in @xcite . in this paper we briefly report on related work in section 2, before describing the dataset used in the shared task in section 3. in section 4 we outline our various approaches to the task, before reporting on the results and briefly discussing them in section 5. in section 6 we conclude the paper. ## related work previous work aimed at automatically assessing the text complexity level of sentences has focused * * equal contribution mostly on the english language. @xcite use the newsela corpus (english-language newspaper articles, simplified at multiple levels for different aged school children) and calculate scores for unigrams, bigrams and trigrams by looking at what levels of the corpus they occur in. they experiment with different classifiers and achieve the best results with a random forest. @xcite conduct a small-scale analysis on 30 articles from the wall street journal which have been manually annotated on a scale from 1-5 for the question of how well-written the article is. they investigate how various linguistic features correlate with these scores. vocabulary and discourse relations are the strongest predictors of readability, followed by average number of verb phrases and length of the text. @xcite work with three english-language datasets and produce hybrid models which consist of a transformer based model combined with a feature-based model. they predict 3 and 5 classes (depending on the dataset) and achieve the state of the art, with a roberta-based transformer model performing best. work on german-language text complexity assessment is fairly rare. @xcite look at text-level binary readability classification using a corpus of 1627 articles in original form and a version aimed at children. their classifier uses the sequential minimal optimization algorithm with five groups of features (traditional readability formulas, lexical, syntactic, language model, and morphological), with a best accuracy score of 89.7%. @xcite work with various corpora from different languages from the text simplification domain and evaluate 104 different features using statistical tests, with the aim to determine differences between simplified and complex texts. they also work with a german-language corpus of 1888 texts @xcite and find that the feature lexical complexity, in particular, is relevant specifically for german texts. battisti et al. (2020) build on the same corpus and release a newer version with 6217 documents. hewett and stede (2021) create a corpus of 2655 texts from online lexica at three different levels (adults, children, children who are beginner readers) and use knowledge graph based features to estimate conceptual complexity. in a pairwise classification task they achieve an accuracy score of 91%. ## dataset the dataset for the challenge consists of sentences that have been taken from 41 wikipedia articles from different article genres. groups of german learners, with language levels between a2 and b2, rated the sentences according to complexity, understandability and lexical difficulty on a scale from 1 to 7. for each aspect, the arithmetic mean (or mean opinion score; mos) was calculated and the task was to predict the mos complexity score of the sentences. more information on the dataset can be found in @xcite . the training dataset consists of 1000 sentences, the validation set (for development phase) of 100 and the test set (for the evaluation phase) of 210 sentences. figure 1 shows a histogram of the target variable (mos) in the training set (@xmath0.02, @xmath1.18). some examples from the training set can be seen in it is also worth mentioning that 'complexity' can be subjective. for example as can be seen in ## approaches in this section we outline our different approaches. as a baseline, we take the simple approach of predicting the mean mos value (3.02) for all samples. using this baseline, the root mean squared error (rmse) is 1.18. foot_0 ## ensemble results & discussion our final approach is to combine our feature-based and neural approach by averaging the outputs of these two models. 3 while both our transformer and feature-based models perform better than the baseline rmse, among them, the transformer model does generally better on different data splits. thus, it might seem paradoxical that our final model is a weighted average (ensemble) of the two. using an ensemble method is, however, a theoretically sound practice, and quite common in machine learning competitions. in the words of @xcite , "to rely on a single model is hubris. it invites disaster. [...] wisdom can be achieved by averaging models." simply explained, due to both over-fitting and under-fitting, any one model will predict some samples (especially among the unobserved) quite wrongly. as long as the individual models in the collection do not share a common bias, then any diverse collection of the models will be more accurate than the average member-an implication of the so called diversity prediction theorem @xcite . to illustrate the point, we can compare the predictions from both models on the test dataset (figure 2 ). the pearson correlation coefficient between the two is 0.85. on average, the predictions are close, with the transformer model predicting slightly lower scores. in about ten percent of the samples, the difference between the predictions is bigger than 1, and crucially, in both directions. the final results on the test dataset are presented in ## conclusion in this paper we explained our contribution to the shared task text complexity de challenge 2022. we experimented with both neural and featurebased approaches. our best-performing model is a weighted average of a fine-tuned xlm-r transformer model and a classic feature-based model with linear regression. the ensemble achieves a mapped root square mean error of 0.446 on the test data which is better than either of the models alone.