TUM Social Computing at G erm E val 2022: Towards the Significance of Text Statistics and Neural Embeddings in Text Complexity Prediction (2022)

Abstract: In this paper, we describe our submission to the GermEval 2022 Shared Task on Text Complexity Assessment of German Text. It addresses the problem of predicting the complexity of German sentences on a continuous scale. While many related works still rely on handcrafted statistical features, neural networks have emerged as state-of-the-art in other natural language processing tasks. Therefore, we investigate how both can complement each other and which features are most relevant for text complexity prediction in German. We propose a fine-tuned German DistilBERT model enriched with statistical text features that achieved fourth place in the shared task with a RMSE of 0.481 on the competition’s test data.

## introduction text readability describes how easy a given text is understood by a specific reader @xcite . factors that influence the readability are, for example, the number of technical terms in the text or the length and convolution of the sentences. assessing a text's readability can be used to select the proper texts for a specific user group or provide authors feedback about their texts. moreover, it can be integrated into an automatic text simplification system. on the one hand, it helps to decide whether and, if so, how much a text should be simplified. on the other hand, readability assessment is a measure to evaluate a simplification system by checking if the output has a higher readability @xcite . text complexity is inversely related to text readability; thus, in this work, the terms text complexity prediction and readability assessment are used interchangeably. this paper is a contribution to the germeval 2022 shared task on text complexity assessment of german text that aims to predict the complexity of a german text on a continuous scale @xcite . we propose a model based on fine-tuned german distilbert @xcite combined with traditional readability formulas and statistical text features. this model achieved fourth place in the competition. moreover, we used shap @xcite to explain our model's predictions and discuss which features contribute to higher complexity. by knowing the feature relevance, authors and machine learning engineers can pay attention to them when generating new texts. our code is released on github for further research and development. 1 this paper is structured as follows: section 2 gives an overview of existing readability formulas and prediction models. in section 3, we present the organization of the shared task and introduce its dataset. then, section 4 walks through our proposed approaches and entails their performance. finally, in section 5, we apply explainability methods to discuss text features relevant to complexity prediction. ## related work we investigated two approaches for readability assessment, traditional readability formulas, and deep learning. therefore, this section gives an overview of existing formulas and models. moreover, we analyze which text features yielded promising prediction results in previous work. ## shared task and dataset this paper explains our submission to the ger-meval 2022 shared task on text complexity assessment of german text @xcite . the shared task was split into two different phases, a development and a final phase. during development, participants were provided a labeled training and an unlabeled validation dataset. predictions on this validation data could be uploaded to the competition page with immediate evaluation feedback. in contrast, during the final phase, the results on the final test dataset were only published at the end of the competition. the two evaluation datasets, the validation and the final test data, consist of 100 sentences each. the training dataset for this shared task originates in work by @xcite . it contains 1000 sentences from the german wikipedia together with a complexity score ranging from 1 to 7. @xcite used crowdsourcing to let non-native speakers of a b level annotate the respective sentences by their perceived readability and averaged the scores among the participants. the mean complexity value is 3.016 with a standard deviation of 1.181. there are 76 sentences with an observed complexity of 1.0, but only two samples with a complexity higher than six, making the dataset unbalanced towards the easier sentences. to counteract this imbalance, we replicated sentences with a complexity higher than 5.5 multiple times, yielding a dataset with 1054 samples. the rooted mean squared error (rmse) between predicted and correct complexity scores was used to evaluate a model's performance. in addition, a third-order polynomial function was fitted between the predicted and correct scores to counteract the bias by subjective annotation of text complexity. then, the predicted scores were projected using this function, and the error was calculated on the mapped predictions as well @xcite . ## approaches in this section, we explain the three approaches we explored to predict the complexity score of a sentence. we did not apply any preprocessing to the data, i.e., fed the sentences into the model's tokenizer directly. ## explaining the predictions to evaluate which of the suggested statistics and formulas help to predict the complexity of german texts, we calculated the shapley additive explanations (shap) values (lundberg and lee, 2017) for each of our models. shap measures each feature's contribution by masking their different combinations and rerunning the predictions with these masks. features for which the masked predictions deviate strongly from the initial prediction have a substantial impact and are, thus, the most relevant ones. the shap values are calculated per sample and averaged among them. figure 1 shows the mean shap values for each feature in the statistical svr model (section 4.1). the most relevant statistic is the average sentence length, i.e., the longer a sentence is, the more likely it is complex. the fre score uses this statistic; thus, it is reasonable that it has high importance. even though the wiener sachtext formulas also include this statistic, their contribution to the predicted score is smaller. they incorporate more advanced measures like the percentage of words with more than three syllables. as indicated by the small shap values, these additional statistics are not helping our complexity prediction model. the third most relevant feature is the maximum tree depth, indicating how convoluted a sentence is. for a neural network, it is unknown what functionality a specific neuron models. therefore, a feature-relevance analysis is not beneficial for interpreting a neural network. instead, we selected the example sentence "dieser vorgang wird gletscherschwund oder gletscherschmelze genannt." ("this process is called glacier recession or glacier melt.") and investigated which words have an impact on the prediction. the correct complexity for this sentence is 2.266667, and our model (section 4.2) predicts a complexity of 2.373029. figure 2 0.00 0.05 0.10 0.15 0.20 0.25 mean(|shap value|) (average impact on model output magnitude) distil_606 wstf4 distil_635 distil_354 distil_571 distil_665 distil_73 fre_amstad asl distil_149 figure 3: feature relevance analysis for combined model: the ten features with highest shap values. "distil_i" indicates the ith index in the distilbert embedding. ## discussion readability is a subjective measure that depends on the reader's background knowledge and reading ability @xcite . our work is based on the shared task's dataset labeled with a crowdsourcing approach among non-native speakers. therefore, the findings in this paper should be tested for transferability to other datasets and groups of readers. in addition, the dataset is unbalanced with an overrepresentation of simple sentences and contains some noise. for example, the sentence "martin luther king jr (* 15 januar 1929 in atlanta als michael king jr; † 4 april 1968 in memphis) war ein us-amerikanischer baptistenpastor und bürgerrechtler." ("martin luther king jr (born january 15, 1929 in atlanta as michael king jr; † april 4, 1968 in memphis) was a u.s. baptist pastor and civil rights activist.") has a complexity of 1.0, indicating it was a very easy sentence. this shows that some samples have lower complexity than they would have when relabeling the dataset. ## conclusion in this paper, we have demonstrated three approaches for text complexity prediction in german, one model that relies on handcrafted statistical features only, one fine-tuned transformer network, and a combination of both. in addition, we found that the feature most indicative of a complex sentence is the sentence length and that the fre formula by @xcite gives a good indication of text complexity. modern transformer architectures with deep textual understanding can build accurate complexity prediction pipelines. however, they can still be improved with handcrafted statistical features, showing that they have not yet superseded traditional approaches. in future work, these findings will be extended to a paragraph and full-text level instead of a sentence-wise prediction.