Everybody likes short sentences - A Data Analysis for the Text Complexity DE Challenge 2022 (2022)

Abstract: The German Text Complexity Assessment Shared Task in KONVENS 2022 explores how to predict a complexity score for sentence examples from language learners’ perspective. Our modeling approach for this shared task utilizes off-the-shelf NLP tools for feature engineering and a Random Forest regression model. We identified the text length, or resp. the logarithm of a sentence’s string length, as the most important feature to predict the complexity score. Further analysis showed that the Pearson correlation between text length and complexity score is about \rho ≈ 0.777. A sensitivity analysis on the loss function revealed that semantic SBert features impact the complexity score as well.

## introduction we create and extract features from pre-trained nlp models and train a random forest model to predict scores of the textcomplexityde dataset @xcite because we want to find out what evaluation criteria the annotators, here language learners, used. using handcrafted features was the common approach before the breakthrough and wide adoption of deep learning models. for example, @xcite combine transformer models with random forest models based on 255 manually specified features for readability assessments. @xcite predict the cefr-level of a text with support-vector machines and linguistic features, e.g., lexical, syntactic, discourse-based. @xcite and @xcite measure text difficulty with word familiarity, false cognates, morphological inflections, and phonetic complexity in c-tests. @xcite handcraft linguistic features assuming these may be relevant due to human cognition, or resp., working memory limits. the advantage of manual feature engineering is that it allows to assess the impact of each feature or group of features later on, e.g., sensitivity on the loss function, and feature importance in random-forest. in other words, the model becomes partially explainable, and allows deriving feedback for practitioners such as language teachers. ## feature engineering we use sentence-level features addressing different language levels by using different types of features generated by or derived from off-the-shelf nlp tools ( ## experiments dataset. the subject of this shared task is the textcomplexityde dataset by @xcite . its training set contains 1000 german sentence example from wikipedia. each sentence example had 3 items with likert-scale from 1 to 7 resulting in a) complexity, b) understandability, and c) lexical difficulty scores. and 369 german language learners provided, 10650 valid sentence ratings. random-forest feature importance. we trained the multi-output random-forest @xcite implementation of scikit-learn package @xcite with 100 trees, max. tree depth of 16, and at least 10 samples per leaf, as well as bootstrap aggregation with subsample size of 50% and out-of-bag errors. the text length. the linear relationship between complexity score and the logarithm of the number of characters per sentence has a pearson correlation coefficient of ρ ≈ 0.777 with a p-@xmath0. ## discussion an explanation for the text length as the dominant feature for the textcomplexityde dataset could be the working memory @xcite , or cognitive load theory for sentence comprehension @xcite . foreign language texts are new to a language learner to varying degrees. dealing with new things can require more conscious and analytical information processing, which is more cognitively demanding. respondents may have developed and applied text length as a heuristic while answering the survey, what can be explained by the effort-reduction framework @xcite . in extreme cases, a study participant could only measure the black and white contrast of the dark letters on a light background as an approximation for the text length, i.e., a person do not even have to read the text to assign a score. however, some part of the complexity score is related to semantic sbert features, i.e., the text content still mattered to the survey participants. the other proposed evaluation criteria (e.g., node distance, consonant cluster, word frequency) cannot explain the dependent variables of the textcomplexityde dataset. ## conclusion although the study designer can ask for thoughtful responses, this does not prevent study participants or annotators from using or developing heuristics such as text lengths. we suggest two solutions to prevent annotators from using text length as scoring heuristic. first, use text length as a control variable during the survey, i.e., a participant assess a set of sentence examples of a similar text length. this would force the participant to consider other evaluation criteria related to the survey question. although the implementation is easy, the annotation time would increase because participants might develop more differentiated sets of evaluation criteria. second, ask the participant to translate each german sentence example into their native language before assigning a score. this countermeasure would ensure that participants spend time for details, and may weight less obvious evaluation criteria higher, e.g., they became aware of the syntactic or lexical similarity between both languages. the drawback is that the annotation time would increase considerably when survey participants create a parallel corpus. roland schäfer. 2015. processing and querying large web corpora with the cow14 architecture. proceedings of the 3rd workshop on challenges in the management of large corpora (cmlc-3), lancaster, 20 july 2015, pages 28 -34, mannheim. institut für deutsche sprache. roland schäfer and felix bildhauer. 2012. building large corpora from the web using a new efficient tool chain. in proceedings of the eight international conference on language resources and evaluation (lrec'12), pages 486-493, istanbul, turkey. european language resources association (elra). helmut schmid. 2006. a programming language for finite state transducers. in anssi yli-jyrä, lauri karttunen, and juhani karhumäki, editors, finite-state methods and natural language processing, volume 4002, pages 308-309. springer berlin heidelberg, berlin, heidelberg. series title: lecture notes in computer science. helmut schmid, arne fitschen, and ulrich heid. 2004. smor: a german computational morphology covering derivation, composition and inflection. in proceedings of the fourth international conference on language resources and evaluation (lrec'04), lisbon, portugal. european language resources association (elra). anuj k shah and daniel m. oppenheimer. 2008. heuristics made easy: an effort-reduction framework. psychological bulletin, 134 2:207-22. menglin xia, ekaterina kochmar, and ted briscoe. 2016. text readability assessment for second language learners. in proceedings of the 11th workshop on innovative use of nlp for building educational applications, pages 12-22, san diego, ca. association for computational linguistics.