Tackling Data Drift with Adversarial Validation: An Application for G erman Text Complexity Estimation (2022)

Abstract: This paper describes the winning approach in the first automated German text complexity assessment shared task as part of KONVENS 2022. To solve this difficult problem, the evaluated system relies on an ensemble of regression models that successfully combines both traditional feature engineering and pre-trained resources. Moreover, the use of adversarial validation is proposed as a method for countering the data drift identified during the development phase, thus helping to select relevant models and features and avoid leaderboard overfitting. The best submission reached 0.43 mapped RMSE on the test set during the final phase of the competition.

## introduction automatically assessing how easy to read a text is has many applications, ranging from text simplification for language learners and people with disabilities to customizing content for a particular audience. for this reason, the natural language processing (nlp) research community have been organizing shared tasks and compiled linguistic resources aiming to solve this problem, not only in english but also for other languages. the text complexity de challenge 2022 @xcite proposes the evaluation of systems able to predict the complexity of german texts by rating each sentence using the mean opinion score (mos), derived from annotations from a 7 point likert-scale. in order to solve this problem and addressing the unexpected data drift between training and testing sets, meta-modeling and adversarial validation techniques were applied by combining predictions from multiple estimators via stacked generalization @xcite and leveraging both traditional feature engineering and pre-trained resources. the best submission generated by the described approach and selected through adversarial validation won the competition by achieving the lowest mapped root mean squared error (rmse) score 1 . this paper is organized as follows: first, related work is reviewed in section 2. next, section 3 contains an analysis of the individual models and feature engineering approaches used for this task. in section 4, model selection and adversarial validation strategies are discussed. further on, the performance of the system and its components are detailed in section 5. finally, in section 6 the author draws the main conclusions and outlines future work. ## related work the application of nlp techniques for automatic textual complexity assessment has received attention in several languages other than english @xcite @xcite , although in an smaller scale. despite the differences between languages, the use of lexical, morphological and word list-derived features are also common in research works focused on german @xcite . likewise, related nlp applications such as readability assessment @xcite or evaluation of text simplification pipelines @xcite demonstrated that similar approaches used to estimate the complexity of english texts could be suitable for german as well, although with some known shortcomings. ## methodology the textcomplexityde @xcite dataset that consists of 1000 sentences in german language taken from 23 wikipedia articles was the only resource provided by the organizers. in order to solve the challenge, this dataset was used as training data following two main approaches: feature engineering based on morphological and lexical information @xcite and transfer learning via pre-trained transformers. the regression models trained using these two different strategies and the methodology applied to combine their predictions are described in detail below. ## model selection and adversarial validation in the final phase of the competition it became clear that validation and test data had relevant dissimilarities. some potential reasons were identified by the participants such as the application of nonrandom splits or different pre-processing 8 . while this is not a totally uncommon phenomenon in nlp @xcite to the best of the authors' knowledge there have not been many efforts to address this problem in comparison with other domains. the use of adversarial validation as a solution to identify concept drift has been explored recurrently in the literature @xcite . however, due the relatively small data sizes involved, the usual approach of training a binary classifier between 7 https://huggingface.co/amine/bert-base-5lang-cased 8 https://codalab.lisn.upsaclay.fr/forums/4964/741/ train/dev/test sets and selecting the data points with the closest distribution @xcite was deemed sub-optimal. therefore, principal component analysis (pca) was used instead in order to calculate low-dimensional projections of the evaluation datasets and estimate their drift from the training data by analyzing the reconstruction errors. taking that into account, the author hypothesized that models using features that would remain stable across different data splits based on the criteria define above would exhibit better correlation between the errors estimated during cross-validation and the final scores. in ## conclusions and future work this paper introduces a meta-model for german text complexity estimation using both manual feature engineering and neural networks. the use of adversarial validation by comparing feature distribution changes between different datasets is proposed as a mechanism to detect data drift via pca reconstruction errors. the described system has achieved the first ranking in mapped rmse in the text complexity de challenge of konvens 2022. in a future work, this approach can be extended through autonlp techniques in order to build multi-lingual text complexity estimation solutions that could be integrated in other nlp pipelines.