Pseudo-Labels Are All You Need (2022)

Abstract: Automatically estimating the complexity of texts for readers has a variety of applications, such as recommending texts with an appropriate complexity level to language learners or supporting the evaluation of text simplification approaches. In this paper, we present our submission to the Text Complexity DE Challenge 2022, a regression task where the goal is to predict the complexity of a German sentence for German learners at level B. Our approach relies on more than 220,000 pseudolabels created from the German Wikipedia and other corpora to train Transformer-based models, and refrains from any feature engineering or any additional, labeled data. We find that the pseudo-label-based approach gives impressive results yet requires little to no adjustment to the specific task and therefore could be easily adapted to other domains and tasks.

## introduction what makes some texts more difficult to read for learners of a foreign language than others? how does a complicated sentence construction or the use of rare vocabulary increase complexity? the prediction of text complexity with machine learning methods addresses these questions. in contrast to last years' shared tasks at konvens, which focused on the disambiguation of german verbal idioms @xcite , the identification of toxic, engaging, and fact-claiming comments @xcite , and scene segmentation in narrative texts @xcite , the task of 2022 is about text complexity. in this paper, we present our submission to this text complexity de challenge 2022. it is a shared task addressing the automatic estimation of the complexity of german sentences for readers, in particular, german learners at level b. the provided training dataset contains about 1000 sentences and the test dataset about 300 sentences in german. figure 1 shows an exemplary sentence from the shared task dataset in german, an english translation, and the arithmetic mean of ratings from all annotators. with a seven-level likert-scale with values ranging from very easy (1) to very complex (7), this task is a regression task and it is evaluated using the root mean squared error (rmse). a third-order mapping is applied before the error is measured so that the impact of any systematic bias in the predictions on the metrics is reduced. thereby, the focus of the evaluation is shifted towards ranking sentences correctly with regards to their complexity rather then assigning the correct absolute complexity score. we refer to the overview paper of the shared task for more details about the dataset and the overall results @xcite . the remainder of this paper is structured as follows. section 2 summarizes related work on text complexity estimation and on pseudo-labeling techniques for machine learning. we describe our approach in section 3 and its evaluation in section 4, with experiments on the validation dataset provided by the shared task organizers. we conclude in section 5 and provide an outlook on future work. ## related work research on reading complexity of german texts is so far relatively scarce with several papers introducing datasets of annotated german sentences or longer texts and mostly feature-based approaches for text complexity prediction. first of all, there is a dataset with sentence-level annotations, which is the basis of this shared task @xcite . @xcite introduce a dataset for document-level text complexity with the application focus of text simplification and there are two other document-level text complexity datasets by @xcite and by hewett and stede german sentence: als versauerung der meere wird die abnahme des ph-wertes des meerwassers bezeichnet. english translation: ocean acidification is the term used to describe the decrease in the ph of seawater. compexity score: 2.13 german sentence: nach chemischer härtung des rußes war er in der lage, auf galvanoplastischem wege ein zink-positiv und von diesem ein negativ der platte anzufertigen, das als stempel zur pressung beliebig vieler positive genutzt werden konnte -die schallplatte war erfunden. english translation: after chemical hardening of the carbon black, he was able to produce a zinc positive by galvanoplastic means and from this a negative of the record, which could be used as a stamp for pressing any number of positives -the record was invented. complexity score: 4.70 (2021). the latter follows the format of a similar study @xcite based on a dataset of english newspaper articles @xcite . another dataset is from a kaggle challenge called commonlit readability prize, where the task is to rate the complexity of literary passages for school grades 3-12. foot_0 last but not least, there are unlabeled datasets of german texts with simple language, such as the tagesschau/logo corpus and the geo/geolino corpus @xcite or klexikon @xcite . these datasets cannot be used directly for fine-tuning models on the text complexity prediction task due to the lack of annotations. however, we show in our approach that they can be used in combination with pseudo-labeling. similar to the pseudo-labeling approach that we use, there is a data augmentation technique where a slow but more accurate cross-encoder model is used to label a large set of otherwise unlabeled data samples @xcite . this technique augments the training data for a faster, less complex ## a semi-supervised learning approach for text complexity prediction our semi-supervised learning approach uses neural language models based on the transformer architecture @xcite . as pre-trained models that are not fine-tuned to a specific natural language processing task yet, we use gbert and gelectra models by @xcite and an xlm-roberta model by @xcite . given that the training dataset provided by the shared task organizers is relatively small for fine-tuning these pre-trained models, the core idea of our approach is to increase the number of training samples by automatically generating pseudolabels. figure 2 visualizes the different steps of the entire approach with its three main steps: pseudolabeling, fine-tuning, and ensembling. for the implementation of these steps, we use the two open train base model on human labels collect large unlabeled corpus pseudo-label corpus with base model finetune on pseudo-labels finetune on human labels with cross-validation train linear model for ensembling on out-of-fold predictions source frameworks farm foot_2 and haystack foot_3 . the first step is to create a large corpus of german sentences with varying text complexity to serve as a source for the pseudo-label sentences. this corpus comprises the following resources: the pseudo-labels are used to fine-tune different transformer-based models on the task of complexity scoring. we fine-tune deepset/gelectralarge, deepset/gbert-large and xlmroberta-large using three different seeds for each model, resulting in a total of nine models. subsequently, we fine-tune each of these models using five-fold cross-validation with the original training set, resulting in a total of 45 models. finally, to combine these 45 models into an ensemble providing a single prediction score per data sample, we train linear regression models on the out-of-fold predictions from the previous cross-validations. ## experiments we evaluate four different settings using five-fold cross-validation: we submitted the predictions of each of the last three settings to the shared task competition. 9 the first setting serves as our baseline with a deepset/gbert-large model fine-tuned on the provided training data. the model is trained on each of the cross-validation folds using early stopping for a maximum of four epochs. each training run was tracked using mlflow and can be found here. the second setting is an ensemble of the language models deepset/gbert-large, deepset/gelectra-large and xlmroberta-large. each of these models is fine-tuned for two epochs on the pseudo-labels described in section 3 using three different random seeds. this results in an ensemble of nine models. one training run takes approximately three hours on an nvidia tesla v100 gpu with 16 gb of ram. for the third and fourth setting, we fine-tune the resulting models of the previous step on the provided training dataset. to further increase the number of models in the ensemble, we perform five-fold cross-validation on each of the nine models, resulting in a total of 45 models. again, each training run was tracked using mlflow and can be found here. to ensemble these 45 models, we use two different techniques. the third setting aggregates each individual score into a single score 9 https://codalab.lisn.upsaclay.fr/ competitions/4964 by simply taking the arithmetic mean of all scores. the fourth setting trains a linear ridge regression model on the out-of-fold predictions for each model that we trained, resulting in five linear regression models. applying these linear regression models decreases the number of scores from 45 to 5. to get a single score out of these five scores, we calculate their arithmetic mean. ## conclusion in this paper, we presented our submission to the text complexity de challenge 2022. we leveraged pseudo-labeled sentences from wikipedia and several other publicly available, unlabeled corpora. based on the labeled training dataset from the shared task and the additional pseudo-labeled data, we fine-tuned transformer-based neural language models. our best ensemble model achieved an rmse of 0.433 in cross-validation on the public dataset without third-order mapping and an rmse of 0.454 on the private test dataset with third-order mapping (0.484 without third-order mapping). for future work, our trained model could be used to create more pseudo-labels for another iteration of the entire approach, presumably resulting in a model that generalizes even better to unseen test data.