Text Complexity DE Challenge 2022 Submission Description: Pairwise Regression for Complexity Prediction (2022)

Abstract: This paper describes our submission to the Text Complexity DE Challenge 2022 (Mohtaj et al., 2022). We evaluate a pairwise regression model that predicts the relative difference in complexity of two sentences, instead of predicting a complexity score from a single sentence. In consequence, the model returns samples of scores (as many as there are training sentences) instead of a point estimate. Due to an error in the submission, test set results are unavailable. However, we show by cross-validation that pairwise regression does not improve performance over standard regression models using sentence embeddings taken from pretrained language models as input. Furthermore, we do not find the distribution standard deviations to reflect differences in “uncertainty” of the model predictions in an useful way.

## introduction this paper describes our submission to the text complexity de challenge 2022 @xcite . the task is to predict the linguistic complexity of a given sentence. the task is defined as a regression task, where labels @xmath0. labels are averaged human ratings, who rated the sentences for complexity, understandability, and lexical defficulty (see @xcite ) for details). only complexity labels are taken into account in this shared task. the train set consists of 1000 labelled sentences, the development set consists of 100 sentences, and the test set contains 210 sentences. only the labels of training sentences where ever revealed to participants. in this paper, we evaluate pairwise regression for complexity score prediction. instead of predicting a single complexity score from a single sentence, we predict the relative difference in complexity of two sentences. in practise, this results in a distribution over complexity scores instead of a point estimate, because we predict the relative difference for each training sentence. however, further analysis reveals that pairwise regression neither performs better than standard regression nor does the standard deviation of score distributions contain useful information about model performance. furthermore, due to an erroneous submission we do not have test set score for this shared task. therefore, all our analyses and observations are based on 10-fold cross-validation on the training data. ## related work readability scoring of texts is has been researched for over a century. research started by developing readability formulas based on surface features such as token counts or type-token ratios. modern approaches use statistical methods, especially supervised learning, to learn readability models. here, readability scoring can be defined both as a regression task @xcite and a classification task @xcite . features usually rely on broad linguistic modelling @xcite . recently, deep neural networks have also been proposed for predicting readability labels @xcite . furthermore, the utility of linguistic features compared to deep representations was put into question by @xcite . however, the main disadvantage of deep neural networks is their black-box nature. this is especially problematic, because practical applications of readability models generally require an especially high level of transparency, for example in an educational context (for giving feedback) or for essay scoring (where grades should be explainable and fair). ## method in this section, we describe our approach at predicting the linguistic complexity of given sentences. we train deep learning models (described in section 3.1) and also compare them to a traditional machine learning model based on linguistic features (see section 3.2). ## results here, we present performance results of the pairwise regression model (see section 3.1) and baselines (see section 3.2). unfortunately, we cannot present the shared task's test set scores due to an erroneous submission: instead of submitting results on the real test set, we accidentally submitted results on a custom test set that we had created for internal evaluation. this error remained unnoticed until after the submission deadline. therefore, we decide to report 10-fold cross-validation results on the training set, because we do also not have development set scores for all baselines and ablations. results for the pairwise regression model are in in figure 3 , we show the loss curve for a pairwise regression model only predicting the relative difference and using all sentence embeddings. the curve shows that the loss starts to decrease quickly after about half an epoch. this may be an artifact of the initially very low learning rate due to the one-cycle-scheduler. after about 2 epochs, the loss only shows little improvements. this suggests that training for fewer epochs may already be sufficient. however, given that we did not observe better generalisation performance with shorter training, this may also suggest that the model is somewhat robust to longer training and still does not overfit the data. results for the baselines are in ## analysis in section 4, we have established that pairwise regression does not achieve better performance than direct prediction of absolute complexity scores. however, we are still interested in whether having a distribution of scores instead of a single score can provide additional insights. for example, it would be of advantage if we could use the score sample standard deviation to detect uncertain predictions, i.e. sentences where the model is not confident about the complexity. to be able to do this, the sample standard deviation has to correlate with the prediction error. this is, however, not the case: figure 4 shows that while most errors are small, sentence score distributions that result in larger prediction errors do not have larger standard deviation. in fact, pearson correlation is -0.18, however the negative value could be an artifact of the small number of large errors. therefore, we conclude that the score distribution predicted by pairwise regression models does not provide further insights into the model predictions. finally, we also evaluate whether we can find linguistic features that are informative about which ## discussion we evaluated pairwise regression in comparison to standard regression (predicting a single complexity score from a single sentence). our results are largely negative, showing that pairwise regression does not perform better than standard regression and the resulting score distribution does not seem to have additional use over the point estimates returned by standard regression. furthermore, there seems to be no trend that can be captured by linguistic features about which sentences are more difficult to score by deep learning based models. on the positive side, our evaluations show that pairwise regression and standard regression can be exchanged with only very little difference in prediction quality, and that deep learning based models perform somewhat better than models based on linguistic features.