Improving Readability Assessment with Ordinal Log-Loss (2024)

Abstract: Automatic Readability Assessment (ARA) predicts the level of difficulty of a text, e.g. at Grade 1 to Grade 12. ARA is an ordinal classification task since the predicted levels follow an underlying order, from easy to difficult. However, most neural ARA models ignore the distance between the gold level and predicted level, treating all levels as independent labels. This paper investigates whether distance-sensitive loss functions can improve ARA performance. We evaluate a variety of loss functions on neural ARA models, and show that ordinal log-loss can produce statistically significant improvement over the standard cross-entropy loss in terms of adjacent accuracy in a majority of our datasets.

## introduction automatic readability assessment (ara) aims to predict the level of difficulty of a text, e.g. at grade 1 to grade 12. it can be helpful for teachers and students in identifying and revising text to the desirable level of difficulty. ara is an ordinal classification task since the levels follow an underlying order, from easy to difficult. yet, in ara models trained with traditional machine learning, the use of ordinal classification has yielded mixed results @xcite @xcite . further, most neural ara models treat the task as multi-class classification @xcite @xcite @xcite @xcite and ignore the distance between the gold level and predicted level. in these models, a classifier is typically trained with the standard cross-entropy loss function, which treats the difficulty levels as independent labels. further, performance evaluation often penalizes incorrect predictions equally, regardless of their distance from the gold. recognizing the ordinal nature of ara could potentially enhance performance and enable more accurate evaluation. a loss function that reflects label distance could be suitable, since the boundary between difficulty levels may not be clear-cut, especially on fine-grained scales. while severe mistakes are never desirable, a sufficiently close prediction may be acceptable in some applications, such as retrieval of extra-curricular reading materials. evaluation metrics that reflect the average distance from the gold label would therefore be more informative. distance-sensitive loss functions have received relatively little attention in neural ara. @xcite showed that soft labels could improve performance, but the evaluation was limited to bert and only one loss function. we present a more comprehensive study on a variety of loss functions, evaluated on a range of pre-trained language models, hyper-parameters, and performance metrics. experimental results show that ordinal logloss @xcite performs best overall for neural ara models. it achieves a statistically significant improvement over the standard crossentropy loss in terms of adjacent accuracy in a majority of our datasets, though sometimes at the expense of accuracy. the rest of the paper is organized as follows. after a review of the major loss functions in section 2, we give details on the experimental set-up in section 3. we then report results in section 4. 1 ## previous work many text classification tasks, ranging from ara and essay scoring, to sentiment and review rating prediction, have an ordinal structure. let @xmath0, r 2 , ..., r k } be the set of possible labels. ordinal binary classification exploits the structure 1 code and data can be accessed at https://github.com/hhlim333/readability-assessmentwith-ordinal-log-loss with k-1 binary classifiers @xcite . ordinal multi-class classification with voting was found to be potentially helpful in improving ara performance @xcite . ordinal regression models have been applied to ara models trained in traditional machine learning. while @xcite found that the proportional odds model offered competitive performance, @xcite reported that ordinal classifiers did not perform better than standard classifiers. losssensitive classification, which is the focus of this paper, utilizes loss functions that impose higher penalty to predictions further from the gold label, based on a distance function d(r i , r j ) that specifies the distance between labels r i and r j . two main families of these loss functions are as follows. ## experimental set-up this section describes the loss functions (section 3.1), the datasets (section 3.2) and training procedure (section 3.3). ## experimental results all results are averaged based on stratified 5-fold cross-validation with a 8:1:1 split for train/validation/test. we first report overall results based on mean absolute error (mae) and mean squared error (mse) (section 4.1), and then analyze their performance in terms of adjacent accuracy and accuracy. 10 henceforth, all chinese results are based on macbert, and the english results on roberta, sicne they performed best among the four plms evaluated (see ## conclusion since ara is an ordinal classification task, the magnitude of classification error should in principle be taken into account. this paper has presented a comprehensive evaluation of a variety of loss functions that are sensitive to the distance between the predicted label and gold label. our experiments on neural ara models suggest that ordinal log-loss (oll) is able to capture the ordinal nature of the task, reducing the mean absolute error and mean squared error on most datasets. it produces significant improvement over the standard cross-entropy function in terms of adjacent accuracy, but at the expense of accuracy in some settings. these results suggest that future ara models should consider using oll for applications that need to avoid severe errors but do not require precise classification. a number of research directions may be pursued. first, ara accuracy could be further improved by optimizing the distance function in the ordinal logloss and soft label models. second, the usability of the ara model in an educational setting, for example assisting teachers and students in text selection and revision, is also worth investigating.