A Simple Log-based Loss Function for Ordinal Text Classification (2022)

Abstract: The cross-entropy loss function is widely used and generally considered the default loss function for text classification. When it comes to ordinal text classification where there is an ordinal relationship between labels, the cross-entropy is not optimal as it does not incorporate the ordinal character into its feedback. In this paper, we propose a new simple loss function called ordinal log-loss (OLL). We show that this loss function outperforms state-of-the-art previously introduced losses on four benchmark text classification datasets.

## introduction for many classification tasks, there is an order on the labels of the target variable. in particular, in natural language processing (nlp) when, for example, we are trying to predict the number of stars associated with a review: it is obvious that when the label is 1 star, predicting 2 stars is better than predicting 5 stars. this type of classification is called ordinal classification (or ordinal regression) and many techniques have been developed in recent years around it. among the most used techniques, the ordinal binary classification consists in decomposing the ordered target variable in several binary ones @xcite . the threshold methods treat the target variable (with n classes) as a continuous real-valued variable and n -1 thresholds are introduced @xcite @xcite . in the loss-sensitive classification the loss function is built such that a higher penalty is assigned if the distance between the prediction and the label is higher. several losses can be used here: mapping the labels {c 1 ; c 2 ; ...; c n } into values {1; 2; ...; n } and use the mean squared error. the margin loss or the hinge loss can also be extended for ordinal regression @xcite . the weighted kappa loss (de la @xcite , the earth mover's distance @xcite , the soft labels @xcite or the coral method @xcite are other examples of recent modified losses introduced in ordinal classification problems. in order to measure the performance of the ordinal regression there are well known metrics such as the off-by-k-accuracy, the mean absolute error, the mean squared error or kendall tau for instance @xcite . ## experiments and results in this section we first introduce the public datasets (section 3.1) and the metrics (section 3.2) used to compare our loss function to existing ones. then in section 3.3 we present the different results obtained. ## conclusion we introduced a simple and novel loss function specially designed for the ordinal classification task. this loss is intuitive and easy to use. we evaluated our method on four benchmark ordinal text classification datasets and against five different metrics. our loss outperforms state-of-the-art comparable and previously introduced losses. we also experimentally find good hyper-parameters to use. thus, the contribution of this article is to introduce a new loss (oll) that is easier to use than the majority of recently introduced losses and which gives better results for ordinal classification applied to nlp tasks. we believe that those results could be extended to other machine learning tasks in computer vision, speech or structured data for instance.