Error-Sensitive Evaluation for Ordinal Target Variables (2021)

Abstract: Product reviews and satisfaction surveys seek customer feedback in the form of ranked scales. In these settings, widely used evaluation metrics including F1 and accuracy ignore the rank in the responses (e.g., ‘very likely’ is closer to ‘likely’ than ‘not at all’). In this paper, we hypothesize that the order of class values is important for evaluating classifiers on ordinal target variables and should not be disregarded. To test this hypothesis, we compared Multi-class Classification (MC) and Ordinal Regression (OR) by applying OR and MC to benchmark tasks involving ordinal target variables using the same underlying model architecture. Experimental results show that while MC outperformed OR for some datasets in accuracy and F1, OR is significantly better than MC for minimizing the error between prediction and target for all benchmarks, as revealed by error-sensitive metrics, e.g. mean-squared error (MSE) and Spearman correlation. Our findings motivate the need to establish consistent, error-sensitive metrics for evaluating benchmarks with ordinal target variables, and we hope that it stimulates interest in exploring alternative losses for ordinal problems.

## introduction organizations have a vested interest in ensuring customer happiness. to measure this quantity, analysts often use surveys containing numerical likert scales addressing various aspects of the customer experience @xcite . one popular question asks customers the likelihood that they will recommend a product or service to others. from these answers, analysts calculate a "net promoter score" (nps) representing the percentage of customers who will recommend a product or service to others minus those who will recommend against it @xcite . additionally, many companies are also interested in tracking product reviews @xcite . collecting, measuring, and analyzing customer feedback is essential to the profitability and long-term success of many companies, but it is prohibitively expensive to survey the entire customer base and even the feedback that a company does receive is often too massive for systematic human evaluation. therefore, it is important to develop effective machine learning models for predicting customer satisfaction and to maintain consistent and accurate methods and metrics for evaluating their performance. an important aspect of modeling customer sentiment is the subjective numerical ranking of the customer response. feedback is often in the form of ranked scales, e.g. rating scales 1-5 or 1-10, or textual "strongly agree," "disagree," etc. crucially, these likert scales are ordinal and should not be confused with scalar values: a rating of 5 or "strongly agree" is not necessarily 5 times greater than a rating of 1 or "strongly disagree" (allen and @xcite . to predict ordinal target variables from textual input, we explored a variety of commonplace and cutting-edge nlp techniques, ranging from linear models such as naive bayes and logistic regression to transformer-based approaches such as bert @xcite and "performer" @xcite . one of the most striking observations from our experiments was that the most impactful experimental variable was not necessarily the model architecture itself, but rather the loss function employed and classification scheme, i.e. target variable encoding. specifically, we highlight our use of ordinal regression (or), which is an approach that is underutilized in the field of nlp. this approach has a long history @xcite @xcite and has recently been applied to deep neural network models predicting ordinal labels @xcite . here, we extend this framework into the domain of nlp and apply it to train transformer models as a novel application of previous work on or and transformers. our results showed that or produced a distribution of predictions significantly closer to ground truth distributions than multi-class classification (mc) for both nps and survey ratings (i.e. lower k-l divergence), while producing similar accuracies and f1. additionally, we found that or improved the correlation between model predictions and ground truth. our findings highlight that common nlp metrics are insufficient to distinguish the better model(s) for our task, which motivated us to explore and identify error-sensitive metrics more consistent and effective for evaluating models with ordinal target variables. ## related work there has been a longstanding debate @xcite within the survey, psychometrics, and crowdsourcing methodology community regarding the use of likert scales. in particular, a debate on whether the level of measurement in question is interval or ordinal @xcite and whether parametric or nonparametric tests should should be used @xcite . in a classification setting involving ordinal data, a related question involves the use of multinomial versus ordinal modeling for such data. intuitively, the inclusion of ordinality in the classification model should improve performance relative to a multinomial approach, as shown in campbell and donner 1989. despite the prevalence of ordinally-scaled tasks in nlp, such as in sentiment analysis @xcite , stance classification @xcite , lexical specificity @xcite , political bias @xcite , and commonsense inference @xcite , approaches to such tasks have tended to ignore the ordinal nature of the data, treating them as mc tasks. for example, only 2 of 11 participants in subtask c of semeval 2016 @xcite (a 5-point-scale twitter sentiment classification task) chose to exploit the ordinal nature of the task in their models. justifying this choice would involve systematically comparing the use of ordinal versus mc models for these tasks, yet most such comparisons have been reported as an incidental part of experiments for tasks such as sentiment analysis in tweets (saad and yang, 2019) and classification of psychiatric symptom severity in clinical notes @xcite . ## results our experiments with mc vs or classifiers for benchmark datasets ( ## discussion we have shown that while mc outperformed or for some datasets in terms of accuracy and f1, or is significantly better than mc for minimizing error between predictions and targets for all datasets, as revealed by error-sensitive metrics such as mean-squared error and spearman rho. this can lead to better performance in terms of representing distributions, as measured by smoothed k-l divergence, and min-imizing the magnitude of errors, as shown by a @ n. we attribute or's superiority over mc on mse and spearman correlation to the difference in their loss functions. while both mc and or utilize cross-entropy, the label encodings and model constraints account for major differences. mc assumes that each rating is independent and uses one-hot encodings. or encodes the ratings as rank thresholds, v, where each index, v i , represents a binary indicator of threshold, i.e. v 1 = 1 if ŷ1 > 0.5 and v 2 = 1 if ŷ2 > 0.5, etc. this encoding combined with shared weight parameters and independent biases enforces rank-consistency, meaning a model prediction of ŷ3 > 0.5 is accompanied by both ŷ2 > 0.5 and ŷ1 > 0.5 because b 1 ≥ b 2 ≥ b 3 @xcite . consequently, we have not observed rank-inconsistent predictions in our or models. this rank-consistency places a constraint on the model, forcing it to learn the ordinal information separating different ratings. in terms of bias-variance tradeoff, mc results in a lower bias, higher variance model, while or produces a higher bias, lower variance model for ordinal target variables. therefore, or optimizes for rank-error between prediction and target, leading to lower mse and higher spearman correlations compared to mc. despite not directly optimizing for raw accuracy, we observe that or still outperforms mc in accuracy and f1 for the mr and specificity datasets. we hypothesize that this is related to the independence assumption that mc places during training, which may omit useful ordinal signal. while likert-like scales can be highly subjective and inconsistent from one reviewer to another @xcite due to cultural differences @xcite , among other reasons, there is significant overlap in rating schemes among reviewers (e.g. all reviewers should agree that higher ratings are better than lower). this standardization is particularly apparent when ratings are consistently generated, as is the case with mr, where all reviews originate from four authors. each author, while unique, has a self-consistent way of expressing their reviews, which allows their readers to understand their reasoning for assigning movie ratings. this intra-author consistency may en-hance the ordinal signal contained in the mr dataset. the specificity dataset provides a related explanation. the ranks in the specificity and mr datasets derive from labeled ratings that were collapsed from a more-continuous scale into 4 ranks. this may help to reduce the variance among the annotators. additionally, the creators of the specificity dataset took care to ensure that annotators agreed with oneanother, assigning multiple annotators to label each data point @xcite . or may also benefit from having fewer ranks, as more ranks create more opportunities for inconsistencies among reviewers as to what each rank represents. the complement to our previous observation is that mc outperforms or in accuracy and f1 for imdb and sst-5. for imdb, we hypothesize that this is related to the granularity of the ranks (8 total classes), and the large number of reviewers (likely tens of thousands). in this case, each person's different notions of ratings introduces considerable noise into the ordinal signal. it seems that or may be more sensitive to rater inconsistencies compared to mc because or fits the model on the ordinal signal whereas mc makes a rank-independence assumption. for sst-5, the answer is less clear, as each sentence is annotated by 3 judges, and the labels are collapsed down to 5-classes from a continuous range obtained from a sliding bar. it is possible that or may struggle to make the generalizations necessary to successfully distinguish different ranks when the input sequences are short, as words may not appear in more than one example. for sst-5, 56% of the words in the training split appear in only one sentence. this may create difficulties in learning to generalize across ranks. it also highlights the impact of pre-training either in the form of language models or word embeddings for performance @xcite . this pitfall is not apparent in the specificity dataset, and we hypothesize that it is because the specificity task has significant correlations with the tweet length itself @xcite , so that learning word associations is less important. in addition to rank-error minimization, or avoids the class dropping problem we observe in mc. we hypothesize that mc drops classes because probability mass is shared among all classes via the softmax activation. this results in a model bias to take probability mass away from minority classes to give to majority classes when it is uncertain, which we observe in mc over-predicting majority classes (figure 2 ). this is likely to happen in cases where the model cannot find reliable signal for particular minority class(es). in the extreme case, this leads to nearly no predictions for those minority classes, i.e. class dropping. or, in contrast, avoids class dropping because its predictions do not share probability mass, i.e. each rank threshold represents its own prediction after sigmoidal activation, without a softmax. this builds inherent robustness into the prediction. for example, if the model is quite confident that a review is higher than 2 and less than 5, i.e. ŷ2 >> 0.5 and ŷ5 << 0.5, but unsure if it is a 3 or a 4, i.e. ŷ3 ≈ 0.5 and ŷ4 ≈ 0.5, rather than splitting probability mass between 3 and 4 as in mc (thus making other classes more likely in relation via softmax), it can adjust the probability mass of v 3 independent of the other thresholds by adjusting b 3 . for inputs near the decision boundary for both v 3 and v 4 , the model will predict 3 and 4 in roughly equal proportions, avoiding a drop of either rank, whereas for mc, the model's bias towards majority classes coupled with softmax activation may lead to dropping 3 or 4. therefore, or may produce better results in tasks with data imbalance, e.g. with highly-skewed or bi-/multi-modal distributions. while our results are only empirically demonstrated for or implemented with coral, we expect that our observations would likely generalize to other or implementations, especially latent-variable models and other variants that produce rank-ordered thresholds, as the salient features would be the same. ## conclusion common model evaluation metrics such as accuracy, precision, recall, and f1 are insufficient for capturing the degree of error between prediction and target for multi-class prediction where the target is ordinal. therefore, selecting models based on these traditional metrics may result in selecting an underperforming model. error-sensitive metrics such as mse (similarly, mae and rmse) and spearman correlation capture the ordinal error, resulting in selecting models that have more representative distributions and improved generalization across data segments, as measured by k-l divergence. this is particularly important in use cases like predictive nps, where accurate scores are necessary not just in aggregate, but across time and different customer segments as well. the independence assumption made by mc can remove useful ordinal signal, especially in cases where there is greater consistency across reviewers, their language, or their ratings. for benchmarks involving ordinal target variables, it is important to evaluate the mse (or a similar error-sensitive metric like mae, a @ n, and m ae m ) and spearman correlations in addition to the usual metrics in determining whether a new model outperforms previous models. we hope to see these metrics included in future benchmarks with ordinal target variables. additionally, some of our experimental observations appear to be tied to the softmax activation used in mc compared to the sigmoidal activation in or, such as class dropping in mc and substantially lower author-level k-l divergence in or. these observations motivate exploration into alternative losses and activations for ordinal problems compared to traditional classification. for example, it would be interesting to compare the mc approach to one that does not involve the softmax, such as "one-vs-rest" or "one-vs-one," to observe whether class dropping persists.