Evaluating Evaluation Measures for Ordinal Classification and Ordinal Quantification (2021)

Abstract: Ordinal Classification (OC) is an important classification task where the classes are ordinal. For example, an OC task for sentiment analysis could have the following classes: highly positive, positive, neutral, negative, highly negative. Clearly, evaluation measures for an OC task should penalise misclassifications by considering the ordinal nature of the classes. Ordinal Quantification (OQ) is a related task where the gold data is a distribution over ordinal classes, and the system is required to estimate this distribution. Evaluation measures for an OQ task should also take the ordinal nature of the classes into account. However, for both OC and OQ, there are only a small number of known evaluation measures that meet this basic requirement. In the present study, we utilise data from the SemEval and NTCIR communities to clarify the properties of nine evaluation measures in the context of OC tasks, and six measures in the context of OQ tasks.

## introduction in nlp and many other experiment-oriented research disciplines, researchers rely heavily on evaluation measures. whenever we observe an improvement in the score of our favourite measure, we either assume or hope that this implies that we have managed to moved our system a little towards what we ultimately want to achieve. hence it is of utmost importance to examine whether evaluation measures are measuring what we want to measure, and to understand their properties. this paper concerns evaluation measures for ordinal classification (oc) and ordinal quantification (oq) tasks. in an oc task, the classes are ordinal, not nominal. for example, task 4 (sentiment analysis in twitter) subtask c in semeval-2016/2017 is defined as: given a set of tweets about a particular topic, estimate the sentiment conveyed by each tweet towards the topic on a five-point scale (highly negative, negative, neutral, positive, highly positive) @xcite . on the other hand, an oq task involves a gold distribution of labels over ordinal classes and the system's estimated distribution. for example, task 4 subtask e of the semeval-2016/2017 workshops is defined as: given a set of tweets about a particular topic, estimate the distribution of the tweets across the five ordinal classes already mentioned above @xcite . the dialogue breakdown detection challenge @xcite and the dialogue quality subtasks of the ntcir-14 short text conversation @xcite and the ntcir-15 dialogue evaluation @xcite tasks are also oq tasks. 1 clearly, evaluation measures for oc and oq tasks should take the ordinal nature of the classes into account. for example, in oc, when a highly positive item is misclassified as highly negative, that should be penalised more heavily than when it is misclassified as positive. surprisingly, however, there are only a small number of known evaluation measures that meet this requirement. in the present study, we use data from the semeval and ntcir communities to clarify the properties of nine evaluation measures in the context of oc tasks, and six measures in the context of oq tasks. some of these measures satisfy the aforementioned basic requirement for ordinal classes; others do not. section 2 discusses prior art. section 3 provides formal definitions of the measures we examine, as this is of utmost importance for reproducibility. section 4 describes the data we use to evaluate the measures. sections 5 and 6 report on the results on the oc and oq measures, respectively. finally, section 7 concludes this paper. ## task data ## conclusions we conducted extensive evaluations of nine measures in the context of oc tasks and six measures in the context of oq tasks, using data from semeval and ntcir. as we have discussed in sections 5.4 and 6.4, our recommendations are as follows. oc tasks use (linear weighted) κ as the primary measure if the task does not involve multiple runs that always return the same class (e.g., one that always returns class 1, another that always returns class 2, etc.). otherwise, use α-ord (i.e., krippendorff's α for ordinal classes) as the primary measure. in either case, use both measures. oq tasks use rnod as the primary measure, and nmd as a secondary measure. all of our evaluation measure score matrices are available from https://waseda.box.com/ acl2021packocoq , to help researchers reproduce our work. among the above recommended measures, recall that linear weighted κ and rnod assume equidistance (i.e., they rely on w @xmath0, while α-ord and nmd do not. hence, if researchers want to avoid relying on the equidistance assumption (i.e., satisfy the ordinal invariance property @xcite ), α-ord can be used for oc tasks and nmd can be used for oq tasks. however, we do not see relying on equidistance as a practical problem. for example, note that the linear weighted κ is just an instance of the weighted κ family: if necessary, the weight w ij can be set for each pair of classes i and j according to practical needs. similarly, w @xmath1. 12) for rnod (and other equidistance-based measures) may be replaced with a different weighting scheme (e.g., something similar to the prox ij weights of cem ord ) if need be. our final and general remark is that it is of utmost importance for researchers to understand the properties of evaluation measures and ensure that they are appropriate for a given task. our future work includes evaluating and understanding evaluation measures for tasks other than oc and oq.