An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results (2020)

Abstract: In Ordinal Classification tasks, items have to be assigned to classes that have a relative ordering, such as “positive”, “neutral”, “negative” in sentiment analysis. Remarkably, the most popular evaluation metrics for ordinal classification tasks either ignore relevant information (for instance, precision/recall on each of the classes ignores their relative ordering) or assume additional information (for instance, Mean Average Error assumes absolute distances between classes). In this paper we propose a new metric for Ordinal Classification, Closeness Evaluation Measure, that is rooted on Measurement Theory and Information Theory. Our theoretical analysis and experimental results over both synthetic data and data from NLP shared tasks indicate that the proposed metric captures quality aspects from different traditional tasks simultaneously. In addition, it generalizes some popular classification (nominal scale) and error minimization (interval scale) metrics, depending on the measurement scale in which it is instantiated.

## introduction in ordinal classification (oc) tasks, items have to be assigned to classes that have a relative ordering, such as positive, neutral, negative in sentiment analysis. it is different from n-ary classification, because it considers ordinal relationships between classes. it is also different from ranking tasks, which only care about relative ordering between items, because it requires category matching; and it is also different from value prediction, because it does not assume fixed numeric intervals between categories. most research on ordinal classification, however, evaluates systems with metrics designed for those other problems. but classification measures ignore the ordering between classes, ranking metrics ignore category matching, and value prediction metrics are used by assuming (usually equal) numeric intervals between categories. in this paper we propose a metric designed to evaluate ordinal classification systems which relies on concepts from measurement theory and from information theory. the key idea is defining a general notion of closeness between item value assignments (system output prediction vs gold standard class) which is instantiated into ordinal scales but can be also be used with nominal or interval scales. our approach establishes closeness between classes in terms of the distribution of items per class in the gold standard, instead of assuming predefined intervals between classes. we provide a formal (section 4) and empirical (section 5) comparison of our metric with previous approaches, and both analytical and empirical evidence indicate that our metric suits the problem best than the current most popular choices. ## state of the art in this section we first summarize the most popular metrics used in oc evaluation campaigns, and then discuss previous work on oc evaluation. ## theoretical evidence following a methodology previously applied for classification @xcite , clustering @xcite @xcite , and document ranking tasks @xcite , here we define a formal framework for oc via desirable properties to be satisfied, which are illustrated in figure 2 and introduced below. ## empirical study meta-evaluating metrics is not straightforward. a common criterion is robustness, defined as consistence (correlation) of system rankings across data sets. however, although robustness is relevantand we do report it at the end of this section -it does not reflect to what extent a metric captures the quality aspects of systems. as many authors have pointed out, an oc metric should capture diverse aspects of systems: class matching, ordering, and imbalance. in our exper-iments, in addition to robustness, we select three complementary metrics, each focused on one of these partial aspects, and we evaluate to what extent existing oc metrics are able to capture all these aspects simultaneously. the selected metrics are: (i) accuracy, as a partial metric which captures class matching; (ii) kendall's correlation coefficient tau-a (without counting ties), in order to capture class ordering 2 ; and (iii) mutual information (mi), a clustering metric which reflects how much knowing the system output reduces uncertainty about the gold standard values. this metric accentuates the effect of small classes (imbalance property). ## conclusions our findings can be summarized as follows: (i) metrics commonly used for ordinal classification problems are highly heterogeneous and, in general, inconsistent with the notion of ordinal scale in measurement theory; (ii) the notion of closeness between classes can be modelled in terms of measurement theory and information theory and particularized for different scales; and (iii) our proposed ordinal closeness evaluation measure (cem ord ) is the only one that satisfies all desirable formal properties, it is as robust as the best state-of-the-art metrics, and it is the one that better captures the different quality aspects of oc problems in our ex-perimentation, with both synthetic and naturalistic datasets. from a methodological perspective, the evidence that we have presented covers the four approaches pointed out in @xcite : we have compared metrics in terms of desirable formal properties to be satisfied (theoretic top-down), we have generalized existing approaches (theoretic bottomup), and we have compared effectiveness on human assessed and on synthetic data (empirical bottomup and top-down). future work includes the application of cem at scales other than the ordinal.