A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice (2024)

Abstract: Classification systems are evaluated in a countless number of papers. However, we find that evaluation practice is often nebulous. Frequently, metrics are selected without arguments, and blurry terminology invites misconceptions. For instance, many works use so-called ‘macro’ metrics to rank systems (e.g., ‘macro F1’) but do not clearly specify what they would expect from such a ‘macro’ metric. This is problematic, since picking a metric can affect research findings and thus any clarity in the process should be maximized. Starting from the intuitive concepts of bias and prevalence, we perform an analysis of common evaluation metrics. The analysis helps us understand the metrics’ underlying properties, and how they align with expectations as found expressed in papers. Then we reflect on the practical situation in the field, and survey evaluation practice in recent shared tasks. We find that metric selection is often not supported with convincing arguments, an issue that can make a system ranking seem arbitrary. Our work aims at providing overview and guidance for more informed and transparent metric selection, fostering meaningful evaluation.

## introduction classification evaluation is ubiquitous: we have a system that predicts some classes of interest (aka classifier) and want to assess its prediction skill. we study a widespread and seemingly clear-cut setup of multi-class evaluation, where we compare a classifier's predictions against reference labels in two steps. first, we construct a confusion matrix that has a designated dimension for each possible prediction/label combination. second, an aggregate statistic, which we denote as metric, summarizes the confusion matrix as a single number. already from this it would follow that 'the perfect' metric can't exist, since important infor-mation is bound to get lost when reducing the confusion matrix to a single dimension. still, we require a metric to rank and select classifiers, and thus it should characterize a classifier's 'skill' or 'performance' as well as possible. but exactly what is 'performance' and how should we measure it? such questions do not seem to arise (as much) in other 'performance' measurements that are known to humans. for example, a marathon's result derives from a clear and broadly accepted criterion (time over distance) that can be measured with validated instruments (a clock). however, in machine learning, criterion and instrument are often less clear and lie entangled in the term 'metric'. since metric selection can influence which system we consider better or worse in a task, one would think that metrics are selected with great care. but when searching through papers for reasons that would support a particular metric choice, we mostly (at best) find only weak or indirect arguments. for example, it is observed ( however, what is meant with phrases like 'imbalanced data' or 'macro' is rarely made explicit, and how the metrics that are then selected in this context are actually addressing a perceived 'imbalance' is unclear. according to a word etymology, evaluation with a 'macro' metric may involve the expectation that we are told a bigger picture of classifier capability (greek: makrós, 'long/large'), whereas a smaller picture (greek: mikrós, 'small') would perhaps bind the assessment to a more local context. regardless of such metric cited motivation/argument macro f1 ''macro-averaging (...) implies that all class labels have equal weight in the final score'' macro p/r/f1 ''because (...) skewed distribution of the label set'' macro f1 paper outline. after introducing preliminaries ( §2) and five metric properties ( §3), we conduct a thorough metric analysis ( §4) of common classification measures: 'accuracy' ( §4.1), 'macro recall and precision' ( §4.2, §4.3), two different 'macro f1' ( §4.4, §4.5), 'weighted f1' ( §4.6), as well as 'kappa' and 'matthews correlation coefficient (mcc)' in §4.7. we then show how to create simple but meaningful metric variants ( §5). we wrap up the theoretical part with a discussion ( §6) that includes a short summary ( §6.1) of our main analysis results in ## preliminaries we introduce a set of intuitive concepts as a basis. classifier, confusion matrix, and metric. for any classifier f : @xmath0, . . . , n} and finite set s ⊆ d × c, let m f,@xmath1,s @xmath2. 2 we omit superscripts whenever possible. so generally, m i,j contains the mass of events where the classifier predicts i, and the true label is j. that is, on the diagonal of the matrix lies the mass of correct predictions, and all other fields indicate the mass of specific errors. a metric : r n×@xmath3, 1] then allows us to order confusion matrices, respectively, rank classifiers (bounds are chosen for convenience). we say that (for a data set s) a classifier f is better than (or preferable to) a classifier g iff metric(m f,s ) > metric(m g,s ), i.e., a higher score indicates a better classifier. let us now define five basic quantities: class precision. p i is the precision for class i: it approximates the probability of observing a correct class given a specific prediction. class recall. r i denotes the recall for class i, ## defining metric properties to understand and distinguish metrics in more precise ways, we define five metric properties: monotonicity, class sensitivity, class decomposability, prevalence invariance, and chance correction. ## metric property analysis equipped with the appropriate tools, we are now ready to start the analysis of classification metrics. we will study 'accuracy', 'macro recall', 'macro precision', 'macro f1', 'weighted f1', 'kappa', and 'matthews correlation coefficient' (mcc). ## recommendations overall, we would like to refrain from making bold statements as to which metric is 'better', since different contexts may easily call for different metrics. still, from our analyses, we can synthesize some general recommendations: other classification evaluation methods. these can be required when class labels reside on a nominal 'likert' scale @xcite , or in a hierarchy @xcite , or they are ambiguous and need be matched (e.g., 'none'/'null'/'other') across annotation styles @xcite , or their number is unknown @xcite . classifiers are also evaluated with p-r curves @xcite or a receiver-operator characteristic @xcite . the checklist (ribeiro et al., 2020) proposes behavioral testing of classifiers, and the neatclass workshop series @xcite is an effort to find novel ways of evaluation. ## conclusion starting from a definition of the two basic and intuitive concepts of classifier bias and class prevalence, we examined common classification evaluation metrics, resolving unclear expectations such as those that pursue a 'balance' through 'macro' metrics. our metric analysis framework, including definitions and properties, can aid in the study of other or new metrics. a main goal of our work is to provide guidance for more informed decision making in metric selection.