Why only Micro-F1? Class Weighting of Measures for Relation Classification (2022)

Abstract: Relation classification models are conventionally evaluated using only a single measure, e.g., micro-F1, macro-F1 or AUC. In this work, we analyze weighting schemes, such as micro and macro, for imbalanced datasets. We introduce a framework for weighting schemes, where existing schemes are extremes, and two new intermediate schemes. We show that reporting results of different weighting schemes better highlights strengths and weaknesses of a model.

## introduction relation classification (rc) models are typically compared with either micro-f 1 or macro-f 1 , often without discussing the measure's properties (see e.g. @xcite . each measure highlights different aspects of model performance @xcite . however, using an inappropriate measure can lead to the preference of an unsuitable model @xcite , e.g., tasks with an imbalanced or long-tailed class distribution. we argue that model evaluation should better reflect this, particularly as rare phenomena become more important in nlp @xcite . for instance, popular datasets for rc, such as tacred @xcite , nyt @xcite , chemprot @xcite , do-cred @xcite , and semeval-2010 task 8 @xcite , often exhibit a highly imbalanced label distribution (see in this paper, we introduce a framework for weighting schemes of measures to address these evaluation deficits. we present and motivate two new weighting schemes that are in between the extremes of micro-and macro-weighting. we demonstrate these, micro-, class-weighted-and macro-f 1 on tacred and semeval with two popular models each. we show that more information about models can be inferred from our results and point out what further steps should be taken to improve evaluation in relation classification. ## methods we first give background on the f 1 -score and existing f 1 weighting schemes. we present our framework of weighting schemes. we introduce two new weighting schemes. finally, we outline statistical tests. ## experiments we evaluate and compare three rc methods with our proposed measures on two datasets. we choose these methods, as recent @xcite and bert em @xcite are based on vanilla fine-tuning of a pre-trained language model, with a classification head on top. ptr @xcite recent proposes a model-agnostic paradigm that exploits entity types to narrow down the candidate relations. given an entity-type combination, a separate classifier is trained on the restricted classes. baldini @xcite compare various strategies that extract relation representation from transformers and claim entity start (i.e. insert entity markers at the start of two entity mentions) yields the best performance. ptr also takes entity types into consideration and constructs prompts composed of three subprompts, two corresponding to the fill-in of the entity types and one predicting the relation. in our experiments we use recent gcn for re-cent, bert em with entity start, and unreversed prompts for ptr. we use the official repositories for recent and ptr, we reimplement bert em 4 . we use the hyperparameters proposed in the original papers and conduct five runs for each model. additional implementation and training details can be found in appendices a and b. the main focus is unearthing performance information about these methods that was previously 4 our reimplementation is available at https:// github.com/dfki-nlp/mtb-bert-em . ## outlook we suggest creating and using a bidimensional leaderboard like @xcite where measures and models can be contributed. to this end, benchmarking of rc models could be done on a centralized site where a model or test set predictions are submitted and measures are calculated automatically through a script. for measures that modify weighting of classes and intra-class scoring, this does not require additional training computation. due to the reproducibility crisis @xcite , not all state-of-the-art scores can be replicated. possible future work includes a comprehensive evaluation study of papers on leaderboards of rc tasks. this would enable an in-depth discussion of strength and weaknesses (including reproducibil-ity) of these models. the analysis we present can also be extended to other nlp tasks with imbalanced datasets, such as named entity recognition @xcite , part-of-speech tagging @xcite and coreference resolution @xcite . ## conclusion we criticise the current practice of reporting a single score when evaluating imbalanced rc datasets. we propose a new framework to weight scores for multi-class evaluation of imbalanced datasets. we provide two new weighting schemes, dodrans and entropy, which are positioned between classweighted and macro. in our experiments, we show that model performance on both tacred and semeval, especially on the long-tail relations, is not adequately captured by a single score. thus, we advocate the use of multiple weighing schemes when reporting model performance on imbalanced datasets.