Macro-Average: Rare Types Are Important Too (2021)

Abstract: While traditional corpus-level evaluation metrics for machine translation (MT) correlate well with fluency, they struggle to reflect adequacy. Model-based MT metrics trained on segment-level human judgments have emerged as an attractive replacement due to strong correlation results. These models, however, require potentially expensive re-training for new domains and languages. Furthermore, their decisions are inherently non-transparent and appear to reflect unwelcome biases. We explore the simple type-based classifier metric, MacroF1, and study its applicability to MT evaluation. We find that MacroF1 is competitive on direct assessment, and outperforms others in indicating downstream cross-lingual information retrieval task performance. Further, we show that MacroF1 can be used to effectively compare supervised and unsupervised neural machine translation, and reveal significant qualitative differences in the methods’ outputs.

## introduction model-based metrics for evaluating machine translation such as bleurt @xcite , esim @xcite , and yisi (lo, 2019) have recently attracted attention due to their superior correlation with human judgments @xcite . however, bleu @xcite remains the most widely used corpus-level mt metric. it correlates reasonably well with human judgments, and moreover is easy to understand and cheap to calculate, requiring only reference translations in the target language. by contrast, model-based metrics require tuning on thousands of examples of human evaluation for every new target language or domain macroavg-naacl21. @xcite . model-based metric scores are also opaque and can hide undesirable biases, as can be seen in reference: you must be a doctor. hypothesis: must be a doctor. he -0.735 joe -0.975 sue -1.043 she -1.100 reference: it is the greatest country in the world. hypothesis: the source of model-based metrics' (e.g. bleurt) correlative superiority over model-free metrics (e.g. bleu) appears to be the former's ability to focus evaluation on adequacy, while the latter are overly focused on fluency. bleu and most other generation metrics consider each output token equally. since natural language is dominated by a few high-count types, an mt model that concentrates on getting its if s, ands and buts right will benefit from bleu in the long run more than one that gets its xylophones, peripatetics, and defenestrates right. can we derive a metric with the discriminating power of bleurt that does not share its bias or expense and is as interpretable as bleu? as it turns out, the metric may already exist and be in common use. information extraction and other areas concerned with classification have long used both micro averaging, which treats each token equally, and macro averaging, which instead treats each type equally, when evaluating. the latter in particular is useful when seeking to avoid results dominated by overly frequent types. in this work we take a classification-based approach to evaluating machine translation in order to obtain an easy-to-calculate metric that focuses on adequacy as much as bleurt but does not have the expensive overhead, opacity, or bias of model-based methods. our contributions are as follows: we consider mt as a classification task, and thus admit macrof 1 as a legitimate approach to evaluation (section 2). we show that macrof 1 is competitive with other popular methods at tracking human judgments in translation (section 3.2). we offer an additional justification of macrof 1 as a performance indicator on adequacy-focused downstream tasks such as cross-lingual information retrieval (section 3.3). finally, we demonstrate that macrof 1 is just as good as the expensive bleurt at discriminating between structurally different mt approaches in a way bleu cannot, especially regarding the adequacy of generated text, and provide a novel approach to qualitative analysis of the effect of metrics choice on quantitative evaluation (section 4). ## nmt as classification neural machine translation (nmt) models are often viewed as pairs of encoder-decoder networks. viewing nmt as such is useful in practice for implementation; however, such a view is inadequate for theoretical analysis. gowda and may (2020) provide a high-level view of nmt as two fundamental ml components: an autoregressor and a classifier. specifically, nmt is viewed as a multi-class classifier that operates on representations from an autoregressor. we may thus consider classifier-based evaluation metrics. consider a test corpus, @xmath0,h (i) ,y (i) ) @xmath1,2,3...m} where x (i) , h (i) , and y (i) are source, system hypothesis, and reference translation, respectively. let @xmath2. let v h ,v y ,v h∩y , and v be the vocabulary of h, the vocabulary of y, v h ∩v y , and v h ∪v y , respectively. for each class @xmath3, preds(c) = m @xmath4,h (i) ) in the following sections, we verify and justify the utility of macrof 1 while also offering a comparison with popular alternatives such as microf 1 , bleu, chrf 1 , and bleurt. 4 we use kendall's rank correlation coefficient, τ, to compute the association between metrics and human judgments. correlations with p-values smaller than α = 0.05 are considered to be statistically significant. ## discussion and conclusion we have evaluated nlg in general and mt specifically as a multi-class classifier, and illustrated the differences between micro-and macro-averages using microf 1 and macrof 1 as examples (section 2). macrof 1 captures semantic adequacy better than microf 1 (section 3.1). bleu, being a micro-averaged measure, served well in an era when generating fluent text was at least as difficult as generating adequate text. since we are now in an era in which fluency is taken for granted and seman-tic adequacy is a key discriminating factor, macroaveraged measures such as macrof 1 are better at judging the generation quality of mt models (section 3.2). we have found that another popular metric, chrf 1 , also performs well on direct assessment, however, being an implicitly micro-averaged measure, it does not perform as well as macrof 1 on downstream clir tasks (section 3.3.1). unlike bleurt, which is also adequacy-oriented, macrof 1 is directly interpretable, does not require retuning on expensive human evaluations when changing language or domain, and does not appear to have uncontrollable biases resulting from data effects. it is both easy to understand and to calculate, and is inspectable, enabling fine-grained analysis at the level of individual word types. these attributes make it a useful metric for understanding and addressing the flaws of current models. for instance, we have used macrof 1 to compare supervised and unsupervised nmt models at the same operating point measured in bleu, and determined that supervised models have better adequacy than the current unsupervised models (section 4). macro-average is a useful technique for addressing the importance of the long tail of language, and macrof 1 is our first step in that direction; we anticipate the development of more advanced macroaveraged metrics that take advantage of higherorder and character n-grams in the future.