Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing (2020)

Abstract: Current standards for designing and reporting human evaluations in NLP mean it is generally unclear which evaluations are comparable and can be expected to yield similar results when applied to the same system outputs. This has serious implications for reproducibility testing and meta-evaluation, in particular given that human evaluation is considered the gold standard against which the trustworthiness of automatic metrics is gauged. %and merging others, as well as deciding which evaluations should be able to reproduce each other’s results. Using examples from NLG, we propose a classification system for evaluations based on disentangling (i) what is being evaluated (which aspect of quality), and (ii) how it is evaluated in specific (a) evaluation modes and (b) experimental designs. We show that this approach provides a basis for determining comparability, hence for comparison of evaluations across papers, meta-evaluation experiments, reproducibility testing.

## introduction human evaluations play a central role in natural language generation (nlg), a field which has always been wary of automatic evaluation metrics and their limitations @xcite @xcite . nlg has trusted human evaluations perhaps more than any other nlp subfield, and has always gauged the trustworthiness of automatic evaluation metrics in terms of how well, and how consistently, they correlate with human evaluation scores @xcite @xcite @xcite @xcite . if they do not, even in isolated cases, the reliability of the metric is seen as doubtful, regardless of the quality of the human evaluation, or whether the metric and human evaluation involved aimed to assess the same thing. more generalised conclusions are sometimes drawn, for example that bleu scores do not correlate well with human judgements of specific quality criteria foot_0 such as 'fluency,' 'naturalness,' 'readability' or 'overall quality' foot_1 in the general case @xcite @xcite @xcite @xcite . however, such comments make the assumption that, and only really make sense if, multiple evaluations of, say, 'fluency' do in fact assess the same aspect of quality in the output texts. we argue that we do not currently have a way of establishing whether any two evaluations, metric or human, do or do not assess the same thing. in fact, we have plenty of evidence (section 2) that in many cases, when two evaluations use the same name for a quality criterion, they do in fact assess different aspects of quality, even for seemingly straightforward criteria like 'fluency' and 'readability.' and conversely, evaluations that do use different terms often assess identical aspects of quality. in this situation, not only are we on shaky ground when drawing conclusions from meta-evaluations of metrics via correlations with human evaluations, but not knowing when two different evaluations should produce the same results also has clear implications for reproducibility assessments. in this paper, we propose a classification system that disentangles the properties of evaluation methods, providing a basis for establishing comparability. we start with issues in how human evaluations are currently designed and reported in nlg (section 2). we then discuss the difficulties of disentangling the properties of evaluation methods (section 3), and present the proposed classification system consisting of three quality-criterion properties, three evaluation modes, and 12 experimental design properties (section 4). next we demonstrate how these combine to form a classification system that supports comparability (section 5), and show how the system can be used in the context of de-signing and reporting evaluations, meta-evaluations and reproducibility testing (section 6). we finish with some discussion and conclusions (section 7). ## issues in comparing human evaluations in nlg human evaluations in nlg currently paint a confused picture foot_2 with very poor standards for designing and reporting evaluations (van der @xcite . in this section we focus on those aspects that make it hard to compare different evaluations. ## example classifications in the papers in actual fact assess distinct aspects of quality, as do the four criteria called coherence. the third example of fluency in this group in fact assesses three distinct aspects of quality, which is likely to place a high cognitive load on evaluators. the three evaluation measures in group 2 have identical classifications but different names. based on our classifications and information in the original paper, these criteria are not actually distinct. the four evaluation measures in group 3 present a similar case, with reading time and ease of reading on the one hand, and task success and usefulness on the other. however, here the evaluation modes are different within each pair. group 4 has two examples of feature-type criteria with different names but the same qualitycriterion classification; evaluation modes are different, with one involving system rankings (relative mode), and the other direct ratings (absolute mode). the names used (text complexity and simplicity) indicate two ends of the same scale, either one of which may be preferable depending on context. the evaluation measures in group 5 involve quality criteria that appear at first glance closely related (see ## use cases the proposed classification system provides a basis for systematically comparing evaluation methods. we can see at least three contexts in which this is either a prerequisite or at least useful, as outlined in the next three subsections. ## future work and conclusions the present paper is intended as a step towards full comparability of human evaluation methods in nlg. there are clear directions for further development. e.g. we have remained agnostic about what happens within the 27 groups of quality criteria defined by the proposed system (visualised in figure 1 ). do the groups map to 27 quality criteria that are enough for all evaluation contexts, merely needing to be 'localised' to a specific task and domain? this might work for correctness criteria and goodness criteria, but new criteria can be almost arbitrarily added to the feature-type groups. another question is how to ensure that experimental design matches a chosen quality criterion and does not end up evaluating something else entirely. we have pointed to using the terms and definitions of the proposed classification properties in experimental design, but not given details of how this can be done. we can see relevance also to recent machine-learned evaluation metrics (to clarify what it is they are emulating). we plan to address the above lines of inquiry in future work. while this paper proposes a standard way of classifying evaluation methods, we do not propose a standardised nomenclature of quality criterion names and definitions. if such a standard did become widely adopted in the field, it would go a long way towards addressing the issue of comparability. however, given the deeply ingrained habit in nlg of using ad-hoc, tailored evaluation methods that differ widely even within small nlg subfields, this seems unrealistic for now. our aim in this paper has instead been to find a way of teasing apart the similarities and dissimilarities of evaluation methods used in the current, highly diverse context, to yield a set of clearly defined properties that provides a firm basis for designing and reporting evaluation methods, establishing comparability for meta-evaluation, and specifying repeatability/reproducibility conditions for reproducibility tests.