Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons (2022)

Abstract: Recent studies have shown the advantages of evaluating NLG systems using pairwise comparisons as opposed to direct assessment. Given k systems, a naive approach for identifying the top-ranked system would be to uniformly obtain pairwise comparisons from all {k \choose 2} pairs of systems. However, this can be very expensive as the number of human annotations required would grow quadratically with k . In this work, we introduce Active Evaluation, a framework to efficiently identify the top-ranked system by actively choosing system pairs for comparison using dueling bandit algorithms. We perform extensive experiments with 13 dueling bandits algorithms on 13 NLG evaluation datasets spanning 5 tasks and show that the number of human annotations can be reduced by 80%. To further reduce the number of human annotations, we propose model-based dueling bandit algorithms which combine automatic evaluation metrics with human evaluations. Specifically, we eliminate sub-optimal systems even before the human annotation process and perform human evaluations only on test examples where the automatic metric is highly uncertain. This reduces the number of human annotations required further by 89%. In effect, we show that identifying the top-ranked system requires only a few hundred human annotations, which grow linearly with k . Lastly, we provide practical recommendations and best practices to identify the top-ranked system efficiently. Our code has been made publicly available at https://github.com/akashkm99/duelnlg

## introduction in the last few years, the field of nlg has made rapid progress with the advent of large-scale models trained on massive amounts of data @xcite @xcite . however, evaluation of nlg systems continues to be a challenge. on the one hand, we have automatic evaluation metrics which are easy to compute but unreliable. in particular, many studies have shown that they do not correlate well with human judgments @xcite @xcite . on the other hand, we have human evaluations, which are relatively more reliable but tedious, expensive, and time-consuming. further, recent studies have highlighted some limitations of human evaluations that involve direct assessment on an absolute scale, e.g., likert scale. specifically, human evaluations using direct assessment have been shown to suffer from annotator bias, high variance and sequence effects where the annotation of one item is influenced by preceding items @xcite @xcite @xcite . in this work, we focus on reducing the cost and time required for human evaluations while not compromising on reliability. we take motivation from studies which show that selecting the better of two options is much easier for human annotators than providing an absolute score, which requires annotators to maintain a consistent standard across samples @xcite . in particular, recent works show that ranking nlg systems using pairwise comparisons is a more reliable alternative than using direct assessment @xcite @xcite . while this is promising, a naive approach for identifying the top-ranked system from a set of k systems using uniform exploration is prohibitively expensive. specifically, uniform exploration obtains an equal number of annotations for all the k 2 system pairs; as a result, the required human annotations grows as o(k 2 ). to reduce the number of pairwise annotations, we introduce active evaluation, a framework to efficiently identify the top-ranked nlg system. our active evaluation framework consists of a learner that selects a pair of systems to compare at each time step. the learner, then, receives a feedback signal indicating the (human) preference between the selected systems on one input context, randomly sampled from the test dataset. the learner's objective is to reliably compute the topranked system with as few human annotations as possible. we adopt algorithms from the stochastic dueling bandits literature @xcite to decide which pair of nlg systems to compare at each time step. to check if existing dueling bandits algorithms can indeed provide reliable top-rank estimates with minimal annotations, we evaluate 13 such algorithms on 13 nlg evaluation datasets spanning five tasks viz., machine translation, summarization, data-to-text generation, paraphrase generation, and grammatical error correction. we show that the best performing dueling bandit algorithm can reduce the number of human annotations by 80% when compared to uniform exploration. to further reduce human annotations, we leverage automatic evaluation metrics in our active evaluation framework. we utilize existing automatic metrics such as bleu @xcite , bertscore @xcite , etc for pairwise evaluations by converting the direct evaluation scores into preference probabilities using pairwise probability models. we also develop trained pairwise metrics that directly predict the comparison outcome given pairs of generated texts and context or reference as input. to incorporate such evaluation metrics in our active evaluation framework, we propose three model-based dueling bandits algorithms, viz., (i) random mixing: human annotations and evaluation metric predictions are randomly mixed, (ii) uncertainty-aware selection: human annotations are obtained only when the predictions from the evaluation metric is highly uncertain, (iii) ucb elimination: poorly performing nlg systems are eliminated using an upper confidence bound (ucb) on the evaluation metric scores. through our experiments, we show that the number of human annotations can be further reduced by 89% on average (this reduction is over and above the 80% reduction that we got earlier). in effect, we show that given k systems, we can find the top-ranked nlg system efficiently with just a few hundred comparisons that vary as o(k). lastly, we provide practical recommendations to efficiently identify the top-ranked nlg system based on our empirical study on various design choices and hyperparameters. ## active evaluation framework we introduce the problem and our active evaluation setup in section 2.1. later in section 2.2, we describe the different approaches to decide which pairs of nlg systems to compare at each time step. finally, in section 2.3, we formalize the notion of top-ranked system. ## pairwise probability models our active evaluation framework, which we described in the previous section, completely relied on human annotators to compare pairs of generated texts (y 1 , y 2 ) to provide the preference feedback w. we can further reduce the number of required human annotations by estimating the human preference feedback using automatic evaluation metrics. however, most existing evaluation metrics are designed for direct assessment and not directly suitable for pairwise evaluations. in this section, we de-scribe three pairwise probability models to convert direct evaluation scores into pairwise preference probabilities. let f (y ) denote the score provided by a direct assessment metric f to a generated text y (the dependence of f on the reference/context is omitted for brevity). the pairwise preference probability p(y 1 ≻ y 2 ) between any two hypotheses y 1 and y 2 can be modeled in 3 different ways: we choose τ 1 and τ 2 using grid search on the validation set. refer appendix c.2 for more details. ## model-based dueling bandits in the previous section, we discussed pairwise probability models to obtain the estimated preference probability p(y 1 ≻ y 2 ) and the comparison outcome ŵ using scores assigned by direct assessment metrics. we now propose three model-based dueling bandit algorithms wherein we combine such predictions from evaluation metrics with human annotations in the active evaluation framework. ## experimental setup in this section, we describe the (i) nlg tasks and datasets used in our experiments, (ii) automatic evaluation metrics used in our model-based algorithms, and (iii) annotation complexity measure used for comparing dueling bandit algorithms. ## results & discussion we discuss the performance of dueling bandits algorithms in 6.1, automatic metrics in 6.2 and our proposed model-based algorithms in 6.3. lastly in 6.4, we analyze the variation of annotation complexity with the number of nlg system. ## practical recommendations we summarize the key insights from this study and provide practical recommendations on efficiently identifying the top-ranked nlg system. ## related work several works @xcite @xcite in machine translation and grammatical error correction adopt the trueskill algorithm @xcite , originally used for ranking xbox gamers, to efficiently rank nlg systems from pairwise annotations. a recent work @xcite proposes an online algorithm to rank nlg systems when we receive pairwise preference feedback in the form of a continuous scalar with bounded support. the key difference in our work is that we focus on the problem of identifying the top-rank system instead of ranking all the systems. experimental study of dueling bandit algorithms have been limited to synthetic simulations in a few works @xcite . most others @xcite @xcite focus on information retrieval applications that involve evaluating search retrieval algorithms @xcite . to the best of our knowledge, ours is the first work to extensively study the effectiveness of dueling bandit algorithms for nlg evaluation. ## conclusion & future work in this work, we focused on the problem of identifying the top-ranked nlg system with few pairwise annotations. we formulated this problem in an active evaluation framework and showed that dueling bandit algorithms can reduce the number of human annotations by 80%. we then proposed modelbased algorithms to combine automatic metrics with human evaluations and showed that human annotations can be reduced further by 89%; thereby requiring only a few hundred human annotations to identify the top-ranked system. in future work, we would like to extend our analysis to the general problem of finding the top-k ranked systems.