Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2022)

Abstract: Natural language processing researchers have identified limitations of evaluation methodology for generation tasks, with new questions raised about the validity of automatic metrics and of crowdworker judgments. Meanwhile, efforts to improve generation models tend to depend on simple n-gram overlap metrics (e.g., BLEU, ROUGE). We argue that new advances on models and metrics should each more directly benefit and inform the other. We therefore propose a generalization of leaderboards, bidimensional leaderboards (Billboards), that simultaneously tracks progress in language generation models and metrics for their evaluation. Unlike conventional unidimensional leaderboards that sort submitted systems by predetermined metrics, a Billboard accepts both generators and evaluation metrics as competing entries. A Billboard automatically creates an ensemble metric that selects and linearly combines a few metrics based on a global analysis across generators. Further, metrics are ranked based on their correlation with human judgments. We release four Billboards for machine translation, summarization, and image captioning. We demonstrate that a linear ensemble of a few diverse metrics sometimes substantially outperforms existing metrics in isolation. Our mixed-effects model analysis shows that most automatic metrics, especially the reference-based ones, overrate machine over human generation, demonstrating the importance of updating metrics as generation models become stronger (and perhaps more similar to humans) in the future.

## introduction recent modeling advances have led to improved natural language generation in applications such as machine translation and summarization @xcite , when a generator developer submits output text (output.txt), billboard computes all metric scores. when a metric developer submits an executable program (e.g., metric.py), billboard computes correlation with the human judgments, updates the ensemble metric ( §2.2), and measures how much the metric overrates machines ( §2.3). meanwhile, many evaluation metrics that improve correlation with human judgments have been proposed @xcite @xcite , but this progress has yet to be broadly adopted by the community of researchers focused on advancing models. indeed, consistent with prior metaevaluations @xcite , we found that 68% of the machine translation papers from naacl and acl 2021 evaluated their models solely by bleu, and only 5% measured the performance using recent metrics with contextual representations such as comet @xcite . similarly, automatic evaluation in 66% of the summarization papers was done only in terms of rouge. 2 we believe this separation between generation modeling and automatic evaluation represents a missed opportunity for each subcommunity to more rapidly benefit from the advances of the other. we therefore propose an abstraction of conventional leaderboards, bidimensional leaderboards (billboards), that simultaneously facilitates progress in natural language generation and its evaluation (fig. 1 ). a billboard accepts two types of submissions related to a given task and dataset: generators and metrics. unlike conventional leaderboards, model ranking is not tied to a predetermined set of metrics; the generators are ranked based on the metric that currently correlates best with human judgments. metric submissions are ranked by their correlations to human judgments, and each is stored as an executable program, which will then be used to evaluate future generation submissions. our billboard includes a sparse regression that selects and linearly combines three existing metrics, revealing complementary strengths. all leaderboard scores are readily reproducible, allowing research on generation models and automatic metrics to benefit from each other. we release four billboard interfaces ( https://nlp.cs.washington.edu/ billboard/ ) spanning three generation tasks: the wmt20 en-de and wmt20 zh-en machine translation tasks @xcite , the cnndm summarization task @xcite , and the mscoco image captioning task @xcite . ## bidimensional leaderboards we propose billboards to simultaneously drive progress in natural language generation and its evaluation, which are often disconnected in current research. we first describe the general framework ( §2.1) and the automatic analyses they provide ( §2.2-2.3). we then discuss our design choices ( §2.4) and the rubric-based, human judgment data necessary to initialize billboards ( §2.5). ## experiments having established the framework, we set up bill-boards for three natural language generation tasks: machine translation, summarization, and image captioning. to maximize the performance of reference-based metrics, we use as many references as possible for each task. see §4 for an analysis on the effect of varying numbers of references. ## results and analysis here we discuss the current results and make several key observations about the state of language generation evaluation. ensemble metric the rightmost section of removed metric -comet comet-qe bleurt correlation 0.61 0.61 0.57 0.61 mixed-effects models seen in ## conclusion we introduced billboards, a simple yet powerful generalization of leaderboards that bridges the gap between generation modeling and evaluation research. we established and released four bill-boards on machine translation, summarization, and image captioning tasks. we demonstrated that their built-in analysis of metric ensembling and mixed-effects modeling revealed key insights into the current state of natural language generation and its evaluation methods. billboards allow for a standard leaderboard experience both on the modeling and evaluation sides. we invite submissions from researchers through our website.