The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021)

Abstract: We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.

## introduction natural language generation is the task to automatically generate understandable texts, typically using a non-linguistic or textual representation of information as input @xcite . these texts aim to fulfill an underlying communicative goal (e.g., to produce a summary of an article) while remaining faithful to the input information, fluent, grammatical, and natural-looking. an nlg system needs to be robust to shifts in the data distribution and be able to produce text in many different languages. finally, it is often desired that repeated interactions with the model produce diverse outputs, for example, to explain concepts in multiple ways or to become a more interesting conversational agent. these optimization objectives can often be conflicting @xcite and, as a result, evaluations that focus only on a single aspect may fail to recognize the drawbacks of a particular method. to demonstrate this trade-off, consider an improvement on the cnn-dm summarization dataset @xcite measured by the rouge-l met-ric @xcite . since rouge only tests the extent to which a generated summary has a lexical overlap with a reference summary, it can erroneously produce high scores for fluent, yet meaningless and unfaithful outputs as long as many of the same words are used @xcite . moreover, rouge tends to favor systems that produce longer summaries @xcite . it is thus crucial to carefully assess the progress of nlg toward all of its goals at the same time in ways that evolve alongside the models. this is currently not the case; new models are evaluated on different datasets, most of which focus only on the english language @xcite , and using these flawed metrics. moreover, while human evaluations of generated texts can provide complementary insights to automatic evaluation @xcite , it can also lead to contradicting results since studies often omit crucial replication details and assume different definitions of the measured quantities @xcite . we propose a living benchmark called gem (generation, evaluation, and metrics) that aims to enable research on a wide range of nlg challenges. to avoid the fallacy of encouraging hill climbing on a leaderboard (linzen, 2020), gem focuses on an in-depth evaluation of model outputs across human and automatic evaluation that aims to uncover shortcomings and opportunities for progress. as datasets, metrics, and models improve, the benchmark environment will improve as well, replacing "solved" tasks with more challenging ones, incorporating newly developed metrics, and addressing discovered flaws in the experimental setup, as demonstrated in figure 1 . making all model outputs available under an open-source license will support evaluation research and integrating new metrics will, in turn, help their adoption and increase the robustness of model evaluations. the initial set of eleven included datasets is presented in improving data improving metrics improving models consistent human eval evaluation on "solved" data evaluation with gameable metrics varying experimental setups non-repeatable human evaluation figure 1 : the opportunities of living benchmarks and pitfalls of evaluation. as models improve, we need consistent evaluations such that models can be compared to each other. this can only happen if we develop robust human evaluation standards and improve our automated metrics. otherwise, results are challenging to interpret and compare to each other. finally, as models improve and metrics saturate, we need to evaluate them on more challenging datasets instead of continuing to move sideways on old ones. gem aims to provide this environment for natural language generation. ## benchmarks in nlg in this section, we summarize common criticisms of benchmarks in nlp, discuss how they apply to nlg, and how we plan to address them. then, we describe opportunities that gem can provide. nlp benchmarks such as glue @xcite are common for natural language understanding dataset communicative goal language(s) size input type commongen (lin et al., 2020) produce a likely sentence which mentions all of the source concepts. en 67k concept set czech restaurant (dušek and jurčíček, 2019) produce a text expressing the given intent and covering the specified attributes. cs 5k meaning representation dart (radev et al., 2020) describe cells in a table, covering all information provided in triples. en 82k triple set e2e clean (novikova et al., 2017) (dušek et al., 2019) describe a restaurant, given all and only the attributes specified on the input. en 42k meaning representation mlsum (scialom et al., 2020) summarize relevant points within a news article *de/es *520k articles schema-guided dialog (rastogi et al., 2020) provide the surface realization for a virtual assistant en *165k dialog act totto (parikh et al., 2020) produce an english sentence that describes the highlighted cells in the context of the given table. en 136k highlighted table xsum (narayan et al., 2018) highlight relevant points in a news article en *25k articles webnlg (gardent et al., 2017) produce a text that verbalises the input triples in a grammatical and natural way. en/ru 50k rdf triple wikiauto + turk/asset (jiang et al., 2020) (xu et al., 2016) (alva-manchego et al., 2020) communicate the same information as the source sentence using simpler words and grammar. en 594k sentence wikilingua (ladhak et al., 2020) produce high quality summaries of an instructional article. *ar/cs/de/en es/fr/hi/id/it ja/ko/nl/pt/ru th/tr/vi/zh *550k article however, they also pose a risk that progress is reduced to the single number shown in a benchmark's leaderboard and thus may encourage blindly optimizing it without regard to other considerations like model size or fairness @xcite . this is especially challenging for benchmarks in nlg since, as discussed above, the performance cannot be described through a single metric and it is often not clear what metric to optimize for. this shortfall can be seen in benchmarks like decanlp @xcite and glge @xcite which include nlg tasks but focus only on a single metric and, as a result, may mischaracterize a system's performance. moreover, an easy-to-use data infrastructure also disincentivizes researchers from interacting with and conducting in-depth analyses of the data sets that models are trained on. the limited analysis delegates the responsibility to ensure that all included datasets have been collected fairly to the creators of the benchmark @xcite . the dataset and benchmark creators thus must provide in-depth statements that describe the data characteristics and surface potential issues and consider these issues when selecting datasets for a benchmark @xcite . these dangers emphasize selecting datasets for a benchmark needs to be carefully done, that the setup has to remain flexible to be able to address newly found limitations, and that the benchmark should focus on climbing a leaderboard. instead, a living benchmark that can adjust its datasets and specific evaluation metrics can be much more powerful and long-lived. this can, for example, be seen in dynabench, 1 @xcite which has a static evaluation, but interactively adds more test data through a human-in-the-loop approach. ## dataset selection as highlighted in figure 1 , the selection of included datasets is an integral part of a benchmark. they should be challenging for models, but it should still be possible to evaluate models trained on them. moreover, the datasets should cover a wide range of relevant generation challenges that allow for findings to be as general as possible. finally, the datasets should cover tasks that are interesting for contributors to work on to facilitate the wide adoption of the benchmark. to collect datasets with those desired properties, the selection methodology for gem is composed of three steps. first, we elicited a set of proposals from everyone involved in the effort. second, we identified criteria for the selection. third, all gem members voted on individual dataset and criteria utilities. the final selection maximizes the utility under constrained resources, similar to a knapsack solver. 4 this can be seen as an extension of the selection process of superglue @xcite that had similar first and second steps but made the final decision based on which were harder for a baseline model to solve after identifying a final set of candidate datasets. since we are going to introduce challenge sets, the baseline performance of models on a dataset matters less. ## experimental setup since the gem test sets and final metrics selection have not been released yet, we describe an experimental setup that will ensure that participating models are trained correctly and evaluated on publicly available data with available metrics that will give a sufficient indication of a model's performance. to do this, we are reporting the results of the baseline models on the validation sets. ## results one of the central aims of gem is to measure the progress in nlg without misrepresenting the complex interactions between the sometimes contradicting measures. we thus will not distill the complex interplay of the data, metrics, and model outputs into a single number or statement, and we do not present results in a traditional leaderboard. instead, we developed an interactive result exploration system that allows analyses of model results, and which we describe in this section. to further motivate this change, consider the following conclusion someone may draw from looking at a leaderboard: system foo performs the best. our interactive system aims to enable more nuanced statements such as: system foo leads to consistent performance increases in bar-type metrics on challenges that measure baz while maintaining equal performance on most metrics of type qux. a screenshot of our system is presented in figure 2 . 12 in addition, our baseline results are presented in a tabular view in tables 3 and 4 . our interactive system is centered around a parallel coordinates plot @xcite which shows all results as lines through parallel axes. every line intersects the axes at the corresponding mapped value. for instance, see the red line representing the results for task "totto" of baseline "t5-small". filters can be applied along axes (see bleurt axis in figure 2 ) and the filtered selection is highlighted through bold lines. a selection can be a set of metrics, systems, or tasks. this style of presentation has not been used before for a benchmark. the closest prior work is by @xcite for namedentity recognition which allows similar filtering and sorting, but presents the results in a table. however, the parallel coordinates approach can scale to a much greater number of metrics than a table. moreover, by using a parallel coordinates plot instead of a table, it is easy to spot patterns that span multiple metrics, systems, or tasks. for example, the highlighted line in figure 2 uncovers that, for the t5 baseline on totto, the diversity metrics score higher than other systems while scoring lower on reference-based metrics. since we only have a single baseline for totto, it is unclear whether this difference can be attributed to the dataset or the system but this relationship will be uncovered once we receive submissions. the final system will additionally be able to display the model cards and other related metainformation associated with submissions. it will also be able to show (and compare) exemplary outputs for each test set. those two features will improve the transparency of the results and systems to those who are not familiar with a task and provide necessary information to those who consider using a particular system. the combination of all components will enable analysis on quantitative, individual, and qualitative level which can support formulating new research hypotheses and gather in-depth insights about system performance. for example, the functionality to compare human anno-tation and automatic measures could lead to a better understanding how fluency affect bertscore. in addition to the interactive self-directed result exploration, our shared task features an evaluation and analysis part. instead of dictating the interpretation of the modeling shared task results, we will release all system outputs and metrics in this second part and participants of this part may run their own evaluation and conduct interesting analyses. ## submitting to the benchmark while we ask submitters to try to cover as many tasks as possible, we acknowledge potential restrictions on computation resources. we thus do not require that a submissions to gem has to include predictions on every included test and challenge sets. all predictions from a model should be formatted and added into a single file as outlined on our website. in addition, we require every submitter to answer a series of questions that we will use to construct a model card @xcite and externalize potential concerns regarding the social impact of a model and its use, or its training data. the card will additionally display information to replicate the experiments. while we require responses to these questions at submission time, we allow the information about a model to remain anonymous during required anonymization periods should a paper describing the model be under submission elsewhere. all submitted model outputs will be made publicly available for download. after a submission, we will run the evaluation suite on the submitted outputs and additionally collect human annotations. human evaluation gem will be used to develop reproducible and consistent human evaluation strategies for generated text. this task involves selecting and defining which quantities of the generated text should be measured, developing annotation schemes and rater guidelines to capture these quantities accurately, and infrastructure to annotate system outputs. we aim to develop these setups for all task setups such as summarization, dialogue, simplification, and data-to-text. to approach this task, we will follow the recently proposed taxonomy of human evaluation measures by @xcite and follow the reporting strategies proposed by @xcite . the detailed setups will be described in a evaluation datasheet @xcite . all shared task participants will be asked to provide gold annotations on system outputs, which we will then use to evaluate the consistency of crowdsourced annotations. 13 ## next steps this section lists the currently active developments and the long-term steps we will take to ensure that gem will continue to evolve and improve. ## conclusion in this paper, we have introduced gem, a living natural language generation benchmark with a focus on evaluation. while gem does not claim to instantly solve all issues of benchmarks in nlg, we aim to provide an environment in which systems can be tested in a principled manner and which can elevate the prominence of interesting evaluation approaches. by providing a testbed to easily conduct experiments across many datasets and evaluate in a repeatable, consistent, and more interpretable way, we will be able to track progress toward the goals in nlg research much more clearly. moreover, we will be able to extend and shape gem in the future to include more multilingual datasets, which will assist in their adoption across the wider research community.