GEM v2: Multilingual NLG Benchmarking in a Single Line of Code (2022)

Abstract: Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances. We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other’s work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.

## introduction the standard evaluation process in natural language processing involves comparisons to prior results in a fixed environment, often facilitated through benchmarks and leaderboards. this process, if executed correctly, can advance reproducibility @xcite and standardize evaluation choices that lead to better dataset diversity. but static benchmarks also prevent the adoption of new datasets or metrics @xcite , and many evaluation advancements are thus put aside. that means that the focus on surpassing the best prior reported scores reinforces outdated evaluation designs. furthermore, this process ignores properties that do not match the leaderboard metric @xcite @xcite . this issue is particularly pertinent in natural language generation (nlg) since the model quality cannot be estimated using accuracy and instead, nlg relies on automatic and human evaluation approaches that constantly improve @xcite . to bridge the gap between advantages of leaderboards and in-depth and evolving evaluations, the generation, evaluation, and metrics benchmark @xcite proposed a "living" benchmark. as such, gem is participatory in that contributors propose new datasets and expand the selection of metrics. model developers using gem retain full agency over the evaluation process but are able to choose from a wider range of tasks and metrics. gem further introduced evaluation suites @xcite that are compatible with its datasets and test various robustness and fairness aspects of models. we uncovered several shortcomings in gemv1 that hindered its scaling and adoption: (1) centralized data management made adding new datasets too complex. (2) computing all metrics in a single framework led to dependency issues and was challenging for those with limited compute resources. (3) participants needed more guidance in our dataset documentation process @xcite to guarantee data card quality. we introduce gemv2, a modular and extendable nlg evaluation infrastructure which allows for continuous integration of newly developed datasets. we release a data card collection and rendering tool that makes it easier to follow for both card creators and readers. these improvements led to an expansion of gem from 13 to 40 tasks and from 18 to 51 supported languages. we also introduce an online evaluation process that collects model outputs and computes metrics for all datasets. ## features and functionality since evaluation practices evolve, we focus on modularity and maintainability to ensure that new dataset and metrics are compatible with all other features. model developers are able to use new datasets and metrics without any changes to their existing setup. in this section, we describe the supported user [j]ourneys for various stakeholders in generation research. j1 -document a dataset every gem dataset is documented using the data card template by @xcite , which we revised using the data card playbook @xcite . a new card can be filled out or an existing one updated via an interactive form that provides detailed instructions for each field. 1 j2 -choose a dataset the data card viewer presents information at multiple detail levels in separate columns. anyone can quickly get a high-level overview of a dataset or read extended information on a documentation category (see figure 1 ). j3 -create a data loader each dataset has a separate repository at huggingface.co/gem, with a loader using the datasets library @xcite . 2 through this, all supported datasets can be loaded via the same code, ## dataset selection and loading to identify candidate datasets, continued to follow the superglue process @xcite by soliciting tasks to be included from the research community. our request to suggest multilingual, challenging, and/or interesting nlg tasks led to 40 submissions. to avoid quality judgments, we imposed only three requirements for selection: (1) consent from dataset authors, (2) availability under a permissive license, (3) the task needs to be able to be cast as a text-to-text problem. 27 tasks were selected in addition to the 13 existing ones @xcite .three datasets are simplification evaluation sets added to the wiki-auto loader @xcite , while all others have independent data loaders. 5 all data loaders and cards were produced as part of a month-long hackathon, and we invited the dataset authors and gem participants to contribute to one or more of the datasets. afterwards, the organizers managed the ongoing maintenance. new datasets can be added on an ongoing basis, subject to the three requirements. gemv2 currently supports 40 datasets, listed in appendix a and described in this section. figure 2 shows the distributions of training example count, task types, and their input and target lengths. data-to-text and summarization are most common, followed by response generation. while data-to-text tasks are spread across resource availability categories, summarization datasets tend to @xcite . summarization tasks have input lengths of over 1,000 while all other tasks remain under 1,000 tokens. there is a lot more between-task variance in output length. four dataset configurations are hidden due to the axis truncation. exceptions to this are summarization, with input lengths beyond what is supported by most models (e.g., wikicatsum @xcite and xlsum @xcite ), and a class of data-to-text datasets with the communicative goal to generate game summaries from large sports statistic tables (e.g., @xcite @xcite . we put an emphasis on language diversity, as prior work has found that fewer than 30% of nlg publications (even counting evaluations on machine translation) evaluate on non-english tasks @xcite . while a lot of this focus on english can be traced to a lack of multilingual resources, many non-english nlg datasets have been released in recent years (e.g., @xcite @xcite . as shown in of the 40 datasets, 14 have multiple configurations which can differ in task setup, languages, their encoding in romanized or original script, or domain. for example, we modified wikilingua @xcite to have splits from and to any of the 18 supported languages, enabling better crosslingual evaluations. seventeen datasets have challenge splits, many of which were created for gem. for example, the challenge set for the conversational weather dataset @xcite selects examples from the original test split with complex discourse relations. ## data cards each dataset is accompanied by documentation about how it was created, who created it, how it should be used, and the risks in using it @xcite . our original data documentation process @xcite required filling out a markdown template following instructions in a separate guide. we count languages 1 amharic, azerbaijani, bengali, burmese, dutch, gujarati, hausa, igbo, javanese, kirundi, kyrgyz, marathi, nepali, oromo, pashto, persian, pidgin, punjabi, scottish gaelic, serbian, sinhala, somali, sundanese, swahili, swedish, tamil, telugu, tigrinya, ukrainian, urdu, uzbek, welsh, yoruba 2 czech, italian, thai, turkish, vietnamese 3 arabic, finnish, hindi, japanese, korean, portuguese 4 indonesian 6 the output of the tool is a structured json file that tax. languages 0 west african pidgin english, sinhala 1 azerbaijani, burmese, gujarati, igbo, javanese, kirundi, kyrgyz, nepali, oromo, pashto, scottish gaelic, somali, sundanese, telugu, welsh 2 amharic, hausa, marathi, punjabi, swahili, tigrinya, yoruba 3 bengali, indonesian, tamil, thai, ukrainian, urdu, uzbek 4 czech, dutch, finnish, hindi, italian, korean, persian, portuguese, russian, serbian, swedish, turkish, vietnamese 5 arabic, chinese, english, french, german, japanese, spanish the structured format enables us to study trends in dataset construction practices beyond those shown in section 3. 6 for example, 66% of the data cards report that pii is unlikely or definitely not included, while it is likely or definitely included in 33%. in the free-text explanations, we find four types of justifications for absent pii: the majority (7) stated that the data format or domain was restricted to avoid pii. two stated that the data is in the public domain (e.g., wikipedia) and another two used fully simulated data. one response described that crowd raters were instructed to avoid mentioning pii. we found that multiple of the piilikely datasets only use public domain data, indicating that there is confusion about pii definitions. another typically hidden aspect is the data sourcing. our datasets present an almost even split between automatically-, crowdworker-, and expertcreated datasets, with crowdworker-created ones being slightly more common, possibly confounded if experts were hired through crowdworking platforms, as was done for squality @xcite . it may thus also possible to compare which of these collection methods leads to more insightful modeling results. we follow up by asking which crowdworking platform was used and unsurprisingly, amazon mechanical turk was the most frequent answer, followed by participatory experiments and other non-specified platforms. ## system design to support the automatic evaluation of outputs, we use the hugging face hub to integrate datasets, metrics, and user interfaces for gem users to submit their outputs. the system architecture is shown in figure 3 , and consists of five main components: spaces we host streamlit applications on spaces foot_6 for the submission of predictions, downloading of results, and visualization of model performance. datasets dataset repositories are used to host the datasets, submissions, evaluations, and results. autotrain we use autotrain foot_7 , hugging face's automl platform, to run all evaluation jobs using hugging face benchmarks, a library that defines how metrics are computed within autotrain. 9 metrics we use gem-metrics to perform the metric computations. in addition to supporting common metrics like bleu @xcite and rouge @xcite , the docker integration simplifies the calculation of multiple model-based metrics like bleurt @xcite . on submission, a dataset repository with the model outputs is created under the gem-submissions organisation on the hugging face hub. in parallel, an evaluation job is triggered in autotrain which downloads the submission from the hub, along with all the reference splits of the gem datasets. these references are used to compute a wide variety of nlg metrics via gem-metrics. the resulting metrics are then pushed to a dataset repository on the hub, and used to source the visualization of results on the gem website foot_9 and space. 11 ## conclusion we introduce gemv2 which unifies infrastructure for generation research. we propose a consistent workflow from documenting and choosing datasets to loading and evaluating on them while keeping all supported datasets and metrics compatible with each other. we demonstrate the scalability by releasing the initial version with support for 40 datasets in 51 languages. of the supported datasets, 23 are improved through configurations, filtering, and re-splitting processes and 17 datasets have challenge sets. we release a submission tool that computes metrics and makes model outputs available to download for evaluation researchers. researchers who are interested in integrating their dataset are welcome to contact us for support.