The 2024 GEM Shared Task on Multilingual Data-to-Text Generation and Summarization: Overview and Preliminary Results (2024)

Abstract: We present an overview of the GEM 2024 shared task, which comprised of both data-to-text generation and summarization. New datasets were compiled specifically for the task to reduce data contamination in the large language models, which the participants were likely to use. The paper describes the tasks, the datasets, the participating systems, the evaluation methods, and some preliminary results. The full results will be presented at INLG ‘24.

## introduction since its inception, the generation, its evaluation and metrics initiative (gem @xcite ) has had the objective to contribute to measuring progress in the field of natural language generation (nlg), via the creation of datasets and tools for automatic and human assessments of text generation systems on different nlg tasks @xcite @xcite @xcite @xcite . in the past few years, large language models (llms) have been widely used in nlg; they have been trained on enormous amounts of data, to the point that it can be unclear what they have seen or not during training time @xcite . to challenge these models, the nlg community has recently been developing methods for creating ad-hoc input data that the models cannot have been exposed to. for instance, @xcite propose to build dynamically counterfactual and fictional inputs for data-to-text generation, and @xcite released a tool for collecting new test sets using public apis; the creation or compilation of reference texts for the collected inputs remains an open issue. in parallel, the interest for multilingual natural language processing has been growing, with the organisation of shared tasks that included underresourced languages, such as universal dependency parsing @xcite for syntactic parsing, msr @xcite for surface realisation, lowresourceeval @xcite for morphological analysis, @xcite and wmt @xcite for machine translation, as well as webnlg @xcite for data-to-text generation. inspired by the current state of affairs, this edition of the gem shared task 1 has two main objectives: (i) to assess llms-and more broadly nlg systems-using new ad-hoc datasets that no model could have already been exposed to, and (ii) to encourage participants to come up with approaches suitable across languages (including lowresource languages). we created data for two tasks, namely data-to-text generation and text summarization. the data-to-text task comprises 6 types of inputs: in-domain factual data, in-domain counterfactual data, in-domain fictional data, out-ofdomain factual data, out-of-domain counterfactual data, and out-of-domain fictional data. we accepted output texts in 9 languages: arabic, english, chinese, german, hindi, korean, russian, spanish and swahili; small sets of new humanwritten references were compiled for all 6 test sets in english and swahili. for the summarization task, we scraped recent news articles in swahili, extracting a summary from the web page they appeared in. the other two summarization subtasks we planned (cross-lingual summarization and book chapter summarization) did not attract participants, so we do not elaborate on them here. for all tasks, we apply both automatic and human evaluation methods. in the remainder of this paper, we present the timeline of the task and comment on the incomplete results (section 2). we then provide an overview of the tasks and datasets involved (section 3), followed by descriptions of the participating systems (section 4) and the evaluation methods employed (section 5). finally, we present the results available at the time of publication (section 6). as a result, at the time of publication of this paper, several evaluations are still ongoing. we only sent the participants the following completed evaluation results: the data-to-text metrics results for english (6 test sets, 7 systems), and the summarization metrics results for swahili (1 test set, 2 systems). the data-to-text metrics results for swahili (6 test sets, 3 systems), the human evaluation results for english (6 test sets, 7 systems), swahili (6 test sets, 3 systems) and spanish (6 test sets, 3 systems), and the summarization human evaluation results for swahili (1 test set, 2 systems) are not yet released and are planned to be presented during the inlg conference in september 2024. ## overview of tasks the gem 2024 shared task consists of two different types of tasks: data-to-text generation and text summarization. ## participating systems about 40 teams pre-registered, and 9 submitted outputs; one team eventually withdrew their submission. pre-registration after handing out a preliminary survey to collect interest in the tasks and languages for the shared task, we asked all registered teams to carry out a pre-registration of their planned experiment(s). the objective of the preregistration is to log in the details of a specific experiment before it is carried out; it is an important initial step to guarantee that the experiment is conducted fairly, and to help avoid potential biases derived from the researchers' interest @xcite . we asked participants to pre-register selected information (i.e. intended systems, hardware, additional data, automatic metrics, etc.) through a qualtrics form (see appendix e for screenshots of the form). in the following, the summarization baseline and the team submissions are briefly described; an overview of participation to the tasks is provided in the summarization baseline uses gpt-3.5 following the prompt design from @xcite . the specific prompt is "summarize the above ar-ticle briefly in 1 sentence" translated into swahili, "fanya muhtasari wa kifungu kilicho hapo juu kwa kifupi katika sentensi 1.". the system prompt is the default. all output is checked for language id to ensure that the output is in swahili. cuet_sstm @xcite uses an integrated extractive-abstractive summarizer. for the extractive summarizer, the authors used the bert extractive summarizer, which shortens long texts of more than 512 tokens. for the abstractive summarizer, they used two pre-trained models (t5-small, mbart-50) to generate the summaries. the integrated model is trained on the xlsum swahili dataset combined with 1,000 manually summarized texts from the given swahili news classification dataset. dcu-adapt-modpb (osuji et al., 2024) adopts an nlg+mt approach based on a pipeline neural architecture. it leverages the fine-tuned flan-t5-large model for the ordering and structuring of input triples. additionally, a gpt-4 prompt-based model was integrated for surface realisation, generating the final text outputs and employing few-shot prompting with five examples for the final text generation tasks in english. for multilingual text generation in korean, arabic, and swahili, a promptbased model-the cohere-command-r-plus neural machine translation model-was incorporated, also using five examples for the translation. for hindi, the indictrans2 model was used. dcu-nlg-pbn @xcite finetuned the mistral 7b instruct model, using low-rank adaptation (lora) to enhance performance while maintaining computational efficiency. the system generates text in english, which is then translated into multiple languages (chinese, german, russian, spanish, korean, hindi, swahili, and arabic) using a machine translation system (google translate). dcu-nlg-small @xcite combined the forge rule-based generator and a post-processing step with a t5-base model fine-tuned on a parallel dataset of english rule-based-generated texts and human-or llm-produced texts. for languages other than english, they used the off-theshelf machine translation system nllb, which is freely available on huggingface. dipinfo-unito @xcite focuses on english and employs a three-step pipeline called the sga (split-generate-aggregate) pipeline to generate verbalizations. the process begins with a data unit splitting phase, where the initial triples are divided into subsets of three or fewer triples, with an effort to maintain the relationships between them. the next step involves generating verbalizations for each subset of triples using mistral-7b, which has been fine-tuned on a training and development set from webnlg 3.0 dataset for english. finally, in the last step, a pre-trained mistral-7b model is used for sentence aggregation with a zeroshot prompting technique, merging the generated sentences into a more fluent and coherent text. osu compling @xcite experimented with a data filtering and knowledge distillation approach for english, spanish, chinese, and english. they leverage the expertise of chat-gpt (gpt 4.0) to generate training data for factual, counterfactual, and fictional triple sets. data filtering was done with automatic model judgments for error detection. spanish and english filtered synthetic data was used to fine-tune llama2. rdfpyrealb (lapalme, 2024) employs a symbolic method to address the english d2t challenge. one objective is to contrast the outcomes of computationally demanding techniques that may not always be easy to control with a streamlined, swift, and reliable symbolic method. the design is straightforward: every rdf triple represents a statement, where the subjects and objects of the triple are nearly identical to those of the sentence. the predicate in the triple represents a verb phrase that defines the sentence's syntax. the narrativebuilding mechanism arranges predicates sequentially, giving rise to a coherent tale. it also combines sentence components when they share the same subject or predicate. the final realization is performed using pyrealb, a french-english realizer which is used in some data to text applications. saarlst (jobanputra and demberg, 2024) employs a retrieval-augmented generation (rag) pipeline to generate verbalization. most rag pipelines use a dense retriever while this pipeline contains a sym-bolic retriever -propertyretriever. the propertyretriever leverages available webnlg training and validation sets to retrieve instances with the most similar properties. the retrieved examples and prompting instructions combined form the final few-shot prompt. in the final verbalization step, the pipeline prompts an ensemble of mixtral and command-r models to generate coherent verbalization. ## evaluation methods in section 3, we detailed the procedure for creating the inputs used in both the d2t and summ tasks. initially, these inputs lacked corresponding reference texts. due to the significant time and resource investments required to create input-output pairs, we strategically delayed collecting human references until we had identified the languages submitted by participants. this section first provides an overview of the language selection and the reference text creation procedures, and then describes the automatic and human evaluations we ran on each submission to the shared task. ## results in this section, we present the results of the metrics evaluation for the english data-to-text task and the swahili summarization task. ## conclusions we presented an overview of the two tasks of the 2024 gem shared task, multilingual data-totext generation and news article summarization in swahili. for both tasks, we collected new data with the objective provide challenging inputs to the large language models that we supposed most teams were going to use. for the data-to-text task, 7 teams submitted outputs in one or more languages, and we report on the metrics evaluation for english outputs only. the results of the evaluation show that despite the variety of system types (llms, rule-based, combination of the two), all systems seem to suffer when exposed to (i) out-of-domain data, and (ii) counterfactual or fictional data. the unexpectedly high scores obtained by all systems on the counterfactual out-of-domain dataset remain to be explained, possibly in the light of the human evaluation results. for the summarization task in swahili, we received only one submission, which is competitive with a zero-shot gpt-3.5 baseline according to the metrics evaluation. we were not able to complete all evaluations at the time the paper is published, and the data-totext metrics results for swahili, the human evaluation results for english, swahili and spanish, and the summarization human evaluation results for swahili will be reported in a separate publication. lining zhang, simon mille, yufang hou, daniel deutsch, elizabeth clark, yixin liu, saad mahamood, sebastian gehrmann, miruna clinciu, khyathi raghavi chandu, and joão sedoc. 2023. a needle in a haystack: an analysis of high-agreement workers on mturk for summarization. in proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 14944-14982, toronto, canada. association for computational linguistics. tianyi zhang, varsha kishore, felix wu, kilian q weinberger, and yoav artzi. 2019. bertscore: evaluating text generation with bert. arxiv preprint arxiv:1904.09675.