T eam S aar LST at the GEM ’24 Data-to-text Task: Revisiting symbolic retrieval in the LLM -age (2024)

Abstract: Data-to-text (D2T) generation is a natural language generation (NLG) task in which a system describes structured data in natural language. Generating natural language verbalization for structured data is challenging as the data may not contain all the required details (here, properties such as gender are missing from the input data and need to be inferred for correct language generation), and because the structured data may conflict with the knowledge contained in the LLM’s parameters learned during pre-training. Both of these factors (incorrect filling in of details, pretraining conflict and input data) can lead to so-called hallucinations. In this paper, we propose a few-shot retrieval augmented generation (RAG) system, using a symbolic retriever – PropertyRetriever. Additionally, we experiment with state-of-the-art large language models (LLMs) to generate data verbalizations. Our system achieves the best results on 4 out of 6 subtasks for METEOR and chrF++ metrics. We present our results along with an error analysis. We release our code for reproducing the results as well as the generated verbalizations from all the experiments for any further explorations here.

## introduction nowadays llms are pretrained using trillions of text tokens foot_1 @xcite . these llms can not only generate grammatical and fluent text, but they are also capable of learning new tasks without any training data using in-context learning techniques @xcite . one central challenge in llms research is to understand the extent to which llms memorize their training data versus how they generalize to new tasks and settings. there has been some empirical evidence that llms do some degree of both: they clearly memorize parts of the training data -for example, llms are often able to reproduce large portions of training data verbatim @xcite -but llms also seem to learn from this data, allowing them to generalize to new tasks. do llms truly produce new content, or do they only remix their training data? until we concretely answer this question, it is essential to test model faithfulness systematically through various data augmentation techniques. the task of data-to-text generation is one of the popular nlg tasks. in this task, the system is given a set of rdf triplets describing facts (i.e., entities and relations between them) and has to produce a fluent text that is faithful to the facts. the gem'24 @xcite challenge brings forth a new shared task on data-to-text generation to test llms for factual information (i.e., information in the model parameters is likely to be in line with the input), vs. counterfactual information (i.e., the information in the prompt contrasts with what the model encodes about this entity) vs. fictional entities (i.e., the model parameters should not contain specific information supporting or contradicting the prompt information.) the gem'24 shared task consists of two subtasks of generating texts from input triple sets (subject | property | object) in the webnlg fashion. we participate in both the subtasks. one of the subtasks (d2t-1) is based on the webnlg dataset. this subtask uses the official webnlg test set foot_2 as input for testing the generation system. the test data contains 1,779 input triples with properties and entities not seen in the training/dev data. the second subtask (d2t-2) is based on the wikidata. this subtask uses 1,800 newly compiled input triples from wikidata for testing the generation system. @xcite proposed this dataset containing 74 new properties and entities, which were not part of the webnlg dataset. in recent years, llms such as gpt-4 (achiam et al., 2023), gemini @xcite and llama @xcite , have made significant advancements in the field of natural language generation (nlg). however, the inherent tendency of these llms to generate inaccurate or non-factual content, commonly referred to as "hallucinations" @xcite , continues to present a significant challenge. this generally occurs because the model parameters from pretraining encode some information, which may "overwrite" the information in the prompt due to its high sequence probability. another challenge with structured data is that the data does not contain all the required details such as entity type, gender and relation explanation. if the model fails to infer these details correctly, it may generate hallucinated verbalization. in the literature, @xcite suggests that providing relevant examples during inference can help in reducing hallucinations. while @xcite suggests using an explicit, symbolic, text planning stage for generating more faithful verbalization of data. in this work, we combine these suggestions and propose a few-shot rag system to solve this task, using a symbolic retriever -propertyretriever. figure 1 illustrates the architecture of our system. we experiment with state-of-the-art open-weight models for generating verbalization. in the following sections, we describe the dataset, our approach and provide a detailed study of the errors made by the system. ## dataset the gem'24 shared task introduced novel augmented test sets for both webnlg and wikidata. these augmented test sets consist of 3 parallel datasets as follows: ## method our final system consists of a few-shot rag pipeline that verbalizes the input data. in the following subsections, we describe the details of each component of our rag pipeline. ## shared task results the gem'24 shared task organizers provide evaluation scores for all participating systems and subtasks using 4 metrics -bleu, meteor, chrf++ and bertscore. these scores are calculated with 1 amt reference text per data point. our system achieves the best results on 4 out of 6 subtasks for meteor and chrf++ metrics. for detailed results and comparison with participating systems, please refer to the overview literature @xcite . ## performance analysis in this section, we provide the results of our human evaluation study. we conduct this study to finalize our primary and secondary model for the final system. we also discuss observed error patterns during the evaluation. ## conclusion in this paper, we describe our solution for the datato-text generation shared task. we propose a symbolic retriever method -propertyretriever, to retrieve better examples for data-to-text generation problems. we further explore the capabilities of two state-of-the-art llms, mixtral and command-r. combining the insights from our human evaluation study and error analysis, we propose an ensemble system as our final solution. in the future, we would like to explore multiturn correction and planning approaches. we believe such approaches may allow the model to selfcorrect its formatting errors and generate verbalizations with better fluency.