D ip I nfo- U ni T o at the GEM ’24 Data-to-Text Task: Augmenting LLM s with the Split-Generate-Aggregate Pipeline (2024)

Abstract: This paper describes the DipInfo-UniTo system participating to the GEM shared task 2024. We participate only to the Data-to-Text (D2T) task. The DipInfo-UniTo system is based on Mistral (Jiang et al., 2023), a recent Large Language Model (LLM). Most LLMs are capable of generating high-quality text for D2T tasks but, crucially, they often fall short in terms of adequacy, and sometimes exhibit “hallucinations”. To mitigate this issue, we have implemented a generation pipeline that combines LLMs with techniques from the traditional Natural Language Generation (NLG) pipeline. In particular, we have a three step process SGA, consisting in (1) Splitting the original set of triples, (2) Generating verbalizations from the resulting split data units, (3) Aggregating the verbalizations produced in the previous step.

## introduction in the last few years, llms have become the state of the art in natural language generation tasks, as can be seen in the most important conferences and challenges, such as the inlg conference and the webnlg challenge. 12 llms enable high performance across various fields of nlp, including rdf-to-text. the use of such models can occur through prompting techniques or, if there is a suitable dataset available for their task, by fine-tuning the models, which involves further training. this latter approach leaded to improved performances in many tasks related to generation. systems like llama2 @xcite and mistral are among the most popular open-weights system for text generation. fortunately, a linguistic resource for fine-tuning these models is provided by the webnlg challenge. this corpus, originally designed for english and later extended to other languages @xcite , russian @xcite and partially maltese @xcite , among others), consists of data units, i.e., sets of rdf triples, composed of subject, predicate, and object, accompanied by their verbalizations, which represent the semantics of the triples. the system employed in the gem shared task @xcite consists of a threestep pipeline, which we called sga @xcite . it includes a data unit splitting algorithm (s) to simplify data units for subsequent steps, an rdf-to-text system (g) designed to generate verbalizations from obtained data units, and a sentence aggregation system (a) to combine the verbalizations produced in the previous steps. the paper is structured as follows: in section 2 we provided a brief selection of related work; in section 3 we give few details about gem shared task; in section 4 we give some details on webnlg 3.0, that is our training corpus; in section 5 we describe the sga pipeline; in section 6 we present the official results of the dipinfo-unito system and, finally, section 7 closes the paper by considering future development. the code and submitted outputs of the dipinfo-unito system can be found on github. 3 2 related works over the years, rdf-to-text has become an increasingly important task. several webnlg challenges have been held @xcite to develop the best rdf-to-text models based on webnlg corpora. 456 a common strategy involves using prompting techniques or fine-tuning to generate verbalizations from given rdf triples @xcite . in the latest webnlg challenge, several pipelines emerged to generate more accurate outputs, incorporating techniques such as data splitting to reduce the input data and backtranslation for low-resource languages @xcite . ## gem 2024 rdf-to-text task description the gem shared task 2024 focuses on text summarization and rdf-to-text generation. our participation is limited to the second task, which involves generating verbalizations from a set of rdf triples. these triples, consisting of a subject, predicate, and object. the shared task provides six files containing rdf triples extracted from the web. three of these files each contain 1, 799 inputs extracted from webnlg and are classified as "seen" inputs, because these data could have corresponding goldstandard verbalizations that can be used to train potential statistical or neural systems. the other three files each contain 1, 800 inputs extracted from wikipedia, for which no gold-standard verbalizations are available online. these inputs are therefore classified as "unseen" inputs. these files contain triples extracted directly from webnlg and wikidata, altered triples where the subject or object has been changed and triples with entities generated using llm prompting. the task is designed for multiple languages, including english, chinese, german, russian, spanish, korean, hindi, swahili, and arabic. in our case, we have chosen to participate in the task using only english. ## english webnlg corpus description webnlg is a corpus containing data units, a set of rdf triples, each paired with one or multiple natural language expressions handwritten by expert annotators, where verbalizations express the semantics of the corresponding data units. for instance: ## the sga pipeline our work is based on the sga pipeline, illustrated in figure 2 , which consists of three main steps: data unit splitting (s), rdf-to-text generation (g), and sentence aggregation (a). while the first step is based on a symbolic deterministic algorithm, the second and third steps rely on llms. we chose this modular approach to mitigate the "hallucinations" of llms' holistic approach. this was done because we hypothesize that as the amount of input data increases, the performance of llms in terms of adequacy and fluency decreases. in the shared task, the provided data units contain up to seven triples. to address these issues, we simplified the problem by dividing the data units into separate sets of triples, which were then verbalized through an rdf-to-text system and unified using a sentence aggregation system. as described in section 4, the data units in the test sets provided by the gem shared task could also have different shape types, representing various relationships between them, namely chain, sibling, and mixed type. to reduce the complexity of the data units, i.e., reducing the number of triples in each unit, the main idea is to divide data units into subsets of triples, with a maximum of three triples per set. to achieve this goal, we analyze the shape of each data unit to identify the relationships between triples. unfortunately, the data units provided by gem do not contain information about the shape type. therefore, we created a splitting algorithm to find the relationship between triples and divide them based on the retrieved information (cf. algorithm 1). the splitting algorithm processes triples within a data unit by storing those with identical subjects in subjects_dict and those whose objects appear as subjects in other triples in objects_dict. it splits each triple into subject, predicate, and object, using the subject as the key in subjects_dict and checking if the subject appears as an object in other triples. if so, those triples are added to objects_dict. after populating both subjects_dict and objects_dict, the algorithm merges these dictionaries into a unified structure called merged_dict. this involves copying entries from subjects_dict into merged_dict. for keys that are present in both dictionaries, the algorithm checks if any triple from subjects_dict is already listed under that key in merged_dict. if a triple is not found in the existing list, it is added to the list, capturing the chain relationships between triples. once the dictionaries are merged, the algorithm addresses cases where any key in merged_dict contains more than three values. it splits these lists into chunks of up to three items each, dividing the triples based on their order. for instance, if there are four triples, the first three are grouped into one subset, while the fourth is placed in a separate subset. this method ensures that no list becomes too large, making the data easier to process and analyze. the choice of three as the chunk size is based on a qualitative analysis of the results from the sga pipeline. for example, given this data unit with four triples: for each set, the corresponding verbalizations will be generated using the approach described in the next section. ## results in this section, we present the results provided by the organizers for the various systems participating in the gem shared task, evaluated on different subsets of data. the performance of these systems are compared using metrics such as bleu, me-teor, chrf++, and bert f1 for the english tasks. the evaluation was conducted using 180 selected data points, each associated with a single reference text. it is important to note that the use of only one reference per data point might lead to lower scores compared to evaluations with multiple references or a larger number of data points. d2t-1-fa the d2t-1-fa subtask consists of data units directly extracted from the webnlg test set. in this task, the dipinfo-unito system demonstrated excellent performance with a bleu score of 32.31, making it the top system in this metric. additionally, it achieved great results across other metrics, ranking among the best systems for this task (see d2t-1-cfa this subtask involves switching entities in the data units extracted from webnlg (e.g., replacing a person entity with another person entity, a date with another date, etc.). dipinfo-unito achieved the highest scores across all metrics, surpassing the other participants by a significant margin, making it the best system for this task (see d2t-1-fi in the d2t-1-fi subtask, which is the most challenging of all the d2t-1 dataset, data units were first extracted from webnlg and then modified with entities generated by an llm. the dipinfo-unito system achieved the highest scores across all metrics, maintaining a significant lead over the other systems, like in the previous subtask (see d2t-2-fa this subtask involves data units directly extracted from wikidata. the dipinfo-unito system achieved the highest score on the bert f1 metric (0.937) and ranked as the second-best system in the other metrics, just behind saarlst (see d2t-2-fi finally, the d2t-2-fi subtask involves data units from wikidata with entities generated by an llm. the dipinfo-unito system demonstrated strong performance, achieving a bleu score of 21.26, the highest among all participants. it also ranked second in the other metrics for this task, being outperformed by saarlst in meteor and chrf++, and by dcu-nlg-pbn in the bert f1 metric (see in conclusion, the dipinfo-unito system has proven to be highly competitive across all tasks, frequently achieving the highest scores among participants and only falling slightly short in other cases, demonstrating excellent generalization ability across various datasets. ## conclusion the main objective of this work was to enhance the performance of llms in rdf-to-text generation. to achieve this, we employed nlg techniques with llms to develop the sga pipeline designed to simplify the task and improve the quality of the outputs. to demonstrate the effectiveness of this technique, we compared the performance of llama-2 and mistral models both with fine-tuning and within the sga pipeline. the results show that our approach improves performance on the rdf-to-text task. the developed system demonstrated strong competitiveness across all tasks in the gem 2024, achieving the highest scores in some cases while narrowly missing out in others. this performance underscores its good ability to generalize across various datasets. future work could involve refining this technique by fine-tuning the models to better specialize in sentence aggregation, developing a more sophisticated data splitting algorithm, and integrating additional nlg techniques to produce more fluent and accurate text.