DCU - NLG - PBN at the GEM ’24 Data-to-Text Task: Open-Source LLM PEFT -Tuning for Effective Data-to-Text Generation (2024)

Abstract: LLMs have been used in various tasks with impressive success, including data-to-text generation. However, one concern when LLMs are compared to alternative methods is data contamination, in other words, for many datasets the data used in training these models may have included publicly available test sets. In this paper, we explore the performance of LLMs using newly constructed datasets in the context of data-to-text generation for English, Chinese, German, Russian, Spanish, Korean, Hindi, Swahili, and Arabic. We performed a testing phase to evaluate a range of prompt types and a fine-tuning technique on Mistral 7B and Falcon 40B. We then fully evaluated the most promising system for each scenario: (i) LLM prompting in English followed by translation, and (ii) LLM PEFT-tuning in English followed by translation. We find that fine-tuning Mistral outperforms all other tested systems and achieves performance close to GPT-3.5. The few-shot prompting with a dynamic selection of examples achieves higher results among prompting. The human evaluation to be carried out by the shared-task organisers will provide insight into the performance of the new datasets. In conclusion, we observed how the fine-tuning of an open-source LLM can achieve good performance close to state-of-the-art closed-source LLM while using considerably fewer resources.

## introduction with the advancement of large language models (llms), their capabilities have been explored in many tasks including data-to-text generation, which maps structured input data into a suitable output text containing all and only provided information. however, the datasets for many data-totext tasks have been available online for years and might have been used to train llms. in the work reported here, we participate in the gem 2024 shared task @xcite using new datasets which are not available online. in more detail, we address the data-to-text generation task using two settings: llm prompting and fine-tuning. however, fine-tuning llms for specific tasks remains challenging, often constrained by computational resources. to mitigate this, we use a parameter efficient fine-tuning (peft) technique to substantially reduce the number of parameters participating in training, making the finetuning process far more computationally efficient while maintaining model performance. in both explored settings, we use an external machine translation (mt) system to translate our englishgenerated texts into chinese, german, russian, spanish, korean, hindi, swahili, and arabic. the paper is structured as follows. section 2 describes data and task, and section 3 presents the general approach, prompt types, testing phase and the specific systems we fully evaluated. experimental set-up and results are outlined in section 4, and section 5 provides conclusions. all the code and generated texts are available on github. 1 2 data and task the data-to-text task converts input data, specifically rdf triples representing subject | predicate | object combinations, into coherent and contextually appropriate text that accurately conveys all and only the information present in the input triples. the gem 2024 shared task provides datasets for two subtasks: (i) webnlg-based, utilising the official webnlg @xcite test set, and (ii) wikidata-based, using newly obtained triples from wikidata. each subtask includes three parallel datasets: factual, counterfactual, and fictional. the factual dataset consists of triples found in webnlg or wikidata. the counterfactual dataset switches entities based on their class, creating hypothetical scenarios. finally, the fictional dataset replaces original entities with those created via llm prompting. for all datasets, only the test set is provided, containing the input triples with predicates in english. no training data is available, and reference texts are not provided. however, for the webnlg-based factual dataset, references can be extracted from the original webnlg english dataset, allowing for some level of automatic evaluation. ## systems we consider two settings to create our systems using pretrained llms (figure 1 ): (i) generate text in english using out-of-the-box llms with prompting, (ii) generate text in english using a fine-tuned llm. in the first setting, we employ pretrained llms without additional training and use various prompting strategies to guide the model in generating text based on the input rdf triples. in the second setting, we fine-tune pretrained llms using low-rank adaptation (lora). regardless of the generation method, the generated english text is then translated into chinese, german, russian, spanish, korean, hindi, swahili, and arabic using a machine translation system. ## experimental set-up and results we executed our experiments using the transformer library 4 of huggingface and the paid-for google translate api 5 in late march/early april 2024. the systems are tested using the six datasets described in section 2. all generated texts are post-processed as described in section 3.2. all systems are executed on a nvidia a100 gpu with 80gb ram. following the webnlg 2023 evaluation setup @xcite , we perform an automatic evaluation on the webnlg-based factual dataset in english computing bleu @xcite , chrf++ @xcite , meteor @xcite , and bertscore @xcite . we compare our two systems against the best system proposed by @xcite , i.e. gpt-3.5 using few-shot prompt with fixed examples. an additional human evaluation will be performed by the organisers of the shared task and at the time of writing the results are not available yet. refer to the shared task report for more details. ## conclusion we explored the effectiveness of pretrained llms for data-to-text generation focusing on two settings: llm prompting and llm fine-tuning with lora. we first conducted a testing phase comparing the performance of mistral 7b and falcon 40b models using various prompting strategies and fine-tuning techniques, evaluated on the webnlg 2020 validation set. the results demonstrated that fine-tuning with lora substantially enhances the performance of the mistral 7b model. this model outperformed all other tested systems, including falcon 40b. among the prompting strategies, the few-shot in-context learning with dynamic examples based on the triple set length and predicates achieved the best results, indicating the importance of contextually relevant example selection. we submitted the two system settings, llm prompting + mt and llm fine-tuning with lora + mt, using mistral 7b to the gem 2024 shared task in english, chinese, german, russian, spanish, korean, hindi, swahili, and arabic. our findings highlight the potential of lora for efficient fine-tuning of llms, offering a competitive performance close to state-of-the-art models like gpt-3.5, but with substantially smaller model sizes and reduced resource requirements. the success of dynamic example selection in prompting also underscores the need for tailored approaches to optimize model performance.