Pipeline Neural Data-to-text with Large Language Models (2024)

Abstract: Previous studies have highlighted the advantages of pipeline neural architectures over end-to-end models, particularly in reducing text hallucination. In this study, we extend prior research by integrating pretrained language models (PLMs) into a pipeline framework, using both fine-tuning and prompting methods. Our findings show that fine-tuned PLMs consistently generate high quality text, especially within end-to-end architectures and at intermediate stages of the pipeline across various domains. These models also outperform prompt-based ones on automatic evaluation metrics but lag in human evaluations. Compared to the standard five-stage pipeline architecture, a streamlined three-stage pipeline, which only include ordering, structuring, and surface realization, achieves superior performance in fluency and semantic adequacy according to the human evaluation.

## introduction advancements in data-to-text natural language generation (nlg) have evolved from seq2seq models @xcite and vanilla encoder-decoder models @xcite towards pretrained language models (plms) @xcite @xcite . initially, plms were fine-tuned on specific datasets to perform text generation tasks. recently, these models are prompted with textual instructions, with or without examples, to guide text generation (zero-shot and few-shot learning). although plms excel in several natural language processing tasks, they face challenges in generating text from complex structured data due to the intricate demands of accuracy and structure @xcite . despite these challenges, plms demonstrate superior performance in generating high-quality text under fine-tuned, fewshot, or zero-shot learning scenarios, leveraging extensive pre-training on general knowledge. in a previous study, @xcite compared traditional 5-stage pipeline approaches to end-to-end neural methods, utilizing systems such as gru @xcite and the bert transformer @xcite . the pipeline approach, despite lacking pretraining or fine-tuning, outperformed the end-to-end method in automatic and human evaluations, especially in domains not seen in the training phase. building on @xcite , this study integrates plms and large language models (llms) into the pipeline architecture to compare their effectiveness against the baseline. we assess the generalization capabilities of pipeline neural architectures and end-to-end systems under fine-tuned and few-shot settings, also proposing a simplified 3stage pipeline architecture. automatic evaluations and human assessments of the results highlight a preference for end2end architecture and the potential for optimized pipeline designs. the code and results are publicly available foot_0 . ## related work end-to-end (e2e) architectures, while simplifying generation processes, face limitations due to the lack of intermediate steps, which can hinder control over semantic fidelity @xcite . researchers have increasingly adopted pipeline architectures for data-to-text tasks, leveraging diverse deep neural network models @xcite @xcite . the data-to-text generation pipeline, originally delineated by @xcite 2020 ) divides text generation into planning and realization stages, using ordered trees or relational graph convolutional networks (r-gcn) @xcite to guide the neural generation system, providing explicit control over the output. recent research has utilized plms like t5 (raffel et al., 2020) and bart @xcite for both pipeline and end-to-end data-to-text generation, achieving more fluent text than human references @xcite . this is evident from the top competitor @xcite in the webnlg'20 (castro ferreira et al., 2020) competition. studies have also shown that these plms when fine-tuned outperform generative llms like gpt-3.5 @xcite in prompt-based scenarios, reducing hallucinations and over-generation issues @xcite , which are pivotal areas of investigation in our current study. by integrating plms into both traditional and simplified pipeline architectures, our research seeks to quantify their impact on the fidelity and fluency of generated text, particularly under fine-tuned and few-shot conditions. ## results fine-tuned models across all domains, flan-t5 surpasses both bart and gpt-2, except for the structuring stage where bart excels. in the seen category, flan-t5 maintains its superiority across all pipeline stages compared to gpt-2 and bart. notably, gpt-2 closely competes with bart, particularly in the ordering stage where bart outperforms. in the unseen domain (referenced in in the surface realization stage of the pipeline architecture, the baseline model seemed to perform best followed by the flan-t5 model. all other model seemed to perform poorly. but in general the fine-tuned models performed best. prompt-based llms due to the substantial costs linked to proprietary models like gpt-4 turbo, we limited their application to specific stages of the pipeline and for end-to-end data-totext generation. to control expenses, we refrained from generating referring expressions for evaluation from the gold standard inputs due to the extensive dataset involved. nonetheless, we did produce results for the referring expression generation (reg) stage within the pipeline, where the inputs were directly sourced from the mapped lexicalization outputs of the pipeline itself. the results of these models in fine-tuned vs. prompt-based models overall, in comparing fine-tuned and instruction-based models in ## conclusion this study demonstrates that plms tend to outperform the baseline, particularly in unseen domains. the baseline in this case is a vanilla transformer model that was trained from scratch on the dataset. it also corroborates existing research which shows that fine-tuned models generally outperform prompt-based models in zero-shot scenarios and exhibit comparable trends in few-shot learning @xcite . however, prompt-based models exhibited fewer errors in numbers and entities, as well as fewer additions and omissions compared to the fine-tuned models. this confirms previous research on fine-tuned models in pipeline architecture generating imaginary numbers @xcite . moreover, the performance of prompt-based models does not decrease in unseen domains, as shown 2 https://platform.openai.com/docs/models/ gpt-4o in previous studies and for fine-tuned models. in the comparison between pipeline and end-toend approaches, our study shows that end-to-end architecture yielded the best results in both automatic and human evaluations. in the comparison between pipeline approaches, our analysis indicates that a pipeline architecture with fewer stages produces better outcomes than a full-stage pipeline. in a combination between model designs, we speculate that fine-tuned models under a 3-stage architecture could outperform prompt-based models. additionally, using fine-tuned models for ordering and structuring, and a prompt-based model for surface realization (i.e., model hybridization) could yield better results. this is intended to be explored as future work.