Evaluating and Improving Graph to Text Generation with Large Language Models (2025)

Abstract: Large language models (LLMs) have demonstrated immense potential across various tasks. However, research for exploring and improving the capabilities of LLMs in interpreting graph structures remains limited. To address this gap, we conduct a comprehensive evaluation of prompting current open-source LLMs on graph-to-text generation tasks. Although we explored the optimal prompting strategies and proposed a novel and effective diversity-difficulty-based few-shot sample selection method, we found that the improvements from tuning-free approaches were incremental, as LLMs struggle with planning on complex graphs, particularly those with a larger number of triples. To further improve LLMs in planning with graph sequences and grounding in truth, we introduce a new graph-to-text dataset, PlanGTG, annotated with two sub-tasks: reordering and attribution. Through extensive automatic and human evaluations, we demonstrate significant improvements in the quality of generated text from both few-shot learning and fine-tuning perspectives using the PlanGTG dataset. Our study paves the way for new research directions in graph-to-text generation.

## introduction recent advancements in large language models @xcite @xcite @xcite have revolutionized natural language processing (nlp) due to their remarkable zero-and few-shot capabilities. while llms have been explored for structured graph tasks @xcite and graph classification @xcite , their potential in verbalizing graphs in natural language (graph verbalization) remains underexplored. graph-to-text generation @xcite is a challenging task that yields text from different graph structures and requires semantic alignment between graph and text. knowledge graphs (kgs) @xcite ,a) store graph-like knowledge in triplets ⟨h, r, t⟩, stating that the head entity h is related to the tail entity t through the relation type r. verbalizing triplets from kgs is essential for a wide range of tasks, such as knowledge graph completion @xcite @xcite as relation prediction, entity typing @xcite and negative triples @xcite for answer validation, as well as creating qa datasets from graph data; e.g. common-senseqa @xcite and scigraphqa @xcite . it also plays a key role in mitigating hallucinations of llms @xcite @xcite . to advance the kg-to-text generation task in the era of llms @xcite , we perform a preliminary evaluation on how well open-source llms perform on different prompts both in zero and few-shot scenarios. the prompt searching results emphasized the role of detailed instructions in unleashing llms' potential to generate fluent and accurate text from graphs. at the same time, we did not observe improvements of llms over various prompt optimizations in the zero-shot case (e.g. different linearizations and triplet expressions). for few-shot prompts, we empirically showed that choosing moderately hard and diverse prompts results in the best performance and propose a novel difficulty-diversity balanced demonstration selection method (ddd), which outperforms both simple difficulty or diversity-based method. however, the absolute value of improvements is marginal, reflecting the limitations of in-context learning. the analysis of graphs with varying complexity, measured by the number of triplets and graph diam-eters (the longest shortest path between any two vertices), further suggests that current llms struggle with planning when handling graphs with a lot of triplets and with small diameters. this motivates us to use instruction-tuning over sub-tasks to strengthen llms' planning abilities, thus improving the performance of graph-to-text generation. inspired by @xcite , we investigate whether instructing llms to explicitly output their decision process for graph-to-text generation can improve the quality of generated text. in addition, we design two subtasks: 1) reordering: reorder the given kg triplets to better align with the generated text. 2) attribution: attribute triplet indexes in the generated text using sequential numbers to enhance the interpretability of the generated text. to achieve this, we create plangtg (sec.4), a new instruction dataset containing approximately 30,000 data pairs, featuring annotated attributions of triplets from the text. this dataset is generated using seeds rewritten from graphnarritive @xcite and gpt-3.5-turbo, incorporating the two subtasks sequentially. we fine-tune llms with plangtg (sec.5), conducting extensive automatic and human evaluations. human evaluation results suggest that models finetuned with instructions are capable of successfully adjusting the order of kg triplets and correctly marking the sequence numbers in the generated text in most cases. comparatively, models trained with plangtg outperform those using the event-narrative @xcite , tekgen @xcite , and graphnarrative @xcite datasets in both zero-shot generalization and fullshot fine-tuning. in summary, our main contributions are: ## related work various approaches have been proposed for transforming knowledge graphs into text. these include graph neural network based methods @xcite and language model based approaches @xcite @xcite . graph neural network based methods typically encode structured inputs explicitly as model representations @xcite @xcite . among llm based approaches, after llm based sub-graph retrieval @xcite , a critical step is linearizing input triplets @xcite , and recent efforts have introduced various planning techniques @xcite , including effective cot based summaries @xcite . there are studies that have focused on evaluating graph-to-text generation quality using pre-trained language models. @xcite address hallucinations in open-domain graph-to-text generation, while @xcite evaluate closedsource llms such as gpt-3 and chatgpt only under the zero-shot setting. in contrast, our work focuses on open-source llms, aiming to provide a comprehensive evaluation across various aspects of the graph-to-text conversion process, including linearization, demonstration selection, and model scaling. furthermore, we propose two new subtasks, reordering and attribution. these tasks are designed to enhance transparency in the generation process of llms and improve their performance. broadly speaking, there are also works on using knowledge graphs and ontologies for text generation @xcite @xcite . ## preliminary study to investigate how to improve the performance of llm's graph-to-text generation, we first perform a preliminary study to answer 1) how can different prompts influence the performance of graph-to-text generation, and will prompt engineering be relevant to attain such improvements? 2) will scaling the parameters of open-sourced llms improve their graph-to-text verbalization abilities? we performed experiments following the settings in section 3.1. for detailed experiment results, please refer to app. ## the plangtg dataset in the perspective of fine-tuning for improving the performance of graph-to-text generation, we construct plangtg (planning for graph-to-text generation), a graph-to-text paired instructiontuning dataset, annotated for both reordering and attribution subtasks. the construction of plangtg is guided by several key objectives: (1) ensuring the diversity of both the structure of graphs (size and diameter) and the topic of texts; (2) avoiding textgraph misalignment (i.e. textual descriptions containing information not found in input graphs), a significant cause of hallucination @xcite ; and (3) ensuring interpretability by annotating the attribution triplets from the text descriptions and automatically formatting the linearization labels. these goals are achieved through sequential generation and parallel annotation. ## experiments 5.1 experimental setting to evaluate whether plangtg can enhance llms generalizability in graph-to-text tasks, we conducted experiments on both zero-and full-shot learning. in our zero-shot experiments, as shown in (b) of figure 1 , we fine-tuned llms on the plangtg dataset and subsequently evaluated their performance on webnlg 2017, webnlg 2020, and dart datasets. for the baselines, we finetuned llms on other graph-to-text generation datasets. we compared three classic datasets: event (eventnarrative) @xcite , gn (graphnarrative) @xcite , and tekgen @xcite . in our training phases, we use the commonly employed cross-entropy loss for generation tasks to align the model's predictions with the grounding generation, which is our plangtg dataset. in the full-shot experiments, after fine-tuning on plangtg, we continued to finetune the models on the training sets of webnlg 2017. for more details on our training experiments, please refer to the app. e. ## conclusion we have conducted a comprehensive analysis over both zero-and few-shot scenarios in graph-to-text generation tasks to evaluate the capabilities and challenges of llms. our findings reveal that (1) llms struggle to understand complex graphs and (2) moderately difficult and diverse demonstrations may help llms for translating graphs to text. based on these findings, we propose solutions from both few-shot learning and fine-tuning perspectives to enhance the effectiveness of llms. in terms of few-shot prompting, we propose the ddd method to select samples considering both difficulty and diversity simultaneously, leading to improvements in both the one-shot and few-shot cases. for model fine-tuning, we construct a highquality graph-to-text dataset, plangtg, and develop two new subtasks. fine-tuning llms on plangtg demonstrates a significant improvement in the alignment generation and generalization abilities of llms. additionally, through learning from simple to complex data, the model's ability to generate text from graphs is further enhanced. our work lays the groundwork for future research aimed at effectively enabling llms to reorganize graph structures and identify sequential information in generated texts.