Evaluating Generative Models for Graph-to-Text Generation (2023)

Abstract: Large language models (LLMs) have been widely employed for graph-to-text generation tasks. However, the process of finetuning LLMs requires significant training resources and annotation work. In this paper, we explore the capability of generative models to generate descriptive text from graph data in a zero-shot setting. Specifically, we evaluate GPT-3 and ChatGPT on two graph-to-text datasets and compare their performance with that of finetuned LLM models such as T5 and BART. Our results demonstrate that generative models are capable of generating fluent and coherent text, achieving BLEU scores of 10.57 and 11.08 for the AGENDA and WebNLG datasets, respectively. However, our error analysis reveals that generative models still struggle with understanding the semantic relations between entities, and they also tend to generate text with hallucinations or irrelevant information. As a part of error analysis, we utilize BERT to detect machine-generated text and achieve high macro-F1 scores. We have made the text generated by generative models publicly available.

## introduction graph-to-text generation is a subtask of data-totext generation and natural language generation (nlg) @xcite . its purpose is to generate fluent descriptive text based on the structure of a given graph (see figure 1 ). with the widespread use of graph structured data, this technique plays a crucial role in various natural language processing applications, including question answering, dialogue systems, and data augmentation @xcite @xcite . previous research on model architectures has achieved significant performance graph as a linearized sequence of text for input to the models (see figure 1 ). to assess the performance of the generative models, we conduct a comprehensive evaluation on each dataset. employing machine translation metrics such as bleu @xcite , me-teor @xcite , and rouge @xcite to the generated texts, we reveal that the generative models fall short of matching the quality achieved by state-of-the-art approaches. to identify patterns of mistakes made by the generative models, we perform error analysis by comparing the generated texts with the reference texts. additionally, we fine-tune a bert model to detect the machinegenerated text. we make the texts generated by the models available on github to facilitate future research on the analysis of machine-generated text and trustworthy ai. in summary, our study aims to assess the performance of generative models in the zero-shot setting for graph-to-text generation using two distinct benchmarks. our contribution lies in conducting a rigorous quantitative analysis of the results, shedding light on the effectiveness of generative models in this domain. ## related work graph-to-text generation. various efforts have been made to enhance graph-to-text generation using neural network models. they can be categorized into two main types: graph neural network (gnn) based models and language model (lm) based models. gnn-based models typically employ a graph encoder to encode the graph struc-ture @xcite @xcite @xcite . in contrast, lm-based models do not rely on the graph structure but purely on the sequence of tokens in the text. as such, graphs have first been transformed into a linearized representation before being fed into lms to generate coherent text @xcite . besides gnn and lm, previous works have also explored the use of recurrent neural networks (rnns) such as lstm and gru for graph-to-text generation @xcite @xcite . we follow the approach of @xcite and other prior works by using a linearized graph as input for generative models. generative models. generative language models, such as gpt-3 @xcite , instruct-gpt @xcite , and chatgpt, have been designed to learn and generate natural language text. these models are based on the transformer decoder architecture @xcite , which enables them to handle large amounts of training data and perform zero-shot applications. while gpt-3 has made a significant breakthrough in text completion, instructgpt and chatgpt possess unique characteristics that align user intent with a conversational style. these models are trained using supervised fine-tuning and reward modeling, allowing them to generate high-quality responses that accurately reflect the user's needs and preferences. instructgpt and chatgpt are first fine-tuned on the gpt-3 model through supervised learning and then further trained using reinforcement learning based on human feedback. as demonstrated by @xcite , this approach substantially improves the model's performance on nlp benchmarks. although there have been numerous reports and research evaluating the performance of generative models in various nlp applications such as summarization @xcite , machine translation @xcite , and multilingual evaluation @xcite , our work focuses on the generative models' capability to handle structured data. ## dataset we evaluate generative models using the agenda and webnlg datasets, as they are widely used in recent research on graph-to-text generation @xcite @xcite and as they represent different domains: scholarly domain and general domain (e.g., as given in wikipedia). we focus on the test sets of agenda and webnlg for our experiments, as the models do not require further training. in the following, we briefly describe the used datasets. agenda. abstract generation dataset (agenda) is a dataset that pairs knowledge graphs with paper abstracts from scientific domains @xcite . the graphs in agenda were automatically extracted from the sciie information extraction system @xcite . each instance in agenda includes the title, entities, graph, and abstract of a paper. we use the title, entities, and graph as input for the models. webnlg. this dataset is a benchmark for mapping sets of rdf triples to text @xcite . the rdf triples are subgraphs of the knowledge graph dbpedia @xcite , while the texts describe the graphs in one or a few sentences. the webnlg challenge foot_1 has released several versions of this dataset since 2017. in order to compare with previous work, we take the test data of ## error analysis we observe that the texts generated by generative models contain errors following similar patterns. in as shown in the example of furthermore, chatgpt ignores the relations "part-of" and "used-for" in the input. this suggests that generative models still face challenges in comprehending the semantic relations between entities, which can result in the generation of inaccurate text. while generating paper abstracts is complex and challenging, generating short descriptive text from a knowledge graph is relatively more straightforward. as llms, gpt-3 and chatgpt are trained on large corpora and thus already contain world knowledge. in the example provided in ## conclusion in this paper, we explored the capabilities of generative models in generating coherent text from structured data, focusing on two benchmarks: agenda and webnlg. to achieve this, we adopted the linearized graph representation approach employed in prior work. leveraging the zero-shot ability of language models, we prepended the prompt to the