Unifying Structured Data as Graph for Data-to-Text Pre-Training (2024)

Abstract: Data-to-text (D2T) generation aims to transform structured data into natural language text. Data-to-text pre-training has proved to be powerful in enhancing D2T generation and yields impressive performance. However, previous pre-training methods either oversimplified structured data into a sequence without considering input structures or designed training objectives tailored for a specific data structure (e.g., table or knowledge graph). In this paper, we unify different types of structured data (i.e., table, key-value data, knowledge graph) into the graph format and cast different D2T generation tasks as graph-to-text generation. To effectively exploit the structural information of the input graph, we propose a structure-enhanced pre-training method for D2T generation by designing a structure-enhanced Transformer. Concretely, we devise a position matrix for the Transformer, encoding relative positional information of connected nodes in the input graph. In addition, we propose a new attention matrix to incorporate graph structures into the original Transformer by taking the available explicit connectivity structure into account. Extensive experiments on six benchmark datasets show the effectiveness of our model. Our source codes are available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/unid2t.

## introduction data-to-text (d2t) generation, which aims to generate a target natural language text conditioned on source structured data, has attracted noticeable attention due to its many applications such as journalism @xcite , medical diagnosis @xcite , financial and weather re-ports @xcite , and sports broadcasting @xcite . the input structured data can include tables of records, simulations of physical systems, spreadsheets, knowledge graphs, and so on. transforming structured data into textual data can facilitate a wide range of users to understand and use the structured data, which is needed in many real-life scenarios. recently, large-scale pre-trained models have proved to be powerful in d2t generation and yield impressive performance @xcite @xcite , which benefit from the rich knowledge contained in large-scale pre-training corpora. @xcite proposed a structure-aware table-to-text pre-training model, which devised three selfsupervised training objectives tailored for modeling tables and their contexts. @xcite adopted a structure-aware semantic aggregation module to model the structure of an input graph at each transformer layer, and explicitly learned graph-text alignments instead of directly finetuning text-to-text pre-trained models on graphto-text corpora. although significant progress has been made in this field, there are still several technical challenges with existing d2t pre-training methods. most prior studies made a cumbersome design tailored for a specific data structure such as tables @xcite or knowledge graphs @xcite , which could not effectively deal with diverse structured data in a unified framework. @xcite were the first to study the ''pretrain and fine-tune'' strategy on several benchmarks spanning task-oriented dialogue, table-to-text, and graph-to-text. however, they oversimplified the input structured data into a flat string and adopted an original transformer without capturing the structural information of source structured data. in this paper, we unify the structured data into the graph format for data-to-text pre-training (denoted as unid2t). we convert diverse types of structured data into a unified graph format, keeping the structural information of the structured data. we treat the items in the structured data as a set of nodes and connect the nodes according to the connectivity of the structured data. in this way, we can cast various d2t tasks as the graph-to-text generation task. to effectively encode the graph structure, we propose a structure-enhanced pre-training model, which can be applied to various downstream d2t generation tasks. our proposed d2t pre-training model is built upon the t5 model @xcite . since the t5 model is a text-to-text transfer transformer framework and cannot effectively encode the graph structure, we propose a structureenhanced transformer to encode the structural information. concretely, we propose an explicit position matrix for the transformer, encoding the relative positional information of connected nodes in the input graph. in addition, we build a new attention matrix to replace the attention mask in self-attention of the original transformer, which encodes graph structures and takes the available explicit connectivity structure into account. our main contributions are three-fold. (1) we unify diverse types of structured data into a graph format and cast all d2t tasks as the graph-to-text generation task taking a graph as input and producing a text as output. (2) we propose a structureaware pre-training method for d2t generation based on the t5 model, which incorporates relative positional information and graph structures into the original transformer via two new position and attention matrices, respectively. (3) we conduct extensive experiments on six d2t benchmarks and achieve substantially better performance than strong baselines. we believe that the release of our unified d2t pre-training model will advance the research in this area. ## pre-training data construction previous data-to-text pre-training datasets are usually tailored to specific structured data. in this paper, we collect eight d2t datasets from previous works and aggregate these datasets into a large corpus for pre-training our model. the statistics of pre-training data are provided in ## conclusion in this paper, we proposed a unified data-to-text pre-training method, which could be applied to various downstream data-to-text generation tasks. concretely, we first converted different types of structured data into graph format. then, we devised a structure-enhanced transformer to capture graph structures by introducing two new position and attention matrices to replace the position embedding and attention mask in the self-attention of the transformer. extensive experiments on six data-to-text benchmark datasets demonstrated that unid2t achieved substantially better performance than strong baselines by enabling better information sharing and representation learning of data structures across diverse data-to-text datasets.