What Makes Data-to-Text Generation Hard for Pretrained Language Models? (2022)

Abstract: Expressing natural language descriptions of structured facts or relations – data-to-text generation (D2T) – increases the accessibility of structured knowledge repositories. Previous work shows that pre-trained language models (PLMs) perform remarkably well on this task after fine-tuning on a significant amount of task-specific training data. On the other hand, while auto-regressive PLMs can generalize from a few task examples, their efficacy at D2T is largely unexplored. Furthermore, we have an incomplete understanding of the limits of PLMs on D2T. In this work, we conduct an empirical study of both fine-tuned and auto-regressive PLMs on the DART multi-domain D2T dataset. We consider their performance as a function of the amount of task-specific data and how the data is incorporated into the models: zero and few-shot learning, and fine-tuning of model weights. In addition, we probe the limits of PLMs by measuring performance on subsets of the evaluation data: novel predicates and abstractive test examples. To improve the performance on these subsets, we investigate two techniques: providing predicate descriptions in the context and re-ranking generated candidates by information reflected in the source. Finally, we conduct a human evaluation of model errors and show that D2T generation tasks would benefit from datasets with more careful manual curation.

## introduction structured data repositories, or knowledge bases, contain a wealth of information organized to facilitate automated access and analysis. automated data-to-text (d2t) generation systems can transform and organize this knowledge into natural language text snippets that enable broader access @xcite . these systems take as input a set of relations, where each relation is a (subject, predicate, object) triple. applications of this technology include story or dialogue generation @xcite , open-domain question-answering @xcite , and text summarization @xcite . domains span journalism @xcite , weather @xcite , finance, sports @xcite @xcite , and summarizing patient medical histories @xcite . historically, d2t systems included pipeline approaches with customized models @xcite , but have now shifted to pretrained transformer-based language models (plms) @xcite @xcite . recent examples include @xcite and @xcite , who use models like gpt-2 @xcite and t5 @xcite to generate natural language descriptions for relations. to support these types of systems, @xcite introduced dart, a large open-domain data-to-text generation corpus. models trained on dart, both larger and more diverse than previous corpora, improve the performance of bart @xcite and t5 on the standard webnlg challenge @xcite . this approach requires a plm to be fine-tuned on a task-specific in-domain dataset @xcite @xcite . the promising results achieved by fine-tuning on dart belie the reality -in spite of dart's aspirations, most domains and relations that one could express fail to appear in dart. a variety of methods have emerged within plm research to address domain or task adaptation. for example, auto-regressive models, like gpt, have demonstrated improved performance on a wide range of tasks via few-shot learning from a handful of examples @xcite . other strategies, such as prompt tuning @xcite , can adapt plms to specific down-stream tasks by up-dating only a small subset of model parameters. while great progress has been made in utilizing plms for d2t generation, the path forward is unclear, as we have an incomplete understanding as to which examples they fall short on and the quantity of training resources they need to achieve acceptable performance. more specifically, it is not clear which classes of d2t examples are challenging for these models. in addition, we do not fully understand what classes of errors plms are prone to and how the adaptation mechanism (e.g., k-shot learning, fine-tuning) affects the prevalence of these errors. in this work, we conduct an evaluation of plms for d2t generation, focusing on two classes of challenging examples: examples with novel (unseen) relations (predicates) and examples where the source and target sequences are lexically very different (not amenable to purely extractive d2t systems). we consider how gpt-2, adapted with few-shot learning, prompt tuning, and the addition of predicate descriptions, performs on these example classes as compared to a state-of-the-art fine-tuned t5. we show that while gpt-2 performs poorly on dart in the 0-shot setting, its performance can be drastically improved by employing the above techniques. we make the following contributions: finally, we provide recommendations for future model and dataset research in d2t generation. ## background and related work in the task of data-to-text generation, we are provided a set of triples that include a predicate, subject, and object. the system then produces a text snippet expressing the predicate in natural language. figure 2 shows examples of predicates about sports. the system can be given a set of triples with related predicates (e.g., club, league, former_team) and must generate text that expresses the facts encoded by these relations. the resulting text is typically evaluated by comparison to a set of reference texts, which represent various ways of expressing this triple set. variations in the formulation of this task depend on the structure of the relations (e.g., tables, triples), the domain of the task (single or open domain), and the source of the data (manually created, automatically derived). harkous et al. (2020) follow a generate-and-rerank paradigm to improve the semantic fidelity of the generated text by fine-tuned gpt-2 model. more recently, @xcite propose a new task-adaptive pretraining strategy to adapt bart @xcite and t5 @xcite models for data-to-text generation. they show that adding an intermediate task-adaptive pretraining step between the task-independent pretraining and fine-tuning further improves the performance of these models on data-to-text generation. despite the progress of these models, it is not clear which types of d2t examples are most challenging for plms or what errors are prevalent in generations. futhermore, how does plm adaptation (tuning/prompting) interact with the occurrence of these errors. on the other hand, d2t datasets are not readily available in many domains. weakly supervised annotation methods (e.g., based on identifying sentences in a corpus that are likely to express a data record) require significant manual effort and often result in annotations with low fidelity between data records and the corresponding textual expression @xcite . training nlg models on such data can result in generations with missing information or hallucination @xcite . these issues render the path forward for d2t generation research zero-shot prompt translate graph to english: graph: <@xmath0.c. english: zero-shot prompt + relation description translate graph to english: definition: club is an organization of players and managers associated with a particular football team. graph: <@xmath1.c. english: ## model adaptation as a supervised task, d2t generation systems rely on previously observed examples to learn the correct generation or level of required "re-writing" for a predicate. on the other hand, large autoregressive plms (such as gpt2-xl) are able to perform d2t generation without any explicit finetuning at all. however, their efficacy on d2t and potential shortcomings are largely unexplored. how well do plms perform on relations with a novel predicate? do plms overly rely on copying verbatim from the input or are they capable of abstraction when required? what classes of errors are prevalent in the generations and how do they interact with the choice of adaptation mechanism? our focus is on the analysis of plms for d2t generation. we study this problem using two types of plms: auto-regressive models like gpt-2 and "supervised" models like t5 @xcite . while prior work has demonstrated that t5 achieves state-ofthe-art results on d2t, these "supervised" models foot_0 expect task-specific training data, whereas generative plms excel at adapting to new tasks. since auto-regressive models have not been fully benchmarked for d2t, we will evaluate them in multiple settings and compare to t5. for both, we will explore the effect of varying training size and their pathological behaviors. while plms can be fine-tuned, their increasing size and training requirements disfavors this approach. instead, current work assumes a single plm capable of performing multiple downstream tasks @xcite . we adopt gpt2-xl, a decoder-only transformer @xcite with 1.5b parameters pre-trained for language modeling @xcite . 2 we utilize gpt2-xl as a d2t generation model by varying the amount of supervised information available. instead of fine-tuning gpt2-xl, we investigate both few-shot learning @xcite , which is better suited to settings where little training data is available, and prompt tuning, which enables us to tractably update a subset of model weights in spite of gpt2-xl's large parameter count. ## experimental setup model training we use the pretrained models gpt2-xl and t5 large released by hugging face @xcite , along with their respective tokenizers, for all experiments. we use beam search with beam size of three for decoding in all models, lightly post-processing the generated text by truncating generations at the newline character. we set maximum generated tokens to 100 and repetition penalty to 1.01 for all experiments. we used a single v100 gpu with 32gb of memory for all prompt tuning experiments, tuning for a single epoch on the dart train set with prefix and infix length both set to 8 tokens. we use the adam optimizer (kingma and ba, 2014) with a maximum learning rate of 0.1 and 100 warm-up steps for the linear learning rate schedule. the training batch size was fixed to 2, with 32 gradient accumulation steps (effective batch size of 64 examples). we use the scripts from ribeiro et al. (2020) to fine-tune t5 on dart, using identical hyperparameter settings. 8 we use the adam optimizer with an initial learning rate of 3e-5 and a linearly decreasing learning rate schedule. we fine-tune the model on four gpus for a maximum of 100 epochs and stop training early if the performance does not improve on the dev set for 15 epochs. each training epoch takes approximately two hours for each model. finally, we include a baseline system to benchmark the performance of our machine learning models. in a "copy baseline" we simply copy the input text and remove the prefix tokens (<h>, <r>, <@xmath2.g., underscores) common in dart predicates. this baseline performs well for examples with high lexical overlap between input triple set and reference. evaluation metrics following previous work, we use automated metrics such as bleu @xcite , meteor @xcite , translation edit rate (ter) @xcite , and chrf++ @xcite for evaluating our generation results. in addition, we also report bertscore @xcite and bleurt @xcite . these metrics go beyond surface form similarities and use contextual embeddings to measure semantic similarity between the generated and reference text. foot_8 ## experiments we evaluate plms with various input types and training regimes to answer the following empirical questions: ## conclusion and future work in this work, we systematically analyze the performance of two plms -t5 and gpt2-xl -for d2t generation by examining performance based on the choice of adaptation mechanism: fine-tuning, prompt tuning, and few-shot learning. we observe that while fine-tuning on more data leads to better performance, when no training data is available, gpt2-xl (0-shot) outperforms t5. with a small number of training examples, few-shot gpt2-xl is a more appropriate solution for d2t. we also conduct a thorough investigation of d2t challenges for plms by evaluating them on two divisions of the dart test set: novel predicates and abstractive examples. we show that the performance of fine-tuned t5 drops significantly on unseen predicates. on the other hand, the performance of few-shot gpt2-xl on unseen predicates can be enhanced even with shots containing unrelated predicates. we also notice that t5 and gpt2-xl both do well at d2t by copying the input. however, they do noticeably worse on examples where significant re-writing is needed. adding domain knowledge (predicate descriptions) to the prompts can improve the performance of few-shot gpt2-xl on this subset by a large amount. we also conduct a human evaluation of the generations and find that prompt tuned gpt2-xl generations can be improved by re-ranking generations by overlap with the input entity spans. future work in d2t generation should consider more challenging examples, and should consider ways in which to generate more diverse variations for expressing a given predicate. this should include more challenging and disparate domains, such as finance or medicine. in these cases, one may see benefits from including predicate descriptions, which performed well on the most abstractive examples.