ASDOT : Any-Shot Data-to-Text Generation with Pretrained Language Models (2022)

Abstract: Data-to-text generation is challenging due to the great variety of the input data in terms of domains (e.g., finance vs sports) or schemata (e.g., diverse predicates). Recent end-to-end neural methods thus require substantial training examples to learn to disambiguate and describe the data. Yet, real-world data-to-text problems often suffer from various data-scarce issues: one may have access to only a handful of or no training examples, and/or have to rely on examples in a different domain or schema. To fill this gap, we propose Any-Shot Data-to-Text (ASDOT), a new approach flexibly applicable to diverse settings by making efficient use of any given (or no) examples. ASDOT consists of two steps, data disambiguation and sentence fusion, both of which are amenable to be solved with off-the-shelf pretrained language models (LMs) with optional finetuning. In the data disambiguation stage, we employ the prompted GPT-3 model to understand possibly ambiguous triples from the input data and convert each into a short sentence with reduced ambiguity. The sentence fusion stage then uses an LM like T5 to fuse all the resulting sentences into a coherent paragraph as the final description. We evaluate extensively on various datasets in different scenarios, including the zero-/few-/full-shot settings, and generalization to unseen predicates and out-of-domain data. Experimental results show that ASDOT consistently achieves significant improvement over baselines, e.g., a 30.81 BLEU gain on the DART dataset under the zero-shot setting.

## introduction data-to-text generation @xcite aims at generating natural language text conditioned on structured data content such as tables and graphs. the task has a broad range of applications such as task-oriented dialog (wen et al., 1 code available at https://github.com/szxiangjn/ any-shot-data2text 2015), weather forecasting @xcite , sports news reporting @xcite , and biography generation @xcite . the problem is challenging in practice due to the vast diversity of the input data in terms of the domains (e.g., finance vs sports), schemata (e.g., the set of predicates, table structures), etc. the inherent ambiguity makes it particularly difficult to learn to understand and describe the data. for instance, in the @xmath0, time, 2008> from a music domain, the predicate word time means the release time of an album, while @xmath1, time, 9.58> from sports it expresses the world record time. recent approaches based on end-to-end neural models, e.g., by finetuning pretrained language models (lms) @xcite @xcite , typically require massive training instances to resolve the ambiguity and are not applicable to many data-scarce scenarios. in practice, a data-to-text problem of interest may have a varying number of training examples, ranging from a (small) set to only a few shots, or even no examples at all, and sometimes may rely on available examples out of the current domain to facilitate the generation. we refer to the diverse practical scenarios as the any-shot data-totext problems. recent work has studied data-to-text solutions when limited examples are available, but is often restricted to single specific settings. for instance, @xcite and @xcite focused on few-shot problems but fail to apply when no examples are accessible, while the zero-shot neural pipeline by @xcite relies on human-crafted templates and thus could not handle out-of-domain data. in this paper, we develop any-shot data-to-text (asdot), a new flexible approach that makes efficient use of any given (or no) examples and achieves stronger generation quality compared to the prior specific methods. asdot draws inspiration from how humans describe data, namely by first disambiguating and understanding the data content, and then fusing and organizing the information together into text paragraphs. as a result, given input data (e.g., a table or graph), asdot consists of two intuitive steps, i.e., data disambiguation and sentence fusion. importantly, each of the two steps is amenable to be solved with the appropriate off-the-shelf pretrained lms with optional finetuning, enabling the unique flexibility of asdot in the presence of any-shot training examples. more specifically, in data disambiguation aiming to understand each data entry (e.g., @xmath2, time, 2008>), we use the prompted gpt-3 model @xcite , which has encoded rich commonsense and world knowledge, to convert the triple into a short sentence (fearless was released in 2008) with greatly reduced ambiguity. the subsequent sentence fusion stage then uses another lm, such as t5 @xcite , to combine all the resulting sentences into a coherent paragraph as the final description. the sentence fusion as a sub-task allows us to incorporate any available in-/out-of-domain training examples as well as existing large weakly supervised corpus @xcite to finetune the lm and boost the performance. we evaluate the proposed approach in a wide range of practical any-shot scenarios, including (1) the zero-/few-/full-shot setting where we have access to a varying number of training examples, (2) the unseen-predicates setting where we describe the data of new predicates that are never seen in the training examples, and (3) the out-of-domain setting where we are presented only with examples from other domains. extensive experiments show that our approach consistently achieves significant gains over the diverse previous methods specifically designed for each of the different scenarios. ## related work data-to-text (d2t) generation is a long-standing problem in natural language processing with broad applications in practice. early research on this task focused on rule-based and pipeline approaches @xcite , decomposing the task into text planning, sentence planning, and linguistic realisation. recent work has developed various neural approaches. @xcite used a neural encoder-decoder for the task, followed by attention @xcite , content selection @xcite , entity modeling @xcite , and style imitation @xcite for further improved performance. recent studies have also incorporated pretrained lms @xcite @xcite . although previous fully-supervised methods have achieved remarkable performances, most of them require a large amount of in-domain training examples, leading to limited applicability to the common low-data scenarios in practice. recent interests are aroused in zero-/few-shot data-to-text generation problems. @xcite first formulated the few-shot setting and incorporated a pretrained model with a pointer generator as a solution. @xcite developed a knowledge-grounded pretrained lm for both zeroand few-shot data-to-text generation. @xcite and @xcite proposed to solve the few-shot task with content matching and prototype memory, respectively. there are also studies on combining templates and pretrained lm for zero-/few-shot generation. for example, @xcite trained a neural model to rewrite templates for few-shot task-oriented dialogue. @xcite applied the idea of template rewriting to build a practical few-shot data-to-text system. most of the previous methods have each focused on a specific setting (e.g., either zero-or few-shot). in comparison, our work studies a wide spectrum of any-shot scenarios with a varying number of training examples from current or different domains. of particular relevance to our work is the approach by @xcite , which performs zeroshot data-to-text generation by rephrasing given templates. however, the approach relies on humanwritten templates for data disambiguation and thus has limited applicability to wide domains. besides, the approach involves several components (ordering, aggregation, compression) to fuse sentences, which restricts the use of any-shot examples for improvement. the approach thus studies only in zero-shot settings, while our work makes a comprehensive study on the diverse any-shot problems. ## any-shot data-to-text generation we propose asdot for any-shot data-to-text generation. §3.1 describes the any-shot problems. we then provide an overview of our method ( §3.2) and prompt 2 table: michael | birth place | usa text: michael was born in the usa. …… table: buzz aldrin | birthplace | glen ridge new jersey text: prompt 1 table: michael | birth place | usa text: michael was born in the usa. …… table: apollo 11 | operator | nasa text: b ir th p la ce se le ct ed by n a sa buzz aldrin glen ridge new jersey essex county new jersey is p ar to f cr ew m em be r operat or apollo 11 ba cku pp ilo t william anders 1963 @xmath3, operator, @xmath4, birthplace, glen ridge new @xmath5. buzz aldrin was born in glen ridge, new jersey. …… …… gpt-3 data disambiguation sentence fusion final output buzz aldrin was born in glen ridge, essex county, new jersey. he went on to become a crew member on apollo 11 which was operated by nasa in 1963. william anders was the backup pilot for apollo 11.… short sentences plm (e.g. t5) weaklysupervised finetuning any-shot finetuning give details of each of the components ( §3.3, 3.4). figure 1 illustrates our method. ## conclusion we have proposed asdot to deal with the diverse any-shot problems for data-to-text generation. as-dot is composed of two stages, data disambiguation that uses prompted gpt-3 to disambiguate input data triples into short sentences, and sentence fusion using state-of-the-art pretrained lms to fuse these sentences into the desired paragraphs. in the process, asdot integrates rich external implicit knowledge from the large lms, which ensures strong generalization capability and broad applicability to zero-/few-/full-shot, unseen-predicates, and out-of-domain training scenarios. extensive experiments show our approach consistently achieves significant improvements over diverse baselines.