Neural Pipeline for Zero-Shot Data-to-Text Generation (2022)

Abstract: In data-to-text (D2T) generation, training on in-domain data leads to overfitting to the data representation and repeating training data noise. We examine how to avoid finetuning pretrained language models (PLMs) on D2T generation datasets while still taking advantage of surface realization capabilities of PLMs. Inspired by pipeline approaches, we propose to generate text by transforming single-item descriptions with a sequence of modules trained on general-domain text-based operations: ordering, aggregation, and paragraph compression. We train PLMs for performing these operations on a synthetic corpus WikiFluent which we build from English Wikipedia. Our experiments on two major triple-to-text datasets—WebNLG and E2E—show that our approach enables D2T generation from RDF triples in zero-shot settings.

## introduction the aim of data-to-text (d2t) generation is to produce natural language descriptions of structured data @xcite . although pipelines of rule-based d2t generation modules are still used in practice @xcite , end-to-end approaches based on plms recently showed superior benchmark performance @xcite @xcite @xcite , surpassing pipeline systems @xcite in both automatic and human evaluation metrics. finetuning plms on human-written references is widely accepted as a standard approach for adapting plms to the d2t generation objective and achieving good performance on a given benchmark @xcite . however, finetuning a model on the domain-specific data leads to overfitting to the particular benchmark, decreasing performance on out-of-domain 1 our code and data is available at https://github. com/kasnerz/zeroshot-d2t-pipeline . in-domain knowledge is included only in the simple hand-crafted templates for each predicate. in this paper, we present a zero-shot alternative to the traditional finetuning paradigm by formulating the d2t generation from rdf triples as a sequence of general-domain operations over text in natural language. we start by transforming individual triples to text using trivial templates, which we subsequently order, aggregate, and compress on the paragraph level to produce the resulting description of the data. in constrast to traditional pipeline systems, all our pipeline modules are built upon plms and operate over sentences in natural language. the modules are trained on our new wiki-fluent corpus, which contains 934k examples of first paragraphs from the english wikipedia, each supplied with a synthesized set of simple templatelike sentences which together convey the meaning of the original paragraph. our approach allows generating natural language descriptions from rdf triples with a minimum amount of domain-specific rules or knowledge and without using training data from the d2t datasets. although our approach is primarily a probe into the territory of zero-shot approaches and cannot yet match the quality of stateof-the-art models, we show that it can yield large improvements upon simple baselines and match older supervised systems on automatic metrics for text fluency. moreover, the semantic accuracy metrics and our manual error analysis suggest that our approach offers a way to prevent omissions and hallucinations common in few-shot approaches. our contributions are the following: @xcite we propose an alternative d2t generation approach based on general-domain text-to-text operations (ordering, aggregation, and paragraph compression). (2) we introduce a synthetic wikifluent corpus containing 934k sentences based on english wikipedia, providing training data for the operations in ( 1 ). ( 3 ) we apply our system on two d2t datasets and evaluate its performance both automatically and manually, including the contribution of individual pipeline modules. (4) we release our code, data, pretrained models, and system outputs to ease future research. 1 2 related work d2t generation with plms large neural language models pretrained on self-supervised tasks @xcite @xcite have recently gained a lot of traction in d2t generation research @xcite . following @xcite , other works adopted plms for few-shot d2t generation @xcite . @xcite and @xcite showed that plms using linearized representations of data can outperform graph neural networks on graph-to-text datasets, recently surpassed again by graph-based models @xcite . although the models make use of generaldomain pretraining tasks, all of them are eventually finetuned on domain-specific data. pipeline-based d2t generation until the recent surge of end-to-end approaches @xcite , using several modules connected in a pipeline was a major approach for d2t generation @xcite @xcite . our approach is inspired by the pipeline approaches, in particular the pipelines utilizing neural modules @xcite . in contrast with these approaches, our pipeline works with unstructured data in natural language and it operates in zero-shot setting, i.e. without using any training data from target d2t datasets. @xcite introduce a three-step pipeline for zero-shot d2t generation similar to ours. unlike the approach we describe here, they use a semiautomatic template generation system, foot_0 their sentence fusion is rule-based, and they do not address content planning. ## method in this section, we provide the formal description of our proposed approach. we focus on the task of producing a natural language description y for a set of n rdf triples x " tx 1 , . . . , x n u. each triple x i " ts i , p i , o i u consists of subject s i , predicate p i , and object o i . our pipeline proceeds as follows. given a set of triples x on the input, we: (1) transform the triples into facts, which are sentences in natural language, (2) sort the facts using an ordering module, (3) insert sentence delimiters between the sorted facts using an aggregation module, (4) input the ordered sequence of facts with delimiters into a paragraph compression module, which generates the final description y . the individual steps are described in the following sections: transforming individual triples to text ( §3.1), ordering ( §3.2), aggregation ( §3.3), and paragraph compression ( §3.4). ## wikifluent corpus here we descibe the process of building a largescale synthetic corpus wikifluent. the corpus provides training data for the neural models which we use in our implementation of the ordering, aggregation, and paragraph compression modules (cf. §5). our goal is to cover a broad range of domains while capturing the sentence style in d2t generation with respect to both the input facts and the target descriptions. in other words, we aim to build a corpus in which (1) the input is a set of simple, template-like sentences, (2) the output is a fluent text in natural language preserving the semantics of the input. as we describe below in detail, we achieve that by using human-written paragraphs in english wikipedia and applying split-and-rephrase and coreference resolution models to obtain synthetic source texts. the process is illustrated in figure 2 ; corpus statistics are included in appendix a. ## implementation in this section, we describe how we implement our pipeline modules ( §3) using simple template transformations ( §5.1) and neural models trained on the wikifluent dataset ( §5.2-5.4). 5 ## experiments we train our pipeline modules on the wikiflu-ent corpus as described in §5. next, we use these modules without finetuning for generating descriptions for rdf triples on two english d2t datasets, webnlg and e2e. ## evaluation and discussion our main aim is the evaluation of our pipeline on the downstream task of d2t generation. we evaluate outputs from the {1,2,3}-stage variants of our pipeline using automatic metrics ( §7.1), and we perform a detailed manual error analysis of the model outputs ( §7.2). we also evaluate the performance of the content planning modules and the ability of the pc module to follow the content plan ( §7.3). in §7.4, we include an intrinsic evaluation of our modules on the wikifluent test set. ## future work our experiments outline several possible future research directions. automatic generation of facts without using hand-crafted templates (cf. §5.1) could allow applying zero-shot generation systems to datasets with a large number of predicates, such as totto @xcite . the task of paragraph compression could be used as a task-specific pretraining @xcite for more efficient finetuning of d2t models, e.g., with a small amount of clean data. consistency checks may be introduced in the pipeline to control the output from the modules, and individual modules may be improved by using more efficient model architectures. more research is also needed regarding the main shortcoming of our approach, i.e., the semantic errors stemming from merging of facts in improper ways. as we suggested in §7.2, explicitly controlling the semantics of sentence fusion could help to mitigate this issue, while still keeping the advantages of a zero-shot approach. ## conclusion we presented an approach for zero-shot d2t generation. the approach uses a pipeline of plms trained on general-domain lexical operations over natural language. the pipeline builds upon traditional approaches and consists of three interpretable intermediate steps. by avoiding noisy human-written references from the d2t datasets, our models produce more semantically consitent output. we believe that training models for zeroshot d2t generation using large cross-domain corpora will help to build d2t generation systems with good performance across various domains.