Data-to-text Generation with Variational Sequential Planning (2022)

Abstract: We consider the task of data-to-text generation, which aims to create textual output from non-linguistic input. We focus on generating long-form text, that is, documents with multiple paragraphs, and propose a neural model enhanced with a planning component responsible for organizing high-level information in a coherent and meaningful way. We infer latent plans sequentially with a structured variational model, while interleaving the steps of planning and generation. Text is generated by conditioning on previous variational decisions and previously generated text. Experiments on two data-to-text benchmarks (RotoWire and MLB) show that our model outperforms strong baselines and is sample-efficient in the face of limited training data (e.g., a few hundred instances).

## introduction data-to-text generation refers to the task of generating textual output from non-linguistic input such as database tables, spreadsheets, or simulations of physical systems @xcite @xcite . recent progress in this area @xcite @xcite has been greatly facilitated by the very successful encoder-decoder neural architecture @xcite and the development of large scale datasets. rotowire @xcite and mlb @xcite constitute such examples. they both focus on the sports domain, which has historically drawn attention in the generation community @xcite @xcite and consider the problem of generating long target texts from database records. figure 1 (reproduced from puduppully and lapata, 2021) provides a sample from the mlb dataset, which pairs human written summaries (table c in figure 1 ) with major league baseball game statistics. these are mostly scores (collectively referred to as box score) which summarize the performance of teams and players, for example, batters, pitchers, or fielders (table a in figure 1 ) and a play-by-play description of the most important events in the game (table b in figure 1 ). game summaries in mlb are relatively long (540 tokens on average) with multiple paragraphs (15 on average). the complexity of the input and the length of the game summaries pose various challenges to neural models which, despite producing fluent output, are often imprecise, prone to hallucinations, and display poor content selection @xcite . attempts to address these issues have seen the development of special-purpose modules that keep track of salient entities @xcite , determine which records (see the rows in tables a and b in figure 1 ) should be mentioned in a sentence and in which order @xcite , and reconceptualize the input in terms of paragraph plans @xcite to facilitate document-level planning (see table d in figure 1 ). specifically, @xcite advocate the use of macro plans for improving the organization of document content and structure. a macro plan is a sequence of paragraph plans, and each paragraph plan corresponds to a document paragraph. a macro plan is shown in table e (figure 1 ). examples of paragraph plans are given in table d @xmath0. verbalizations are sequences of record types followed by their values. document paragraphs are shown in table c and have the same color as their corresponding plans in table e . during training, @xcite figure 1 : example from the mlb dataset reproduced from @xcite with the authors' permission. table a is typically referred to as box score. it summarizes the data of the game per team and player. table b reports statistics pertaining to innings or play-by-play scores. table c contains the game summary. paragraphs in table c are separated @xmath1. table d contains paragraph plans obtained from tables a and b . paragraph plans in the first column correspond to a single entity or event. paragraph plans in the second column describe combinations of entities or events. <v(entity)> verbalizes records pertaining to entities @xmath2. paragraph plans correspond to paragraphs in table c . table e contains the macro plan for the document in table c . a macro plan is a sequence of paragraph plans. plan-document correspondences are highlighted using the same color. lapata (2021) learn to predict a macro plan from a pool of paragraph plans, and produce a game summary based on it. continuing with our example in figure 1 , plan (e) is obtained from paragraph plans (d), to give rise to game summary (c). the intermediate macro plan renders generation more interpretable (differences in the output can be explained by differences in macro planning). it also makes modeling easier, the input is no longer a complicated table but a sequence of paragraph plans, which in turn allows us to treat data-to-text generation as a sequence-to-sequence learning problem. nevertheless, decoding to a long document remains challenging for at least two reasons. firstly, the macro plan may be encoded as a sequence but a very long one (more than 3,000 tokens), which the decoder has to attend to at each time step in order to generate a summary token-by-token. secondly, the prediction of the macro plan is conditioned solely on the input (i.e., pool of paragraph plans (d) in figure 1 ) and does not make use of information present in the summaries. we hypothesize that planning would be more accurate were it to consider information available in the table (and corresponding paragraph plans) and the generated summary, more so because the plans are coarse-grained and there is a one-to-many relationship between a paragraph plan and its realization. for example, we can see that the plan @xmath3.keller)> results in two very different realizations in the summary in figure 1 (see first and third paragraph). in this work, we present a model which interleaves macro planning with text generation (see figure 2 for a sketch of the approach). we begin by selecting a plan from a pool of paragraph plans (see table d in figure 1 ), and generate the first paragraph by conditioning on it. we select the next plan by conditioning on the previous plan and the previously generated paragraph. we generate the next paragraph by conditioning on the currently selected plan, the previously predicted plan, and generated paragraph. we repeat this process until the final paragraph plan is predicted. we model the selection of paragraph plans as a sequential latent variable process, which we argue is intuitive since content planing is inherently latent. contrary to puduppully and lapata (2021), we do not a priori decide on a global macro plan. rather, our planning process is incremental and as a result less rigid. planning is informed by generation and vice versa, which we argue should be mutually beneficial (they are conditioned on each other). during training, the sequential latent model can better leverage the summary to render paragraph plan selection more accurate and take previous decisions into account. we hypothesize that the interdependence between planning and generation allows the model to cope with diversity. in general, there can be many ways in which the input table can be described in the output summary, that is, different plans give rise to equally valid game summaries. the summary in figure 1 (table c ) focuses on the performance of brad keller, who is a high scoring pitcher (first three paragraphs). an equally plausible summary might have discussed a high scoring batter first (e.g., ryan o'hearn). also notice that the summary describes innings in chronological order. however, another ordering might have been equally plausible, for example, describing innings where the highest runs are scored first or innings that are important in flipping the outcome of the match. in the face of such diversity, there may never be enough data to learn an accurate global plan. it is easier to select a paragraph plan from the pool once some of the summary is known, and different plans can be predicted for the same input. in addition, the proposed model is end-to-end differentiable and gradients for summary prediction also inform plan prediction. our contributions can be summarized as follows: (1) we decompose data-to-text generation into sequential plan selection and paragraph generation. the two processes are interleaved and generation proceeds incrementally. we look at what has been already generated, make a plan on what to discuss next, realize the plan, and repeat; (2) in contrast to previous models @xcite , where content plans are monolithic and determined in advance, our approach is more flexible, it simplifies modeling (we do not need to learn alignments between paragraph plans and summary paragraphs), and leads to sample efficiency in low resource scenarios; (3) our approach scales better for tasks involving generation of long multi-paragraph texts, as we do not need to specify the document plan in advance; and (4) experimental results on english and german rotowire @xcite , and mlb @xcite show that our model is well-suited to long-form generation and generates more factual, coherent, and less repetitive output compared to strong baselines. we share our code and models in the hope of being useful for other tasks (e.g., story generation, summarization). foot_1 2 related work a long tradition in natural language generation views content planning as a central component to identifying important content and structuring it appropriately @xcite . earlier work has primarily made use of hand-crafted content plans with some exceptions that pioneered learning-based approaches. for instance, @xcite learn ordering constraints on the content plan, while @xcite learn content planners from semantically annotated corpora, and @xcite predict content plans using grammar rules whose probabilities are learned from training data. more recently, there have been attempts to equip encoder-decoder models @xcite with content planning modules. @xcite introduce micro planning: they first learn a content plan corresponding to a sequence of records, and then generate a summary conditioned on it. @xcite treat content selection as a task similar to extractive summarization. specifically, they post-process pudupully et al.'s (2019a) micro-plans with special tokens identifying the beginning and end of a sentence. their model first extracts sentence plans and then verbalizes them one-by-one by conditioning on previously generated sentences. @xcite propose a two-stage approach that first predicts a document plan and then generates text based on it. the input to their model is a set of rdf subject, object, predicate tuples. their document plan is a sequence of sentence plans where each sentence plan contains a subset of tuples in a specific order. text generation is implemented using a sequence-to-sequence model enhanced with attention and copy mechanisms @xcite . they evaluate their model on the webnlg dataset @xcite , where the outputs are relatively short (24 tokens on average). our approach is closest to @xcite , who advocate macro planning as a way of organizing high-level document content. their model operates over paragraph plans that are verbalizations of the tabular input and predicts a document plan as a sequence of paragraph plans. in a second stage, the summary is generated from the predicted plan making use of attention enriched with a copy mechanism. we follow their formulation of content planning as paragraph plan prediction. our model thus operates over larger content units compared to related work @xcite and performs the tasks of micro-and macroplanning in one go. in contrast to @xcite , we predict paragraph plans and their corresponding paragraphs jointly in an incremental fashion. our approach is reminiscent of psycholinguistic models of speech production @xcite @xcite , which postulate that different levels of processing (or modules) are responsible for language generation; these modules are incremental, each producing output as soon as the information it needs is available and the output is processed immediately by the next module. we assume that plans form a sequence of paragraphs, which we treat as a latent variable and learn with a structured variational model. sequential latent variables @xcite @xcite have previously found application in modeling attention in sequence-to-sequence networks @xcite , document summarization @xcite , controllable generation @xcite , and knowledgegrounded dialogue @xcite . in the context of data-to-text generation, latent variable models have been primarily used to inject diversity in the output. @xcite generate a sequence of groups (essentially a subset of the input) which specifies the content of the sentence to be generated. their plans receive no feedback from text generation, they cover a small set of input items, and give rise to relatively short documents (approximately 100 tokens long). @xcite use latent variables to disentangle the content from the structure (operationalized as templates) of the output text. their approach generates diverse output output by sampling from the template-specific sample space. they apply their model to single-sentence generation tasks @xcite . ## model following puduppully and lapata (2021), we assume that at training time our model has access to a pool of paragraph plans e (see table d in figure 1 ), which represent a clustering of records. we explain how paragraph plans are created from tabular input in section 4. given e, we aim to generate a sequence of paragraphs @xmath4, . . . , y t ] that describe the data following a sequence of chosen plans @xmath5, . . . , z t ]. let y t denote a paragraph, which can consist of multiple sentences, and t the count of paragraphs in a summary. with a slight abuse of notation, superscripts denote indices rather than exponentiation. so, y t i refers to the i-th word in the t-th paragraph. a plan @xmath6, . . . , z t ] is a list of discrete variables where z @xmath7. ## experimental setup data we performed experiments on the rotowire @xcite and mlb @xcite datasets and the german rotowire provided as part of the wngt 2020 dgt shared task on ''document-level generation and translation'' @xcite . statistics on these datasets are shown in we used the official train/dev/test splits: 3,398/727/728 for rotowire, 22,821/1,739/1,744 for mlb, and 242/240/241 for german rotowire. the latter is considerably smaller than its english counterpart and mlb, and serves to illustrate our model's sample efficiency when training data is scarce. all three datasets were preprocessed following the method of @xcite . a paragraph plan for an entity is constructed by verbalizing its records in a fixed sequence of record type followed by its value. for example, pitcher b.keller from figure 1 would be verbalized @xmath8.@xmath9. . . . we denote this using the @xmath10.keller)>. the paragraph plan for an event is the verbalization of the players in the event followed by the verbalization of play-by-plays. candidate paragraph plans e are obtained by enumerating entities and events and their combinations (see table d in figure 1 ). oracle macro plans are obtained by matching the mentions of entities and events in the gold summary with the input table. we make use of these oracle macro plans during training. the versions of mlb and rotowire released by puduppully and lapata (2021) contain paragraph delimiters for gold summaries; we preprocessed the german rotowire in a similar fashion. ## results our experiments were designed to explore how the proposed model compares to related approaches which are either not enhanced with planning modules or non-incremental. we also investigated the sample efficiency of these models and the quality of the predicted plans when these are available. the majority of our results focus on automatic evaluation metrics. we also follow previous work @xcite @xcite in eliciting judgments to evaluate system output. ## discussion in this work, we proposed a novel sequential latent variable model for joint macro planning and generation. key in our approach is the creation of a latent plan in a sequential manner, while interleaving the prediction of plans and the generation of corresponding paragraphs. we proposed to deconstruct monolithic long document generation into smaller units (paragraphs, in our case), which affords flexibility and better communication between planning and generation. taken together, the results of automatic and human evaluation suggest that seqplan performs best in terms of factuality and coherence, it generates diverse, and overall fluent, summaries, and is less data-hungry compared with strong systems like macro and ncp+cc. as seqplan does not have to learn alignments between the macro plan and the output text, it is better suited for long-form generation. potential applications include summarizing books @xcite , where the output can be longer than 1,000 tokens, or generating financial reports @xcite , where the output exceeds 9,000 tokens. existing approaches for long-form generation summarize individual paragraphs independently @xcite or adopt a hierarchical approach @xcite , where summaries of