Data-to-text Generation with Macro Planning (2021)

Abstract: Recent approaches to data-to-text generation have adopted the very successful encoder-decoder architecture or variants thereof. These models generate text that is fluent (but often imprecise) and perform quite poorly at selecting appropriate content and ordering it coherently. To overcome some of these issues, we propose a neural model with a macro planning stage followed by a generation stage reminiscent of traditional methods which embrace separate modules for planning and surface realization. Macro plans represent high level organization of important content such as entities, events, and their interactions; they are learned from data and given as input to the generator. Extensive experiments on two data-to-text benchmarks (RotoWire and MLB) show that our approach outperforms competitive baselines in terms of automatic and human evaluation.

## introduction data-to-text generation refers to the task of generating textual output from non-linguistic input @xcite @xcite such as databases of records, simulations of physical systems, accounting spreadsheets, or expert system knowledge bases. as an example, figure 1 shows various statistics describing a major league baseball (mlb) game, including extracts from the box score (i.e., the performance of the two teams and individual team members who played as batters, pitchers or fielders; table (a)), play-by-play (i.e., the detailed sequence of each play of the game as it occurred; table (b)), and a human written game summary (table (c)). traditional methods for data-to-text generation @xcite @xcite follow a pipeline architecture, adopting separate stages for text planning (determining which content to talk about and how it might be organized in discourse), sentence planning (aggregating content into sentences, deciding specific words to describe concepts and relations, and generating referring expressions), and linguistic realization (applying the rules of syntax, morphology, and orthographic processing to generate surface forms). recent neural network-based approaches @xcite @xcite ) make use of the encoderdecoder architecture @xcite , are trained end-to-end, and have no special-purpose modules for how to best generate a text, aside from generic mechanisms such as attention and copy @xcite . the popularity of end-to-end models has been further boosted by the release of new datasets with thousands of input-document training pairs. the example shown in figure 1 is taken from the mlb dataset @xcite , which contains baseball game statistics and human written summaries (∼25k instances). rotowire @xcite is another widely used benchmark, which contains nba basketball game statistics and their descriptions (∼5k instances). @xcite show that despite being able to generate fluent text, neural data-to-text generation models are often imprecise, prone to hallucination (i.e., generate text that is not supported by the input), and poor at content selection and document structuring. attempts to remedy some of these issues focus on changing the way entities are represented @xcite , allowing the decoder to skip low-confidence tokens to enhance faithful generation @xcite , and making the encoder-decoder architecture more modular by introducing micro planning @xcite . micro planning operates at the record level (see @xcite in figure 1 : mlb statistics tables and game summary. tables summarize the performance of teams and individual team members who played as batters and pitchers as well as the most important actions (and their actors) in each play (tables (a) and (b)). macro plan for the game summary is shown at the bottom @xcite ). <@xmath0. there is a plan for every paragraph in the game summary (correspondence shown in same color); <v(entity)> verbalizes entities, @xmath1.1). set of candidate paragraph plans are shown above macro plan (table (d)) and grouped into two types: plans describing a single entity/event or their combinations. best viewed in color. figure 1 ; e.g., c.mullins bh 2, j.villar team orioles), it determines which facts should be mentioned within a textual unit (e.g., a sentence) and how these should be structured (e.g., the sequence of records). an explicit content planner essentially makes the job of the neural network less onerous allowing to concentrate on producing fluent natural language output, without expending too much effort on content organization. in this work, we focus on macro planning, the high-level organization of information and how it should be presented which we argue is important for the generation of long, multi-paragraph documents (see text (c) in figure 1 ). problematically, modern datasets like mlb @xcite ; and also figure 1 ) and rotowire @xcite do not naturally lend themselves to document planning as there is no explicit link between the summary and the content of the game (which is encoded in tabular form). in other words, the underlying plans are latent, and it is not clear how they might be best represented, namely, as sequences of records from a table, or simply words. nevertheless, game summaries through their segmentation into paragraphs (and lexical overlap with the input) give clues as to how content might be organized. paragraphs are a central element of discourse @xcite @xcite , the smallest domain where coherence and topic are defined and anaphora resolution is possible @xcite . we therefore operationalize the macro plan for a game summary as a sequence of paragraph plans. although resorting to paragraphs describes the summary plan at a coarse level, we still need to specify individual paragraph plans. in the sports domain, paragraphs typically mention entities (e.g., players important in the game), key events (e.g., scoring a run), and their interaction. and most of this information is encapsulated in the statistics accompanying game summaries (see tables (a) and (b) in figure 1 ). we thus define paragraph plans such that they contain verbalizations of entity and event records (see plan (e) in figure 1 ). given a set of paragraph plans and their corresponding game summary (see table (d) and summary (c) in figure 1 ), our task is twofold. at training time, we must learn how content was selected in order to give rise to specific game summaries (e.g., how input (d) led to plan (e) for summary (c) in figure 1 ), while at test time, given input for a new game, we first predict a macro plan for the summary and then generate the corresponding document. we present a two-stage approach where macro plans are induced from training data (by taking the table and corresponding summaries into account) and then fed to the text generation stage. aside from making data-to-text generation more interpretable, the task of generating a document from a macro plan (rather than a table) affords greater control over the output text and plays to the advantage of encoder-decoder architectures which excel at modeling sequences. we evaluate model performance on the rotowire @xcite and mlb @xcite benchmarks. experimental results show that our plan-andgenerate approach produces output that is more factual, coherent, and fluent compared with existing state-of-the-art models. our code, trained models, and dataset with macro plans can be found at https://github.com/ratishsp /data2text-macro-plan-py . ## related work content planning has been traditionally considered a fundamental component in natural language generation. not only does it determine which information-bearing units to talk about, but also arranges them into a structure that creates coherent output. many content planners have been based on theories of discourse coherence @xcite , schemas @xcite , or have relied on generic planners @xcite . plans are mostly based on hand-crafted rules after analyzing the target text, although a few approaches have recognized the need for learning-based methods. for example, @xcite learn ordering constraints in a content plan, @xcite represent plans as grammar rules whose probabilities are estimated empirically, while others make use of semantically annotated corpora to bootstrap content planners @xcite . more recently, various attempts have been made to improve neural generation models @xcite based on the encoder-decoder architecture @xcite by adding various planning modules. @xcite propose a model for data-to-text that first learns a plan from the records in the input table and then generates a summary conditioned on this plan. @xcite introduce a planning-based hierarchical variational model where a plan is a sequence of groups, each of which contains a subset of input items to be covered in a sentence. the content of each sentence is verbalized, conditioned on the plan and previously generated context. in their case, input items are a relatively small list of attributes (∼28) and the output document is also short (∼110 words). there have also been attempts to incorporate neural modules in a pipeline architecture for data-to-text generation. @xcite develop a model with a symbolic text planning stage followed by a neural realization stage. they experiment with the webnlg dataset @xcite which consists of rdf subject, object, predicate triples paired with corresponding text. their document plan is a sequence of sentence plans that in turn determine the division of facts into sentences and their order. along similar @xcite propose an architecture composed of multiple steps including discourse ordering, text structuring, lexicalization, referring expression generation, and surface realization. both approaches show the effectiveness of pipeline architectures, however, their task does not require content selection and the output texts are relatively short (24 tokens on average). although it is generally assumed that taskspecific parallel data is available for model training, @xcite do away with this assumption and present a three-stage pipeline model which learns from monolingual corpora. they first convert the input to a form of tuples, which in turn are expressed in simple sentences, followed by the third stage of merging simple sentences to form more complex ones by aggregation and referring expression generation. they also evaluate on data-to-text tasks which have relatively short outputs. there have also been efforts to improve the coherence of the output, especially when dealing with longer documents. puduppully et al. (2019b ) make use of hierarchical attention over entity representations which are updated dynamically, while @xcite explicitly keep track of salient entities and memorize which ones have been mentioned. our work also attempts to alleviate deficiencies in neural data-to-text generation models. in contrast to previous approaches, @xcite @xcite , we place emphasis on macro planning and create plans representing high-level organization of a document including both its content and structure. we share with previous work (e.g., @xcite ) the use of a two-stage architecture. we show that macro planning can be successfully applied to long document data-totext generation resulting in improved factuality, coherence, and fluency without any postprocessing (e.g., to smooth referring expressions) or recourse to additional tools (e.g., parsing or information extraction). ## problem formulation we hypothesize that generation based on plans should fare better compared to generating from a set of records, since macro plans offer a bird's-eye view, a high-level organization of the document content and structure. we also believe that macro planning will work well for long-form text generation, that is, for datasets that have multi-paragraph target texts, a large vocabulary space, and require content selection. ## model description the input to our model is a set of paragraph plans, each of which is a sequence of tokens. we first compute paragraph plan @xmath2, and then apply a contextualization and content planning mechanism similar to planning modules introduced in earlier work @xcite . predicted macro plans serve as input to our text generation model, which adopts an encoder-decoder architecture @xcite . to reflect the fact that some records will be more important than others, we compute an attention weighted sum of {e i,j } |e i | @xmath3. let @xmath4. we compute attention values α i,j over d and paragraph plan token representation e i,j : figure 2 : paragraph plan representation and contextualization for macro planning. computation of e 3 is detailed in equations ( 1 ) and ( 2 ), e att 3 in equation ( 3 ), and e c 3 in equation ( 4 ). paragraph plan vector e i is the attention weighted sum of e i,j (with j α i,@xmath5: next, we contextualize each paragraph plan representation vis-a-vis other paragraph plans (figure 2 , top left part). first, we compute attention scores β i,k over paragraph plan representations to obtain an attentional vector e att i for each: we use pointer networks @xcite ) to model p(z k |@xmath6, e ) as: once the output points to e i , its representation e c i is used as input to the next step of the lstm decoder. the process stops when the model points to eom, a token indicating end of the macro plan. ## discussion in this work we presented a plan-and-generate approach for data-to-text generation that consists of a macro planning stage representing high-level document organization in terms of structure and content, followed by a text generation stage. extensive automatic and human evaluation shows that our approach achieves better results than existing state-of-the-art models and generates summaries which are factual, coherent, and concise. our results show that macro planning is more advantageous for generation tasks expected to produce longer texts with multiple discourse units, and could be easily extended to other sports domains such as cricket @xcite or american football @xcite . other approaches focusing on micro planning @xcite might be better tailored for generating shorter texts. there has been a surge of datasets recently focusing on single-paragraph outputs and the task of content selection such as e2e @xcite , webnlg @xcite , and wikibio @xcite . we note that in our model content selection takes place during macro planning and text generation. the results in throughout our experiments we observed that template-based systems score poorly in terms of cs (but also @xcite . this is primarily due to the inflexibility of the template approach which is limited to the discussion of a fixed number of (high-scoring) players. yet, human writers (and neural models to a certain extent), synthesize summaries taking into account the particulars of a specific game (where some players might be more important than others even if they scored less) and are able to override global defaults. template sentences are fluent on their own, but since it is not possible to perform aggregation @xcite , the whole summary appears stilted, it lacks coherence and variability, contributing to low bleu scores. the template baseline is worse for mlb than rotowire which reflects the greater difficulty to manually create a good template for mlb. overall, we observe that neural models are more fluent and coherent, being able to learn a better ordering of facts which is in turn reflected in better co scores. despite promising results, there is ample room to improve macro planning, especially in terms of the precision of rg (see for rotowire, the main source of errors is the model's inability to understand numbers. for example, macro generates the following output: the lakers were the superior shooters in this game, going 48 percent from the field and 30 percent from the three-point line, while the jazz went 47 percent from the floor and 30 percent from beyond the arc. here, 30 percent should have been 24 percent for the lakers but the language model expects a higher score for the three-point line, and since 24 is low (especially compared to 30 scored by the jazz), it simply copies 30 scored by the jazz instead. a mechanism for learning better representations for numbers @xcite or executing operations such as argmax or minus @xcite should help alleviate this problem. finally, although our focus so far has been on learning document plans from data, the decoupling of planning from generation allows to flexibly generate output according to specification. for example, we could feed the model with manually constructed macro plans, consequently controlling the information content and structure of the output summary (e.g., for generating short or long texts, or focusing on specific aspects of the game).