A gg G en: Ordering and Aggregating while Generating (2021)

Abstract: We present AggGen (pronounced ‘again’) a data-to-text model which re-introduces two explicit sentence planning stages into neural data-to-text systems: input ordering and input aggregation. In contrast to previous work using sentence planning, our model is still end-to-end: AggGen performs sentence planning at the same time as generating text by learning latent alignments (via semantic facts) between input representation and target text. Experiments on the WebNLG and E2E challenge data show that by using fact-based alignments our approach is more interpretable, expressive, robust to noise, and easier to control, while retaining the advantages of end-to-end systems in terms of fluency. Our code is available at https://github.com/XinnuoXu/AggGen .

## introduction recent neural data-to-text systems generate text "end-to-end" (e2e) by learning an implicit mapping between input representations (e.g. rdf triples) and target texts. while this can lead to increased fluency, e2e methods often produce repetitions, hallucination and/or omission of important content for data-to-text @xcite as well as other natural language generation (nlg) tasks @xcite . traditional nlg systems, on the other hand, tightly control which content gets generated, as well as its ordering and aggregation. this process is called sentence planning @xcite @xcite @xcite . figure 1 shows two different ways to arrange and combine the representations in the input, resulting in widely different generated target texts. in this work, we combine advances of both paradigms into a single system by reintroducing human-authored text sentence plan 1: generated target text 1: sentence plan 2: generated target text 2: sentence planning into neural architectures. we call our system agggen (pronounced 'again'). agggen jointly learns to generate and plan at the same time. crucially, our sentence plans are interpretable latent states using semantic facts foot_0 (obtained via semantic role labelling (srl)) that align the target text with parts of the input representation. in contrast, the plan used in other neural plan-based approaches is usually limited in terms of its interpretability, control, and expressivity. for example, in @xcite the sentence plan is created independently, incurring error propagation; @xcite use latent segmentation that limits interpretability; @xcite sample from a latent variable, not allowing for explicit control; and @xcite aggregate multiple input representations which limits expressiveness. agggen explicitly models the two planning processes (ordering and aggregation), but can directly influence the resulting plan and generated target text, using a separate inference algorithm based on dynamic programming. crucially, this enables us to directly evaluate and inspect the model's planning and alignment performance by comparing to manually aligned reference texts. we demonstrate this for two data-to-text generation tasks: the e2e nlg @xcite and the webnlg challenge @xcite . we work with a triple-based semantic representation where a triple consists of a subject, a predicate and an object. 2 for instance, in the last triple in figure 1 , apollo 8, operator and nasa are the subject, predicate and object respectively. our contributions are as follows: ## related work factual correctness is one of the main issues for data-to-text generation: how to generate text according to the facts specified in the input triples without adding, deleting or replacing information? the prevailing sequence-to-sequence (seq2seq) architectures typically address this issue via reranking @xcite @xcite or some sophisticated training techniques @xcite @xcite . for applications where structured inputs are present, neural graph encoders @xcite @xcite or decoding of explicit graph references @xcite are applied for higher accuracy. recently, large-scale pretraining has achieved sota results on webnlg by fine-tuning t5 @xcite . several works aim to improve accuracy and controllability by dividing the end-to-end architecture into sentence planning and surface realisation. however, all of these pipeline-based approaches either require additional manual annotation or depend on a parser for the intermediate steps. other works, in contrast, learn planning and realisation jointly. for example, @xcite introduce a hierarchical decoding model generating different parts of speech at different levels, while filling in slots between previously generated tokens. @xcite include a jointly trained content selection and ordering module that is applied before the main text generation step.the model is trained by maximizing the log-likelihood of the gold content plan and the gold output text. li and rush (2020) utilize posterior regularization in a structured variational framework to induce which input items are being described by each token of the generated text. @xcite aim for better semantic control by using a hidden semi-markov model (hsmm) for splitting target sentences into short phrases corresponding to "templates", which are then concatenated to produce the outputs. however it trades the controllability for fluency. similarly, @xcite explicitly segment target text into fragment units, while aligning them with their corresponding input. @xcite use a hierarchical variational model to aggregate input items into a sequence of local latent variables and realize sentences conditioned on the aggregations. the aggregation strategy is controlled by sampling from a global latent variable. in contrast to these previous works, we achieve input ordering and aggregation, input-output alignment and text generation control via interpretable states, while preserving fluency. ## joint planning and generation we jointly learn to generate and plan by aligning facts in the target text with parts of the input representation. we model this alignment using a hidden markov model (hmm) that follows a hierarchical structure comprising two sets of latent states, corresponding to ordering and aggregation. the model is trained end-to-end and all intermediate steps are learned in a unified framework. ## intrinsic evaluation of planning we now directly inspect the performance of the planning component by taking advantage of the readability of srl-aligned facts. in particular, we investigate: (1) sentence planning performance. we study the agreement between model's planning and reference planning for the same set of input triples; (2) alignment performance -we use agggen as an aligner and examine its ability to align segmented facts to the corresponding input triples. since both studies require ground-truth triple-to-fact alignments, which are not part of the webnlg and e2e data, we first introduce a human annotation process in section 6.1. 10 the number of fluent generations for transformer and agggen among the examined 100 examples are 96 and 95 respectively. the numbers for agggen-od and agggen-ag are 86 and 74, which indicates that both input ordering and input aggregation are critical for generating fluent texts. ## conclusion and future work we show that explicit sentence planning, i.e., input ordering and aggregation, helps substantially to produce output which is both semantically correct as well as naturally sounding. crucially, this also enables us to directly evaluate and inspect both the model's planning and alignment performance by comparing to manually aligned reference texts. our system outperforms vanilla seq2seq models when considering semantic accuracy and word-overlap based metrics. experiment results also show that agggen is robust to noisy training data. we plan to extend this work in three directions: other generation models. we plan to plug other text generators, e.g. pre-training based approaches @xcite , into agggen to enhance their interpretability and controllability via sentence planning and generation. zero/few-shot scenarios. kale and rastogi (2020)'s work on low-resource nlg uses a pretrained language model with a schema-guided representation and hand-written templates to guide the representation in unseen domains and slots. these techniques can be plugged into agggen, which allows us to examine the effectiveness of the explicit sentence planning in zero/few-shot scenarios. including content selection. in this work, we concentrate on the problem of faithful surface realization based on e2e and webnlg data, which both operate under the assumption that all input predicates have to be realized in the output. in contrast, more challenging tasks such as rotowire @xcite , include content selection before sentence planning. in the future, we plan to include a content selection step to further extend agggen's usability. predicted text: the 11th mississippi infantry monument is located at seminary ridge, gettysburg, pennsylvania, united states. it was established in 2000 and falls under the category of contributing property.