Conditional Generation with a Question-Answering Blueprint (2023)

Abstract: The ability to convey relevant and faithful information is critical for many tasks in conditional generation and yet remains elusive for neural seq-to-seq models whose outputs often reveal hallucinations and fail to correctly cover important details. In this work, we advocate planning as a useful intermediate representation for rendering conditional generation less opaque and more grounded. We propose a new conceptualization of text plans as a sequence of question-answer (QA) pairs and enhance existing datasets (e.g., for summarization) with a QA blueprint operating as a proxy for content selection (i.e., what to say) and planning (i.e., in what order). We obtain blueprints automatically by exploiting state-of-the-art question generation technology and convert input-output pairs into input-blueprint-output tuples. We develop Transformer-based models, each varying in how they incorporate the blueprint in the generated output (e.g., as a global plan or iteratively). Evaluation across metrics and datasets demonstrates that blueprint models are more factual than alternatives which do not resort to planning and allow tighter control of the generation output.

## introduction neural generation models are often prone to hallucination @xcite @xcite , repetition and redundancy @xcite , and struggle to identify which content units are salient @xcite . these phenomena are amplified when generating longform text, i.e., documents with multiple paragraphs @xcite , when dealing with non-linguistic data (e.g., database tables), or very long input-which is common when summariz-ing multiple documents @xcite , books @xcite , or dialogue @xcite . an additional challenge concerns the blackbox nature of deep learning systems, which hides the inherent complexity of modeling multiple interconnected linguistic phenomena in text generation, and makes it difficult to examine model decisions and attribute errors to specific components. the lack of modularity further affects controllability as these systems cannot be easily tailored to individual needs. attempts to remedy some of these issues focus on changing the way entities are represented @xcite , allowing the decoder to skip low-confidence tokens to enhance faithful generation @xcite , modeling graph connections between document elements to better capture salience @xcite , encoding documents hierarchically @xcite @xcite , learning latent alignments between the input and the target text @xcite , adopting sparse attention mechanisms @xcite , and introducing content selection @xcite and planning components @xcite @xcite . in this paper we also aim to render conditional generation more modular via an intermediate, plan-based representation. while autoregressive models of language predict one token at a time, there is evidence that in humans some degree of planning occurs at a higher level than individual words @xcite . a long tradition in natural language generation views our work proposes a new conceptualization of text plans as a sequence of question-answer pairs. specifically, we draw inspiration from the ''questions under discussion'' (qud) theory of discourse structure, which posits that one way of articulating the structure of a text is to identify the questions and sub-questions that are raised and answered by subsequent spans of text @xcite @xcite @xcite . theoretical models of qud assume that discourse contains implicit questions for each of the assertions made, which are thereby turned into answers. these questions and answers can be understood in terms of their use in moving a discourse forward to achieve communicative goals. we propose to make quds explicit by exploiting state-of-the-art question generation technology @xcite and use them as an intermediate representation layer for conditional generation, i.e., a question-answering (qa) blueprint operating as a proxy for both content selection (i.e., what to say) and planning (i.e., in what order). we instantiate our models with a transformer @xcite encoder-decoder architecture and perform experiments on summarization datasets representing different information seeking tasks, application domains, and user requirements. 1 in all cases, we empirically demonstrate that blueprint models are more factual than alternatives which do not resort to planning; we also observe that qa blueprints are a better representation compared to plans based on entity chains @xcite , allowing tighter control of the output, and providing a comprehensive explanation for model predictions (if the plan is erroneous, then the summary will be too). ## related work questions under discussion the qud-based approach to discourse structure assumes an openended inventory of possible questions and subquestions @xcite . recent efforts @xcite @xcite have nevertheless shown that it is possible to manually annotate documents with quds, i.e., to formulate a question for every assertion expressed in a text. de kuthy et al. (2020) even go as far as to partially automate qud annotation in german by automatically generating all potentially relevant questions for a given sentence. related work @xcite focuses on the generation of inquisitive questions that reflect general text understanding and free-form open-ended questions @xcite . our work builds upon qud and related discourse structure theories, although, we do not directly implement any of them in particular. we adopt question answering as a good way of spelling out the connection between the information structure of a sentence and the discourse in which the sentence can function. ## automatic evaluation in this section we present experimental results using automatic evaluation metrics that assess overall summary (and blueprint) quality. moreover, we quantify the extent to which automatically generated output is grounded to the blueprint and faithful to the input document/s. ## human-based evaluation in addition to automatic evaluation, we conducted three human-based studies assessing different dimensions of output quality. wishing to avoid well-documented issues 7 with automated bots on amazon mechanical turk and crowdworkers running through hits as quickly as possible without paying attention to the tasks, we used a few trained annotators. they were given task-specific instructions and went through several pilots to iron out disagreements on edge cases. 8 ## conclusion in this work we proposed a novel plan-based approach to conditional generation. we conceptualized text plans as a sequence of qa pairs operating as a proxy for what to say and in what order. we developed transformer-based models that generate by conditioning on a global qa blueprint plan (e2e, multitask) or iteratively by planning and generating one sentence at a time (iterative). experimental results across three challenging datasets demonstrate that blueprint models are inherently more informative than vanilla sequence-to-sequence approaches without a planning component. among the three presented here (e2e, multitask, iterative), we find that iterative is the best choice for grounded generation and suggests a promising direction for longform generation. blueprint models offer several advantages compared to blackbox generation. model predictions can be examined, and errors can be traced back to the blueprint, which in turn can reveal whether the output is informative and faithful to its input. the formulation of the blueprint plan as question-answer pairs makes it intuitive and user-friendly. we have discussed how blueprint models might be used in a human-in-the-loop setting, where users interact with and influence model predictions directly, e.g., by editing the blueprint length and content (as different blueprints lead to different outputs). in the future, we would like to use blueprints more directly to advance methods for training language models using reward learning @xcite , e.g., based on whether the output answers the blueprint questions. rather than eliciting expensive human feedback @xcite , blueprints could provide a cheaper automatic alternative. finally, although we focused primarily on the generation problem in this work, we believe blueprints might also be useful as a general-purpose approach to retrieving and organizing important content, especially when faced with many and very long inputs.