Text-Blueprint: An Interactive Platform for Plan-based Conditional Generation (2023)

Abstract: While conditional generation models can now generate natural language well enough to create fluent text, it is still difficult to control the generation process, leading to irrelevant, repetitive, and hallucinated content. Recent work shows that planning can be a useful intermediate step to render conditional generation less opaque and more grounded. We present a web browser-based demonstration for query-focused summarization that uses a sequence of question-answer pairs, as a blueprint plan for guiding text generation (i.e., what to say and in what order). We illustrate how users may interact with the generated text and associated plan visualizations, e.g., by editing and modifying the plan in order to improve or control the generated output.A short video demonstrating our system is available at https://goo.gle/text-blueprint-demo

## introduction with the advent of encoder-decoder models @xcite , transformer-based architectures @xcite , and large-scale pretraining @xcite , deep learning models have achieved great performance on conditional generation tasks such as summarization @xcite @xcite or task-oriented dialogue modeling @xcite . however, it remains challenging to control the text generation, as these neural models tend to generate hallucinated @xcite @xcite or repetitive content @xcite , and struggle to identify which information is most relevant to include in the output text @xcite . recent work shows that planning can be a useful intermediate step to address some of these challenges @xcite @xcite . in this work, we present text-blueprint, a demonstration for showcasing the approach described in @xcite , that uses a text plan, formulated as a sequence of question-answer pairs called the blueprint, to serve as an intermediate representation for content selection and organization of the generated text. it draws inspiration from the "questions under discussion" theory of discourse which posits that the structure of a text can be derived by identifying the questions that are answered by each subsequent span of text @xcite @xcite @xcite . we implement this blueprint approach as an interactive web application for query-focused summarization. an example snapshot of our interface is shown in figure 1 . as can be seen, for a given generated summary, users can examine its corresponding blueprint, modify it to make it more faithful or relevant, and control its length by changing the number of question-answer pairs. given a query and relevant documents, there can be multiple semantically-diverse summaries that meet the communicative goal of synthesizing the most important points. traditional generation systems do well at single-best summaries, while our interactive demonstration allows users to explore different summaries for a given input, while directly observing the impact of changes to the plan on the generated text. the formulation of the blueprint plan as question-answer pairs makes it intuitive and user-friendly (e.g., users can inspect and ask questions without any instructions). our demonstration is an example of what can be achieved with human-in-the-loop conditional generation @xcite . it allows users to revise the output text (i.e., by editing the blueprint) subject to their information needs. additionally, it allows researchers to analyze what constitutes a good blueprint for various summarization tasks. ## related work there are several libraries for broad nlp tasks, such as allennlp foot_0 or gluonnlp foot_1 . the language interpretability tool @xcite ) is an interactive platform for examining model behavior, meant for rapid exploration and error analysis. a variety of toolkits have been developed recently that support generation tasks. for instance, texar @xcite is an open-source platform that unifies the development of diverse yet closelyrelated applications, such as machine translation, summarization, and dialog. textbox @xcite is a modular framework that offers interfaces for various common functions in text generation models, allowing researchers and practitioners to reproduce baseline models and compare new models. the giant language model test room, also known as gltr @xcite , helps users differentiate automatically-generated text from human-written text. for conditional generation, many demonstrations are summarization systems. for instance, @xcite present a modular tool for automatic summarization. @xcite showcase a visualization tool for summaries obtained by different summarization methods. the summvis platform @xcite serves a simi-lar goal to the demonstration presented in this paper. it enables users to visually analyze the models, data, and evaluation metrics associated with abtractive summarization, e.g., by highlighting hallucinated entities in the generated text. while previous tools and frameworks are versatile and modular, their focus is not on empowering users with control over the generated text in an interactive environment. in particular, studies on human-ai interaction for text summarization @xcite show that users' overall experience is better when they can control the generation process. users preferred systems that allowed them to adjust and see the impact of their changes on the output directly, and the controllability improved their trust in the system when summarizing unfamiliar topics. systems more geared toward interactive text generation include chatbots such as meena (adiwardana et al., 2020) or other specialized dialogue systems such as parlai @xcite . @xcite present an approach called collaborative semantic inference that exposes latent variables to the user for interactive generation. still, these tasks differ from conditional generation using planning for which our demonstration is designed. ## summarization using planning this demonstration showcases query-focused summarization using planning as described in narayan et al. (2022). 3 in their approach, question-answer text plans, called blueprints, serve as intermediate representations for content selection and structuring of the generated text. we implement three blueprint models in our demonstration, which we briefly describe below; they are all encoder-decoder variants instantiated from a transformer @xcite architecture. let d denote the input to our models, which is a user query concatenated with a document or a set of documents relevant to this query. from this input d, the model generates b; s, the blueprint b and its corresponding summary s. the blueprint itself is a sequence of question-answer pairs {(q 1 , a 1 ), (q 2 , a 2 ), . . . , (q m , a m )}. existing datasets do not contain such blueprints, they are typically designed as (d, s) pairs. narayan et al. (2022) describe a suite of data enhancement methods for obtaining blueprint annotations (we refer the interested reader to their paper for details). end-to-end model the end-to-end blueprint model uses an encoder-decoder model to encode the input documents d and generate b; s, the concatenation of the blueprint and output text, in one go (figure 2a ). the decoder first predicts the blueprint b autoregressively and continues to generate the output s, conditioned on both b and d. in particular, it predicts b as q 1 ; a 1 ; . . . ; q m ; a m , a concatenated sequence of question-answer pairs. in contrast to systems that use prompted encoders, such as ctrlsum @xcite , the blueprint models use decoder prompting. as a consequence, the blueprint plan is entirely generated by the model, without human intervention or relying on external systems. after the generation, users can inspect the question-answer pairs and corresponding summary. if desired, they can then select questionanswer pairs to remove from the plan. the system is fed the updated blueprint b which prompts the decoder to generate the corresponding output s . iterative model it is generally challenging for encoder-decoder models to generate long output sequences @xcite . the end-to-end model ultimately suffers from this problem as it aims to generate sequence b; s instead of just s. the iterative blueprint model mitigates this by adopting an incremental approach that interleaves planning with text generation rather than predicting a global plan before generating the output s (figure 2b ). if we denote the output s as consisting of n sentences {s 1 , s 2 , . . . , s n }, then the corresponding blueprint b can be expressed as {b 1 , b 2 , . . . , b n }, where b i : {(q i 1 , a i 1 ), (q i 2 , a i 2 ), . . . , (q i k , a i k )} consists of k question-answer pairs for sentence s i . this model iteratively plans and generates one sentence at a time, conditioning on the input and the output sentences generated so far. in particular, it is trained such that the encoder first encodes the input d, while the decoder takes the output generated so far {s 1 , . . . , s i } as a forced prompt and generates the blueprint b i+1 for the following sentence s i+1 , followed by sentence s i+1 itself. the iterative approach naturally addresses some of the issues the end-to-end model faces. in particular, it does not run into sequence length limitations as it predicts one sentence at a time. interactive model the third model is an interactive blueprint model that allows the user to modify the blueprint and directly change the generated output. it operates similarly to the end-to-end model previously described but in addition to letting the user select which elements of the plan to keep or remove, we design the system to allow the user to provide their own plan. we do not expect users to be able to provide answers to all the questions they come up with when creating their own plans. therefore, we modify the original paradigm set by @xcite and develop a new model specifically for the interactive mode that uses a question-only blueprint instead of a question-answer blueprint. from the input documents d, we fine-tune this model to generate b; s, where b is a concatenated sequence of questions q 1 ; q 2 ; . . . ; q m . for this new model, we use the same blueprint training data as the iterative and end-to-end models, but only use the question annotations during fine-tuning, ignoring the answers. in the interactive mode, the user can edit the plan by typing in questions they come up with. this process creates an updated blueprint b which prompts the decoder to generate an updated summary s . ## system description our web browser-based demonstration is designed so that researchers and practitioners can inspect and interact with the different blueprint models. we frame the summarization task around a user query, since in a real world scenario we would expect users to have a question or intent in mind. the system retrieves documents relevant to the query and displays their summary and its corresponding blueprint. figure 1 provides a snapshot of the user interface (ui) and its components, namely the model selection, document retrieval, and blueprint output views. ## use cases in the following we explore some of the possibilities of human-in-the-loop summarization and illustrate different use cases for our demonstration. informative blueprints while the inner workings of deep learning models might be opaque to a human user, the formulation of the blueprint as a sequence of questions makes the control of the system's output user-friendly. users do not have to be machine learning experts to interact with the system through questions and answers. moreover, the ability to change the blueprint and observe the result on the summary, provides the user with immediate feedback. the planning step also brings some insight into the often black-box nature of conditional generation. this property is especially valuable when the user summarizes complex or difficult information, since it breaks down the generation process into a sequence of questions. the blueprint plan offers context for the information in the generated output, which has been shown to be a desirable property in human-ai interaction for text summarization @xcite . plan agnostic models do not provide details as to why certain pieces of information were included. in contrast, as seen in figure 4 , the blueprint plan anchors conditional generation, providing the user with a question-answer explanation for each proposition. narayan et al. (2021) evaluate the blueprint models across several datasets and show improvements in faithfulness over models that do not use planning. moreover, they also evaluate the impact of automated blueprint edits on the output summary. for each generated blueprint, they automatically remove question-answer pairs for which the answer is not contained in the input, thus eliminating questions that cannot be answered based on the input documents. they then prompt the decoder with the modified blueprint to generate the summary, following a similar setting as in our system demonstration. their results confirm that this automatic filtering of the generated blueprint further improves source documents, the system can lead to diverse summaries by selecting different blueprints. moreover, while it would be difficult for a user to come up with a plan from scratch if they are unfamiliar with the topic of their query, the provided blueprint can serve as a starting point from which the user can select what they would like to keep. as we discuss next, the user could also elaborate on their initial query by adding questions in the blueprint. ## conclusions this demonstration showcases a novel approach to query-focused summarization that uses a blueprint to plan the generated text. by implementing it within an interactive framework, we transform it into an example of human-in-the-loop conditional generation. our demonstration is designed in a what was the largest passenger liner afloat at the time it entered service? when did the titanic movie with dicaprio come out? who played the female lead in the titanic movie? figure 5 : snapshot of the results obtained with the interactive blueprint model for the query "what is the titanic known for?". questions highlighted in red were manually added by the user, leading to a different output. we hope this demonstration will spur further exploration into controllable and interpretable conditional generation systems and how human interaction can be an integral component in generating personalized outputs. we further expect interactive tools like the one presented here to assist in summary creation and editing, e.g., for data augmentation in low-resource settings or for more robust system evaluation by generating multiple outputs for a given document.