Interactive Text Generation (2023)

Abstract: Users interact with text, image, code, or other editors on a daily basis. However, machine learning models are rarely trained in the settings that reflect the interactivity between users and their editor. This is understandable as training AI models with real users is not only slow and costly, but what these models learn may be specific to user interface design choices. Unfortunately, this means most of the research on text, code, and image generation has focused on non-interactive settings, whereby the model is expected to get everything right without accounting for any input from a user who may be willing to help. We introduce a new Interactive Text Generation task that allows training generation models interactively without the costs of involving real users, by using user simulators that provide edits that guide the model towards a given target text. We train our interactive models using Imitation Learning, and our experiments against competitive non-interactive generation models show that models trained interactively are superior to their non-interactive counterparts, even when all models are given the same budget of user inputs or edits.

## introduction advances in generative modeling have made it possible to automatically generate high-quality texts @xcite , code @xcite , and images @xcite , but these creations can be unsatisfactory in many respects. for example, they often suffer from content errors-e.g., hallucinations @xcite -that may require help from the user. even if the generation is of good quality, it may not be the kind of text, code, or image that the user was hoping to obtain. indeed, open-ended and complex generation tasks are often underspecified (e.g., from a simple prompt), which makes it almost impossible for the model to figure 1 : the itg framework of this paper allows generation models to be trained and evaluated by directly interacting with users or user simulators. this paper introduces a new end-to-end generation task, interactive text generation, that accounts for interactivity without requiring real users during training. the framework is illustrated in figure 1 , where the model and a "user" take turns editing the text until a given stopping criterion is met. as this setup would be impractical to train with real users, we rely on a user simulator that provides a few high-quality edits that guide the model towards a given target text. this interspersion of user edits in the generation process allows text generation models to more efficiently use inferential capabilities of large language models (llm). contrast interactive text generation in figure 1 with conventional text generation, where both systems are given exactly three user input words. interactive text generation leverages llm's capability to infer, e.g., "space-craft" from "nasa" and allows it to focus on parts of the text (e.g., "monday") that are more difficult to predict, and this allows the interactive approach to generate text that better meets user's needs. our work makes the following contributions: • we propose a task and framework for interactive text generation, and release models, a dataset, and user simulators. crucially, this task evaluates generation models with the same budget of user inputs or edits, to ensure the comparison between interactive and non-interactive models is fair. our editing models include both autoregressive and non-autoregressive versions, with the nonautoregressive one achieving the best results. our task models the interaction between a user and system, where the two parties successively take turns making changes to a draft document. as this interaction applies to both training and testing, it would be unrealistic to assume we have real users in both cases, and we therefore rely on user simulators. although building effective user simulators is notoriously hard in tasks such as dialog @xcite @xcite , it is less so in our work given how we frame the interactive generation task: we assume that there is a goal text which one can think of as representing the user's desire or needs. the system does not know the goal, but the user simulator does. the objective of the agent is to get as close to the goal text as possible, while minimizing the number of edits the user simulator needs to make. this framing makes it much easier to design effective user simulators, as a comparison between the current draft and goal text can help infer useful text edits. while agents in this setup are given an advantage by interacting with an oracle that has knowledge of the goal document, the number of oracle edits is generally set to be small, and we ensure comparisons between all models (including non-interactive models) are fair by giving each model the same budget of edits or inputs derived from the goal document. ## user simulator this section describes the user simulator used in our work. throughout this section and the next, we refer the reader to the appendix for more details. where appropriate, we indicate references to the appendix in parentheses. ## imitation learning model the agent policy π θ will parameterize a distribution over documents, so that at each step of the interaction we draw a h ∼ π θ (•|s h ). here s h is the his-tory of drafts (a 0 , u 1 ), . . . , (a h-1 , u h ) exchanged so far. this information is necessary for the agent to make inferences about the goal g. however, this history can become very long, especially since each element is an entire document. therefore, in practice we only give the policy π θ access to the last set of drafts, (a h-1 , u h ), which we represent as a diff (app. b), as illustrated in figure 2 . ## experiments because of the novelty of our task, the main goal of our experiments is to assess the benefit of interactivity in text generation. we also provide example interactions between the learned agent and user simulator. ablations and additional details on experiment settings are in the appendix. ## results automatic evaluation figure 4 presents our main results on interactivity. we can see that for our main model, splitting the same number of user edits over more episodes leads to better scores across bleu, bleurt and bertscore. for example, comparing the setting where the model receives all 6 user edits at the start in one episode, against the setting where the edits are given across 3 episodes, we see improvements of about 7%, 4% and 5% (absolute % gains) in terms of bleu, bleurt, and bertscore respectively. while these differences are not large in absolute terms, we emphasize that this gain comes exclusively from interactivity. the amount of information given to the model is the same in both settings. this suggests that, even in this narrow sense, interactivity helps. note that there may also be many more benefits from a human-computer interaction standpoint that we cannot quantify here. we also note that our token editing model (editor) outperforms the left-to-right, autoregressive sequence to sequence (s2s) model foot_2 . while the difference is not staggering, it is notable given the success of standard s2s models across a variety of text generation tasks. as motivated in section 4, the token editing model is more naturally suited to the task, and we argue that it constitutes the starting point for more interesting interactive models. for example, one could foresee training the model using reinforcement learning, in which case the structure of the editing model based on editing actions is better suited than a s2s model. ## related work text generation prior work on natural language generation (nlg) has largely focused on noninteractive settings that have become increasingly more challenging and open-ended, e.g., with generation from prompts @xcite , outlines (rashkin et al., 2020), topics or keywords @xcite , plots (riedl and young, 2010), descriptions @xcite , events @xcite . this increase of difficulty can make nlg more prone to content quality issues, such as hallucinations @xcite @xcite , that can require post-editing from the user. several works explored ways for llms to improve their outputs by iteration and self-critiquing @xcite @xcite . in particular, @xcite presented models for text generation and self-correction that also incorporated external feedback. however, in their case the feedback is used at training time to learn a corrector. in our task, the user feedback comes at inference time and the agent must use that feedback to guess what the user would like to generate. ## conclusions we presented a new task and benchmark for text generation that operationalizes interactivity between an agent and a user, without the need to involve real users during training. our framework compares interactive and non-interactive systems that are given the same amount of user inputs, and shows that interactive text generation leads to text of higher quality according to multiple automated evaluation metrics and human evaluations. we also present a non-autoregressive editing model that outperforms a standard sequence-to-sequence transformer in various settings. all baseline, data, and models of this paper will be made public.