Plug-and-Blend: A Framework for Controllable Story Generation with Blended Control Codes (2021)

Abstract: We describe a Plug-and-Play controllable language generation framework, Plug-and-Blend, that allows a human user to input multiple control codes (topics). In the context of automated story generation, this allows a human user lose or fine grained control of the topics that will appear in the generated story, and can even allow for overlapping, blended topics. We show that our framework, working with different generation models, controls the generation towards given continuous-weighted control codes while keeping the generated sentences fluent, demonstrating strong blending capability.

## introduction recent advancement in very large pre-trained neural language models (e.g. @xcite ) have enabled a new generation of applications that make use of the text generation capability they provide, ranging from autocompletion of e-mails to solving complicated math equations. however these very large pre-trained neural language models are also difficult to control beyond providing a prompt for a generated continuation. this makes very large language models ill-suited for co-creative tasks wherein a human works with a language model in an iterative fashion to produce novel content, such as stories or poems. co-creative tasks require an ability to not only prompt the language model but to guide the generation with, for example, style, context, or topic constraints. conditional generation is a family of text generation methods that attempt to provide controllability by either directly modifying the model to accept control signals or posing constraints in the generation process. conditional text generation techniques add an extra input feature @xcite and fine-tuning with additional information embedded @xcite @xcite @xcite @xcite , or by sideloading additional discriminators along with a pre-trained model, without changing base model parameters holisticly @xcite @xcite . we seek "plug-and-play" approaches to controllable text generation wherein new language models can be slotted into existing generative systems; new language models are being developed and it becomes intractable to update and retrain controlled generation architectures. plug-and-play techniques such as @xcite aim to only intervene with the outputs-a vector of logits-of a generative language model. this becomes especially important as the latest iteration of very large pre-trained language models such as @xcite restrict access to the hidden states and layer weights of models. as language models improve, they can be easily incorporated into existing, controllable generation frameworks. we present plug-and-blend foot_0 , an efficient plugand-play generative framework for controllable text generation that (a) works with the logit outputs of any language model; (b) facilitates fine control of generated sentences by allowing continuous bias towards specific control codes; and (c) allows multiple control codes representing style and topic constraints to be provided in overlapping contexts. these control codes can be blended together to generate content that meets multiple style or topic constraints. we describe that these key capabilities empower latent space walking in the hyperspace of generated sentences, and show a simple content planning technique that utilizes this feature to generate paragraphs regarding user intentions in a co-authoring. we present our work in the context of automated story generation wherein a human author provides a prompt as well as a high-level control specification for topics. ## preliminaries generative language models (lms), specifically continuation models, take a context ("prompt") and generate a continuation by predicting the next tokens. this is achieved by optimizing the model parameters θ that best estimates the probability density of a sequence of word tokens x 1:@xmath0, . . . , x t } represented as an auto-regressive factorization by iteratively predicting a distribution on the next token given the previous tokens, a continuation can be generated by repeatedly sampling and attach the selected token back to the "previous" tokens for the next step. sequences generated this way are not controlled; to control the generated sequence, an attribute represented as a class variable @xcite that could describe sentiment or topics can be introduced to equation (1) to form a class-conditional language model (cc-lm): to decouple the conditional generation component, c, from the unconditional part, p lm (x 1:t ), @xcite proposed the gedi framework and an algorithm to enable a separate controlling model to guide the generation process of a base language model. instead of tackling p θ (x 1:t | c) directly, they train a contrastive discriminator model on the side to estimate ## the plug-and-blend framework our plug-and-blend framework consists of two components (see figure 1 ): (1) a blending generative model that is responsible for plug-and-play controlled continuations using the control specifications; and (2) a planner that plans and assigns control specifications based on control sketches. a control sketch is a high-level specification of what topics should be present in the story and what portions of the story each topic should approximately appear in. this provides a human co-creator the ability to guide the generator loosely, with a broad range per topic, or tightly, with a narrow range per topic. we envision a co-creative loop wherein the human user provides a control sketch and iteratively updates the control sketch based on generation results, refining the topics and refining the ranges for the topics. the user interface for eliciting control sketches from a human is outside the scope of this paper and experiments about the co-creative loop are left for future work. the next sections provide the algorithmic support for control sketches. ## experiments for our experiments, we use the gpt2-large model fine-tuned on rocstories @xcite as our base language model. fine-tuning gpt2 on rocstories results in a model that generates short stories about common everyday situations. we pair the language model with a pretrained gedi (which in turn is based on gpt-2-medium) trained on ag-news foot_2 as the guiding model. across all setups, at generation time, we use greedy decoding with repetition penalty described in @xcite , and only use the first sentence generated as the output, discarding every token after it if any. since there is no ground truth for any generated sequence, metrics such as bleu and other n-grambased metrics are not applicable. this poses a unique challenge in evaluating our system, limiting us to unsupervised metrics. in this section, we report evaluation of our blending generative model from two aspects: ## discussion our experiments suggest that there is a trade-off between control fidelity and fluency. as figures 2 and 3 show, a higher total control strength results in overall better τ -a scores, meaning more sensitivity and ability to correctly differentiate between topic blends, but worse perplexity, risking less fluent language. in practice, an iterative deepening algorithm where multiple control strengths are used to generate multiple candidate sentences per line, can be used. control strength modifiers of 1x, 2x, 3x, 4x, etc. can be tried and the best generated sentence, as measured by perplexity (or any other task-specific metric), is selected. this can, just like how multiple control codes are handled, be implemented very efficiently. the current planner is heuristic. empirically we find the heuristic to create good blends. we envision a planner that can be parameterized and learn from demonstrations. reinforcement learning, in which the context and control sketches work as world states, can choose control configurations as actions. feedback (reward) from the user would be necessary. this would incorporate the plug-andblend technique into a human-in-the-loop creative in this paper we have made no assumptions about how control sketches are acquired from the human user. we envision a processes whereby the human co-creator has an idea of what the final product-in this case a story-should be and can express that vision at the level of how topics manifest through the story. supporting a co-creative ## conclusions in this paper, we present plug-and-blend, a plugand-play framework that enhances a base lm, enables controllable generation with continuousweighted control codes, along with capability of generating paragraphs based on control sketches, all without access to internal knowledge of this base lm. these capabilities will fuel a new generation of controllable generation applications with the key assets of decoupling between the controllable component and the generative component, and easiness of adapting to new advancements in the field of generative lms.