Parallel Refinements for Lexically Constrained Text Generation with BART (2021)

Abstract: Lexically constrained text generation aims to control the generated text by incorporating certain pre-specified keywords into the output. Previous work injects lexical constraints into the output by controlling the decoding process or refining the candidate output iteratively, which tends to generate generic or ungrammatical sentences, and has high computational complexity. To address these challenges, we proposed Constrained BART (CBART) for lexically constrained text generation. CBART leverages the pre-trained model, BART and transfers part of the generation burden from the decoder to the encoder by decomposing this task into two sub-tasks, thereby improving the sentence quality. Concretely, we extended BART by adding a token-level classifier over the encoder, aiming at instructing the decoder where to replace and insert. Guided by the encoder, the decoder refines multiple tokens of the input in one step by inserting tokens before specific positions and re-predicting tokens at a low confidence level. To further reduce the inference latency, the decoder predicts all tokens in parallel. Experiment results on One-Billion-Word and Yelp show that CBART can generate plausible text with high quality and diversity while largely accelerating inference.

## introduction controllable text generation aims to generate text in a controlled way, such as transferring text style @xcite @xcite and generating text with control codes @xcite . lexically constrained text generation requires that the given keywords must appear in the output, which can be applied to incorporating keywords into a dialog response @xcite , creating a story with keywords @xcite , generating advertisements for products @xcite and writing a concrete meeting summary based on several key phrases. to generate sentences with keywords, @xcite proposed a backward and forward language model (b/f-lm). @xcite applied adversarial learning @xcite to b/f-lm. both models are limited to generating text with one keyword. to incorporate multiple keywords into machine translation, @xcite proposed grid beam search (gbs) by adding an additional constrained dimension to beam search. however, gbs does not consider the future lexical constraints when generating previous tokens, thereby degrading the quality of generated sentences. recently, markov chain monte carlo (mcmc) sampling has been applied to text generation @xcite @xcite . compared with gbs, mcmc-based models can iteratively refine tokens based on contexts. cgmh @xcite uses metropolis-hastings sampling to generate constrained sentences with a series of actions, such as insertion, deletion, and replacement. in most cases, refinements conducted by cgmh are invalid because of the randomly chosen actions and positions. to solve this problem, the gradient information @xcite and a token-level classifier @xcite are used to determine the position to be edited and the action to be taken. these models are computation-intensive as they can update only one token in each step. pointer @xcite reduces the inference latency by refining multiple tokens in one step. however, pointer is based on bert @xcite and imposes all the burden of generation on the decoder. previous work @xcite has shown that bart is more suitable than bert for text generation. nevertheless, bart can not be directly applied to constrained text generation. if we feed the keywords into the encoder, the decoder will not guarantee that the output contains the given keywords. to alleviate the above problems, we propose cbart, a parallel refinement model for lexically constrained text generation, shown in figure 1 . cbart benefits from the large-scale pre-trained model, i.e., bart. in addition, cbart shifts part of the decoder's burden to the encoder. specifically, we put a token-level classifier over bart's encoder, in charge of analyzing the input and providing the decoder with coarse-grained modification information, such as the positions to be refined and the actions to be conducted. the refinement information provided by the encoder enables the decoder to revise multiple tokens of the input in one step to make the sentence more fluent, such as inserting missing tokens before certain positions and replacing inappropriate tokens with other tokens. in addition, the decoder predicts all tokens simultaneously, which further speeds up inference. ## problem definition lexically constrained text generation aims to incorporate the given keywords into the generated text. given a set of lexical constraints, c 1 , c 2 , . . . , c k , this task aims to find a fluent text by maximizing the conditional probability: ## methodology in section 3.1, we will first introduce the proposed model, cbart. then, we will show how to create synthetic datasets to train cbart in section 3.2. finally, we will introduce several parallel decoding strategies for generating text, the repetition penalty mechanism used to discourage the generation of repetitive tokens, and the termination criterion for inference in section 3.3. ## conclusion in this paper, we presented cbart for lexically constrained text generation. compared with previous work, cbart leverages bart and transfers part of the burden from the decoder to the encoder. furthermore, cbart refines multiple tokens in parallel in each refinement step, thus accelerating the inference. experiment results on one-billion-word and yelp datasets show that cbart can generate fluent and diverse text with lexical constraints and dramatically reduce the inference time.