POINTER : Constrained Progressive Text Generation via Insertion-based Generative Pre-training (2020)

Abstract: Large-scale pre-trained language models, such as BERT and GPT-2, have achieved excellent performance in language representation learning and free-form text generation. However, these models cannot be directly employed to generate text under specified lexical constraints. To address this challenge, we present POINTER (PrOgressive INsertion-based TransformER), a simple yet novel insertion-based approach for hard-constrained text generation. The proposed method operates by progressively inserting new tokens between existing tokens in a parallel manner. This procedure is recursively applied until a sequence is completed. The resulting coarse-to-fine hierarchy makes the generation process intuitive and interpretable. We pre-train our model with the proposed progressive insertion-based objective on a 12GB Wikipedia dataset, and fine-tune it on downstream hard-constrained generation tasks. Non-autoregressive decoding yields a logarithmic time complexity during inference time. Experimental results on both News and Yelp datasets demonstrate that Pointer achieves state-of-the-art performance on constrained text generation. We released the pre-trained models and the source code to facilitate future research.

## introduction real-world editorial assistant applications must often generate text under specified lexical constraints, for example, convert a meeting note with key phrases into a concrete meeting summary, recast a user-input search query as a fluent sentence, generate a conversational response using grounding facts @xcite , or create a story using a pre-specified set of keywords @xcite @xcite . generating text under specific lexical constraints is challenging. constrained text generation broadly falls into two categories, depending on whether inclusion of specified keywords in the output is mandatory. in soft-constrained generation @xcite , keyword-text pairs are typically first constructed (sometimes along with other conditioning information), and a conditional text generation model is trained to capture their co-occurrence, so that the model learns to incorporate the constrained keywords into the generated text. while soft-constrained models are easy to design, even remedied by soft enforcing algorithms such as attention and copy mechanisms @xcite @xcite , keywords are still apt to be lost during generation, especially when multiple weakly correlated keywords must be included. hard-constrained generation (hokamp and @xcite @xcite @xcite , on the other hand, requires that all the lexical constraints be present in the output sentence. this approach typically involves sophisticated design of network architectures. hokamp and @xcite construct a lexical-constrained grid beam search decoding algorithm to incorporate constraints. however, @xcite observe that a naive implementation of this algorithm has a high running time complexity. @xcite introduces a sampling-based conditional generation method, where the constraints are first placed in a template, then words in a random position are either inserted, deleted or updated under a metropolis-hastings-like scheme. however, individually sampling each token results in slow convergence, as the joint distribution of all the tokens in a sentence is highly correlated. @xcite propose a tree-based text generation scheme, where a token is first generated in an arbitrary position, and then the model recursively stage generated text sequence 0 (x 0 ) sources sees structure perfectly 1 (x 1 ) sources company sees change structure perfectly legal 2 (x 2 ) sources suggested company sees reason change tax structure which perfectly legal . 3 (x 3 ) my sources have suggested the company sees no reason to change its tax structure , which are perfectly legal . 4 (x 4 ) my sources have suggested the company sees no reason to change its tax structure , which are perfectly legal . motivated by the above, we propose a novel nonautoregressive model for hard-constrained text generation, called pointer (progressive insertionbased transformer). as illustrated in due to the resemblance to the masked language modeling (mlm) objective, bert @xcite can be naturally utilized for initialization. further, we perform large-scale pre-training on a large wikipedia corpus to obtain a pre-trained pointer model that which can be readily finetuned on specific downstream tasks. the main contributions of this paper are summarized as follows. (i) we present pointer, a novel insertion-based transformer model for hardconstrained text generation. compared with previous work, pointer allows long-term control over generation due to the top-down progressive structure, and enjoys a significant reduction over emperical time complexity from o(n) to o(log n) at best. (ii) large-scale pre-training and novel beam search algorithms are proposed to further boost performance. (iii) we develop a novel beam search algorithm customized to our approach, further improving the generation quality. (iv) experiments on several datasets across different domains (including news and yelp) demonstrates the superiority of pointer over strong baselines. our approach is simple to understand and implement, yet powerful, and can be leveraged as a building block for future research. non-autoregressive generation many attempts have been made to use non-autoregressive models for text generation tasks. for neural machine translation, the promise of such methods mostly lies in their decoding efficiency. for example, @xcite employs a non-autoregressive decoder that generates all the tokens simultaneously. generation can be further refined with a post-processing step to remedy the conditional independence of the parallel decoding process @xcite @xcite @xcite . deconvolutional decoders @xcite have also been studied for title generation and machine translation. the insertion transformer @xcite @xcite is a partially autoregressive model that predicts both insertion positions and tokens, and is trained to maximize the entropy over all valid insertions, providing fast inference while maintaining good performance. our pointer model hybridizes the bert and insertion transformer models, inheriting the advantages of both, and generates text in a progressive coarse-to-fine manner. ## experiments we evaluate the pointer model on constrained text generation over news and yelp datasets. details of the datasets and experimental results are provided in the following sub-sections. the pretrained models and the source code are available at github foot_1 . ## conclusion we have presented pointer, a simple yet powerful approach to generating text from a given set of lexical constraints in a non-autoregressive manner. the proposed method leverages a large-scale pre-trained model (such as bert initialization and our insertion-based pre-training on wikipedia) to generate text in a progressive manner using an insertion-based transformer. both automatic and human evaluation demonstrate the effectiveness of pointer. in future work, we hope to leverage sentence structure, such as the use of constituency parsing, to further enhance the design of the progressive hierarchy. our model can be also extended to allow inflected/variant forms and arbitrary ordering of given lexical constraints. keywords children fault left charge oracle my relationship with my children was seriously affected as they were told time and again that everything was my fault , they were even left ' in charge ' of me if my wife went out of the house . cgmh his two children are the rare fault that left the police charge nmstg but despite children from hospitals to last one by fault backing this month , there have arrived as mr hunt has been left charge . pointer (greedy, base) but i found that these children were not at school however this was not their fault , and if so they were left without a parent in charge . pointer (ilbs, base) but my lovely wife and children consider that it is not our own fault and we should not be left alone in charge . pointer (greedy, large) i said to my children : it ' s not his fault the parents left him ; the parents should be in charge of him . wiki zeroshot but for the children who are not at a fault , they are left behind on the charge . keywords estate pay stay policy oracle how many people on the estate does he think will be affected by the new pay -to -stay policy ? cgmh an economic estate developer that could pay for it is that a stay policy nmstg as estate owners , they cannot pay for households for hundreds of middle -income property , buyers stay in retail policy . pointer (greedy, base) if you buy new buildings from real estate company, you may have to pay down a mortgage and stay with the policy for financial reasons . pointer (ilbs, base) but no matter what foreign buyers do , real estate agents will have to pay a small fee to stay consistent with the policy . pointer (greedy, large) but it would also be required for estate agents , who must pay a larger amount of cash but stay with the same policy for all other assets . wiki zeroshot however , his real estate agent agreed to pay him for the stay under the same policy . keywords managers cut costs million oracle he was the third of four managers sent in to cut costs and deal with the city ' s $ 13 million deficit . cgmh the managers , who tried to cut off their costs , added 20 million euros nmstg business managers cut demand for more expensive costs in 2017 -by october -is around 5 million 8 per cent , and has fallen by 0 . 3 per cent in january and 2017 . pointer (greedy, base)