Enconter: Entity Constrained Progressive Sequence Generation via Insertion-based Transformer (2021)

Abstract: Pretrained using large amount of data, autoregressive language models are able to generate high quality sequences. However, these models do not perform well under hard lexical constraints as they lack fine control of content generation process. Progressive insertion based transformers can overcome the above limitation and efficiently generate a sequence in parallel given some input tokens as constraint. These transformers however may fail to support hard lexical constraints as their generation process is more likely to terminate prematurely. The paper analyses such early termination problems and proposes the ENtity CONstrained insertion TransformER(ENCONTER), a new insertion transformer that addresses the above pitfall without compromising much generation efficiency. We introduce a new training strategy that considers predefined hard lexical constraints (e.g., entities to be included in the generated sequence). Our experiments show that ENCONTER outperforms other baseline models in several performance metrics rendering it more suitable in practical applications.

## introduction the field of natural language generation (nlg) @xcite has seen significant improvements in recent years across many applications such as neural machine translation @xcite , text summarization @xcite , poem generation @xcite and recipe generation (h. @xcite . constrained text generation (ctg) is one of the challenging problems in nlg that is important to many real world applications but has not been well addressed. ctg imposes input constraints which may be in the form of objects expected to exist in 1 our code is available at https://github.com/ larc-cmu-smu/enconter the generated text or rules over objects in the generated text @xcite . the objects here can be entities, phrases, predefined nouns, verbs, or sentence fragments. the constraints can be categorized into two types: (1) hard-constraints which require mandatory inclusion of certain objects and complete compliance of given rules @xcite @xcite @xcite ; and (2) softconstraints which allow the some constraint objects or rules to be not strictly enforced in the generated text @xcite . as autoregressive models generate tokens from left to right, they cannot easily support constraints involving multiple input objects, hard-constrained text generation therefore often requires non-autoregressive models. recently, @xcite proposed a non-autoregressive hard-constrained text generation model (pointer) that generates a text sequence in a progressive manner using an insertiontransformer @xcite . to train an insertion transformer to generate a missing token between every two tokens in an input sequence, the training data is prepared by masking "less important" tokens in the original text sequence in an alternating manner. the process is then repeated using the masked input sequence as the new original sequence, and further masking alternate tokens in it. the process ends when the masked sequence meets some length criteria. while pointer shows promising results, it does not consider hard constraints which involve entities that must be included in the generated sequence. such entity constraint requirements are unfortunately prevalent in many applications. for example, we may want to generate a job description with some given skills, or a food recipe with some given ingredients. a naive approach to the problem is to apply con-straints on the pointer's masking strategy forcing it to keep entity tokens. we call this modified model pointer-e. although this allow entity information entering pointer-e, another problem rises. pointer-e suffers from cold start problem which refers to the inability to generate meaningful tokens at the early stages of inference forcing the generation to end prematurely. this issue can be attributed to the pointer-e's top-down masking strategy for training the insertion transformer and the tokens of input entities not evenly spread out across the sequence. to solve the cold start generation problem, we propose enconter that incorporates bottom-up masking strategy. enconter supports hard entity constraints, and encourages more meaningful tokens to be generated in the early stages of generation thus reducing cold start. on top of that, we further introduce the balanced binary tree scheme @xcite to reduce the number of stages in generation and to improve the efficiency of generation. ## entity constrained sequence generation in this section, we first describe the state-of-theart pointer model, its preprocessing of training data and inference process. we highlight the pitfalls of the entity constrained variant of pointer, pointer-e. we then present our proposed entity constrained insertion transformer called encon-ter. ## empirical analysis of pointer-e and enconter in this section, we conduct an analysis of the data preprocessing step in pointer-e, greedy en-conter and bbt-enconter. our objective is to empirically evaluate the characteristics of training data generated for the two models. we have left out pointer as it is inherently not entity-aware and pointer-e is its entity-aware variant. we first present the two datasets used in this study. ## conclusions constrained text generation is an important task for many real world applications. in this paper, we focus on hard entity constraints and the challenges associated with enforcing them in text generation. our analysis of the state-of-the-art insertion transformers reveals issues, namely, cold start problems and inefficient generation. we therefore propose two insertion transformer models, greedy en-conter and bbt enconter, that use a bottomup preprocessing strategy to prepare training data so as to eliminate the cold start problem caused by top-down preprocessing strategy. bbt enconter further incorporates a balanced tree reward scheme to make the generation process more efficient. through experiments on real world datasets, we show that the two models outperform the strong baselines, pointer-e and gpt2, in recall, quality and failure rate while not compromising much generation efficiency. for future research, it will be interesting to consider more diverse constraints (e.g., soft constraint, rules, etc.) and user interaction in the generation process to expand the scope of applications that can benefit from this research.