A uto T emplate: A Simple Recipe for Lexically Constrained Text Generation (2024)

Abstract: Lexically constrained text generation is one of the constrained text generation tasks, which aims to generate text that covers all the given constraint lexicons. While the existing approaches tackle this problem using a lexically constrained beam search algorithm or dedicated model using non-autoregressive decoding, there is a trade-off between the generated text quality and the hard constraint satisfaction. We introduce AutoTemplate, a simple yet effective lexically constrained text generation framework divided into template generation and lexicalization tasks. The template generation is to generate the text with the placeholders, and lexicalization replaces them into the constraint lexicons to perform lexically constrained text generation. We conducted the experiments on two tasks: keywords-to-sentence generations and entity-guided summarization. Experimental results show that the AutoTemplate outperforms the competitive baselines on both tasks while satisfying the hard lexical constraints. The code is available at https://github.com/megagonlabs/autotemplate

## introduction text generation often requires lexical constraints, i.e., generating a text containing pre-specified lexicons. for example, the summarization task may require the generation of summaries that include specific people and places @xcite , and advertising text requires the inclusion of pre-specified keywords @xcite . however, the black-box nature of recent text generation models with pre-trained language models @xcite makes it challenging to impose such constraints to manipulate the output text explicitly. hokamp and @xcite and others tweaked the beam search algorithm to meet lexical constraints by increasing 1: illustration of autotemplate. we build the model input x by concatenating the constraint lexicons z with mask tokens. for the conditional text generation task, we further concatenate input document x. we also build the model output ỹ by masking the constraint lexicons in summary y. then, we can train a standard sequence-to-sequence model, p(ỹ | x), generate masked template ỹ given input x, and post-process to achieve lexically constrained text generation. the weights for the constraint lexicons, but it often misses to include all the constrained lexicons. @xcite and others introduced specialized non-autoregressive models @xcite that insert words between the constraint lexicons, but the generated texts tend to be lower-quality than standard autoregressive models. on the other hand, classical template-based methods @xcite ) can easily produce text that satisfies the lexical constraints as long as we can provide appropriate templates. nevertheless, it is impractical to prepare such templates for every combination of constraint lexicons unless for specific text generation tasks where the output text patterns are limited, such as data-to-text generation tasks @xcite . still, if such a template could be generated automatically, it would be easier to perform lexically constrained text generation. we propose autotemplate, a simple framework for lexically constrained text generations by automatically generating templates given constrained lexicons and replacing placeholders in the templates with constrained lexicons. the autotemplate, for example, can be used for summarization tasks, as illustrated in figure 1 , by replacing the constraint lexicons (i.e., {japan, akihito}) in the output text with placeholder tokens during training and using these constraints as a prefix of the input, creating input-output pairs, and then using a standard auto-regressive encoder-decoder model @xcite to train the autotemplate model. during the inference, the constraint lexicons are prefixed in the same way, the model generates the template for the constraints, and the placeholder tokens are replaced with the constraint lexicons to perform lexically constrained text generation. we evaluate autotemplate across two tasks: keywords-to-sentence generation on one-billion-words and yelp datasets ( §3.1), and entity-guided summarization on cnndm @xcite and xsum datasets @xcite ( §3.2). the autotemplate shows better keywordsto-sentence generation and entity-guided summarization performance than competitive baselines, including autoregressive and non-autoregressive models, while satisfying hard lexical constraints. we will release our implementation of autotemplate under a bsd license upon acceptance. ## autotemplate autotemplate is a simple framework for lexically constrained text generation ( §2.1), divided into two steps: template generation ( §2.2) and lexicalization ( §2.3). the template generation task aims to generate the text with placeholders ỹ, which we defined as a template, given constraint lexicons z, and the lexicalization is to replace these placeholders with the constraints to perform lexically constrained text generation. ## experiments we present experiments across two tasks: keywords-to-sentence generation ( §3.1), and entity-centric summarization ( §3.2). ## analysis does autotemplate generate fluent text? au-totemplate decomposes the lexically constrained text generation task into template generation and lexicalization tasks. the template generation task constraint entities: { game boy , apple , chris gallizzi , nintendo } autotemplate: case adds iconic game boy buttons to apple handset. it also lets gamers play their existing cartridges on their handset. developer chris gallizzi said: 'we wanted to create a retro device that can be easily adapted into any modern gamer's arsenal of devices' nintendo advised keeping cartridges away from dust, where possible, to avoid gameplay glitches. constraint entities: { hyperkin , nintendo , game boy color , start and select } autotemplate: hyperkin has designed a case that adds the iconic directional arrows from the nintendo game boy color . it was originally devised as part of an april fool's joke, but the popularity and demand for a real product was so high the firm has announced plans to sell it. it will feature an eight-way d-pad, two action buttons, a start and select button, and a battery that can be charged through the phone. to this end, we compare the fluency of the output text by autotemplate and baselines. we specifically used the grammatical acceptability classifier based on roberta-large fine-tuned on cola dataset @xcite following @xcite 9 and show the micro averaged accuracy of sentence-level grammaticality. 10 we show the results in for the entity-guided summarization task, au-9 https://huggingface.co/cointegrated/ roberta-large-cola-krishna2020 10 although we can also measure fluency using the perplexity of an external language model, it can assign low perplexity to unnatural texts containing common words @xcite . therefore, we decided to evaluate fluency using the classifier. totemplate shows similar fluency with the stateof-the-art autoregressive text generation models, including bart and ctrlsum, indicating that the autotemplate can generate as fluent text as the state-of-the-art direct generation models. ## further related work template-based text generation for classical text generation systems, templates were an important building block @xcite @xcite . the advantage of a template-based system is that it can produce faithful text, but it can produce disfluent text if an inappropriate template is selected. therefore, the current primary approach is to produce fluent text directly from the input using end-to-end neural generation models. more recent studies have focused mainly on using templates as an auxiliary signal to control the stylistic properties of the output text, such as deriving templates as latent variables @xcite @xcite and using retrieved exemplars as soft templates @xcite @xcite . copy mechanism the copy mechanism was originally introduced to deal with the out-ofvocabulary problem in machine translation by se-lecting the words from the source for the generation in addition to the vocabulary, such as the unknown word replacement with post-processing @xcite , and the joint modeling of unknown word probabilities into encoder-decoder models @xcite , but with the advent of subword units @xcite , the unknown word problem has been diminished. thus, the copy mechanism is not widely used now for handling out-of-vocabulary problems. however, the copy mechanism still plays a vital role in more complex text generation tasks such as involving numerical computation @xcite or logical reasoning @xcite . specifically, they produce special tokens that serve as placeholders and replace them with the desired words in postprocessing. autotemplate adapts a similar copy mechanism to perform lexically constrained text generation, showing that it can cover all the constrained entities in its outputs, even for more complex conditioning (more than ten entities). ## conclusions this study proposes autotemplate, a simple yet effective framework for lexically constrained text generation. the core idea is to decompose lexically constrained text generation into two steps, template generation, and lexicalization, by converting the input and output formats. the template generation can be done with standard encoder-decoder models with beam search so that autotemplate can perform lexically constrained text generation without using dedicated decoding algorithms such as non-autoregressive decoding and constrained beam search. experimental results show that the au-totemplate significantly outperforms the competitive baselines across keywords-to-sentence generation and entity-guided summarization tasks while satisfying the lexical constraints.