Mention Flags ( MF ): Constraining Transformer-based Text Generators (2021)

Abstract: This paper focuses on Seq2Seq (S2S) constrained text generation where the text generator is constrained to mention specific words which are inputs to the encoder in the generated outputs. Pre-trained S2S models or a Copy Mechanism are trained to copy the surface tokens from encoders to decoders, but they cannot guarantee constraint satisfaction. Constrained decoding algorithms always produce hypotheses satisfying all constraints. However, they are computationally expensive and can lower the generated text quality. In this paper, we propose Mention Flags (MF), which traces whether lexical constraints are satisfied in the generated outputs in an S2S decoder. The MF models can be trained to generate tokens in a hypothesis until all constraints are satisfied, guaranteeing high constraint satisfaction. Our experiments on the Common Sense Generation task (CommonGen) (Lin et al., 2020), End2end Restaurant Dialog task (E2ENLG) (Duˇsek et al., 2020) and Novel Object Captioning task (nocaps) (Agrawal et al., 2019) show that the MF models maintain higher constraint satisfaction and text quality than the baseline models and other constrained decoding algorithms, achieving state-of-the-art performance on all three tasks. These results are achieved with a much lower run-time than constrained decoding algorithms. We also show that the MF models work well in the low-resource setting.

## introduction this paper focuses on seq2seq (s2s) constrained text generation where a set of encoder input tokens are required to be present in the generated outputs. for example, keyword-to-text @xcite , data-to-text @xcite and image-to-text (lin et al., 2014; 1 the source code for this paper is released at https: //github.com/garyyufei/acl2021mf figure 1 : an overview of the mention flag mechanism for transformer-based s2s models. here, the tokens flower and bee are required to appear in the generated outputs. each generated token has a corresponding set of mention flags which informs the decoder whether each lexical constraint has been satisfied in the current decoder input sequence. for example, the mention flag for flower is set (indicated by orange dots) from the third token because it is generated at the second step. both token and mention flag embeddings are the input to the decoder, but mention flags are injected into the decoder in a different way to the tokens (see fig. 3 ). note that task specific encoder inputs have been omitted for brevity. @xcite require the models to mention all or some of the input keywords, key-value pairs and image object labels (respectively), potentially with linguistic variants, in the generated outputs. large (pre-trained) transformer-based s2s models such as t5 @xcite can be trained (fine-tuned) to perform this task. however, they only learn to copy the surface tokens from encoder inputs to the decoder outputs and there is no underlying mechanism guaranteeing good constraint satisfaction (the ratio of satisfied lexical constraints to given lexical constraints). constrained beam search (cbs) @xcite and related algorithms can guarantee outputs satisfying all constraints, however they are much slower than the standard beam search algorithm. in addition, as they are all inference-based algorithms, their corresponding models are not aware of the constraint words or phrases, the resulting generation could be poor. ideally, a method for producing constrained text should: a) generate high-quality text; b) achieve high constraint satisfaction; c) have an efficient inference procedure. to this end, we propose mention flags (mf), which trace whether a lexical constraint has been realized in partial decoder outputs. specifically, each decoder input token is provided with a set of flags indicating which constraints have been satisfied up to that token. as shown in fig 1 , the mention flags for flower is set from the third step, because flower is generated at the second step. we represent the three possible mention flags as separate trainable embeddings and inject them into the decoder of the s2s transformer-based text generator. the dynamic mention flags explicitly inform the model about which constraints have been satisfied, which is helpful for the models to produce high-quality text satisfying the constraints (goal a). during training, all the mention flags are set when the model is tasked to generate the end-of-sequence (eos) token, strongly encouraging the model not to stop generation until all constraints are satisfied (goal b). the mf models only require ordinary decoding algorithms. their inference time and memory requirements are similar to their baseline models (goal c). we conduct experiments on three benchmarks: commonsense generative reasoning (common-gen) @xcite , where the only input is a set of words representing concepts, and the output text is constrained to include all of them; end-to-end data-to-text (e2enlg) @xcite , where the constraints are meaning representations with lexicalised attributes and values that the output text should mention; and novel object captioning at scale (nocaps) @xcite , where constraints are salient image objects that should be mentioned in the generated caption. compared to the constrained decoding algorithms, the mf models can produce higher-quality text with a similar level of constraint satisfaction and much less inference run-time and memory. mention flags are a general mechanism that improves constraint satisfaction in the non-pre-trained and pre-trained s2s transformer-based models. furthermore, our experiments show that the mf models can satisfy novel constraints (i.e, involving words or phrases not seen during training) and they work well in low-resource settings. our mf models set a new state-of-the-art in these three tasks. ## background in this paper, we focus on constraining transformerbased text generation models due to their popularity and success in various domains, especially in largescale pre-trained language models @xcite . previous work can be roughly categorized into two streams: s2s training approaches and constrained decoding approaches: training s2s models s2s models can implicitly capture the co-occurrence between encoder and decoder sequences, particularly pre-trained ones such as t5 @xcite and bart @xcite . @xcite uses a special gate to control what information will be generated in the following steps. @xcite have shown that the t5 models achieve state-of-the-art results in various data-to-text tasks, requiring copying from encoder to decoder, after fine-tuning. as an alternative, the copy mechanism @xcite explicitly learns where to copy the input constraints into the output by adding an extra copy pathway to the models. however, these approaches cannot control or guarantee their constraint satisfaction. @xcite also have observed lower constraint satisfaction in the above methods, compared to the constrained decoding approaches. constrained decoding these algorithms, including constrained beam search (cbs) @xcite and grid beam search (gbs) @xcite , maintain a set of states which have their own size-k beams and only allow hypotheses satisfying specific constraints to be considered during inference. each cbs state corresponds to the hypotheses satisfying different constraints (exponential in the number of constraints) and the gbs states correspond to the hypotheses satisfying the same number of constraints (linear to constraint number). @xcite al. (2019); juraska et al. (2018); dušek and jurčíček (2016) also modify their inference algorithm in a similar way to fulfill specific output requirements. however, they significantly increase the inference run-time and memory and can produce sub-optimal outputs. ## method this section first formulates constrained text generation tasks, then introduces mention flags and their integration with transformer-based text generators. ## experiments we conduct experiments on three benchmarks with different forms of constraints including commonsense generative reasoning (commongen) @xcite with keyword constraints, end-to-end restaurants dialog (e2enlg) @xcite with key-value constraints, and novel object captioning at scale (nocaps) @xcite with visual object word constraints. we integrate mention flags with a three-layer standard s2s transformer models (trans, l3) @xcite and pre-trained t5 models @xcite for each task. the t5 models achieve state-of-the-art results in various data-to-text tasks @xcite . for the t5-base and t5-large models, we use the implementation of t5 models in the huggingface transformers foot_0 . the trans, l3 models share the same implementation of the t5-base models, except that it is not initialized with the pretrained parameters and it only uses 3 layers, rather than 12 layers, for both encoder and decoder. in addition, to improve the generalization of our pretrained model, we freeze the parameters in the self-attention module and feed-forward layers in each layer of the t5 decoder. this parameters freezing technology is applied to both t5 baseline models and the mf models in all of our experiments. we report constraint satisfaction for all tasks. we use gbs in the commongen task (max 5 constraints) and cbs in the e2enlg (max 1 constraint) and nocaps (max 2 constraints) task. ## conclusion and future work in this paper, we propose mention flags to constrain transformer-based text generators via injecting mention status embeddings into text decoders. our extensive experiments on three different tasks have shown the effectiveness of mention flags in maintaining high generated text quality and excellent constraint satisfaction, comparing favourably to competitive constrained decoding algorithms. we plan to expand mention flags i) to control larger input source text such as constrained text summarization and machine translation; ii) to handle larger granularity such as sentence-level.