Extract, Denoise and Enforce: Evaluating and Improving Concept Preservation for Text-to-Text Generation (2021)

Abstract: Prior studies on text-to-text generation typically assume that the model could figure out what to attend to in the input and what to include in the output via seq2seq learning, with only the parallel training data and no additional guidance. However, it remains unclear whether current models can preserve important concepts in the source input, as seq2seq learning does not have explicit focus on the concepts and commonly used evaluation metrics also treat them equally important as other tokens. In this paper, we present a systematic analysis that studies whether current seq2seq models, especially pre-trained language models, are good enough for preserving important input concepts and to what extent explicitly guiding generation with the concepts as lexical constraints is beneficial. We answer the above questions by conducting extensive analytical experiments on four representative text-to-text generation tasks. Based on the observations, we then propose a simple yet effective framework to automatically extract, denoise, and enforce important input concepts as lexical constraints. This new method performs comparably or better than its unconstrained counterpart on automatic metrics, demonstrates higher coverage for concept preservation, and receives better ratings in the human evaluation. Our code is available at https://github.com/morningmoni/EDE .

## introduction text-to-text generation is an important research problem with a broad set of applications, such as dialog response generation @xcite , headline generation @xcite , and summarization @xcite . a distinct feature of text-to-text generation (vs. free-form text generation) is that it is often desired to preserve the mainstream text-to-text generation methods are mostly data-driven, which "hope" to learn meaningful mappings between source input and target output via sequence-to-sequence (seq2seq) learning -this is particularly the case for the recent pre-trained language models (plms) @xcite , where seq2seq learning is expected to identify what to attend to in the source input and what to include in the model output, with access to only parallel training data. however, as seq2seq learning does not explicitly focus on key concepts (e.g., named entities) and commonly used evaluation metrics (e.g., bleu and rouge) also treat all the tokens in a sequence equally important, it is unclear how many of the important input concepts can be (or have been) preserved. existing attempts to alleviate the above issue use soft constraints, such as copy mechanism or additional attention, to focus on (certain parts of) the source input @xcite @xcite . nevertheless, they still lack explicit guidance and resort to seq2seq learning itself to figure out what is important, without any guarantee of the model output or evaluation on the input concept preservation. explicit guidance for text generation can be achieved by lexically (hard) constrained generation (lcgen), which specifies lexical constraints (tokens) that must be included in the model output. however, to what extent guiding text-totext generation with lexical constraints works in general remains unknown, as existing studies on lcgen @xcite @xcite focus on scenarios where gold (ground-truth) constraints are given (e.g., generate a story using user-specified keywords), while in generic text-to-text generation tasks the constraints (target output) are unavailable. in this paper, we present a systematic analysis on generic text-to-text generation to understand (1) the abilities of seq2seq models, especially the plms, for preserving important input concepts and (2) whether more explicit guidance that uses important input concepts as lexical constraints can complement seq2seq learning. we select four representative tasks where preserving important concepts (entities) in the source input and incorporating them in the model output is essential, including question generation, knowledge-grounded dialog, headline generation, and abstractive summarization. we examine the effectiveness of guiding generation with the important input concepts as lexical constraints, where the concepts are either obtained by comparing the source input with the target output in an analytical study or automatically extracted from the source input in a practical setting. specifically, in the analytical study, we first evaluate how many of the concepts found in the target output (named gold concepts) are available in the source input, and how many of them are already preserved by seq2seq models. we then investigate the room for improvement if we guide generation with the gold concepts as lexical constraints (named gold constraints). next, when the target output is unavailable, we propose a simple yet effective framework, named ede, to automatically extract, denoise, and enforce input concepts as lexical constraints. ede achieves significant improvement on two tasks and moderately better or comparable performance on the other two under sequence-level automatic evaluation. moreover, ede receives better ratings in the human evaluation and demonstrates higher coverage in the concept-level evaluation on all the tasks that we examine. contributions. (1) we analyze whether current plms for text-to-text generation are good enough for preserving important input concepts via seq2seq learning. (2) we study the usefulness of guiding generation explicitly with important input concepts as lexical constraints. (3) we propose a framework to automatically extract, denoise, and enforce concept constraints, which achieves comparable or better performance than unconstrained generation on a range of text-to-text generation tasks. ## analysis on concept preservation in this section, we conduct a series of analytic experiments on concept preservation, which tries to answer the following two questions: q1: is the current "plms+seq2seq fine-tuning" paradigm for text-to-text generation good at preserving the important concepts in the source input? q2: what is the room for improvement if we guide text-to-text generation by enforcing high-quality (gold) concepts as lexical constraints? ## guiding text-to-text generation with extracted concepts in this section, we study a practical setting where automatically extracted concepts are used as constraints (named automatic constraints), with the following research question in mind: q3: can we automatically extract important input concepts and use them as lexical constraints to guide text-to-text generation? ## experiments in this section, we conduct experiments of guiding generation with automatic constraints and compare with the base as well as state-of-the-art models. the constraints are automatically extracted without access to the target output. ## conclusion in this work, we examine whether current pretrained language models for text-to-text generation are good enough for preserving important concepts in the source input without explicit guidance but pure seq2seq learning. we conduct extensive analytical experiments on a range of text-to-text generation tasks and study when adding important input concepts as lexical constraints can help guide textto-text generation. we propose a simple yet effective framework for automatic constraint extraction, denoising, and enforcement, which is shown to perform comparably or better than unconstrained generation in various text-to-text generation tasks and better preserves important input concepts.