A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation (2022)

Abstract: We propose Composition Sampling, a simple but effective method to generate diverse outputs for conditional generation of higher quality compared to previous stochastic decoding strategies. It builds on recently proposed plan-based neural generation models (FROST, Narayan et al, 2021) that are trained to first create a composition of the output and then generate by conditioning on it and the input. Our approach avoids text degeneration by first sampling a composition in the form of an entity chain and then using beam search to generate the best possible text grounded to this entity chain. Experiments on summarization (CNN/DailyMail and XSum) and question generation (SQuAD), using existing and newly proposed automaticmetrics together with human-based evaluation, demonstrate that Composition Sampling is currently the best available decoding strategy for generating diverse meaningful outputs.

## introduction in many nlg tasks, it is important to be able to generate multiple diverse outputs from a model. tasks like summarization @xcite and question generation @xcite exhibit one-to-many relationships; there can be multiple semantically diverse summaries or questions for the same source, and it may be useful for a model to be able to generate multiple outputs. yet, the primary focus of recent research in nlg has been on improving the quality of single-best outputs @xcite @xcite @xcite , while diversity remains an unsolved problem @xcite . this is particularly challenging in conditional generation, where diversity in the target sequence should not come at the cost of correctness or faithfulness; for example, alternate summaries are not valuable if they are unfaithful to the input document(s) @xcite . in this work, we investigate decoding methods for generating semantically diverse text which is also faithful to its input focusing on two tasks, namely summarization and question generation. beam search @xcite has proven successful for single-best generation @xcite @xcite , but struggles to generate diverse output @xcite . stochastic sampling strategies, such as top-k sampling @xcite and nucleus sampling @xcite , are better at generating diverse sequences but are not suitable for conditional generation as they degenerate, foot_0 producing output that is not faithful to the source. figure 1 exposes degeneration in summary output using nucleus sampling. to address these shortcomings, we propose composition sampling, a simple but effective hybrid decoding method for diverse and faithful conditional generation. it builds on recently proposed generation models @xcite that are trained to first plan a semantic composition of the target and then generate the text conditioned on the composition and the input. composition sampling first samples a composition in the form of an entity chain and then uses beam search to generate the best possible sequence grounded to the sampled entity chain. unlike top-k or nucleus sampling, it avoids degeneration by instilling diversity in composition, rather than directly on the surface form. our contributions can be summarized as follows: (a) we introduce composition sampling, a simple yet effective decoding method for diverse conditional generation, which combines planning with stochastic sampling; @xcite we propose several metrics to compute semantic diversity in generated text; our metrics are complementary to lexical diversity figure 1 : human written summary, single-best predicted summary using beam search (beam size 8), and diverse summaries with nucleus sampling (@xmath0.95) and our composition sampling for a cnn/dailymail article (shown in the appendix, figure 6 ). we highlight spans in orange that are not faithful to the input. evaluation on two popular summarization tasks, namely highlight generation (cnn/dailymail; hermann et al. 2015) and extreme summarization (xsum; @xcite , and question generation (squad; @xcite , shows that composition sampling is most effective in generating diverse summaries or questions. when assessed by humans, composition sampled summaries are as faithful as the best summaries produced with beam search. in comparison, nucleus sampled summaries can be as diverse but far less faithful. taken together our results demonstrate that composition sampling is currently the best available decoding strategy for generating diverse and meaningful output. 2 ## background conditional generation tasks such as summarization @xcite , data-to-text generation @xcite , and machine translation 2 our checkpoints and spacy annotation code are available at https://github.com/google-research/ language/tree/master/language/frost . @xcite , are typically modeled using attention-based encoder-decoder architectures @xcite @xcite . the encoder first encodes the input text d and then the decoder predicts the output s 1:n (e.g., the translation or summary of d) one token at a time as p(s i |s 1 , . . . , s i-1 ; d), where, n is the output length and s i is the ith token in the output. often these models benefit from large scale task-agnostic pretraining @xcite @xcite @xcite . @xcite develop a plan-based approach for neural summarization; their decoder generates a composition c 1:m of target summary s as p(c j |c 1 , . . . , c j-1 ; d), and then the same decoder produces s as p(s i |s 1 , . . . , s i-1 ; c; d) conditioned on input d and composition c 1:m , with m being the composition length. specifically, they adopt entity chains as the composition c of summary s, under the assumption that entities in the chain ought to be observed in the output summary. during inference, the model takes document d as input and generates c; s, the concatenation of composition and summary sequences, instead of generating s directly; c and s are prefixed with special markers "[content]" and "[summary]", respectively, as shown in figure 2 . if s consists of multiple sentences, markers"|||" denote sentence boundaries in composition c. the approach allows to directly manipulate the content of summaries and their quality. for example, we might inspect the predicted chain during inference and drop entities which are not present in the input document, thereby controlling for hallucinations @xcite . outwith summarization, similar constraints can be easily adapted to other conditional generation tasks. ## composition sampling composition sampling is a novel hybrid method which combines stochastic sampling with maximization-based decoding, whilst leveraging plan-based generation @xcite . specifically, we employ nucleus sampling to obtain diverse compositions c sample from p(c|d) where d is the input text and c are entity chains (prefixed with "[content]" in figure 2 ). we first employ nucleus sampling to obtain diverse compositions from p(c|d), where d is the input text. and then employ beam search to generate the most-likely diverse output s (prefixed with "[summary]" in figure 2 ), given input d and composition c sample as p(s|c sample ; d). we will experimentally show that composition sampling enables the generation of fluent, faithful and diverse texts for conditional generation. why entity chains? unlike top-k or nucleus sampling, composition sampling avoids degeneration by introducing diversity in composition, rather than directly on the surface form. for this to effectively work, the choice of c needs to be well correlated with an underlying notion of "semantic composition", which we want to "diversify"; if c 1 and c 2 are two semantic compositions for input d such that c 1 = c 2 , then two summaries s 1 = arg max s p(s|c 1 ; d) and s 2 = arg max s p(s|c 2 ; d) are bound to be diverse. in our work, we have chosen entity chains to model semantic compositions; entity chains have been widely studied to model entity-level lexical cohesion @xcite and coherence @xcite ) in text. also, entity chains are unique to d, and thus can be easily distinguished from compositions for other inputs. moreover, entity chains provide a very effective knob for content control in abstractive generation, e.g., compositions can be constrained to entities only present in the input document, thereby avoiding hallucinations and entity degeneration. hypothesis 1: if the semantic composition c of the output text s corresponds to entity chains, then learning p(c|d) is much easier than learning p(s|d); d is the input. hence, we can sample from p(c|d) with higher confidence than sampling directly from p(s|d), and then compute arg max s p(s|c; d). we demonstrate the effectiveness of entity chains as a choice for c using the summarization example in figure 3 . the negative log likelihood of generating the summary s from scratch without planning (-log p(s|d)) is 121.18, while the negative log likelihood of generating composition c with planning (-log p(c|d)) is 46.95; hence, the model is much more confident when sampling from p(c|d) than directly from p(s|d). why grounded generation? the generation of s is inherently grounded to its entity composition c; following @xcite , the entity chains are extracted from their targets during training. hence, once the hard part of planning the composition is done, the model is less perplexed during generation of the output. in figure 3 , the plan-based model is more confident in predicting entities than its counterpart without planning; perplexities of predicting entities in the summary with and without planning are 0.24 and 1.36, respectively, and perplexities of generating the whole summary with and without planning are 1.15 and 1.48, respectively. in fact, despite the increased length of the target in the plan-based model (i.e., c 1:m ; s 1:n instead of s 1:n ), we find that the perplexity of predicting the longer sequence (c 1:m ; s 1:n ) is lower than predicting just the summary without any planning, due to grounding (1.16 vs 1.48). overall, p(c; s|d), the plan-based approach, learns a more confident distribution at each decoding step compared to no planing, i.e., p(s|d). for the example in figure 3 , the average cumulative probabilities for the top 15 tokens in the vocabulary distribution at each decoding step are 0.283 for p(s|d) and 0.433 for p(c; s|d). in the following we assess composition sampling for its ability to generate semantically diverse output for two tasks, namely summarization (sec-tion 4) and question generation (section 5). ## conclusion we proposed composition sampling, a simple yet effective decoding method for faithful and diverse conditional generation. our method is straightforward to implement and does not require any external system to augment the input during inference. our experiments demonstrate that it is currently the best available decoding strategy for generating diverse and meaningful output. we also introduced self-entailment and self-bertscore, to automatically compute semantic diversity in summaries, and, edna, for jointly measuring faithfulness and diversity.