Don’t Say What You Don’t Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search (2022)

Abstract: Abstractive summarization systems today produce fluent and relevant output, but often “hallucinate” statements not supported by the source text. We analyze the connection between hallucinations and training data, and find evidence that models hallucinate because they train on target summaries that are unsupported by the source. Based on our findings, we present PINOCCHIO, a new decoding method that improves the consistency of a transformer-based abstractive summarizer by constraining beam search to avoid hallucinations. Given the model states and outputs at a given step, PINOCCHIO detects likely model hallucinations based on various measures of attribution to the source text. PINOCCHIO backtracks to find more consistent output, and can opt to produce no summary at all when no consistent generation can be found. In experiments, we find that PINOCCHIO improves the consistency of generation by an average of 67% on two abstractive summarization datasets, without hurting recall.

## introduction abstractive text generation is an important task with the promise of compressing lengthy source material into concise summaries, satisfying application or user needs. pretrained abstractive summarizers (e.g. bart @xcite ) have recently achieved new state-of-the-art (sota) across multiple datasets @xcite . however, these systems remain unusable in most real world scenarios, because they frequently hallucinate information that is inconsistent with the input @xcite . many researchers have proposed methods to assess and improve the consistency 1 of summarization systems. two popular approaches are 1) incorporating extracted knowledge @xcite method text source ...the psni said the tablets were "as yet unidentified" but warned of the "potential dangers" they posed... bart a 17-year-old boy has been charged after a teenager was taken ill after taking what police have described as "potentially lethal" ecstasy tablets. pinocchio a 17-year-old teenager has been charged with drugs offences after a teenager was treated in hospital after taking what police described as an "unidentified" drug. we propose a different approach for generating more consistent summaries. it is based on the observation that today's abstractive summarizers are often trained on target summaries that contain statements unsupported by the source text @xcite . this disconnect arises because the training datasets are acquired from noisy "silver" sources in order to scale, e.g. treating a news headline as a summary of its article or an encyclopedia entry as a summary of a portion of its references. we conjecture that a model optimized for likelihood and trained on target summaries containing unsupported statements will have a strong tendency to hallucinate information rather than say something less "likely," but supported ( §3). further, common automatic evaluation metrics like rouge reward lexical similarity significantly more than consistency, preferring hallucinated lexically similar summaries to completely consistent lexically different ones. our method, called pinocchio, is a novel decoding algorithm that constrains beam search to only consider predicted tokens that are likely to be supported by the source text. pinocchio estimates which tokens are likely supported using simple but effective heuristics based on the model's confidence and attention distribution, and word frequency. when pinocchio reaches a state where no supported token can be generated, it backtracks the search. it can also opt-out from generating a summary at all, rather than produce one expected to be hallucinated. we show how pinocchio significantly improves consistency on two abstractive summarization datasets with only a small decrease in fluency, measured using careful human evaluations. to test pinocchio on diverse domains, we also develop a new abstractive summarization dataset called scientific concept description (scd). inspired by the wikisum @xcite dataset, scd uses wikipedia descriptions as the target summaries and the referenced papers as the source documents, detailed in ( §5). scd is motivated by the goal of automatically generating a high-quality encyclopedia for the long tail of scientific concepts described in papers, and presents a challenging workload for abstractive summarization. it comes with a total of 60k samples of scientific concepts and 118k corresponding paper identifiers, with full text for 8k of the papers. we make the following contributions: the scd dataset, along with our code, trained models, and human evaluations, is available at https://github.com/allenai/ pinocchio . ## related work pretrained language models have recently taken the top spots on summarization leaderboards @xcite . this includes models like bart @xcite , pegasus @xcite , and unilm @xcite . in a recent large scale evaluation of summarization models, fabbri et al. (2020) found bart and pegasus to be the top performing models. we choose to focus on bart in this work. it is widely known that sota summarization models tend to hallucinate facts @xcite , and the most closely related works to ours are those on factual summarization. however, we avoid the term "factuality" and instead use "consistency" to denote that the generated summary is supported by the input text. as noted in @xcite , a summary could be hallucinated but still be factually correct. in this work, we aim to improve consistency and reduce hallucinations, which indirectly improves factuality, without directly optimizing for it. prior works attempt to improve consistency by correcting already-generated summaries @xcite , using a knowledge graph @xcite , filtering training data @xcite , constraining generation with keywords @xcite , using nli models @xcite , among others. some have focused on the data-to-text setting, which presupposes structured input @xcite . some works control the extractiveness of generations @xcite . there have also been multiple works on automatically measuring consistency @xcite @xcite . @xcite noted that hallucinations come from a source-target discrepancy, where many training targets are not fully supported by their source text, and suggested to address it by removing samples with unsupported summaries. we extend their empirical findings with similar measurements on three additional datasets, conjecture that hallucination is unavoidable in such settings, and provide evidence for the conjecture in terms of the lexical statistics of output summaries. we use beam search for decoding, which has become standard practice for neural seq2seq models @xcite . our approach can be viewed as a version of constrained decoding @xcite but with dynamically identified constraints and the ability to backtrack. our constraints come from various model internal signals that indicate attribution to the source text. one such signal is entropy, where @xcite found that low next token entropy indicates the model is copying. unlike previous work, we do not attempt to imbue models with a new level of textual understanding, but rather show that we can improve consistency of generated text using simple signals based on model internals. ## why do models generate inconsistent summaries? in this section, we analyze why models generate inconsistent summaries. here, we use the definition of consistent from fabbri et al. (2020), i.e., the factual alignment between the summary and the summarized source. we hypothesize that there are two factors that contribute to inconsistency: 1) the maximum likelihood training and generation strategy used in summarization models, and 2) imperfect training datasets that contain many instances where the target is difficult or impossible to deduce from the source. specifically, we conjecture that in the presence of these two factors, models are guaranteed to hallucinate because they either 1) default to a background distribution of the most common relevant terms during generation or 2) learn spurious correlations between the source and target texts. in either case, the model generates text that is often inconsistent with the inputs. we present our analysis in terms of a motivating example below, and provide empirical support for it in sec. 7. the analysis inspires the design of the pinocchio method in sec. 4. ## pinocchio: constraining beam search to improve consistency inspired by the previous analysis, we introduce pinocchio; a modification to standard beam search for supported-decoding (alg. 1). beam search for text generation typically works by adding to a small set of candidate generations one token at a time, keeping the top b generations according to model-predicted likelihood after each prediction timestep. @xmath0, those b candidates are rescored with a length penalty @xcite , and the best one is chosen as the final output. pinocchio differs from regular beam search only in its use of the set r, which holds a set of disallowed generation paths; if r is always empty, alg. 1 simplifies to standard beam search. pinocchio modifies the model predicted token scores to avoid inconsistent predictions. in particular, pinocchio applies a function f c (model state, candidate next generation) to the predicted likelihood of the top predicted tokens. if algorithm 1: supported-decoding input: beam size b, generative model m , consistency function fc, vocab v , maximumly allowed backtrack count n priority queue p @xmath1; completed generations @xmath2; rejected paths @xmath3; backtrack count η = 0; while |cg| < b do c := {x + v : @xmath4, @xmath5; t := top 2b items of c scored by m ; r := r ∪ {@xmath6: fc(m, d) = 0}; if t -@xmath7; end r := r ∪ {x[: -1] : @xmath8; p q := {x[: -1] : @xmath9; η := η + 1; continue; end t := t -r; p q := top b elements of t according to m not ending in "<@xmath10; cg := cg ∪ {@xmath11: d scores higher than min in p q and ends in "<@xmath12; end return top-ranked element of cg; within this framework, we present an instantiation of f c based on a set of carefully curated heuristics, determining if a token is allowed to be predicted or not. the function f c consists of a series of binary checks, which take into account both model internals as well as language features. if any of the checks succeeds, f c is 1 and the model continues generating, but if all of the checks fail f @xmath13. first, we consider the model confidence for the current prediction-based on the intuition that a low entropy of the token prediction probability distribution corresponds to more certain, and potentially more correct, predictions. second, we keep track of the source text with high attention scores during the generation process: when the attended texts are semantically or lexically similar to the token to be generated, that suggests that the token may be supported by the source. third, pinocchio also allows tokens that are especially common (such as stopwords), as we expect these are less likely to be hallucinations. we develop a total of 8 different binary functions within the three categories above (details in §d.1). the heuristics do not require additional training steps, and all the associated thresholds or hyperparameters were determined by manual inspection on a small number of samples (e.g., @xmath14. different from prior work @xcite , this non-machine learning approach is based on scrutiny of the model generation process. it is easy to execute and more explainable compared to black-box models. ## tasks and datasets we evaluate pinocchio on two distinct summarization tasks: news summarization (xsum and cnn / daily mail) and scientific concept description (the newly proposed scd dataset). ## experiments 6.1 metrics we rely on human evaluation, as current automatic metrics are unreliable for evaluating factuality (see §6.4). we are not targeting rouge metrics @xcite , but present them for completeness. 6 for human evaluation, we use standard dimensions of consistency (does the source entail the 6 https://github.com/yale-lily/summeval ## conclusion in this work, we present pinocchio, a simple, no-additional-machine-learning required, method for reducing hallucination in generative encoderdecoder models. pinocchio provides a substantial lift in consistency, with only a small decrease in fluency. we analyze why existing summarizers hallucinate, showing that silver abstractive summarization datasets can contain unsupported target summaries, and presenting evidence for our conjecture that models that maximize likelihood trained on such data will tend to hallucinate. we also show that existing factuality metrics are insufficient, and further explore how patterns in the training dataset can produce misleading results on the test test. we also introduce the task of scientific concept description and release a wikipedia-based dataset for it. we would like to clearly acknowledge the limitations of our approach. pinocchio does not add new learned behavior to the model, using simple heuristics and single-step backtracking to steer the model towards more consistent output. the heuristics have settings that require some adaptation for each data set, and while limited manual tuning was sufficient for the two data sets in our experiments, further experiments with additional data sets are necessary. further, preliminary experiments suggest that the settings that were effective for bart do not simply work out of the box for another summarizer, pegasus. we also acknowledge that while pinocchio offers improvements in consistency, the results are still far from perfect, and thus the system is not suitable for certain applications. we hope the approach and insights in this paper help spur further development of models that generate consistent text.