Faithfulness-Aware Decoding Strategies for Abstractive Summarization (2023)

Abstract: Despite significant progress in understanding and improving faithfulness in abstractive summarization, the question of how decoding strategies affect faithfulness is less studied. We present a systematic study of the effect of generation techniques such as beam search and nucleus sampling on faithfulness in abstractive summarization. We find a consistent trend where beam search with large beam sizes produces the most faithful summaries while nucleus sampling generates the least faithful ones. We propose two faithfulness-aware generation methods to further improve faithfulness over current generation techniques: (1) ranking candidates generated by beam search using automatic faithfulness metrics and (2) incorporating lookahead heuristics that produce a faithfulness score on the future summary. We show that both generation methods significantly improve faithfulness across two datasets as evaluated by four automatic faithfulness metrics and human evaluation. To reduce computational cost, we demonstrate a simple distillation approach that allows the model to generate faithful summaries with just greedy decoding.

## introduction recent developments in large pre-trained language models have achieved remarkable performance on abstractive summarization @xcite . however, such models often suffer from the problem of hallucinations, where the generated summary contains facts or entities not present in the original document. prior research has analyzed and defined potential error types and typology @xcite @xcite , and developed methods to improve faithfulness, including post-processing models @xcite @xcite and faithfulness-aware training @xcite @xcite @xcite . one aspect that is less understood on faithfulness of abstractive summarization is the effect of decoding strategies, which determine how the model generates the output strings. our primary objective is to understand whether different types of exploration of the search space, such as traversing and maintaining multiple possible output hypotheses with beam search or encouraging diversity with nucleus sampling @xcite , have an impact on faithfulness. to this end, we first conduct a thorough analysis comparing the faithfulness of popular decoding strategies, including greedy decoding, beam search, and nucleus sampling for two popular summarization datasets xsum @xcite and cnn/dm @xcite . evaluating the generated summaries using four faithfulness metrics, including bertscore @xcite , factcc @xcite , dae @xcite , and questeval @xcite , and human evaluation, we find a consistent trend that beam search provides the most faithful summaries with its large exploration of the search space, and the randomness introduced by sampling hurts faithfulness. to further improve faithfulness beyond the common decoding strategies, we propose two faithfulness-aware decoding methods. first, similar to @xcite , we make use of the multiple candidates generated by beam search and propose a simple re-ranker, which selects the best summary according to a faithfulness metric. instead of using a specific metric, we rank and select the summaries with a composite metric, a weighted combination of popular faithfulness metrics. next, inspired by @xcite , we propose a faithful-a study led by goldsmiths, university of london, found british army infantry troops spent less than 47% of their time on the western front … ## faithfulness behavior of popular decoding strategies we first describe our experiment investigating the effect of popular decoding strategies on faithfulness. we wish to primarily investigate whether better exploration of the search space, such as the candidate expansion with beam search, can improve faithfulness, and how randomness introduced through sampling impacts faithfulness. these investigations in turn motivate our more advanced, faithfulness-aware decoding strategies in section 3. ## faithfulness-aware decoding strategies we hypothesize (and later test and confirm whether it is true in section 6.1 and appendix e) that current decoding methods, such as beam search which explores a large space, may not explore the paths that focus on faithfulness directly and effectively. hence, we propose two faithfulness-aware methods that can be applied on top of the base decoding strategies to modify how the space is explored from two different perspectives: (1) ranking makes use of the large exploration of beam search and picks the explored path that is most faithful; (2) lookahead directly guides the search process by adding faithfulness heuristics when selecting the next token starting from the initial decoding process. ## efficient decoding via distillation one drawback of the proposed decoding methods is the heavy computational cost during decoding. we thus explore using distillation to transfer the knowledge of faithfulness-aware decoding to a student model that can generate summaries of similar faithfulness with just greedy decoding. we note here that our distillation aims at improving the decoding time rather than downsizing the model. similar to @xcite , we assume that we have a teacher model and a student model. in our setting, the teacher model does not necessarily need to be a different model, but it needs to decode with more faithfulness-aware methods. typical distillation methods use the teacher's probability distribution @xcite as the target for the student model to imitate. in our case, however, that distribution is the same for all methods -the difference lies in how the probability is used to generate the next tokens. thus, we propose a new decoding distillation loss. we use the teacher model to generate summaries y gen as additional reference summaries, and interpolate between the cross-entropy loss using the original reference summaries and the crossentropy loss where we consider y gen as reference summaries. formally, the training loss is: iterative distillation. while we use the student model with just greedy decoding to improve decoding speed, the student model can also benefit from using our proposed faithfulness-aware decoding methods. thus, the student models can also serve as a new teacher model to distill more faithfulness knowledge to a new student model. the distillation process thus becomes iterative, illustrated in figure 2. we use the trained student model as a new teacher model, where we decode with our proposed faithfulness methods to create additional reference summaries y gen for the next iteration. ## related work many of the related works of our proposed decoding methods have been discussed in section 3; here we cover other related areas. ## conclusion in this paper, we show a thorough analysis of the effect of decoding strategies on faithfulness for abstractive summarization. we present an analysis of popular decoding strategies, as well as our two newly proposed faithfulness-aware decoding strategies, ranking and lookahead, that can further improve faithfulness upon the base decoding methods. finally, we show a simple (and optionally iterative) distillation trick where the training of a student model incorporates the summaries generated with more faithfulness-aware methods, and the student model generates summaries of similar faithfulness with minimal decoding time. future experiments could extend similar analysis of faithfulness and factuality beyond summarization and develop a combination of heuristics that also encompasses other aspects and styles.