Should We Trust This Summary? B ayesian Abstractive Summarization to The Rescue (2022)

Abstract: We explore the notion of uncertainty in the context of modern abstractive summarization models, using the tools of Bayesian Deep Learning. Our approach approximates Bayesian inference by first extending state-of-the-art summarization models with Monte Carlo dropout and then using them to perform multiple stochastic forward passes. Based on Bayesian inference we are able to effectively quantify uncertainty at prediction time. Having a reliable uncertainty measure, we can improve the experience of the end user by filtering out generated summaries of high uncertainty. Furthermore, uncertainty estimation could be used as a criterion for selecting samples for annotation, and can be paired nicely with active learning and human-in-the-loop approaches. Finally, Bayesian inference enables us to find a Bayesian summary which performs better than a deterministic one and is more robust to uncertainty. In practice, we show that our Variational Bayesian equivalents of BART and PEGASUS can outperform their deterministic counterparts on multiple benchmark datasets.

## introduction state-of-the-art text summarization methods have achieved remarkable performance in various benchmarks @xcite @xcite . the majority of these methods use very large transformer models pre-trained on language generation tasks. although such methods can generate high quality summaries for texts similar to their training set, they suffer from a couple of issues when the inputs lie far from the training data distribution. they are prone to generating particularly bad outputs @xcite and are usually fairly confident about them @xcite . these shortcomings are bound to cause problems once a summarization model is deployed to solve a practical problem. since the output of automatic summarization models is usually expected to be consumed by humans, it is very important to know when such an output is of good enough quality to be served to users. in most cases, it is very much preferable to not serve an output at all, instead of serving a bad output. this will in turn increase users' trust to automated summarization systems. model uncertainty is one way of detecting when a model's output is likely to be poor on the grounds of predicting far away from it's training distribution. recent summarization methods have focused heavily on improving the overall performance, but model uncertainty has been explored very little @xcite . in addition to improving user experience, the development of uncertainty measures for summarization can pave the way for active learning approaches @xcite @xcite . the value of active learning stems from the fact that obtaining labeled samples for training is hard, but it is relatively easy to obtain large amounts of unlabeled samples. summarization is no different in this perspective, since creating good quality target summaries for training can be very costly. this work explores uncertainty estimation for state-of-the-art text summarization models, from a bayesian perspective. we extend the bart @xcite and pegasus @xcite summarization models with monte carlo dropout @xcite , in order to create corresponding variational bayesian pega-sus (varpegasus) and bart (varbart) models. sampling multiple summaries from those models allows us to approximate bayesian inference in a practical way, which in turn enables us to estimate summarization uncertainty. to the best of our knowledge this is the first attempt to apply bayesian summary generation with large transformer models. based on bayesian approximation, we adapt the monte carlo bleu variance metric @xcite to the summarization task, and investigate its efficacy as a measure of summarization uncertainty. our findings suggest that this uncertainty metric correlates well with the quality of the generated summaries and can be effective at identifying cases of questionable quality. finally, we take the summarization uncertainty study one step further, and select the summary with the lowest disagreement out of multiple summaries sampled from our variational models. experiments across multiple benchmark datasets show that this method consistently improves summarization performance (see the rest of this paper is structured as follows. section 2 discusses related work on bayesian deep learning and uncertainty estimation methods. section 3 presents our approach. section 4 describes our experimental setup, while section 5 presents and discusses the results. finally, section 6 concludes our work and considers its broader impact. ## related work uncertainty estimation in deep learning is a topic that has been studied extensively. bayesian deep learning includes a family of methods that attempt to capture the notion of uncertainty in deep neural networks. such methods have gained increased popularity in the deep learning literature and there exist multiple applications in subfields such as computer vision @xcite @xcite and natural language processing (nlp) @xcite @xcite . despite their obvious advantage of modeling uncertainty, the main problem with bayesian deep learning methods is the computational cost of full bayesian inference. to tackle this problem, @xcite propose using standard dropout @xcite as a practical approximation of bayesian inference in deep neural networks and call this method monte carlo dropout. @xcite use a convolutional neural network with monte carlo dropout in order to obtain an uncertainty estimate for active learning in the task of image classification. @xcite sample many networks with monte carlo simulation and propose an objective function that takes into account the disagreement and confidence of the predictions coming from these networks. similar methods have also been applied to nlp. in machine translation, xiao et al. (2020) extend the transformer architecture with mc dropout to get a variational transformer, and use it to sample multiple translations from the approximate posterior distribution. they also introduce bleuvar, an uncertainty metric based on the bleu score @xcite between pairs of the generated translations. lyu et al. (2020) extend the work of @xcite to question answering and propose an active learning approach based on a modified bleuvar version. similarly, @xcite use a conditional random field to obtain uncertainty estimates for active learning and apply their method to named entity recognition. although summarization is a prominent nlp task, summarization uncertainty has not been widely studied. @xcite is the only work that focuses on uncertainty for summarization, but their work does not make use of bayesian methods. they define a generated summary's uncertainty based on the entropy of each token generated by the model during the decoding phase. their study includes experiments on cnn/dm and xsum using the pegasus and bart summarization models. their main focus is on understanding different properties of uncertainty during the decoding phase, and their work is not directly comparable to ours. ## methods we first introduce bayesian inference, in the context of deep neural networks and show how it can be used to measure uncertainty. subsequently, we show how bayesian inference can be applied to summarization in order to estimate the uncertainty of a summary generated for a given input. finally, we show how bayesian inference can be employed for producing better summaries. ## experimental setup we first present the three datasets that are involved in our experiments, their main statistics and the reasons for including them in our empirical study. then we present the two summarization models that we employed, along with their parameters and details on stochastic summary generation. ## results our main experiment evaluates bleuvarn's effectiveness in quantifying uncertainty for summarization models. a second experiment investigates the potential of the bayesian summarization method proposed in section 3.3 as a way of improving summarization performance at test time. ## conclusions and future work this work explored bayesian methods in the context of text summarization. we extended state-ofthe-art summarization models with mc dropout to approximate bayesian inference, and demonstrated how bleuvarn can be used to quantify model uncertainty. this allows us to effectively identify high uncertainty summaries at prediction time, which can be a significant advantage. furthermore, we show that ensembling multiple stochastic summaries generated by variational bayesian models can lead to improved performance compared to similar deterministic models. this finding is verified by experiments for two different models and across 3 benchmark datasets. it should be noted here that the proposed methods are directly applicable to other abstractive summarization datasets as well. we acknowledge that some of the more interesting summarization problems involve much longer summaries, for example scientific abstracts. in this work we focused on datasets of short summaries in order to be more resource efficient and conduct more experiments. there's a lot of interesting work that focuses on the summarization of longer documents @xcite ) that could potentially be applied in combination with the methods we propose here. our work can have a broader impact in several ways. to the research community, being the first work to study bayesian uncertainty for abstractive summarization and paving the way for other similar methods. to the industry, because it improves automatic summarization systems and can be paired nicely with active learning and humanin-the-loop approaches. finally, to the end users, improving their experience and building up confidence towards automatic summarization systems.