M i RAN ews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization (2021)

Abstract: One of the most challenging aspects of current single-document news summarization is that the summary often contains ‘extrinsic hallucinations’, i.e., facts that are not present in the source document, which are often derived via world knowledge. This causes summarisation systems to act more like open-ended language models tending to hallucinate facts that are erroneous. In this paper, we mitigate this problem with the help of multiple supplementary resource documents assisting the task. We present a new dataset MiraNews and benchmark existing summarisation models. In contrast to multi-document summarization, which addresses multiple events from several source documents, we still aim at generating a summary for a single document. We show via data analysis that it’s not only the models which are to blame: more than 27% of facts mentioned in the gold summaries of MiraNews are better grounded on assisting documents than in the main source articles. An error analysis of generated summaries from pretrained models fine-tuned on MIRANEWS reveals that this has an even bigger effects on models: assisted summarisation reduces 55% of hallucinations when compared to single-document summarisation models trained on the main article only.

## introduction the vast majority of current research on abstractive summarization is aimed at single-document news summarization due to the widespread availability of data, e.g. (ny times; @xcite , @xcite , newsroom; @xcite , xsum; @xcite , mlsum; @xcite . the datasets are curated by pairing a single document with human authored highlights/description as the summary. this task is typically approached using figure 1 : an example where the summary (top section) contains information that is not explicitly included in its main document (middle section), but is covered in the related assisting document (bottom section). we highlight the information in the summary that is aligned to its corresponding main and assisting documents with yellow and pink colors, respectively. while these sota summarization models reach a high level of fluency and coherence, they are also highly prone to hallucinating content that is not grounded by the input document. @xcite classified hallucinations into intrinsic that mistakenly manipulate information from the source document resulting in counterfactual output, and extrinsic that introduce information not grounded in the document (see figure 1 ). extrinsic hallucinations are further broken down into 'factual', i.e., holding true in real life, and 'counterfactual'. similar to @xcite , we find not only the models are to blame, but also the datasets: human-written summaries contain up to 36% external facts which are not faithful, i.e., covered by the single input document. in other words: the summaries also contain 'extrinsic hallucinations'. moreover, facts which are present are often re-phrased or shortened in the summary in ways which requires world knowledge. consider the example in figure 1 , where the surname "rivers" used throughout the document (middle section), is elaborated as the full name "joan rivers" in the summary (top section), i.e. adding information. meanwhile, "celebrities lined up outside the fifth avenue synagogue" in the document is specified as "say goodbye at joan rivers funeral" in the summary, which requires world knowledge. moreover, the fact about an "a star-studded funeral" is not mentioned explicitly in the document. any summarization model that is agnostic to such data divergence issues between the source and target texts @xcite will function more as an open-ended language model and will be prone to extrinsic hallucinations. in this work, we tackle the problem of extrinsic hallucinations by introducing a new task, multi-resource-assisted news summarization and a novel dataset (miranews). following maynez et al. (2020), we regard the incorporation of background knowledge within a generated summary as the desired property. however, instead of sourcing this knowledge via pretraining on large datasets, 2 2 although they report berts2s @xcite to output more factual hallucinations in the summary than their non-pre-trained counterparts on xsum @xcite , we base our work on the assumption that articles from alternative news resources covering the same news event can complement the background knowledge in an easier to learn, more direct, and explainable way. consider the example in figure 1 , where the assisting document (bottom section) from another news resource recounts some facts in the summary (highlighted in pink) in a more explicit way. note that, as shown in figure 2 (left), our task is different from both single-document summarization (sds, middle) and multi-document summarization (mds, right): sds aims at generating a summary for a single main document, while we aim to generate a target summary s for a single document d with supporting facts from multiple assisting documents a. in this paper: news cluster and content extraction. we consider each article on newser.com, together with the pages cited therein, as a cluster about one news event. we extract the document and the corresponding summary from each webpage automatically, following the method introduced in newsroom @xcite . 3 specifically, the documents are constructed from the html main text body excluding html markups, inline advertising, images/videos, and captions, while the target summary s is extracted from the document's metadata fields, e.g. og:description, twitter:description, description, which are often written by editors and journalists to appear on social media services or as search engine webpage descriptions. hence, for each cluster c, we collect paired documents and summaries @xmath0, s 1 ) , (d 2 , s 2 ) • • • (d m , s m )}, where m is the number of webpages in the cluster. ## data analysis miranews contains 150k examples in total, with an average of 1.7 assisting documents per instance foot_1 ## experiment results similarity to reference. the results of reference-based automatic metrics are shown in on most metrics, the concatenation variants (-c) of the models perform worse than the pipeline approaches (-p) and sds-trained systems (-s). on the other hand, both -p outperform the -s systems in most cases. the gold systems (-g) achieve the best performance with a large margin. the performance of bart-g is even comparable with the upper bound of the extractive models (eo generated from miranews(s-d&a)). hence, we conclude that (1) introducing assisting documents benefits the abstractive summary generation; (2) better content selection improves the performance of the abstractive models; (3) the margin between the gold upper baseline and the rest is notable, which suggests that there is room for improvement for content selection. extractiveness level. the results are shown in the left section of support from assisting documents. the middle section of extrinsic hallucination. the results in the right section of within each of the two models, summaries generated by each variant preserves a roughly similar level of extractiveness. in both models, concatenation and pipeline systems achieve a lower ## hallucination analysis we manually identify 4 types of hallucinations from a small random sample (30 main/assisting documents and summaries) from the development set of miranews, as summarised in in both types with up to 55% on extrinsic hallucinations when using the assisting documents for training efficiently (bart-g). at the same time, we observe 'extrinsic hallucinations' that are correctly grounded only on the assisting documents (11), and rarely guessed based on pre-training (only 1 fact based on world knowledge). interestingly, we also observed a number of facts (10) in the gold summary that are grounded exclusively on the assisting documents, further supporting the value of our approach. an example of outputs from variants of bart is shown in figure 4 . ## related work single document summarization aims to compress a single textual document while keeping salient information. sds includes two directions: extractive summarization @xcite which aims at extracting salient sentences from the input document, and abstractive summarization @xcite @xcite @xcite @xcite which generates a novel short representation of the input. multi-document summarization aims to compress multiple textual documents to a shorter summary @xcite . approaches mainly focus on increasing the capacity of the encoder to process longer inputs @xcite @xcite @xcite , leveraging knowledge graphs @xcite @xcite , and including content selection steps @xcite @xcite @xcite . hallucinations in summarization are a well established problem @xcite @xcite . previous research aimed to reduce hallucination by adapting model architec-tures, training and decoding, e.g. cao et al. (2018 @xcite . however, we are the first research aiming to reduce the hallucinations by adapting the dataset. ## conclusions and future work in this work, we found that up to 36% facts in the ground truth summaries in traditional sds datasets are not faithful to the source article. in other words, the ground truth summaries also contain 'extrinsic hallucinations'. summarization models trained on such data will be prone to extrinsic hallucinations. to tackle this problem, we introduce a new task, multi-resource-assisted news summarization, which produces a summary based on the events present in the main article while reaching to a set of assisting documents for complementary background. we release the miranewsdataset, which includes multiple assisting news articles from different news resources for each documentsummary pair. our newly introduced evaluation metrics confirm that introducing assisting documents offers better grounding to more than 27% facts in the reference summaries. we report benchmark results on miranews. we also show that the model trained with assisting documents produces 55% less counterfactual hallucinations than a model trained only with main documents. in future work, we plan to explore a retrievalbased approaches @xcite that are able to search and filter relevant assisting documents for a given news event, without the help of human-edited resources such as newser.com. in the paper, we demonstrated that the assisting documents contain useful facts to support the summarization of the main news event. thus, efficient content selection that eliminates noise and grounds in the relevant facts appearing in either main or assisting documents will also be explored in our future work.