CCS um: A Large-Scale and High-Quality Dataset for Abstractive News Summarization (2024)

Abstract: Training a supervised news summarization model requires large amounts of high-quality training data consisting of news articles paired with reference summaries. However, obtaining such data is costly, and existing datasets contain considerable amount of noise. We present a new large-scale and high-quality dataset for supervised abstractive news summarization containing 1.3 million training samples, which we call CCSum. In creating this dataset, we take advantage of the journalistic inverted-pyramid style in news writing: In some articles, the first sentence can be considered a summary of the reported story. Accordingly, among 35 million CommonCrawl News articles, we identify pairs of articles about the same news story and use one article’s first sentence as the summary for the other article. To ensure high quality, we apply strict filters whose parameters we optimize using Bayesian optimization. We show that the resulting dataset is more factual and informative than established summarization datasets; less than 1% of the summaries have major factual inconsistencies with the corresponding news articles, compared to 5.5% to 15.4% in existing datasets, according to our human evaluation. Summarization models trained on our dataset are more favored compared to those trained on CNN/Daily Mail. The proposed dataset can open new opportunities for future research in abstractive summarization.

## introduction news summarization is the task of automatically converting news articles into concise summaries that correctly convey the main information of the articles. particularly, in abstractive news summarization, summaries are generated as coherent reformulations of the articles' main points, rather than verbatim extraction of the articles' phrases and sentences. abstractive news summarization has made figure 1 : news summaries in our large-scale dataset ccsum (variants: ccsum-ext and ccsum-abs) have higher factual consistency than those of existing datasets, while written in an abstractive style comparable to cnn/dm, multi-news and newsroom. bubble size corresponds to dataset size. (details in section 4.) significant progress since the advent of transformers @xcite : summarization models have grown larger @xcite , objective functions have evolved @xcite and strategies for more factual @xcite or coherent @xcite summaries have been presented. however, the datasets on which abstractive news summarizers are trained have remained constant for several years. cnn/daily mail (cnn/dm; @xcite ) and xsum @xcite are used most frequently, multi-news @xcite and newsroom @xcite have found adoption as well. while these datasets differ in size (figure 1 ), they contain considerable amount of noise @xcite . in human evaluations, summaries generated in zero-shot fashion by large language models have been preferred over sum-a political storm is brewing ahead of prince harry's and meghan markle's may 19 wedding over whether to crack down on homeless people and beggars in the well-to-do english town of windsor (…) borough council leader simon dudley kicked off the controversy by tweeting over the christmas holidays about the need to clean up windsor's streets (…) homeless charities reacted angrily thursday to his suggestion that homelessness should be treated as a police matter so windsor can make a positive impression on visitors during the royal nuptials (…). murphy james of the windsor homeless project called dudley's views offensive. "it's absolutely abhorrent that anybody has got these views in this day and age (…) news event: royal wedding: windsor homeless removal controversy maries generated by supervised models trained on the above datasets @xcite . to mitigate these shortcomings, we introduce a new dataset for abstractive news summarization that is both large-scale and of high quality. it contains 1.3 million pairs of articles and summaries. less than 1% of the summaries have major factual inconsistencies with the corresponding news articles, compared to 5.5% to 15.4% in existing datasets, according to our human evaluation in our method is based on the insight that news articles often follow the journalistic principle of the inverted pyramid @xcite : articles start with the most important information, followed by important details and, finally, background information and other facts. the most fundamental and important information is often contained in the first sentence-the lead sentence (see figure 2 ). accordingly, we pair the lead sentence from one article as summary for another article describing the same news story. in particular, we obtain 35 million news articles from commoncrawl news and cluster them such that each cluster contains multiple articles that describes the same news story. in each cluster, we consider all pairs of articles; in each pair, we consider one article's lead sentence as the summary of the other article (figure 3 ). this results in over 100 million candidate pairs, each one consisting of a full article and an associated lead sentence from a related article as its potential sum-mary. only a small percentage of these candidate pairs are of high quality. therefore, we construct a multi-aspect filter involving automatic metrics of factual consistency, information coverage, abstractiveness, and other dimensions. we optimize this filter using bayesian optimization and apply it to remove 99% of the candidate pairs, resulting in the final dataset of articles and lead sentence summaries, which we split into a train, dev, and test portion. we call this dataset ccsum because it is derived from the commoncrawl corpus. in summary, our contributions are: our dataset can offer new opportunities for text generation research; this could include exploring the effects of model and data scaling, improving factual consistency in generated summaries, and developing better instruction-following models. article c cluster 1 article clusters cluster 3 cluster 2 sentencen.1 sentencen.2 sentencen.3 … article n sentenceb.1 sentenceb.2 sentenceb.3 … article b sentencea.1 sentencea.2 sentencea.3 … article a commoncrawl-news filters factual consistency coverage abstractiveness before filtering heuristics entity precision quotation match bertscore-precision bertscore-recall title-title similarity summary-title similarity simhash mint bayesian optimization c d f d …… figure 3 : process to construct the ccsum dataset. news articles from commoncrawl-news are clustered into news events, from which we generate candidate article-summary pairs, and apply extensive filtering to obtain the final dataset. for instance, out of 312,544 articles published between september 21 and september 23, 2020, 71,731 unique news events were identified, creating 1.8 million candidate pairs. after extensive filtering, 99% of these pairs were eliminated ensuring only high-quality pairs were included in the final dataset. and used as summary for the remaining part of the article. its training set has 204k samples. the introductory sentences were not intended to be abstractive summaries and might contain information not present in the article. newsroom @xcite was constructed from web-scale crawling and uses the "description" field in each article's html metadata as the summary. the summaries are split into extractive, abstractive, and mixed bins, according to the density metric. the summaries tend to be noisy due to the inconsistent quality of html metadata. it contains 1.3 million training samples. multi-news @xcite pairs human-written news aggregation summaries with the articles originally linked from those summaries. this makes it a multi-document summarization dataset-in contrast to the other datasets discussed. it contains 44k training samples. gigaword @xcite pairs articles with their headlines, e.g., "five dead as powerful quake hits southern iran", rather than fully formulated summaries. ## dataset overview and evaluation train/validation/test splits. the dataset is divided into training, validation, and testing segments, categorized by the publication date of each article ( abstractiveness. the dataset is divided equally into "extractive" and "abstractive" bins, which we denote as ccsum-ext and ccsum-abs, respectively, based on the median mint score (0.4665). ## conclusion we present a large-scale and high-quality dataset, ccsum, for supervised abstractive news summarization. it contains 1.3 million pairs of articles and reference summaries derived from 35 million news articles from commoncrawl news. in creating this dataset, we cluster commoncrawl news articles into news events from which we generate candidate article-summary pairs and apply strict filtering and a bayesian optimization method that eliminates 99% of the candidate summaries. the human evaluation shows the proposed dataset has higher quality-in terms of factual consistency, in-