The State and Fate of Summarization Datasets: A Survey (2025)

Abstract: Automatic summarization has consistently attracted attention due to its versatility and wide application in various downstream tasks. Despite its popularity, we find that annotation efforts have largely been disjointed, and have lacked common terminology. Consequently, it is challenging to discover existing resources or identify coherent research directions. To address this, we survey a large body of work spanning 133 datasets in over 100 languages, creating a novel ontology covering sample properties, collection methods and distribution. With this ontology we make key observations, including the lack of accessible high-quality datasets for low-resource languages, and the field’s overreliance on the news domain and on automatically collected distant supervision. Finally, we make available a web interface that allows users to interact and explore our ontology and dataset collection, as well as a template for a summarization data card, which can be used to streamline future research into a more coherent body of work.

## introduction summarization is the task of shortening a text while preserving the most important information it contains. this task has been explored for over 60 years @xcite yet it remains a longstanding challenge, as evidenced by the steadily increasing number of summarization datasets released every year (see figure 1 ). despite the many research efforts in creating summarization datasets, we find they lack standardization and common terminology. this shortcoming makes it difficult to discover existing resources and bottlenecks, and to identify general directions for future research. in this work, we conduct the most comprehensive survey of summarization datasets to date, covering a wide collection of 133 datasets across 104 languages. in section 3, we develop a comprehensive ontology geared towards modern summarization datasets, positioning each dataset along seven different scales, including information about the samples (language, language modality, domain and summarization shape) as well as collection methods (annotation efforts and sources of supervision), and the manner in which the dataset is distributed. in section 4 we show that our proposed ontology surfaces five major trends and gaps in the field. first, we find that the lack of common terminology thus far often leads to inconsistency and to the omission of important details when presenting datasets, making comparison challenging. an important example of this inconsistency is the traditional abstractive-extractive distinction, which we argue has gradually shifted away from a binary trait to a more nuanced spectrum. second, we observe that the field is over-reliant on the news domain, which acts as a double-edged sword -being easy to obtain while prone to low quality. this is especially evident for low-resource languages, where the vast majority of datasets are based on news articles. third, we find that the datasets available for lowresource languages tend to be of subpar standard and especially challenging to discover. this is because many languages are currently supported only through multilingual datasets, which tend to compromise quality for diversity @xcite , while also making it harder to know which languages are annotated in which dataset. fourth, we find that while the majority of annotations are gathered automatically, they are typically not originally created to serve as summaries, relying instead on distant supervision. fifth, we observe that copyright concerns, especially in the news domain, have been changing data distribution, with many datasets moving from publishing the data to publishing scripts that reconstruct it from urls. furthermore, a few seminal summarization datasets that were published publicly have been either taken down or are considered legally questionable due to copyright violations. in section 5 we publish two resources to standardize and help organize the vast research in the field. first, we make our dataset collection publicly available through a web ui which allows users to search datasets matching different criteria according to our terminology, for example, retrieving all manually created datasets in a certain language. this ui can help researchers discover datasets, as well as identify new research directions. we welcome community contributions to keep this resource updated with new research. second, inspired by work on model cards @xcite , we propose a standard summarization data card which covers the different items in our ontology as well as additional important properties. this can benefit future dataset developers in making their work more accessible, easy to discover, and comparable against other datasets. we hope that our work helps chart the way for future work in the field, as well as enable researchers from various disciplines in exploring the wide range of existing summarization datasets. to conclude we make the following contributions: ## dataset collection our survey covers 133 datasets collected over nearly 20 years, spanning from 2005 to 2024. the full list of references can be found in appendix a. venues. most datasets are drawn from papers published in top-tier nlp, ml and ai venues and workshops. 3 we also include datasets published through shared tasks and in language-specific journals and workshops. filtering. we manually verify that each dataset is available for use, and exclude papers describing currently inaccessible resources. we include papers that explicitly focused on creating summarization datasets or that focused on summarization models and produced data as a by-product. collected data. for each dataset, we gather information on its properties, collection methods and distribution. this methodology allows categorizing the current resources available, analyzing emerging trends and identifying the gaps in the field. ## ontology of summarization datasets we propose a categorization of summarization datasets across seven orthogonal axes, and provide an ontology for each, depicted in figure 2 , and elaborated below. this ontology helps contextualize the extensive research in summarization and identifies missing types of datasets which can be addressed in follow up work. ## the state of summarization datasets we thoroughly examine 133 summarization datasets under the lens of our ontology. figure 2 presents the quantitative dataset distribution as well as categorization examples for several datasets. following, we discuss five major observations regarding the current state and challenges in summarization datasets. ## standardizing summarization datasets our survey surfaces a lack in terminology and reporting across the vast number of summarization datasets. this challenges researchers both in determining whether there exists a dataset specifically tailored to their needs (e.g., in terms of language, domain, or shape), and in identifying bottlenecks and directions for future research. here we outline two contributions we make towards making datasets more standard and accessible. interactive web interface. we make available a web ui populated with our ontology of 133 summarization datasets, along with our annotations of the seven axes that characterize each dataset (see figure 7 ). users of this interface can search for datasets by specifying the criteria they are looking for. we welcome community contributions to add new datasets, or past datasets not already included in it. we hope that this interface can allow users to easily find relevant datasets, without the need to manually sift through hundreds of papers. template for a summarization data card. in recent years, model and data cards have helped set agreed reporting standards, formalizing what details are reported when publishing a new resource @xcite . moving forward, we suggest that summarization research adopt a similar approach. in specifically, we focus on four key elements to report: (1) sample information: shape, languages, domain; (2) annotation information, indicating how were the summaries created: e.g., via scraping or manual effort; (3) data quality assessment: we propose to report the level of abstraction via ngram novelty, the compression rate @xcite , as well as human evaluation of the data, where applicable; and (4) availability: indicating copyright licensing and download information. ## related work several surveys have explored different aspects of automatic summarization @xcite @xcite ), yet they focus on models rather than datasets. to the best of our knowledge, none of the previous work has focused on the state of low-resource languages, nor looked at the dataset collection method and distribution. dernoncourt et al. (2018) ## conclusion we carried out an extensive survey of 133 summarization datasets in 104 languages, introducing a new ontology covering sample information, collection methods and data distribution. we offer a detailed report on the current state of resources in the field, surfacing challenges due to the lack of terminology, as well as findings on low resource languages, data quality and availability. to support researchers, we release two resources: a platform to search available datasets by our categorization and a template for a summarization data card, to help streamline future datasets.