ACLS um: A New Dataset for Aspect-based Summarization of Scientific Publications (2024)

Abstract: Extensive efforts in the past have been directed toward the development of summarization datasets. However, a predominant number of these resources have been (semi)-automatically generated, typically through web data crawling. This resulted in subpar resources for training and evaluating summarization systems, a quality compromise that is arguably due to the substantial costs associated with generating ground-truth summaries, particularly for diverse languages and specialized domains. To address this issue, we present ACLSum, a novel summarization dataset carefully crafted and evaluated by domain experts. In contrast to previous datasets, ACLSum facilitates multi-aspect summarization of scientific papers, covering challenges, approaches, and outcomes in depth. Through extensive experiments, we evaluate the quality of our resource and the performance of models based on pretrained language models (PLMs) and state-of-the-art large language models (LLMs). Additionally, we explore the effectiveness of extract-then-abstract versus abstractive end-to-end summarization within the scholarly domain on the basis of automatically discovered aspects. While the former performs comparably well to the end-to-end approach with pretrained language models regardless of the potential error propagation issue, the prompting-based approach with LLMs shows a limitation in extracting sentences from source documents.

## introduction the availability of high-quality datasets annotated with ground-truth human judgements has been a staple of the elements required to advance research in nlp for a very long time, dating back to the very dawn of statistical nlp @xcite . unfortunately, the availability of such resources 1 our data is released at https://github.com/ sobamchan/aclsum . existing statistical phrasal or hierarchical machine translation system relies on a large set of translation rules which results in engineering challenges. they propose to use factorized grammar from the field of linguistics as more general translation rules from xtag english grammar. the proposed method consistently outperforms existing methods in bleu on various low-resource language translation tasks with less training data. ## related work a common practice to build summarization datasets is to find data on the internet which can be used as a silver-standard proxy for documentsummary pairs @xcite @xcite , e.g., news articles and their lead sentences @xcite . while scalable and very practical to build datasets on a large scale, the resulting summaries exhibit a few fundamental limitations. unfaithful summaries. @xcite show that xsum @xcite ) contains summaries that are unfaithful to source documents in the sense that these 'summaries' are rather 'eyecatching' sentences to draw readers to the corresponding articles for the news platform from which the dataset is extracted. noisy data. @xcite report that cnn/dailymail @xcite and newsroom @xcite contain much noise, such as urls and placeholder texts in their summaries. more recently, @xcite show that more than 60% of the reference summaries in the test set of the arxiv dataset which is built for long document summarization contain noises. moreover, in a preliminary study, we investigated automatic language detection models and found that 0.4% samples in the test set of xsum are, for instance, written in welsh (as opposed to english). ## aclsum we validate the quality of the annotations in our dataset by taking 75 summaries (25 aics times three aspects) from our validation split and let two additional domain experts evaluate the quality. we use the four criteria proposed by @xcite , namely relevance, consistency, coherence, and fluency. we do not evaluate fluency for extractive summary annotations by assuming the texts published at acl conferences are well polished, and also consistency since the sentences are extracted without modifications from source documents. for abstractive summaries, coherence is excluded since this measures "the quality of all sentences collectively" @xcite while our reference summaries are composed of a single sentence. the complete annotation guidelines can be found in the appendix a.2. on all aspects and criteria, our both extractive and abstractive annotations achieve above 4 on a 1-5 likert scale ( we show in figure 2 the relative positions of sentences annotated with aspects, highlighting how the three different aspects are highly interspersed across documents, thus indicating that indeed models are required to attend to many different parts of the document for each aspect. ## experiments and results we next use aclsum to answer the following research questions: ## conclusion in this paper, we presented aclsum, a manually crafted and validated multi-aspect summarization dataset for both extractive and abstractive summarization systems. using aclsum, we performed experiments using summarization models based on pretrained language models and more recent large language models such as llama, as well as evaluating a standard algorithm to create extractive summarization datasets. in future work, we plan to explore ways to use our annotated data to bootstrap and extend our dataset through (semi-)automatic data augmentation methods, as well as build datasets for other fields, including other areas of computer science and other domains, possi-bly in languages other than english, such as, e.g., publications from the social sciences and humanities. we additionally plan to explore ways to use our aspect-based single document summarization models to enable multi-document summarization of scientific publications, a yet under-researched setup with much potential to provide challenging tasks in the age of large-scale text understanding and generation.