O pen A sp: A Benchmark for Multi-document Open Aspect-based Summarization (2023)

Abstract: The performance of automatic summarization models has improved dramatically in recent years. Yet, there is still a gap in meeting specific information needs of users in real-world scenarios, particularly when a targeted summary is sought, such as in the useful aspect-based summarization setting targeted in this paper. Previous datasets and studies for this setting have predominantly concentrated on a limited set of pre-defined aspects, focused solely on single document inputs, or relied on synthetic data. To advance research on more realistic scenarios, we introduce OpenAsp, a benchmark for multi-document open aspect-based summarization. This benchmark is created using a novel and cost-effective annotation protocol, by which an open aspect dataset is derived from existing generic multi-document summarization datasets. We analyze the properties of OpenAsp showcasing its high-quality content. Further, we show that the realistic open-aspect setting realized in OpenAsp poses a challenge for current state-of-the-art summarization models, as well as for large language models.

## introduction when faced with a large body of text, a summary is an effective means to get a concise version of the salient content. however, informational needs of users vary, calling for summarizers that can focus a summary around a given request. the summarization community has addressed this demand mainly through query-focused summarization (qfs) and aspect-based summarization (abs). accordingly, several datasets and benchmarks have been compiled over time to enable research on these tasks (see collecting datasets for these tasks is a major obstacle. this kind of data does not naturally occur in available resources, and manually annotating is highly burdening, particularly for the multidocument setting. indeed, the very few existing oabs datasets are synthetically gathered. in addition, they address a single-document setting, even though real-world scenarios involve information dataset domain mds open collection # queries|aspects # instances qfs squality @xcite scifi ✗ ✓ manual 437 625 qmsum @xcite meeting ✗ ✓ manual 1,566 1,808 aquamuse @xcite general ✓ ✓ automatic 5,519 7,168 td-qfs @xcite medical ✓ ✓ manual 40 120 abs facetsum @xcite science ✗ ✗ automatic 4 60,532 ma-news @xcite news ✗ ✗ automatic 6 286,701 aspectnews @xcite news ✗ ✗ manual 4 400 space @xcite reviews ✓ ✗ manual 6 900 wikiasp @xcite wikipedia ✓ ✗ automatic 200 504,546 oabs anyaspect @xcite ) news ✗ ✓ automatic 363,440 2,204,270 oasum @xcite wikipedia ✗ ✓ automatic 1,045,895 3,747,569 openasp (ours) news ✓ ✓ manual 1,266 1,310 our annotation protocol enables efficient collection of high-quality multi-document open-aspect-based summaries. "# queries|aspects" is the number of unique queries or aspects appearing in the dataset, and "# instances" is the number of data instances in the dataset (document set + query|aspect + summary). applying our protocol, we present openasp, an oabs dataset for the multi-document setting. 1 the aspects and respective summaries are based on the prominent duc @xcite and multi-news @xcite datasets. see figure 1 for an example of open-aspect-based summaries extracted from a generic summary. the dataset contains 1,310 aspect-based summaries, split to train, validation and test sets, enabling methodological modeling for the task. we further implement and analyze several baseline models that demonstrate the challenging nature of the task and dataset, even for recent high-performing large language models. early research on abs @xcite recognized the need for more structured information around subtopics of the source text. relevant datasets (middle of an additional direction stemming from abs permits open aspects that still concisely target subtopics, but can be unique for the individual input text (bottom of collecting datasets for the abs task poses a challenging undertaking that includes reading large input texts and writing out the focused summaries. unlike in generic summarization, where large sources of summaries can be scraped from the web @xcite @xcite , summaries for our setting are not generally available. meanwhile, manually writing high quality aspect-based summaries from scratch is an expensive labor-intensive task. @xcite reported 20-40 minutes just for reading a 3000-6000 word story. summarizing multi-document sets is even more complex since the total input-length may be much larger (e.g., tens of thousands of words; see section 6.2), while information-overlap further requires content consolidation by the summarizer. for instance, dang (2005) reported 5 hours of labor for generating one multi-document summary by an expert. employing crowdsource workers, on the other hand, has been shown to lead to poor extractive multi-document summaries @xcite . our protocol exploits existing mds benchmarks and applies controlled crowdsourcing @xcite , and is hence substantially cheaper and more efficient than previous manual collection processes. our openasp dataset addresses all the above issues by supporting open-aspects in the multidocument setting, with manually annotated realistic summaries. moreover, the annotation protocol can be applied across any available generic summarization dataset to produce even more like-quality aspect-based summaries. ## task formulation following prior work @xcite , given a set of texts about a topic, we define an aspect as a central theme within a topic. the aspect can be referred by certain phrases, denoted aspect labels. as an example, research in antarctica and territorial claims are aspect labels of the antarctica topic (see figure 1 ). similar to previous work on abs @xcite , our aspect label is short and concise. in contrast, our aspect definition is open allowing ad-hoc aspects with freeform labels, contrary to having pre-defined domainspecific aspects. relative to a query in queryfocused summarization (qfs; dang, 2005), which might specify a complex information need, our aspects are restricted to relevant subtopics. @xcite @xcite . the oabs task definition follows previous work @xcite , and is extended to the multi-document setting as follows: given a set of documents d on the same topic and an arbitrary aspect label a, the task is to output a short aspect-based summary s a . the summary should consolidate salient information from the document set that is relevant to the aspect. ## annotation protocol as emphasized in section 2, manually collecting aspect-based summaries is very costly. we propose a novel and cost-effective protocol for generating aspect-based multi-document summaries, executed through controlled crowdsourcing @xcite and a specially-designed annotation tool (figure 3 in the appendix). the key idea of our protocol is the extraction of gold aspect-based summaries from generic summaries in existing mds datasets. notably, the process is accomplished by reading the generic summary text only, as described below, while saving the strenuous need to read the entire set of source documents and to write the aspect-based summary from scratch. ## baseline models in this and the subsequent sections we demonstrate the challenges that our dataset lays out for summarization models, and suggest initial directions to cope with these challenges. a major hurdle to overcome is the large input length of a document set, averaging ∼8k tokens in our data, and stretching to ∼30k (section 6.2). even with current advancements made to support growing input sizes, properly attending to relevant information in a large input remains a hurdle. there were no feasible models available to us that would fit all of our document-sets. to cope with this we investigate two common schemas used in the mds setting: (a) filter-thensummarize (e.g., @xcite , runs a sentence selector to extract aspectrelated sentences, and passes them on to a summarization model (7.1); (b) recursive summarization (e.g. shapira and levy, 2020) summarises subsets of documents separately, and then summarizes these summaries (7.2). ## baseline evaluation and analysis to assess the overall capabilities of our baseline models, we show an overall comparison of the methods ( §8.1), an ablation analysis on the bart summ -based configurations ( §8.2), and end with a human evaluation on our best baselines ( §8.3). ## conclusion summarizing texts around an open-aspect is a basic necessity when consuming information. our new openasp benchmark serves this demand, as the first open abs dataset in the multi-document setting, with high-quality summaries collected via an efficient protocol. our protocol overcomes the major hurdle of manually collecting summaries, by tapping into existing generic summaries in multidocument summarization datasets. our proposed baselines, based on strong models, reveal the gap towards solving this task, posing a challenge even for the best current models. overall, our efficient data collection protocol can be expanded to supply even more data for real-world open-abs and related information-seeking tasks.