SUBSUME : A Dataset for Subjective Summary Extraction from W ikipedia Documents (2021)

Abstract: Many applications require generation of summaries tailored to the user’s information needs, i.e., their intent. Methods that express intent via explicit user queries fall short when query interpretation is subjective. Several datasets exist for summarization with objective intents where, for each document and intent (e.g., “weather”), a single summary suffices for all users. No datasets exist, however, for subjective intents (e.g., “interesting places”) where different users will provide different summaries. We present SUBSUME, the first dataset for evaluation of SUBjective SUMmary Extraction systems. SUBSUME contains 2,200 (document, intent, summary) triplets over 48 Wikipedia pages, with ten intents of varying subjectivity, provided by 103 individuals over Mechanical Turk. We demonstrate statistically that the intents in SUBSUME vary systematically in subjectivity. To indicate SUBSUME’s usefulness, we explore a collection of baseline algorithms for subjective extractive summarization and show that (i) as expected, example-based approaches better capture subjective intents than query-based ones, and (ii) there is ample scope for improving upon the baseline algorithms, thereby motivating further research on this challenging problem.

## introduction traditional non-generic extractive summarization systems allow users to express their summarization intent via a query or a natural-language question @xcite @xcite . while this simplifies the interaction between the user and the system, queries are not the best means for expressing very subjective intents. consider a user trying to summarize the wikipedia pages of all us states to find places that would be interesting to them. a query such as "interesting places" may report places that are of * equal contribution. the example-based paradigm programming-byexample (pbe) has been successful for a variety of tasks, such as: code synthesis @xcite ; data wrangling @xcite , integration @xcite , and extraction (le and @xcite ); text processing and normalization @xcite ; querying relational databases @xcite , and even creative tasks such as music composition @xcite ). an interface for extractive summarization by example was proposed in sudocu @xcite , offering an easy and natural way for users to annotate documents to construct example summaries: the user browses through the document, optionally performing keyword search, and simply clicks on sentences that should be included in the summary. the system then infers the user's intent from the provided examples, and learns the mechanism to automatically summarize the rest of the unseen documents. figure 1 contrasts the traditional query-based interface (left) with an example-based one (right). the interface makes it easy for users to construct a few example summaries from a corpus. summarization by example is powerful for several reasons: first, it allows the system to access more information than what a query might provide, and, thus, such a paradigm is expected to produce better results than the traditional query-based approaches. second, it allows users to express very ## related work several datasets exist for generic summarization tasks, including the cnn/daily mail dataset (nallapati et al., 2016) which contains 300,000 news article-summary pairs, webis-tldr-17, which contains three million document-summary pairs extracted from reddit forums @xcite , multi-news dataset, which is a multi-document summarization dataset containing over 50,000 articles-summary pairs @xcite , and the gigaword @xcite and x-sum @xcite datasets, both of which contain singlesentence summaries of news articles. scisummnet @xcite is a manually annotated corpus for scientific papers on computational linguistics to generate summaries that include the impacts of the articles on the research community. talksumm @xcite is for scientific paper summarization based on conference talks. however, it does not consider personalization, where different people might want different summaries of the same paper. in general, none of the above datasets are suitable for the task of subjective summarization, which is our focus. a task close to ours is query or topic-based extractive summarization. suitable datasets include duc 2004, duc 2005, and duc 2006, which contain query-based (multi-)document summaries (duc). webis-snippet-20 consists of 10m web pages together with their query-based abstractive snippets @xcite . in these datasets, each document (or set of documents) has one or more summaries with respect to a single query. in contrast, subsume contains multiple summaries of each document corresponding to different intents. furthermore, each document-intent pair is summarized by multiple individuals. @xcite , in the context of "aspect-based" summarization, provide a dataset having multiple topic-focused summaries mostly objective (i1) how is the weather of the state? (i2) how is the government structured in this state? (i3) what is the state's policy regarding education? (i4) what are the available modes of transport in this state? balanced subjective/objective (i5) what drives the economy in this state? (i6) what are the major historical events in this state? mostly subjective (i7) what about this state's arts and culture attracts you the most? (i8) which places seem interesting to you for visiting in this state? (i9) what are some of the most interesting things about this state? (i10) what are the main reasons why you would like living in this state? for each document. the dataset is synthetic, however, and does not involve human annotators. to the best of our knowledge, subsume is the first human-generated dataset for subjective, extractive document summarization, where interpretation of intents varies across individuals. ## dataset description we now describe our data collection process and design choices, and analyze the statistical properties of the dataset. the dataset is available publicly at https://github.com/afariha/subsume . intents. we devised ten intents with different degrees of subjectiveness, ranging from mostly objective to mostly subjective, as shown in figure 2 . "objective" intents refer to unambiguous facts (weather, modes of transport), "subjective" intents refer to opinions (interesting, attractive), and the "balanced" intents correspond to a ranking of unambiguous facts according to subjectively estimated importance (drivers, major events) that are expected to vary only moderately between individuals. a statistical analysis (see below) supports our heuristic classification of intents. ## experiments in this section, we benchmark existing summarization techniques over subsume in two settings: query-based (qb) and example-driven (ex). recall that for every user-intent pair, subsume consists of summaries of eight documents. in the ex setting, we use summaries of five documents, chosen at random from the eight summaries, as example summaries to learn the user's intent, and evaluate on the remaining three documents. in the qb setting, the baselines summarize the documents using only the query (intent text), and we evaluate on the same set of three documents as in the exampledriven setting. we repeat this over ten different splits of the eight document-summary pairs, and average out results across all splits, and over all data points. we report f 1 scores of rouge-1, rouge-2, and rouge-l metrics @xcite ) for all the baselines. ## conclusions in this paper, we present subsume, the first dataset for evaluation of subjective summarization systems, and evaluate existing baselines on the dataset. the results presented in this paper show that even the best-performing approaches leave significant room for improvement for subjective document summarization, encouraging further research. in future, we plan to investigate transfer-learning and few-shot learning approaches that naturally fit the task of subjective summarization by example. another direction of future work is to investigate how our data-collection method can be extended to other domains beyond the domain of wikipedia documents of us states.