W iki A sp: A Dataset for Multi-domain Aspect-based Summarization (2021)

Abstract: Aspect-based summarization is the task of generating focused summaries based on specific points of interest. Such summaries aid efficient analysis of text, such as quickly understanding reviews or opinions from different angles. However, due to large differences in the type of aspects for different domains (e.g., sentiment, product features), the development of previous models has tended to be domain-specific. In this paper, we propose WikiAsp,1 a large-scale dataset for multi-domain aspect- based summarization that attempts to spur research in the direction of open-domain aspect-based summarization. Specifically, we build the dataset using Wikipedia articles from 20 different domains, using the section titles and boundaries of each article as a proxy for aspect annotation. We propose several straightforward baseline models for this task and conduct experiments on the dataset. Results highlight key challenges that existing summarization models face in this setting, such as proper pronoun handling of quoted sources and consistent explanation of time-sensitive events.

## introduction aspect-based summarization is a subtask of summarization that aims to provide targeted summaries of a document from different perspectives @xcite @xcite @xcite . unlike generic summarization, this gives more concise summaries that are separated according to specific points of interest, allowing readers to fulfill focused information needs more easily and quickly. however, existing aspect-based summarization work is somewhat narrowly focused; for example, a great majority of the work focuses specifically on the domain of product or restaurant reviews. in contrast, generic summarization models are tested on a much wider variety of genres, from newswire @xcite , to academic papers @xcite , to movie scripts @xcite . for each genre, the types and characteristics of aspects that will need to be touched upon in a good summary will differ greatly. one natural source of such multi-domain articles is wikipedia, and the section boundaries and titles in each article form natural annotations of aspects and corresponding text. there have recently been a number of attempts to generate the lead section of wikipedia articles from the linked external sites in the reference section @xcite @xcite , an approach that does not explicitly consider the different aspects covered by the article. @xcite also examine domain differences in wikipedia text summarization. however, existing datasets and analyses lack structure, broad domain coverage, or both. we argue that (1) generating structured summaries is of inherent interest, as these will allow humans consuming the information to browse specific aspects of interest more readily, and (2) the structure will vary across domains, with different domains demonstrating very different characteristics. in this paper, we construct a dataset for multidomain aspect-based summarization that allows us to train models for this unique variety of summarization task, and examine the challenges posed therein. figure 1 illustrates the overview of our task. specifically, we turn to section titles of wikipedia articles and construct sets of ''aspects'' through steps of automatic extraction, curation, and filtering. the section texts then serve as corresponding aspect-based summaries. we devise a baseline two-stage method consisting of aspect identification and summarization using extractive and abstractive models, and conduct experiments on the proposed dataset. the analysis of experimental results and the generated summaries reveals the unique challenges posed by our multi-domain and multi-document setting. for example, aspects that require summarizing contents in a particular order (e.g., time series events) in a multi-document setting adds extra difficulty because of the need for correctly ordering scattered (and possibly duplicate) pieces of information from different sources. certain domains that involve interviews or quotes of people also exhibit challenges in correctly modifying pronouns based on the relationship to the topic of interest. ## generating wikipedia as aspect-based summarization wikipedia articles exhibit a specific way of organizing information about a focused topic. an article s consists of two parts: section titles a, and their contents p. the contents are further split into sections, where each section describes information about the main topic from different viewpoints. the goal of our task is to learn a model f : @xmath0, which can 1) identify and gather information from cited references and 2) generate a sectionby-section summary where each section contains the appropriate type of information. formally, let @xmath1, r 2 , . . . , r m } be a collection of m cited references for an article @xmath2, s 2 , . . . , s n } of n sections. each section s i is essentially a tuple of a section title and one or more paragraphs: although there is a fair amount of variety in section titles across different articles, articles that belong to the same domain tend to share aspects that are particularly salient for that domain. because of this, we select a fixed-size subset of all section titles that appear in each domain as the set of aspects a that we will target; details on how we select this subset will be elucidated in the following section. hence, our task is cast as multi-document aspect-based summarization. ## the wikiasp dataset in this section, we describe our concrete steps to create our dataset. ## baseline models next, in this section we describe two baseline models for solving this task. both of these models decompose the overall process into two stages: aspect discovery and aspect-based summarization of classified sentences. both baseline models share the same methodology for aspect discovery, but differ in terms of summarization models. the model overview is shown in figure 2 . ## evaluation we evaluate models along two axes: aspect discovery and summarization. we note that the primary task in this dataset is aspect-based summarization, thus aspect discovery evaluation discussed below is only for diagnostic purposes. because the aspect sets differ in different domains, evaluation is performed separately for each domain. aspect discovery models have to correctly predict the right set of aspects about which they generate summaries. the aspect discovery criterion aims to evaluate the similarity between the set of aspects about which a model decides to generate summaries and the set of aspects that appear in the target article. 7 for comparing these two sets, we use precision, recall, and f1 scores. aspect-based summarization gold standard summaries only exist for each of the aspects that appear in an article. therefore in this evaluation, we focus on evaluating the model's ability to summarize inputs particularly on these aspects. specifically, generated summaries are paired to corresponding reference summaries with the same aspects and are evaluated using rouge @xcite . because rouge is a recall-based measure, the number of tokens in the model outputs directly affect the performance. controlling the length is particularly important for our dataset because average summary length for each aspect in different domains varies (e.g., ''description'' and ''location'' from historicplace domain has 396 and 90 average tokens, respectively). we take this into account by explicitly setting the maximum number of words for extractive and abstractive summaries to be the average number of words in the target summaries in the training set for each aspect and for each domain. ## experiments we provide two baseline models for the task and evaluate on the proposed dataset. ## analysis we discuss the model outputs and analysis below. ## conclusion and future work in this paper, we propose a large-scale, multi-domain multi-aspect summarization dataset derived from wikipedia. through experiments, we perform an extensive analysis of performance across different genres and aspect types. our analysis has demonstrated that there are both general challenges regarding summarization into various aspects, as well as specific challenges in particular genres such as time-consistent mentions and proper pronoun conversion depending on the writer of the original content. because of this, the proposed datas et also provides a testbed for several potential directions for future work. for example, better aspect discovery models may take into account the coherence of the discourse in the original documents when extracting aspects. better summarization models may take into account the provenance of the information, appropriately determining when the information is written by a first or third party. wikiasp also invites a focus on domains of interest to investigate various problems of text summarization, such as correct pronoun handling and description of chronological timeline.