ASPECTNEWS : Aspect-Oriented Summarization of News Documents (2022)

Abstract: Generic summaries try to cover an entire document and query-based summaries try to answer document-specific questions. But real users’ needs often fall in between these extremes and correspond to aspects, high-level topics discussed among similar types of documents. In this paper, we collect a dataset of realistic aspect-oriented summaries, AspectNews, which covers different subtopics about articles in news sub-domains. We annotate data across two domains of articles, earthquakes and fraud investigations, where each article is annotated with two distinct summaries focusing on different aspects for each domain. A system producing a single generic summary cannot concisely satisfy both aspects. Our focus in evaluation is how well existing techniques can generalize to these domains without seeing in-domain training data, so we turn to techniques to construct synthetic training data that have been used in query-focused summarization work. We compare several training schemes that differ in how strongly keywords are used and how oracle summaries are extracted. Our evaluation shows that our final approach yields (a) focused summaries, better than those from a generic summarization system or from keyword matching; (b) a system sensitive to the choice of keywords.

## introduction recent progress in text summarization @xcite @xcite has been supported by the availability of large amounts of supervised data, such as the cnn/daily mail and xsum datasets @xcite , which provide a single, generic, topic-agnostic summary. however, a document often contains different aspects @xcite that might be relevant to different users. for example, a political science researcher studying responses to earthquakes may want a summary with information about government-led recovery efforts and broader social impacts, not a high-level generic summary of what happened. systems should be able to produce summaries tailored to the diverse information needs of different users. crucially, these systems should be usable in realistic settings where a user is interested in vague aspects of the document, instead of a highly focused query. in this work, we present a new dataset for evaluating single-document aspect-oriented extractive summarization which we call aspectnews. we derive subsets of examples from cnn/daily mail following certain topics, namely earthquakes and fraud reports. these domains are special in that the articles within them have several aspects which are repeatedly mentioned across articles and form coherent topics, e.g., impact on human lives of an earthquake. we ask annotators to select sentences relevant to such information needs, which correspond to imagined use cases. interannotator agreement on full summaries is low due to the inherent subjectivity of the task, so rather than coming up with a consensus summary, we instead primarily evaluate against soft labels based on the fraction of annotators selecting a given sentence. to benchmark performance on this dataset, we build a system that can summarize a document conditioned on certain aspect-level keywords without assuming annotated training data for those aspects. since there are no large-scale supervised training sets suitable for this purpose, we explore methods to generate aspect-oriented training data from generic summaries. we compare these with past approaches @xcite on their ability to adapt to our aspect-oriented setting, which requires taking aspectual keyword inputs (as opposed to specific entities or queries) and being appropriately sensitive to these keywords. our experiments on our aspectnews dataset 1. at least 42 people have died with hundreds more injured after a 6.2-magnitude earthquake hit indonesia's sulawesi island early friday, according to indonesia's disaster management agency. to […] 7. many people are still trapped under collapsed buildings, according to local search and rescue teams. 8. rescuers search for survivors at a collapsed building in mamuju city in indonesia. 9. "our priority is saving victims who are still buried under the buildings," safaruddin sanusi, head of west sulawesi's communications and information department, told cnn friday. […] 12. "most…of the people in mamuju city are now displaced. they are afraid to stay at their houses." 15. "we need more extrication equipment and more personnel to work fast on saving victims trapped under the building." generic geo recovery 1 ✓ ✓ ✓ 2 ✓ 3 ✓ ✓ 4 ✓ 7 8 ✓ 9 ✓ 12 15 and the space dataset @xcite find that our model produces summaries that score higher on agreement with human aspect-oriented annotations than generic summarization models, previous aspect-oriented models, and baselines such as keyword matching. second, we find that the summaries our model generates are sensitive to the choice of keywords. third, we find that our model performs competitively with leading models on the space dataset in the multi-document setting. finally, we find that abstractive query-focused systems @xcite hallucinate significantly in this setting, justifying our choice of an extractive framework here. ## related work relatively little recent work has focused on aspectoriented summarization. one line of research focuses on summarization of documents with respect to specific queries @xcite @xcite @xcite . however, a query such as "what facilities were damaged in the oaxacan region?" is a document specific query, which cannot be applied to other earthquake news articles and bears more resemblance to the task of long-form question answering @xcite . our focus is closer to work on attribute extraction from opinions or reviews @xcite , as factors like geographic details and recovery efforts are usually mentioned in many earthquake stories. recent work has also begun to study summarization from an interactive perspective @xcite ; our approach could be naturally extended in this direction. methods historically, most work on queryfocused summarization has addressed the multidocument setting. @xcite apply regression models to this task, and @xcite approach the problem from the perspective of ranking sentences by their similarity to the query. these classic methods rely integrally on the multidocument setting, and so cannot be easily adapted to our setup. more recently, xu and lapata (2020b) focus on multi-document summarization by modeling the applicability of candidate spans to both the query and their suitability in a summary. @xcite explore a method using quantized transformers for aspect-oriented summarization, which we compare to. datasets there are several differences between aspectnews and other existing aspect-oriented summarization datasets. firstly, aspectnews focuses on single-document summarization, while similar aspect-oriented datasets such as the space dataset of reviews @xcite and other attribute extraction settings @xcite are multi-document. second, our dataset focuses on generalization to new aspect types, rather than assuming we've trained on data with those same aspects; that is, how can we produce appropriate aspect-oriented summaries of earthquake articles even if we have not trained on any? third, compared to query-focused settings, our aspect-oriented dataset is closer to the actual information needs of users, since users are often interested in summaries about broad subtopics rather than specific queries. the tac 2010/2011 summarization datasets 2 domain aspect prompt keywords earthquake geo geography, region, or location region, location, country, geography, miles recv recovery and aid efforts (death toll and injuries, foreign/domestic government assistance, impact on survivors) recovery, aid, survivor, injury, death fraud pen penalty or consequences for the fraudster, or for others penalty, consequences, jailed, fined, court nature nature of the fraud: the amount of money taken, benefits for the fraudster, and how the fraud worked amount, money, bank, stolen, time other previous work @xcite @xcite proposes constructing keyword sets for each individual document for training. @xcite ; @xcite condition on topic tokens referring to the topic tags in metadata. compared to these other approaches, we focus more on evaluation of aspects, as opposed to a purely keywordand query-driven view. ## aspect-oriented data collection we begin by considering our target application: users who have specific information needs that they want to be satisfied. this consideration broadly falls under the category of purpose factors defined by jones (1998) and should be accounted for in the summarization process. our data collection process involves the following steps: (1) identifying clusters of articles in our target domains from a large corpus of news summaries. (2) manually specifying multiple user intents per target domain, representing the aspect of the summarization process. (3) crowdsourcing annotation of extractive summaries in these domains based on the user intents. ## building an aspect-oriented system our aspect-oriented data collection works well to create labeled evaluation data, but it is difficult to scale to produce a large training set. identifying suitable domains and specifying user intents requires significant human effort, and collecting real test cases at scale would require a more involved user study. we build an aspect-oriented model without goldlabeled aspect-oriented training data. we do this by generating keywords for each article in cnn/dm, and training the model to learn the relationship between these keywords and a summary. our system follows broadly similar principles to he et al. @xcite , but in an extractive setting. ## experiments we evaluate our model on the aspectnews dataset, comparing performance on aspect-oriented summarization to several baselines. we additionally experiment on the space multi-document dataset @xcite to provide a point of comparison on a prior dataset and show that our aspect-oriented method is competitive with other systems. ## qualitative evaluation & comparison extractive vs. abstractive comparison it is difficult to directly compare the quality of summaries produced by an extractive model to those produced by an abstractive model. abstractive models do not extract individual sentences from a summary so direct f 1 evaluations cannot be compared in the manner of comparison of factuality of output ## conclusion in this paper, we present a new dataset for aspectoriented summarization of news articles called as-pectnews. unlike query-focused summarization datasets which are often driven by document specific facts or knowledge, this aspect-oriented task is designed to mimic common user intents in domain-specific settings. we present a keywordcontrollable system trained on synthetic data and show that it can perform well on aspectnews without training on the target domains, performing a appendices a.1 training details for all models, we split cnn/daily mail set into the standard 287,226 training pairs, 13,368 validation pairs and 11,490 test pairs following see et al. (2017). we follow the training procedure for bertsum (liu and lapata, 2019) with modifications. we use the cased variant of bert-base-cased available through huggingface (wolf et al., 2019) instead of uncased and do not lowercase the dataset during preparation. our learning rate schedule follows vaswani et al. (2017) with @xmath0.5 , step • warmup -1.5 ) where @xmath1. for fine-tuning aosumm on the modified cnn/dm dataset, the training completes in 8 hours on a single nvidia quadro rtx 8000.