News Editorials: Towards Summarizing Long Argumentative Texts (2020)

Abstract: The automatic summarization of argumentative texts has hardly been explored. This paper takes a further step in this direction, targeting news editorials, i.e., opinionated articles with a well-defined argumentation structure. With Webis-EditorialSum-2020, we present a corpus of 1330 carefully curated summaries for 266 news editorials. We evaluate these summaries based on a tailored annotation scheme, where a high-quality summary is expected to be thesis-indicative, persuasive, reasonable, concise, and self-contained. Our corpus contains at least three high-quality summaries for about 90% of the editorials, rendering it a valuable resource for the development and evaluation of summarization technology for long argumentative texts. We further report details of both, an in-depth corpus analysis, and the evaluation of two extractive summarization models.

## introduction news summarization has been, and still is subject of active research to this day @xcite @xcite . however, the inverted pyramid structure of news reports, where the lead paragraphs often have a summarizing quality in and of themselves @xcite , induces a bias in many recent news summarization approaches to just copy the opening sentences @xcite . hence, these approaches fail at argumentative news articles, such as editorials, whose structure differs from that of news reports. editorials represent the views of an organization (newspaper) on long-standing societal issues and aim to shape public opinion @xcite . compared to news reports, which aim to inform objectively about current events, editorials subjectively assess a controversial topic in order to persuade its audience of a specific stance toward it @xcite . this difference in their goals leads to a difference in linguistic choices. an editorial is usually composed of three discourse parts, namely lead, body, and conclusion @xcite . the lead introduces the issue at hand by starting with an anecdote or a question. the body elaborates on arguments and background information, while the conclusion provides an evaluation as well as (possibly) implicit suggestions and calls to action @xcite . the summary of an editorial must hence be constructed with care in order to preserve its argumentative structure and its persuasive means. research on automatic (news) summarization has so far neglected argumentative texts in general, and the genre of editorials in particular. with this paper we contribute to closing this gap, taking effective steps toward editorial summarization: the evaluation indicates a high suitability of the corpus for research and development: for 90% of the editorials there are at least three high-quality summaries, and for 52% all five are. the analyses also reveal that multiple summaries can be collected for an editorial with low content overlap, that good summaries include more third party evidence to justify an editorial's thesis, and that editorials' summaries have a distinct structure compared to those of news reports, with a specific contribution from each editorial discourse unit (lead, body, and conclusion). the corpus and other resources are publicly available. foot_0 ## related work the summarization of argumentative text has hardly been studied: @xcite automatically summarize political online debates by extracting key content from their arguments as "points" (verbs and their syntactic arguments). similarly, @xcite propose to map crowdsourced arguments to "key points." they created the argkp corpus, containing 24,000 argument, key point pairs, extracted from the ibm-rank-30k dataset @xcite . to the best of our knowledge, no argument corpus comprises summaries of long-form monological argumentative text. most of the commonly used corpora for automatic news summarization, such as the nyt corpus @xcite , gigaword @xcite , cnn/dailymail @xcite , xsum @xcite , and newsroom @xcite , primarily consist of (non-argumentative) news reports and only one ground truth summary per report. although the duc shared task datasets @xcite provide multiple summaries per document (500 news reports), they are very short (up to 14 words or 75 bytes), similar to gigaword and xsum (up to two sentences). these corpora, stemming from the news domain, may contain some editorials; the ones in the nyt corpus were studied by @xcite , @xcite , el @xcite , and el @xcite for tasks such as summarization, analysis of rhetorical strategies, and argumentation quality assessment. but @xcite observes that the accompanying summaries in this corpus are teasers rather than actual summaries. in our work, we focus on composing summaries exclusively for news editorials by aiming to capture their core argumentation, providing multiple and comparably longer ground truth summaries (20% of an editorial's segments) for each editorial. a scheme for annotating argumentative roles of sentences for summarizing research articles was presented by @xcite . however, they only analyzed the effectiveness of this scheme and did not collect or evaluate any summaries. the key difference between other news summarization corpora and ours is the use of a (genre-specific) annotation scheme that unifies the summary acquisition and evaluation. other summarization corpora lack such unification and only adopt the notion of "salience" (importance) of sentences in a text @xcite to automatically extract summaries or crowdsource their acquisition @xcite . in the absence of a human-written ground truth, parts of a text, such as the title, highlighted sentences, or lead sentences, are used as proxies for summaries of the source documents. while such heuristics help create large corpora, the infeasibility of evaluating all the ground truth summaries leads to increased noise in the datasets, severely limiting the task of summarization and its evaluation @xcite . we evaluated each summary in our corpus for its quality and provide labels for high (low) quality per quality dimension defined in our annotation scheme. ## annotation scheme for editorial summaries as per @xcite , humans produce two types of summaries: writer-based ones and reader-based ones. a writer-based summary is produced to facilitate one's own comprehension of a text. a reader-based summary intends to inform others about a text's core message, possibly to evoke further interest in the reader. news, in particular, may also be accompanied by a teaser, namely an incomplete summary that aims to attract people to read the entire news article (report or editorial) @xcite . in extreme cases, teasers can become clickbait @xcite , constructed to manipulate their readers to visit an online news article (e.g., by invoking strong curiosity). for our corpus, we strive for reader-based summaries. the intention of a reader-based summary is often to substitute the original text. for informational texts, such as news reports, this is roughly performed by the omission of irrelevant sentences (deletion), the subsumption of details into higher-level categories (generalization), and the integration of details into topic sentences (construction) @xcite . reorganization and rewording are possible @xcite , but new ideas must not be introduced @xcite . in this regard, an editorial aims to persuade its readers of one central claim (thesis) through its monological argumentation @xcite . it is composed of argumentative discourse units (adus, typically statements) that form arguments to support the thesis @xcite . these arguments implement the author's strategy, incorporating not only logical, but also emotional and credible means of persuasion @xcite . the core message of an editorial corresponds to its thesis and its most persuasive segments; thus, an editorial's summary-and that of long argumentative texts in general-should aim to preserve both. we propose an annotation scheme tailored for editorial summaries, defining five quality dimensions that emphasize argumentation as well as summarization quality: ## summary acquisition based on a corpus of news editorials that have previously been annotated with regard to argumentative discourse units, we crowdsource the generation of multiple reader-based summaries using amazon's mechanical turk. the summarization is framed as an annotation / extraction task, where segments from an editorial are selected to compose a summary.