Automatically Discarding Straplines to Improve Data Quality for Abstractive News Summarization (2022)

Abstract: Recent improvements in automatic news summarization fundamentally rely on large corpora of news articles and their summaries. These corpora are often constructed by scraping news websites, which results in including not only summaries but also other kinds of texts. Apart from more generic noise, we identify straplines as a form of text scraped from news websites that commonly turn out not to be summaries. The presence of these non-summaries threatens the validity of scraped corpora as benchmarks for news summarization. We have annotated extracts from two news sources that form part of the Newsroom corpus (Grusky et al., 2018), labeling those which were straplines, those which were summaries, and those which were both. We present a rule-based strapline detection method that achieves good performance on a manually annotated test set. Automatic evaluation indicates that removing straplines and noise from the training data of a news summarizer results in higher quality summaries, with improvements as high as 7 points ROUGE score.

## introduction automatic text summarization is a challenging task. recent progress has been driven by benchmarks that were collected by scraping a large collection of web-pages, including gigaword @xcite , cnn/dailymail @xcite , newsroom @xcite , and xsum @xcite . due to the way they are collected, these datasets contain a substantial portion of articles that are paired with texts that are not summaries. this flaw in data quality negatively impacts research in two ways: (i) models trained on these benchmarks tend to reproduce flaws in the data, making them less useful for summarization, and (ii) any evaluation against a reference text is meaningless if the reference is not actually a summary. in this work, we present methods for improving the data quality in scraped news summarization corpora, focusing on the newsroom benchmark @xcite . we identify two main issues with the data quality: (i) noise in the extraction process (the wrong field being scraped, markup, ...), which was previously also identified to be an issue by @xcite , and (ii) straplines. according to the writing guidelines used by cern foot_0 , "[t]he strap @xcite gives added "teaser information not included in the headline, providing a succinct summary of the most important points of the article. it tells the reader what to expect, and invites them to find out more." figure 1 shows an example of a strapline ("don't expect to be riding one by 2020") below the regular headline. while the cern guidelines emphasize the function of straplines to provide a summary, we find that most straplines in the newsroom corpus are not summaries of their associated articles. therefore, in order to obtain high quality data, it is necessary to distinguish a strapline aimed at piquing a reader's interest from an abstractive summary. to the best of our knowledge, no work has tried to distinguish straplines from summaries before, and even the word "strapline" does not appear in the acl anthology in a research paper. in our work, one pair of us designed a strapline annotation guideline through discussions and manual pre-annotations ( §3.1) and then annotated a development and test set for evaluating strapline classifiers. based on the guideline, a separate pair created heuristics for a rule-based classifier that distinguishes straplines from summaries ( §3.2). we empirically verify the usefulness of these heuristics for strapline detection ( §4.2). automatic evaluation indicates that removing straplines and noise from the training data with our heuristics results in higher quality summaries, with improvements as high as 7 points rouge score when compared to reference summaries ( §4.3). ## related work several works have analyzed existing summarization datasets from different aspects but none have identified straplines as an issue. @xcite quantified html artifacts in two large scraped summarization datasets which are cnn/dm @xcite , and newsroom @xcite . they found that "summaries" containing such artifacts were found in ≈ 3.2% of the newsroom data. they also argued that many of these artifacts could be detected using simple regular expressions and heuristics. @xcite define three sub-aspects of text summarization and analyze how different domains of summarization dataset are biased to these aspects. bommasani and cardie (2020) evaluate the quality of ten summarization datasets, and their results show that in most summarization datasets there are a sizable number of low quality examples and that their metrics can detect generically low quality examples. @xcite analyzed 600 samples from three popular datasets, studying the data quality issues and varying degrees of sample complexity, and their analysis of summarization models demonstrate that performance is heavily dependent on the data and that better quality summarization datasets are necessary. given that research has shown that the training data of summarization models are noisy, researchers have proposed methods for training summarization models based on noisy data. for example, @xcite propose a model that can quantify noise to train summarization models from noisy data. the improvement of the models indicates that the noisy data has noticeable impacts for the training of the models. ## methodology the newsroom corpus contains articles from 38 news sources that vary in style and topics. news articles were scraped from html pages, where the page's title tag is parsed as the article's headline, while the page's body tag is parsed as the article's body. since there was no consistent metadata tag indicating the summary of an article, @xcite used different metadata tags to extract summaries. these tags are generally added to be used by social media platforms, and search engines. news publishers do not share a single format for organizing metadata. nevertheless, all (or most) use the metadata label description, albeit for different things. since the creators of newsroom take as the summary of each article, the first tag in its metadata having the keyword description, this might be one reason that a strapline appears in the extract for an article in place of the real summary. knowing that the "summaries" in the newsroom corpus are of mixed quality, we call what @xcite scraped from the web extracts, which may or may not be a genuine summary. @xcite classify extracts according to how much text they repeat verbatim from the article into three categories: extractive (nearly everything appears verbatim in the article), abstractive (summarize in different words) and mixed. we have focused on extracts classified as "abstractive". we have also limited our study to two of the 38 news sources -ones with different styles and covering different topics, specifically the new york times (nyt) and time.com. ## conclusion we present methods for improving the data quality in scraped news summarization corpora, focusing on the new york times and time magazine sections of newsroom @xcite . we identify two main issues with the data quality that make newsroom less appropriate as a summarization benchmark: (i) noise in the extraction process and (ii) presence of straplines in place of genuine summaries. after identifying common characteristics of straplines, we develop a set of effective heuristics for detecting straplines and noise. our work shows that when straplines and noisy data are excluded from the training data, the result- news source output of original model (m original ) output of cleaned model (m clean ) nyt a day in the life of a olympic athlete. the australian swimmer mack horton was booed by chinese swimmers after his victory in the 200-meter freestyle, and russian swimmer irina efimova was booed. nyt a day in the life of a yankees fan. the yankees victory parade on friday was a celebration of the teamâs success, but not everyone was there. nyt a new york times blog about comic book publishing and design. kevin conroys performances as batman in the comic books, movies and television series stand out. nyt new york times reporters and editors are reporting from washington, d.c. a new hampshire biologist turned to film school to learn how to communicate scientific information. nyt reading, watching, discussing and blogging the day's local, national, and international news at the new york times. the university of illinois, chicago, has a bright spot in its diversity.