Updated Headline Generation: Creating Updated Summaries for Evolving News Stories (2022)

Abstract: We propose the task of updated headline generation, in which a system generates a headline for an updated article, considering both the previous article and headline. The system must identify the novel information in the article update, and modify the existing headline accordingly. We create data for this task using the NewsEdits corpus by automatically identifying contiguous article versions that are likely to require a substantive headline update. We find that models conditioned on the prior headline and body revisions produce headlines judged by humans to be as factual as gold headlines while making fewer unnecessary edits compared to a standard headline generation model. Our experiments establish benchmarks for this new contextual summarization task.

## introduction automatic text summarization condenses the most important and salient information from a large quantity of text. the task takes many different forms depending on the type of information being summarized, the modality of the information, the type of summary desired and the needs of the end user. examples include news headline generation @xcite @xcite @xcite , summarization of social media @xcite @xcite , and medical documents @xcite @xcite . in many settings, users encounter information progressively instead of all at once. for instance, news stories are revised as events unfold @xcite , social media streams evolve as people post content @xcite , and biomedical texts are revised as clinical trial results emerge @xcite . in such dynamic settings, existing summaries should be updated as new information becomes available. to address this, we could in principle leverage static summarization systems for generating a summary of the underlying content at any given point in time. however, a more natural approach would be to produce a new summary based on what the reader already knows and what content changed. consider the case of a news article being updated as events unfold (figure 1 ). the article first reports that a man is charged with stealing an ice cream van, and the article is later updated when the man admits to the crime. by the time the article is updated, the reader already knows what was stolen, who was charged, and where it happened. at this point, the reader is most interested in what changed, namely the admission of guilt. in the case of news articles, the new headline must both convey critical new information and provide a holistic overview for readers unfamiliar with the story. updating a summary instead of wholesale replacement falls outside the scope of static summarization systems. to address these shortcomings, we envision a summarization system that combines an existing summary with information updates. more concretely, following prior work of using headlines as article summaries @xcite , we consider the task of news headline generation. we instead propose updated headline generation, which entails updating headlines based on changes to the content of the article. in this work, we make the following contributions: • conduct a human evaluation demonstrating that leveraging this additional context leads to headlines which are as factual as standard headline generation models, while applying fewer unnecessary edits. ## updated headline generation a news article consists of a body (b) and a headline (h). headline generation @xcite @xcite @xcite asks a system to consider b and produce h. we propose updated headline generation as a modification of this task. a system receives an existing article (b 1 , h 1 ) and an updated version of the article body (b 2 ). the goal is to update h 1 to produce a new headline (h 2 ) that reflects important new information in b 2 . this task introduces several challenges. first, a system must identify the most critical new information in b 2 . changes to the article can be small or very significant, and it must determine which of these changes, if any, should be reflected in the headline. second, it needs to consider how to modify h 1 . oftentimes a revision to an article will preserve most of the structure of h 1 , even if a completely rewritten headline might convey the same information. new information should be reflected in an updated headline with minimal edits, for the sake of continuity and minimizing cognitive load on a reader who is following an evolving story. third, there are different types of updated stories that each require a different style of headline update. stories can be updated as the underlying event progresses (e.g., criminal investigations, natural disasters, voting on legislation or appointments, live events), new or corrected information becomes available (e.g., number of people injured following an accident), or public figures react to the event (e.g., political figure commenting on a situation). see ## dataset the newsedits @xcite corpus contains articles with revision histories derived from 22 wires: 5 from news sniffer foot_1 and the remainder from twitter accounts powered by diff-engine. 3 it consists of over one million articles with 4.6 million revisions. in this work, we focus on the 5 english language wires from news sniffer (washington post, ny times, independent, bbc, guardian), as we found them to have cleaner revision histories. from the revision history of a given article, we extract body-headline pairs by examining consecutive versions, (b k , h k ), (b k+1 , h k+1 ), resulting in examples of the form {(b 1 , h 1 ), (b 2 , h 2 )}. we b1 h1 b2 h2 nearly a million people in southern vietnam face evacuation from the path of a deadly tropical storm... exclude cases without a change in the body, and group examples into two different classes: positiveexamples where the headline is updated (i.e., h 1 ̸ = h 2 ) -and negativethe headline remains unchanged (i.e., h 1 = h 2 ). we observed that the headline change associated with a particular body change sometimes occurred in the subsequent revision (not contemporaneous). so, we also include positive examples which have the following property across three consecutive revisions: only the body is changed between the first and second versions and only the headline is changed between the second and third versions, i.e., (b 1 , h 1 ) → (b 2 , h 1 ) → (b 2 , h 2 ). we do not include (b 1 , h 1 ) → (b 2 , h 1 ) as a negative example for such cases. to avoid spurious positive examples, we tried removing versions that were incorrectly paired together, 4 or where the headline change was trivial. 5 this process produced a dataset of 144,218 positive 4 b1 and b2 are sometimes completely unrelated, likely due to an error in the news sniffer collection. we removed examples in which b2 was published more than a week after b1, and we exclude articles that yield more than 8 version pairs (95th percentile). 5 trivial headline changes included modifications limited to spacing and punctuation, as well as simple rephrasing (i.e., changes to stopwords or the surface form of a lemma). and 794,372 negative examples. even after filtering by heuristic, we found that many of the remaining headline changes still do not reflect a substantive update to the article. these include purely stylistic changes, embellishments, and rephrasings. to filter such cases, we develop a classifier which is trained to determine whether h 1 needs to be updated based on the changes between b 1 and b 2 . the classifier achieves 51.9 f1, indicating that this is a challenging problem; the training and evaluation data are silver-labeled, and noisy. we filter the remaining positive examples with this classifier. empirically, we find that training on this filtered subset leads to improved performance. we provide a complete description of the classifier and attendant experiments in appendix a. ## sources of information we study the importance of several types of information -in the form of baselines and inputs to models -for updated headline generation. ## models we evaluate two encoder-decoder models that utilize each of the representations described in section 4.2. note that we first preprocess all representations using the penn treebank tokenizer foot_4 to tokenize and split text into sentences and words, prior to model-specific preprocessing. pointer networks consist of separate lstm encoders for body and headline text, and these are concatenated to form the initial states for an lstm decoder, equipped with attention @xcite . the hidden states are concatenated for both attention and copy mechanisms. we posit that this model might be effective at headline updating, as this task benefits from copying tokens from the input context (especially h 1 ). we initialize embeddings for the model with glove @xcite . bart @xcite ) is a pretrained transformer network considered state-of-the-art for summarization. because we focus on the news domain, we consider a version of bart already fine-tuned for summarization on news articles from @xcite . 8 we further fine-tune on our data, by concatenating inputs into a single sequence, separated by special tokens (e.g., <old_headline>, <@xmath0. we evaluate all context representations with both of these architectures, with the exception of h 1 + b 2 + b 1 for bart, due to limitations in fitting the entire input context within bart's 1024 token limit. we use beam search with a beam size of 20 to decode for all models along with bigram blocking @xcite . 9 these decoding hyperparameters were found to work well across models during preliminary experimentation based on an unweighted average across automated metrics. ## experiments we evaluate with common text-generation metrics: meteor @xcite , rouge-l @xcite @xcite . given the editing nature of our task, we also use two edit-specific metrics: gleu @xcite and sari @xcite . sari measures the average n-gram f1 scores corresponding to edit operations (add, delete, and keep). gleu closely follows bleu except that it places more importance on n-grams which have been correctly changed. we compute statistical significance at the @xmath1.05 level using bootstrap tests @xcite . rule-based baselines: our results ( to h 1 will learn to copy many parts of this input, and consequently will make fewer edits. using body edits: to investigate whether providing body edits can further improve performance by helping a model learn to correlate them with h 1 and apply analogous updates, we consider different ways of incorporating b 1 . first, in the pointer network, we evaluate performance when just feeding it in as another input (h 1 + b 2 + b 1 ). we observe no improvement in performance, suggesting that the model fails to implicitly learn the edits. next, we consider collapsing b 1 and b 2 into a sequence of edits (h 1 + b edits ), with which we see a slight improvement in performance over h 1 + b 2 for the pointer network but a reduction in performance for bart. we believe this is because bart struggles to model longer input sequences. when we reduce the context length and provide only the changes in the edit sequence (h 1 + b edits (change only)), we see an improvement in bart. note that the performance of h 1 + b edits (change only) is lower for the pointer network across most metrics. this may be due to a lack of pretraining, whereas bart is already equipped with a strong language model. pointer network vs. bart: while both model classes perform well, bart models tend to perform better overall, demonstrating the value of bart's larger transformer-based architecture and pretraining. nonetheless, the benefits of using h 1 and body edits generalize across both architectures. we expect that the performance of more recent summarization models such as pegasus @xcite or simcls @xcite will exhibit a similar trend as bart, but we welcome evaluation of other large pretrained summarization models on hren. ## discussion case study performance by edit level headlines require more extensive edits when there are more substantial changes to the article. we perform a finegrained analysis to better understand how various context representations fare for these different types of examples. we group examples based on the jaccard similarity between h 1 and the gold h 2 ; low similarity means significant edits, high similarity means minimal edits. for the bartbased model we calculate the gleu score for each bucket because we find that it is better suited for simultaneously evaluating whether appropriate edits were made along with generation quality. figure 2 shows the change in performance attributed to each of the context representations relative to headline generation (b 2 ). for low similarity values, none of the specialized context representations outperform standard headline generation. this suggests that when more substantial edits are needed, starting from scratch may be best. as the similarity increases, models which utilize h 1 perform substantially better. for moderate similarity, having b 2 instead of body edits performs marginally better, but this changes as the similarity score increases, with b edits (change only) leading to drastic improvements. analyzing attention to better understand how models explicitly make use of the old headline, we analyze how the h 1 + b edits (change only) bart model's decoder attends to h 1 . for this, we follow the methodology of @xcite . namely, we label each context token by which span it belongs to: h 1 , one of the edit spans, or a span delimiter token (other). we compute the average attention paid to each context token class by the bart decoder across all examples and layers. we find that even though only 1.5% of the context tokens are from h 1 , they attract over 17% of the bart decoder's attention. interestingly, the attention paid to added content and headline tokens increases in later layers at the expense of other tokens (figure 3 ). we posit that this is because the initial layers need to attend to special tag tokens in order to understand which type of span each enclosed token belongs to. this may also arise from the fact that initially the decoder attends to all tokens relatively uniformly (49.9% of tokens are other on average). however, even in the initial layers, the tokens in h 1 are attended to more than would be expected by a uniform attention distribution, likely because h 1 text always appears near the start of the context. because of this, locating the h 1 tokens is less dependent on identifying enclosing tags -absolute position also helps. we also find that the decoder attends to tokens in h 1 more often than would be expected under a uniform attention model, until it needs to refer to a new piece of information that was added to the article body. figure 4 displays the relative attention paid to each token type for a decoded headline exemplifying this phenomenon. see appendix e for additional detail on the decoder attention analysis. error cases finally, we inspected cases where annotators assigned very low or high scores. we observe with b 2 alone, the headline generation model makes factual errors by mixing up important details when two similar types of entities are discussed in the article (e.g., mixing up the victim and suspect of a crime, mixing up locations and dates). additionally, it makes factual errors by omitting something important, which drastically changes the meaning (e.g., missing a letter in the acronym for an organization). on the other hand, because h 1 often includes important background that can be directly copied, we find fewer such factual errors caused by omission for the h 1 + b 2 and h 1 + b edits (change only) models. having h 1 also helps in maintaining important details (e.g., event location) and specifying the level of detail that is needed. in general, h 1 is most useful when there is high lexical overlap with the lead sentences of b 2 . if the content is significantly different (e.g., the focus of the article changes), it becomes less useful and can even hurt performance in some cases, since h 2 is likely very different from h 1 . body edits are most useful when there are few edits and these edits can be easily grounded in h 1 . for h 1 + b edits (change only), we also noticed errors where the model incorrectly correlates body edits with h 1 , resulting in it erroneously inserting body tokens that are edited into the headline. natural language edits: our work focuses on learning from edits in news articles to apply update and existing headline. prior work studies the nature of edits in various texts including news @xcite and wikipedia @xcite . there has also been extensive work on generating edits for tasks such as grammatical error correction @xcite , sentence simplification @xcite , style transfer @xcite , fact-based sentence editing @xcite ), text improvement @xcite , and comment updating based on source code changes @xcite . ## conclusion in this work, we show that headline generation models can benefit from access to the past state of the article. our proposed model, h 1 + b edits (change only), can generate headline predictions that are statistically tied with gold headlines in terms of factuality, while making fewer unnecessary edits. by releasing the hren dataset, we hope to encourage the community to produce higher quality tools for aiding journalists, as well as encourage research in nlp over dynamic texts.