Improving Truthfulness of Headline Generation (2020)

Abstract: Most studies on abstractive summarization report ROUGE scores between system and reference summaries. However, we have a concern about the truthfulness of generated summaries: whether all facts of a generated summary are mentioned in the source text. This paper explores improving the truthfulness in headline generation on two popular datasets. Analyzing headlines generated by the state-of-the-art encoder-decoder model, we show that the model sometimes generates untruthful headlines. We conjecture that one of the reasons lies in untruthful supervision data used for training the model. In order to quantify the truthfulness of article-headline pairs, we consider the textual entailment of whether an article entails its headline. After confirming quite a few untruthful instances in the datasets, this study hypothesizes that removing untruthful instances from the supervision data may remedy the problem of the untruthful behaviors of the model. Building a binary classifier that predicts an entailment relation between an article and its headline, we filter out untruthful instances from the supervision data. Experimental results demonstrate that the headline generation model trained on filtered supervision data shows no clear difference in ROUGE scores but remarkable improvements in automatic and manual evaluations of the generated headlines.

## introduction automatic text summarization aims at condensing a text into a shorter version while maintaining the essential information @xcite . methods on summarization are broadly categorized into two approaches: extractive and abstractive. the former extracts important words, phrases, or sentences from a source text to compile a summary @xcite @xcite . in contrast, the latter involves more complex linguistic operations (e.g., abstraction, paraphrasing, and compression) to generate a new text @xcite . until 2014, abstractive summarization had been less popular than extractive one because of the difficulty of generating a natural text. however, research on abstractive summarization has attracted a lot of attentions recently with the advances on encoder-decoder models @xcite @xcite @xcite . english gigaword @xcite is a representative dataset for abstractive summarization. @xcite regarded gigaword as a corpus containing a large number of article-headline pairs for training an encoder-decoder model. their work assumed a task setting where the first sentence of an article is a source text and its corresponding headline is a target text (summary). since then, it has been a common practice to use the gigaword dataset with this task setting and to measure the quality of generated headlines with rouge scores @xcite between system-generated and reference headlines. apparently, a summarization method is desirable to achieve a rouge score of 100, i.e., a system output is identical to the reference. however, this is an unrealistic goal for the task setting on the gigaword dataset. the summarization task is underconstrained in that the importance of a piece of information highly depends on the expectations and prior knowledge of a reader @xcite . in addition, the gigaword dataset (as well as other widely-used datasets) was noisy for summarization research because it was not created for the research objective but other professional activities (e.g., news production and distribution). thus, the state-of-the-art method could only reach rouge-1 scores less than 40 on the dataset. while a number of methods compete with each other for the underconstrained task on the noisy data, we have another concern about the truthfulness of generated summaries: whether all facts of a generated summary are mentioned in the source text. unlike extractive summarization, abstractive summarization has no guarantee of truthfulness. this may result in a serious concern of practical applications of abstractive summarization when a generated summary includes fake facts that are not mentioned in the source document. in this paper, we explore improving the truthfulness in abstractive summarization on two datasets, english gigaword and japanese multi-length headline corpus (jamul) @xcite . in section 2, we analyze headlines generated by the state-of-the-art encoder-decoder model and show that the model sometimes generates unexpected words. in order to estimate the truthfulness to the original text, we measure the recall-oriented rouge-1 scores between the source text and the generated headlines. this analysis reveals that a high rouge score between a reference and headline does not necessarily mean a high truthfulness to the source and that there is only a weak correlation between the two. in section 3, we conjecture that one of the reasons why the model sometimes exhibits such an untruthful behavior lies in untruthful article-headline pairs, which are used for training the model. in order to quantify the truthfulness of article-headline pairs, we consider the textual entailment of whether an article (source document) entails its headline. we will show that about 30-40% of source documents do not entail their headlines under the widely-used experimental settings. in other words, the current task setting is inappropriate for abstractive summarization. we release the annotations of textual entailment for both english gigaword and jamul foot_0 . after confirming the untruthfulness of articleheadline pairs in the datasets, we hypothesize that removing untruthful instances from the training data may remedy the problem of the untruthful behavior of the model. in section 4, we build a binary classifier that predicts an entailment relation between an article and its headline and use the classifier to filter out untruthful instances in the training data. we train a model on the filtered supervision data in section 5. experimental results demonstrate that the filtering procedure shows no clear difference in rouge scores but remarkable improvements when we manually and automatically evaluate the truthfulness of the generated headlines. these results suggest the importance of evaluating truthfulness in addition to relevance. ## improving the truthfulness of data based on the analysis in the previous section, we can consider two strategies to improve the task setting: using full-text articles as source documents instead of leading sentences; and removing nonentailment instances from the dataset. although the former strategy reduces the ratio of non-entailment pair to 7.2% (english gigaword) and 5.8% (ja-mul), we must consider the trade-off: the use of full-text articles increases the cost for training, and may decrease the quality of headlines because of longer inputs to encoder-decoder models. furthermore, jnc does not provide full-text articles but only lead three sentences. therefore, we take the latter strategy, removing non-entailment pairs from the supervision data for headline generation. ## improving the truthfulness of models in this section, we examine whether the supervision data built in the previous section reduces untruthful headlines. ## conclusion and future work in this paper, we showed that the current headline generation model yields unexpected words. we conjectured that one of the reasons lies in the defect in the task setting and data set, where generating a headline from the source document is impossible because of the insufficiency of the source information. we presented an approach for removing from the supervision data headlines that are not entailed by their source documents. experimental results demonstrated that the headline generation model trained on filtered supervision data showed no clear difference in rouge scores but remarkable im-provements in automatic and manual evaluations of the truthfulness of the generated headlines. we also presented the importance of evaluating truthfulness in abstractive summarization. in the future, we explore a more sophisticated method to improve the relevance and truthfulness of generated headlines, for example, removing only deviated spans in untruthful headlines rather than removing untruthful headlines entirely from the supervision data. other directions include an extensive evaluation of relevance and truthfulness of abstractive summarization and an establishment of an automatic evaluation metric for truthfulness. moreover, it will be also interesting to see whether the same issue occurs in other related tasks such as data-to-text generation. we believe that the concern raised in this paper is beneficial to other tasks.