Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency (2022)

Abstract: The topic of summarization evaluation has recently attracted a surge of attention due to the rapid development of abstractive summarization systems. However, the formulation of the task is rather ambiguous, neither the linguistic nor the natural language processing communities have succeeded in giving a mutually agreed-upon definition. Due to this lack of well-defined formulation, a large number of popular abstractive summarization datasets are constructed in a manner that neither guarantees validity nor meets one of the most essential criteria of summarization: factual consistency. In this paper, we address this issue by combining state-of-the-art factual consistency models to identify the problematic instances present in popular summarization datasets. We release SummFC, a filtered summarization dataset with improved factual consistency, and demonstrate that models trained on this dataset achieve improved performance in nearly all quality aspects. We argue that our dataset should become a valid benchmark for developing and evaluating summarization systems.

## introduction while the revolutionary success of the transformer @xcite architecture has drawn a surge of attention to automatic summarization, most research has been focused on improving performance metrics of summarization models on a set of popular datasets. it is often taken for granted that these datasets provide representative examples of high quality summaries, and that models capable of producing summaries that are similar to the ones of the dataset are superior in the task of automatic summarization. but is this really the case? numerous works @xcite have already pointed out deficien-cies in the most widely employed summarization datasets (e.g. cnn/dailymail @xcite and xsum @xcite ). these datasets are typically composed of news articles automatically extracted from news websites, paired together with a highlight or introduction sentence which serves as the summary. however, the qualities we seek in highlights and introductions are fundamentally different from the ones we seek in summaries. it is thus time for the nlp community to take a step back, revisit the formulation of the summarization task and reconsider the appropriateness of currently employed datasets. the goal of automatic summarization is to take an information source, extract content from it and present the most important content to the user in a concise form and in a manner sensitive to the user's or application's needs @xcite . in the scope of this paper, we solely focus on generic summarization @xcite , where we make few assumptions about the audience or the goal for generating the summary. while the importance of the selected content and the conciseness of its form are rather subjective, there is one criterion for summarization that is certain: the summary content should be extracted from the information source. in other words, the summary should be factually consistent with the source document. this is however not the case with summaries constructed from highlights or introductions. it is common for highlights and introductions to contain information that are not mentioned in the main article or even exaggerate certain facts to achieve the goal of click baiting. therefore, the models trained on these datasets perform the task of "pitch generation" rather than the intended summary generation. an example of such a reference summary is given in in this paper, we aim to identify these erroneous data samples by scoring them with factual consistency models. we push towards answering two main research questions : q1 what kind of factual inconsistency can different factuality models capture and how can they be combined for better detection of erroneous samples? this question is answered in sections 3.2 and 5.1 . q2 does having more reliable datasets with more factually consistent reference summaries lead to better performing summarization models? and do the results only improve in factual consistency or also in other quality aspects such as informativeness and saliency? we address this question in section 5.4. our research aims to answer these questions by filtering out samples with problematic reference summaries. we achieve this by leveraging state-of-the-art factual consistency models. we test the performance of different models on different categories of factuality errors @xcite and eventually combine them for an optimized filtration methodology. our contributions are summarized below: we prove that finetuning summarization models on this dataset leads to better performance on not only factuality but also other quality aspects. we believe that regularly amending issues in widely employed benchmark datasets should become a common practice in nlp. source document: archibald, 22, dominated the event, with dutch rider kirsten wild second and belgium's lotte kopecky third. that came 24 hours after she won her third consecutive women's individual pursuit title, having gained silver in thursday's elimination race. the scottish cyclist won gold in the pursuit quartet at the olympics in rio. factually inconsistent reference summary: british olympic champion katie archibald won omnium gold at the european track championships, her second title in two nights in paris. ## related work summarization benchmark datasets summarization benchmark datasets are typically composed of a large number of news documents paired together with "gold-standard" human reference summaries. unfortunately, the human reference summaries in these datasets are often constructed in a suboptimal way. @xcite analyze a sample of 20 papers proposing summarization approaches published in 2021; they find 27 datasets that models were being evaluated on. the most popular ones, cnn/dm @xcite and xsum @xcite , were used five and four times respectively. however, both of these datasets have multiple issues. taking the cnn/dm dataset for example, the construction is done by pairing an article with the bullet points written for it on the cnn and daily mail websites. this design works well for its initial use as a question answering dataset @xcite , but does not function at the same level after its adaptation for summarization. regarding the xsum dataset, the main issue concerns its factuality. the reference summaries were never meant to be a real summary, there is thus no requirement for it to be faithful to the source article. an analysis of xsum finds that over 70% of reference summaries contain factual inconsistencies @xcite . ## analyzing factual consistency models in this section, we present the factual consistency models we employ for detecting factual errors in the reference summaries. we introduce three state-of-the-art models that each rely on a distinctive mechanism. we rely on human annotations from the frank benchmark @xcite to validate the effectiveness of these models and analyze the variations in their performance when shifting dataset domain or error category. the frank benchmark is the most recent benchmark proposed for evaluating factual consistency models, it is the only benchmark to date that provides a categorization of factual errors. ## examining summarization benchmark datasets to question the validity of current summarization benchmark datasets, we first provide an overview of the context in which each dataset was created and the methodology employed in the construction procedures. in addition, we make use of the previously mentioned factual consistency models to evaluate the factuality of human reference summaries collected in each dataset. our selection of summarization datasets is based on their popularity in recently published papers introducing new summarization systems. we believe that it is crucial to examine their validity as they play a fundamental role in both the development and evaluation of new systems. the most popular benchmark datasets according to our criteria are cnn/dm @xcite and xsum @xcite . we also include the very recently released dataset xl-summ @xcite . it is not yet widely employed but including it alongside the other two can help us inspect the most recent advances made in the field of summarization dataset creation. does the most recent benchmark dataset exhibit any improvement on factuality in comparison to the previous ones with well-known issues? ## introducing the summfc dataset the approach we choose to address the factuality problem in these summarization datasets is to filter the samples by their factual consistency scores obtained in section 4.2. we apply this idea to construct summfc, the new summarization benchmark dataset with improved factual consistency. in the remainder of this section, we first present our filtration methodology, followed by statistics of the summfc dataset. we then introduce the evaluation metrics used in the comparison of summaries generated by models trained on the original benchmarks and models trained on summfc. finally, we perform experiments on the three datasets introduced in section 4.1 and observe advantages of the summfc dataset. ## conclusion in this paper, we demonstrate that popular summarization datasets suffer from the lack of factual consistency and that summarization models trained on these datasets are not adequate for the task of abstractive summarization. we show that this problem can be solved by filtering the benchmark datasets with scores from factual consistency models. we propose a filtration methodology combining three stateof-the-art factual consistency models and introduce the summfc dataset. summfc is a unified summarization benchmark consisting of factually consistent samples chosen from cnn/dm, xsum and xlsum. experiments indicate that, in general, models trained on the smaller summfc generate summaries with higher quality than models trained on the larger original datasets. our findings suggest that more deliberate considerations should be made in the construction of benchmark datasets and that continuous revisions for the already existing ones are particularly necessary. however, constructing adequate benchmark datasets with textual contents and labels matching the initial formulation of nlp tasks remains an open question, major obstacle, and unresolved issue for the whole community.