Reference Matters: Benchmarking Factual Error Correction for Dialogue Summarization with Fine-grained Evaluation Framework (2023)

Abstract: Factuality is important to dialogue summarization. Factual error correction (FEC) of model-generated summaries is one way to improve factuality. Current FEC evaluation that relies on factuality metrics is not reliable and detailed enough. To address this problem, we are the first to manually annotate a FEC dataset for dialogue summarization containing 4000 items and propose FERRANTI, a fine-grained evaluation framework based on reference correction that automatically evaluates the performance of FEC models on different error categories. Using this evaluation framework, we conduct sufficient experiments with FEC approaches under a variety of settings and find the best training modes and significant differences in the performance of the existing approaches on different factual error categories.

## introduction factuality (also known as factual consistency, faithfulness) is a crucial dimension in evaluating summary quality. the summaries generated by current summarization models, and even some reference summaries, still have much room for improvement in factuality @xcite @xcite . dialogue summarization, a recently popular subfield of text summarization, has more challenging factual issues involved @xcite . the prior approaches to enhance the factuality of summaries can be broadly classified into two categories: one is to introduce factuality-related objectives in training or inference process to make the summarization models more faithful, which is a direct generation of factually better summaries @xcite @xcite @xcite ; the other is to design a factual error correction 1 code and data will be available at https://github.com/ kite99520/dialsummfactcorr (fec) model independent of the summarization models, which takes the source document and the summary to be corrected as input and outputs a corrected summary @xcite @xcite @xcite . there are a number of studies on news summarization that can fall into both categories. to the best of our knowledge, there has been no work on factual error correction for dialogue summarization. considering the importance of factual issues in dialogue summarization, we would like to try to correct factual errors in dialogue summaries. however, after carefully examining and considering the motivations and practices of previous fec studies, we argue that there are flaws in the way fec models are evaluated, which may have diverted the fec for summarization from its original purpose. previous studies evaluate the effectiveness of fec models mainly by judging whether the scores of factuality metrics (e.g. factcc @xcite ) of the corrected summaries increase compared to the original summaries. first, this evaluation mechanism is so vague that it is difficult to evaluate the effectiveness of factual error correction accurately: we neither know which parts of the original summary have factual errors nor whether the corrected summary addresses them as expected. second, this evaluation mechanism also blurs the line between fec for summarization and the direct generation of factually better summaries: the factual error correction model can ignore the content of the original summary and directly generate a different but more factually correct summary. we argue that it is necessary to introduce manually annotated reference correction to address the above issues. factual error correction for summarization has its basic requirement: to correct factual errors in the original summary by as few substitution, insertion, and deletion operations as possible to obtain a fluent and non-redundant summary. this can be reflected in the manual annotation. the introduction of reference correction, on the one hand, provides more valuable data for the training of fec models compared to pseudo data; on the other hand, and more importantly, it creates the condition for a more comprehensive and accurate evaluation of the performance of fec models. we construct an evaluation framework that can assess the performance of fec models on different factual error categories based on manually annotated references. using this framework, we are able to comprehensively evaluate and analyze the performance of various fec methods on dialogue summarization. our work has the following three main contributions: ## test for factuality metrics we perform a simple test of the reliability of factuality metrics using the above dataset. in general, the factuality metric f takes the source document s and the summary h as inputs and outputs a score f (s, h). a reliable factual indicator needs to satisfy the condition that for summaries with factual errors, the factual score of the corrected summary c is greater than that of the original summary o, i.e., f (s, c) > f (s, o). we select four commonly used factuality metrics: factcc @xcite , dae @xcite , @xcite , and bartscore @xcite . ## reference-based evaluation framework we find that it is difficult for manual annotation to determine the boundaries of erroneous spans accurately sometimes, which hinders the fine-grained evaluation of fec models by error categories. considering these error categories have clear linguistic characteristics, it is more feasible to use a rulebased approach to automatically align and determine error categories when reference correction is already available. we propose ferranti, a factual error annotation toolkit designed for fec. noting the great practice of @xcite , our implementation builds on it. as shown in figure 3 , it mainly consists of three steps: alignment, classification, and comparison. miley needs some rest and does not want to go to work tomorrow. miley doesn't want to go to work tomorrow. u "needs some rest and" → "" r "wants" → "doesn't want" r "wants" → "does not want" (3) our error classification is implemented by explicit linguistic rules rather than manual annotation. ## experiments 6.1 fec approaches we select a few representative fec approaches. among them, we are most interested in such methods: generating corrected summaries directly based on data augmentation because of their flexibility. rule-based transformation cao et al. (2020) use a set of rules that swap the entities, numbers, dates, and pronouns of the reference summaries to construct the summaries to be corrected for training. we call this approach rule. infilling-based transformation balachandran et al. ( 2022 ) mask and predict the subjects, relations, and objects of sentences in the source documents to train an infilling model. the reference summaries are then masked in the same way, and the trained infilling model is used to fill the masked reference summaries to construct the summaries to be corrected. for the infilling model, we experiment with two different setups: (1) using the trained infilling model from the original study, denoted as infill. in addition to the method of generating a corrected summary directly, we also select other approaches, which aim at correcting extrinsic hallucinations: ccgs chen et al. (2021a) replace named entities and numbers in reference summary with the compatible semantic type of content from the source document to generate candidates to train a factual classifier based on bart. at the time of inference, the candidates for the summary to be corrected are generated in a similar way, the trained classifier is used to re-rank the candidates, and the best one is selected as the corrected summary. the spacy library, and it removes or replaces entities and related content in the summary that do not appear in the source document. ## conclusion our work establishes a new benchmark for modelagnostic factual error correction for dialogue summarization. unlike previous studies, we manually correct factual errors in summaries. we point out the shortcomings of factuality metrics in fec evaluation: they are not reliable enough and cannot provide more detailed information. for better evaluation, we propose ferranti, a reference-based evaluation framework and conduct thorough experiments on the performance of multiple fec approaches under various settings. we have the following important findings: 1) training fec models with reference summaries from dialogue summarization datasets yields the best results of unreliable factuality metrics. there is an urgent need to change the evaluation methods for fec models. for future work, it is feasible to apply fer-ranti to fec for other summarization tasks.