Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization (2021)

Abstract: With the continuous upgrading of the summarization systems driven by deep neural networks, researchers have higher requirements on the quality of the generated summaries, which should be not only fluent and informative but also factually correct. As a result, the field of factual evaluation has developed rapidly recently. Despite its initial progress in evaluating generated summaries, the meta-evaluation methodologies of factuality metrics are limited in their opacity, leading to the insufficient understanding of factuality metrics’ relative advantages and their applicability. In this paper, we present an adversarial meta-evaluation methodology that allows us to (i) diagnose the fine-grained strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets, (ii) search for directions for further improvement by data augmentation. Our observations from this work motivate us to propose several calls for future research. We make all codes, diagnostic test datasets, trained factuality models available: https://github.com/zide05/AdvFact .

## introduction with the rapid development of neural networks in text summarization @xcite @xcite @xcite @xcite , especially the use of contextualized pre-trained models @xcite , the state-of-the-art performance, measured by automated metrics such as rouge @xcite and bertscore @xcite ) has been constantly updated. however, although these systems can generate informative, and fluent summaries, they suffer from the problem of making factual errors-generating incorrect facts that can not be supported by the source document @xcite . among this background, a large body of recent works @xcite @xcite are trying to search for new automated metrics that can assess the factuality of generated summaries due to the fact that existing metrics (e.g., rouge) are not correlated well with factual consistency @xcite . generally, the process of designing these evaluation metrics w.r.t factuality is commonly formulated into different forms of nlp tasks, ranging from text entailment @xcite at sentence level or more fine-grained level @xcite to question answering @xcite . improving the understanding of these factuality metrics with diverse paradigms is critical for further metric improvement. however, the evaluation methodologies of factuality metrics are limited in their opacity-they are opaque to their results, which are usually holistic scores (e.g., accuracy) and not interpretable. specifically, different from traditional non-learnable metrics like rouge, whose scores are relatively straightforward to interpret, e.g., lower rouge-2 recall implies fewer bigrams from reference summaries are covered by generated summaries, there are diverse factors that could lead to lower score of factuality metrics (e.g., entity replacement, number inference). however, most of existing meta-evaluation strategies fail to tell (i) which types of factual errors the metric evaluated at hand are better at identifying, (ii) on which categories the error recognition ability of factuality metrics can not be well generalized. as a result, (1) the relative advantages between a better-and worse-performing systems w.r.t factuality are unclear. (2) the lack of understanding of factuality metrics' applicability reduces their reliability, and users may take the risk of over-estimating their generalization ability so as to apply them to inappropriate evaluation samples. (3) it's unclear how to improve the metric further. thus instead of further pursuing a new method, we take a step back to understand the shortcomings of existing metrics. we present an adversarial meta-evaluation framework which can perform fine-grained evaluation of factuality metrics. methodologically, we (i) first conduct error analysis of existing state-of-the-art factuality metrics, (ii) define effective adversarial transformations based on the results of error analysis. we (iii) construct diagnostic examples by applying adversarial transformations to test datasets with different distributions and then diagnose existing top-scoring factuality metrics. (iv) we finally show that, the technique of data augmentation, driven by adversarial transformations, can increase the diversity of training samples, making factuality metrics more robust and reliable. our contributions can be summarized as follows: (1) we figure out several representative errors made by the existing top-performing factuality metrics ( §4.2), inspiring the direction for further improvement. (2) we propose effective adversarial transformations that can either be applied to test set for model diagnosis ( §5) or applied to training set for data augmentation ( §6.2), by which we further improve the performance of current checkers. (3) we propose a fine-grained meta-evaluation methodology for factuality metrics and re-evaluate existing top-performing metrics to assess their relative strengths and weaknesses. (4) we call for a more fine-grained and interpretable meta-evaluation of factuality metrics for future research. as a first step, we released our constructed diagnostic test sets with various characteristics, as well as augmented training data and more robust factuality metrics. ## related work factuality in text summarization recent studies on factuality of text generation revolve around metric design and system optimization. regarding the metric perspective, researchers formulate the design of automated metrics w.r.t factuality as different problems: text entailment over sequential @xcite or tree @xcite structures; question answering @xcite and sequence labeling @xcite . concurrent to our work, @xcite constructs human annotated test sets for factuality metrics while using a different typology. additionally, their method is difficult to be used as automatic data augmentation. other works aim to learn factuality-aware summarization systems, which can be achieved by leveraging open information extraction and dependency parsing @xcite . @xcite explore how factuality metrics are influenced by domain shift and conclude that out-of-domain systems can even surpass in-domain systems in terms of factuality and factuality checkers like factcc is limited in predictive power of positive samples. adversarial evaluation of nlp systems adversarial evaluation has been extensively explored in many nlp tasks recently. the adversarial challenge sets have been introduced into tasks of natural language inference @xcite question answering @xcite , machine translation @xcite and language model (marvin and linzen, 2018) to examine system drawbacks. more recently, @xcite introduces the concept of "contrast set" and proposes to use it to measure the generalization of different nlp systems. instead of adversarially evaluate an nlp system, we perform an adversarial metaevaluation of evaluation metrics. meta-evaluation for automated metrics metaevaluation aims to evaluate the reliability of automated metrics based on their correlation with human judgments @xcite @xcite . most existing works perform meta-evaluation on metrics that measure semantic equivalence, such as rouge @xcite and bertscore @xcite . @xcite more recently propose bartscore and meta evaluate it on multiple evaluation perspectives. by contrast, in this paper, we focus on the evaluation of factuality metrics using our constructed diagnostic test sets. concurrent with our work, @xcite durrett (2021b); pagnoni et al. (2021b) also look into the error patterns of existing factuality checkers. 1 3 preliminaries ## construction of diagnostic set it is not realistic to produce large scale human annotated test sets with multiple error reasons observed above. as a consequence, former work @xcite and @xcite construct diagnostic test sets automatically. in this section, we first introduce automatic rule-based transformation methods based on error analysis ( §5.1). then we construct 24 diagnostic test sets based on three types of baseline test sets. ## implications and future directions in this paper, we present an adversarial metaevaluation methodology driven by our fine-grained analysis, which not only allows us to re-evaluate existing top-performing factuality metrics, diagnosing their limitations, but also instructs us to further improve current metrics by data augmentation. based on what we have explored and observed in this work, we suggest following potentially promising future directions: models base advbase ref advref factcc origin (100 m) × × × factccsub sub (50 m) × × × factcc adv sub sub (50 m) × × factcc ref sub sub (50 m) × factcc ref adv sub sub (50 m)