QAF act E val: Improved QA -Based Factual Consistency Evaluation for Summarization (2022)

Abstract: Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.

## introduction text summarization aims to compress long document(s) into a short and fluent form that preserves salient information. the field has benefited from the application of pretrained methods @xcite @xcite . however, state-of-the-art models are not always factually consistent with the source documents they are conditioned on @xcite . thus, determining the factual consistency of a summary remains an essential task. recent metrics for summarization factual consistency can be broadly split into two categories: 1) entailment-based metrics that determine whether the content in the summary is entailed by the input document @xcite ## related work evaluating factual consistency within entailment-based factual consistency evaluation, @xcite propose the task of ranking summary pairs for factual consistency based on entailment models, while @xcite explore factual consistency classification jointly with source support or contradiction span extrac-tion. other work on entailment-based metrics has examined input granularity @xcite , trained on adversarial datasets @xcite , and explored entailment-based models as the backbone of others metrics such as bertscore @xcite as in @xcite . metric comparisons, however, were often conducted on isolated datasets. @xcite unify work in entailment-based metrics for factual consistency, showing the effect of granularity, base models, and other hyperparameter choices. this work also proposes a learned metric built on top of the output of an entailment model, with parameters fine-tuned on synthetic data. while this work fills a gap in the use of entailment-based metrics for factual consistency, our work analogously unifies qa-based metrics for factual consistency and proposes to combine entailment and qa-based metrics in a single learned metric. qa-based evaluation metrics have received attention for summary quality dimensions beyond factual consistency @xcite @xcite . recent work has shown that qa-based metrics better measure the overlap of information units for determining summary relevance over embedding-based metrics @xcite , further driving our study of qa-based metrics for factual consistency. while several qa-based metrics with similar structures have been applied for factual consistency, @xcite @xcite , they differ in their underlying answer selection, question generation, question answering, and answer overlap components, reporting different performances. we perform a comprehensive evaluation of qa-based metric components and propose improved model components for the task of answer overlap and question filtering. summarization benchmarking a recent line of work aims to take stock of the current state of summarization models and progress, both within factual consistency and across summarization more broadly. @xcite note biases and failure modes of abstractive summarization models, while other work analyzes and collects annotations over the output of recent summarization models across multiple dimensions, including factual consistency @xcite @xcite . @xcite propose a typology of errors found in summarization models, while @xcite propose a framework for meta-evaluation of factual consistency metrics. @xcite propose to combine recent work in factual consistency evaluation for summarization through a single benchmark. our work directly makes use of this benchmark while emphasizing qa-based metrics. we also include correlation analysis for a more comprehensive understanding of current factual consistency metrics. ## evaluation metrics in this section, we introduce the factual consistency metrics studied, which we divide into entailment metrics, qa-based metrics, and learned metrics. ## methodology we present the datasets explored for binary classification and correlation analyses. we also describe settings for reporting ablation and final results. ## results in this section, we first study the effects of model component choices on qafacteval . we then compare metric results across both the summac binary classification task and correlation analysis. ## conclusion in this work, we demonstrated that qa-based metrics, when its components are properly optimized, outperform entailment-based metrics on a comprehensive factual consistency evaluation benchmark. we identify question generation and answerability detection as key components for improving qabased metrics in future work. furthermore, we show that entailment and qa-based metrics offer complementary signals through a combined metric that achieves state-of-the-art performance on this benchmark. we believe that our work lays the foundation for future work in qa-based metrics for factual consistency by offering a fairer comparison to other metrics across datasets and settings.