F act KB : Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge (2023)

Abstract: Evaluating the factual consistency of automatically generated summaries is essential for the progress and adoption of reliable summarization systems. Despite recent advances, existing factuality evaluation models are not robust, being especially prone to entity and relation errors in new domains. We propose FactKB—a simple new approach to factuality evaluation that is generalizable across domains, in particular with respect to entities and relations. FactKB is based on language models pretrained using facts extracted from external knowledge bases. We introduce three types of complementary factuality pretraining objectives based on entity-specific facts, facts extracted from auxiliary knowledge about entities, and facts constructed compositionally through knowledge base walks. The resulting factuality evaluation model achieves state-of-the-art performance on two in-domain news summarization benchmarks as well as on three out-of-domain scientific literature datasets. Further analysis of FactKB shows improved ability to detect erroneous entities and relations in summaries and is robust and easily generalizable across domains.

## introduction generating factually accurate document summaries in addition to fluent and informative ones is critical to the adoption of summarization models @xcite . however, evaluating the factual consistency of summaries is still challenging, especially in specialized domains like scientific or legal @xcite @xcite . the key reason is that the majority of existing approaches employ neural classifiers trained on synthetic data constructed from a relatively small set of documents @xcite . these factuality figure 1 : existing factuality models struggle to identify semantic frame errors encompassing entities and relations. in the example, they fail to identify an error in the generated summary about who was hit by the stone. to address these issues, we develop a new factuality evaluation model with improved factual knowledge representation, specifically focusing on entities and relations. entity-oriented pretraining objectives have been shown to improve qa and reasoning tasks @xcite ; we thus hypothesize that similar objectives can aid factuality evaluation in better detecting semantic frame errors in generated summaries. we propose factkb, a novel factuality evaluation model built upon language models (lms) augmented with factual knowledge ( §2). the lms are pretrained with knowledge-focused objectives using text synthesized from external knowledge bases (kbs) which store high-quality facts about entities and relations. we propose three types of complementary pretraining strategies: (1) entity wiki, with a focus on improving entity understanding; (2) evidence extraction, with a focus on incorporating supporting evidence from surrounding context; and (3) knowledge walks, with a focus on augmenting compositional reasoning about entities. for factuality evaluation, we first pretrain a language model using these three entity-centric pretraining strategies, and then fine-tune the enhanced lm on a factual error detection dataset. we evaluate factkb's correlation with human factuality judgments across three settings ( §3). in in-domain (news) summarization, factkb significantly outperforms baselines by 2-7 balanced accuracy (bacc) points on the factcollect dataset @xcite and 10-12 correlation points on the frank benchmark @xcite , particularly showing marked improvements in semantic frame errors. in out-of-domain experiments, factkb consistently outperforms existing approaches by 3-5 bacc points on three datasets in biomedical and scientific domains @xcite @xcite , demonstrating stronger generalizability to unseen documents in new domains. further analysis shows that factkb is compatible with different lms and kbs while presenting a lightweight and easy-to-use approach to factuality evaluation. code, data, and trained factuality evaluation models are publicly available. ## factkb methodology factkb aims to improve the robustness and generalizability of factuality evaluation by a simple factuality pretraining, which improves entity and relation representations in lms. we first propose three pretraining strategies ( §2.1). we then describe the training process to ( 1 ) pretrain an lm using the proposed strategies and (2) fine-tune the factenhanced lm on a factuality error detection dataset, resulting in factkb ( §2.2). figure 2 presents an overview of our approach. ## related work factuality evaluation recent advances in text summarization have presented models and systems that are capable of generating increasingly fluent, controllable, and informative summaries of documents @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . however, they suffer from hallucination and might not be factually faithful towards the source document @xcite @xcite @xcite , leading to increased research in factuality evaluation. qa-based approaches @xcite @xcite attempt to generate and answer questions based on summaries and documents and judge the factuality by comparing answers. later approaches are generally entailment-based @xcite @xcite @xcite , proposing to classify (summary, document) pairs into factual or non-factual labels. among them, factcc @xcite ) is one of the first entailment-based metrics and is trained on synthetic data; dae @xcite proposes to leverage the dependency structure of summaries and documents; factgraph @xcite builds abstract meaning representation graphs and adopts graph neural networks for joint representation learning along the textual content. in addition, hypothesis re-ranking (garneau and lamontagne, 2021), counterfactual estimation @xcite , nli models @xcite , phrase-level localization @xcite , and weighting facts in the source document @xcite were also explored in factuality evaluation. moving beyond a binary concept of factuality, frank @xcite promotes a fine-grained understanding of factuality and proposes a typology of factuality errors. inspired by its analysis that semantic frame errors, errors regarding entities and relations, are a major source of factuality errors yet under-explored by existing factuality metrics, we propose factkb to leverage external kbs for factuality pretraining and help enforce better factuality towards entities and relations discussed in summaries and documents. knowledge bases in nlp knowledge base is a standard format for structured knowledge representation. one application of kbs in nlp is to inject knowledge and augment lms, where different approaches focused aspects such as pretraining @xcite @xcite , document graphs @xcite , kb structure (yasunaga ## conclusion we propose factkb, a simple and novel approach to factuality evaluation using language models pretrained on facts from external kbs to improve entity and relation representations. specifically, we leverage kbs to construct three factuality pretraining objectives: entity wiki, evidence extraction, and knowledge walk. factkb pretrains an lm using the three objectives and fine-tunes the resulting model on factuality evaluation datasets. extensive experiments demonstrate that factkb advances the state-of-the-art in both in-domain and out-ofdomain factuality evaluation, better correlates with human factuality annotations, and better detects semantic frame errors. factkb presents an easy-touse and generalizable factuality metric, facilitating research on factually-consistent summarization.