F act G raph: Evaluating Factuality in Summarization with Semantic Graph Representations (2022)

Abstract: Despite recent improvements in abstractive summarization, most current approaches generate summaries that are not factually consistent with the source document, severely restricting their trust and usage in real-world applications. Recent works have shown promising improvements in factuality error identification using text or dependency arc entailments; however, they do not consider the entire semantic graph simultaneously. To this end, we propose FactGraph, a method that decomposes the document and the summary into structured meaning representations (MR), which are more suitable for factuality evaluation. MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form, and reducing data sparsity. FactGraph encodes such graphs using a graph encoder augmented with structure-aware adapters to capture interactions among the concepts based on the graph connectivity, along with text representations using an adapter-based text encoder. Experiments on different benchmarks for evaluating factuality show that FactGraph outperforms previous approaches by up to 15%. Furthermore, FactGraph improves performance on identifying content verifiability errors and better captures subsentence-level factual inconsistencies.

## introduction recent summarization approaches based on pretrained language models (lm) have established a new level of performance @xcite , generating summaries that are grammatically fluent and capable of combining salient parts of the source document. however, current models suffer from a severe limitation, generating summaries that are not factually consistent, that is, the content of the summary does not meet the in the face of such a challenge, recent works employ promising ideas such as question answering (qa) @xcite and weakly supervised approaches @xcite to assess factuality. another line of work explores dependency arc entailment to improve the localization of subsentence-level errors within generated summaries @xcite . however, these methods have a reduced correlation with human judgments and may not capture well semantic errors @xcite . one reason for the poor performance is the lack of good quality factuality training data. second, it is challenging to properly encode core semantic content from the document and summary @xcite and reason over salient pieces of information in order to assess the summary factuality. third, previous work @xcite treats semantic relations as isolated units, not simultaneously considering the entire semantic structure of both document and summary texts. to mitigate the above issues, we explore meaning representations (mr) as a form of content representation for factuality evaluation. we present factgraph, a novel graph-enhanced approach that incorporates core information from the document and the summary into the factuality model using graph-based mrs, which are more suitable for factuality evaluation: as shown in figure 1 , graphbased mrs capture semantic relations between entities, abstracting away from syntactic structure and producing a canonical representation of meaning. different from previous methods @xcite , fact-graph is a dual approach which encodes both text and graph modalities, better integrating linguistic knowledge and structured semantic knowledge. as shown in figure 2 , it is composed of parameterefficient text and graph encoders which share the same pretrained model and differ by their adapter weights @xcite . the texts from the document and summary are encoded using the adapter-based text encoder whereas the entire semantic structures that represent document and summary facts are used as input to the graph encoder augmented structure-aware adapters @xcite . the representations of the two modalities thus are combined to generate the factuality score. in particular, we explore abstract meaning representation (amr) @xcite , a semantic formalism that has received much research interest @xcite @xcite @xcite @xcite and has been shown to benefit downstream tasks such as spoken language understanding @xcite , machine translation @xcite , commonsense reasoning @xcite , and question answering @xcite . intuitively, amr provides important benefits: first, it encodes core concepts as it strives for a more logical and less syntactic representation, which has been shown to benefit text summarization @xcite @xcite . furthermore, amr captures semantics at a high level of abstraction explicitly modeling relations in the text and reducing the negative influence of diverse text surface variances with the same meaning. lastly, recent studies @xcite demonstrate that there is a trade-off between factuality and abstractiveness. structured semantic representations are potentially beneficial for reducing data sparsity and localizing generation errors in abstractive scenarios. figure 1 shows examples of (c) document and (d) summary amrs, where the summary amr is missing a crucial modifying node present in the document amr, which indicates a factual error in the summary. we consolidate a factuality dataset with human annotations derived from previous works @xcite @xcite . this dataset is constructed from the widely-used cnn/dm @xcite and xsum @xcite benchmarks. extensive experimental results demonstrate that factgraph achieves substantial improvements over previous approaches, improving factuality performance by up to 15% and correlation with human judgments by up to 10%, capturing more content verifiability errors and better classifying factuality in semantic relations. ## factgraph model we introduce factgraph, a method that employs semantic graph representations for factuality evaluation in text summarization, describing its intuition ( §3.3) and defining it formally ( §3.4). ## results and analysis we compare factgraph with different methods for factuality evaluation: two qa-based methods, namely qags @xcite and quals @xcite , and factcc @xcite . we fine-tune factcc using the training set, that is, it is trained on both synthetic data and factcollect. we call this approach factcc+. on synthetic data derived from this dataset. however, the factcc's performance does not transfer to xsum. factcc+ has a large increase in performance, especially on xsum, demonstrating the importance of human-annotated data for training improved factuality models. factgraph outperforms factcc+ by 2.4 bacc points in both subsets and by 10.3 bacc in xsum, even though factcc+ was pretrained on millions of synthetic examples. this indicates that considering semantic representations is beneficial for factuality evaluation and factgraph can be trained on a small number of annotated examples. pretraining structural adapters improves the performance on cnn/dm and xsum. finally, fact-graph's performance further improves when both structural and text adapters are pretrained, improving over factcc+ by 3.7 bacc points. foot_1 ## conclusion we presented factgraph, a graph-based approach to explicitly encode facts using meaning representations to identify factual errors in generated text. we provided an extensive evaluation of our approach and showed that it significantly improves results on different factuality benchmarks for summarization, indicating that structured semantic representations are beneficial to factuality evaluation. future work includes (i) exploring approaches to develop document-level semantic graphs @xcite , (ii) an explainable graph-based component to highlight hallucinations and (iii) to combine different meaning representations in order to capture distinct semantic aspects.