AMRF act: Enhancing Summarization Factuality Evaluation with AMR -Driven Negative Samples Generation (2024)

Abstract: Ensuring factual consistency is crucial for natural language generation tasks, particularly in abstractive summarization, where preserving the integrity of information is paramount. Prior works on evaluating factual consistency of summarization often take the entailment-based approaches that first generate perturbed (factual inconsistent) summaries and then train a classifier on the generated data to detect the factually inconsistencies during testing time. However, previous approaches generating perturbed summaries are either of low coherence or lack error-type coverage. To address these issues, we propose AMRFact, a framework that generates perturbed summaries using Abstract Meaning Representations (AMRs). Our approach parses factually consistent summaries into AMR graphs and injects controlled factual inconsistencies to create negative examples, allowing for coherent factually inconsistent summaries to be generated with high error-type coverage. Additionally, we present a data selection module NegFilter based on natural language inference and BARTScore to ensure the quality of the generated negative samples. Experimental results demonstrate our approach significantly outperforms previous systems on the AggreFact-SOTA benchmark, showcasing its efficacy in evaluating factuality of abstractive summarization.

## introduction recent advances in text summarization, driven by pre-trained language models, have enabled the generation of coherent abstractive summaries @xcite @xcite . however, studies have shown that these generated summaries can be factually inconsistent with the source documents @xcite @xcite us president donald trump has said he will consider fire special counsel robert mueller, who is investigating alleged russian interference in the us election. ## abstract meaning representations introduced by @xcite , amr is a representation language that effectively captures the essence of texts by encoding abstract-level semantic information, including named entities, negations, coreferences, and modalities. this intrinsic capability positions amr as a critical asset for various semantic-related nlp tasks, such as summarization @xcite and machine translation @xcite . in our study, we harness the potential of amr in the summarization factuality evaluation task by perturbing the graphs of factually consistent summaries. each perturbation reflects a factual error in summarization. in amr graphs, nodes symbolize entities and concepts, connected through different relational edges. figure 1 provides an example of an amr graph for a summary. ## amrfact in amrfact, we start by parsing reference summaries into amr graphs and then introduce factual errors into these graphs. subsequently, we backtranslate the perturbed amr graphs into text and use data filtering to acquire high-quality negative training samples. finally, we combine the positive samples (reference summaries) with the negatives to train a classifier, which serves as our factuality evaluation metric. more specifically, the process of generating factually inconsistent summaries involves two primary states: introducing summarylevel perturbations based on amr and invalid negative data filtering. they are discussed in more detail in the following sections dedicated to explaining how factual errors are introduces ( §3.1) and the criteria for choosing valid negative examples ( §3.2). ## conclusion we introduce a novel framework, amrfact, which leverages abstract meaning representations (amrs) to generate factually inconsistent summaries for summarization factuality evaluation. our method addresses the prevalent issues of low coherence and insufficient error-type coverage observed in prior entailment-based approaches. by parsing factually consistent reference summaries into amr graphs and injecting controlled factual inconsistencies, we successfully create coherent but factually inconsistent summaries with broad error-type coverage. the introduction of the neg-filter module further enhances the quality of the generated negative samples. the effectiveness and generalizability of this approach are validated through its state-of-the-art performance on the aggrefact-ftsota dataset. our contributions not only advance the field in generating highquality, factually inconsistent summaries but also provide a scalable and efficient solution for enhancing the factuality evaluation of summarization systems. the results underscore the potential of amrbased perturbations in improving the integrity and reliability of natural language generation.