P ivot FEC : Enhancing Few-shot Factual Error Correction with a Pivot Task Approach using Large Language Models (2023)

Abstract: Factual Error Correction (FEC) aims to rectify false claims by making minimal revisions to align them more accurately with supporting evidence. However, the lack of datasets containing false claims and their corresponding corrections has impeded progress in this field. Existing distantly supervised models typically employ the mask-then-correct paradigm, where a masker identifies problematic spans in false claims, followed by a corrector to predict the masked portions. Unfortunately, accurately identifying errors in claims is challenging, leading to issues like over-erasure and incorrect masking. To overcome these challenges, we present PivotFEC, a method that enhances few-shot FEC with a pivot task approach using large language models (LLMs). Specifically, we introduce a pivot task called factual error injection, which leverages LLMs (e.g., ChatGPT) to intentionally generate text containing factual errors under few-shot settings; then, the generated text with factual errors can be used to train the FEC corrector. Our experiments on a public dataset demonstrate the effectiveness of PivotFEC in two significant ways: Firstly, it improves the widely-adopted SARI metrics by 11.3 compared to the best-performing distantly supervised methods. Secondly, it outperforms its few-shot counterpart (i.e., LLMs are directly used to solve FEC) by 7.9 points in SARI, validating the efficacy of our proposed pivot task.

## introduction chatgpt is an artificial intelligence chatbot released by openai on november 30, 2022 and built upon on the company's generative pretrained transformer (gpt) series of large language (llms) @xcite . since its launch, chatgpt has garnered significant global attention due to its comprehensive and eloquent responses across various knowledge domains. within just two months, by january 2023, †corresponding authors. the most straightforward way to develop fec systems is by fine-tuning pre-trained models, such as bart @xcite and t5 @xcite , on parallel data, consisting of false claims along with their corresponding corrections. nevertheless, the availability of such paired data is restricted due to the tremendous labor and time required for human annotations. to overcome the data scarcity, researchers @xcite @xcite ) make use of distant supervision from the fact verification dataset, fever @xcite . fever is a large resource consisting of claims paired with evidence from wikipedia, where each instance is labeled as either supported or re-futed based on whether the claim is supported or refuted by the corresponding evidence. existing distantly supervised models typically follow the mask-then-correct approach @xcite . concretely, the fact verification classifier (fvc), trained on fever, acts as the masker, designed to find problematic spans within false claims. the token-level explanations @xcite of fvc are usually exploited as masks. the corrector is trained on the supported data from fever, with the objective of restoring/generating the original/correct claim based on the masked claim and evidence, during training/inference. furthermore, page: 2013 nba draft context: the 2013 nba draft was held on june 27, 2013, at barclays center in brooklyn, new york. ## problem statement factual error correction aims to revise the factual errors in claim c with minimal edits and generate a revised claim c ′ based on the provided evidence e. c ′ should be grammatical, supported by the evidence, and correct the factual errors in c. ## preliminary in this section, we will introduce in-context learning and how to solve fec using llms with incontext few-shot learning via prompting. ## approach in this section, we will first provide an overview of pivotfec in §4.1. as illustrated in figure 2 , our method comprises three main steps. we will begin by introducing the fei task and demonstrating the utilization of llms to address the fei task in §4.2. next, we will present the process of gathering synthetic paired data for the fec task in §4.3. finally, we will demonstrate the training of the corrector using the synthetic data in §4.4. ## conclusion in this paper, we present pivotfec, which introduces a pivot task, factual error injection, to improve factual error correction. specifically, we first intentionally introduce factual errors into correct claims using llms under the few-shot setting. by doing so, we can obtain enough synthetic paired data for fec, consisting of correct claims paired with their corresponding false claims, which will be used to train the fec corrector. as a result, pivot-fec demonstrates a significant improvement over previous distantly supervised baselines, establishing a new sota performance on fecdata. furthermore, our approach significantly outperforms its few-shot counterpart, providing strong evidence for the effectiveness of the pivot task.