Zero-shot Faithful Factual Error Correction (2023)

Abstract: Faithfully correcting factual errors is critical for maintaining the integrity of textual knowledge bases and preventing hallucinations in sequence-to-sequence models. Drawing on humans’ ability to identify and correct factual errors, we present a zero-shot framework that formulates questions about input claims, looks for correct answers in the given evidence, and assesses the faithfulness of each correction based on its consistency with the evidence. Our zero-shot framework outperforms fully-supervised approaches, as demonstrated by experiments on the FEVER and SciFact datasets, where our outputs are shown to be more faithful. More importantly, the decomposability nature of our framework inherently provides interpretability. Additionally, to reveal the most suitable metrics for evaluating factual error corrections, we analyze the correlation between commonly used metrics with human judgments in terms of three different dimensions regarding intelligibility and faithfulness.

## introduction the task of correcting factual errors is in high demand and requires a significant amount of human effort. the english wikipedia serves as a notable case in point. it is continually updated by over 120k editors, with an average of around six factual edits made per minute foot_1 . using machines to correct factual errors could allow the articles to be updated with the most current information automatically. this process, due to its high speed, can help retain the integrity of the content and prevent the spread of false or misleading information. in addition, the hallucination issues have been shown to be a prime concern for neural models, ## proposed methods our framework, zerofec, faithfully corrects factual errors using question-answering and entailment. specifically, we represent the input claim c as question-answer pairs {(q 1 , a input claim question generation claim answer generation 1. a spanish comic book 2. night of the living dead ... claim answers generated questions 1. what is night of the living dead? 2. what is the name of the spanish comic book? ... evidence night of the living dead is a 1968 american independent horror film, directed by george a. romero, starring duane jones and judith o'dea … question answering 1. a horror film 2. la guairita ... evidence answers qa-to-claim correction scoring 1. night of the living dead is a horror film. 2. la guairita is a spanish comic book. ... ## conclusions and future work we have presented zerofec, a zero-shot framework that asks questions about an input claim and seeks answers from the given evidence to correct factual errors faithfully. the experimental results demonstrate the superiority of our approach over prior methods, including fully-supervised methods, as indicated by both automatic metrics and human evaluations. more importantly, the decomposability of zerofec naturally offers interpretability, as the questions and answers generated directly reflect which information units in the input claim are incorrect and why. furthermore, we reveal the most suitable metric for assessing faithfulness of factual error correction by analyzing the correlation between the reported automatic metrics and human judgments. for future work, we plan to extend our framework to faithfully correct misinformation in social media posts and news articles to inhibit the dissemination of false information. in addition, it may be meaningful to explore extending zero-shot factual error correction to multimedia task settings, such as identifying inconsistencies between chart and text @xcite .