Retrieval Augmented Fact Verification by Synthesizing Contrastive Arguments (2024)

Abstract: The rapid propagation of misinformation poses substantial risks to public interest. To combat misinformation, large language models (LLMs) are adapted to automatically verify claim credibility. Nevertheless, existing methods heavily rely on the embedded knowledge within LLMs and / or black-box APIs for evidence collection, leading to subpar performance with smaller LLMs or upon unreliable context. In this paper, we propose retrieval augmented fact verification through the synthesis of contrasting arguments (RAFTS). Upon input claims, RAFTS starts with evidence retrieval, where we design a retrieval pipeline to collect and re-rank relevant documents from verifiable sources. Then, RAFTS forms contrastive arguments (i.e., supporting or refuting) conditioned on the retrieved evidence. In addition, RAFTS leverages an embedding model to identify informative demonstrations, followed by in-context prompting to generate the prediction and explanation. Our method effectively retrieves relevant documents as evidence and evaluates arguments from varying perspectives, incorporating nuanced information for fine-grained decision-making. Combined with informative in-context examples as prior, RAFTS achieves significant improvements to supervised and LLM baselines without complex prompts. We demonstrate the effectiveness of our method through extensive experiments, where RAFTS can outperform GPT-based methods with a significantly smaller 7B LLM.

## introduction as the scope of social media and digital forums continue to expand, increasing amount of misinformation has been observed across multiple platforms (e.g., twitter), posing risks to public interest @xcite . therefore, fact-checking methods are proposed to prevent the spreading of false information before it leads to severe consequences @xcite @xcite . for example, online fact-checking services (e.g., snopes foot_1 ) employ professional factcheckers to identify instances of misinformation. nevertheless, human fact-checking involves a significant amount of manual work, proving to be less efficient confronted with the vast volume of misinformation, particularly as it evolves and spreads online @xcite . to perform fact-checking at scale, automated methods have emerged by leveraging large language models (llms) @xcite @xcite . for example, rarg proposes to train and align llms for generating faithful explanations upon detected misinformation @xcite . despite their effectiveness, these methods typically require extensive training data and may demonstrate performance deterioration upon domain / concept shifts @xcite @xcite . moreover, many models are unaware of external evidence / knowledge and must be frequently re-trained to incorporate up-to-date domain knowledge for accurate factverification @xcite @xcite . as a solution, evidence-based fact-checking methods are proposed to collect evidence (e.g., documents, graphs etc.), followed by extracting relevant information and assessing the credibility of input claims through llms @xcite @xcite @xcite @xcite . an example is folk, which leverages first-order-logic to construct sub-claims and perform question answering-based verification to generate predictions and explanations @xcite . yet current approaches rely on the assumption that input claims can be decomposed input claim (e.g., facebook has a policy against users' posting the text of the lord's prayer.) document retrieval fact chechking reliable source of documents retrieved documents output prediction: false explanation: facebook's community standards make no mention of prohibiting the posting of the lord's prayer or other forms of religious expression. only hate speech directed at a person's religious affiliation is disallowed under facebook's standards. into a series of predicates (i.e., sub-claims) through complex prompts. moreover, they depend on the embedded knowledge within llms and / or blackbox apis (e.g., serpapi 3 ) to collect external information, leading to subpar performance with smaller llms or provided with unreliable evidence @xcite . consequently, we consider a retrieval augmented generation (rag) framework designed to extract relevant information from reliable documents (i.e., wikipedia, scholarly articles etc.), where the extracted information can be used as supporting facts to assess the claim credibility through llms. that is, given the input statement, our first objective is to retrieve (and optionally re-rank) relevant documents among an extensive collection of documents from verifiable sources. subsequently, we utilize the retrieved documents to fact-check the input claim, aiming to either confirm the input or uncover opposing information that identifies misinformation. our framework is visually illustrated in figure 1 , where the rag-based fact verification framework retrieves relevant documents, and then generates both prediction and explanation regarding the validity of the input statement. to this end, we propose retrieval augmented fact verification through the synthesis of contrastive arguments (rafts), which effectively retrieves relevant documents and performs few-shot fact verification using pretrained llms. rafts is structured into three components: (1) demonstration retrieval, where informative in-context examples are collected to improve fact-checking performance; (2) document retrieval, in which we design a retrieve and re-rank pipeline to accurately identify relevant documents for input claims; and we summarize our contributions: ## preliminary we consider the following problem setup: given input claim x (with label y) and k-shot demonstrations {(x i , y i )} k @xmath0, we aim to: (1) retrieve a set of m documents {d i } m @xmath1; and learning: our retrieval pipeline f retrieve is parameterized by θ. to learn f retrieve , we maximize the score of the sampled input-document pair (x, d). meanwhile, the fact-checking model f check (i.e., pretrained llm) remains unchanged to minimize training expenses. to optimize fact-checking performance of f check , we employ a lightweight embedding model f embed to select informative incontext demonstrations {(x i , y i )} k @xmath2, we elaborate the details in section 4.1. ## limitations despite introducing rafts for retrieval augmented fact verification, we have not discussed the setting in which the document retrieval domain significantly differs from the fact-checking domain (e.g., using wikipedia documents to factcheck covid misinformation), which can cause performance deterioration for domain-generalized applications. furthermore, we have not examined the robustness and reliability of our example retrieval and document retrieval, which could unlock additional improvements for fact verification. consequently, we plan to explore a more generalized and domain-adaptive solution for retrieval augmented fact verification as future work.