Contrastive Learning to Improve Retrieval for Real-World Fact Checking (2024)

Abstract: Recent work on fact-checking addresses a realistic setting where models incorporate evidence retrieved from the web to decide the veracity of claims. A bottleneck in this pipeline is in retrieving relevant evidence: traditional methods may surface documents directly related to a claim, but fact-checking complex claims requires more inferences. For instance, a document about how a vaccine was developed is relevant to addressing claims about what it might contain, even if it does not address them directly. We present Contrastive Fact-Checking Reranker (CFR), an improved retriever for this setting. By leveraging the AVeriTeC dataset, which annotates subquestions for claims with human written answers from evidence documents, we fine-tune Contriever with a contrastive objective based on multiple training signals, including distillation from GPT-4, evaluating subquestion answers, and gold labels in the dataset. We evaluate our model on both retrieval and end-to-end veracity judgments about claims. On the AVeriTeC dataset, we find a 6% improvement in veracity classification accuracy. We also show our gains can be transferred to FEVER, ClaimDecomp, HotpotQA, and a synthetic dataset requiring retrievers to make inferences.

## introduction retrieval-augmented generation (rag) systems are now widely used across nlp applications including question answering @xcite @xcite ) and text generation @xcite , but one particular application of interest is fact-checking. while older fact-checking systems would often not consider evidence at all @xcite or consider oracle evidence @xcite , the real fact-checking task involves finding evidence to support or refute complex claims in the wild @xcite claim us president donald trump following his positive diagnosis of covid-19, was given treatment developed from the use of fetal tissue, made by the company regeneron. ## methodology we consider a setting following work in averitec and claimdecomp @xcite . we assume we are given a collection of claims (c 1 , . . . , c n ). for claim c i , we define q ij as the jth subquestion for the ith claim in the dataset and a g ij define its answer. we also assume access to a document set d(c i , q ij ) for each subquestion, created by querying bing with c i appended to q ij and scrape the top-k articles to form a document corpus. each document d is a 200 token span gathered from the scraped articles. the title of the document is prepended to the start of each document. the dataset also comes with a gold article which contains the gold answer. like the bing-retrieved documents, it is chunked into 200 token span documents {d g } and added to d(c i , q ij ). we refer to documents belonging to these articles as gold. given a query @xmath0; q ij ] and a document d @xmath1, we want to generate embeddings in r e using an encoder network (e.g. contriever). let h y , h d i denote the representations of y and d i . then we define our scoring function the goal is to optimize our encoder via f to rank documents for answering questions in-context with the claim above topically relevant documents that do not ultimately contain information for an answer. we choose to optimize this for downstream veracity classification accuracy. we also track more upstream metrics such as using a relevance score for the top document or measuring how close its extracted answer matches the gold answer. ## experimental setup we evaluate contriever fine-tuned on the supervision signals outlined in section 3. the datasets selected for evaluation, namely averitec @xcite , claimdecomp @xcite , fever @xcite , and hot-potqa @xcite , encompass a wide range of scenarios for document retrieval. for evaluation, a random subset of 200 answerable examples (subquestions contain an answer) were selected from each of these not overlapping with the training sets. ## retriever reasoning capabilities our hypothesis about our contrastive training was that it would impart a greater ability for our retriever to "reason" about content rather than directly locating an answer. we conduct an additional study of whether our retriever can exhibit basic 1-hop reasoning capabilities via a synthetic data experiment. we construct positive and negative documents where the positive documents do not directly state the answer, similar to what we found in several averitec examples. ## conclusion this work presents an improved retrieval system, cfr, for fact-checking complex claims. we present two supervision signals for finetuning retrievers under a contrastive objective, and their integration results in improved downstream veracity classification. furthermore, cfr is able to improve retrieval in settings where inferences are required to identify the correct documents. the gains found in this paper encourage explorations into improving retrieval for fact-checking, as surfacing relevant information proved to be a hard task even for sota dense retrievers.