C limate C heck2025: Multi-Stage Retrieval Meets LLM s for Automated Scientfic Fact-Checking (2025)

Abstract: Misinformation on social media poses significant risks, particularly when it concerns critical scientific issues such as climate change. One promising direction for mitigation is the development of automated fact-checking systems that verify claims against authoritative scientific sources. In this work, we present our solution to the ClimateCheck2025 shared task, which involves retrieving and classifying scientific abstracts as evidence for or against given claims. Our system is built around a multi-stage hybrid retrieval pipeline that integrates lexical, sparse neural, and dense neural retrievers, followed by cross-encoder and large language model (LLM)-based reranking stages. For stance classification, we employ prompting strategies with LLMs to determine whether a retrieved abstract supports, refutes, or provides no evidence for a given claim. Our approach achieves the second-highest overall score across both subtasks of the benchmark and significantly surpasses the official baseline by 53.79% on average across Recall@2, Recall@5, Recall@10, and B-Pref. Notably, we achieve state-of-the-art performance in Recall@2. These results highlight the effectiveness of combining structured retrieval architectures with the emergent reasoning capabilities of LLMs for scientific fact verification, especially in domains where reliable human annotation is scarce and timely intervention is essential.

## introduction the rapid proliferation of online misinformation, particularly in scientific, health, and policy contexts, has intensified the demand for reliable automated fact-checking systems @xcite . these systems aim to assess the veracity of natural language claims by retrieving and evaluating relevant evidence from large text corpora. this process hinges on two core challenges: (1) retrieving relevant information from vast knowledge sources, and (2) determining whether the retrieved content supports, refutes, or fails to inform the claim. traditional keyword-based retrieval methods often struggle with these tasks, especially in domains requiring deep semantic understanding or domainspecific reasoning @xcite . recent advances in neural retrievers and large language models (llms) have improved retrieval and reasoning capabilities across diverse topics @xcite @xcite . nonetheless, integrating high-recall retrieval with robust, claim-sensitive reasoning remains a key bottleneck -particularly in scientific domains, where evidence is often sparse, nuanced, and hedged @xcite . in this paper, we present our system for the cli-matecheck2025 shared task @xcite , which consists of two subtasks: (1) for each climate-related claim extracted from social media, retrieve the top-10 most relevant abstracts from a corpus of nearly 400,000 scientific abstracts, and (2) classify each claim-abstract pair as supports, refutes, or not enough information (nei). our retrieval pipeline (subtask 1) is a threestage architecture. first, we combine bm25 @xcite , a fine-tuned dense retriever, and a sparse neural retriever using reciprocal rank fusion (rrf) @xcite . second, we train a cross-encoder reranker with a two-phase hard negative mining strategy, leveraging both model uncertainty and relevance judgments. finally, we apply an adapted rankgpt @xcite , prompting an llm in a fewshot setting to rerank the top candidates using permutation-based generation informed by crossencoder scores. for evidence classification (subtask 2), we eval-uate zero-and few-shot llm prompting, and finetune transformer-based models for multi-class classification of claim-abstract pairs. our key contributions are: ## related work automated fact-checking aims to assess the veracity of claims using evidence, a task traditionally performed by human experts but increasingly addressed with automated methods due to scalability concerns @xcite . numerous datasets have been developed to support this research. general-domain resources include fever (wikipedia-based claims) @xcite , vitaminc (contrastive evidence) @xcite , liar @xcite , and multifc (realworld political/media claims) @xcite . mumin further expands this scope to multilingual, multimodal misinformation on social media (nielsen and mcconville, 2022). scientific fact-checking, a more specialized subfield, introduces challenges such as complex language, evolving knowledge, and domain-specific reasoning. key datasets include scifact (wadden et al., 2020) (scientific claims and abstracts), healthver @xcite and covid-fact @xcite (biomedical), and climat-eviz (climate science) @xcite . these corpora underscore the risks of domain-specific misinformation, from harmful medical decisions @xcite to distorted climate discourse @xcite . the fact-checking process is typically modeled as a pipeline: (1) claim detection @xcite , (2) check worthiness estimation @xcite , (3) document retrieval @xcite , (4) claim verification via natural language inference (nli) @xcite . some systems also generate explanations, though these face challenges with hallucination @xcite . our work focuses on document retrieval and claim verification. for retrieval, sparse methods such as bm25 use lexical matching, while dense methods, such as dense passage retrieval @xcite , leverage neural encoders for semantic similarity. hybrid systems combining both have shown improved performance @xcite , and retrieval-augmented generation (rag) models further integrate retrieval with generation for grounded responses @xcite . claim verification is often framed as an entailment task, with transformer-based models, such as bert @xcite , fine-tuned to classify claim-evidence pairs as support, refute, or neutral @xcite . prompt-based methods using llms offer zero-shot alternatives, though performance varies across models and prompt designs @xcite . despite advances, scientific fact-checking remains challenging due to long-context reasoning, subtle hedging, contradictory evidence, and the need for up-to-date knowledge. llms have shown promise as rerankers or classifiers, but often lag behind supervised models in consistency and interpretability @xcite . in this work, we investigate how traditional retrieval methods can be effectively integrated with llms to leverage their complementary strengths. ## dataset the climatecheck dataset @xcite was prepared by the task's organizers and comprises (i) claims sourced from clima-convo @xcite , debagree-ment @xcite , climate-fever @xcite , multifc @xcite , and climatefeedback foot_1 , including both real and synthetically generated social media-style content, (ii) a corpus of scientific abstracts from openalex foot_2 and s2orc @xcite ## methodology below, we describe our multi-stage pipeline for scientific fact-checking, summarized in figure 1 , and our technical implementation details. ## conclusion scientific fact verification poses unique challenges due to complex domain language and the need for precise evidence interpretation. in this work, we introduced a multi-stage retrieval and classification pipeline tailored to these challenges, integrating hybrid retrieval methods, cross-encoder reranking, and llm-based reasoning modules. our experiments on the climatecheck benchmark demonstrate consistent improvements across all retrieval metrics, with each additional component -especially llm-based reranking -contributing meaningfully to performance. in the classification subtask, prompting strategies based on llms outperformed traditional fine-tuned models, even when the latter were trained on task-specific human annotations. these findings highlight the flexibility and effectiveness of instruction-tuned llms for complex scientific reasoning tasks, especially in data-scarce or rapidly evolving domains. overall, our work underscores the importance of combining structured retrieval pipelines with the emergent reasoning abilities of llms. future work could explore more tightly integrated retrieval-generation models, few-shot active learning for stance classification, and methods for im-proving the interpretability and trustworthiness of llm-based decisions in scientific verification contexts.