The C limate C heck Shared Task: Scientific Fact-Checking of Social Media Claims about Climate Change (2025)

Abstract: Misinformation in public discourse on global and significant issues like climate change is often facilitated through social media. However, current systems do not address fact-checking climate-related claims against trustworthy, evidence-based sources, such as scientific publications. We organised the ClimateCheck shared task at the 5th Scholarly Document Processing (SDP) Workshop, co-located with ACL 2025 in Vienna, Austria. The task featured two subtasks: 1. Abstracts retrieval given a claim, and 2. Claim verification based on the retrieved abstract. ClimateCheck had 27 registered users with active participation from 13 teams, ten of which submitted results for the first subtask and three for the second. The winning team achieved a Recall@10 score of 0.66 and a Binary Preference score of 0.49 for subtask I, and an F1 score of 0.73 for subtask II. Their method combined sparse retrieval using BM25, an ensemble of fine-tuned cross-encoder models using BGE-rerankers, and large language models for classification.

## introduction the widespread use of social media has transformed the way people engage with crucial global challenges such as climate change. while these platforms enable a public dialogue, they also fasttrack the spread of inaccurate and misleading information @xcite . recent work in natural language processing (nlp) offers promising advances in decoding and analysing complex discourse online @xcite . researchers have used methods to detect misinformation @xcite , extract scientific claims and entities @xcite ; hughes and song, 2024), and fact-check statements @xcite . at the same time, work on scholarly document processing has advanced methods for extracting and structuring scientific knowledge @xcite , making it easier to link it to public discourse. shared tasks are effective tools for mobilising the research community around challenging tasks, driving innovation and the development of stateof-the-art methods @xcite . previous shared tasks targeted fact-checking by retrieving relevant evidence for a given claim and classifying their relation. however, they mainly focused on non-scientific evidence corpora, e. g., wikipedia @xcite , or were limited to the biomedical domain @xcite . to the best of our knowledge, no previous effort has tackled the challenge of connecting claims posted online about climate change to credible scientific sources. to address this, we present the climatecheck shared task, focusing on automatic fact-checking of climate-related claims from social media against scientific publications. the task was hosted at the 5th scholarly document processing (sdp) workshop 1 and consisted of two subtasks: (i) retrieving relevant scientific documents for a given claim, and (ii) classifying the claim's veracity based on the retrieved evidence. subtask i was evaluated using the average scores of recall@k (@xmath0, 5, 10) and binary preference (bpref, @xcite , and subtask ii was evaluated using the f1 score in addition to recall@10 from subtask i. we used the codabench platform to host the task @xcite , attracting registrations from 27 users and 13 active teams, ten of which submitted results to the leaderboard. 2 the competition followed the timeline below: 2 related shared tasks several shared tasks have been introduced to support research on automatic evidence retrieval and claim verification. these tasks differ in the domain of claims, the type of evidence corpora, and the complexity of the verification process. fact extraction and verification (fever, thorne et al., 2018) and its extension, fever over unstructured and structured information (fever-ous, aly et al., 2021), were tasks focused on claim verification against wikipedia articles, the latter expanding into structured evidence such as tables and lists. fever established the widely adopted three-stage pipeline of document retrieval, sentence selection, and natural language inference (nli). however, despite their scale and influence, fever and feverous differ from our effort in their evidence domain, which is encyclopedic rather than scientific, potentially affecting the applicability of certain retrieval methods. the automated verification of textual claims (averitec) shared task was a recent effort presented at the fever 2024 workshop @xcite . the task focused on evidence retrieval and veracity prediction of general real-world claims with linked evidence from the web using search engines. this task differs from ours in two main aspects: claims are not domain-specific, and the evidence is retrieved from the web rather than the more trustworthy scientific literature. the sciver shared task was organised at the sdp 2021 workshop, aiming to verify scientific claims extracted from research articles against a given corpus of publications @xcite . although the task is similar in its focus on scientific evidence, sciver's claims originate from research papers and are limited to the biomedical domain, in contrast to our task, which focuses on climate-related claims from public discourse. finally, checkthat!, organised annually as a clef lab since 2018 @xcite , focuses on mitigating misinformation online across different platforms and several languages. previous editions have addressed claim detection, stance verification, and evidence retrieval, focusing primarily on political and journalistic content. most recently, the 2025 edition included the scientific web discourse task @xcite , focusing on 1. detecting whether a post contains references to scientific entities, and 2. linking posts with implicit references of studies to their relevant publications. these tasks are similar to our work in their objective of connecting public discourse to scientific publications, with task 1 being especially relevant to the pre-processing steps of preparing the climatecheck dataset. however, unlike task 2, our work does not assume any mention of a study in a post, rather processing general claims. ## task description climatecheck consisted of two subtasks: ## evaluation subtask i: abstracts retrieval as an information retrieval (ir) task, subtask i tackles identifying relevant pieces of information from large corpora based on a user query. evaluating ir is an inherently difficult task due to the problem of incomplete relevance annotations when the evidence corpus contains a large number of documents @xcite . that is because not all potentially relevant documents can be annotated, making it hard to know whether a system truly failed to retrieve relevant items or simply retrieved items that were never judged. various metrics are employed to evaluate ir based on rankings @xcite , including mean average precision, mean reciprocal rank, and normalised discounted cumulative gain @xcite . however, in our specific task, we faced two primary challenges: the absence of annotated ranking information and the problem of incomplete relevance judgements. given these constraints, we selected recall@k and bpref as our evaluation metrics. recall@k measures the proportion of relevant documents retrieved in the top k results. it does not consider the order of the retrieved documents, making it suitable for scenarios where gold ranking information is unavailable. the metric has been widely used to evaluate dense retrieval systems @xcite . in subtask i, we ask participants to retrieve the top 10 abstracts per claim, hence we use @xmath1, 5, 10 to compare systems on different levels. bpref is a score designed to handle situations with incomplete relevant judgements. it evaluates how many judged non-relevant documents are retrieved before judged relevant ones, mitigating potential bias introduced by unjudged documents @xcite . the final evaluation of subtask i, which decides the rankings, is the average of the four scores mentioned above. we considered a retrieved abstract to be relevant if it was annotated as evidentiary (i. e., supports or refutes) in our gold data. however, this data was bound to be biased towards our own retrieval method used to create the annotation corpus. thus, to ensure a fair evaluation, we collected participants' outputs weekly during the test phase, subsequently adding more human-annotated instances to the gold data (see section 5). ## dataset the foundation of the shared task is the climate-check dataset (abu ahmad et al., 2025), foot_2 consisting of 435 unique english climate-related claims in lay language linked to scientific abstracts, resulting in 1,815 claim-abstract pairs. each pair was reviewed by two graduate students in climate sciences and annotated as supports, refutes, or nei. in cases of disagreements, a third student curated the claim-abstract pair, deciding its final label. claims were collected from available datasets @xcite @xcite , and underwent several pre-processing steps: scientific check-worthiness detection, atomic claim generation, and text style transfer, the latter for those not originating directly from social media. the abstracts were collected from openalex @xcite and s2orc @xcite , resulting in a corpus of 394,269 climate-related publications. 4 claims and abstracts were then linked using bm25 @xcite followed by a cross-encoder trained on the ms-marco data and a trec-like pooling approach using six models to create the annotation corpus. in abu ahmad et al. (2025), we describe the development of the dataset in more detail. week 1 week 2 week 3 -2 active teams -405 annotated documents -3 active teams -351 annotated documents -7 active teams -477 annotated documents 21-25 april, 2025 25 april -02 may, 2025 02-13 may, 2025 total -7 active teams -1,233 annotated documents ing sets, the former consisting of 259 unique claims and a total of 1,144 claim-abstract pairs, while the latter 176 unique claims and 671 claim-abstract pairs. the annotated pairs of the test set were not released publicly for participants, since they were used as the reference test set for evaluation. in an attempt to make the evaluation less biased towards the gold test set, which is based on our own linking approach, we annotated more documents on a weekly basis as the task was running. these were based on participants' submissions using the following approach: the agreement threshold was decided each week depending on the number of submitting teams, taking into account our limited human annotation capacity (four student annotators). if needed, we filtered further based on the rankings of claimabstract pairs across submitted systems. we summarise the result of this process in figure 1 , and report more details in appendix b. to accommodate the timeline of the sdp 2025 workshop and the pace of the annotators, we were able to gather new documents from runs submitted until may 13, 2025, one week before the competition deadline. this process resulted in the addition of 1,233 new claim-abstract pairs added to the gold testing data, with an overall number of 1,904 manually annotated pairs in the gold test set. ## baselines for subtask i, we developed a multi-stage retrieval approach as a baseline, combining sparse and dense retrieval with a neural reranker. bm25 has proven to be a fast and efficient method for initial retrieval @xcite . we used it as a sparse retrieval step to get an initial set of the top 1000 relevant abstracts per claim. next, we computed embeddings for each claim and abstract using the msmarco-minilm-l-12-v3 sentence transformer, 5 and calculated the cosine similarity for each claim-abstract pair. we selected the top 20 ranked abstracts per claim, filtering out lexically relevant but semantically irrelevant candidates. finally, a neural reranker, ms-marco-minilm-l6-v2, foot_5 provided cross-encoder scores, resulting in the final candidate pool of the top 10 abstracts per claim. to obtain labels for each claim-abstract pair as a baseline for subtask ii, we used the open source yi-1.5-9b-chat-16k model @xcite , selected based on our experiments with several models when creating the dataset @xcite . the model was prompted in a zero-shot manner with the following prompt: you are an expert claim verification assistant with vast knowledge of climate change , climate science , environmental science , physics , and energy science. your task is to check if the claim is correct according to the evidence. generate 'supports' if the claim is correct according to the evidence, 'refutes' if the claim is incorrect or cannot be verified, or 'not enough information' if you there is not enough information in the evidence to make an informed decision. only return the verification verdict. ## discussion the submissions to climatecheck reveal key design patterns and trade-offs in building claim verification pipelines grounded in scientific literature. although architecture choices varied, several common effective strategies emerged across topperforming teams. we summarise the approaches for subtasks i and ii in tables 4 and 5 , respectively, and compare their results visually in figure 3 . a clear pattern from subtask i is the use of hybrid pipelines, combining sparse retrievers (e.g., bm25 and splade) with different dense retrievers, as well as cross-encoder rerankers (e.g., bge and minilm). three teams extended this by utilising more advanced components: llm-based reranking (akiepura_jlam and pranav) and graph-based reranking (alexunlp-fmt). although the teams achieved competitive scores, they were still outperformed by the relatively simpler ensemble of fine-tuned cross-encoders using rrf presented by ant bridge. despite variations in retrieval strategies, all teams, except pranav, followed a similar paradigm of fine-tuning models with the available training data in a contrastive learning approach. the main difference in their approaches was the way negative samples were selected, with some incorporating nei-labelled abstracts, while others using the least relevant abstracts from their own retrieval approach. we hypothesise that this enabled models to distinguish subtle semantic differences in scientific discourse @xcite . notably, the top two systems fine-tuned the cross-encoders, while the others did so on their dense retrieval models. when comparing systems, we additionally note the impact of retrieval depth and recall preservation. the top-ranked system retrieved up to 5000 abstracts per claim before reranking, enabling a high coverage of potentially relevant documents. in contrast, systems that retrieved a limited number of abstracts early on could have missed documents, impacting the effectiveness of reranking. this highlights that in tasks where relevant evidence is sparse and semantically complex, such as scientific abstracts, high recall in retrieval is effective. for subtask ii, all leaderboard results employed an llm classification approach, resulting in relatively small margins in their scores. notably, the top two teams used closed-source, commercial llms, while the third ranked team and the baseline employed open-source models. that being said, team efc showed that a more lightweight architecture, fine-tuned correctly, can still yield competitive results, highlighted by the results they achieved using deberta. this emphasises the practical trade-off between performance and efficiency, which is an important consideration for real-world applications such as content moderation or misinformation detection. in such scenarios, latency, scalability, and interpretability matter. thus, systems optimised for low-resource settings remain very relevant, while other systems that employ commercial llms might be less useful. ## conclusion this paper presented the climatecheck shared task, which focused on fact-checking claims from social media about climate change against scholarly articles. the task ran during april/may 2025 and was hosted as part of the 5th sdp workshop in 2025. given a claim, two subtasks were available: (i) retrieving the top 10 most relevant (i. e., evidentiary) abstracts, and (ii) classifying the veracity of the claim given the abstract. the first subtask was evaluated using recall@k (@xmath2, 5, 10) and bpref, while the second using f1 with additional scaling based on correctly retrieved abstracts. the task received ten leaderboard submissions, three of which for both subtasks. participants explored a wide range of retrieval and classification strategies, including sparse and dense retrieval fusion, supervised reranking with cross-encoders, promptbased classification with llms, and fine-tuned transformer classifiers. despite methodological differences, the most effective systems shared an emphasis on high-recall retrieval, robust reranking, and careful label calibration. the climate-check datasets are publicly available, 15,16 and a test suite can be accessed for further submissions by the community. 17 while the task results are encouraging, it remains an open question whether these systems are reliable enough for practical deployment. key open challenges include ensuring system robustness under noisy or multilingual in-put, reducing inference latency for real-time use, and scaling evidence retrieval across large scholarly corpora. addressing these challenges will be essential to transition from prototype systems to real-world fact-checking tools that can support climate literacy and policy discourse.