The C limate C heck Dataset: Mapping Social Media Claims About Climate Change to Corresponding Scholarly Articles (2025)

Abstract: The rapid spread of misinformation on and through social media poses a significant challenge to public understanding of climate change and evidence-based policymaking. While natural language processing techniques have been used to analyse online discourse on climate change, no existing resources link social media claims to scientific literature. Thus, we introduce ClimateCheck, a human-annotated dataset that connects 435 unique, climate-related English claims in lay language to scientific abstracts. Each claim is connected to at least one and at most seventeen abstracts, resulting in 3,048 annotated claim-abstract pairs. The dataset aims to facilitate fact-checking and claim verification by leveraging scholarly document processing to improve access to scientific evidence in online discussions about climate change.

## introduction social media serves as a powerful tool to discuss critical issues such as climate change. however, it also accelerates the spread of misinformation @xcite , making it increasingly difficult to ensure an informed public and create evidence-based policies. natural language processing techniques have proven valuable in analysing online discourse on pressing topics @xcite . a particularly promising application is linking social media discussions to peer-reviewed scholarly articles, fostering an evidence-based public dialogue @xcite . however, previous efforts have primarily focused on the biomedical domain @xcite and, to the best of our knowledge, no resources have been developed to facilitate this connection for climate change discourse among the public. to address this, we introduce climatecheck, a human-annotated dataset that links atomic english claims in lay language to scientific abstracts related to climate change. our work aims to support fact-checking efforts and promote scientifically grounded discussions on climate change. this paper describes the detailed process of developing the climatecheck dataset, which consists of four main stages, illustrated in figure 1: (1) collection of claims, (2) collection of publications, (3) linking claims to abstracts, and (4) manual annotation of claim-abstract pairs. we collected claims from several existing sources and decomposed them into an atomic, scientifically check-worthy form. claims were either directly extracted from social media or synthetically generated using text style transfer techniques. we then sourced abstracts from scholarly articles in climate change and environmental sciences using existing research registries. to efficiently link claims and abstracts, we employed a pooling strategy popularised by trec @xcite , as seen in similar datasets @xcite . the relevant abstracts for each claim were first retrieved via a sparse retrieval method, followed by a neural cross-encoder for re-ranking. then, state-of-the-art models identified abstracts containing supporting or refuting evidence. this resulted in claim-abstract pairs manually annotated by five graduate students in climate sciences. we adopt an existing annotation scheme @xcite , where each pair is annotated as supports, refutes, or not enough information (nei). this process resulted in 1,325 unique english claims, of which we employ 435 for running the climatecheck shared task @xcite . we split the data into training and testing sets with 259 and 176 unique claims, respectively. each claim in the training data is linked to at least one and at most five abstracts based on our own linking approach, while for the testing data, we annotate additional claim-abstract pairs based on the submissions of participants, result- collection of claims linking claims to abstracts human annotation and curation collection of publications top 1000 relevant abstracts (bm25) top 20 relevant abstracts neural re-ranker sota pooling models climate-fever debagreement climaconvo climatefeedback multifc manual annotation curation + post-processing (5 annotators) ing in a maximum number of seventeen connected abstracts per claim. the overall process results in 3,048 claim-abstract pairs manually annotated with a total inter-annotator agreement (iaa) score of 0.69 using cohen's κ. the rest of the paper is structured as follows. section 2 reviews existing datasets for scientific fact-checking in general, mentioning those specific to the climate sciences domain. sections 3, 4, 5, and 6 respectively explain the processes of collecting claims, collecting publications, linking them, and annotating claim-abstract pairs. importantly, each process is followed by either manual or automatic evaluation to ensure that no errors propagate to the next step. 1 lastly, section 7 presents the performance of pooling models on the annotated data and section 8 concludes our paper. ## related work several datasets have been developed for factchecking across various domains. fact extraction and verification (fever, @xcite is a benchmark that established a baseline for evidence-based claim verification using wikipedia as a corpus. likewise, wikipedia citation entailment (wice, @xcite is a dataset of fine-grained natural claims mapped to wikipedia articles that also provides entailment judgements for each subclaim. although fever and wice cover a broad range of general knowledge claims, their evidence pool lacks the depth of scientific expertise required to verify domainspecific claims. similarly, x-fact @xcite is a multilingual benchmark designed for fact-checking general claims across 25 languages, but does not focus on scholarly publications as evidence sources. several fact-checking datasets use scientific literature as an evidence source. for example, scifact @xcite ) is a benchmark for scientific claim verification in which claims are fact-checked against peer-reviewed biomedical abstracts, with claims annotated as supported or refuted based on evidence sentences. scifact-open @xcite extends this to an open-domain setting, requiring the retrieval of relevant abstracts before verification. however, a key distinction from our approach is that scifact and scifact-open derive claims from scientific documents rather than public discourse. closer comparisons to our work are healthver @xcite , covid-fact @xcite , and covert @xcite , which link social media claims to scientific publications. however, all three are limited to the biomedical domain. when it comes to climate-related fact-checking efforts, climate-fever @xcite is, to our knowledge, the only publicly available dataset designed to assess the veracity of claims about climate change. it follows a fever-like approach, where claims sourced from english news articles are linked to evidential sentences from wikipedia. while climate-fever is a valuable resource, it does not align directly with our goal of connecting public discourse in lay language to scientific publications. other climate-focused fact-checking efforts @xcite rely on claims extracted from dedicated fact-checking websites such as science feedback foot_1 and skeptical science. 3 ## collection of claims to collect claims about climate change, we used five existing sources: (1) climate-fever @xcite , (2) debagree-ment @xcite , (3) clima-convo @xcite , (4) climatefeedback, 4 and ( 5 ) multifc @xcite . some of these directly extract text from social media, while others utilise claims from other sources, such as news outlets. in order to match the claims with the purpose of this dataset, text that was extracted directly from social media platforms underwent a process of claim detection (section 3.1) and atomic claim generation (section 3.2), while text extracted from other sources was converted to social media text style using text style transfer techniques (section 3.3). ## collection of publications to build the corpus of scholarly publications, we first queried s2orc @xcite via its bulk search api using "climate change" as the search term and filtered the results to the environmental sciences field, yielding 210,237 publications. to better simulate a real-life fact-checking environment with millions of available studies, we expanded the corpus using openalex @xcite , filtering for open-access english publications on climate change, which yielded 826,531 articles. after merging and deduplicating by doi, we further filtered out non-english foot_8 publications and those missing abstracts or full-text urls, resulting in a corpus of 835,659 publications. upon inspection, we noticed that some publications in the corpus were noisy, consisting of think-pieces and various non-peer-reviewed documents. thus, we filtered out publications with less than 10 citations as a quality measure, chosen based on similar prior research @xcite . the final ## linking claims to publications following retrieval strategies popularised by the trec competitions @xcite @xcite , we linked each claim to relevant abstracts of scholarly articles using a sparse retrieval method, followed by a neural reranker. we then employed a pooling approach, using six state-of-the-art models to classify each claim-abstract pair as "supports", "refutes", or "nei". if an abstract was classified as evidentiary (i. e., either supports or refutes) by at least three models, the claim-abstract pair was added to the annotation pool. for the sparse retrieval step, we used bm25 @xcite , a method that relies on tf-idf keyword matching, to get the top 1000 abstracts per claim. then, we used a bertbased neural cross-encoder 11 trained on the ms marco passage ranking task 12 to re-rank the retrieved abstracts per claim. for the pooling step, we chose six models based on the following criteria: (1) open source, (2) available on hugging face for ease of implementation, (3) parameter size falls between 120m and 15b due to a limit in compute resources, (4) state-of-the-art performance on language understanding, natural language inference (nli), and/or claim verification tasks. we checked the last criterion using the superglue, 13 openllm, 14 and mteb 15 leaderboards. consequently, three sequence classification models and three causal llms were chosen: 1. roberta-large, fine-tuned on the mnli dataset, 16 2. deberta-xxlarge, fine-tuned on the mnli dataset, 17 3. xlm-roberta, fine-tuned on the xnli dataset, 18 4. yi-1.5-chat with a 16k context window @xcite , 5. qwen 1.5-14b-chat @xcite , and 6. llama3.1 8b-instruct. 19 due to compute and time limitations, the top 20 abstracts from the re-ranking phase were considered. causal models were prompted using zero-shot role prompting and chain-of-thought @xcite techniques (see prompt in appendix a). we used huggingface's pipeline object 20 with the default text generation hyperparameters, disabling sampling, thus effectively selecting the most likely next token at each 11 https://huggingface.co/cross-encoder/ ms-marco-minilm-l-12-v2 12 https://github.com/microsoft/ msmarco-passage-ranking 13 https://gluebenchmark.com/leaderboard 14 https://huggingface.co/spaces/open-llm-leaderboard/ open_llm_leaderboard#/ 15 https://huggingface.co/spaces/mteb/leaderboard 16 https://huggingface.co/facebookai/roberta-large-mnli 17 https://huggingface.co/microsoft/ deberta-v2-xxlarge-mnli 18 https://huggingface.co/joeddav/xlm-roberta-large-xnli 19 https://huggingface.co/meta-llama/llama-3. 1-8b-instruct 20 https://huggingface.co/docs/transformers/en/main_ classes/pipelines step. for sequence classification models, we used the default forward pass without any adjustments. to prepare the annotation corpus, a maximum of five abstracts per claim were considered, preferring higher-ranking evidentiary abstracts. interestingly, the output of the linking phase resulted in a total of 1,167 unique claims connected to a minimum of 1 and a maximum of 5 abstracts. hence, 158 claims were naturally filtered out due to not resulting in any connected abstracts in the pooling process. these were indeed non-scientifically check-worthy claims that were mistakenly left in the data during the claim collection process (see examples in appendix b). ## annotation process to annotate the corpus of claim-abstract pairs, we hired five part-time graduate students (master's). all students have strong expertise in climate sciences, as evidenced by their academic records, and are enrolled in english-language programmes dealing with different aspects of climate sciences. their english proficiency was proven by providing official results from certified english language tests. we used the inception @xcite annotation tool, which offers an automatic calculation of iaa and allows multiple users and roles @xcite . the annotation process followed these steps: (1) read the claim carefully. (2) read the abstract carefully. (3) label the pair as one of the following: "supports": if the abstract supports the claim. "refutes": if the abstract refutes the claim. "nei": if the abstract does not provide sufficient information. annotators were explicitly asked to decide only based on the given abstract, not on their prior knowledge. to account for mistakes in the data preparation process in terms of creating atomic claims, the annotators were asked to report cases that were manually reviewed. if a claim was shortened to an atomic form, both annotators were updated and asked to annotate with the new version of the claim. additionally, if an annotator encountered a claim that is not check-worthy against scientific articles, it was disregarded. 21 due to time and resource restrictions, the first version of the dataset contains a total of 435 unique claims resulting in 1,815 annotated claimabstract pairs. we split those into training and testing sets, where the former includes 259 unique claims and 1,144 claim-abstract pairs, while the latter 176 unique claims with 671 claim-abstract pairs. each document was annotated by two students and curated by a third in case of disagreement. for administrative reasons, we had two annotation groups for the training data, and three for the testing data, each group consisting of two annotators given the same documents. the dataset was used for the climatecheck shared task @xcite , where we annotated a subset of claim-abstract pairs from the submissions of participants on a weekly basis, using claims from the test set. this resulted in 1,233 additional manually annotated claimabstract pairs, with a total of 3,047 documents overall. figure 4 shows the number of claims as a function of the number of connected abstracts, and the quality of annotations is evaluated based on iaa using cohen's κ for pairwise comparisons 22 the climatecheck dataset is publicly available at https: //huggingface.co/datasets/rabuahmad/climatecheck @xcite . while this metric was introduced to account for chance agreement, interpretive and widely-used guidelines, such as those by @xcite , suggest that values between 0.61 and 0.80 indicate substantial agreement. our annotation process achieved an overall cohen's κ score of 0.69, suggesting a high level of consistency among annotators. throughout the project, special attention was paid to the agreement score, with the curator flagging claims with low iaa across all associated abstract pairs as potentially vague. those were then reviewed and rephrased when necessary. the final iaa results are shown in ## performance of pooling models after finalising the annotations for both the train and test sets, we evaluated the pooling models on the task of claim verification using all annotated documents. the results are shown in in contrast, instruction-tuned llms show a significantly better performance, with yi-1.5-9b-chat-16k achieving the best f1 and accuracy scores, both 0.61. this suggests that such models have more generalised reasoning abilities and contextual understanding, likely due to their exposure to such data during training. interestingly, yi outperformed qwen by a large margin, despite having fewer parameters, showing that more parameters does not necessarily mean better performance. among causal lms, yi achieves a relatively balanced precision and recall scores, suggesting that it captures claim-abstract relations with minimal trade-off. however, other models show a clear precision-recall gap, with a pronounced emphasis on weighted precision at the expense of recall. this indicates that while models are highly reliable to make an accurate classification, they miss a significant portion of true instances, generating more false negatives. ## conclusion this paper introduces our work of constructing climatecheck, a human-annotated dataset designed to bridge the gap between social media claims about climate change and corresponding scientific literature. our dataset consists of 435 unique english claims in lay language, each linked to up to seventeen relevant scientific abstracts, resulting in 3,048 claim-abstract pairs. claims were fetched from existing resources and refined into atomic, scientifically check-worthy statements, while abstracts were retrieved from open-access climate science publications. we employed bm25 and a neural cross-encoder to rank abstracts per claim, followed by a pooling approach using stateof-the-art models to select the most relevant evidentiary abstracts for annotation. to ensure highquality annotations, we conducted a structured human annotation process with five graduate students in climate sciences. with this work, our aim is to advance climate-related fact-checking research, fostering a more scientifically grounded public discourse on climate change. further work can utilise our dataset for tasks such as detecting scientifically check-worthy statements on social media, retrieving relevant publications, and verifying climate-related claims.