Winning C limate C heck: A Multi-Stage System with BM 25, BGE -Reranker Ensembles, and LLM -based Analysis for Scientific Abstract Retrieval (2025)

Abstract: The ClimateCheck shared task addresses the critical challenge of grounding social media claims about climate change in scientific literature. This paper details our winning approach. For abstract retrieval, we propose a multi-stage pipeline: (1) initial candidate generation from a corpus of ~400,000 abstracts using BM25; (2) fine-grained reranking of these candidates using an ensemble of BGE-Reranker cross-encoder models, fine-tuned with a specialized training set incorporating both random and hard negative samples; and (3) final list selection based on an RRF-ensembled score. For the verification aspect, we leverage Gemini 2.5 Pro to classify the relationship (Supports, Refutes, Not Enough Information) between claims and the retrieved abstracts, guided by carefully engineered prompts. Our system achieved first place in both subtasks, demonstrating the efficacy of combining robust sparse retrieval, powerful neural rerankers, strategic negative sampling, and LLM-based semantic analysis for connecting social media discourse to scientific evidence. Part of the example code: https://anonymous.4open.science/r/climatecheck_solution-1120

## introduction the widespread dissemination of misinformation on social media platforms represents a serious threat to informed public discourse @xcite , particularly regarding critical global issues such as climate change @xcite . addressing this challenge, the climatecheck shared task at the fifth workshop on scholarly document processing @xcite aims to bridge the gap between social media discourse on climate change and scientific literature. the cli-matecheck dataset @xcite is also provided in this shared task for model training and evaluation. participants develop systems to: (1) retrieve relevant scholarly abstracts for social media claims (subtask i: abstract retrieval), and (2) verify claims against these abstracts (subtask ii: claim verification). ## method our winning solution employs a multi-stage process designed to maximize both recall and precision as illustrated in figure 1 . this approach for subtask i involves: (1) coarse retrieval with bm25 @xcite for top 5,000 abstracts; ## discussion in this section, we discuss why we chose this cascaded architecture to solve the problem. for algorithm competitions, practicality in solving problems within limited timeframes needs consideration. since we needed to iteratively submit results and receive feedback to refine our models, an efficient and accurate coarse-ranking solution became essential. we ultimately selected bm25 for recall because it could rapidly screen 5,000 candidate samples within a short time frame. in the subsequent second stage, we experimented with various models repeatedly and finally chose the bge-reranker through offline evaluation. ensemble learning effectively improved model accuracy, as demonstrated in our previous practice at kdd cup 2024 @xcite . for task ii, we directly used gemini pro 2.5 for reasoning to obtain final results. this was because we believed that large-parameter closed-source llm would achieve better performance in deep semantic understanding tasks compared to fine-tuned smallerparameter models. however, resource constraints prevented us from conducting more sophisticated experiments. another benefit of our approach is that for subtask i, the time required to process the data does not increase significantly when the data volume grows. this is because the first step, bm25, will only retain a fixed set of 5,000 entries. naturally, for subtask ii, the processing time will increase linearly with the amount of data. ## conclusion this paper describes our climatecheck shared task system, which won first place in both subtasks. our approach combines a bm25 sparse retriever for candidate pooling with an ensemble of finetuned bge-reranker models for semantic reranking. training the rerankers on a dataset incorporating hard negative mining significantly improved their performance. for claim verification, gemini 2.5 pro effectively classified claim-abstract relationships. this hybrid pipeline demonstrates the efficacy of combining optimized retrieval with large language models.