Benchmarking the Benchmarks: Reproducing Climate-Related NLP Tasks (2025)

Abstract: Significant efforts have been made in the NLP community to facilitate the automatic analysis of climate-related corpora by tasks such as climate-related topic detection, climate risk classification, question answering over climate topics, and many more. In this work, we perform a reproducibility study on 8 tasks and 29 datasets, testing 6 models. We find that many tasks rely heavily on surface-level keyword patterns rather than deeper semantic or contextual understanding. Moreover, we find that 96% of the datasets contain annotation issues, with 16.6% of the sampled wrong predictions of a zero-shot classifier being actually clear annotation mistakes, and 38.8% being ambiguous examples.These results call into question the reliability of current benchmarks to meaningfully compare models and highlight the need for improved annotation practices. We conclude by outlining actionable recommendations to enhance dataset quality and evaluation robustness.

## introduction as climate change becomes a more and more urgent problem (the world meteorological organization (wmo)), there has been a surge of interest in the field of climate-related natural language processing (nlp). over 60 recent works study tasks such as detecting climate-related discourse, identifying green claims in corporate communications, answering climate-related questions, and detecting deceptive patterns in climate-related communication @xcite . and yet, the progress in this area is hard to gauge, for several reasons: first, the approaches are evaluated on different datasets even if they treat the same tasks; second, the evaluations differ in their metrics, which makes the results incomparable; third, the approaches are rarely compared to simple tf-idf baselines, so that it is unclear to what degree the proposed problems can be solved by simpler methods as well; and finally, few of the works use zero-shot llms as competitors, so that it is hard to judge to what degree the proposed methods have become obsolete with the arrival of generative models. in this paper, we set out to make a transversal and unified comparison of climate-related nlp tasks. we compiled 29 climate-related nlp datasets and evaluated a suite of baselines that cover traditional machine learning approaches such as tf-idf with logistic regression, as well as finetuned transformer models, and open and closed source llms in a standardized setting -effectively building a benchmark. we also performed an indepth error analysis, with more than 500 manually annotated errors. our works yields the following insights: (1) tf-idf, finetuned models, and zero-shot llms perform similarly, with finetuned models performing slightly better. while this confirms that current models are capable of solving these tasks, it also questions the difficulty of the tasks. if tf-idf performs well, as we systematically show, then this means that the datasets consist of simple examples that can be predicted using word-frequency only. (2) 96% of our datasets contain annotation issues, with 16.6% of the sampled wrong predictions of a zero-shot classifier being actually clear annotation mistakes, and 38.8% being ambiguous examples where multiple labels could fit or where it is not clear which label to assign. (3) our manual analysis shows that finetuned models may perform better not because they generalize better, but because they overfit on annotation noise, bias, or unclear guidelines. these findings highlight the limitations of the datasets to meaningfully compare models, because their simplicity and inconsistencies compress performance into a narrow band, masking real differences in model capability. we believe our findings motivate the need for more challenging datasets, with examples that go beyond keywords, require reasoning, and come with clearer, more robust annotation schemes. we propose recommendations for construction such datasets at the end of our paper. we make all cleaned-up datasets, baselines, and a python library to run comparisons publicly available at https://github.com/tcalamai/acl_ climatenlptoolbox . ## conclusion in this work, we curated, cleaned, and standardized 29 climate-related nlp datasets, ensuring consistency across tasks. we systematically evaluated a diverse range of models. our results indicate that all approaches achieved competitive performance. however, since even the tf-idf baseline performs well, the datasets may overall be too simple. our analysis also revealed that nearly all datasets contain annotation inconsistencies, which may introduce noise in model evaluations. these findings highlight the importance of dataset quality in benchmarking and call for more rigorous annotation protocols in climate-related nlp research. all our data, scripts, and models are publicly available at at https://github.com/tcalamai/ acl_climatenlptoolbox . in particular, we provide a python library that can run all models on entire files by parsing the pdfs, splitting them into paragraphs, doing model inference on each paragraph (individually or as an ensemble), and aggregating the results into statistics (example in appendix d). acknowledgement. this work was performed using hpc resources from genci-idris (grant ad011014244r1). this work is supported by amundi technology and the anrt with a cifre fellowship.