WXI mpact B ench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models (2025)

Abstract: Climate change adaptation requires the understanding of disruptive weather impacts on society, where large language models (LLMs) might be applicable. However, their effectiveness is under-explored due to the difficulty of high-quality corpus collection and the lack of available benchmarks. The climate-related events stored in regional newspapers record how communities adapted and recovered from disasters. However, the processing of the original corpus is non-trivial. In this study, we first develop a disruptive weather impact dataset with a four-stage well-crafted construction pipeline. Then, we propose WXImpactBench, the first benchmark for evaluating the capacity of LLMs on disruptive weather impacts. The benchmark involves two evaluation tasks, multi-label classification and ranking-based question answering. Extensive experiments on evaluating a set of LLMs provide first-hand analysis of the challenges in developing disruptive weather impact understanding and climate change adaptation systems. The constructed dataset and the code for the evaluation framework are available to help society protect against vulnerabilities from disasters.

## introduction climate change adaptation @xcite , referring to actions that help reduce societal vulnerability to climate change, demands a sophisticated understanding of the disruptive weather impacts on society (e.g., the perspective of economy and policy) @xcite . societal reactions to past disruptive weather events are stored in reliable historical sources @xcite . among them, historical newspapers provide irreplaceable information, recording not just meteorological conditions @xcite , but crucially, how societies adapted and recovered from disasters @xcite . in addition, historical newspapers usually report regional disruptive weather impacts with local experiences, which is valuable to understanding longterm climate patterns and social effects @xcite but are often absent in official meteorological records @xcite . understanding complex patterns in disruptive weather events is important for society with forecasts, societal responses, and public policy @xcite . the challenge of identifying impacts and responses often lies in climate-related text processing, which contains period-specific narratives and domain-specific linguistic phenomena. for example, disambiguation and taxonomy polysemy can occur in newspaper articles, where climate-related terms frequently appear in diverse linguistic contexts beyond their meteorological meanings. figure 1 shows that the term "blizzard" can refer to a severe snowstorm or the name of the sports team (e.g., "toronto blizzard"). similarly, the term "flood" can describe an overflowing body of water or can be used metaphorically (e.g., "flood the venue"). this polysemy occurs commonly in newspapers and thus requires the system to distinguish the literal weather-related meanings and alternate usages by improving the climate-related semantic understanding @xcite . another challenge lies in extracting information from the original paper content. although it is commonly achieved by optical character recognition (ocr) @xcite , errors remain due to mixed content formats, and complex narrative structures @xcite . these errors can negatively affect the extracted text for disruptive weather impact analysis, which renders the texts difficult to serve as a high-quality corpus. existing studies on climate-related language processing focus on extracting climate patterns (alaparthi and mishra, 2020), wildfire resilience @xcite and analyzing extreme weather events @xcite . intuitively, llms @xcite @xcite offer a powerful alternative for understanding disruptive weather impacts. however, their effectiveness is unexplored @xcite due to the lack of a corresponding benchmark. the resources used in previous studies cannot comprehensively evaluate the ability of llms for weather impacts. this is because i) compared to informative reports in newspapers, previously used meteorological records do not contain long-term and detailed regional information @xcite ; and ii) the previous meteorological records are easily obtained and have been available for a long while. thus they might be already included in the pre-training of llms and should not be included in benchmark build-up to avoid potential bias @xcite . to develop a system that assesses the impact of disruptive weather on society, the first step is to establish a domain benchmark for the evaluation protocol. in this study, we design a four-stage data construction pipeline that begins with a disruptive weather impact dataset in which we correct ocr errors in digitalized newspaper article extraction. we extract a sample of articles from two time periods, which cover linguistic and social changes across different eras and increase linguistic complexity due to the different descriptions of weather events in different times @xcite . historical newspapers often employed more descriptive and elaborate narratives compared to modern reporting styles @xcite . these narratives frequently included outdated terminology, spelling variations, and evolving writing conventions @xcite . the articles are selected by topic modeling, including six impact categories (infrastructural, political, financial, ecological, agri-cultural, and human health), which are informed by previous studies @xcite and align with modern disaster impact assessment frameworks @xcite . with our constructed dataset, we develop a benchmark, wximpactbench, to investigate the capacity of llms to understand disruptive weather impacts with two tasks: i) multi-label classification and ii) ranking-based question-answering. the multi-label classification task employs the previous six impact categories as labels for each article whose ground-truth is annotated by human labor. the question and the candidate pools for the ranking-based question-answering task are constructed based on the context and annotation of the multi-label classification task. this can facilitate any future development of retrieval-augmented generation (rag) systems in the climate-related domain @xcite @xcite . extensive experiments on evaluating a set of off-the-shelf llms provide first-hand analysis of their capacity to understand disruptive weather impacts and reveal the challenges in developing climate change adaptation systems to help society protect against vulnerabilities from disasters. our contributions are summarized as follows: (1) we construct a high-quality disruptive weather impact dataset from digitalized newspaper articles in the climate-related domain with a four-stage pipeline. (2) we propose wximpactbench with two typical tasks for evaluating the capacity of llms on disruptive weather impact understanding, which is the first benchmark to facilitate the development of such domain-specific systems. the constructed dataset and the evaluation framework code are available at our github repository foot_0 . (3) we conduct extensive experiments on benchmarking a set of llms, providing first-hand analysis of challenges in disruptive weather impact understanding and climate change adaptation. ## wximpactbench: disruptive weather impact benchmark our wximpactbench benchmark aims to evaluate to what extent existing llms can understand disruptive weather impacts, which also shows the evolution of vulnerability and resilience strategies from society across various periods. it involves two main stages: i) dataset construction; and ii) task definition and evaluation. ## conclusion in this study, we propose a disruptive weather impact understanding benchmark, wximpact-bench, to address the lack of datasets and evaluation frameworks in climate change adaptation. we first process the corpus from newspaper articles and provide comprehensive instruction for impact annotation with each processed article. then, we design multi-label classification and ranking-based qa tasks to evaluate the llms in understanding various defined disruptive weather impacts. extensive experiments on wximpact-bench reveal that the existing llms struggle with understanding disruptive weather impacts across different style narratives and context settings. our wximpactbench enables the community to evaluate the developed systems on disruptive weather impact understanding, which sup-ports the society to learn from and prepare for the impacts of climate change.