D isast IR : A Comprehensive Information Retrieval Benchmark for Disaster Management (2025)

Abstract: Effective disaster management requires timely access to accurate and contextually relevant information. Existing Information Retrieval (IR) benchmarks, however, focus primarily on general or specialized domains, such as medicine or finance, neglecting the unique linguistic complexity and diverse information needs encountered in disaster management scenarios. To bridge this gap, we introduce DisastIR, the first comprehensive IR evaluation benchmark specifically tailored for disaster management. DisastIR comprises 9,600 diverse user queries and more than 1.3 million labeled query-passage pairs, covering 48 distinct retrieval tasks derived from six search intents and eight general disaster categories that include 301 specific event types. Our evaluations of 30 state-of-the-art retrieval models demonstrate significant performance variances across tasks, with no single model excelling universally. Furthermore, comparative analyses reveal significant performance gaps between general-domain and disaster management-specific tasks, highlighting the necessity of disaster management-specific benchmarks for guiding IR model selection to support effective decision-making in disaster management scenarios. All source codes and DisastIR are available at https://github.com/KaiYin97/Disaster_IR.

## introduction natural disasters and technological crises cause severe threats to human lives, infrastructure, and the environment, necessitating timely and effective management responses @xcite @xcite . in such critical scenarios, stakeholders, including emergency responders, government agencies, and the general public, require rapid access to reliable and contextually relevant information to make informed decisions @xcite . are people rebuilding homes after california wildfires? my car is trapped in water, how to save myself? power outage occurred in texas in this winter storm, is it right? what emergency supplies should i prepare for covid lockdown? did nasa confirm a meteorite will hit florida in 2025? ## related work existing ir benchmarks target mainly generalpurpose or specialized domains, such as medicine and finance. beir @xcite evaluates zero-shot retrieval models across 18 tasks, such as fact verification, qa, and scientific document ranking. instruction-based benchmarks like followir @xcite , instructir @xcite , and mair @xcite reformulate ir tasks using natural language instructions. some domain-specific ir benchmarks, such as mirage @xcite and finmteb @xcite , focus on biomedical and financial domains. while effective in their respective domains, they fail to capture the linguistic and contextual patterns in disaster management areas. despite the critical role of information retrieval in disaster management, existing benchmarks are limited in scope, scale, and task diversity. prior datasets-such as the fire irmidis track @xcite and event-specific corpora from disasters in nepal, italy, and indonesia @xcite @xcite -primarily focus on twitter microblogs, targeting short-text retrieval or keyword matching with narrow task coverage. case-based systems like @xcite use proprietary data for concept-based retrieval in search and rescue planning. these benchmarks typically rely on single-source or scenario-specific data and lack support for realistic, multi-intent retrieval. in contrast, disastir provides a large-scale, multi-intent, and multi-source benchmark covering diverse disaster types and information needs, enabling comprehensive evaluation in real-world contexts. ## conclusion in this work, we introduce and publicly release dis-astir, the first comprehensive retrieval benchmark for evaluating model performance in disaster management contexts. disastir consists of 9,600 user queries and more than 1.3 million labeled querypassage pairs, spanning 48 retrieval tasks defined by six search intents and eight general disaster event types, covering 301 specific event types. using disastir, we evaluate 30 sota opensource retrieval models under both exact and ann search settings. our findings provide practical guidance for selecting appropriate ir models based on task type and computational constraints, supporting timely and effective access to critical information in disaster management scenarios.