H um S et: Dataset of Multilingual Information Extraction and Classification for Humanitarian Crises Response (2022)

Abstract: Timely and effective response to humanitarian crises requires quick and accurate analysis of large amounts of text data – a process that can highly benefit from expert-assisted NLP systems trained on validated and annotated data in the humanitarian response domain. To enable creation of such NLP systems, we introduce and release HumSet, a novel and rich multilingual dataset of humanitarian response documents annotated by experts in the humanitarian response community. The dataset provides documents in three languages (English, French, Spanish) and covers a variety of humanitarian crises from 2018 to 2021 across the globe. For each document, HUMSET provides selected snippets (entries) as well as assigned classes to each entry annotated using common humanitarian information analysis frameworks. HUMSET also provides novel and challenging entry extraction and multi-label entry classification tasks. In this paper, we take a first step towards approaching these tasks and conduct a set of experiments on Pre-trained Language Models (PLM) to establish strong baselines for future research in this domain. The dataset is available at https://blog.thedeep.io/humset/.

## introduction during humanitarian crises caused by reasons ranging from natural disasters, wars ,or epidemics such as covid-19, a timely and effective humanitarian response highly depends on fast and accurate analysis of relevant data to yield key information. early in the response phase, namely in the first 72 hours after a disaster strikes, the humanitarian response analysts in international organizations 1 review large amounts of data loosely or strongly relevant to the crisis to gain situational awareness. 1 such as the international federation of red cross (ifrc), the united nations high commissioner for refugees (un-hcr), or the united nations office for the coordination of humanitarian affairs (unocha) a large portion of this data appears in the form of secondary data sources i. e. reports, news, and other forms of text data, and is integral in revealing which type of relief activities to undertake. analysis in this phase involves extracting key information and organizing it according to sets of pre-defined domain-specific structures and guidelines, referred to as humanitarian analysis frameworks. while typically only small workforces are available to analyze such information, an automatic document processing system can significantly help analysts save time in the overall humanitarian response cycle. to facilitate such systems, we introduce and release humset, a unique and rich dataset of document analysis in the humanitarian response domain. humset is curated by humanitarian analysts and covers various disasters around the globe that occurred from 2018 to 2021 in 46 humanitarian response projects. the dataset consists of approximately 17k annotated documents in three languages of english, french, and spanish, originally taken from publicly-available resources. 2 for each document, analysts have identified informative snippets (entries) with respect to common humanitarian frameworks and assigned one or many classes to each entry (details in §2). humset provides a large dataset for the training and evaluation of entry extraction and classification models, enabling the research and development of further nlp systems in the humanitarian response domain. we take the first step in this direction, by studying the performance of a set of strong baseline models (details in §3). our released dataset expands the previously provided collection by @xcite with a more recent and comprehensive set of projects, as well as additional classification labels. other similar datasets in the humanitarian domain, @xcite present humanannotated twitter corpora collected during 19 dif-ferent crises between 2013 and 2015, @xcite provide a combination of various socialmedia crisis-related existing datasets, and adel and wang (2020) and later alharbi and lee (2021) publish arabic twitter classification datasets for crisis events. humset, in contrast to the current resources which mostly originated from social media, is created by humanitarian experts through an annotation process on official documents and news from the most recognized humanitarian agencies, conferring high reliability, continuous updating, and accurate geolocation information. ## humset dataset the collection originated from a multiorganizational platform called the data entry and exploration platform (deep), foot_1 developed and maintained by data friendly space (dfs) 4 the platform facilitates classifying primarily qualitative information with respect to analysis frameworks and allows for collaborative classification and annotation of secondary data. the dataset is available at https://blog.thedeep.io/humset/ . ## experiments and results to conduct a set of baseline experiments on hum-set according to the mentioned tasks, we split the data into training, validation, and test sets for all our experiments (80%, 10%, and 10%, respectively). we apply stratified splitting @xcite to maintain the same distribution of labels for each set. implementation details of entry extraction (section 3.1) and entry classification (section 3.2) are available at https: //github.com/the-deep/humset . ## conclusion we presented humset, a new dataset of annotated humanitarian data, containing 148,621 entries with a total of 62 different tags. we have shown two nlp-based tasks that can be applied to it, providing initial experiments of its applications. humset is a multilingual human-annotated humanitarian text dataset not composed of social network data, providing a valuable and highly reliable resource for the development of automation tools regarding crisis response and humanitarian aid activities.