H um VI : A Multilingual Dataset for Detecting Violent Incidents Impacting Humanitarian Aid (2024)

Abstract: Humanitarian organizations can enhance their effectiveness by analyzing data to discover trends, gather aggregated insights, manage their security risks, support decision-making, and inform advocacy and funding proposals. However, data about violent incidents with direct impact and relevance for humanitarian aid operations is not readily available. An automatic data collection and NLP-backed classification framework aligned with humanitarian perspectives can help bridge this gap. In this paper, we present HumVI – a dataset comprising news articles in three languages (English, French, Arabic) containing instances of different types of violent incidents categorized by the humanitarian sector they impact, e.g., aid security, education, food security, health, and protection. Reliable labels were obtained for the dataset by partnering with a data-backed humanitarian organization, Insecurity Insight. We provide multiple benchmarks for the dataset, employing various deep learning architectures and techniques, including data augmentation and mask loss, to address different task-related challenges, e.g., domain expansion. The dataset is publicly available at https://github.com/dataminr-ai/humvi-dataset.

## introduction violent events that impact humanitarian efforts, such as the looting of aid trucks and the kidnapping of aid workers, frequently hinder the delivery of life-saving aid and protection efforts, with devastating consequences for conflict-affected populations. conflicts also severely disrupt existing food systems, healthcare, and education structures @xcite , leading to food insecurity, malnutrition @xcite , increased mortality due to inadequate healthcare @xcite , and significant educational disruptions for children @xcite . the system-* work done during author's internship at dataminr. figure 1 : a sample input/output to the two nlp models for (i) relevance classification, followed by (ii) categorization of the event into humanitarian aid categories. aggregated information about violent events that impact humanitarian efforts can be operationalized by humanitarian organizations to systematically inform a wide range of different responses, including security risk management, program planning, and advocacy prioritization. resources are severely limited during crisis response, and manual data collection in conflict-affected and highly insecure environments is rarely a priority. the use of nlp techniques can readily reduce the labor and resources required to maintain such a data collection pipeline. importantly, it can also be carried out remotely, thus reducing the security exposure of those supporting a humanitarian response through data collection and analysis. most nlp capacity in the humanitarian context has focused on english, limiting the use of models in conflict-affected countries where information is shared in french, arabic, and other languages. in order to process large amounts of disparate data to detect violent events, it is essential to build nlp models that can automatically detect such incidents. next, these incidents should be tagged appropriately with the relevant type of humanitarian operation, e.g., an event where aid workers are threatened or harmed would be tagged as aid security. we identify two primary gaps in the current publicly available datasets: (1) most are focused on identifying a single type of event, such as disasterrelated information @xcite , specific threats against human rights defenders @xcite , or civil unrest @xcite . even when datasets cover a wide range of events and associated tags @xcite , these tags often do not indicate the specific humanitarian sectors impacted by the event and lack information related to downstream humanitarian efforts, such as healthcare or food security @xcite . (2) most of the humanitarian event detection datasets are disproportionately english and this has been identified as a key issue for improving adoption of nlp techniques in the humanitarian sector @xcite . the dataset presented in this paper is multilingual, and we hope it will support and encourage nlp practitioners to create multilingual models for humanitarian purposes. to address these two issues, we introduce a new dataset humvi (humanitarian violent incidents). this dataset comprises 17, 497 articles in three languages, with each article tagged for its relevance and, if relevant, categorized by the key humanitarian response sector where it is relevant, i.e., aid security, education, food security, health, and protection (a sample is shown in figure 1 ). there are three qualities of this dataset that ensure its value to the field. ## dataset split and variants we provide two different variants of the dataset described in the previous section. humvi-core. the core dataset is english only, and contains labels for four categories relevant to humanitarian work -it does not include any label for food security. the dataset consists of 15, 631 news articles. ## models we experiment with multiple different model types to establish benchmarks for two tasks on humvi: ## conclusions in this paper, we presented humvi, a multilingual dataset of 17k news articles classified by their relevance to humanitarian aid efforts and tagged with appropriate categories. developed in collaboration with insecurity insight, this dataset addresses real-world challenges in optimizing data workflows with limited resources using nlp techniques. we trained and evaluated multiple nlp models for relevance and event classification, showing that both fine-tuned models and zero-shot llms can perform well, leaving room for improvement. we believe that humanitarian organizations can utilize this dataset to classify real-time news articles based on relevance and tag them with specific humanitarian aid sector relevant categories. the results can help determine whether llms or finetuned transformer-based models are suitable for their processes. additionally, organizations needing extra tagging for humanitarian events associated with the text can benefit from the models trained using the provided data and code. for nlp researchers we hope that they will find this dataset a valuable resource for addressing realworld problems in the ai4sg space, particularly for developing techniques in resource-limited scenarios. the dataset presents interesting challenges, with significant potential for improving baseline performance.