M ulti H um ES : Multilingual Humanitarian Dataset for Extractive Summarization (2021)

Abstract: When responding to a disaster, humanitarian experts must rapidly process large amounts of secondary data sources to derive situational awareness and guide decision-making. While these documents contain valuable information, manually processing them is extremely time-consuming when an expedient response is necessary. To improve this process, effective summarization models are a valuable tool for humanitarian response experts as they provide digestible overviews of essential information in secondary data. This paper focuses on extractive summarization for the humanitarian response domain and describes and makes public a new multilingual data collection for this purpose. The collection – called MultiHumES– provides multilingual documents coupled with informative snippets that have been annotated by humanitarian analysts over the past four years. We report the performance results of a recent neural networks-based summarization model together with other baselines. We hope that the released data collection can further grow the research on multilingual extractive summarization in the humanitarian response domain.

## introduction the disaster risk management cycle consists of four stages: mitigation, preparedness, response, and recovery @xcite . the review of secondary data sources (i.e., reports, news, and other forms of text data) is embedded in all these stages, with varying levels of importance from stage to stage. the work of secondary data review is characterized by a high and ever-increasing amount of information to be analyzed. at the same time, typically, only a small workforce is available to analyze such information. early in the response phase, namely in the first 72 hours after a disaster strikes, secondary data review to gain situational awareness is essential, as it brings to light which type of relief activities to undertake. after this stage, primary data collection (such as surveys) begins while still supported by the secondary data review processes. the disaster information cycle by its part consists of the collection, collation, analysis, dissemination, decision-making, and reporting stages. an effective summarization tool can provide meaningful support for the collection stage when analysts prioritize what documents to read first (i.e., offering an overview of a document). in the collation stage, when analysts need to take and merge the most important findings from several documents, and even in the reporting stage, analysts are asked to bring in a few sentences containing the key findings of what they have written. an auto summarization system aims to provide analysts with a starting point from which they can continue their work rather than replace it. such a system can significantly save time in the overall disaster information cycle. in this work, we take the initial steps of creating such an extractive summarization system for humanitarian responders by curating and releasing the novel publicly available multilingual humanitarian response dataset for extractive summarization (multihumes) foot_0 . this collection is annotated by humanitarian experts and consists of the data related to various disasters around the globe that occurred in the last four years. our contribution in this work occurs in two ways: the collection and consultation process for the possible release of the dataset and the dataset curation for performing extractive summarization tasks. the dataset consists of approximately 50k documents in three languages: english, french, and spanish. among these documents, approximately 35k are annotated with informative snippets and can be used for the training and evaluation of ex-tractive summarization models. we evaluate the performance of lead4, textrank @xcite ) -an unsupervised graph-based model -, and neusum @xcite ) -a recently created supervised neural model in the dataset. to the best of our knowledge, this is the first multilingual dataset released for summarization in the humanitarian domain. the most similar initiatives are done by @xcite , which released a social-media based dataset for classification in the humanitarian domain, and by appen, which released a set of short messages from social media and news articles for classification in the humanitarian domain 2 . this article is structured as follows: section 2 describes the background, annotation process, curation process, and main statistics of the multihumes collection. section 3 explains the experiment design of the summarization models on the dataset, and section 4 presents and discusses the results. ## multihumes collection this section first provides the background on the humanitarian response domain and the ecosystem from which the collection originated. then it explains the annotation process, the curation process and finalizes presenting key statistics from the collection. ## experiment design this section explains the baseline extractive summarization models and their corresponding parameter settings. ## results and discussion the french corpus had the lowest performance, but the fact that the oracle presents a low performance indicates that this result may be more related to the corpus's nature or the preprocessing of the data rather than the model. lead4 bases its success on the position of the relevant sentences. textrank bases its success on the content of the relevant sentences. neusum higher performance may be explained by the joint encoding of the position information and the content information of a sentence. it is also key to remember that we truncated each document to a maximum length of 200 sentences because of our gpu capacity. this meant that reports such as the humanitarian needs overview with around 60 pages (900 sentences) were reduced to 200 sentences and were therefore not well summarized. our neural approach for the summarization task of humanitarian documents was useful for setting a precedent of how good these models are in the domain. however, until higher computational resources are more readily available, it would be necessary to have a simpler model to treat longer documents. ## conclusions and future work automatic summarization in the humanitarian domain is crucial for supporting fast and effective responses to crises. to facilitate this process, we provide multihumes, a novel multilingual collection for extractive summarization in the humanitarian response domain. the collection enables the training and evaluation of machine/deep learning-based models, revealing new horizons for research in this domain. the collection consists of approximately 50k documents, from which 35k have related annotated snippets by experts. we test and evaluate the performance of three strong baselines on the collection. we consider the following points as potential future directions. first, a key aspect of automatic summarization for the humanitarian response domain is the human evaluation of the output. it is indeed important to understand whether the reported performance results correlate with the evaluation of domain experts. the second direction is learning multilingual models, trained over all the data, and investigating whether this approach can improve performance, especially in languages with a smaller amount of available training data.