Extracting Victim Counts from Text (2023)

Abstract: Decision-makers in the humanitarian sector rely on timely and exact information during crisis events. Knowing how many civilians were injured during an earthquake is vital to allocate aids properly. Information about such victim counts are however often only available within full-text event descriptions from newspapers and other reports. Extracting numbers from text is challenging: numbers have different formats and may require numeric reasoning. This renders purely tagging approaches insufficient. As a consequence, fine-grained counts of injured, displaced, or abused victims beyond fatalities are often not extracted and remain unseen. We cast victim count extraction as a question answering (QA) task with a regression or classification objective. We compare tagging approaches: regex, dependency parsing, semantic role labeling, and advanced text-to-text models. Beyond model accuracy, we analyze extraction reliability and robustness which are key for this sensitive task. In particular, we discuss model calibration and investigate out-of-distribution and few-shot performance. Ultimately, we make a comprehensive recommendation on which model to select for different desiderata and data domains. Our work is among the first to apply numeracy-focused large language models in a real-world use case with a positive impact.

## introduction timely and accurate information during crisis events is crucial for rescue operations and the allocation of humanitarian aid @xcite . however, crisis information is often scarce, subjective, or biased, which renders reported numbers in text extremely important @xcite @xcite . for instance, the count of injured or missing people provides quantitative information about the catastrophic impact of an earthquake. in this work, we focus on human victims in crisis events, e.g., fatalities in floods, herein referred to as victim counts. a reliable estimate of victim counts is helpful during crisis @xcite , and also post-crisis, benefiting research to diversify measures of crisis intensity. as of now, most intensity measures are either limited to event types @xcite , fatality counts @xcite or both @xcite . more fine-grained measures such as injured, displaced, or abused victims are not captured in most popular databases and remain unmonitored @xcite @xcite . many victim counts are reported in full-text form within event descriptions in news media. this makes their systematic collection and analysis technically complex. manual extraction of victim counts from text is very labor-intensive and does not scale to big data collections @xcite . computerized approaches such as the event coding software tabari @xcite and petrarch2 @xcite focus on extracting actor and event types. they rely on lambda calculus and syntactic pattern matching, but disregard mentions of victim counts. as we will show, parsing-based approaches perform decently well at extracting explicitly reported victim counts. they can identify the mention of the count "5" in "5 people were injured". however, they are often inadequate when the description implies a correct count -for example, from the description that "one logger was shot but survived", a human reader may infer that one person is injured. since neither a count nor the injury is mentioned explicitly, a parsing-based system may fall short. another difficulty stems from the fact that the counts can be reported in many, different formats. a reported count may be digit-based or spelled out, define an exact quantity or a range as in "dozens of people were injured". as a conse-quence, formulating the task of victim count extraction is not an easy endeavor ( §3). most prior work assumes a setting where the count is explicitly mentioned in an event description @xcite @xcite . such settings can be tackled by sequence labeling models that select a relevant span from the given description. however, if the victim count does not appear verbatim, as in the above "one logger" example, models with some form of abstract reasoning capacity may be needed @xcite . recently, large language models have shown promising results in answering number-focused questions with and without explicit mentions of relevant numbers @xcite @xcite . this paper is concerned with studying these different approaches ( §4): as baselines, we compare regular expression, dependency parsing, and semantic role labeling. we consider the nt5 @xcite model as a representative numeracyenhanced pre-trained language model. we use the representation of this model in a generation, a classification, and a regression setting. we evaluate all models along three dimensions: accuracy ( §5), reliability ( §6), and robustness ( §7). we find that the fine-tuned language model outperforms the baseline models, especially when the victim count extraction requires reasoning. reliability and robustness are particularly important in high-stake, human-centric tasks such as victim count extraction @xcite @xcite . model reliability indicates to which extent model behavior can be trusted within decision-making settings @xcite . one dimension of reliability is model calibration which indicates if a model's confidence is aligned well with it making correct predictions @xcite . while calibration has been widely studied for classification, we add to the discussion of calibrated regression @xcite and generation settings @xcite . finally, the dimension of robustness describes how stably a model performs. for instance, when the training set is limited or when the test data is out-of-distribution, a less robust model will forfeit more of its predictive performance. to shed light on this dimension, we conduct experiments in few-shot learning and out-of-distribution settings. we conclude with an application to showcase the extraction of fine-grained and highly specialized types of victim counts. lastly, we discuss the benefits and drawbacks of the different approaches to assist practitioners in choosing the most suitable task formulation and model. ## data we use publicly available datasets covering natural disasters and armed conflicts, namely: (1) world atrocities dataset (wad) @xcite , (2) non-violent and violent campaigns and outcomes 3.0 (navco) @xcite , and (3) european media monitor (emm) (jrc science @xcite . for each dataset, we use the event text description and two types of victim counts: the death count and the injury count that we refer to as "wad death" or "wad injury". we pre-process the data by removing the samples with missing values (nan) in the victim counts . for emm, we only consider samples with a non-zero victim count since "0" is over-represented. ## task formulation in this section, we discuss some questions and challenges faced in formulating the task of extracting victim counts from event descriptions. we justify some of the choices we make and describe why it is not possible to have a single formulation that fits all needs: is the victim count always present in the text? victim counts can be expressed in various ways in the text. when the count is expressed explicitly in the text, say "5 people were injured", a span extraction model can effectively extract the injury count 5. however, in certain cases, a single explicit number might not be mentioned, and the victim count needs to be logically or algebraically inferred from the text. consider the description "a 4-year-old girl and her mother were found dead"; a model would need to logically deduce that the victim count of death is 2. to handle this, we not only look at span extraction models but also experiment with models that can understand the text at a deeper level and produce a victim count. is the victim count always a single number? often, in the event description, the victim count is described as a range, such as "at least 330 people died", or in vague terms, like "dozens were injured". additionally, even within a description, the victim counts for the same event can be varying, possibly because of recording the counts from different sources. this makes extracting a single exact count almost impossible. in such cases, the best a model can do is to output a close estimate of the actual victim count. another solution would be to provide a range within which the count could lie. for a humanitarian section deciding on the quantity of aid to be deployed, a range might suffice over a single exact count. to account for this, we also look at models that are trained to output a range by classifying the victim counts into a set of binned categories. ## models in §4.1, we introduce baselines models that parse an event description and heuristically extract a victim count. we then specify the model implementation for the different task formulations in §4.2. ## accuracy of counts extraction we begin by evaluating the efficacy of our proposed methods for victim count extraction. we examine the model accuracy by comparing baselines and the fine-tuned model with a generation objective ( §5.1). we then show the results of using classification and regression formulations ( §5.2). ## evaluating reliability another important dimension is reliability which we evaluate through the lens of calibration ( §6.1). as we approach the task with multiple formulations, calibration analysis is especially needed to understand whether a model is calibrated ( §6.2), and how post-hoc calibration techniques may adjust models to be better calibrated ( §6.3). ## application: overlooked victim types most event datasets feature only one column detailing victim counts. this column typically quantifies fatalities, as they are considered least ambivalent and most important @xcite . the armed conflict location & event data project (acled) @xcite recently published curated datasets containing violence against healthcare workers, media personnel, and women. con-sidering the acled dataset on political violence targeting women & demonstrations featuring women, we find that more than 85% of events have zero fatalities. this means, many forms of violence remain non-quantified, often those against "marginalized" groups of society. using the methods presented in this work, we can extract much more fine-grained victim types such as "injured women" and "abducted women". to this end, we rely on the nt5-gen model that we fine-tuned on the navco data, without specifically asking for "women". in fig. 4 , we present exemplary two-month time series of events in syria. we find that our model has a higher recall than precision on the ground truth annotations for fatality counts. this may be desirable since we would like to avoid overlooking true victim counts. ## discussion this work surveys different task formulations of victim count extraction and inspects desiderata like accuracy, reliability, and robustness of different models. we now summarize our findings and conclude which approach performs best under which circumstances (tab. 5). some of the parsing-based approaches have the advantage of requiring no ground truth annotations of the extracted victim counts. this means, there is no need for training, but instead, a manually curated list of patterns and rules has to be assembled. the regex approach, for instance, has minimum requirements regarding hardware, but writing regex patterns is very time-intensive and can be prone to mistakes. overall, the baseline models shine when it comes to speed, and they perform reasonable when victim counts are explicitly mentioned. yet they fail at complex reasoning. for instance, when asking for the count of deaths in "one child and four women lost their lives", all baselines mistakenly output "1". this is where language model-based methods have a competitive edge. the fine-tuned nt5-gen model has high accuracy both in exact-match metric and relative error metric. surprisingly, it is also well-calibrated and relatively robust in the few-shot and out-of-distribution setting. this performance comes at the costs of reduced speed, the requirement of large amounts of training data, and the need for resources like gpus to be deployed on a large scale. comparing classification and regression objec-tives, we conclude that classification is easier to handle. in most settings, it may be sufficient to extract a range rather than an exact number anyways. in comparison to generation, in classification and regression settings, models show higher calibration errors and require post-hoc calibration to adjust the model confidence. ## related works this work interfaces with related works from different disciplines to improve the measurement of crisis intensity. it draws inspiration from recent advancements in question answering models with a focus on numbers and math word problems. this includes number-enhanced language models more generally. our work also connects with model calibration in natural language processing (nlp) more generally. ## conclusion we presented victim count extraction, a challenging and impactful task. the task can be tackled using different formulations and models. models should be evaluated along different dimensions such as accuracy, reliability, and robustness. we survey this ambiguity of victim count extraction, identify promising directions, and discuss outlooks and applications.