A Meta-framework for Spatiotemporal Quantity Extraction from Text (2022)

Abstract: News events are often associated with quantities (e.g., the number of COVID-19 patients or the number of arrests in a protest), and it is often important to extract their type, time, and location from unstructured text in order to analyze these quantity events. This paper thus formulates the NLP problem of spatiotemporal quantity extraction, and proposes the first meta-framework for solving it. This meta-framework contains a formalism that decomposes the problem into several information extraction tasks, a shareable crowdsourcing pipeline, and transformer-based baseline models. We demonstrate the meta-framework in three domains—the COVID-19 pandemic, Black Lives Matter protests, and 2020 California wildfires—to show that the formalism is general and extensible, the crowdsourcing pipeline facilitates fast and high-quality data annotation, and the baseline system can handle spatiotemporal quantity extraction well enough to be practically useful. We release all resources for future research on this topic at https://github.com/steqe .

## introduction events are often associated with quantities -how many covid-19 patients are on ventilators, how many people are injured during protests, or how large is the extent of a wildfire. we often need to figure out the type of an event, and where and when it happened for these quantities for coherent discussion of public policy on sociopolitical events in rapidly evolving situations: "19 deaths" is different from "19 recoveries;" "19 deaths in a small city yesterday" apparently describes a more severe situation than "19 deaths in the whole country last month." however, until dedicated channels are established, these quantities are typically first reported on social media and local news articles, which then have to slowly make their way to some aggregate location for decision-makers to use. this calls for a general framework to extract and analyze quantities associated with events, so that we can automatically summarize quantitative information from news streams, rapidly respond to emergencies, investigate incidents, and potentially combat misinformation through comparisons with trusted sources. prior work on events focused on extracting event mentions, attributes, and relationships @xcite @xcite @xcite @xcite , and paid little attention to quantities associated with those events, which presents an opportunity to perform targeted information extraction on these quantity events. this paper studies spatiotemporal quantity extraction (steqe): finding quantities of certain types and extracting their associated times and locations. we develop a general meta-framework to help researchers overcome challenges and extend to new domains easily. specifically, the contributions of this meta-framework are: task formulation we draw on ideas from existing nlp tasks to create the first formalism that defines steqe as four information extraction tasks: quantity recognition, typing, spatial grounding, and temporal grounding. while each of these has analogues in the literature, our combination of them into a complete picture of quantity events is novel. ## steqe the steqe problem aims to extract information about quantity events in text, consisting of four parts: determining which numerical expressions actually correspond to events ( §2.1), the type of the event that a quantity is referring to ( §2.2), where that event happened ( §2.3), and the temporal extent to which the quantity refers ( §2.4). note that for each of these subparts, there could have been other definition and formulation choices. we describe our formalism's design choices, and discuss why they would lead to better-defined learning problems and more reliable data collection, along with their limitations and how to extend our formalism for more specialised applications. ## data annotation we have walked through the definition of the tasks in our steqe framework, with discussions on various design choices. next we explain how to collect annotations via this framework in practice. ## model quantity recognition is a typical span selection problem and we use the standard token classification model based on bert (large, cased) @xcite that comes with huggingface @xcite . for typing, spatial, and temporal grounding, we use the t5-large language model @xcite for its flexibility across tasks and easy domain transfer. we format data from each task to fit into t5's sequence to sequence (seqto-seq) nature. specifically, for each quantity, the input sequence to t5 is the string of the previous 3 sentences, the current sentence with a special marker token right before the quantity span, the next 3 sentences, the title, and document creation time (dct). for typing, the output sequence is a single token representing each label mapped from a reserved vocabulary. for spatial grounding, the output sequence is the location names from the highest hierarchy to the lowest ended by an end-ofsentence (eos) marker. for temporal grounding, the output sequence is the start time followed by the end time. both times are either "unknown" or a date string in iso 8601 format (e.g., "2021-01-15"). we view the start time of an overall quantity as "unknown". to get complete date predictions, we enforce the decoding length to be at least 12 and use a date parser to find "unknowns" or dates. ## experiments in our evaluation of quantity recognition using the aforementioned bert model on a random set of 300 sentences (100 from each domain), we find the precision 99% for all domains, and the recall 95% (covid), 87% (blm), and 87% (fire). the recall is slightly lower because of poor performance on article words ("a" and "an"). however, since most missed quantities are not associated with event types that we are interested in (e.g., "[a] post office" or "[a] comment"), the adjusted recall is 98% (covid), 94% (blm), and 93% (fire) if we do not consider those irrelevant quantities. t5 (in-domain) on quantity typing, t5 improves by a large margin over the naive baseline in all domains. the naive baseline performs reasonably well on spatial grounding at the state level (82-92% em-state across three domains), but often fails to provide more granular information at the city level (58-74% em-city). this is expected because a city mentioned close to the quantity does not necessarily mean that the quantity is for the city. 9 this phenomenon also varies across domains: blm protests were in a few major cities, the em-city score of the naive method is thus relatively high (74%), while for calfornia wildfires, there were more cities to choose from, leading to a low em-city of 58%. in contrast, t5 can produce more granular information at the city level, and maintain a relatively stable score across domains (70-81% em-city). as for temporal grounding, due to the nature of news articles, the naive baseline that treats all quantities as an overall quantity ending on dct yields reasonably good performances in all domains; but for quantities with a non-trivial start time or end time, the naive baseline largely fails. ## related works existing nlp works on events have focused on detection (e.g., detecting life and business events; ace ( 2005 2014 )). there is also a line of recent works specifically on temporal semantics: time expression extraction and normalization @xcite , temporal relation extraction @xcite @xcite , temporal common sense @xcite , temoral slot filling @xcite , and timeline construction @xcite @xcite . these tasks may help understanding the temporal aspects of events in general, but they cannot directly associate temporal values with quantities, and calls for a dedicated framework such as steqe. prior works on quantities either focus on math calculations @xcite or common sense reasoning (e.g., mass distribution of animals; @xcite ), and not on quantity events and the associated spatiotemporal extent studied in this work. existing works on spatial semantics have focused on natural language navigation @xcite , human-machine interaction @xcite , dialogue systems @xcite , and clinical analysis @xcite . works on geocoding @xcite ) map spatial mentions to coordinates, which can be applied to our work for finer geolocation mapping. @xcite proposes a qa dataset that considers time and location of the question when judging answer correctness, which may benefit from our information extraction framework. a recent work from zong et al. (2020), which extracts covid-19 related events from tweets, is closely related to our work. besides that they worked on tweets instead of news articles, the key differences are: (1) instead of span selection used in zong et al. ( 2020 ), we propose formalisms deeper into the spatiotemporal extent of quantity events and capture more nuances in spatiotemporal semantics; (2) we show that our steqe framework generally applies to multiple domains and not only for the covid-19 pandemic; (3) we release our entire data collection pipeline on crowdaq for public use and extension. ## discussion as §5 shows, the performance bottleneck of steqe is mainly at temporal grounding: with almost perfect quantity recognition and very good typing and spatial grounding results, temporal grounding performance is typically much lower than the other tasks. while typing and spatial grounding are ready for practical research into fewand zero-shot settings along the lines of what is done in entity typing @xcite @xcite , temporal grounding still requires more investigation even in in-domain settings. why is temporal grounding so challenging? first, news articles tend to mention many overall quantities ending on publication time, leading to imbalanced datasets. for instance, 86% in fire fall into this category, leaving little training data for other quantities; in contrast, this number is only 32% in blm, and the s-n and e-n scores are much higher in blm than those in fire. second, temporal grounding often requires reasoning, an effect known to be difficult in many works on temporal semantics @xcite . for instance in fig. 4 , to figure out the time span of "80," we need to understand that (1) it happened on "sunday" (2) the "sunday" is a sunday in the past instead of in the future, and (3) it is most likely the most recent sunday instead of earlier ones. dct: tuesday, 06/16/2020 text: black lives matter demonstrators in a tiny ohio town...sunday. the small demonstration has about [80] people, organized by local bethel residents. another direction to improve on steqe is to aggregate from multiple articles, given that the same quantity or similar quantities are typically covered by multiple sources. cross-document event coreference has many unique difficulties (e.g., see upadhyay et al. (2016); bugert et al. (2020) ), but knowing the quantity event type, location, and time span may make it relatively easy to find coreference to strengthen one's belief in its prediction, or demote outliers that are likely wrong predictions. the proposed steqe framework may also be used to detect misinformation and perhaps in social science studies too. for instance, we have anecdotes where a website mistakenly reported virginia's covid-19 case number on apr 2, 2020 to be 17k, while the correct number was 1.7k; we also found signs that news agencies might have mentioned case numbers in new york city less frequently after a sharp increase, but turned to report case numbers in new jersey in april 2020. these social science analyses are beyond the scope of this work, but the examples above point to interesting potential uses of these information extraction systems. ## conclusion many important news events are associated with quantities. with practicality in mind, we dive deep into the semantics of quantity events and propose a meta-framework for spatiotemporal quantity extraction: we formulate the problem as four information extraction tasks which lead to quick and reliable data annotation via crowdsourcing; we also build a t5 baseline to study the difficulties of the task and discuss transfer learning opportunities. we use this meta-framework to build datasets on three separate sociopolitical events: the covid-19 pandemic, blm protests, and california fires. our meta-framework is shown to be readily extensible to different domains of quantity events, an appealing feature for quick response to future events. the new datasets we collect as examples of this framework can also directly contribute to future studies on spatiotemporal quantity extraction.