Tracking COVID -19 protest events in the U nited S tates. Shared Task 2: Event Database Replication, CASE 2022 (2022)

Abstract: The goal of Shared Task 2 is evaluating state-of-the-art event detection systems by comparing the spatio-temporal distribution of the events they detect with existing event databases. The task focuses on some usability requirements of event detection systems in real worldscenarios. Namely, it aims to measure the ability of such a system to: (i) detect socio-political event mentions in news and social media, (ii) properly find their geographical locations, (iii) de-duplicate reports extracted from multiple sources referring to the same actual event. Building an annotated corpus for training and evaluating jointly these sub-tasks is highly time consuming. One possible way to indirectly evaluate a system’s output without an annotated corpus available is to measure its correlation with human-curated event data sets. In the last three years, the COVID-19 pandemic became motivation for restrictions and anti-pandemic measures on a world scale. This has triggered a wave of reactions and citizen actions in many countries. Shared Task 2 challenges participants to identify COVID-19 related protest actions from large unstructureddata sources both from mainstream and social media. We assess each system’s ability to model the evolution of protest events both temporally and spatially by using a number of correlation metrics with respect to a comprehensive and validated data set of COVID-related protest events (Raleigh et al., 2010).

## introduction state-of-the-art evaluation methods for event detection are based on manually coded corpora with annotated document and sub-document units, including annotation of syntactic fragments, such as event reporting verbal phrases, as well as entities having specific semantic roles, such as victim, perpetrator, weapons, etc., see @xcite and @xcite among the others. while this type of benchmarks provide accurate means for measuring the performance of event detection approaches, their development implies significant efforts: many person-hours of annotations by journalists or linguists, which make such annotated corpora limited in number and size and generally developed for the english language only, with a few exceptions @xcite . moreover, such evaluation methods do not assess the overall usability of machine-coded event data sets for micro-level modelling of social processes. also, in the domain of socio-political and armed conflicts, spatio-temporal analysis has become standard and state-of-the-art evaluation methods come short in evaluating exhaustively the spatial and temporal aspects of event detection systems. extracting spatio-temporal information from online text sources has developed in the late 2000's, with the advent of the so-called 'web 2.0' and social networks @xcite , @xcite . since then, applications have been developed in fields as diverse as disaster management @xcite , traffic monitoring (d' @xcite , or fight against crime @xcite . detecting socio-political events (and in particular, protests) emerged as an important use case @xcite , as many applications in this field need to rely on comprehensive, timely and high-quality data that is often not available (e.g. high quality commercial data is produced on a weekly, or even monthly basis, while applications need near-real-time data). this is a gap that case workshops, and this shared task in particular, are aiming to address. the dynamics of the covid-19 protests and their varied media coverage by news outlets and social media makes it a particularly relevant use case for assessing the capability of automated event extraction systems to analyse socio-political processes. the database replicability shared task 2 aims at doing so by challenging event extraction systems to extract a collection of protest events from two heterogeneous text collections (i.e., news and social media). the task's evaluation is done by measuring a number of spatio-temporal correlation coefficients against a gold standard data set of protest incidents, provided by the the armed conflict location and event data (acled) project @xcite . this task is the second in a series of shared tasks at the case 2022 workshop @xcite . the first task is concerned with protest news detection at multiple text resolutions (e.g., the document and sentence level) and in multiple languages: english, hindi, portuguese, and spanish @xcite . task 3 is about detecting event causality in a corpus of sentence pairs that have been annotated with labels on whether there is a causal relations or not between them @xcite . teams which participated in task 1 were invited to participate in this second task. this is an evaluation only task, where all models are (i) trained on the data provided in task 1, (ii) applied to raw news and social media data, specifically gathered for the task (i.e, news collection crawled from the web from various news sources, as well as twitter data), and (iii) evaluated on a manually curated, covid-19 protest event list, gathered from the web page of the acled project @xcite . ## related work some recent studies show that performance measures such as precision, recall, and f1 are limited in their capacity to asses the efficiency of an nlp system @xcite . moreover, evaluating a system on detecting socio-political events from text requires additional metrics such as spatio-temporal correlation of the system output and the actual distribution of the events @xcite . in a detailed study cook and weidmann (2019) demonstrates the usefulness of disaggregating event reports when considering data from event coding. several approaches deal with assessing the correlation of automatically generated event data sets with gold standards based on disaggregated event counts, see example @xcite and schrodt and analytics (2015) among the others. hammond and weidmann (2014) applied disaggregation of events across prio-grid geographical cells @xcite to assess the spatiotemporal pattern of conflicts in the global database of events, language and tone (gdelt) @xcite . in a later work @xcite adapted the aforementioned approach to administrative units for measuring the impact of event de-duplication on increasing correlation with acled event data sets. ## data the goal of this task is to evaluate the performance of automatic event detection systems on modeling the spatial and temporal pattern of a social protest movement. we evaluate the capability of participant systems to reproduce a manually curated covid-19 related protest event data set, by detecting protest event reports, enriched with location and date attributes, from a news corpus collection, a twitter collection (both pre-filtered for covid-19 topic occurrence) and from the union of the two. ## evaluation system performance is evaluated by computing correlation coefficients on event counts aggregated on cell-days, using uniform grid cells of approximately 55 kilometers sides from the prio-grid data set @xcite . we use these analytical measures as a proxy to the spatio-temporal pattern of the coronavirus-related protest events. ## results here, the correlations are between the total number of cells per day where the system found an event vs. the number of cells where at least one event occurred according to the gold standard. the figures show no correlation between the automatically detected conflict cells and the gold standard over time. this is evident from figure 2 , where we plot the time series of total daily protest cells of the participant system against gold standard. we see the system evaluated on news data failing to pick up both temporal variation (i.e., the gradually declining weekly picks of protests from early august through october) and the overall magnitude of the protest movement (e.g., it detects a maximum of less then 10 protest cells per day). while this correlation analysis is overall more tolerant to errors in geocoding foot_2 , a more in-depth error analysis showed that geocoding inaccuracy caused: a. several detected events to wrongly activate the same cells in system output, causing the geographical spread to be significantly lower than gold standard; b. some highly recurrent place names to be wrongly resolved to multiple homonym locations, activating additional cells. here the variables range over the whole set of prio-grid cells included in the us territory and, thus, show the correlation of event numbers across geo-cells, thus better evaluating the system's finegrained geolocation capabilities. as expected, no significant correlation with gold standard is found here either. a more lenient representation of the agreement with gold standard is shown in ## conclusions the goal of the "covid protest events" shared task was to explore novel performance evaluations of pre-trained event detection systems. these systems are applied to large noisy, heterogeneous text data sets (i.e., news articles and social media data) related to a specific protest movement or, as in this case, a wave of protests induced by the coronavirus crisis. thus, the systems are being evaluated out-of-domain in terms of both data type (i.e., the systems are trained on news data and evaluated on both news and social media) and protest movement context (i.e., the training data are not necessarily related to covid-19 pandemic). systems are evaluated on their ability to identify both events across time as well as their distribution across space. this evaluation scenario proved challenging for the system participating in the shared task, confirming the finding from the previous edition @xcite . a major problem, as shown on the low recall at this years shared task may be due to the pre-filtering of the news data for the presence of covid-19 mentions. differently than for an organized protest movement (like black lives data r ρ rmse *classbases news 0.0247 0.0342 0.0101 matter), inferring a relationship of single protest events to the pandemic might not be trivial and thus explicitly stated in the protest news report: therefore, filtering for covid-19 keywords might remove relevant protest reports. however, absolute low recall does not necessarily affect correlation measures as much as inaccurate geocoding of the detected events, as shown. overall, this year's edition of the task was compromised by the low attendance and it is not possible to draw further significant conclusions. we therefore decided to re-open the evaluation window open and welcome further system run submissions. researchers interested to have their models run and evaluated on the input data provided can check the github https://github. com/zavavan/case2022_task2 or contact the authors.