Discovering Black Lives Matter Events in the U nited S tates: Shared Task 3, CASE 2021 (2021)

Abstract: Evaluating the state-of-the-art event detection systems on determining spatio-temporal distribution of the events on the ground is performed unfrequently. But, the ability to both (1) extract events “in the wild” from text and (2) properly evaluate event detection systems has potential to support a wide variety of tasks such as monitoring the activity of socio-political movements, examining media coverage and public support of these movements, and informing policy decisions. Therefore, we study performance of the best event detection systems on detecting Black Lives Matter (BLM) events from tweets and news articles. The murder of George Floyd, an unarmed Black man, at the hands of police officers received global attention throughout the second half of 2020. Protests against police violence emerged worldwide and the BLM movement, which was once mostly regulated to the United States, was now seeing activity globally. This shared task asks participants to identify BLM related events from large unstructured data sources, using systems pretrained to extract socio-political events from text. We evaluate several metrics, accessing each system’s ability to identify protest events both temporally and spatially. Results show that identifying daily protest counts is an easier task than classifying spatial and temporal protest trends simultaneously, with maximum performance of 0.745 and 0.210 (Pearson r ), respectively. Additionally, all baselines and participant systems suffered from low recall, with a maximum recall of 5.08.

## introduction typically, performance evaluations of automated event coding engines are carried out with respect to benchmarks made of annotated linguistic units (e.g. clause, sentence or document). while this is crucial in order to factorize the individual, linguistic subtasks composing the event extraction process, it does not estimate the overall usability of machinecoded event data sets for micro-level modelling of social processes, particularly in the domain of socio-political and armed conflict, where spatial analysis has become standard. the complex dynamics of the black lives matter movement and its varied media coverage by news outlets and social media make it a particularly relevant use case for assessing the capability of automated, event extraction systems to model socio-political processes. the task 3: "discovering black lives matter events" foot_0 organized in the context of the challenges and applications of automated extraction of socio-political events from text (case) 2021 workshop aims at doing so by challenging event extraction (ee) engines to extract a collection of protest events from two heterogeneous text collections (i.e., news and social media) and then measuring a number of spatiotemporal correlation coefficients against a curated gold standard data set of protest incidents from the blm movement. during may and june of 2020, protests occurred across the globe in response to the murder of george floyd, an unarmed black man, by derek chauvin, a white police officer. in the u.s., the number of locations holding demonstrations related to this murder outnumbered any other demonstration in u.s. history @xcite . these events were more often than not associated with the black lives matter (blm) movement, either (1) directly through organizing or (2) indirectly through the slogan "black lives matter" or shared political agendas such as police abolition and protests against police violence towards black communi-ties. since its inception in 2013, the black lives matter movement, a loose network of affiliated organizations, has organized demonstrations around a large number of police shootings and killings and sought to raise awareness of systematic violence against black communities. while support for black lives matter has varied over its lifetime @xcite , the work done over the past years laid the foundation for the global response seen in the wake of george floyd's murder. this task is the third in a series of tasks at case 2021 workshop @xcite . the first task is concerned with protest news detection at multiple text resolutions (e.g., the document and sentence level) and in multiple languages: english, hindi, portuguese, and spanish @xcite . teams which participated in task 1 were invited to participate in this third task: "discovering black lives matter events in the united states". this task is an evaluation only task, where all models are (1) trained on the data supplied in task 1, (2) applied to the news and social media data (i.e, new york times and twitter data), and (3) evaluated on a manually curated, gold standard blm protest event list. each team's system is compared to simple baselines in order to properly evaluate their accuracy. ## related work summary measures such as precision, recall, and f1 are limited in their capacity to inform about the quality of the predictions of an automated system @xcite . moreover, evaluating capabilities of a system on detecting socio-political events from text requires additional metrics such as spatio-temporal correlation of the system output and the actual distribution of the events @xcite . several studies focused on assessing the correlation of machine-coded event data sets with gold standards based on disaggregated event counts, for example @xcite and @xcite . @xcite applied disaggregation of events incidents across prio-grid geographical cells @xcite to assess the global database of events, language and tone (gdelt) data approximation of the spatio-temporal pattern of conflicts. @xcite adapted this method to administrative units for measuring the impact of event de-duplication on increasing correlation with the armed conflict location and event data (acled) data sets for a number of conflicts in africa. in this report we report on an evaluation task, which we refer as task 3, we provide a detailed analysis of the capabilities of the best performing systems on task 1 @xcite in this respect. we believe this effort will shed light on system performances beyond precision, recall, and f1. ## data the goal of this task is to evaluate the performance of automatic event detection systems on modeling the spatial and temporal pattern of a social protest movement. we evaluate the capability of participant systems to reproduce a manually curated blm-related protest event data set, by detecting blm event reports, enriched with location and date attributes, from a news corpus collection, a twitter collection, and from the union of the two. ## evaluation system performance is evaluated by computing correlation coefficients on event counts aggregated on cell-days, using uniform grid cells of approximately 55 kilometers sides from the prio-grid data set @xcite . we use these analytical measures as a proxy to the spatio-temporal pattern of the blm protest movement. ## results in figure 2 we plot the time series of total daily protest cells for the best performing instance of each system on new york times (left) and twitter (right) data, respectively. we see the systems evaluated on the nyt data failing to pick up both variation in the temporal patterns (i.e., a large number of protests early in late may and early june, which gradually declines with weekly spikes) and the magnitude of the events (i.e, most systems pick up less than 100 events per day). systems evaluated on twitter data pick up more events in late may and early june, but still fail to pick up the magnitude of the events. a more lenient representation of the agreement with gold standard is shown in ## conclusions the goal of the "discovering black lives matter events" shared task was to explore novel performance evaluations of pretrained event detection systems. these systems were applied to large noisy, multi-modal text data sets (i.e., news articles and social media data) related to a specific protest movement, namely, black lives matter. thus, the systems are being evaluated out-of-domain in terms of both data type (i.e., the systems are trained on news data and evaluated on both news and social media) and protest movement context (i.e., the training data are not necessarily related to blm). systems are evaluated in their ability to identify both events across time as well as events their distribution across space. this evaluation scenario proved difficult for all systems participating in the shared task. a major problem, as shown on