Detecting and Geocoding Battle Events from Social Media Messages on the Russo- U krainian War: Shared Task 2, CASE 2023 (2023)

Abstract: The purpose of the shared task 2 at the Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) 2023 workshop was to test the abilities of the participating models and systems to detect and geocode armed conflicts events in social media messages from Telegram channels reporting on the Russo Ukrainian war. The evaluation followed an approach which was introduced in CASE 2021 (Giorgi et al., 2021): For each system we consider the correlation of the spatio-temporal distribution of its detected events and the events identified for the same period in the ACLED (Armed Conflict Location and Event Data Project) database (Raleigh et al., 2010). We use ACLED for the ground truth, since it is a well established standard in the field of event extraction and political trend analysis, which relies on human annotators for the encoding of security events using a fine grained taxonomy. Two systems participated in this shared task, we report in this paper on both the shared task and the participating systems.

## introduction automatic discovery of an event's location is an important sub-task of event extraction: most events occur at a defined location, reported in text. usually the event time can be guessed by the time of the publication of the news article or the social media post and the presence of temporal adverbs. however, it is far more difficult to detect the location: multiple events can be reported in the same story, each with potentially no, one, or multiple locations mentioned in the text @xcite @xcite . event geoparsing, as distinguished from simple geoparsing, is an important part of the event extraction process @xcite @xcite . the purpose of this shared task was to provide a real-world evaluation of event geoparsing and challenge the researchers, working on event detection, to propose solutions for event geocoding. another critical aspect of this evaluation is the comparison between automated and manually curated datasets in line with @xcite and @xcite . our evaluation methodology is based on spatiotemporal correlation, using the prio grid geographical cells @xcite : we measured the correlation between the geographical cells in which armed clashes were detected by the participating systems and the cells containing events from the gold standard data. details about the evaluation methodology are given in the section data set and evaluation methodology. in the previous two years the shared task has featured protest events with complex geographical patterns. this year data, referring to russo ukrainian conflict, features battles situated along the russian ukrainian border. conflict has a different structure than protest. protests are followed instantly by journalists, there is a civilian population, you can get information about the same protest from different news sources. in a military conflict it is difficult to access information as there is much less reporting from open source. and the information is often unreliable and imprecise. conflict or their shape and size can be hidden or difficult to assess. all these are the main reasons why this work is both valuable and difficult. this year we had two submissions, which used two different paradigms to event detection, exhibiting different behaviour: the tma system, a combination of transformer-based classification model and a geoparser, which achieved better correlation and nexus, a rule based system also combined with a geoparser. ## related work socio-political event extraction (spe) has long been a challenge for the natural language processing (nlp) community, as reflected in previous editions of the challenges and applications of automated extraction of socio-political events from text (case) workshops @xcite . specifically, event extraction in the security domain has been identified as an important application area in the automatic information retrieval domain @xcite . similarly, deriving geolocated information from social networks has been seen early identified as an application-rich discipline @xcite . despite the fact that detection and geocoding of events from social media sources have been studied for more than a decade, the field is still vibrant and innovative as advances in artificial intelligence make new approaches possible, and as the evolution of the web and its social media services constitute a "moving target" for automatic information extraction efforts. ## data the goal of this task is to evaluate the performance of automatic discovery of event locations systems on modeling the spatial and temporal patterns of violence in the russo-ukrainian war. the data consists of telegram messages from channels reporting about developments in the russo-ukrainian war. we evaluate the capability of participant systems to reproduce the manually curated russo-ukrainian war-related dataset. ## evaluation the performance of event geolocation is evaluated by computing correlation coefficients on event counts aggregated on cell-days, using uniform grid cells of approximately 55 kilometers sides from the prio-grid data set @xcite . we use these analytical measures as a proxy to the spatio-temporal pattern of violence in the russo-ukrainian war. ## results a more lenient representation of the agreement with gold standard is shown in ## conclusions the purpose of the database replication shared task is to provide a flexible benchmark for evaluation and comparison between event geocoding systems without annotated corpus of events and locations. this year we tested the capabilities of the event detection systems to detect and geolocate battles event type in the russo-ukrainian war from telegram messages in english, comparing the extracted events against a subset of the acled database, dedicated to the war in ukraine. two systems participated this year: each system was an aggregation of two subsystems -event detection and classification and a geoparser, based on different paradigms. the first system was a combination of nexus and the mordecai3 geoparser and the second consisted of event classifier based on xlm-roberta combined with nerone geoparser. xlm roberta and nerone obtained much better correlation in both evaluation scenarios: temporal and spaciotemporal. a conclusion from this year shared task is that tracking armed conflicts is a challenging task, due to the incompleteness of the information: biased because of political consideration or unavailable because of security reasons, and in most case difficult to verify. nevertheless, one of the participating systems achieved a medium level of correlation, which is a satisfactory result, given the difficulty of this year task.