Challenges and Applications of Automated Extraction of Socio-political Events from Text ( CASE 2021): Workshop and Shared Task Report (2021)

Abstract: This workshop is the fourth issue of a series of workshops on automatic extraction of socio-political events from news, organized by the Emerging Market Welfare Project, with the support of the Joint Research Centre of the European Commission and with contributions from many other prominent scholars in this field. The purpose of this series of workshops is to foster research and development of reliable, valid, robust, and practical solutions for automatically detecting descriptions of socio-political events, such as protests, riots, wars and armed conflicts, in text streams. This year workshop contributors make use of the state-of-the-art NLP technologies, such as Deep Learning, Word Embeddings and Transformers and cover a wide range of topics from text classification to news bias detection. Around 40 teams have registered and 15 teams contributed to three tasks that are i) multilingual protest news detection detection, ii) fine-grained classification of socio-political events, and iii) discovering Black Lives Matter protest events. The workshop also highlights two keynote and four invited talks about various aspects of creating event data sets and multi- and cross-lingual machine learning in few- and zero-shot settings.

## introduction today, the unprecedented quantity of easily accessible data on social, political, and economic processes offers ground-breaking potential in guiding data-driven analysis in social and human sciences and in influencing policy-making processes. the need for precise and high-quality information about a wide variety of events ranging from political violence, environmental catastrophes, and conflict to international economic and health crises has rapidly escalated @xcite . governments, multilateral organizations, and local and global ngos present an increasing demand for this data to prevent or resolve conflicts, provide relief for those that are afflicted, or improve the lives of and protect citizens in a variety of ways. for instance, black lives matter protests foot_0 , conflict in syria 2 and covid-19 related events 3 are only a few examples where we must understand, analyze, and improve the real-life situations using such data. a recent report from reliefweb 4 clearly demon-strates that the number of wars and other armed conflicts is on an increasing trend. in particular, the so-called internationalized conflicts are on a rise in the last two decades. in this situation, it is important to provide solutions for situation awareness, using various branches of artificial intelligence (ai), natural language processing (nlp), machine learning (ml), and advanced statistical methods. in this clue, event detection and extraction plays an important role, because of its capacity to detect conflict developments in news and social media and to extract important information about them. such information involves the quantity and the profiles of the victims, the participating entities, the conflict dynamics, its spatio-temporal characteristics, the weaponry used, as well as infrastructural, technical and human impact. this information extracted through various nlp methods can throw light on the intensity and the trend development of each conflict, as it is reflected in the media. event detection has been used by political analysts to write their daily situation reports for decision makers, to create long-term analyses, as well as for conflict forecasting and prediction. automation offers scholars not only the opportunity to improve existing practices, but also to vastly expand the scope of data that can be collected and studied, thus potentially opening up new research frontiers within the field of socio-political events, such as political violence and social movements. event information collection has long been a challenge for the nlp community as it requires sophisticated methods in defining event ontologies, creating language resources, and developing algorithmic approaches @xcite @xcite . we believe that this workshop and the shared task contribute strongly towards putting emphasis on this important technology, providing a gathering point for scientists and developers in nlp, ai, conflict studies and related areas. social and political scientists have been creating event databases such as acled @xcite , embers @xcite , gdelt @xcite , icews (o'brien, 2010), mmad @xcite , phoenix, poldem @xcite , speed @xcite , ter-rier @xcite , and ucdp @xcite for decades. these projects and the new ones increasingly rely on machine learn-ing (ml) and nlp methods to deal better with the vast amount and variety of data in this domain @xcite . nonetheless, automated approaches suffer from major issues like bias, low generalizability, class imbalance, training data limitations, ethical issues, and lack of recall quantification which affect the quality of the results and their use drastically @xcite @xcite . moreover, the results of the automated systems for socio-political event information collection may not be comparable to each other or not of sufficient quality @xcite . socio-political events are varied and nuanced. both the political context and the local language used may affect whether and how they are reported. therefore, all steps of information collection (event definition, language resources, and manual or algorithmic steps) may need to be constantly updated. this leads us to a series of challenging questions such as: do events related to minority groups are represented well? are new types of events covered? are the event definitions and their operationalization comparable across systems? we organize the workshop on challenges and applications of automated extraction of socio-political events from text (case 2021) 5 and the shared task sociopolitical and crisis events detection foot_3 to seek answers to these and related questions, to inspire innovative technological and scientific solutions for tackling the aforementioned issues, and to quantify the quality of the automated event extraction systems. moreover, the workshop aims to trigger a deeper understanding of the performance of the computational tools used and the usability of the resulting socio-political event datasets. the workshop is co-located with the joint conference of the ## accepted papers the workshop attracted 21 submissions. the competition was high and 7 of them were accepted based on reviewer evaluations, which vary between ing approach for detecting incidents from industrial reports. incidents in industries have huge social and political impacts. however, automated analysis of repositories of incident reports has remained a challenge. due to absence of event annotated datasets for industrial incidents authors employ a transfer learning based approach. a detailed analysis is provided on how amount of data utilized affect pre-training and why pre-training improves the performance. data is gathered from aviation and construction incident reports. different deep learning methods are evaluated for the task, including bilstm and transfer learning. transfer learning consistently outperforms the baseline and achieves f1 measure of 0.81. radford (2021) presents a study on geocoding and a new data set. geocoding is an important subtask of event detection, in which the goal is to find the geographic coordinates associated with event descriptions. the paper presents an "end-to-end probabilistic model" for geocoding from text data. a novel data set has been created for evaluating the performance of geocoding systems. the output of the new model is compared with a state-of-the-art model, called mordecai. the comparison clearly shows an improvement provided by the proposed model. @xcite report on a study on the political bias in hong kong published news reporting about protest events. the paper reports on lexical differences between home and western news sources about protests happening in hong kong in the period 1998-2020. experiments on topic modeling, sentiment analysis, lexical distribution and comparative lexical analysis between westernand hong kong-based sources reveal a bias in the reporting from the hong kong press. the evaluation reveals that during the anti-extradition law amendment bill movement reports from hong kong made fewer references to police violence compared to the western media. the study also reveals that the lexical contexts of salient keywords changed in hong kong sources when the movement emerged. @xcite describe an algorithm for event argument detection and aggregation. the paper reports on document level aggregation of the following argument types: time, place, casualties, after-effect, reason, and participant. the arg-fuse algorithm is based on a bert based active learning classifier, which identifies whether a pair of event arguments is redundant, and a biased text rank argument ordering function. authors report f1 measure of 0.61, which beats all the other 5 baseline algorithms with which the argfuse performance is compared. ## shared task: socio-political and crisis events detection the work on event database creation comprises of three steps that are collecting events, classifying them, and measuring utility of the system output, which is an event database, against ground-truth. each of these steps contains pitfalls and subject to limitations. for instance, the data source utilized maybe biased or a ground-truth may not be available. although aforementioned issues in sociopolitical and crisis event studies have been studied by numerous scholars for decades to date, there are still no answers or solutions to them @xcite @xcite @xcite . therefore, we aim at contributing to the understanding and resolution of event database creation via quantifying performance of the state-of-the-art text processing systems in the shared task socio-political and crisis events detection. 7 the shared task consists of three tasks that are on collection (task 1), classification (task 2) @xcite , and evaluation (task 3) of event databases. shared task and submission details are reported in the overview papers of the tasks @xcite @xcite and the system description papers in this proceedings respectively. we provide a summary of the tasks and the findings in the following subsections. ## keynotes kristine eck and elizabeth boschee will deliver the keynote talks. @xcite addresses the responsibility of the scholars that create event datasets to define and apply what is right, suggests data sources alternative to news data that may report event information inconsistently, and emphasizes the need for interdisciplinary collaboration for creating data sets that advance conflict studies. @xcite ; boschee (2021) share the concerns addressed by prof. eck and presents a detailed study that compares various approaches for utilizing multilingual data in a cross-lingual zero-shot setting to improve quality of the event datasets. ## invited talks the workshop contains an invited talks session as well. the authors of the papers published in findings of acl and related to workshop theme are invited to present their work in this session. the papers are @xcite propose an event-driven trading strategy that predicts stock movements by detecting corporate events from news articles; halterman et al. (2021) introduce the indiapo-liceevents corpus-all 21,391 sentences from 1,257 times of india articles about events in the state of gujarat during march 2002; halterman and radford (2021) show utility of "upsampling" coarse document labels to finegrained labels or spans for protest size detection; and tsarapatsanis and aletras (2021) discuss the importance of academic freedom, the diversity of legal and ethical norms, and the threat of moralism in the computational law field. ## conclusion this workshop is the fourth event from a series of workshops on automatic extraction of sociopolitical events from news, organized by the emerging market welfare project, with the support of the joint research centre of the european commission, with contributions from many other prominent scholars in this field. the purpose of this series of workshops is to foster research and development in the area of event extraction of socio-political events. the topics cover a wide range of applications and technologies: event detection via text classification, detection of news bias, fake news detection, modality analysis through syntactic parsing, event argument extraction and aggregation, a new geocoding algorithm and the creation of a new geocoding dataset. most of the papers are dedicated to protest events, one paper is about industrial reports, and one paper discusses generic events, not related to the socio-political topic. the papers in this issue of the workshop make use of state-of-the-art nlp technologies, such as deep learning, word embeddings and transformers. most of the papers use the bert model: some use the pre-trained existing models, others train domain-specific ones, and one of the paper introduces a modified version of bert. most papers use bert embeddings as features in their models and one paper discusses an algorithm, which uses a full syntactic parser. sentiment analysis is used in one paper, which studies the political bias of the news. the shared task results shed light on critical aspects of the automated socio-political extraction and evaluation methodology. 8