Fine-grained Event Classification in News-like Text Snippets - Shared Task 2, CASE 2021 (2021)

Abstract: This paper describes the Shared Task on Fine-grained Event Classification in News-like Text Snippets. The Shared Task is divided into three sub-tasks: (a) classification of text snippets reporting socio-political events (25 classes) for which vast amount of training data exists, although exhibiting different structure and style vis-a-vis test data, (b) enhancement to a generalized zero-shot learning problem, where 3 additional event types were introduced in advance, but without any training data (‘unseen’ classes), and (c) further extension, which introduced 2 additional event types, announced shortly prior to the evaluation phase. The reported Shared Task focuses on classification of events in English texts and is organized as part of the Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021), co-located with the ACL-IJCNLP 2021 Conference. Four teams participated in the task. Best performing systems for the three aforementioned sub-tasks achieved 83.9%, 79.7% and 77.1% weighted F1 scores respectively.

## introduction the task of event classification is to assign to a text snippet an event type using a domain specific taxonomy. it constitutes an important step in the the views expressed in this article are those of the authors and not necessarily those of erste digital. this paper describes the shared task on finegrained event classification in news-like text snippets. the task is divided into three subtasks: (a) classification of text snippets reporting sociopolitical events (25 classes) for which vast amount of training data exists, although exhibiting slightly different structure and style vis-a-vis test data, (b) enhancement to a generalized zero-shot learning problem @xcite , where 3 additional event types were introduced in advance, but without any training data ('unseen' classes), and (c) further extension, which introduced 2 additional event types, announced shortly prior to the evaluation phase. the reported shared task focuses on classification of events in english texts and is organized as part of the workshop on challenges and applications of automated extraction of socio-political events from text (case 2021) @xcite , co-located with the acl-ijcnlp 2021 conference. four teams actively participated in the task. the main rationale behind organising this shared task is not only to foster research on fine-grained event classification, a relatively understudied area, but to specifically explore robust and flexible solutions that are of paramount importance in the context of real-world applications. for instance, often available training data is slightly different from the data on which event classification might be applied (data drift). furthermore, in real-world scenarios one is interested in quickly tailoring an existing solution to frequent extensions of the underlying event taxonomy. the paper is organized as follows. section 2 reviews prior work. section 3 describes the shared task in more detail. section 4 describes the training and test datasets. next, the evaluation methodology is introduced in section 5. baseline and participant systems are described in section 6. subsequently, section 7 presents the results obtained by these systems, whereas section 8 discusses the main findings of the shared task. we present the conclusions in section 9. ## prior work the research on event detection and classification in free-text documents was initially triggered by the message understanding contests @xcite and the automatic content extraction challenges (ace) @xcite . the event annotated corpora produced in the context of the aforementioned challenges fostered research on various techniques of event classification, which encompass purely knowledge-based approaches @xcite , shallow @xcite and deep machine learning approaches @xcite . multi-lingual event detection and co-reference challenge was introduced more recently in the text analysis conference (tac) in 2016 foot_0 and 2017 foot_1 . in particular, it included an event nugget detection subtask, which focused on detection and finegrained classification of intra-document event mentions, covering events from various domains (e.g., finances and jurisdiction). one could observe in the last decade an ever growing interest in research on fine-grained event classification. @xcite compared svm-based models against word-vectorbased lstms for classification of 10 types of company-specific economic events from news texts, whereas @xcite studied the performance of various models, including ones that exploit word embeddings as features, for detection and classification of natural disaster and crisis events in news articles. jacobs and hoste (2020) reports on experiments of exploiting bert embedding-based models for fine-grained event extraction for the financial domain. although most of the reported work in this area focuses on processing english texts, and in particular, news-like texts as presented in @xcite , some efforts on event classification for non-english language were reported too. for instance, sahoo et al. (2020) introduced a benchmark corpus for fine-grained classification of natural and man-made disasters (28 types) for hindi, accompanied with evaluation of deep learning baseline models for this task. furthermore, an example of fine-grained classification of cyberbullying events (7 classes) in social media posts was presented in @xcite . work on classification of socio-political events and the related shared tasks, although not focusing on fine-grained classification, but covering event types which are in the scope of our task, was presented in @xcite and @xcite . ## task description the overall objective of this shared task is to evaluate the 'flexibility' of fine-grained event classifiers. firstly, we are interested in the robustness vis-a-vis the input text structure, i.e., how classifiers trained on short texts from a curated database perform on news data taken from diverse sources where this structure is somewhat different. this corresponds to subtask 1, which can be considered as a regular classification task. secondly, we wanted to study how classifiers can be made flexible regarding the taxonomy used, with the aim of easily tailoring them for specific needs. this corresponds to subtask 2 and 3, which were framed as generalized zero-shot learning problems: the label set for subtask 2 was announced in advance, while the label set for subtask 3 was announced on the day of the competition. the aforementioned objectives arise from the practical constraints of working with real data, being exposed to data drift and having different users being interested in different facets of the same events. in order to train a fine-grained event classifier, we proposed to use acled @xcite event database and the corresponding taxonomy described in the acled codebook foot_2 , which has 25 subtypes of events related to socio-political events and violent conflicts. acled created a large dataset of events over several years which are manually curated with a common pattern in the way of reporting events and uses a complex event taxonomy: the boundary between the definition of similar classes can be highly intricate, and can seem at point quite arbitrary. nevertheless, acled presented itself as the best possible training material for the specific objectives of this shared task. more precisely, the formal definitions of the different subtasks are as follows: classification of text snippets that are assigned to all acled types plus three unseen (non-acled) types, namely: organized crime, natural disaster and man-made disaster, these new types were announced in advance, but no training data was provided, classification of text snippets that are assigned to two additional unseen event types (diplomatic event and attribution of responsibility) on top of the ones of subtask 2, these new types were not announced in advance. the participating teams had the possibility to submit solutions to any number of subtasks without condition, whereas per subtask up to 5 system responses could be submitted for evaluation. more information on the event types for this shared task is provided in appendix a. ## systems 6.1 baseline systems we provide two baseline systems: a simple character n-gram based l2-regularized logistic regression model and a system based on two transformerbased deep neural representation models. ## evaluation results the results for all submitted system responses for all 3 subtasks in terms of precision, recall and f 1 weighted average scores are provided in the two teams that reported using undersampling due to lack of sufficient computational resources, are also the ones having the overall lowest score on subtask 1. in for subtask 3, the best weighted f 1 score for zero-shot classifier restricted to the 5 new classes only are the following: 65.1% for icic, 52.9% for ibm-mnlp and 26.2% for uncc, c.f. ## conclusions this paper reported on the outcome of the shared task on fine-grained event classification in newslike text snippets that has been organized as part of the workshop on challenges and applications of automated extraction of socio-political events from text (case 2021), co-located with the acl-ijcnlp 2021 conference. further documentation and related material on the reported shared task can be found at https://github.com/emerging-welfare/ case-2021-shared-task/tree/main/task2 , whereas the test dataset alone is also available at: http://piskorski.waw.pl/resources/ case2021/data.zip for research purposes. we believe that the reported results, findings and the annotated test dataset will contribute to stimulating further research on fine-grained event classification.