Extended Multilingual Protest News Detection - Shared Task 1, CASE 2021 and 2022 (2022)

Abstract: We report results of the CASE 2022 Shared Task 1 on Multilingual Protest Event Detection. This task is a continuation of CASE 2021 that consists of four subtasks that are i) document classification, ii) sentence classification, iii) event sentence coreference identification, and iv) event extraction. The CASE 2022 extension consists of expanding the test data with more data in previously available languages, namely, English, Hindi, Portuguese, and Spanish, and adding new test data in Mandarin, Turkish, and Urdu for Sub-task 1, document classification. The training data from CASE 2021 in English, Portuguese and Spanish were utilized. Therefore, predicting document labels in Hindi, Mandarin, Turkish, and Urdu occurs in a zero-shot setting. The CASE 2022 workshop accepts reports on systems developed for predicting test data of CASE 2021 as well. We observe that the best systems submitted by CASE 2022 participants achieve between 79.71 and 84.06 F1-macro for new languages in a zero-shot setting. The winning approaches are mainly ensembling models and merging data in multiple languages. The best two submissions on CASE 2021 data outperform submissions from last year for Subtask 1 and Subtask 2 in all languages. Only the following scenarios were not outperformed by new submissions on CASE 2021: Subtask 3 Portuguese & Subtask 4 English.

## introduction we aim at determining event trigger and its arguments in a text snippet in the scope of an event extraction task. the performance of an automated system depends on the target event type as it may be broad or potentially the event trigger(s) can be ambiguous. the context of the trigger occurrence may need to be handled as well. for instance, the 'protest' event type may be synonymous with 'demonstration' or not in a specific context. moreover, the hypothetical cases such as future protest plans may need to be excluded from the results. finally, the relevance of a protest depends on the actors as in a contentious political event only citizen-led events are in the scope. this challenge is even harder in a cross-lingual and zero-shot setting in case training data are not available in new languages. we provide a benchmark that consists of four subtasks and multiple languages in the scope of the 5th workshop on challenges and applications of automated extraction of socio-political events from text at the 2022 conference on empirical methods in natural language processing (case @ emnlp 2022) @xcite . 1 bgenfrhumil: to paraphrase: the work presented in this paper is a continuation of the work initiated in case 2021 task 1 @xcite and consists in adding new documents in already available languages, as well as adding new languages to the evaluation data. task 1 consists of the following subtasks that ensure the task is tackled incrementally: i) document classification, ii) sentence classification, iii) event sentence coreference identification, and iv) event extraction. the training data consist of documents in english, portuguese, and spanish, while the evaluation texts are in english, hindi, mandarin, portuguese, spanish, turkish, and urdu. subtask 1 ensures documents with relevant senses of event triggers are selected. next, subtask 2 focuses on identifying event sentences in a document. discriminating sentences that are about separate events and grouping them is done in subtask 3 @xcite . finally, the sentences that are about the same events are processed to identify the event trigger and its arguments in subtask 4. in addition to accomplishing the event extraction task, the subtask division improves significantly the annotation quality, as the annotation team can focus on a specific part of the task and errors in previous levels are corrected during the preparation of the following subtask @xcite . the significance of this specific task division is twofold: i) facilitating the work with a random sample of documents by first identifying relevant documents and sentences before annotating or processing a sample or a complete archive of documents respectively; ii) increasing the generalizability of the automated systems that may be developed using this data @xcite . the current report is about task 1 in the scope of case 2022. task 2 @xcite and task 3 @xcite complement task 1 by evaluating task 1 systems on events related to covid-19 and detecting causality respectively. the following section, which is section 2 describes the data we use for the shared task. next we describe the evaluation setting in section 3. the results are provided in section 4. finally, the section 5 conclude this report. ## data we used the case 2021 training data as those for case 2022. 2 the case 2022 test data are the union of case 2021 test data and additional new documents in both available and new languages. the new languages are mandarin, turkish, and urdu. the new document level data, which are used to extend case 2021 data, were randomly sampled from mot v1.2 (palen-michel et al., 2022) foot_3 and were annotated by co-authors of this report. documents were annotated by native speakers of the respective language. a single label was attached to each document. the annotation manual followed in the annotation process @xcite was the same as that used in case 2021. the total number of case 2022 documents with labels is 3,870 for english, 267 for hindi, 300 for mandarin, 670 for portuguese, 399 for spanish, 300 for turkish, and 299 for urdu. teams that developed systems for subtasks 2, 3, and 4 evaluated their systems on case 2021 test data. ## evaluation setting we utilized codalab for evaluation of task 1 for case 2022. 4 the evaluation for case 2021 was performed on an additional scoring page foot_5 of the original foot_6 case 2021 codalab page. moreover, we launched an additional scoring page for case 2022 after completion of the official evaluation period. 7 five submissions per subtask and language pair could be submitted in total for case 2022. the additional scoring phase of both case 2021 and case 2022 allow only one submission per subtask and language combination per day. the test data of case 2021 were shared with participants at the same time with the training data. but the case 2022 evaluation data were shared around two weeks before the deadline for submission. the same evaluation scores that are f1-macro for subtasks 1 and 2, conll-2012 foot_8 for subtask 3, and conll-2000 foot_9 script for subtask 4 were utilized. and https://github.com/emerging-welfare/ case-2022-multilingual-event for case 2022. eighteen teams were registered for the task and obtained the training and test data for both case 2022 and case 2021. ten and seven teams submitted their results for case 2021 and case 2022 respectively. seven papers were submitted as system description papers to the case 2022 workshop in total. the scores of the submissions are calculated on two different codalab pages for case 2021 10 and case 2022 11 . the teams that have participated are arc-nlp @xcite , campros @xcite , ceia-nlp (fernandes et al., 2022), classbases (wiriyathammabhum, 2022), eventgraph @xcite , nsut-nlp @xcite , sparta @xcite . we provide details of the results and submissions of the participating teams for each subtask in the following subsections. 12