Multilingual Protest News Detection - Shared Task 1, CASE 2021 (2021)

Abstract: Benchmarking state-of-the-art text classification and information extraction systems in multilingual, cross-lingual, few-shot, and zero-shot settings for socio-political event information collection is achieved in the scope of the shared task Socio-political and Crisis Events Detection at the workshop CASE @ ACL-IJCNLP 2021. Socio-political event data is utilized for national and international policy- and decision-making. Therefore, the reliability and validity of these datasets are of the utmost importance. We split the shared task into three parts to address the three aspects of data collection (Task 1), fine-grained semantic classification (Task 2), and evaluation (Task 3). Task 1, which is the focus of this report, is on multilingual protest news detection and comprises four subtasks that are document classification (subtask 1), sentence classification (subtask 2), event sentence coreference identification (subtask 3), and event extraction (subtask 4). All subtasks had English, Portuguese, and Spanish for both training and evaluation data. Data in Hindi language was available only for the evaluation of subtask 1. The majority of the submissions, which are 238 in total, are created using multi- and cross-lingual approaches. Best scores are above 77.27 F1-macro for subtask 1, above 85.32 F1-macro for subtask 2, above 84.23 CoNLL 2012 average score for subtask 3, and above 66.20 F1-macro for subtask 4 in all evaluation settings. The performance of the best system for subtask 4 is above 66.20 F1 for all available languages. Although there is still a significant room for improvement in cross-lingual and zero-shot settings, the best submissions for each evaluation scenario yield remarkable results. Monolingual models outperformed the multilingual models in a few evaluation scenarios.

## introduction every day across the globe, hundreds of different socio-political protest events against various decisions taken by the respective governments or authorities take place. these events are of interest to political scientists, policy makers, democracy watchdogs and other stakeholders for multiple reasons including analysing the nature, scope and extent of such events, forming public opinion about various causes, gauging the state of freedom and democracy across different nations and others. however, manually keeping track of such events at a national level itself is a very challenging task and it is more so if we are trying to get a sense of these events across the globe. given this, automated methods of collecting and, possibly, processing protest news events from multiple countries and locations gain great significance. but the automated identification and collection of such events in multiple languages also comes with its own set of significant challenges. this task was designed to address some of these challenges.. the task of event information detection, in general, could be divided into multiple subsequent steps and the efficiency at each of these steps could drastically affect the quality of the resultant event database. thus, we believe one must consider a complete pipeline including the following steps i) classification of documents and sentences as relevant or not (in the sense that whether they describe an event or not -in this specific case event is a protest event); ii) identification of the sentences that provide information about the same event; and iii) extraction of event information. finally the resultant database of the events should be tested against a manually created list of events to evaluate the performance of the state-of-the-art systems on this task. we have formulated these different steps into three inter-dependent tasks -task 1 is a multilingual protest news detection task, task 2 complements the first task with fine-grained semantic event classification @xcite using data reported by @xcite and task 3 evaluates the performance of the systems developed for task 1 on a real-world scenario, in this it specifically evaluates the system for the task of identifying the events surrounding black lives matter movement, using data from twitter and new york times @xcite . in order to benchmark the state-of-the-art in these three tasks, we organized the shared task socio-political and crisis events detection foot_0 . the shared task is held in the scope of the workshop challenges and applications of automated extraction of socio-political events from text (case 2021) 2 @xcite that is held at the joint conference of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (acl-ijcnlp 2021). 3 we report results of the task 1 that is multilingual protest news detection in this report. task 1 follows extracting protests from news (protestnews) and event sentence coreference identification (esci) tasks that were organized at conference and labs of the evaluation forum (clef 2019) @xcite and automated extraction of socio-political events from news (aespen 2020) at language resources and evaluation conference (lrec) @xcite respectively. the protestnews and esci were monolingual tasks comprising only english data from various countries and evaluated for cross-context generalization of automated text processing systems across texts collected from different countries. this edition of the shared task series focuses on language generalization of the event information collection systems in four languages viz. english, hindi, portuguese, and spanish. the task 1 we present in this report follows all the steps we find essential for event information collection in a multilingual setting. it is divided into the following subtasks. ## related work automated socio-political event information collection has a long history @xcite . many event ontologies such as idea @xcite , cameo @xcite , acled @xcite , and plover foot_2 have been proposed in this domain. these ontologies facilitated development of automated event information collection tools such as mpeds @xcite , pe-trarch @xcite , tabari, and bbn accent, embers @xcite . the databases that are created using automated methods at various levels are gdelt (leetaru and schrodt, 2013), icews (o'brien, 2010), mmad @xcite , phoenix, poldem @xcite , speed @xcite , terrier @xcite , and ucdp @xcite . although majority of this work is on western countries and english language, there are considerable number of similar studies on collecting socio-political event information from text originated from countries other than western countries and in languages other than english @xcite . the main data source of the event information has been text of news articles. but the use of social media posts has gradually increased in recent times @xcite . the application of state-of-the-art automation using machine learning and computational linguistics techniques requires gold standard annotated corpora that can be utilized for the task and benchmarks that facilitate comparison of the proposed methods for protest event information collection @xcite . however, there are only a few corpora shared for research purposes in this domain @xcite @xcite and to the best of our knowledge, there is no benchmark available. our efforts via this task establishes a common ground for comparison and benchmarking in a multilingual setting. the multilingual text processing has become a critical target in computational linguistics and machine learning. tackling this task enables us to collect information about global events that are reported and to trace occurrence of similar events in many languages. moreover, this technology facilitates event information collection from local sources, which provide detailed information about events. new benchmark data sets such as xtreme @xcite and system proposals such as mbert @xcite , xlm (lample and conneau, 2019), mbart @xcite , and xlm-r @xcite have demonstrated promising results on various tasks @xcite . multilingual embedding creation is the other major research line, in which the approaches such as laser @xcite and labse @xcite have been proposed. these methodological advancements extend the exploration space for detecting event information. consequently, this technology contributes to the resolution of the popularity or ideological bias of the sources toward popular and mainstream events both at global and local levels. in general, it is not an optimum decision to work with a single language due to biases, absence of event information in a single source or international sources etc. we must invest in generalizability and multilinguality of the event information collection systems and therefore in the current task we incor-porate these aspects as well. by design, zero-or few-shot learning is required to tackle some subtask and language combinations @xcite in this task since the released data set contained relatively small training data in spanish and portuguese and no training data in hindi. thus the final evaluation provides some insights into these approaches for contentious socio-political event data collection and classification task. ## annotation the multilingual version of the corpus glo-con gold @xcite , which was reported as containing data only in english, is utilized in this task. this corpus is created by random sampling from news archives and double annotation @xcite for the data in english, spanish, and portuguese. there are document, sentence, and token level annotations that are performed on the whole news articles. the quality of the annotations are ensured by a detailed annotation manual foot_3 , adjudications, spot-checks, and semiautomated quality checks before the next level of annotation starts. a cascaded annotation workflow is applied. for instance, quality of the document level annotations is ensured before the sentence level annotation starts. the inter-annotator aggrements (iaa) that are measured using krippendorf's alpha @xcite are .75 and .65 in average for document-and sentence-level annotations. the token level iaa is between .35 and .60 for the information types in scope. all disaggrements are resolved by the annotation supervisor. moreover, spot-checks and semi-automated error corrections have fixed 10% of the annotation errors in total @xcite . the document and sentence level annotations yielded the data for subtasks 1 and 2 respectively. the token level annotations produced the data for subtasks 3 and 4. the data in hindi is prepared applying a slightly different methodology but using the same annotation manual. a native graduate student from india has annotated these articles at the document level. twenty hindi newspapers and periodicals available on the web are used as sources for this data set. this data set contains all possible articles and editorials related to ongoing farmer protest in india against the three farm bills (1.bill on agri market, 2.bill on contract farming, and 3.bill relating to commodities) passed by the government of india in august, 2020. 6 the current annotated data set cov-ers equal proportion of articles from each source, which are twenty except the periodical panchjanya, which has only 19 articles. all articles are searched and collected manually from web pages of each newspaper and periodical with the metadata date, date of article retrieval, url, location of incident, and location of newspaper. overall, the news articles used in this task are obtained from china and south africa in english, from brazil in portuguese, from argentine in spanish, and from india in english and hindi. the annotation team consists of graduate students in social and political sciences. students from turkey, brazil, and india have annotated text in english, spanish and portuguese, and hindi respectively. these students are trained on contentious politics of their target country and annotation methodology before they started the annotation. news reports that are not related to a target country are excluded from the token level annotations in order to improve precision of the annotations. ## task description task 1 consists of four subtasks that are at document, sentence, and token levels. the subtasks are as follows. subtask 1 aims at classifying news articles. if the document reports an event that has happened or is ongoing, it should be labelled as relevant. scheduled events, speculations, and anything else should be marked as irrelevant. subtask 1 is a binary classification problem. subtask 2 has the same aim as subtask 1 but for sentences of a document. 7 a sentence should have some token(s) that qualify as event trigger or a reference to an event trigger in another sentence. subtask 3 is about determining event sentences that provide information about the same event. all event sentences in a document are clustered according to the events they report. subtask 4 marks all tokens in an event sentence based on the information they hold 7 . the event trigger and its arguments such as participant, place, target, organizer, time, and facility name are annotated. the event trigger can be a coreferent of a trigger in another sentence. the subtasks are multilingual by means of comprising data in english, portuguese, and spanish languages both for training and evaluation of the automated text processing systems. moreover, the tasks are a few-shot scenario since portuguese and e2%80%932021_indian_farmers%27_protest, accessed on june 9, 2021. 7 the annotators see the whole document during the annotation spanish training data is significantly less than english data. finally, the subtask 1 includes a zeroshot setting in which participants do not have access to data in hindi language, but they should predict documents in hindi language. @xcite have showed that, although, event information collection could be performed utilizing systems developed only for subtask 4 with potential contribution of the systems developed for subtask 3 in principle, this setting is not possible in practice due to challenge of reliable annotation of event information at token level for development and evaluation of event extraction systems. document and sentence annotation significantly facilitates reliable annotation of event information at token level. moreover, authors have demonstrated a considerable increase in f1 in case document and sentence classification systems are applied before token level event extraction. thus, we consider application of these subtasks in this order indispensable for reliable collection of event information. ## data description we share text data in english, spanish and portuguese for training and evaluation. also, there is data in hindi language for evaluation of the subtask 1. finally, participants are free to use any additional data they may think that will help to improve their systems. this section provides details on the format, size, and preparation of the data shared with the participants. moreover, we describe how data across subtasks depend on each other and how we deal with copyright issues in the subsection on the data preparation. ## evaluation although the subtasks form a coherent flow, task participants can focus on one or more of them. therefore, participants can choose the tasks or subtask(s) they would like to participate in. participants have access to all of the data for all tasks and subtasks. any combination of these resources to achieve high performance for any of the tasks is allowed. for instance, task 1 data could be used to potentially improve the performance on task 2 and vice versa. participants had access to the test data for a week and could submit up to five submissions for each subtask and language combination. the best score of each team is reflected to the leaderboard. 10 additional submissions are allowed (after the competition ended) on a separate codalab page foot_7 in case participating teams would like to run additional experiments or create multiple submissions of the same system for measuring standard deviation of their systems. however, the additional submission page allows only one submission for each language and subtask combination per day. f1-macro is calculated on the predictions on the test data for the subtasks 1 and 2. we use a python implementation foot_8 of the original foot_9 conlleval evaluation script for subtask 4. the subtask 3 is evaluated using scorch -a python implementation of conll-2012 average score for the test data @xcite . 14 . we carry out separate evaluation for each subtask using the test data for each language separately. ## baseline systems we created baseline models for the subtasks 1 and 2 in english, portuguese, and spanish. document classification is a challenging task. for simplicity, we have done document classification on the summaries of documents, which are the most important sentence in the document generated using the lexrank extractive summarization method @xcite . thus document summarization task was converted into an important part of the sentence classification task pipeline. as such, the input text for the document classification is a sentence rather than a set of sentences. we have used an attention (i.e. transformer) @xcite based neural network model for feature representation @xcite and multilingual sentence representations @xcite for the subtasks 1 and 2 with three languages -english, spanish and portuguese. among available approaches @xcite @xcite , @xcite provide efficient representation for sentences for 50+ languages from various language families. the main motivation of using the multilingual approach is to learn efficient representation for the low resource (non-english) languages. specifically, we have used 'distiluse-base-multilingual-cased' for learning sentence representation of the three languages. we have also used language-specific sentence representation @xcite for the english language. specifically, for the experiment, we used 'paraphrase-distilroberta-base-v1', which is a 'distilbert-base-uncased' model fine-tuned on a large dataset of paraphrase sentences. we apply a linear support vector machine (svm) classifier trained on these features. the multilingual representation yields competitive results in our experiments for the language-specific representation in english language classification. we have used 70% of the training data to train the model and 30% of the data to validate the models for subtasks 1 and 2. for the document classification task, the validation scores are 74.85 for the english language, 49.27 for the spanish language, and 56.67 for the portuguese language. the test scores are 76.78, 64.45, 64.13 for english, portuguese and spanish respectively. for the sentence classification task (subtask 2) the f1-macro on the validation data is 79.67 with the language-specific representation of the english language. with multilingual representation, the validation f1-macro is 76.90 for the english language, 73.93 for the portuguese, and 73.42 for the spanish language. the score on the test data is 67.08, 67.42 , and 66.75, for english, portuguese and spanish respectively. ## conclusion and future work this shared task shows that multilingual and cross-lingual approaches perform surprisingly well for subtasks of protest event information collection. we observed that merging the training data from multiple languages improves the performance. moreover, the performance of the first and second submissions, which are prepared by two different teams, for the cross-lingual zero-shot setting for subtask 1 in hindi language are 78.77 and 78.53 in terms of f1-macro, thereby demonstrating the promising suitability of the approach for zero-shot multilingual setting. another significant outcome is that "ibm mnlp ie" has outperformed all other teams by more than 4 points of f1-macro in all languages in subtask 4, which is the most challenging subtask. monolingual models outperforms multilingual models in case sufficient training data (subtask 1, english) or additional further pre-training data is available (subtask 2, english). these conditions are satisfied mostly for evaluation scenarios pertaining to english language. although, multilingual models yield best performance in some scenarios, the monolingual models ranked second or third place. automated event information collection approaches are prone to major issues like bias toward majority class and popular content and limited generalizability that affect reliability and validity of them @xcite @xcite @xcite @xcite @xcite . we consider this benchmark as the first step to obtain comparable results across various automated approaches in a multilingual setting. we form a basis for increasing variety of the data that can be utilized for developing and evaluating event information collection systems by extending the language data that has various levels of availability such as few-shot and zero-shot settings. furthermore, this benchmark allows determination of the most suitable text processing approaches for this task by identifying the performance levels that can be achieved applying recent technology. last but not least, the random sampling of the corpus utilized in the shared task enables realistic recall quantification that has been challenging to measure to date @xcite . we will be extending available training data and include additional data in different languages in the future iterations of this benchmark.