IBM MNLP IE at CASE 2021 Task 1: Multigranular and Multilingual Event Detection on Protest News (2021)

Abstract: In this paper, we present the event detection models and systems we have developed for Multilingual Protest News Detection - Shared Task 1 at CASE 2021. The shared task has 4 subtasks which cover event detection at different granularity levels (from document level to token level) and across multiple languages (English, Hindi, Portuguese and Spanish). To handle data from multiple languages, we use a multilingual transformer-based language model (XLM-R) as the input text encoder. We apply a variety of techniques and build several transformer-based models that perform consistently well across all the subtasks and languages. Our systems achieve an average F_1 score of 81.2. Out of thirteen subtask-language tracks, our submissions rank 1st in nine and 2nd in four tracks.

## introduction event detection aims to detect and extract useful information about certain types of events from text. it is an important information extraction task that discovers and gathers knowledge about past and ongoing events hidden in huge amounts of textual data. the case 2021 workshop @xcite focuses on socio-political and crisis event detection. the workshop defines 3 shared tasks. in this paper we describe our models and systems developed for "multilingual protest news detection -shared task 1" @xcite . shared task 1 in turn has 4 subtasks: event extraction on news has long been popular, and benchmarks such as ace @xcite and ere @xcite annotate event triggers, arguments and coreference. most previous work has addressed these tasks separately. @xcite also focused on detecting social-political events, but case 2021 has added more subtasks and languages. case 2021 addresses event information extraction at different granularity levels, from the coarsest-grained document level to the finestgrained token level. the workshop enables participants to build models for these subtasks and compare similar methods across the subtasks. the task is multilingual, making it even more challenging. in a globally-connected era, information about events is available in many different languages, so it is important to develop models that can operate across the language barriers. the common languages for all case task 1 subtasks are english, spanish, and portuguese. hindi is an additional language for subtask 1. some of these languages are zero-shot (hindi), or low resource (portuguese and spanish) for certain subtasks. in this paper, we describe our multilingual transformer-based models and systems for each of the subtasks. we describe the data for the subtasks in section 2. we use xlm-r @xcite 2020) as the input text encoder, described in section 3. for subtasks 1 (document classification) and 2 (sentence classification), we apply multilingual and monolingual text classifiers with different window sizes (sections 4 and 5). for subtask 3 (event sentence coreference identification), we use a system with two modules: a classification module followed by a clustering module (section 6). for subtask 4 (event extraction), we apply a sequence labeling approach and build both multilingual and monolingual models (section 7). we present the final evaluation results in section 8. our models have achieved consistently high performance scores across all the subtasks and languages. ## data the data for this task has been created using the method described in @xcite . the task is multilingual but the data distribution across languages is not the same. in all subtasks there is significantly more data for english than for portuguese and spanish. there is no training data provided for hindi. as there are no official train and development splits, we have created our own splits. the details are summarized in for the final submissions, we use all the provided data, and train various types of models (multilingual, monolingual, weakly supervised, zero-shot) with details provided in the appropriate sections. ## multilingual transformer-based framework for all the subtasks we use transformer-based language models @xcite as the input text encoder. recent studies show that deep transformer-based language models, when pretrained on a large text corpus, can achieve better generalization performance and attain state-ofthe-art performance for many nlp tasks @xcite @xcite . one key success of transformer-based models is a multi-head self-attention mechanism that can model global dependencies between tokens in input and output sequences. due to the multilingual nature of this shared task, we have applied several multilingual transformerbased language models, including multilingual bert (mbert) @xcite , xlm-roberta (xlm-r) @xcite , and multilingual bart (mbart) @xcite . our preliminary experiments showed that xlm-r based models achieved better accuracy than other models. hence we decided to use xlm-r as the text encoder. we use huggingface's pytorch implementation of transformers @xcite . xlm-r was pre-trained with unlabeled wikipedia text and the commoncrawl corpus of 100 languages. it uses the sentencepiece tokenizer @xcite with a vocabulary size of 250,000. since xlm-r does not use any cross-lingual resources, it belongs to the unsupervised representation learning framework. for this work, we fine-tune the pre-trained xlm-r model on a specific task by training all layers of the model. ## subtask 1: document classification to detect protest events at the document level, the problem can be formulated as a binary text classification problem where a document is assigned label "1" if it contains one or more protest event(s) and label "0" otherwise. various models have been developed for text classification in general and also for this particular task @xcite . in our approach we apply multilingual transformerbased text classification models. ## subtask 2: sentence classification to detect protest events at the sentence level, one can also formulate the problem as a binary text classification problem where a sentence is assigned label "1" if it contains one or more protest event(s) and label "0" otherwise. as for document classification, we use xlm-r as the input text encoder. the difference is that for sentence classification, we set max seq length (a parameter of the model that specifies the maximum number of tokens in the input) to be 128; while for document classification where the input text is longer, we set max seq length to be 512 (for documents longer than 512 tokens, we truncate the documents and only keep the first 512 tokens). we train the models for 10 epochs, taking 80 minutes to train a model with training data from all the languages on a single nvidia v100 gpu. for this subtask we are provided with training data in english, spanish and portuguese, and evaluation is on test data for all three languages. the sizes of the train/development/test sets are shown in as for document classification, we build two types of xlm-r based sentence classification models: a multilingual model and monolingual models. the results of these models on the development sets are shown in we prepared three submissions on the test data for each language (english, spanish, portuguese), similar to those described in section 4.2. ## subtask 3: event sentence coreference identification typically, for the task of event coreference resolution, events are defined by event triggers, and are usually marked in a sentence. two event triggers are considered coreferent when they refer to the same event. in this task, however, the gold event triggers are not provided; the sentences are deemed coreferent, possibly, on the basis of any of the multiple triggers that occur in the sentences being coreferent, or if the sentences are about the same general event that is occurring. given a document, this event coreference subtask aims to create clusters of coreferent sentences. there is good variety in the research for coreference detection. cattan et al. (2020 ) rely only on raw text without access to triggers or entity mentions to build coreference systems. barhom et al. (2019) do joint entity and event extraction using a feature-based approach. yu et al. (2020) use transformers to compute the event trigger and argument representation for the task. following the recent work on event coreference, our system is comprised of two parts: the classification module and the clustering module. the classification module uses a binary classifier to make pair-wise binary decisions on whether two sentences are coreferent. once all sentence pairs have been classified as coreferent or not, the clustering module clusters the "closest" sentences with each other with agglomerative clustering, using a certain threshold, a common approach for coreference detection @xcite ; choubey and huang (2017); @xcite ). agglomerative clustering is a popular technique for event or entity coreference resolution. at the beginning, all event mentions are assigned their own cluster. in each iteration, clusters are merged based on the average inter-cluster link similarity scores over all mentions in each cluster. the merging procedure stops when the average link similarity falls below a threshold. formally, given a document d with n sentences {s 1 , s 2 , ..., s n }, our system follows the procedure outlined in algorithm 1 while training. the input to the algorithm is a document, and the output is a list of clusters of coreferent event sentences. the evaluation of the event coreference task is based on the conll coref score @xcite , which is the unweighted average of the fscores produced by the link-based muc @xcite , the mention-based b 3 @xcite , and the entity-based ceaf e @xcite metrics. as there is little spanish and portuguese data, we use it as a held out development set. our system uses xlm-r large pretrained model to obtain token and sentence representations. pairs of sentences are concatenated to each other along with the special begin-of-sentence token and separator token as follows: we feed the bos token representation to the binary classification layer to obtain a probabilistic score of the two sentences being coreferent. once we have the score for all sentence pairs, we call the clustering module to create clusters using the coreference scores as clustering similarity scores. we use xlm-r large pre-trained models. we trained our system for 20 epochs with learning rate of 1e-5. we experimented with various thresholds and chose 0.65 as that gave the best performance on development set. it takes about 1 hour for the model to train on a single v100 gpu. ## subtask 4: event extraction the event extraction subtask aims to extract event trigger words that pertain to demonstrations, protests, political rallies, group clashes or armed militancy, along with the participating arguments model en-dev es-dev pt-dev s1 80.57 --s2 80.25 64.09 69.67 s3 80.87 -- in such events. the arguments are to be extracted and classified as one of the following types: time, facility, organizer, participant, place or target of the event. formally the event extraction task can be summarized as follows: given a sentence @xmath0, w 2 , .., w n } and an event label set @xmath1, t 2 ..., t j }, identify contiguous phrases (w s , ..., w e ) such that l(w s , .., w e ) ∈ t . most previous work @xcite ; nguyen and grishman ( 2018 )) for event extraction has treated event and argument extraction as separate tasks. but some systems @xcite treat the problem as structured prediction and train joint models for event triggers and arguments. lin et al. (2020) built a joint system for many information extraction tasks including event trigger and arguments. following the work of m'hamdi et al. (2019); awasthy et al. (2020), we treat event extraction as a sequence labeling task. our models are based on the stdbert baseline in awasthy et al. ( 2020 ), though we extract triggers and arguments at the same time. we use the iob2 encoding @xcite to represent the triggers and the argument labels, where each token is labeled with its label and an indicator of whether it starts or continues a label, or is outside the label boundary by using b-label, i-label and o respesctively. the sentence tokens are converted to token-level contextualized embeddings {h 1 , h 2 , .., h n }. we pass these through a classification block that is comprised of a dense linear hidden layer followed by a dropout layer, followed by a linear layer mapped to the task label space that produces labels for each token {l 1 , l 2 , .., l n }. the parameters of the model are trained via cross entropy loss, a standard approach for transformerbased sequence labeling models @xcite . this is equivalent to minimizing the negative log-likelihood of the true labels, ## final results and discussion the final results of our submissions and rankings are shown in to recap, our s1 systems are multilingual models trained on all three languages. s2 are monolingual models: for subtasks 1 and 2 they are languagespecific, but for subtasks 3 and 4 they are englishonly. s3 is an ensemble system with voting for subtasks 1, 2 and 4, and an extra-feature system for subtask 3. among our three systems, the multilingual models achieved the best scores in three tracks, the monolingual models achieved the best scores in six tracks, and the ensemble models achieved the best scores in four tracks. for subtask 1 (document-level classification), the language-specific monolingual model (s2) performs better than the multilingual model (s1) for english, portuguese and spanish; while for subtask 2 (sentence-level classification), the multilingual model outperforms the language-specific monolingual model for portuguese and spanish. this shows that building multilingual models could be better than building language-specific monolingual models for finer-grained tasks. the monolingual english-only model (s2) performs best on all three languages for subtask 3. this could be because the multilingual model (s1) here was trained with machine translated data. adding the trigger, time and place markers (s3) did not help, even when these features showed promise on the development sets. the multilingual model (s1) does better for spanish and portuguese on subtask 4. this is consistent with our findings in @xcite where training multilingual models for named entity recognition, also a token-level sequence la-belling task, helps improve performance across languages. as there is much less training data for spanish and portuguese, pooling all languages helps. ## conclusion in this paper, we presented the models and systems we developed for multilingual protest news detection -shared task 1 at case 2021. we explored monolingual, multilingual, zero-shot and ensemble approaches and showed the results across the subtasks and languages chosen for this shared task. our systems achieved an average f 1 score of 81.2, which is 2 f 1 points higher than best score of other participants on the shared task. our submissions ranked 1 st in nine of the thirteen tracks, and ranked 2 nd in the remaining four tracks.