C lass B ases at the CASE -2022 Multilingual Protest Event Detection Task: Multilingual Protest News Detection and Automatically Replicating Manually Created Event Datasets (2022)

Abstract: In this report, we describe our ClassBases submissions to a shared task on multilingual protest event detection. For the multilingual protest news detection, we participated in subtask-1, subtask-2 and subtask-4 which are document classification, sentence classification and token classification. In subtask-1, we compare XLM-RoBERTa-base, mLUKE-base and XLM-RoBERTa-large on finetuning in a sequential classification setting. We always use a combination of the training data from every language provided to train our multilingual models. We found that larger models seem to work better and entity knowledge helps but at a non-negligible cost. For subtask-2, we only submitted an mLUKE-base system for sentence classification. For subtask-4, we only submitted an XLM-RoBERTa-base for token classification system for sequence labeling. For automatically replicating manually created event datasets, we participated in COVID-related protest events from the New York Times news corpus. We created a system to process the crawled data into a dataset of protest events.

## introduction a shared task on multilingual protest event detection at case-2022 is the second installment from the previous event at case-2021 about sociopolitical and crisis events detection @xcite . the shared task focuses on protest events where people complain, put their objections, or display their unwillingness to a course of action whether that action is from an authority or a government @xcite . as in the previous installment, this shared task organizes the automated multilingual protest event detection pipeline into multiple subsequent steps at different granularity levels as four subtasks, document classification, sentence classification, event sentence coreference identification, and event extraction. moreover, the shared task contains many languages in many different magnitudes of data sizes, from ten thousand data points to hundreds of data points to no data points. in other words, many settings are varying from full training to lowresource training to few-shot learning to zero-shot learning. the shared task provides a full training setting for english, spanish and portuguese on the scale of ten thousand data points for english and thousands of data points for spanish and portuguese. we participate in the first, second, and fourth subtasks. we build our system solutions upon huggingface's multilingual transformer language models @xcite , specifically, xlm-roberta language models @xcite and mluke multilingual transformer language models with entity embedding @xcite . we also participated in creating covid-related protest event datasets from the new york times news corpus @xcite . the codes for our systems are open-sourced and available at our github repository foot_0 . ## models as in the ibm mnlp team report @xcite , whose systems top-scored in most subtasks of the previous case-2021, we consider xlm-roberta language models (xlm-r) @xcite : the architecture of our systems. we concatenated data from all languages and randomly sample them into batches. the batches are inputs to our models. the model part consists of a tokenizer, a multilingual language model, and a sequential classifier, all are from the huggingface's library @xcite . for subtask4, we replace a sequential classifier with a token classifier. the entity embeddings are cross-lingual mappings of entities learned from wikipedia. the language model part was pretrained as a masked language model and the entity embedding part was pretrained in a masked entity prediction task. despite the performance gains on entity-related downstream tasks, a major limitation of incorporating entity embeddings is the large memory footprint. that is, using only an mluke-base model requires about the same gpu memory as an xlm-roberta-large model. ## experimental results all of our experiments were done in the google colab setting on nvidia tesla t4 gpus. we used the batch size in the range of 16 -36 and the learning rate for an adamw optimizer @xcite in the set of {2.5e-5, 5e-5} for all experiments. we considered a linear annealing scheduler. also, adding a warm-up step does not make any difference so we set the warm-up step to zero in all experiments. except otherwise stated, we concatenated the given training data in all languages as our combined training set for every subtask. we also employed the early stopping with zero patience training strategy schema @xcite . we varied the training epoch until the training metric saturated with manual monitoring, and then stopped right at the end of that epoch. however, we mostly tried with one or two candidate numbers of training epochs since training large language models takes a few hours and gooogle's colab gpu time just runs out. ## conclusions this report describes our systems for a shared task on multilingual protest event detection at case-2022. we compared a small multilingual language model (xlm-roberta-base), a knowledge-based multilingual model (mluke-base), and a large multilingual language model (xlm-robertalarge). from all experimental results, we observed consistent outperforms from xlm-roberta-large over smaller language models, xlm-robertabase, and mluke-base. therefore, we concluded that language model capacity matters a lot for multilingual tasks. also, we observed that mluke-base mostly outperforms xlm-roberta-large. hence, incorporating entity knowledge helps improve performance but with a nonnegligible computational cost. from our visualizations, we found that our finetuned multilingual language models cram data from various languages into the same space by their class information.