AMU - EURANOVA at CASE 2021 Task 1: Assessing the stability of multilingual BERT (2021)

Abstract: This paper explains our participation in task 1 of the CASE 2021 shared task. This task is about multilingual event extraction from news. We focused on sub-task 4, event information extraction. This sub-task has a small training dataset and we fine-tuned a multilingual BERT to solve this sub-task. We studied the instability problem on the dataset and tried to mitigate it.

## introduction event extraction is becoming more and more important as the number of online news increases. this task consists of extracting events from documents, especially news. an event is defined by a group of entities that give some information about the event. therefore, the goal of this task is to extract, for each event, a group of entities that define the event, such as the place and time of the event. this task is related but still different from named entity recognition (ner) as the issue is to group the entities that are related to the same event, and differentiate those related to different events. this difference makes the task harder and also complicates the annotation. in the case of this shared task, the type of events to extract is protests @xcite . this shared task is in the continuation of two previous shared tasks at clef 2019 @xcite and aespen @xcite . the first one deals with english event extraction with three sub-tasks: document classification, sentence classification, and event information extraction. the second focuses on event sentence coreference identification, whose goal is to group sentences related to the same events. this year, task 1 is composed of the four aforementioned tasks and adds another difficulty: multilinguality. this year's data is available in english, spanish, and portuguese. thus, it is important to note that there is much more data in english than in the other languages. for the document classification sub-task, to test multilingual capabilities, hindi is available on the testing set only. we have mainly focused on the last sub-task (event information extraction), but we have also submitted results for the first and second sub-tasks (document and sentence classification). we used multilingual bert @xcite , henceforth m-bert, which is a model known to obtain near state-of-the-art results on many tasks. it is also supposed to work well for zero-or-few-shot learning on different languages @xcite . we will see the results on these sub-tasks, especially for sub-task 4 where the training set available for spanish and portuguese is small. thus, one of the issues with transformer-based models such as m-bert is the instability on small datasets @xcite . the instability issue is the fact that by changing some random seeds before the learning phase but using the same architecture, data and hyper-parameters the results can have a great variance. we will look at some solutions to mitigate this issue, and how this issue is impacting our results for sub-task 4. foot_0 2 tasks and data sub-tasks 1 and 2 can be seen as binary sequence classification, where the goal is to say if a given sequence is part of a specific class. in our case, a classifier must predict whether a document contains information about an event for sub-task 1 or if a sentence contains information about an event for sub-task 2. document and sentence classification tasks, subtasks 1 and 2, are not our main research interest. moreover, the datasets provided for these tasks are less interesting (reasonable amount of training data). on the other hand, sub-task 4 not only has less training data available but also requires more finegrained token-based prediction. the goal of subtask 4 is to extract event information from snippets that contain sentences speaking about the same event. @xcite have defined that an event has the following information classes (example in figure 1 ): thus, not all the snippets contain all the classes, and they can contain several times the same classes. each information can be composed of one or several adjacent words. each snippet contains information related to one and only one event. as the data is already separated into groups of sentences related to the same event, our approach consists of considering a task of named entity recognition with the aforementioned classes. multilingual bert has already been used for multilingual named entity recognition and showed great results compared to state-of-the-art models @xcite . the data is in bio format @xcite , where each word has a b tag or an i tag of a specific class or an o tag. the b tag means beginning and marks the beginning of a new entity. the tag i means inside, which has to be preceded by another i tag or a b tag, and marks that the word is inside an entity but not the first word of the entity. finally, the o-tag means outside, which means the word is not part of an entity. ## system overview our model is based on pre-trained multilingual bert @xcite . this model has been pretrained on multilingual wikipedia texts. to balance the fact that the data is not equally distributed between all the languages the authors used exponential smoothed weighting to under-sample the most present languages and over-sample the rarest ones. this does not perfectly balance all the languages but it reduces the impact of low-resourced languages. the authors of the m-bert paper shared the weights of a pretrained model that we use to do finetuning. fine-tuning a model consists of taking an already trained model on a specific task and using this model as a starting point of the training for the task of interest. this approach has reached stateof-the-arts in numerous tasks. in the case of m-bert, the pre-training tasks are masked language modeling (mlm) and next sentence prediction (nsp). to be able to learn our task, we add a dense layer on top of the outputs of m-bert and learn it during the fine-tuning. all our models are fine-tuning all the layers of m-bert. the implementation is the one from hugging-face's 'transformers' library @xcite . to train it on our data, the model is fine-tuned on each sub-task. ## conclusion as we can see in it is then hard to compare to other models as we have chosen to use one model for all the languages and we do not know the other approaches.