Event-Based Knowledge MLM for A rabic Event Detection (2022)

Abstract: With the fast pace of reporting around the globe from various sources, event extraction has increasingly become an important task in NLP. The use of pre-trained language models (PTMs) has become popular to provide contextual representation for downstream tasks. This work aims to pre-train language models that enhance event extraction accuracy. To this end, we propose an Event-Based Knowledge (EBK) masking approach to mask the most significant terms in the event detection task. These significant terms are based on an external knowledge source that is curated for the purpose of event detection for the Arabic language. The proposed approach improves the classification accuracy of all the 9 event types. The experimental results demonstrate the effectiveness of the proposed masking approach and encourage further exploration.

## introduction our lives are a sequence of events. some of them concern the individual, some have their effect extended to a greater population, where others can even have a global effect. as the sources of news about events vary and the speed of the reporting has increased dramatically, event extraction has become an important challenge for governments and different agencies to have appropriate responses to the concerning events. event extraction composes mainly of 2 tasks. the first is event detection, in which the event is detected, usually by a trigger, and then classified. a subsequent task is event argument extraction. it aims to identify different semantic entities related to the detected and classified event. there are several challenges related to event extraction and annotation, such as having multiple event types for the same piece of news, i.e. multi-label problem. also, multiple roles for the same entity, commonly referred to as the role overlap problem. in addition, similar sentences that contain the event trigger and the same entities may be classified as being an event or not based on the tense, whether it is an event that happened or something that is planned for in the future. all these challenges contribute to the complexity of the event extraction problem. as with many downstream tasks, a sophisticated text representation, through contextual representation and attention mechanisms, was able to improve the performance of event detection models as shown in various studies related to events in the english language @xcite @xcite . however, this has not been widely explored yet in event extraction for events reported in the arabic language. event detection studies related to arabic language mainly focus on feature extraction using statistical approaches such as tf-idf @xcite and n-gram along with part-of-speech (pos) and named entity recognition (ner) @xcite or using rule-based approaches as in @xcite . in addition, domain adaptation through continuing to pre-train a contextual model, such as bidirectional encoder representations from transformers (bert) @xcite , on domain-specific corpora such as events @xcite , or modifying the masked language modeling (mlm) learning task to focus on entities have shown significant effect on the performance of the downstream task that they are catered towards (e.g., ner, medical domain nlp tasks, stance detection tasks) @xcite . as imposing an inductive bias to the mlm learning task has yet to be explored for the event extraction task, or when modeling the arabic language, we propose event-based knowledge masked language model (ebk-mlm) for the purpose of better detection and classification of events reported in the arabic language. the contribution of this work is as follows: (1) we collect and annotate an arabic event dataset namely araevent foot_0 which consists of 2 sub-datasets araevent(november) and araevent(july) that is sourced from 4 popular twitter news accounts. (2) we customize the mlm learning task to have an inductive bias towards the most significant terms in events, which achieves an average of 3.67% accuracy improvement in the event detection and classification task when tested on a non-homogeneous dataset, with up to 6.25% improvement in some event types foot_1 . the rest of this paper is structured as follows: in section 2, we present some related work to our study. then in section 3, we propose the pretraining method. in section 4, we present our finetuning experiments, whose results we discuss in section 5. in section 6, we list the limitations of our work. finally, in section 7, we conclude and set some future directions. ## pre-training ebk-bert we propose event knowledge-based bert (ebk-bert), which leverages knowledge extracted from events-related sentences to mask words that are significant to the events detection task (section 3.1). this approach aims to produce a language model that enhances the performance of the down-stream event detection task, which is later trained during the fine-tuning process. the bert-base configuration is adopted which has 12 encoder blocks, 768 hidden dimensions, 12 attention heads, 512 maximum sequence length, and a total of 110m parameters. the details of the implementation is in the following subsections. ## evaluation results and discussion to evaluate the proposed approach, we compare between the classification results of the fine-tuning of both the baseline randmask and our proposed approach ebk-bert. starting with the first and second experiment, as presented in as for the third and fourth experiments, where the test set is from a different time period, the results are presented in ## limitations araevent is drawn from a short period, introducing some bias towards events happening in that period such as ˙⌦m k . @ p and …çªòj ⌦ k. also, errors from the lemmatization tool propagate to the ed task, as shown in the ## conclusion and future work this work aims to propose using the event-based knowledge (ebk) approach for selecting the mask for the mlm training task in order to improve the model's performance for the event detection task. in this (ebk), the most significant words are extracted from an araevent(november) using odds ratio. this dataset is pulled from news channels' twitter accounts and then annotated manually to 9 event types, inspired by ace2005 event extraction dataset, with some modifications. the event classification experiment results show improvement over random masking by 0.56 3.645% across all event types when tested on a homogeneous dataset, an average of 3.67% when tested on a non-homogeneous dataset with limited fine-tuning data. this shows the effectiveness of the proposed masking technique for event detection. the classification results, although higher than random masking, raise several questions on the reasons for the varying performance across the types. running the experiment with different data sizes may asssist to answer the question of whether the data size plays a role in narrowing the effect of the proposed masking, in other words: is (ebk) or similar masking approaches more suitable to perform tasks with small anno-tated datasets? another improvement to be made to the approach is constructing the event datasets gradually over an expanded period of time to mitigate the bias towards the data collection period. also, following the log-odds-ratio with dirichlet prior approach should help us mitigate the bias of the collection period and rare words, in general. we consider this study as preliminary work as a proof of concept that mask approaches catered to a certain downstream task are beneficial to the downstream task for language models built for the arabic language. this is illustrated in this study on a language model built on a considerably limited amount of data. it is interesting to see if this approach can be applied to pre-train large-scale arabic language models for different downstream tasks.