A Hybrid Knowledge and Transformer-Based Model for Event Detection with Automatic Self-Attention Threshold, Layer and Head Selection (2022)

Abstract: Event and argument role detection are frequently conceived as separate tasks. In this work we conceive both processes as one taskin a hybrid event detection approach. Its main component is based on automatic keyword extraction (AKE) using the self-attention mechanism of a BERT transformer model. As a bottleneck for AKE is defining the threshold of the attention values, we propose a novel method for automatic self-attention thresholdselection. It is fueled by core event information, or simply the verb and its arguments as the backbone of an event. These are outputted by a knowledge-based syntactic parser. In a secondstep the event core is enriched with other semantically salient words provided by the transformer model. Furthermore, we propose an automatic self-attention layer and head selectionmechanism, by analyzing which self-attention cells in the BERT transformer contribute most to the hybrid event detection and which linguistic tasks they represent. This approach was integrated in a pipeline event extraction approachand outperforms three state of the art multi-task event extraction methods.

## introduction event extraction, argument and semantic role detection are frequently conceived as separate tasks @xcite @xcite @xcite where a multi-word event is first split into a verb as single-word event to process, after which its argument roles (subject, direct and indirect object(s)) and semantic roles (such as time and location) are extracted. these are typically trained in a multi-task setup for event extraction, which combines event span detection and classification. in this work, we tackle multi-word event extraction and conceive event span detection and argument extraction as one task in a hybrid knowledge and transformer-based event detection method. the verb, subject and object(s) (svo) are first outputted by a knowledge-based syntactic parser and combined with automatic keyword extraction (ake). in this latter step, the most relevant keywords in a sentence or most salient semantic information is selected, exploiting the attention mechanism of a transformer, i.e., bert (bidirectional encoder representations from transformers) @xcite . a bottleneck for ake is defining the threshold of the attention values to take into account @xcite . hence, we propose and outline a method for automatic attention threshold selection by exploiting the interaction between self-attention based ake and rulebased event detection. as the main function of the rule-based component is to provide the necessary information for the automatic attention threshold mechanism, it targets only minimal event information, i.e., the core or backbone of the event or the verb and its svo arguments. this allows the transformer's main component to complement it with other semantic roles and semantically salient information. however, the latter type of information is often essential to constitute the core meaning of the event. for example, omitting the adverb "conditionally" in the event "he was conditionally released from detention." changes its semantics and causes a misunderstanding of it. this kind of semantically salient information can only be provided by the transformer model, and not by the knowledge-based component in our hybrid model. our hybrid event detection mechanism is embedded in a pipeline event extraction approach that goes beyond short event spans: in a first step, event classification is applied to raw input sentences, whereas in a second step, the event span is detected. for a fair evaluation, we compare this approach with three event detection approaches as part of a multi-task event extraction method that jointly predicts event spans and classes. the main contributions of this paper are the following: our approach is positioned with respect to the state of the art in section 2 and is presented in sections 3 and 4. an overview of the data is given in section 5. section 6 presents the results of experiments, followed by a thorough analysis and discussion. the paper is concluded in section 8. ## related work knowledge-based event detection methods were initially based on ontologies @xcite @xcite or rule-sets @xcite which represent expert knowledge. these also include extracting candidate event words with part-of-speech tags @xcite , which can also satisfy predefined syntactic patterns @xcite . statistical methods spot event spans using n-grams @xcite , term frequency inverse document frequency (tf-idf), word frequency and word co-occurrence @xcite . early supervised machine learning approaches recast event detection as a binary classification problem @xcite to decide whether an input word is part of an event or not. to that end, maximum entropy @xcite , support vector machines (svm) @xcite and conditional random fields (crf) @xcite were applied. as the event detection field initially concentrated on fixed event types using single-word or event spans with a short length @xcite , these supervised machine learning approaches have successfully used the ace 2005 corpus @xcite comprising single-word event span length annotations. with feature engineering approaches emerging, the scope became larger than a one-word event span @xcite . in lefever and hoste (2016) multiword events in dutch news text are detected using an svm binary classifier combining lexical, syntactic and semantic features. these feature-based machine learning techniques, however, have been superseded by deep learning techniques which are able to learn hidden feature representations automatically from data. in @xcite , a multiword event detection approach using convolutional neural networks (cnn) outperforms an svm approach. spearheaded by their success in dealing with long-term dependencies in longer sequences, the lstm (long short-term memory) and attention mechanism allow the decoder to learn which parts of the sequence should be attended to in an encoderdecoder architecture @xcite , hence taking more context information into account. @xcite presents a supervised attention-based rnn event detection approach that outperforms an rnn and cnn, both without attention mechanism. deep learning approaches that were in recent years combined with word2vec @xcite , glove @xcite and fast-text @xcite word embeddings have led to the rise of the transformer architecture @xcite . its contextual language models have been successfully integrated in a range of nlp tasks using pre-trained contextual bert (bidirectional encoder representations from transformers) word embeddings @xcite . on top of that, the bert model fully exploits the attention mechanism for multi-word event detection, which is illustrated in @xcite , where a multi-attention event detection tool, using bert, fine-tuned on the civil unrest gold standard report data @xcite , outperforms a cnn. the hybrid target event detection method that is proposed here also fully benefits from the bert multi-head self-attention, but is combined with subject, verb and object (svo) information, as outputted by a knowledge-based syntactic parser. ## pipeline event extraction approach an event can be defined as the smallest extent of text that expresses its occurrence @xcite and is identified by a word or phrase called event trigger, nugget, event span or mention. event mentions can be single-word event triggers that are usually (main) verbs, nouns, adjectives and adverbs. multi-word event triggers can be consecutive tokens, complete sentences, or discontinuous when on top of the verb, its participants, or argument roles are also involved @xcite . our hybrid event detection approach targets multiword continuous event spans. it goes beyond the scope of approaches tackling single-word events that are frequently using the ace 2005 corpus (section 2). hence our models are trained on the (dutch) eventdna corpus, annotated with multiword event spans and class labels (section 5). ## baseline multi-task event extraction approaches we compare the target pipeline event extraction model (section 3) with three baseline multi-task event extraction models. to the best of our knowledge, we are not aware of other baseline approaches applied to languages with a similar event prominence annotation scheme (section 3.1). in the multi-task approach, event detection and classification tasks are performed simultaneously to benefit from their interplay @xcite . the first model is an attention-based rnn model with lstm from @xcite , with an encoder-decoder architecture. its atten we finally added a crf on top of the multi-task bertje-based approach, resulting in our third baseline model where the joint bert+crf replaces the softmax classifier with crf @xcite . the target event sequence is labeled in iob format. tokens at the begin of an event mention are labelled as b-ev, tokens inside the mention as i-ev, and tokens outside the mention as o. the eventdna data set comprises raw sentences with more than one event span. as a first step, only unique sentences with one event span were kept for our experiments. ## results analysis and discussion in spite of the interaction between event classification and event detection, the multi-task baseline models could not outperform the classifier of the target pipeline model. on top of that, the pretrained bertje models of the bert-based multitask baseline models outperform the attentionbased rnn multi-task model without bert. this shows that a pre-trained bert transformer model improves performances, when fine-tuning on a small data set. for event detection with automatic self-attention threshold selection, the target pipeline event extraction model did not outperform the bert-based baseline models. however, combined with automatic self-attention layer and head selection, layer 7 and head 1 show the best event detection performances. hence, we analysed the latter result by correlating the order of transformer block layers and heads with event detection f1 scores. in a next step, attention attributions of the transformer model are visualised. finally we check the impact on attention attribution stability by changing the word order of the input sentences. ## conclusion and future work this study outlines a pipeline hybrid knowledgeand transformer self-attention based event detection approach. it outperforms three state of the art multi-task baseline event extraction models. for keyword-based event detection, we solved the bottleneck of defining the threshold of the attention values to take into account. automatic self-attention threshold, layer and head selection was applied, exploiting the interaction between a rule-based svo (subject-verb-object) extraction and self-attention based automatic keyword extraction (ake). analysis of the bertje transformer model shows that syntactic dependencies are most active in the middle layers and contribute most to event detection. we also found evidence for consistency of attention attributions of the transformer model. as a next step, the behaviour and stability of the surrounding layers, should be further investigated. other data sets in dutch or other languages can be used, comprising more than one event span per sentence.