Event Prominence Extraction Combining a Knowledge-Based Syntactic Parser and a BERT Classifier for D utch (2021)

Abstract: A core task in information extraction is event detection that identifies event triggers in sentences that are typically classified into event types. In this study an event is considered as the unit to measure diversity and similarity in news articles in the framework of a news recommendation system. Current typology-based event detection approaches fail to handle the variety of events expressed in real-world situations. To overcome this, we aim to perform event salience classification and explore whether a transformer model is capable of classifying new information into less and more general prominence classes. After comparing a Support Vector Machine (SVM) baseline and our transformer-based classifier performances on several event span formats, we conceived multi-word event spans as syntactic clauses. Those are fed into our prominence classifier which is fine-tuned on pre-trained Dutch BERT word embeddings. On top of that we outperform a pipeline of a Conditional Random Field (CRF) approach to event-trigger word detection and the BERT-based classifier. To the best of our knowledge we present the first event extraction approach that combines an expert-based syntactic parser with a transformer-based classifier for Dutch.

## introduction recently, news publishers have shifted from newspapers to digital means which provide news readers easy access to a wide range of news information. however, the challenge is to find the right content that also corresponds to the user's personal interests. therefore, many of today's major media and news websites offer automated news recommendation and personalization @xcite @xcite . news personalization paradigms define good news recommendations in terms of similarity to the user's previous reading behaviour. hence, news articles are recommended based on proximity to other articles the user has read @xcite . however, this contrasts with the normative concept of journalism that stimulates diversity of topics and events in unfiltered news streams @xcite . in this study we consider the news event as a means to model both diversity and similarity in news articles in the context of a news recommendation system. we present an event extraction approach that will be integrated in a news recommender for dutch foot_0 . as current typology-based event detection fails to handle the variety of events in real-world situations we applied event prominence classification. this allows us to detect unrestricted news events and to overcome the sparsity of a small training data set. our event extraction approach combines an expertbased syntactic parser with a transformer-based classifier: we also motivate the use of syntactic clauses as event spans, by comparing baseline and target classifier performances on other multi-word event span formats. on top of that we outperform a pipeline of a crf event-trigger word detection approach and our bert-based classifier. furthermore, our approach is positioned with respect to the state of the art in section 2 and is outlined in section 3. an overview of the data set is given in section 4. section 5 presents the results of experiments on the held-out test set followed by a results analysis and discussion, conclusion and outlook on future work. ## related work knowledge-based approaches are still frequently used for event extraction. such methods are based on ontologies @xcite @xcite or rule-sets @xcite which represent expert knowledge. information is mined from corpora based on lexical, syntactic @xcite and semantic patterns or frames @xcite @xcite . as the manual creation of rule-sets and ontologies is difficult and time-consuming, data-driven event extraction approaches made their entrance. the ace (automatic context extraction) annotation standards foot_1 , ere (entities, relations, events) annotation standards @xcite and tac-kbp (text analysis conference knowledge base population) foot_2 workshops and competitions stimulated the creation of data sets labeled with entities and events, e.g. the ace 2005 corpus @xcite . as a consequence, supervised methods became predominant but initially concentrated on fixed event types using single-word event spans @xcite . as compensation for small event spans, sentence or cross-sentential context information was used. in @xcite and @xcite events were extracted through cross-document and cross-sentence inference, respectively. @xcite improved event extraction performances adding topic classification information. as feature engineering approaches emerged, a larger scope than one-word event spans was targeted. hand-designed sets of lexical, semantic or syntactic features were extracted and fed into classifiers, allowing the model to take more context into account @xcite . event extraction tasks are typically applied in a pipeline architecture where event trigger word identification, argument and event classification are conceived as separate tasks @xcite . other than a pipeline architecture, multi-task architectures perform several subtasks simultaneously to benefit from their interdependencies. in @xcite events were extracted incorporating features that capture dependencies of multiple triggers and arguments. @xcite and @xcite extracted events combined with named entity and argument role prediction. however, the choice of features is a manual and elaborate process that requires extensive linguistic domain expertise. more recently deep neural networks superseded methods that show a strong dependency on feature resources, although the latter ones are still not definitely outperformed. @xcite and @xcite used lexical, syntactic features, word2vec @xcite , glove @xcite and fasttext @xcite word embeddings. better performances were reported for an svm classifier compared to a recurrent neural network (rnn). in contrast, nguyen and grishman (2015) demonstrated that convolutional neural networks (cnn) significantly outperformed feature-based methods on the ace 2005 task. meanwhile, contextual language models have proven successful in a transformer architecture @xcite that fully benefits from the attention mechanism. it has been integrated in a range of nlp tasks using pre-trained contextual bert (bidirectional encoder representations from transformers) word embeddings @xcite , predominantly for english. @xcite report encouraging results for an event factuality classifier using bert. @xcite report svm event classifications with term frequency-inverse document frequency (tf-idf) that are outperformed by a fine-tuned bert event classifier. the results of these studies inspired us to combine an expert-based syntactic parser with a bert-based language model classifier for dutch in order to extract multi-word events. ## data our baseline and target event prominence classification models were trained on the eventdna corpus. it comprises 1,771 dutch news articles ( for our experiments, both data sets with annotated events and 5 token windows with verbal head, extracted from the corpus, were randomized and split into 80% train, 10% development (dev) and 10% held-out test data as shown in in order to verify the feasibility of our approach to classify events based on the test sentences, split into syntactic clauses, with the alpino syntactic parser (section 5.2), we counted the syntactic constituents in the training data annotated with events. as the test sentences were split into syntactic clauses, the number of test instances (syntactic clauses) in data instances instances set training set test set annotated events 7362 934 annotated events2 7362 780 5 token windows 6248 780 raw sentences -904 syntactic clauses -1030 syntactic clauses2 -904 ## experiments and results we trained and tested our svm baseline event classifier and target bert event classifier on 5 token windows and annotated events (section 4). then we fed the syntactic clauses from the syntactic parser into the baseline svm and target bert classifiers. finally, we positioned our approach w.r.t. a pipeline of a crf approach to event-trigger word detection and target prominence classifier (section 5.2). ## results analysis and discussion analysis of bertje attention heatmaps indicated the feasibility of our event extraction approach combining a syntactic parser and a bert classifier. the sentence "then an adviser to the president was convicted because he lied" foot_6 (figure 3 -left) consists of a main clause "then an adviser to the president was convicted" foot_7 (middle) with a main event, and a subclause "because he lied" foot_8 , the background event (right). figure 3 (left) shows that most attention in the raw sentence is erroneously attributed to the past participle in the subclause, "gelogen" (lied). after splitting the sentence in its main and subclause, most attention is now correctly attributed to the verbs in the main (middle) and background (right) event. although the bertje classifier performances on the syntactic clauses are better, compared to the crf detected events ( in spite of the advantages of using the trans-former model, minimal loss was already obtained after only 3 epochs of training for bertje (section 5.1). the bertje model pre-trained on a large corpus, allows a small dataset being used for finetuning on the event prominence classification task. however, applying data augmentation on the small newsdna dataset might increase training time during fine-tuning. although the pre-trained bertje model is large (2.4 billion tokens), it contains other data than news corpora, whereas our training set consists entirely of news. this raises the question whether it is not better to use a domain-specific pretrained model consisting entirely of news corpora. a bottleneck of classifying prominence labels only based on the sentence level, is the lack of context information. this has an impact mainly on the background prominence class ( ## conclusion and future work this study shows that an event extraction approach of an expert-based syntactic parser in combination with a transformer-based classifier (bertje) is feasible. the resulting model outperforms (62.95% f-score) a pipeline of a crf approach to eventtrigger word detection and a bert-based event classifier. we also demonstrated that a syntactic clause can be used as event span. prominence classification is our answer to take into account a realworld situation where event types in held-out test data are frequently not covered because of training data scarcity. the bertje model benefits from self-attention heads with linguistic notions such as syntax and coreference and outperformed (70.75% f-score) an svm baseline classification model. a bottleneck of classifying prominence labels only based on the sentence level, is the lack of context. this has an impact mainly on the background prominence class. therefore further work includes exploring ways to provide more context information in the transformer model. it can be fine-tuned on training data where previous and following annotated events to the current single event instances are inserted. as a next step the bertje self-attention mechanism will be leveraged to select the tokens in the syntactic clause with the highest attention values. this will allow e.g. the generation of nominal constituents on top of the clauses generated by the syntactic parser. although the transformer model exhibits promising performances fine-tuned on a small dataset, data augmentation of the training set might optimize the fine-tuning and boost performances. finally the classifier output will be fed into a news recommender system.