New Benchmark Corpus and Models for Fine-grained Event Classification: To BERT or not to BERT ? (2020)

Abstract: We introduce a new set of benchmark datasets derived from ACLED data for fine-grained event classification and compare the performance of various state-of-the-art models on these datasets, including SVM based on TF-IDF character n-grams and neural context-free embeddings (GLOVE and FASTTEXT) as well as deep learning-based BERT with its contextual embeddings. The best results in terms of micro (94.3-94.9%) and macro F1 (86.0-88.9%) were obtained using BERT transformer, with simpler TF-IDF character n-gram based SVM being an interesting alternative. Further, we discuss the pros and cons of the considered benchmark models in terms of their robustness and the dependence of the classification performance on the size of training data.

## introduction since an ever-growing amount of information on events of any type is transmitted via web in the form of free texts (e.g. online news) one has witnessed in the last decades an emergence of research on development of methods and tools for automated detection and extraction of structured information on events from textual sources @xcite @xcite @xcite @xcite . one particular step in the event extraction process is event classification, i.e., assigning to a text snippet including event trigger an event type using a domain specific taxonomy, which is the main focus of this paper. while vast amount of tasks and challenges on automated event extraction, including event classification, has been organised over the years, relatively little work has been reported on approaches for fine-grained event classification. furthermore, the existing freely available datasets used for training and evaluation purposes are rather of tiny size, ranging usually up to 5-10k events. due to the emergence of deep learning-based approaches for the entire range of nlp tasks, there is a particular need to have larger event classification corpora in order to gain better insights into the performance of such methods and their comparison with shallow learning approaches, e.g., in terms of training data sizes required to obtain 'acceptable' performance, types of embeddings and model robustness vis-a-vis different data characteristics. the rise of deep learning-based approaches allowing for model pre-training in an unsupervised manner using only plain text and then utilizing transfer learning (via re-using the pre-trained model and only fine-tuning it in a supervised manner), poses additional questions with regard to the required data sizes and the robustness of transfer learning against various data characteristics. one of the most groundbreaking moments in the field of nlp was the release of bert (bidirectional encoder representations from transformers) model @xcite in october 2018, as the first deeply bidirectional, unsupervised language representation model, leading to a significant uplift in multiple performance benchmarks with limited task-specific fine-tuning. hence, among the benchmark machine learning models included in the work reported in this article we study in particular the performance of bert on the fine-grained event classification tasks derived from acled. the main contributions of the work reported in this paper can be summarized as follows: the exploitation of the acled data for evaluation of fine-grained event classification models has, in particular, the following two major advantages: (a) the acled event descriptions resemble very much texts that can be found in online news reporting on events, and (b) acled data is to a certain extent noisy in terms of grammatical correctness, which provides an excellent material to test models robustness visa-vis lower quality data. to our best knowledge, no similar corpora in terms of size for the task at hand exist, and given the specific nature of the dataset (i.e. text snippets resembling news reporting), we believe that measuring the event classification performance of a given method on these dataset might constitute a good approximation of the to-be-expected performance when applying the same method on real news articles. the rest of the paper is structured as follows. first, an overview of related work is provided in section 2. subsequently, section 3 describes the corpora derived from acled data. next, section 4 introduces the benchmark models for the event classification task, whereas section 5 presents the results of the performance of these models on the acled corpora and basic error analysis. the main findings and practical implications thereof are summarized in section 6. finally, section 7 gives conclusions and an outlook on future work. ## related work the early research on event detection and classification in textual documents was driven by the message understanding contests @xcite and the automatic content extraction (ace) challenges @xcite . many approaches to event detection and classification have been reported and evaluated on the event corpora (ca. 6000 event mentions in ca. 500 documents) developed in the context of the aforementioned ace challenges, which range from shallow @xcite to deep machine learning approaches @xcite . the more recently introduced multi-lingual event detection and co-reference challenge in the context of the text analysis conference (tac) in 2016 foot_1 and 2017 foot_2 , included an event nugget detection subtask, which focused on detection and fine-grained classification of intra-document event mentions (9 types and 38 subtypes), covering events from various domains (e.g., finances and jurisdiction). the evaluation datasets used in the context of tac are rather tiny though (<10k events). in the last decade other efforts on more fine-grained event classification that cover various domains were reported too. for instance, @xcite compared svm-based models against wordvector-based lstms for classification of 10 types of company-specific economic events from online news, whereas @xcite studied the performance of various models, including ones that exploit word embeddings as features, for detection and classification of natural disaster and crisis events (7 types) in news articles. while most of the work in this area focused on english language and processing news texts in particular, some efforts on event classification for non-english language and other domains were reported too. a benchmark corpus for fine-grained classification of man-made and natural disasters (28 types) for hindi, accompanied with evaluation of deep learning baseline models for this task, has been presented in @xcite ). furthermore, an example of fine-grained classification of cyberbullying events (7 classes) in social media posts was reported in @xcite . this paper reports on the creation of benchmark corpora for fine-grained event classification of political violence, conflict situation and protest events from short text snippets, where the main difference vis-a-vis the benchmark corpora reported elsewhere is the size of the corpora, significantly bigger (ca.. 600k events) versus other known event classification corpora (usually of the size in the range of 2-10k). the work reported in this paper builds on the preliminary study of the acled data for event classification presented in @xcite and extends it in various dimensions. ## event classification corpora derived from acled data the event classification corpus was derived from event data collected in the context of the armed conflict location & event data project (acled) foot_3 . acled @xcite gathers human-moderated records on most important facts about political violence and protest events across various continents with a specific focus on africa, asia, the middle east, and southeastern and eastern europe. the collected event records contain information on the date of the event, location, the key actors involved, type of violence and number and description of fatalities. for the sake of creating corpora for the event classification task we have extracted from circa 615k manually-curated event records available on the acled web page foot_4 three elements, including: (a) event snippets, being free-text descriptions of the events, which mention basic information on all key information on the event, (b) event type, and (c) and event subtype. the acled event type ontology has 6 main even types (battles, explosion and remote violence, violence against civilians, protests, riots, strategic developments), which are further subdivided into 25 fine-grained subtypes. the detailed definitions of the various event types and subtypes in acled are reported in the so called acled codebook @xcite . some examples of event descriptions for violent demonstration, peaceful protest, and armed clash events resp. are given below. while the texts in the first two examples resemble texts that could as well appear in news articles, the third example contains some comments in brackets provided by the human experts. from the raw data extracted from acled event records three event corpora were created, each being result of cleaning and normalising the original free-text event descriptions. the main drive behind creating three corpora was to move from textual data that contain some "noise" and some not fully grammatically correct constructions to a corpus containing grammatically correct sentences and constructions which are very close to texts appearing in the news reporting on events. 6666 acled-i was created through carrying out most basic cleaning of the texts, including: (a) removing from the event descriptions quotation and similar non-content relevant characters, (b) removing too obvious markers that would artificially hint a classifier to guess the correct event (sub)type, e.g., initial phrases like "attack:" corresponding directly to the definition of the event type, and (c) filtering out event records, whose event descriptions consist of less than 20 characters, which were deemed as non informative. acled-ii was created via applying the following treatments on acled-i: (a) removing outliers, i.e., events, whose description is longer than 650 characters, (b) removing from event descriptions circa 100 unique phrases (provided that they appear at the beginning of the event description and are followed by a colon) that might indicate the event type foot_5 that are, however, not identical with the event type definitions (e.g. detonation:), (c) removing references to urls, (d) removing comments in brackets introduced by human experts in texts, e.g., acled-iii resulted from further cleaning and normalisation of acled-ii, which included: (a) removing events whose description is shorter than 60 characters, (b) removing additional non-sentencelike structures corresponding to comments introduced by human experts encoding the events (i.e, constructions in brackets like in acled-ii d) above, but significantly longer), (c) normalisation of various non-alphanumeric symbols, (d) removing numeric encoding of locations (coordinates), and (f) removing all event descriptions that contain at least one sentence, which could not be parsed by stanford pcfg parser @xcite and resulting in a tree with a root labelled as "s", "frag" or "np". in this context we made an assumption that parse trees with roots labelled with tags other than the ones mentioned before constitute potential indicators of non grammatically correct sentences/utterances/nominal phrases. for instance, the parse tree for the event description 'reports that the csnpd attacked a truck near gore killing two people and wounding six' was labelled with x, and thus eliminated (subject missing). for testing robustness of the benchmark models in the event classification task, an additional version of acled-iii was generated, in which two main type of modifications were carried out on the acled-iii corpus: (a) all day and month names were replaced with randomly selected days and months, and (b) each occurrence of a toponym referring to a populated place was replaced with randomly chosen toponym selected from a geonames gazetteer foot_6 of circa 200k populated cities, whose population is at least 500. the main drive behind these modifications was to simulate data drift that can be expected in the domain of the data at hand. this alternate version of acled-iii will be referred to with acled-iii-∆. the basic statistics for all three acled datasets are provided in the dimensionality of the tf-idf vectors varies depending on the training set size, and each event description is represented by a large sparse vector instead of the short full vector used in the word embedding representation. for the acled-iii corpus, the tf-idf vectors vary from 41 054, when using 1% of the training set, to 364 023 when using the full training set. regarding non-linear kernels, due to the fact that with the given size of the data the standard nonlinear svms do not scale well, we have run some initial experiments following the common approach of using kernel map approximations and applying linear svm on the top of it. using nystroem method @xcite for approximating rbf kernel as well as using monte-carlo sampling from the fourier transformation of the rbf @xcite and chi-squared kernels (vedaldi and zisserman, 2012) resulted in either worse or similar performance than plain linear svm. although only several alternatives for non-linear kernels have been examined and all are subject to sampling errors inherent in the applied approximations, we hypothesise that these results are an indication that with the underlying bow feature space the problem is either linearly separable or close to linearly separable. ## experiments we have evaluated five svm-based classifiers (two of which are feeded with bert embeddings) and one end-to-end deep nn-based bert classifier. more specifically, the following models are included in our experimental setup: (a) svm with tf-idf char n-grams (svm-char), (b) svm with fasttext embeddings (svm-fast), (c) svm with glove embeddings (svm-glove), (d) svm with non finetuned bert embeddings (svm-bert), (e) svm with fine-tuned bert embeddings (svm-f-bert), and finally (f) the deep bidirectional transformer encoder bert (bert). all models were used for running experiments on all four acled datasets. ## discussion based on the results presented in section 5.3 we can draw some general conclusions regarding real-world usability of the benchmark models evaluated. all svm-based and fine-tuned bert-based transformer model happened to be resistant to somewhat more noisy data, where in the case of the latter model we were expecting to see some more visible performance loss given that the acled event descriptions appeared to be different from the corpora on which the pre-trained bert embeddings were learned. as regards the classification performance bert appears to be the winner among the compared models and evaluation metrics, however, when time complexity is a concern (bert being known to be of magnitudes slower), the runner-up, namely tf-idf character n-gram based svm performed surprisingly well, and is not lagging far behind, would be definitely the better choice. we have also observed that dropping all n-grams with tf-idf below 0.001 reduces the dimensionality of the feature space by the factor of at least x10 with only marginal impact on classification performance, which makes svm-char even more attractive. in addition, in the case of selecting a solution for obtaining the best results in terms of macro f 1 and having available only a tiny fraction of the training data (i.e. less than ca. 20k events) with some very sparsely populated event classes like in acled, using bert would not be convenient vis-a-vis other models that managed better to tackle the data sparseness problem. furthermore, all models did not turn to dramatically suffer from significant data drift based on some rudimentary robustness tests carried out. ## conclusions in this paper we presented large datasets for evaluation of fine-grained event classification (25 classes), which were derived from acled data, a human-created event repository. we compared the performance of 6 state-of-the-art benchmark models, spanning svm and nn-based classifiers that exploit tf-idf character n-grams and off-the-shelf pre-trained non-contextual and contextual word embeddings as features. the best results in terms of micro (94.3-94.9%) and macro f 1 (86.0-88.9%) were obtained using the popular bert transformer, however the significantly simpler tf-idf character n-gram based svm constitutes an interesting alternative. there are various avenues to explore in the future, including, i.a., (a) evaluation of models that exploit the hierarchy of the event types, (b) carrying out more advanced robustness tests @xcite , (c) exploration of other transformer-based approaches @xcite , (d) more in-depth cross-model error analysis, and (e) alleviating the problem with uneven event type distribution, and creating somewhat more "balanced" version of the datasets. all raw acledderived corpora can be downloaded at: http://piskorski.waw.pl/resources/acled/ acled-datasets.zip , whereas the corresponding versions with partitions into training and test data are accessible at: http://cidportal.jrc.ec.europa.eu/ftp/jrc-opendata/ language-technology/2020_annotated_event_dataset/folds/ . please note that for the purpose of carrying out the evaluations reported in this paper the test data from fold 1 was used.