PROTEST - ER : Retraining BERT for Protest Event Extraction (2021)

Abstract: We analyze the effect of further retraining BERT with different domain specific data as an unsupervised domain adaptation strategy for event extraction. Portability of event extraction models is particularly challenging, with large performance drops affecting data on the same text genres (e.g., news). We present PROTEST-ER, a retrained BERT model for protest event extraction. PROTEST-ER outperforms a corresponding generic BERT on out-of-domain data of 8.1 points. Our best performing models reach 51.91-46.39 F1 across both domains.

## introduction and problem statement events, i.e., things that happen in the world or states that hold true, play a central role in human lives. it is not a simplification to claim that our lives are nothing but a constant sequence of events. nevertheless not all events are equally relevant, especially when the focus of attention and analysis moves away from individuals and touches upon societies. in this broader context, socio-political events are of particular interest since they directly impact and affect the lives of multiple individuals at the same time. different actors (e.g., governments, multilateral organizations, ngos, social movements) have various interests in collecting information and conducting analyses on this type of events. this, however, is a challenging task. the increasing availability and amount of data, thanks to the growth of the web, calls for the development of automatic solutions based on natural language processing @xcite . besides the good level of maturity reached by nlp systems in many areas, numerous challenges are still pending. portability of systems, i.e., the reuse of previously trained systems for a specific task on different datasets, is one of them and it is far from being solved @xcite @xcite @xcite @xcite . as such, portability is a domain adaptation problem. following @xcite , we consider a domain to be a variety where each corpus, or dataset, can be described as a multidimensional region including notions such as topics, genres, writing styles, years of publication, socio-demographic aspects, annotation bias, among other unknown factors. every dataset belonging to a different variety poses a domain adaptation challenge. unsupervised domain adaptation has a long tradition in nlp @xcite @xcite @xcite @xcite . the availability of large pre-trained transformer-based language models (tlms), e.g., bert @xcite , has inspired a new trend in domain adaptation, namely domain adaptive retraining (dar) @xcite @xcite . the idea behind dar is as simple as effective: first, additional textual material matching the target domain is selected, then the masked language modeling (mlm) objective is used to further train an existing tlms. the outcome is a new tlm whose representations are shifted to better suit the target domain. fine-tuning domain adapted tlms results in improved performance. this contribution applies this approach to develop a portable system for protest event extraction. our unsupervised domain adaptation setting investigates two related aspects. the first concerns the impact of the data used to adapt a generic tlm to a target domain (i.e., protest events). the second targets the portability in a zero-shot scenario of a domain-adapted tlms across protest event datasets. our experimental results provide additional evidence that further pretraining tlm on domain-related data is a "cheap" and successful method in single-source single-target unsupervised domain adaptation settings. furthermore, we show that fine-tuned retrained tlms results in models with a better portability. ## task and data we focus on the protest event detection task following the 2019 clef protestnews lab @xcite . 1 protest events are identified as politically motivated collective actions which lay outside the official mechanisms of political participation of the country in which the action takes place. the lab is organised around three nonoverlapping subtasks: (a.) document classification; (b.) sentence classification; and (c.) event extraction. tasks (a.) and (b.) are text classification tasks, requiring systems to distinguish whether a document/sentence is referring to a protest event. the event extraction task is a sequence tagging problem requiring systems to identify event triggers and their corresponding arguments, similarly to other event extraction tasks, e.g., ace @xcite . the lab is designed to challenge models' portability in an unsupervised setting: systems receive a training and development data belonging to one variety and are asked to test both against a dataset from the same variety and a different one. we report in we quantified differences and similarities by comparing the training data (india train ) against the two test ones (india test and china test ) using the jensen-shannon (j-s) divergence and the out-of-vocabulary rate (oov) that previous work has shown to be particularly useful for this purpose @xcite . the figures in a further challenge is posed by the limited amount of training material. a comparison against the training portion of ace shows that protest-news has 5 times less triggers and 4 times less arguments. 2 unlike ace, event triggers are not further classified into subtypes. however, seven argument types are annotated, namely participant, organiser, target, etime (event time), place, fname (facility name), and loc (location). the role set is inspired by ace attack and demonstrate event types but they are more fine-grained. the markables are encoded in a bio scheme (beginning, inside, outside), resulting in different alphabets for triggers (e.g. b-trigger, i-trigger and o) and each of the arguments (e.g. o, b-organiser, i-organiser, b-etime, i-etime, etc.). ## continue pre-training to adapt we applied dar to english bert base-uncased to fill a gap in language variety between bert, trained on the bookscorpus and wikipedia, and the protestnews's data. we collected two sets of domain related data from the trec washington post corpus version we apply each data collection separately bert base-uncased by further training for 100 epochs using the mlm objective. the outcomes are two pre-trained language models: news-bert and protest-er. the differences between the models are assumed to be minimal but yet relevant to assess the impact of the data used for dar. to further support this claim we report in ## experiments and results event extraction is framed as a token-level classification task. we adopt a joint strategy where triggers' and arguments' extent and labels are predicted at once @xcite . we used india test to identify the best model (news-bert vs. protest-er) and system's input granularity. with respect to this latter point, we investigate whether processing data at document or sentence level could benefit the tlms as a strategy to deal with limited training materials. we compare each configuration against a generic bert counterpart. we fine-tune each model by training all the parameters simultaneously. all models are evaluated using the official script from the protestnews lab. triggers and arguments are correctly identified only if both the extent and the label are correct. we apply to china only the best model and input format. india data results for india are illustrated in the language variety of the data distributions used for dar has a big impact on the performance of fine-tuned systems, with news-bert being the worst model. the extra training should have made this model more suited for working with news articles than the corresponding generic bert. this indicates that selection of suitable data is an essential step for successfully applying dar. globally, the results show that dar has a positive effect on precision, especially when sentences are used as input for fine tuning the models. positive effects on recall can only be observed for protest-er. with the exclusion of news-bert, the systems achieve satisfying results for the trigger component. argument detection, as expected, is more challenging, with no model reaching an f1-score above 50%. protest-er always performs better, especially when processing the data at sentence level. in numerical terms, protest-er provides an average gain of 11.74 points. 4 we observe a relationship between argument type frequency in the training data and models's performance where the most frequent arguments, i.e., participant (26.43%), organizer (18.31%), and place (14.45%), obtain the best results. however, protest-er improves performances also on the least frequent argument types, i.e., loc (6.49%) and fname (5.85) of, respectively, 12.00 and 5.38 points on average, when compared to bert. ## discussion and conclusions our results indicate that dar is an effective strategy for unsupervised domain adaptation. however, we show that not every data distribution matching a potential target domain has the same impact. in our case, we measure improvements only when using data that more directly target the content of the task, i.e., protest events, possibly supplementing limitations in training materials. we have gathered interesting cues that processing data at document level can actually be an effective strategy also for a sequence labeling task with small training data. we think that this approach allows the tlms to gain from processing longer sequences and acquire better knowledge. however, more experiments on different tasks (e.g., ner) and with different training sizes are needed to test this hypothesis. a further positive aspect of dar is that it requires less training material to boost system's performance, pointing to new directions for few-shot learning. we projected the learning curves of bert and protest-er using increasing steps of the training data. protest-er achieves an overall f1-score ∼30% with only 10% of the training data, while bert needs minimally 30% to achieve comparable performances (see @xcite . disappointingly, protest-er falls way back the best model that participated in protest-news. @xcite propose a bi-lstm-crf architecture using flair contextualized word embeddings @xcite . they also adopt a joint strategy for trigger and argument prediction. protest-er obtains a better precision only on china for the overall evaluation and for trigger. quite surprisingly, on india it is bert that achieves better results on trigger, although the model appears to be quite unstable, as shown by the standard deviation. at this stage, it is still unclear whether these disappointing performances are due to the retraining (i.e., need to extend the number of documents used) or the small training corpus. future work will focus on two aspects. first, we will further investigate the impact of the size of the training data when using tlms. this will require to experiment with different datasets and tasks. secondly, we will explore solutions for multilingual extensions of protest-er. we used the original train, validation, and test splits of the event extraction task of the 2019 clef protestnews lab. we conducted all the experiments using the google colaboratory platform. the time required to run all the experiments on the free plan of colaboratory is approximately 20 hours. figure 1 graphically illustrates the base architecture.