How Does Context Matter? On the Robustness of Event Detection with Context-Selective Mask Generalization (2020)

Abstract: Event detection (ED) aims to identify and classify event triggers in texts, which is a crucial subtask of event extraction (EE). Despite many advances in ED, the existing studies are typically centered on improving the overall performance of an ED model, which rarely consider the robustness of an ED model. This paper aims to fill this research gap by stressing the importance of robustness modeling in ED models. We first pinpoint three stark cases demonstrating the brittleness of the existing ED models. After analyzing the underlying reason, we propose a new training mechanism, called context-selective mask generalization for ED, which can effectively mine context-specific patterns for learning and robustify an ED model. The experimental results have confirmed the effectiveness of our model regarding defending against adversarial attacks, exploring unseen predicates, and tackling ambiguity cases. Moreover, a deeper analysis suggests that our approach can learn a complementary predictive bias with most ED models that use full context for feature learning.

## introduction event detection (ed), a crucial subtask of event extraction (ee), aims to identify and categorize event triggers in texts. for example, in a sentence s1: "during a war, invaders destroyed the whole town", ed requires a system to detect an event trigger destroyed, along with its event type attack 1 . building a robust ed system is shown to benefit a wide range of applications including document summarization @xcite , knowledge base population @xcite , question answering @xcite , and others. in recent years, great advances have been made in ed (ji and grishman, 2008; li et al., 2013; chen 1 according to ace event ontology. s1: during a war, invaders destroyed the whole town. s2: during a war, invaders annihilated the whole town. et @xcite @xcite @xcite . however, the vast majority of existing studies focus on improving the overall performance of an ed model (usually on a fixed test set), which rarely consider the robustness (and generalization capability) of an ed model. for example, most of existing methods do not answer questions such as when/why an ed system would fail, how to handle new, previously unseen data, despite these considerations are especially crucial for designing real-world ed systems. this paper focuses on the robustness aspect of ed models. we first emphasize the necessity of this research by pinpointing three stark cases demonstrating the vulnerability of existing ed models. these cases are: 1) adversarial attack, which refers to adding small perturbations in the original sentences @xcite . as shown in figure 1 , a well-trained event detector can correctly recognize the event trigger destroyed at first. but when we replace destroyed with a rare trigger annihilated, despite the meaning of the sentence does not change, we note the same event detector fails to identify the trigger. a quantitative evaluation suggests that the performance of a stateof-the-art (sota) ed model @xcite drops significantly from 69.1% to 19.2% facing adversarial attack. 2) unseen predicates, which measures whether an ed model can tackle new, previously unseen data. we note the existing ed models demonstrate a rather poor generalization capability: they achieve only 14.2% in f1 for the previously unseen triggers, despite 74.9% in f1 for the already seen triggers. 3) event type disambiguation, which refers to assign a correct event type to ambiguous triggers, considering that over 70% of triggers can express different types of events @xcite . while, our pilot experiments suggest that a sota ed model obtains only 50.4% in f1 in tackling the high-ambiguity cases, comparing to 70.6% in f1 in tackling the low-ambiguity cases. the above phenomena reflect the fact that current ed models have a poor ability in modeling contexts, where the underlying reason may highly relate to reasoning shortcuts @xcite -owing to the limited (and biased) training data, an ed model may have only learned lexical pattern, i.e., word-to-trigger mapping (such as @xmath0, owing to its prevalence in data. by adopting such reasoning shortcuts, an ed model may explain the training data well, but fail in the more context-dependent scenarios noted above, as they never capture the underlying regularities about how event triggers appear in texts. in light of the above analysis, we propose a new training paradigm, termed as context-selective mask generalization, aiming to prevent reasoning shortcuts and robustify an ed model. our method is intuitive and straightforward: to prevent lexical bias, we explicitly delexicalize triggers for training/testing, by replacing them with placeholders. this forces our model to make predictions using contexts solely. for instance, a training example s1 is transferred as: "during a war, invaders [mask] the whole town", and our model is forced to predict the event label of the masked word. as the lexical information of the trigger is completely masked, our model has to mine the more essential contextual clues for reasoning. this prevents our model simply remembering word-to-trigger shortcuts, but to learning the underlying regularities regarding how events are described in texts. the proposed learning paradigm consists of two complementary training objectives: contextselective discriminative learning and contextualized similarity learning. the former is an intrasentence objective, considering that contextual words are usually of different importance, for example, in s1, "wars" and "invaders" may be more important than "town" for predicting the attack event. we devise a method combing selective attention @xcite with model uncertainty @xcite to weigh contexts and select the salient parts for learning. the latter is an inter-sentence objective, with an assumption that: event triggers have same types may occur in similar contexts, derived from the well-known distributional hypothesis of words @xcite . we take in pairs of mask-containing sentences as input, and encourage their contextual representations to be similar if the masked triggers express the same type of events. to verify the effectiveness of our approach, we have conducted extensive experiments on the benchmark event dataset, and we show the definite advantages of our approach over previous methods with respect to: 1) defending against adversarial attack, 2) tackling unseen predicates, and 3) handling ambiguity cases. moreover, a deeper analysis suggests that our approach can learn a complementary predictive bias with the existing ed models using full context for reasoning. contributions. 1) in this work, we stress the importance of robustness modeling in ed, a problem less studied in the existing literature. we pinpoint three stark cases demonstrating the brittleness of existing ed methods, with qualitative evaluation, and analyze the underlying reason. 2) we propose a new training paradigm, called contextselective mask generalization, which can effective mine context-specific patterns for ed, shedding lights on building ed systems of decent robustness. 3) we report on extensive experiments demonstrating the advantages of our model in defending against adversarial attack, handling unseen predicates, and tackling ambiguous cases. we also give a deeper analysis exploring the predictive bias of our method. ## approach figure 2 visualizes the overview of our approach, by taking s1 as an example. let a sentence of n words be @xmath1, w 2 , ..., w n ]. following previous works @xcite @xcite , we formulate the ed task as a token-level classification problem. that is, for each word in s, we consider it as a candidate trigger, and our goal is to assign a correct event label to it (a type of nil is used to indicate a non-trigger word). the technical details of our approach are presented in the following, including: trigger delexicalization ( § 3.1), context-selective discriminative learning ( § 3.2), contextualized similarity learning ( § 3.3), attentive representation fusion ( § 3.4), and the training strategy ( § 3.5). ## experimental setups datasets and evaluations. we take ace 2005 and kbp 2017 as the benchmark datasets. for ace 2005, we split the corpus as training/developing/testing sets as recommend in previous works @xcite . for kbp 2017, we adopt the official evaluation settings for training and testing. for evaluations, we adopt precision (p), recall (r), and f1-score (f1) as evaluation metrics, same as previous works for a meaningful comparison. we use two-tailed wilcoxon test for significant test, with a significance level @xmath2.05. implementation details. our model is implemented with bert large , which has 24 layers, 1024 hidden units, and 16 heads, and is pre-trained on large text corpora. we tune hyper-parameters via grid search on the developing set. finally, the learning rate is set as 1e -5 (from [1e -5 , 2e -5 to 1e -4 ]); the batch size is set as 10 (from [2, 5 to 10]). a negative sampling rate of 0.7 is adopt to tackle the unbalance of positive and negative examples @xcite . as in kbp 2017 one event trigger might express multiple event types simultaneously, we adapt the multi-label cross entropy loss to binary cross-entropy loss, and a threshold of 0.3 is used for prediction. baselines. the following models are used as baselines: 1) dnned, which adopts a feedforward neural network for the task -it completely ignores context information; 2) dmcnn @xcite and 3) rnned ( nguyen et al., model pre. rec. f1 dnned 68.6 64.9 66.7 dmcnn (2015) 75.6 63.6 69.1 jrnn (2016) 66.0 73.0 69.3 jmee (2018b) 76.3 71.3 73.7 delta-adv (2019) 76.3 71.9 74.0 m bert 74.1 73.1 73.6 m full 75.2 74.4 74.8* m mask 47.7 42.7 45.0 m mask w/o sel 46.2 40.2 43.0 m mask w/o sim 45.0 40.6 42.7 ## conclusion and future work this paper focuses on the robustness of ed. we highlight three stark cases showing the brittleness of existing ed models. then we propose a new approach called context-selective masking generalization shedding lights on robustifying an ed model. in future, we would like to extend our method to other tasks where exploiting context information is crucial, such as named entity recognition and relation extraction.