Event Detection: Gate Diversity and Syntactic Importance Scores for Graph Convolution Neural Networks (2020)

Abstract: Recent studies on event detection (ED) have shown that the syntactic dependency graph can be employed in graph convolution neural networks (GCN) to achieve state-of-the-art performance. However, the computation of the hidden vectors in such graph-based models is agnostic to the trigger candidate words, potentially leaving irrelevant information for the trigger candidate for event prediction. In addition, the current models for ED fail to exploit the overall contextual importance scores of the words, which can be obtained via the dependency tree, to boost the performance. In this study, we propose a novel gating mechanism to filter noisy information in the hidden vectors of the GCN models for ED based on the information from the trigger candidate. We also introduce novel mechanisms to achieve the contextual diversity for the gates and the importance score consistency for the graphs and models in ED. The experiments show that the proposed model achieves state-of-the-art performance on two ED datasets.

## introduction event detection (ed) is an important task in information extraction of natural language processing. the main goal of this task is to identify event instances presented in text. each event mention is associated with a word or a phrase, called an event trigger, which clearly expresses the event @xcite . the event detection task, precisely speaking, seeks to identify the event triggers and classify them into some types of interest. for instance, consider the following sentences: the dominant approaches for ed involve deep neural networks to learn effective features for the input sentences, including separate models @xcite and joint inference models with event argument prediction @xcite . among those deep neural networks, graph convolutional neural networks (gcn) @xcite have achieved state-of-the-art performance due to the ability to exploit the syntactic dependency graph to learn effective representations for the words @xcite @xcite . however, two critical issues should be addressed to further improve the performance of such models. first, given a sentence and a trigger candidate word, the hidden vectors induced by the current gcn models are not yet customized for the trigger candidate. as such, the trigger-agnostic representations in the gcn models might retain redundant/noisy information that is not relevant to the trigger candidate. as the trigger candidate is the focused word in the sentence, that noisy information might impair the performance of the ed models. to this end, we propose to filter the noisy information from the hidden vectors of gcns so that only the relevant information for the trigger candidate is preserved. in particular, for each gcn layer, we introduce a gate, computed from the hidden vector of the trigger candidate, serving as the irrelevant information filter for the hidden vectors. besides, as the hidden vectors in different layers of gcns tend to capture the contextual information at different abstract levels, we argue that the gates for the different layers should also be regulated to exhibit such abstract representation distinction. hence, we additionally introduce a novel regularization term for the overall loss function to achieve these distinctions for the gates. second, the current gcn models fail to consider the overall contextual importance scores of every word in the sentence. in previous gcn models, to produce the vector representation for the trigger candidate word, the gcn models mostly focus on the closest neighbors in the dependency graphs @xcite . however, although the non-neighboring words might not directly carry useful context information for the trigger candidate word, we argue that their overall importance scores/rankings in the sentence for event prediction can still be exploited to provide useful training signals for the hidden vectors in ed. in particular, we propose to leverage the dependency tree to induce a graph-based importance score for every word based on its distance to the trigger candidate. afterward, we propose to incorporate such importance scores into the ed models by encouraging them to be consistent with another set of model-based importance scores that are computed from the hidden vectors of the models. based on this consistency, we expect that graph-based scores can enhance the representation learning for ed. in our experiments, we show that our method outperforms the state-of-the-art models on the benchmark datasets for ed. ## related work prior studies on ed involve handcrafted feature engineering for statistical models @xcite @xcite @xcite and deep neural networks, e.g., cnn @xcite @xcite , @xcite @xcite , attention mechanism @xcite , contextualized embeddings @xcite , and adversarial training @xcite . the last few years witness the success of graph convolutional neural networks for ed @xcite @xcite where the dependency trees are employed to boost the performance. however, these graph-based models have not considered representation regulation for gcns and exploiting graph-based distances as we do in this work. our ed model consists of three modules: (1) sentence encoder, (2) gcn and gate diversity, and (3) graph and model consistency. sentence encoder: we employ the pre-trained bert @xcite to encode the given sentence x. in particular, we create an input sequence of our gcn module contains l stacked gcn layers @xcite , operating over the sequence of hidden vectors h 0 . the hidden vector the major issue of the current gcn for ed is that its hidden vectors h l i are induced without special awareness of the trigger candidate x t . this might result in irrelevant information (for the trigger word candidate) in the hidden vectors of gcns for ed, thus hindering further performance improvement. to address this problem, we propose to filter that unrelated information by introducing a gate for each gcn layer. the vector g l for the gate at the l-th layer is computed from the embedding vector e t of the trigger candidate: as each layer in the gcn module has access to a particular degree of neighbors, the contextual information captured in these layers is expectedly distinctive. besides, the gates for these layers control which information is passed through, therefore, they should also demonstrate a certain degree of contextual diversity. to this end, we propose to encourage the distinction among the outcomes of these gates once they are applied to the hidden vectors in the same layers. particularly, starting with the hidden vectors h l of of the l-layer, we apply the gates g k (for all (1 ≤ @xmath0, which results in a sequence of filtered vectors mk,l @xmath1. afterward, we aggregate the filtered vectors obtained by the same gates using max-pooling: mk,@xmath2,l 1 , • • • , mk,l n ). to encourage the gate diversity, we enforce vector separation between ml,l with all the other aggregated vectors from the same layer l (i.e., mk,l for @xmath3. as such, we introduce the following cosinebased regularization term l gd (for gate diversity) into the overall loss function: graph and model consistency: as stated above, we seek to supervise the model using the knowledge from the dependency graph. inspired by the contextual importance of the neighboring words for the event prediction of the trigger candidate x t , we compute the graph-based importance scores @xmath4, • • • , p n in which p i is the negative distance from the word x i to the trigger candidate. in contrast, the model-based importance scores for each word x i is computed based on the hidden vectors of the models. in particular, we first form an overall feature vector v t that is used to predict the event type for x t via: in this work, we argue that the hidden vector of an important word in the sentence for ed should carry more useful information to predict the event type for x t . therefore, we consider a word x i as more important for the prediction of the trigger candidate x t if its representation m l i is more similar to the vector v t . we estimate the model-based important scores for every word x i with respect to the candidate x t as follow: to predict the event type, we feed v t into a fully connected network with softmax function in the end to estimate the probability distribution p (ŷ|x, t). to train the model, we use the negative log-likelihood as the classification loss l @xmath5, t). finally, we minimize the following combined loss function to train the proposed model: as the sizes of the ed dataset are generally small, the pre-processing procedures (e.g., tokenization, sentence splitting, dependency parsing, and selection of negative examples) might have a significant effect on the models' performance. for instance, the current best performance for ed on ace-2005 is reported by @xcite (i.e., 80.7% f1 score on the test set). however, once we re-implement this model and apply it to the data version pre-processed and provided by the prior work @xcite , we are only able to achieve an f1 score of 76.2% on the test set. as the models share the way to split the data, we attribute such a huge performance gap to the difference in data pre-processing that highlights the need to use the same pre-processed data to measure the performance of the ed models. consequently, in this work, we employ the exact data version that has been pre-processed and released by the early work on ed for @xcite @xcite and for litbank in @xcite . the hyper-parameters for the models in this work are tuned on the development datasets, leading to the following selected values: one layer for the bilstm model with 128 hidden units in the layers, @xmath6, 128 hidden units for the layers of all the feed-forward networks in this work, and 5e-5 for the learning rate of the adam optimizer. these values apply for both the ace-2005 and litbank datasets. for the trade-off coefficients α and β in the overall loss function, we use α = 0.1 and β = 0.2 for the ace dataset while α = 0.3 and β = 0.2 are employed for litbank. finally, we use the case bert base version of bert and freeze its parameters during training in this work. to obtain the bert representations of the word pieces, we use @xmath7. @xcite . note that these baselines use the same pre-processed data like ours. the second class of baselines concern the models with the contextualized embeddings, i.e., dmbert: a model with dynamic pooling @xcite and bert+mlp: a mlp model with bert @xcite . these models currently have the bestreported performance for ed on ace-2005. note that as these works employ different pre-processed versions of ace-2005, we re-implement the models and tune them on our dataset version for a fair comparison. for litbank, we use the following baselines reported in the original paper @xcite : bilstm: a bilstm model with word2vec, bert+bilstm: a bilstm model with bert, and dmbert @xcite . ablation study: the proposed model involves three major components: (1) the gates to filter irrelevant information, (2) the gate diversity to encourage contextual distinction for the gates, and (3) the consistency between graph and model-based importance scores. ## conclusion