GTN - ED : Event Detection Using Graph Transformer Networks (2021)

Abstract: Recent works show that the graph structure of sentences, generated from dependency parsers, has potential for improving event detection. However, they often only leverage the edges (dependencies) between words, and discard the dependency labels (e.g., nominal-subject), treating the underlying graph edges as homogeneous. In this work, we propose a novel framework for incorporating both dependencies and their labels using a recently proposed technique called Graph Transformer Network (GTN). We integrate GTN to leverage dependency relations on two existing homogeneous-graph-based models and demonstrate an improvement in the F1 score on the ACE dataset.

## introduction event detection is an important task in natural language processing, which encompasses predicting important incidents in texts, e.g., news, tweets, messages, and manuscripts @xcite @xcite @xcite @xcite @xcite . as an example, consider the following sentence: the plane arrived back to base safely. here, the word arrived is an event trigger that denotes an event of the type "movement:transport," while "the plane" and "base" are its arguments. given a sentence, the objective of the event detection task is to predict all such event triggers and their respective types. recent works on event detection @xcite @xcite employ graph based methods @xcite ) using the dependency graph (shown in fig. 1 ) generated from syntactic dependencyparsers. these methods are able to capture useful non-local dependencies between words that are * s. dutta was a research intern at dataminr. relevant for event detection. however, in most of these works (with the notable exception of cui et al. ( 2020 )), the graph is treated as a homogeneous graph, and the dependency labels (i.e., edge-types in the graph) are ignored. dependency labels can often better inform whether a word is a trigger or not. consider the two sentences in fig. 1 . in both the sentences, there is an edge between "police" and "fired". a model that does not take into account dependency labels will only have access to the information that they are connected. however, in the first sentence, "fired" is an event trigger of type "conflict:attack," whereas in the second sentence, it is of type "personnel:end position." the fact that the edge label between "police" and "fired" is a nominal-subject or an object relation serves as an indicator of the type of event trigger. hence, leveraging the dependency labels can help improve the event detection performance. in this work, we propose a simple method to employ the dependency labels into existing models inspired from a recently proposed technique called graph transformer networks (gtn) @xcite . gtns enable us to learn a soft selection of edge-types and composite relations (e.g., multi-hop connections, called meta-paths) among the words, thus producing heterogeneous adjacency matrices. we integrate gtns into two homogeneousgraph-based models (that previously ignored the dependency relations), namely, a simple gated-graphconvolution-based model inspired by nguyen and grishman (2018); @xcite ; @xcite , and the near-state-of-the-art moganed model @xcite , enabling them to now leverage the dependency relations as well. our method demonstrates a relative improvement in the f1 score on the ace dataset @xcite for both models, proving the value of leveraging dependency relations for a graph-based model. while the goal of this paper is not to establish a state-ofthe-art (sota) method, but rather to show the merit of our approach, we do note that the improvements with our method approach the current sota @xcite (which leverages dependency relations using embeddings instead of gtns). to summarize, our main contribution is a method of enabling existing homogeneous-graph-based models to exploit dependency labels for event detection, inspired from gtns. incorporating gtns in nlp tasks has received less attention (also see recent related work veyseh et al. ( 2020 )). notations: we denote matrices and vectors in bold, e.g., a (matrix) or a (vector). note that, a(u, v) denotes the element at index (u, v) in matrix a. ## proposed method in this work, we incorporate gtns onto two homogeneous-graph-based models: (i) model i: a gated-graph-convolution-based model inspired by nguyen and grishman (2018); @xcite ; @xcite ; and (ii) model ii: moganed model @xcite . both models have a similar initial embedding and bilstm module, followed by a graph-based module (where their differences lie), and finally a classification module. embedding and bilstm module: our initial module (shown in fig. 2 ) is similar to existing works (e.g., @xcite ). each word of the sentence is represented by a token which consists of the word embedding, the pos tag embedding, the named-entity type embedding, and its positional embedding. for a sentence of n words, we denote this sequence of tokens as @xmath0, x 1 , . . . , x n-1 . next, we introduce a bilstm to encode x into its context @xmath1, p 1 , . . . , p n-1 where here, w k a and b k a are the weight matrix and bias item for the adjacency matrix a at layer k, and a dependency parser, e.g., stanford core nlp @xcite , generates a directed heterogeneous graph g for each sentence (recall fig. 1 ). existing works typically do not use the dependency labels (e.g., nominal-subject); they only derive three homogeneous adjacency matrices from g as follows: (i) a f wd where a f wd (i, j) = 1 if there is an edge from node i to j; (ii) a rev where a rev (i, j) = 1 if there is an edge from node j to i; and (iii) a loop which is an identity matrix. for model i (see fig. 4 (left)), the output of the k-th layer (input to k +1-th layer) is given by h k+1 @xmath2,arev,a loop } f u (h k , a)). the first layer of gated-graph-convolution network captures dependencies between immediate neighbors (1-hop). to capture k-hop dependencies, model i has k consecutive layers of such gatedgraph-convolution networks. the output of this graph-based module is then fed to a multi-layer perceptron (mlp) with attention weights for classifying each word into its event-type (or "not an event"). in model ii, instead of passing the bilstm output p through a series of k consecutive gatedgraph-convolution layers (to capture k-hop connections), this model separately aggregates the outputs of t parallel graph-convolution layers with separate adjacency matrices representing hops of length 1, 2, . . . , t (see fig. 4 (right)). let h 0 (= p ) be the input and h foot_0 be the output of the graph-based module of model ii (which effectively has only one layer, i.e., @xmath3. in @xcite , the homogeneous adjacency matrices a f wd , a rev , and a loop are considered with their corresponding t-hop adjacency matrices a t f wd , a t rev , and a t loop (multiplied t times) respectively. the output of the graph-based module is given by: h 1 @xmath4,t v t where v @xmath5,arev,a loop } σ(f u (h 0 , a t )). here, w att,t is an attention-weight (further details in @xcite ) and σ(•) is the exponential linear unit 1 . finally, these outputs are passed through an mlp with attention weights for classification. remark. the reason for using only three matrices instead of a separate adjacency matrix for each edge-type is that it results in an explosion of parameters for the gated-graph-convolution network, as individual weight matrices have to be learnt for each type of edge (see also @xcite ). in this work, we replace the homogeneous matrices a f wd and a rev with heterogeneous adjacency matrices without a significant overhead in the number of parameters, as discussed next. obtaining heterogeneous adjacency matrices with gtn: consider a directed heterogeneous graph g with each edge belonging to one of l types. this graph can be represented using a set of l adjacency matrices {a f wd,0 , a f wd,1 , . . . , a f wd,l-1 }, each corresponding to a different edge-type (dependency label). a f wd,l (i, j) = 1 if there is a directed edge from node i to j of type l. a gtn obtains a heterogeneous adjacency matrix by learning a convex combination q f @xmath6,l (see fig. 5 ) where α = softmax(w) and w is a weight vector that the model learns. the matrix q f wd is a heterogeneous adjacency matrix with an "appropriately weighted" edge between any two nodes that have an edge in any of the l original matrices. for model i, we first generate a set of l adjacency matrices (for l edge-types) corresponding to the directed forward edges, and another set of l adjacency matrices corresponding to the reverse edges. next, we learn heterogeneous adjacency matrices, i.e., a f @xmath7. our technique enables baseline model i to leverage dependency relations by learning only 2l more scalar parameters which is significantly less than learning individual weight matrices for l edge-types. for model ii, we not only aim to learn heterogeneous adjacency matrices to replace the homogeneous a f wd and a rev , but also learn heterogeneous adjacency matrices that have an "appropriately weighted" edge between every two nodes that are t-hops apart in the original graph g (called a meta-path of length t) so as to replace a t f wd and a t rev . specifically, for the case of @xmath8, gtn first learns two convex combinations q f wd,0 and q f wd,1 (each corresponds to meta-paths of length 1), and then computes the product q f wd,0 q f wd,1 . similarly, one can compute a product of t such adjacency matrices to learn meta-paths of length t. we replace all t-hop adjacency matrices with heterogeneous adjacency matrices of meta-path length t, learnt through gtns, e.g., a t f wd is replaced by q f wd,0 q f wd,1 . . . q f wd,t-1 , where each q f wd,i is a convex combination of l-adjacency matrices corresponding to the directed forward edges. similar heterogeneous adjacency matrices of meta-path length t are learnt for the reverse edges as well to replace a t rev . this modification enables the baseline model ii to leverage the dependency relations, by only learning 2lt more scalar parameters for each t, which is practicable. ## results dataset and evaluation metrics: we use the benchmark ace2005 english dataset @xcite with the same data split as in prior works (where the sentences from 529, 30, and 40 docu-type conflict:attack. our model is able to identify "war," and "conflict" as non-triggers based on their context in the sentence, while the baseline seems to be over-emphasizing on their literal meaning. in some cases, the baseline model also leads to misclassification. for instance, the apache troop opened its tank guns,[...]. here, "opened," is an event trigger of type conflict:attack, as is correctly identified by our model; however, the baseline misclassifies it as type movement:transport. another interesting example is: [...] beatriz walked into the uscf offices in new windsor and immediately fired 17 staff members. here, "walked" is an event trigger of type movement:transport, and "fired" is of type personnel:end-position. the baseline model misclassifies "fired" as conflict:attack, while using gtns help classify it correctly. however, using gtns can sometimes miss certain event triggers while attempting to be more precise, e.g., "walked" is missed when using gtns while the baseline model identifies it correctly. lastly, there are examples where both the baseline and proposed models make the same errors. e.g., i visited all their families. or, i would have shot the insurgent too. here, both models misclassify "visited," (type contact:meet) as movement:transport, and "shot," (type life:die) as conflict:attack. as future work, we are examining alternate techniques that better inform the context of the event trigger in such sentences. another interesting example is: "it is legal, and it is done." both models miss "it," (type transaction:transfer-money). for this example (and some other similar examples of anaphora resolution), we believe that it might be quite non-intuitive to classify the event trigger from the sentence alone, and dependencies among sentences from the same article might need to be leveraged to better inform the context, as we will examine in future work. ## conclusion we developed a novel method of enabling existing event extraction models to leverage dependency relations without a significant rise in the number of parameters to be learnt. our method relies on gtn, and demonstrates an improvement in f1 score over two strong baseline models that do not leverage dependency relations. the benefits of using gtn in an nlp task suggests that other nlp tasks could be improved in the future.