Self-Attention Graph Residual Convolutional Networks for Event Detection with dependency relations (2021)

Abstract: Event detection (ED) task aims to classify events by identifying key event trigger words embedded in a piece of text. Previous research have proved the validity of fusing syntactic dependency relations into Graph Convolutional Networks(GCN). While existing GCN-based methods explore latent node-to-node dependency relations according to a stationary adjacency tensor, an attention-based dynamic tensor, which can pay much attention to the key node like event trigger or its neighboring nodes, has not been developed. Simultaneously, suffering from the phenomenon of graph information vanishing caused by the symmetric adjacency tensor, existing GCN models can not achieve higher overall performance. In this paper, we propose a novel model Self-Attention Graph Residual Convolution Networks (SA-GRCN) to mine node-to-node latent dependency relations via self-attention mechanism and introduce Graph Residual Network (GResNet) to solve graph information vanishing problem. Specifically, a self-attention module is constructed to generate an attention tensor, representing the dependency attention scores of all words in the sentence. Furthermore, a graph residual term is added to the baseline SA-GCN to construct a GResNet. Considering the syntactically connection of the network input, we initialize the raw adjacency tensor without processed by the self-attention module as the residual term. We conduct experiments on the ACE2005 dataset and the results show significant improvement over competitive baseline methods.

## introduction event detection(ed) is an important information extraction task that aims to match patterns of events in a context.the event patterns and the event contexts define event types. @xcite generally, the * corresponding authors. event type of each event context is labeled by a trigger word or phrase, called "event trigger". however, when performing event detection from an sentence-level perspective, one of the most common scenarios is that there will be multiple event triggers in the same sentence, which means that it is necessary to recognize all the event triggers and classify them into specific event types. taking figure 1 as an example, ed is supposed to recognize the event trigger "death" and "wounded" and classify them to the event type "die" and "injure". however, existing gcn-based ed methods @xcite updates the graph by an adjacency tensor which results from the syntactic analysis. such a graph structure only pay attention to the directly connected nodes. as shown in figure 1 , the dependency labels "nsubj" (nominal subject) and "dobj"(direct object) show that "center"and "deaths" are directly connected to the root node "recorded" respectively. however, for the event type "injure", event trigger "wounded" can not be recognized even though it's connected to the "dobj" node "deaths" with "nmod" (noun compound modifier) dependency. such an observation indicates that for multiple ed tasks, some event triggers may be ignored if they are indirectly connected to the root node. therefore, to perform multiple ed tasks on a single sentence, it's infeasible to rely solely on the dependency labels obtained by syntactic analysis. it's necessary to pay attention to the event triggers which are indirectly connected to the root node. in addition, even for the nodes which are indirectly connected with the dependency labels "nsubj" and "dobj" may assist detecting the single event trigger. specifically, taking figure 1 as an example: the "nsubj" dependency label associates "center" with "death", but the "appos" (apposition modifier) of "center", "hospital", has a closer dependency relations with "death". as the entity labeled by the ace2005, both "center" and "hospital" are also classified as "org-medical-science". in this case, modifier node like "hospital" should be viewed as a key node to discover latent dependency relations, in other words, node like "hospital" should be giving a higher attention weights when updating the graph. although ee-gcn @xcite proposed to explore latent dependency relations by aggregating information from neighbors of each node through specific edge, it will also edgeenahance noisy dependency relations. nosiy nodes like "punct", "det"connected with key nodes will interfere the event detection and dependency relations like "nmod" or "nummod" will be wrongly constituted between irrelevant neighboring nodes. it cannot efficiently locate the event trigger nodes in the whole graph. therefore, it's reasonable to apply the attention mechanism for more in-depth tap the latent node-to-node dependency relations. further more, as mentioned in the gres-net @xcite , the suspended animation limit problem of the existing gnns caused the information vanishing phenomenon, which means the model will respond nothing to the training data and become unlearnable when the model depth reaches such an limit. considering the syntactic connection between the word sequences, we used the encoded gcn input as the graph residual term to solve this problem. in this paper, we propose a novel architecture named self-attention graph residual convolutional networks(sa-grcn), which making full use of syntactic dependency labels to generate the graph and simultaneously giving different attention weights to the nodes on the dependency graph. resembling the existing methods, we transforms a sentence to a graph by viewing words and dependency labels as nodes and typed edges. an asymmetric adjacency tensor is initialized to represent the graph, considering the directionality of the dependency labels. to explore node-to-node latent dependency relations, a self-attention module is constructed to update the self-attention tensor of the whole graph. node-aware edge update module and edge-aware node update module are used to update the network iterativly, such mutual update process make the latent dependency relations expressed in the node representations can be effectively mined and injected to the attention tensor. via a dynamic updated self-attention tensor and a initialized adjacency tensor, our architecture can better capture the interrelations between candidate trigger words and related entities. our contributions are summarized as follows: we propose the novel self-attention graph residual convolutional network, introducing the attention mechanism into gcn, which simultaneously integratse syntactic structure and latent dependency relations to improve the performance of event detection based on existing gcn methods. what's more, gresnet is introduces to deal with the phenomenon of graph information vanishing. to our best our knowledge, this is the first time to apply self-attention mechanism to the ed task. ## related works in earlier studies, researchers perform event detection task based on statistic features, modeling the statistic distributions to cluster the event. however, all of these manually defined features, which are highly dependent on a specific data set, demand high for the preprocessing module. consequently, the robustness of all these models are poor. in recent years, with the rapid development of machine learning, many ed-oriented neural networks have been proposed @xcite @xcite . the existing approaches can be categorized into two classes: the first class is to improve ed through special learning techniques including adversarial training @xcite , knowledge distillation @xcite and model pre-training @xcite . the second class is to improve ed by introducing extra resource, such as argument information @xcite , document information @xcite @xcite , knowledge base @xcite and syntactic dependency labels @xcite . dependcy labels can be viewed as a kind of syntactic information @xcite @xcite , which is essential for the gcn-based ed task. @xcite exploited typed-dependency labels to integrated dependency tree into gcn models. compare with the binary adjacency tensor of vanilla gcn, @xcite construct a symmetric adjacency tensor via the dependency trees to denote the graph structure. @xcite also proposed edge-enhanced to explore the latency dependency relations, but it also brings a lot of useless noisy relations. although @xcite tried to apply attention ## methods in this section, we will introduce our self-attention graph residual convolutional network (sa-grcn) architecture to explore potential event triggers including the embedding method, rnn encode module,self-attention collect module and gresnet. furthermore, we introduce a residual term to our novel architecture to solve the information vanishing phenomenon caused by the symmetric adjacency tensor. ## effectiveness of self-attention collect module as shown in fig3, the overall outputs of the selfattention collect module can be divided into the sub attention output and obj attention output. since the hiddern layer of the self-attention collect module represent the latent dependency relations, the dimension of the output is n×dim. experiment results show that the sub attention output of the 9th node "hospital" and 13th node "deaths" are marked as dark blue, which confirms that our selfattention module explore the key node "hospital" to detect event "die". we also notice that "medical center" which directly connected to the root node in fig1, weigh less than the key node "hospital". in obj attention output, event trigger "wounded" is given a high attention weight to explore latent dependency relations. what's more, noisy node like "," and "with" as we mentioned above, are marked in light blue or white, representing a lower attention weight. finally, both "die" and "injure" are detected by sa-grcn. ## conclusions and future works in this paper, we propose a novel model named self-attention graph residual convolutional networks (sa-grcn) for event detection. sa-grcn introduces the typed dependency label information into the graph modeling process, and learns to update the relation representations in a context-dependent manner. experiments show that our model achieves the start-of-the-art results on the ace2005 dataset. however, the shortcomings of our architecture is that as the depth of the network layer increases, the introduction of the attention mechanism can greatly improve the accuracy of prediction and reduce the recall score. currently, gresnet provides a solution that via introducing manually defined residual term and hyperparameter α to balance the overall performance of the model. our follow-up research work is through mathematical modelling analysis to find a more suitable residual term, and set α to be a learnable parameter.