Learning to Prune Dependency Trees with Rethinking for Neural Relation Extraction (2020)

Abstract: Dependency trees have been shown to be effective in capturing long-range relations between target entities. Nevertheless, how to selectively emphasize target-relevant information and remove irrelevant content from the tree is still an open problem. Existing approaches employing pre-defined rules to eliminate noise may not always yield optimal results due to the complexity and variability of natural language. In this paper, we present a novel architecture named Dynamically Pruned Graph Convolutional Network (DP-GCN), which learns to prune the dependency tree with rethinking in an end-to-end scheme. In each layer of DP-GCN, we employ a selection module to concentrate on nodes expressing the target relation by a set of binary gates, and then augment the pruned tree with a pruned semantic graph to ensure the connectivity. After that, we introduce a rethinking mechanism to guide and refine the pruning operation by feeding back the high-level learned features repeatedly. Extensive experimental results demonstrate that our model achieves impressive results compared to strong competitors.

## introduction relation extraction (re) aims to detect the semantic relationship between two specific entities appearing in a sentence (often termed subject and object, respectively). this task plays an important role in many downstream nlp applications that require a relational understanding of unstructured text such as question answering @xcite and dialogue systems @xcite . models leveraging the dependency tree of the input sentence have proven to be effective in relation extraction because they can effortlessly exploit long-term relations that are obscure from the surface form @xcite . recent studies also stated that not all tokens in the dependency tree are needed to express the relation of the target entity pair @xcite , and some target-irrelevant tokens could introduce noise and cause confusion to the classification. therefore, multiple pruning strategies are proposed to eliminate unimportant tokens and distill the dependency information. @xcite applied neural networks only on the shortest dependency path (sdp) between the two entities in the dependency tree, which soon became dominant with many works demonstrating that using sdp brings better experimental results than using the whole sentence @xcite . @xcite reduced the full tree to the subtree below the lowest common ancestor (lca) of the entities. @xcite expanded sdp by including tokens that are up to distance k away from the dependency path in the lca subtree. however, these hand-crafted pruning rules may lead to the omission of useful information due to the variability and ambiguity of natural language. look at a concrete example shown in figure 1 , the key relational token "founded" is always excluded from the pruned tree no matter what kind of pruning rule mentioned above is deployed. in fact, it's unrealistic to expect an empirical rule to deal with all situations, and an ideal dependency-based comp appos founded in 1947 by two brothers, eugene and quentin fabris, new fabris started out making sewing machine out case nmod nmod num case cc comp prt conj comp nsubj xcomp amod relcl dobj root figure 1 : an example dependency tree expressing a relation org:founded-by between "eugene" and "new fabris". note that even the most relaxed pruning rule lca subtree (highlighted in bold) still excludes the relational token "founded", resulting in the loss of crucial information. in this paper, we propose a novel architecture named dynamically pruned graph convolutional network (dp-gcn), which takes the full dependency tree as input and learns to prune the tree with rethinking in an end-to-end training manner. at the heart of dp-gcn is a selection module that dynamically identifies a subset of critical nodes in the dependency tree that provide sufficient information to extract the relation between two entities. this module takes into account the semantics of each node and the target entities and generates a set of input-dependent binary gates to determine whether each node should be kept. one problem coming with dynamic pruning is that selecting sub-structure from the dependency tree directly may result in a disconnected topology because the dependency tree is sparse, which hinders the message propagation between nodes. to address this issue, we enhance the pruned tree with a pruned semantic graph generated by the self-attention mechanism. then on top of the resulting graph, a gcn module (kipf and welling, 2016) is exploited to update the entity-specific context representations, and such a prune-then-update process can be stacked over l layers. furthermore, instead of pruning the tree based on one-pass of the data through the network, we introduce feedback connections and endow the network with the ability to "rethink" the pruning operation by transferring the high-level features into the selection module of each layer. benefiting from the rethinking mechanism, the model is able to reselect nodes with consideration of previous pruning information and extract more discriminative target-specific features with the guidance from the high-level semantic. to summarize, our contributions are three-fold: our work is inspired by two lines of research: enhancing relation extractor through syntactic dependency information and refining neural network with the rethinking mechanism. ## methodology the goal of our model is to predict the relationship between two entities in a given sentence. figure 2 illustrates the overall architecture of the proposed model, which can be classified into three components : (1) the left panel is a bilstm encoder that transforms the input words into the contextualized representations. (2) the middle part, as the core of the whole model, contains l layers of dp-gcns, which incorporate entity information into the graph modeling process and filter useless information for the given entities. (3) the right panel is a pooling module used to aggregates node representations induced from the former dp-gcn layers. next, we detail all components sequentially from left to right.