Exploiting the Syntax-Model Consistency for Neural Relation Extraction (2020)

Abstract: This paper studies the task of Relation Extraction (RE) that aims to identify the semantic relations between two entity mentions in text. In the deep learning models for RE, it has been beneficial to incorporate the syntactic structures from the dependency trees of the input sentences. In such models, the dependency trees are often used to directly structure the network architectures or to obtain the dependency relations between the word pairs to inject the syntactic information into the models via multi-task learning. The major problem with these approaches is the lack of generalization beyond the syntactic structures in the training data or the failure to capture the syntactic importance of the words for RE. In order to overcome these issues, we propose a novel deep learning model for RE that uses the dependency trees to extract the syntax-based importance scores for the words, serving as a tree representation to introduce syntactic information into the models with greater generalization. In particular, we leverage Ordered-Neuron Long-Short Term Memory Networks (ON-LSTM) to infer the model-based importance scores for RE for every word in the sentences that are then regulated to be consistent with the syntax-based scores to enable syntactic information injection. We perform extensive experiments to demonstrate the effectiveness of the proposed method, leading to the state-of-the-art performance on three RE benchmark datasets.

## introduction one of the fundamental tasks in information extraction (ie) is relation extraction (re) where the goal is to find the semantic relationships between two entity mentions in text. due to its importance, re has been studied extensively in the literature. the recent studies on re has focused on deep learning to develop methods to automatically induce sen-tence representations from data @xcite @xcite . a notable insight in these recent studies is that the syntactic trees of the input sentences (i.e., the dependency trees) can provide effective information for the deep learning models, leading to the stateof-the-art performance for re recently @xcite @xcite . in particular, the previous deep learning models for re has mostly exploited the syntactic trees to structure the network architectures according to the word connections presented in the trees (e.g., performing graph convolutional neural networks (gcn) over the dependency trees @xcite ). unfortunately, these models might not be able to generalize well as the tree structures of the training data might significantly differ from those in the test data (i.e., the models are overfit to the syntactic structures in the training data). for instance, in the cross-domain setting for re, the domains for the training data and test data are dissimilar, often leading to a mismatch between the syntactic structures of the training data and test data. in order to overcome this issue, the overall strategy is to obtain a more general representation of the syntactic trees that can be used to inject the syntactic information into the deep learning models to achieve better generalization for re. a general tree representation for re is presented in @xcite where the dependency trees are broken down into their sets of dependency relations (i.e., the edges) between the words in the sentences (called the edge-based representation). these dependency relations are then used in a multitask learning framework for re that simultaneously predicts both the relation between the two entity mentions and the dependency connections between the pairs of words in the input sentences. although the dependency connections might be less specific to the training data than the whole tree structures, the major limitation of the edge-based representation is that it only captures the pairwise (local) connections between the words and completely ignores the overall (global) importance of the words in the sentences for the re problem. in particular, some words in a given sentence might involve more useful information for relation prediction in re than the other words, and the dependency tree for this sentence can help to better identify those important words and assign higher importance scores for them (e.g., choosing the words along the shortest dependency paths between the two entity mentions). we expect that introducing such importance information for the words in the deep learning models might lead to improved performance for re. consequently, in this work, we propose to obtain an importance score for each word in the sentences from the dependency trees (called the syntax-based importance scores). these will serve as the general tree representation to incorporate the syntactic information into the deep learning models for re. how can we employ the syntax-based importance scores in the deep learning models for re? in this work, we first use the representation vectors for the words from the deep learning models to compute another importance score for each word (called the model-based importance scores). these model-based importance scores are expected to quantify the semantic information that a word contributes to successfully predict the relationship between the input entity mentions. afterward, we propose to inject the syntax-based importance scores into the deep learning models for re by enforcing that the model-based importance scores are consistent with the syntactic counterparts (i.e., via the kl divergence). the motivation of the consistency enforcement is to promote the importance scores as the bridge through which the syntactic information can be transmitted to enrich the representation vectors in the deep learning models for re. in order to implement this idea, we employ the ordered-neuron long short-term memory networks (on-lstm) @xcite to compute the model-based importance scores for the words in the sentences for re. on-lstm extends the popular long short-term memory networks (lstm) by introducing two additional gates (i.e., the master forget and input gates) in the hidden vector computation. these new gates controls how long each neuron in the hidden vectors should be activated across different time steps (words) in the sentence (i.e., higher-order neurons would be maintained for a longer time). based on such controlled neurons, the model-based importance score for a word can be determined by the number of active neurons that the word possesses in the operation of on-lstm. to our knowledge, this is the first time on-lstm is applied for re in the literature. one of the issues in the original on-lstm is that the master gates and the model-based importance score for each word are only conditioned on the word itself and the left context encoded in the previous hidden state. however, in order to infer the importance for a word in the overall sentence effectively, it is crucial to have a view over the entire sentence (i.e., including the context words on the right). to this end, instead of relying only on the current word, we propose to obtain an overall representation of the sentence that is used as the input to compute the master gates and the importance score for each word in the sentence. this would enrich the model-based importance scores with the context from the entire input sentences, potentially leading to the improved re performance of the model in this work. finally, to further improve the representations learned by the deep learning models for re, we introduce a new inductive bias to promote the similarity between the representation vectors for the overall sentences and the words along the shortest dependency paths between the two entity mentions. the intuition is that the relation between the two entity mentions of interest in a sentence for re can be inferred from either the entire sentence or the shortest dependency path between the two entity mentions (due to the demonstrated ability of the shortest dependency path to capture the important context words for re in the prior work @xcite ). we thus expect that the representation vectors for the sentence and the dependency path should be similar (as both capture the semantic relation) and explicitly exploiting such similarity can help the models to induce more effective representations for re. our extensive experiments on three benchmark datasets (i.e., ace 2005, spouse and scierc) demonstrate the effectiveness of the proposed model for re, leading to the state-of-the-art performance for these datasets. ## model the re problem can be formulated as a multi-class classification problem. formally, given an input sentence @xmath0, w 2 , . . . , w n where w t is the t-th word in the sentence w of length n , and two entity mentions of interest at indexes s and o (1 ≤ @xmath1, our goal is to predict the semantic relation between w s and w o in w . similar to the previous work on deep learning for re @xcite , we first transform each word w t into a representation vector x t using the concatenation of the three following vectors: (i) the pre-trained word embeddings of w t , (ii) the position embedding vectors (to encode the relative distances of w t to the two entity mentions of interest w s and w o (i.e., t -s and t -o)), and (iii) the entity type embeddings (i.e., the embeddings of the bio labels for the words to capture the entity mentions present in x). this word-to-vector transformation converts the input sentence w into a sequence of representation vec-tors @xmath2, x 2 , . . . , x n to be consumed by the next neural computations of the proposed model. there are three major components in the re model in this work, namely (1) the ceon-lstm component (i.e., context-enriched on-lstm) to compute the model-based importance scores of the words w t , (2) the syntax-model consistency component to enforce the similarity between the syntaxbased and model-based importance scores, and (3) the similarity component between the representation vectors of the overall sentence and the shortest dependency path. ## conclusion we introduce a new deep learning model for re (i.e., ceon-lstm) that features three major proposals. first, we represent the dependency trees via the syntax-based importance scores for the words in the input sentences for re. second, we propose to incorporate the overall sentence representation vectors into the cells of on-lstm, allowing it to compute the model-based importance scores more effectively. we also devise a novel mechanism to project the syntactic information into the computation of on-lstm via promoting the consistency between the syntax-based and model-based importance scores. finally, we present a novel inductive bias for the deep learning models that exploits the similarity of the representation vectors for the whole input sentences and the shortest dependency paths between the two entity mentions for re. extensive experiments are conducted to demonstrate the benefits of the proposed model. we achieve the state-of-the-art performance on three datasets for re. in the future, we plan to apply ceon-lstm to other related nlp tasks (e.g., event extraction, semantic role labeling) @xcite .