The Dots Have Their Values: Exploiting the Node-Edge Connections in Graph-based Neural Models for Document-level Relation Extraction (2020)

Abstract: The goal of Document-level Relation Extraction (DRE) is to recognize the relations between entity mentions that can span beyond sentence boundary. The current state-of-the-art method for this problem has involved the graph-based edge-oriented model where the entity mentions, entities, and sentences in the documents are used as the nodes of the document graphs for representation learning. However, this model does not capture the representations for the nodes in the graphs, thus preventing it from effectively encoding the specific and relevant information of the nodes for DRE. To address this issue, we propose to explicitly compute the representations for the nodes in the graph-based edge-oriented model for DRE. These node representations allow us to introduce two novel representation regularization mechanisms to improve the representation vectors for DRE. The experiments show that our model achieves state-of-the-art performance on two benchmark datasets.

## introduction an important task of information extraction is relation extraction (re) that seeks to identify the semantic relationships between entities mentioned in text. the prior works have mainly focused on the intra-sentence scenario where the two entity mentions appear in the same sentences @xcite @xcite . in this work, we study a more recent setting for re that additionally considers relations between two entity mentions in different sentences of the documents (i.e., inter-sentence relations) (called document-level re (dre)). the current methods for document-level re have intensively relied on deep learning to induce effective representation vectors for relation prediction. among these deep learning models, graphbased neural networks have been demonstrated as one of the most effective approaches for dre due to their ability to capture long-distance and intersentential information in text @xcite @xcite . in particular, @xcite has recently introduced a graph-based edge-oriented network that achieves state-of-the-art performance for dre. the key idea in this model is to build a interaction graph for each input document where the nodes include the entity mentions, the entities, and the sentences. note that this is fundamentally different from the prior graph-based models for re that have mostly used words as the nodes for the graphs @xcite . in this model, the edges between these nodes are determined by the coreferences of the entity mentions and the appearance of the entity mentions in the sentences. the representation vectors for the edges of the graph (thus called edge-oriented) are then computed via several inference layers, serving as the features to predict the relations between the pairs of entities in the documents. in this way, the model can leverage the interactions between the nodes and edges of different types to obtain richer representation vectors for the edges between the entity nodes @xcite . despite its good performance, a major limitation of the graph-based edge-oriented model for dre is that it only focuses on the edge representations and ignores the representations for the nodes of the graphs. on the one hand, this edge-only representation approach cannot explicitly encode the information that is specific to the entities/entity mentions in the documents (as it only captures the representations of pairs of entities/entity mentions), potentially missing an important clue to boost the performance for dre (e.g., entity subtypes). one the other hand, the lack of the node representations prevents the models from exploiting the relations between the node and edge representations (e.g., the translation relation @xcite ) and the similarity for the representation vectors of the entity mention nodes (i.e., of the same entities) to enhance the representation learning for dre. consequently, in this work, we propose to improve the graph-based edge-oriented model for dre by explicitly computing the representations for the nodes in the graphs. based on such node representations, we introduce two novel regularization techniques to improve the representations for drp, capturing the similarities between the node and edge representations of the same edges or the same entities. we conduct extensive experiments to demonstrate the effectiveness of the proposed method for dre, yielding the state-of-the-art performance for this task on two benchmark datasets. ## related work re has been extensively studied in the intersentence @xcite @xcite @xcite @xcite and distant supervision settings @xcite @xcite @xcite @xcite . recently, document-level re has gained more attention from the community. two major approaches have been considered for dre, i.e., the graph-based @xcite @xcite @xcite and non-graph-based @xcite @xcite @xcite @xcite @xcite approaches. the closest work to ours is the graph-based edgeoriented model in @xcite that introduces new document graphs for dre based on entity mentions, entities and sentences as the nodes. ## model formally, in the dre problem, the input involves a document d with s, m , and e as the sets of the sentences, entity mentions, and entities (respectively) in d. the goal of dre is to predict the semantic relationships between each pair of entities in e (i.e., including the type none for the entity pairs with no relations). in this section, we will first describe the graph-based edge-oriented model (eog) in @xcite . after-ward, we will present our novel node representation computation and regularization for this model. ## experiments datasets & parameters: we evaluate the models on two benchmark datasets for dre, i.e., cdr and gda. the cdr (chemical-disease reactions) dataset is manually annotated for the binary interactions between chemical and disease concepts @xcite while gda (gene-disease associations) @xcite provides the annotations for the binary interactions between gene and disease concepts using distant supervision. for both datasets, we follow the same data preprocessing and spits (i.e., for training/development/test data) as the prior work @xcite to achieve a fair comparison. also, similar to @xcite , we use the pubmed pre-trained word embeddings @xcite for the models on cdr while randomly initialized word embeddings are employed for gda. these word embeddings are optimized during the training process of the models in this work. we implement the eogane model in this work by extending the code for the eog model that is provided in its original paper @xcite . as such, we inherit the values for the common hyper-parameters between eog and eogane from @xcite for the proposed model eogane (e.g., 0.002 for the learning rate of the adam optimizer, 2 and 3 for the batch sizes with the cdr and gda datasets respectively, 100 for the dimension of the node/edge representation vectors in the inference layers). the values for the specific hyper-parameters of eogane (i.e., tuned with the development data for each dataset) include: γ = 0.4 for the controlling constant in the node representation computation of the inference layers (for both cdr and gda), α @xmath0.5 and α @xmath1.1 for the trade-off parameters in the overall loss function l for the cdr dataset, and α @xmath2.4 and α @xmath3.6 for the gda dataset. comparing to the state of the art: this part compares the proposed model eogane with the state-of-the-art models for dre. ## conclusion we present a novel deep learning model for dre that explicitly computes the node representations for the document graphs in the graph-based edgeoriented models for dre. this enables the models to better capture the specific information of the nodes and facilitates the incorporation of two novel consistency constraints to improve the representation vectors. the experiments demonstrate the effectiveness of the proposed method for dre. in the future, we plan to apply the models in this work to the related tasks in information extraction.