Dependency Position Encoding for Relation Extraction (2022)

Abstract: Leveraging the dependency tree of the input sentence is able to improve the model performance for relation extraction. A challenging issue is how to remove confusions from the tree. Efforts have been made to utilize the dependency connections between words to selectively emphasize target-relevant information. However, these approaches are limited in focusing on exploiting dependency types. In this paper, we propose dependency position encoding (DPE), an efficient way of incorporating both dependency connections and dependency types into the self-attention mechanism to distinguish the importance of different word dependencies for the task. In contrast to previous studies that process input sentence and dependency information in separate streams, DPE can be seamlessly incorporated into the Transformer and makes it possible to use an one-stream scheme to extract relations between entity pairs. Extensive experiments show that models with our DPE significantly outperform the previous methods on SemEval 2010 Task 8, KBP37, and TACRED.

## introduction relation extraction (re) has been a long standing goal in natural language processing (nlp) and plays a crucial role in supporting many downstream task @xcite @xcite . nowadays, methods with powerful encoders (e.g. transformer) have achieved promising success in re @xcite @xcite @xcite due to their effectiveness in capturing contextual information. in addition, previous studies @xcite @xcite try to utilize the extra syntactic knowledge (e.g. word dependency) to further improve the ability to encode relations * corresponding author. figure 1 : we introduce an one-stream scheme for relation extraction. by modeling the dependency information as a labeled and fully-connected graph, we extend self-attention to consider words dependencies and seamlessly incorporate them into transformer-based encoder. in this paper, we present dependency position encoding (dpe), an efficient way of incorporating dependency information into the self-attention mechanism of the transformer to distinguish the importance of different word dependencies for relation extraction. specially, we first encode dependency information obtained from an off-the-shelf dependency parser and map it into embeddings, then assign different weights to different labeled dependency connections between any two words through attention calculation. different from previous studies that process input sentence and dependency tree in separate streams, our dpe can be seamlessly incorporated into the encoder and makes it possible to introduce an one-stream scheme for relation extraction. we conduct experiments on three english benchmark datasets (i.e., semeval 2010 task 8, kbp37 and tacred) where the results demonstrate the effectiveness of our method. ## background and motivation bert @xcite is a pre-trained language model which has transformer-based model architecture and is widely used as sentence encoder in re methods @xcite @xcite . the selfattention mechanism in transformer does not explicitly model relative or absolute position information. to this end, transformer @xcite adopts explicit absolute sinusoidal positional encodings added into input embeddings. absolution positions can be more naturally encoded as relative positional encodings (rpe) as follows: inspired by rpe, we argue that dependency types associated with the dependency connections among words also can be regarded as dependency graphs, which contain highly useful information for re. we propose dependency positional encoding (dpe) and use it as an extension of self-attention to consider the dependency information between words. different from previous studies @xcite that treat the dependency connections among words equally, dpe is able to distinguish the importance of different labeled connections through self-attention calculation. ## methodology following the previous studies, we perform re as a classification task. specifically, given an input sentence @xmath0, ..., x n ) with e1 and e2 denoting two entities in it, we predict the relation r between entities as fowllows: ## conclusion in this paper, we propose dependency position encoding (dpe) and use it as an extension of selfattention to consider dependency information. different from previous studies, dpe is able to seamlessly incorporated into the transformer and makes it possible to use an one-stream scheme to extract illuminating word dependencies for relation extraction. experimental results on several public datasets illustrate the effectiveness of our approach.