Entity Relative Position Representation based Multi-head Selection for Joint Entity and Relation Extraction (2020)

Abstract: Joint entity and relation extraction has received increasing interests recently, due to the capability of utilizing the interactions between both steps. Among existing studies, the Multi-Head Selection (MHS) framework is efficient in extracting entities and relations simultaneously. However, the method is weak for its limited performance. In this paper, we propose several effective insights to address this problem. First, we propose an entity-specific Relative Position Representation (eRPR) to allow the model to fully leverage the distance information between entities and context tokens. Second, we introduce an auxiliary Global Relation Classification (GRC) to enhance the learning of local contextual features. Moreover, we improve the semantic representation by adopting a pre-trained language model BERT as the feature encoder. Finally, these new keypoints are closely integrated with the multi-head selection framework and optimized jointly. Extensive experiments on two benchmark datasets demonstrate that our approach overwhelmingly outperforms previous works in terms of all evaluation metrics, achieving significant improvements for relation F1 by +2.40% on CoNLL04 and +1.90% on ACE05, respectively.

## introduction the entity-relation extraction task aims to recognize the entity spans from a sentence and detect the relations holds between two entities. generally, it can be formed as extracting triplets (e 1 , r, e 2 ), which denotes that the relation r holds between the head entity e 1 and the tail entity e 2 , i.e., (john smith, live-in, atlanta). it plays a vital role in the information extraction area and has attracted increasing attention in recent years. traditional pipelined methods divide the task into two phases, named entity recognition (ner) and relation extraction (re) @xcite @xcite . as such methods neglect the underlying correlations between the two phases and suffer from the error propagation issue, recent works propose to extract entities and relations jointly. these joint models fall into two paradigms. the first paradigm can be denoted as (e 1 , e 2 ) → r, which first recognizes all entities in the sentence, then classifies the relation depend on each extracted entity pairs. however, these methods require enumerating all possible entity pairs and the relation classification may be affected by the redundant ones. while another paradigm is referred as @xmath0, e2), which detects head entities first and then predicts the corresponding relations and tail entities @xcite @xcite . comparing with the first paradigm, the second one can jointly identify entities and all the possible relations between them at once. a typical approach is the multi-head selection (mhs) framework @xcite . it first recognizes head entities using the bilstm-crf structure and then performs tail entity extraction and relation extraction in one pass based on multiclass classification. the advantage of the mhs framework is obvious -it is efficient to work with the scenario, that one entity can involve several relational triplets, making this solution suitable for large scale practical applications. in this paper, we focus on the second paradigm of the joint models, especially on the mhs framework. despite the efficiency of the mhs framework, it is weak for the limited performance comparing with other complex models. intuitively, the distance between entities and other context tokens provide impor-louis vuitton founded louis vuitton inc. in 1854 . ## related work in this section, we introduce the related studies for this work, entity and relation extraction as well as the positional representation. ## method in this section, we briefly present the details of the relative position representation based multi-head selection framework. the concept of multi-head means that any head entity may be relevant to multiple relations and tail entities @xcite . formally, denote e and r as the set of pre-defined entity types and relation categories, respectively. given an input sentence with n tokens @xmath1, s 2 , . . . , s n }, the entity-relation extraction task aims at extracting a set of named entities @xmath2, e 2 , . . . , e m } with specific types @xmath3, y 2 , . . . , y m }, and predict the relation r ij for each entity pair (e i , e j ), where y @xmath4. triplets such as bert encoder john smith headed the disease control center in atlanta . [cls] b-per i-per o o b-org i-org i-org o b-loc o o null null null null located-in null null null null headed the disease control atlanta in atlanta . [cls] null john work-for, live-in center, atlanta global relation classification erpr-based self-attention (x 2) encoder module crf module context fusion module multi-head selection module we take a sentence from conll04 dataset as an example. in this sentence, the golden relational triplets are: (john smith, live-in, atlanta), (john smith, work-for, disease control center) and (disease control center, located-in, atlanta). the null label denotes a case of no relation. as illustrated in fig. 2 , our framework consists of four modules as follows: the encoder module, the crf module, the context fusion module and the multi-head selection module. the token sequence is taken as the input of the framework and is fed into the bert encoder to capture contextual representations. the crf module is applied afterward to extract potential head entities (i.e., boundaries and types). then, the hidden states of bert and the entity information are feed into the context fusion module to encoder the entity position-based features. finally, a multi-head selection module is employed to simultaneously extract tuples of relation and tail entity for the input token (e.g., (work-for, center) and (live-in, atlanta) for the head entity simth). additionally, we present the strategy of global relation classification. we will elaborate on each of the modules in the following subsections. ## experiment in this section, we conduct extensive experiments to verify the effectiveness of our framework, and make detailed analyses to show its advantages. ## conclusion in this paper, we propose a relative position representation based multi-head selection framework for joint entity and relation extraction. different with the existing multi-head selection method, we introduce the relative position representation to capture the distance information of entities. we then propose a global relation classification to guide the learning of local features. additionaly, bert is incorperated in the framework for sematic representation. experimental results on conll04 and ace05 datasets show that our framework siginificantly outperforms all the baseline models for both entity and relation extraction.