Synchronous Dual Network with Cross-Type Attention for Joint Entity and Relation Extraction (2021)

Abstract: Joint entity and relation extraction is challenging due to the complex interaction of interaction between named entity recognition and relation extraction. Although most existing works tend to jointly train these two tasks through a shared network, they fail to fully utilize the interdependence between entity types and relation types. In this paper, we design a novel synchronous dual network (SDN) with cross-type attention via separately and interactively considering the entity types and relation types. On the one hand, SDN adopts two isomorphic bi-directional type-attention LSTM to encode the entity type enhanced representations and the relation type enhanced representations, respectively. On the other hand, SDN explicitly models the interdependence between entity types and relation types via cross-type attention mechanism. In addition, we also propose a new multi-task learning strategy via modeling the interaction of two types of information. Experiments on NYT and WebNLG datasets verify the effectiveness of the proposed model, achieving state-of-the-art performance.

## introduction joint entity and relation extraction is a fundamental and important task in information extraction, providing necessary information for knowledge base construction @xcite , question answering @xcite , and dialogue systems @xcite , etc. this task can be decomposed into two subtasks: named entity recognition (ner) and relation extraction (re), aiming respectively to detect text spans with specific types (entities) and semantic relations among those text spans (relations) from unstructured texts. early studies employ pipeline models @xcite @xcite , which first extract all entities of the sentence by an entity model, and then these extracted entities are used as the inputs of a relation model to identify semantic relations between entity pairs. however, pipeline models disregard the correlation between ner and re @xcite . in recent years, joint learning models have gained increasing attention. among them, multitask learning methods @xcite @xcite are popular and they utilize a shared network to learn common features, but making independent predictions for the two tasks. later, @xcite propose a new multi-task learning method to dynamically learn the interactions between the two tasks. @xcite propose a joint neural framework oneie to study the interaction of different feature categories by a set of global feature templates. other methods such as novel tagging @xcite @xcite and generative models @xcite @xcite adopt a unified model to directly extract relational triplets. although these methods show effectiveness for joint entity and relation extraction, they only apply a shared network or a unified model to capture the connection between ner and re without taking into account the interdependence of entity types and relation types. intuitively, the relation types of relational triplets are not only relevant to the textual context and entities, but also to entity types @xcite . conversely, the entity types of subject and object entities are also constrained by relation types in the triplets. for example, the instance in figure 1 contains three entities: "jackie r. brown" (per), "washington" (loc), and "united states of america" (loc). the relation type of ("jackie r. brown", "washington") and ("jackie r. brown", "united states of america") is birth_place, while the relation type of ("washington", "united states of america") is capital_of. when the subject entity type is per and the object entity type is loc, the relation type between them may be birth_place, but never capital_of. conversely, when the relation type is capital_of, the corresponding entity type does not include per. in addition, although @xcite adopt graph convolutional network to handle the joint type inference problem, they fail to discuss the fine-grained correlation between entity types and relation types. in this paper, we propose a synchronous dual network (sdn) with cross-type attention to separately and interactively capture the vector representations related to entity types and relation types. first, sdn adopts two isomorphic bi-directional type-attention lstm to learn two different feature representations, where one is the entity type enhanced representations and the other is the relation type enhanced representations. then the entity type information and the relation type information are introduced into the relation type enhanced representations and the entity type enhanced representations respectively to explicitly model the interaction between entity types and relation types via cross-type attention mechanism. these type-related representations are concatenated together for ner and re via a multi-task learning strategy. the above typeattention lstm is a general structure to select the preferred type distribution. the main idea is to inject all possible entity types simultaneously via multiple type-related cells based on the standard lstm. in this way, our model obtains the preferred type information via training auxiliary tasks. to summarize, the main contributions of this work are as follows 1 : tagging model. zheng et al. (2017) firstly convert the joint extraction task to a sequence labeling problem and propose a unified tagging scheme. later wei et al. (2020) and wang et al. (2020) propose different tagging schemes. yuan et al. (2020) propose a relation-attentive sequence labeling framework. yu et al. (2020) adopt a novel decomposition strategy, which first recognizes head entities and then extracts corresponding tail entities and relations. generative model. zeng et al. (2018) firstly propose a sequence-to-sequence model with copy mechanism to generate relational triplets but fail to generate multi-word entities. subsequently, zeng et al. (2019), nayak and tou ng (2020) and sui et al. (2020) adopt different encoder-decoder architectures to generate relational triplets. ye et al. (2021) propose contrastive triple extraction with a generative transformer. word embedding ℎ 𝑡 𝑐 lstm ⨁ ner & re entity type prediction 𝑘 1 (𝑡) 𝑣 1 (𝑡) 𝑘 𝑚 (𝑡) 𝑣 𝑚 (𝑡) ⋯ 𝛼 1 (𝑡) 𝛼 𝑚 (𝑡) 𝛼 𝑘 (𝑡) 𝑣 𝑘 (𝑡) 𝑘 ℎ 𝑡-1 ℎ 𝑡 ℎ 𝑡 𝑙 𝑥 𝑡 ℎ 𝑡 compared with these works, we propose a new multi-task learning strategy to explicitly model the interdependence between entity types and relation types within one relation to synchronize their information and make them mutually beneficial. ## type-attention lstm in this section, we will introduce the general framework of type attention lstm (ta-lstm). as shown in figure 2 (a), each ta-lstm unit is composed of a lstm unit and a type-attention unit. for each token w t , the lstm unit is used to obtain the contextual representation h c t , while the typeattention unit uses the scaled dot-product attention @xcite to obtain the type representation h l t by integrating m type-related cells for controlling type information flow. we adopt the standard lstm @xcite to encode the contextual representation for each token. at each time step t(@xmath0, ..., n]), the current hidden state h c t based on a memory cell c t is calculated as follows: type-attention mechanism. given x t and h t-1 , the key-value pair of the k-th (@xmath1, . . . , m]) type-related cell is computed as: the above operations are repeated for m typerelated cells. at the time step t, we finally acquire a set of key-value pairs we regard the contextual representation h c t as the query. the scaled dot-product attention first computes the dot products of the query with all corresponding keys, divide each by √ d e , and apply a softmax function to obtain the weight α (t) on the values. then the type representation h l t is computed as a weighted sum of the values: finally, the hidden state h @xmath2: in this section, we will describe how to build a synchronous dual network on top of the proposed ta-lstm and how to interact between entity types and relation types via cross-type attention mechanism for joint entity and relation extraction. as shown in figure 2 (b), synchronous dual network adopts two bi-directional ta-lstm to encode the vector representations with corresponding type information from the perspective of entity types and relation types by synchronous dual learning, respectively. then synchronous dual learning adopts the cross-type attention mechanism to model the interaction between entity types and relation types through introducing entity type information into relation type learning or introducing relation type information into entity type learning. finally, the vector representations with type information are concatenated together for joint learning. ## experiments in this section, we empirically verify the effectiveness of our proposed sdn on two public datasets. in addition, we also investigate how different components in the model impact the performance of joint entity and relation extraction with different settings. ## conclusion in this paper, we propose a synchronous dual network (sdn) with cross-type attention for joint entity and relation extraction. firstly, we use two isomorphic bi-directional type-attention lstm as encoders to learn the entity type enhanced representations and the relation type enhanced representations from two different perspectives. then the entity type information (or the relation type information) is introduced into the relation type enhanced representations (or the entity type enhanced representations) to explicitly model the interaction between entity types and relation types via crosstype attention mechanism. in addition, the proposed type-attention lstm is a general structure to obtain the preferred type distribution. experiments on two public datasets verify the effectiveness of the proposed model, achieving state-of-the-art performance.