ENPAR :Enhancing Entity and Entity Pair Representations for Joint Entity Relation Extraction (2021)

Abstract: Current state-of-the-art systems for joint entity relation extraction (Luan et al., 2019; Wad-den et al., 2019) usually adopt the multi-task learning framework. However, annotations for these additional tasks such as coreference resolution and event extraction are always equally hard (or even harder) to obtain. In this work, we propose a pre-training method ENPAR to improve the joint extraction performance. ENPAR requires only the additional entity annotations that are much easier to collect. Unlike most existing works that only consider incorporating entity information into the sentence encoder, we further utilize the entity pair information. Specifically, we devise four novel objectives,i.e., masked entity typing, masked entity prediction, adversarial context discrimination, and permutation prediction, to pre-train an entity encoder and an entity pair encoder. Comprehensive experiments show that the proposed pre-training method achieves significant improvement over BERT on ACE05, SciERC, and NYT, and outperforms current state-of-the-art on ACE05.

## introduction joint extraction of entities and relations is a fundamental task in information extraction, it aims to extract entities and relations with a unified model. current approaches @xcite usually adopt the multi-task learning framework that optimizes many objectives simultaneously, including entity recognition, relation extraction, coreference resolution, and event extraction. however, as large-scale manually labeled data required by these methods is unavailable in many domains, their applicability is severely restricted. therefore, we expect to catch or even surpass the multi-task based joint models with less annotation cost. compared with the annotations of coreference resolution and event extraction, entity annotations can be easily obtained through automatic ner annotation tools (e.g., spacy foot_0 ). in this paper, we focus on improving the model's performance with just extra entity annotations. although pre-trained models, like bert, have shown impressive performance in many downstream tasks, they have mainly two limitations when applied in the joint entity relation extraction task. one is that currently pre-trained objectives are insufficient for this task. specifically, these commonly used universal pre-trained model (e.g., bert) do not consider the entity-related knowledge that is crucial for better extracting entities and relations. the other is that these models only provide pre-trained representations for tokens and sentences, but not entities and entity pairs. to obtain the representations for entities and entity pairs, additional parameters that are not pre-trained are introduced in the fine-tuning stage, which may futher impair the joint extraction performance. to address the first limitation, recent several works try to incorporate entity-related information into pre-training objectives. @xcite fuses heterogeneous information from both texts and knowledge graphs and proposes a denoising entity auto-encoder objective based on bert. @xcite presents two knowledge masking strategies in the pre-training stage (entity-level masking and phrase-level masking). both of them utilize extra entity annotations (i.e., entities in knowledge graphs and automatic entity annotations, respectively). in this paper, we follow this line of works and build a large-scale entity annotated corpus using the spacy ner tool. for the second limitation, we propose enpar, a pre-training method customized for entity relation extraction. enpar consists of an underlying sentence encoder, an entity encoder, and an entity pair encoder. compared with bert (figure 1(a) ), the proposed entity encoder and entity pair encoder directly provide representations of entities and entity pairs. to train the three encoders, we devise four novel pre-training tasks: masked entity typing, masked entity prediction, adversarial context discrimination and permutation prediction (figure 1(b) ). in the first two tasks, we randomly mask some entity words and then predict the masked tokens and the entity type. these two tasks are natural extensions of the masked language model. to learn a better entity pair encoder, we draw inspirations from the denoising auto-encoder @xcite and propose the last two tasks. specifically, when the entity pair or its context in a sentence are perturbed, we hope that the entity pair encoder is capable of tracking such changes. we employ the parameter-sharing method for these four tasks and train these objectives jointly. to sum up, our main contributions are as follows foot_1 : ## approach given an input sentence @xmath0, . . . , x |s| and a set of entities e (automatically annotated) in s, enpar is to encode each entity @xmath1, e 2 ) into a contextual representation vector. as shown in figure 2 , enpar is composed of a shared transformer @xcite , an entity-level cnn followed by an mlp (multi-layer perceptron), a context-level cnn followed by an mlp, and the last mlp. in the pre-training stage, we optimize enpar with four objectives, namely, masked entity typing, masked entity prediction, adversarial context discrimination and permutation prediction. these pre-training objectives can integrate rich entityrelated information into the proposed network. after pre-training, we can easily fine-tune the pretrained network for entity relation extraction task. ## experiments we conduct experiments on three benchmark entity relation extraction datasets: ace05, scierc, and nyt. for space limitation, we will mainly discuss the results on ace05 and report basic results on the remaining two datasets. ## related work joint entity relation extraction is an important task that has been extensively studied. one simple method to achieve joint learning is through parameters sharing, which usually share some input embeddings or sentence encoders @xcite . to further explore the interactions between the outputs of the entity model and the relation model, many joint decoding algorithms were introduced into this joint task @xcite @xcite @xcite @xcite . besides, @xcite tackle this task under the framework of multi-turn qa. and @xcite conduct joint type inference via gcn on a bipartite graph composed of entities and relations. recently, transfer learning @xcite , multi-task learning @xcite @xcite were also applied in this task. in this work, we investigate the pre-trained model for entity relation extraction. for simplicity, we restrict the joint model of parameters sharing, which can be easily extended to jont decoding methods. pre-trained models @xcite @xcite have made many amazing breakthroughs on various nlp downstream tasks. pre-training paradigm first pretrain networks with some pre-training objectives on large-scale unlabeled text corpora, and then fine-tune the pre-trained networks on downstream tasks. these pre-training objectives determine the knowledge absorbed by the pre-trained models. for example, bert @xcite adopts masked language model and next sentence prediction to learn deep contextual representations and the relation between two sentences respectively. xlnet @xcite uses permutation language model to learn bidirectional representations with autoregressive model. unilm @xcite fuses three types of language model: unidirectional, bidirectional and seq2seq prediction. however, these models and objectives all ignore the entity-related information, which is crucial for entity relation extraction task. recently, there are several public works that explore how to properly integrate entity inforamtion into pre-trained models. specifically, @xcite achieve enhanced language representation by injecting informative entities in kgs into pre-training models. and @xcite propose two higherlevel masking strategies: entity-level masking and phrase-level masking. in this work, we not only integrate entity information, but also extend to entity pair information and learn more powerful representations for entities and entity pairs. ## conclusion we propose enpar, a pre-training method customized for entity relation extraction only with additional entity annotations. instead of only pretraining sentence encoder in universal pre-trained models, we also pre-train an entity encoder and an entity pair encoder. then the proposed four objec-tives can incorporate entity and entity-pair knowledge into the pre-trained encoders to enhance the encoders' representations. experiments on three datasets demonstrate that enpar achieves comparable or even superior performances compared with multi-task based joint models.