Current state-of-the-art systems for joint entity relation extraction ( Luan et al . , 2019 ; Wadden et al . , 2019 ) usually adopt the multi-task learning framework . However , annotations for these additional tasks such as coreference resolution and event extraction are always equally hard ( or even harder ) to obtain . In this work , we propose a pre-training method ENPAR to improve the joint extraction performance . EN-PAR requires only the additional entity annotations that are much easier to collect . Unlike most existing works that only consider incorporating entity information into the sentence encoder , we further utilize the entity pair information . Specifically , we devise four novel objectives , i.e. , masked entity typing , masked entity prediction , adversarial context discrimination , and permutation prediction , to pretrain an entity encoder and an entity pair encoder . Comprehensive experiments show that the proposed pre-training method achieves significant improvement over BERT on ACE05 , SciERC , and NYT , and outperforms current state-of-the-art on ACE05 . Joint extraction of entities and relations is a fundamental task in information extraction , it aims to extract entities and relations with a unified model . Current approaches ( Luan et al . , 2019 ; Wadden et al . , 2019 ) usually adopt the multi-task learning framework that optimizes many objectives simultaneously , including entity recognition , relation extraction , coreference resolution , and event extraction . However , as large-scale manually labeled data required by these methods is unavailable in many domains , their applicability is severely restricted . Therefore , we expect to catch or even surpass the multi-task based joint models with less annotation cost . Compared with the annotations of coreference resolution and event extraction , entity annotations can be easily obtained through automatic NER annotation tools ( e.g. , spaCy1 ) . In this paper , we focus on improving the model 's performance with just extra entity annotations . Although pre-trained models , like BERT , have shown impressive performance in many downstream tasks , they have mainly two limitations when applied in the joint entity relation extraction task . One is that currently pre-trained objectives are insufficient for this task . Specifically , these commonly used universal pre-trained model ( e.g. , BERT ) do not consider the entity-related knowledge that is crucial for better extracting entities and relations . The other is that these models only provide pre-trained representations for tokens and sentences , but not entities and entity pairs . To obtain the representations for entities and entity pairs , additional parameters that are not pre-trained are introduced in the fine-tuning stage , which may futher impair the joint extraction performance . To address the first limitation , recent several works try to incorporate entity-related information into pre-training objectives . Zhang et al . ( 2019 ) fuses heterogeneous information from both texts and knowledge graphs and proposes a denoising entity auto-encoder objective based on BERT . Sun et al . ( 2019c ) presents two knowledge masking strategies in the pre-training stage ( entity-level masking and phrase-level masking ) . Both of them utilize extra entity annotations ( i.e. , entities in knowledge graphs and automatic entity annotations , respectively ) . In this paper , we follow this line of works and build a large-scale entity annotated corpus using the spaCy NER tool . For the second limitation , we propose ENPAR , a pre-training method customized for entity relation extraction . ENPAR consists of an underlying sentence encoder , an entity encoder , and an entity pair encoder . Compared with BERT ( Figure 1 ( a ) ) , the proposed entity encoder and entity pair encoder directly provide representations of entities and entity pairs . To train the three encoders , we devise four novel pre-training tasks : masked entity typing , masked entity prediction , adversarial context discrimination and permutation prediction ( Figure 1(b ) ) . In the first two tasks , we randomly mask some entity words and then predict the masked tokens and the entity type . These two tasks are natural extensions of the masked language model . To learn a better entity pair encoder , we draw inspirations from the denoising auto-encoder ( Zhang et al . , 2019 ) and propose the last two tasks . Specifically , when the entity pair or its context in a sentence are perturbed , we hope that the entity pair encoder is capable of tracking such changes . We employ the parameter-sharing method for these four tasks and train these objectives jointly . To sum up , our main contributions are as follows2 : • We introduce an entity encoder and an entity pair encoder to incorporate not only the entity information but also the entity pair information , which were ignored in current universal pre-trained models . • We propose four novel pre-training tasks that help to learn the proposed encoders . These tasks only require additional entity annotations ( with commonly used entity types ) , which can be automatically generated by public annotation tools , such as spaCy NER . • We conduct comprehensive experiments and demonstrate that the proposed method achieves significant improvement on ACE05 and NYT dataset and is comparable with the state-of-the-art on the SciERC dataset . We propose ENPAR , a pre-training method customized for entity relation extraction only with additional entity annotations . Instead of only pretraining sentence encoder in universal pre-trained models , we also pre-train an entity encoder and an entity pair encoder . Then the proposed four objec-tives can incorporate entity and entity-pair knowledge into the pre-trained encoders to enhance the encoders ' representations . Experiments on three datasets demonstrate that ENPAR achieves comparable or even superior performances compared with multi-task based joint models .

Challenge: Current joint entity relation extraction models follow a multitask learning setup however datasets with multiple types of annotation are not available for many domains.
Approach: They propose to pre-train a language model entity relation extraction with four newly introduced objective functions which utilize automatically obtained annotations by NER models.
Outcome: The models pre-trained by the proposed method significantly outperform BERT and current state-of-the-art models on three entity relation extraction datasets.