Pre-training Entity Relation Encoder with Intra-span and Inter-span Information (2020)

Abstract: In this paper, we integrate span-related information into pre-trained encoder for entity relation extraction task. Instead of using general-purpose sentence encoder (e.g., existing universal pre-trained models), we introduce a span encoder and a span pair encoder to the pre-training network, which makes it easier to import intra-span and inter-span information into the pre-trained model. To learn the encoders, we devise three customized pre-training objectives from different perspectives, which target on tokens, spans, and span pairs. In particular, a span encoder is trained to recover a random shuffling of tokens in a span, and a span pair encoder is trained to predict positive pairs that are from the same sentences and negative pairs that are from different sentences using contrastive loss. Experimental results show that the proposed pre-training method outperforms distantly supervised pre-training, and achieves promising performance on two entity relation extraction benchmark datasets (ACE05, SciERC).

## introduction extraction of entities and relations from free texts is an important task in nlp. its goal is to recognize text spans with specific types (entities) and semantic relations among those entities (relations). current state-of-the-art systems usually employ the supervised joint learning algorithm @xcite @xcite , which can alleviate error propagation caused by the pipeline method. in this paper, we focus on joint entity relation extraction. recently, pre-trained models @xcite have substantially advanced a variety of nlp tasks, including entity relation extraction @xcite . @xcite adopt bert as a sentence encoder and build a multi-task framework for information extraction. however, universal pre-trained models are usually trained without explicitly handling text spans and relation among text span pairs. for example, the objectives of bert are masked language model and next sentence prediction, which are defined at the token level and sentence level, respectively. it rarely considers incorporating spanrelated knowledge, which can provide rich information for better extracting entities and relations ( the traditional way to introduce more entity relation related information is through distant supervision, which aligns triples in knowledge bases and free texts. however, the distantly supervised dataset contains lots of noise samples, which may have a negative impact on other datasets as prior works @xcite . besides, the distantly supervised dataset's annotated labels are usually inconsistent with that of the target dataset. as expected, in the preliminary experiment, we observe that the performance of the model directly pre-trained with annotated data provided by distantly supervised dataset (such as nyt) is not improved or even gets worse when it is fine-tuned on other entity relation dataset (such as ace05). in addition, there are several existing works for incorporating entity information into pre-training objectives @xcite . however, these methods rely on entity annotations, which brings additional cost. in this work, we focus on the unsupervised pretraining objectives. we present a novel pre-training network architecture customized for entity relation extraction. in addition to the default sentence encoder in existing pre-trained models (e.g., the transformer encoder of bert), we also pre-train a span encoder and a span pair encoder. to learn the two encoders, we propose three pre-training objectives corresponding to three levels: token boundary objective (token level), span permutation objective (span level), and contrastive span pair objective (span pair level). token boundary objective can help to enhance the representation of the first subtoken of each token. span encoder is trained by recovering the correct order of span tokens from its random shufflings. span pair encoder is trained by the contrastive loss. specifically, the predictions are made discriminatively with a sampledsoftmax that contrasts positive pairs against negative pairs. positive pairs are from the same sentences, while negative pairs are from different sentences. these three objectives share parameters and will be trained jointly. a closely related work to span level pre-training objective is spanbert @xcite , which adopts the span boundary objective to incorporate the span information. different from @xcite , we introduce not only a new objective at the span level but also a new objective at the span pair level ( ## background of contrastive learning infonce contrastive learning is a framework that builds representations by learning to encode what makes two things similar or dissimilar foot_1 . recently, @xcite regrad contrastive learning as a dictionary look-up task. an effective contrastive loss function, called @xcite , is as follows. formally, for any data point x, to learn a query encoder f q and a key encoder f k (the two encoders can be different, partially shared, or identical, we adopt two identical encoders), in-fonce is to minimize the following loss function foot_2 ## approach given an input sentence @xmath0, . . . , x |x| and a set of spans s (randomly sampling) in x, the target of our pre-training model is to obtain a contextualized vector representation for each span @xmath1, and a contextualized vector representation for each span pair (s 1 , s 2 ). as shown in figure 2 , the pre-training task optimizes a shared transformer @xcite network, a span level cnn and attention parameters with respect to a token boundary objective, a span permutation objective, and a contrastive span pair objective. different from universal pretrained language models @xcite , the proposed network incorporates rich intra-span and inter-span information foot_3 . once our network is pre-trained, we can fine-tune it for entity relation extraction task. ## experiments we conduct experiments on two benchmark entity relation extraction datasets: ace05 and scierc. for the distantly supervised dataset, we choose the nyt dataset. ## related work research on entity relation extraction has been extensively investigated. early pipeline methods suffer the error propagation problem @xcite . joint model can make better use of the complementarity between the entity model and the relation model to alleviate error propagation. a simple method is joint learning through sharing parameters, which means the entity model and the relation model can share some input vectors or sentence encoder. the typical works include tree lstm-based model over dependency tree @xcite and attention-based model without dependency tree @xcite . however, this kind of method does not perform joint decoding, and it can not fully exploit the interaction between output entities and relations. to mitigate the above question, many joint decoding algorithms @xcite @xcite are applied into this task, such as ilp-based joint decoding algorithms @xcite , joint sequence labelling tag set @xcite , structured perceptron @xcite , joint mrt @xcite , joint relational triplets extracting @xcite , and transition system @xcite . besides, @xcite perform the joint type inference with gcn on an entity-relation bipartite graph. especially, our joint model for entity relation extraction is derived from @xcite without gcn. in this work, we mainly investigate whether pre-training can help entity relation extraction task. for simplicity, our joint model does not perform joint decoding (only sharing parameters). in addition, transfer learning @xcite , multi-task learning @xcite @xcite , and reinforcement learning @xcite were also studied. pre-trained models have achieved impressive performance on a wide range of downstream tasks in nlp. different training objectives have been used for different pre-trained models. for example, elmo @xcite , @xcite and unilm @xcite learn different token level objectives and sentence level objectives. despite their great success, they ignore the incorporation of span-related information, which is vital for the entity relation extraction task. @xcite demonstrate that informative entities in kgs can enhance language representation with external knowledge. @xcite propose entity-level masking and phrase-level masking to enhance language representation. comparing with @xcite @xcite , the proposed objectives can not only integrate span information, but also span pair information. recently, unsupervised contrastive pretraining methods, moco @xcite , sim-clr @xcite , infoword @xcite and curl @xcite , have led to great empirical success in computer vision, reinforcement learning and nlp. constrastive learning learns representations by contrasting positive and negative samples. inspired by @xcite , we learn better span pair representations with contrastive methods, utilizing a large set of negatives without requiring large training batches and extending the moco framework with the proposed span pair objective. ## conclusion we propose a pre-training network architecture with three objectives, which can incorporate intraspan and inter-span information into pre-trained models. in comparison to universal pre-trained model, we introduce a span encoder and a span pair encoder. by designning three pre-training objectives, we can learn better pre-trained encoders customized for entity relation extraction task. experiments on two benchmark datasets demonstrate the effectiveness of the proposed pre-training method.