ERICA : Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2021)

Abstract: Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks. However, conventional pre-training objectives do not explicitly model relational facts in text, which are crucial for textual understanding. To address this issue, we propose a novel contrastive learning framework ERICA to obtain a deep understanding of the entities and their relations in text. Specifically, we define two novel pre-training tasks to better understand entities and relations: (1) the entity discrimination task to distinguish which tail entity can be inferred by the given head entity and relation; (2) the relation discrimination task to distinguish whether two relations are close or not semantically, which involves complex relational reasoning. Experimental results demonstrate that ERICA can improve typical PLMs (BERT and RoBERTa) on several language understanding tasks, including relation extraction, entity typing and question answering, especially under low-resource settings.

## introduction pre-trained language models (plms) @xcite @xcite have shown superior performance on various natural language processing (nlp) tasks such as text classification @xcite , named entity recognition @xcite , and question answering @xcite . benefiting from designing various effective self-supervised learning objectives, such as masked language modeling @xcite , plms can effectively capture the syntax and semantics in text to generate informative language representations for downstream nlp tasks. figure 1 : an example for a document "culiacán", in which all entities are underlined. we show entities and their relations as a relational graph, and highlight the important entities and relations to find out "where is guadalajara". however, conventional pre-training objectives do not explicitly model relational facts, which frequently distribute in text and are crucial for understanding the whole text. to address this issue, some recent studies attempt to improve plms to better understand relations between entities @xcite . however, they mainly focus on within-sentence relations in isolation, ignoring the understanding of entities, and the interactions among multiple entities at document level, whose relation understanding involves complex reasoning patterns. according to the statistics on a human-annotated corpus sampled from wikipedia documents by @xcite , at least 40.7% relational facts require to be extracted from multiple sentences. specifically, we show an example in figure 1 , to understand that "guadalajara is located in mexico", we need to consider the following clues jointly: (i) "mexico" is the country of "culiacán" from sentence 1; (ii) "culiacán" is a rail junction lo-cated on "panamerican highway" from sentence 6; (iii) "panamerican highway" connects to "guadalajara" from sentence 6. from the example, we can see that there are two main challenges to capture the in-text relational facts: in this paper, we propose erica, a novel framework to improve plms' capability of entity and relation understanding via contrastive learning, aiming to better capture in-text relational facts by considering the interactions among entities and relations comprehensively. specifically, we define two novel pre-training tasks: (1) the entity discrimination task to distinguish which tail entity can be inferred by the given head entity and relation. it improves the understanding of each entity via considering its relations to other entities in text; we conduct experiments on a suite of language understanding tasks, including relation extraction, entity typing and question answering. the experimental results show that erica improves the performance of typical plms (bert and roberta) and outperforms baselines, especially under lowresource settings, which demonstrates that erica effectively improves plms' entity and relation understanding and captures the in-text relational facts. ## related work dai and @xcite and @xcite propose to pre-train universal language representations on unlabeled text, and perform task-specific fine-tuning. with the advance of computing power, plms such as openai gpt @xcite , bert @xcite and xlnet @xcite based on deep transformer @xcite architecture demonstrate their superiority in various downstream nlp tasks. since then, numerous plm extensions have been proposed to further explore the impacts of various model architectures @xcite , larger model size @xcite @xcite , more pre-training corpora @xcite , etc., to obtain better general language understanding ability. although achieving great success, these plms usually regard words as basic units in textual understanding, ignoring the informative entities and their relations, which are crucial for understanding the whole text. to improve the entity and relation understanding of plms, a typical line of work is knowledgeguided plm, which incorporates external knowledge such as knowledge graphs (kgs) into plms to enhance the entity and relation understanding. some enforce plms to memorize information about real-world entities and propose novel pretraining objectives @xcite @xcite . others modify the internal structures of plms to fuse both textual and kg's information @xcite @xcite . although knowledge-guided plms introduce extra factual knowledge in kgs, these methods ignore the intrinsic relational facts in text, making it hard to understand out-of-kg entities or knowledge in downstream tasks, let alone the errors and incompleteness of kgs. this verifies the necessity of teaching plms to understand relational facts from contexts. another line of work is to directly model entities or relations in text in pre-training stage to break the limitations of individual token representations. some focus on obtaining better span representations, including entity mentions, via span-based pre-training @xcite @xcite . others learn to extract relation-aware semantics from text by comparing the sentences that share the same entity pair or distantly supervised relation in kgs @xcite . however, these methods only consider either individual entities or within-sentence relations, which limits the performance in dealing with multiple entities and relations at document level. in contrast, our erica considers the interactions among multiple entities for an entity pair with its distantly supervised relation in text, the ed task requires the ground-truth tail entity to be closer to the head entity than other entities. and relations comprehensively, achieving a better understanding of in-text relational facts. ## methodology in this section, we introduce the details of erica. we first describe the notations and how to represent entities and relations in documents. then we detail the two novel pre-training tasks: entity discrimination (ed) task and relation discrimination (rd) task, followed by the overall training objective. ## experiments in this section, we first describe how we construct the distantly supervised dataset and pre-training details for erica. then we introduce the experiments we conduct on several language understanding tasks, including relation extraction (re), entity typing (et) and question answering (qa). we test erica on two typical plms, including bert and roberta (denoted as erica bert and erica roberta ) 6 . we leave the training details for downstream tasks and experiments on glue benchmark @xcite in the appendix. ## analysis in this section, we first conduct a suite of ablation studies to explore how l ed and l rd contribute to erica. then we give a thorough analysis on how pre-training data's domain / size and methods for entity encoding impact the performance. lastly, we visualize the entity and relation embeddings learned by erica. ## conclusions in this paper, we present erica, a general framework for plms to improve entity and relation understanding via contrastive learning. we demonstrate the effectiveness of our method on several language understanding tasks, including relation extraction, entity typing and question answering. the experimental results show that erica outperforms all baselines, especially under low-resource settings, which means erica helps plms better capture the in-text relational facts and synthesize information about entities and their relations. models for three epochs, other hyper-parameters are kept the same as ernie.