L earning from C ontext or N ames? A n E mpirical S tudy on N eural R elation E xtraction (2020)

Abstract: Neural models have achieved remarkable success on relation extraction (RE) benchmarks. However, there is no clear understanding what information in text affects existing RE models to make decisions and how to further improve the performance of these models. To this end, we empirically study the effect of two main information sources in text: textual context and entity mentions (names). We find that (i) while context is the main source to support the predictions, RE models also heavily rely on the information from entity mentions, most of which is type information, and (ii) existing datasets may leak shallow heuristics via entity mentions and thus contribute to the high performance on RE benchmarks. Based on the analyses, we propose an entity-masked contrastive pre-training framework for RE to gain a deeper understanding on both textual context and type information while avoiding rote memorization of entities or use of superficial cues in mentions. We carry out extensive experiments to support our views, and show that our framework can improve the effectiveness and robustness of neural models in different RE scenarios. All the code and datasets are released at https://github.com/thunlp/RE-Context-or-Names .

## introduction relation extraction (re) aims at extracting relational facts between entities from text, e.g., extracting the fact (spacex, founded by, elon musk) from the sentence in figure 1 . utilizing the structured knowledge captured by re, we can construct or complete knowledge graphs (kgs), and eventually support downstream applications like question answering @xcite , dialog systems @xcite and search spacex was founded in 2002 by elon musk . founded by type: organization id: q193701 other info: country: us product: falcon … type: person id: q317521 other info: citizenship: us occupation: entrepreneur … engines @xcite . with the recent advance of deep learning, neural relation extraction (nre) models @xcite @xcite have achieved the latest state-of-the-art results and some of them are even comparable with human performance on several public re benchmarks. the success of nre models on current re benchmarks makes us wonder which type of information these models actually grasp to help them extract correct relations. the analysis of this problem may indicate the nature of these models and reveal their remaining problems to be further explored. generally, in a typical re setting, there are two main sources of information in text that might help re models classify relations: textual context and entity mentions @xcite . from human intuition, textual context should be the main source of information for re. researchers have reached a consensus that there exist interpretable patterns in textual context that express relational facts. for example, in figure 1 , "... be founded ... by ..." is a pattern for the relation founded by. the early re systems @xcite formalize patterns into string templates and determine relations by matching these templates. the later neural models @xcite prefer to encode patterns into distributed representations and then predict relations via representation matching. compared with rigid string templates, distributed representations used in neural models are more generalized and perform better. besides, entity mentions also provide much information for relation classification. as shown in figure 1 , we can acquire the types of entities from their mentions, which could help to filter out those impossible relations. besides, if these entities can be linked to kgs, models can introduce external knowledge from kgs to help re @xcite . moreover, for pre-trained language models, which are widely adopted for recent re models, there may be knowledge about entities inherently stored in their parameters after pre-training @xcite . in this paper, we carry out extensive experiments to study to what extent re models rely on the two information sources. we find out that: the above observations demonstrate how existing models work on re datasets, and suggest a way to further improve re models: we should enhance them via better understanding context and utilizing entity types, while preventing them from simply memorizing entities or exploiting biased cues in mentions. from these points, we investigate an entity-masked contrastive pre-training framework for re. we use wikidata to gather sentences that may express the same relations, and let the model learn which sentences are close and which are not in relational semantics by a contrastive objective. in this process, we randomly mask entity mentions to avoid being biased by them. we show its effectiveness across several settings and benchmarks, and suggest that better pre-training technique is a reliable direction towards better re. ## pilot experiment and analysis to study which type of information affects existing neural re models to make decisions, we first introduce some preliminaries of re models and settings and then conduct pilot experiments as well as empirical analyses in this section. ## contrastive pre-training for re from the observations in section 2, we know that both context and entity type information is beneficial for re models. however, in some cases re models cannot well understand the relational patterns in context and rely on the shallow cues of entity mentions for classification. in order to enhance the ability to grasp entity types and extract relational facts from context, we propose the entity-masked contrastive pre-training framework for re. we start with the motivation and process of relational contrastive example generation, and then go through the pre-training objective details. ## experiment in this section, we explore the effectiveness of our relational contrastive pre-training across two typical re tasks and several re datasets. ## conclusion in this paper, we thoroughly study how textual context and entity mentions affect re models respectively. experiments and case studies prove that (i) both context and entity mentions (mainly as type information) provide critical information for relation extraction, and (ii) existing re datasets may leak superficial cues through entity mentions and models may not have the strong abilities to understand context as we expect. from these points, we propose an entity-masked contrastive pre-training framework for re to better understand textual context and entity types, and experimental results prove the effectiveness of our method. in the future, we will continue to explore better re pre-training techniques, especially with a focus on open relation extraction and relation discovery. these problems require models to encode good relational representation with limited or even zero annotations, and we believe that our pre-trained re models will make a good impact in the area.