Neural models have achieved remarkable success on relation extraction ( RE ) benchmarks . However , there is no clear understanding which type of information affects existing RE models to make decisions and how to further improve the performance of these models . To this end , we empirically study the effect of two main information sources in text : textual context and entity mentions ( names ) . We find that ( i ) while context is the main source to support the predictions , RE models also heavily rely on the information from entity mentions , most of which is type information , and ( ii ) existing datasets may leak shallow heuristics via entity mentions and thus contribute to the high performance on RE benchmarks . Based on the analyses , we propose an entity-masked contrastive pre-training framework for RE to gain a deeper understanding on both textual context and type information while avoiding rote memorization of entities or use of superficial cues in mentions . We carry out extensive experiments to support our views , and show that our framework can improve the effectiveness and robustness of neural models in different RE scenarios . All the code and datasets are released at https://github.com / thunlp/ RE-Context-or-Names . Relation extraction ( RE ) aims at extracting relational facts between entities from text , e.g. , extracting the fact ( SpaceX , founded by , Elon Musk ) from the sentence in Figure 1 . Utilizing the structured knowledge captured by RE , we can construct or complete knowledge graphs ( KGs ) , and eventually support downstream applications like question answering ( Bordes et al . , 2014 ) , dialog systems ( Madotto et al . , 2018 ) engines ( Xiong et al . , 2017 ) . With the recent advance of deep learning , neural relation extraction ( NRE ) models ( Socher et al . , 2012 ; Liu et al . , 2013 ; Baldini Soares et al . , 2019 ) have achieved the latest state-of-the-art results and some of them are even comparable with human performance on several public RE benchmarks . The success of NRE models on current RE benchmarks makes us wonder which type of information these models actually grasp to help them extract correct relations . The analysis of this problem may indicate the nature of these models and reveal their remaining problems to be further explored . Generally , in a typical RE setting , there are two main sources of information in text that might help RE models classify relations : textual context and entity mentions ( names ) . From human intuition , textual context should be the main source of information for RE . Researchers have reached a consensus that there exist interpretable patterns in textual context that express relational facts . For example , in Figure 1 , " ... be founded ... by ... " is a pattern for the relation founded by . The early RE systems ( Huffman , 1995 ; Califf and Mooney , 1997 ) formalize patterns into string templates and determine relations by matching these templates . The later neural models ( Socher et al . , 2012 ; Liu et al . , 2013 ) prefer to encode patterns into distributed representations and then predict relations via representation matching . Compared with rigid string templates , distributed representations used in neural models are more generalized and perform better . Besides , entity mentions also provide much information for relation classification . As shown in Figure 1 , we can acquire the types of entities from their mentions , which could help to filter out those impossible relations . Besides , if these entities can be linked to KGs , models can introduce external knowledge from KGs to help RE ( Zhang et al . , 2019 ; Peters et al . , 2019 ) . Moreover , for pre-trained language models , which are widely adopted for recent RE models , there may be knowledge about entities inherently stored in their parameters after pre-training ( Petroni et al . , 2019 ) . In this paper , we carry out extensive experiments to study to what extent RE models rely on the two information sources . We find out that : ( 1 ) Both context and entity mentions are crucial for RE . As shown in our experiments , while context is the main source to support classification , entity mentions also provide critical information , most of which is the type information of entities . ( 2 ) Existing RE benchmarks may leak shallow cues via entity mentions , which contribute to the high performance of existing models . Our experiments show that models still can achieve high performance only given entity mentions as input , suggesting that there exist biased statistical cues from entity mentions in these datasets . The above observations demonstrate how existing models work on RE datasets , and suggest a way to further improve RE models : we should enhance them via better understanding context and utilizing entity types , while preventing them from simply memorizing entities or exploiting biased cues in mentions . From these points , we investigate an entity-masked contrastive pre-training framework for RE . We use Wikidata to gather sentences that may express the same relations , and let the model learn which sentences are close and which are not in relational semantics by a contrastive objective . In this process , we randomly mask entity mentions to avoid being biased by them . We show its effectiveness across several settings and benchmarks , and suggest that better pre-training technique is a reliable direction towards better RE . In this paper , we thoroughly study how textual context and entity mentions affect RE models respectively . Experiments and case studies prove that ( i ) both context and entity mentions ( mainly as type information ) provide critical information for relation extraction , and ( ii ) existing RE datasets may leak superficial cues through entity mentions and models may not have the strong abilities to understand context as we expect . From these points , we propose an entity-masked contrastive pre-training framework for RE to better understand textual context and entity types , and experimental results prove the effectiveness of our method . In the future , we will continue to explore better RE pre-training techniques , especially with a focus on open relation extraction and relation discovery . These problems require models to encode good relational representation with limited or even zero annotations , and we believe that our pre-trained RE models will make a good impact in the area .

Challenge: Which textual information models are used for relation extraction tasks remain unknown hindering to improvement of their performance.
Approach: They perform an analysis and use the findings to propose an entity-masked contrastive pretraining to avoid models memorizing entities or using biased cues in mentions.
Outcome: They found that existing benchmarks leak superficial cues from entity mentions, and the proposed model can achieve robust performance without relying on these features.