CIL : Contrastive Instance Learning Framework for Distantly Supervised Relation Extraction (2021)

Abstract: The journey of reducing noise from distant supervision (DS) generated training data has been started since the DS was first introduced into the relation extraction (RE) task. For the past decade, researchers apply the multi-instance learning (MIL) framework to find the most reliable feature from a bag of sentences. Although the pattern of MIL bags can greatly reduce DS noise, it fails to represent many other useful sentence features in the datasets. In many cases, these sentence features can only be acquired by extra sentence-level human annotation with heavy costs. Therefore, the performance of distantly supervised RE models is bounded. In this paper, we go beyond typical MIL framework and propose a novel contrastive instance learning (CIL) framework. Specifically, we regard the initial MIL as the relational triple encoder and constraint positive pairs against negative pairs for each instance. Experiments demonstrate the effectiveness of our proposed framework, with significant improvements over the previous methods on NYT10, GDS and KBP.

## introduction relation extraction (re) aims at predicting the relation between entities based on their context. several studies have been carried out to handle this crucial and complicated task over decades as the extracted information can serve as a significant role for many downstream tasks. since the amount of training data generally limits traditional supervised re systems, current re systems usually resort to distant supervision (ds) to fetch abundant training data by aligning knowledge bases (kbs) and texts. however, such a heuristic way inevitably introduces some noise to the generated data. training a robust and unbiased re system under ds data with awareness of the existing ds noise, @xcite introduces the multi-instance learning (mil) framework to dsre by dividing training instances into several bags and using bags as new data units. regarding the strategy for selecting instances inside the bag, the soft attention mechanism proposed by @xcite is widely used for its better performance than the hard selection method. the ability to form accurate representations from noisy data makes the mil framework soon become a paradigm of following-up works. however, we argue that the mil framework is effective to alleviate data noise for dsre, but is not data-efficient indeed: as figure 1 shows: the attention mechanism in the mil can help select relatively informative instances (e.g.h 1 , h 2 ) inside the bag, but may ignore the potential information of other abundant instances (e.g.h m ). in other words, no matter how many instances a bag contains, only the formed bag-level representation can be used for further training in the mil, which is quite inefficient. thus, our focus is on how to make the initial mil framework efficient enough to leverage all instances while maintaining the ability to obtain an accurate model under ds data noise? here, we propose a contrastive-based method to help the mil framework learn efficiently. in detail, we regard the initial mil framework as the bag encoder, which provides relatively accurate representations for different relational triples. then we develop contrastive instance learning (cil) to utilize each instance in an unsupervised manner: in short, the goal of our cil is that the instances sharing the same relational triples (i.e.positive pairs) ought to be close in the semantic space, while the representations of instances with different relational triples (i.e.negative pairs) should be far away. experiments on three public dsre benchmarks -nyt10 @xcite , gds @xcite and kbp @xcite demonstrate the effectiveness of our proposed framework cil, with consistent improvements over several baseline models and far exceed the state-of-the-art (sota) systems. furthermore, the ablation study shows the rationality of our proposed positive/negative pair construction strategy. accordingly, the major contributions of this paper are summarized as follows: ## methodology in this paper, we argue that the mil framework is effective to denoise but is not efficient enough, as the initial mil framework only leverages the formed bag-level representations to train models and sacrifices the potential information of numerous instances inside bags. here, we go beyond the typical mil framework and develop a novel contrastive instance learning framework to solve the above issue, which can prompt dsre models to utilize each instance. a formal description of our proposed cil framework is illustrated as follows. ## experiments our experiments are designed to verify the effectiveness of our proposed framework cil. ## related work our work is related to dsre, pre-trained language models, and recent contrastive learning methods. dsre traditional supervised re systems heavily rely on the large-scale human-annotated dataset, which is quite expensive and time-consuming. distant supervision is then introduced to the re field, and it aligns training corpus with kb facts to generate data automatically. however, such a heuristic process results in data noise and causes classical supervised re models hard to train. to solve this issue, lin et al. (2016) applied the multi-instance learning framework with selective attention mechanism over all instances, and it helps re models learn under ds data noise. following the mil framework, recent works improve dsre models from many different aspects: (1) yuan et al. (2019) adopted relation-aware attention and constructed super bags to alleviate the problem of bag label error. (2) ye et al. (2019) analyzed the label distribution of dataset and found the shifted label problem that significantly influences the performance of dsre models. (3) vashishth et al. (2018) employed graph convolution networks @xcite to encode syntactic information from the text and improves dsre models with additional side information from kbs. (4) @xcite extended the gpt to the dsre, and finetuned it to achieve sota model performance. pre-trained lm recently pre-trained language models achieved great success in the nlp field. @xcite proposed a self-attention based architecture -transformer, and it soon becomes the backbone of many following lms. by pre-training on a large-scale corpus, bert @xcite obtains the ability to capture a notable amount of "common-sense" knowledge and gains significant improvements on many tasks following the fine-tune scheme. at the same time, gpt @xcite , xl-net @xcite and gpt2 @xcite are also well-known pre-trained representatives with excellent transfer learning ability. moreover, some works @xcite found that considerably increasing the size of lm results in even better generalization to downstream tasks. contrastive learning as a popular unsupervised method, contrastive learning aims to learn representations by contrasting positive pairs against negative pairs @xcite @xcite . @xcite proposed to use the non-parametric instance-level discrimination to leverage more information in the data samples. our approach, however, achieves the goal of data-efficiency in a more complicated mil setting: instead of contrasting the instance-level information during training, we find that instance-bag negative pair is the most effective method, which constitutes one of our main contributions. in the nlp field, @xcite proposed to use contrastive learning for image caption, and @xcite trained a discriminative model for language representation learning. recent literature @xcite has also attempted to relate the contrastive pre-training scheme to classical supervised re task. different from our work, @xcite aims to utilize abundant ds data and help classical supervised re models learn a better relation representation, while our cil focuses on learning an effective and efficient dsre model under ds data noise. ## conclusion in this work, we discuss the long-standing dsre framework (i.e.mil) and argue the mil is not efficient enough, as it aims to form accurate bag-level representations but sacrifices the potential informa-tion of abundant instances inside mil bags. thus, we propose a contrastive instance learning method cil to boost the mil model performances. experiments have shown the effectiveness of our cil with stable and significant improvements over several baseline models, including current sota systems.