Relation Extraction with Weighted Contrastive Pre-training on Distant Supervision (2023)

Abstract: Contrastive pre-training on distant supervision has shown remarkable effectiveness in improving supervised relation extraction tasks. However, the existing methods ignore the intrinsic noise of distant supervision during the pre-training stage. In this paper, we propose a weighted contrastive learning method by leveraging the supervised data to estimate the reliability of pre-training instances and explicitly reduce the effect of noise. Experimental results on three supervised datasets demonstrate the advantages of our proposed weighted contrastive learning approach compared to two state-of-the-art non-weighted baselines. Our code and models are available at: https://github.com/YukinoWan/WCL .

## introduction relation extraction (re) is the task of identifying the relationship between entities mentioned in the text, which can benefit many downstream tasks such as question answering and knowledge base population. since most of the existing re models @xcite @xcite @xcite ) are trained on the labeled data, the amount of training data limits the performance of supervised re systems. to tackle this problem, recent work leverage semi-supervised distant supervision (ds) @xcite @xcite approach to generate abundant training data by aligning knowledge bases (kbs) and raw corpora. however, distantly supervised relation extraction (dsre) inevitably suffers from wrong labeling noise. introducing a robust framework that utilizes both the abundant but noisy data from ds and the scarce but accurate data from human annotations becomes a new research line to improve re systems. recent works @xcite @xcite two-stage re framework that they first design a re-oriented task to pre-train bert on ds data and then fine-tune on human-annotated (ha) datasets. @xcite use wikipedia articles as the corpus and wikidata as the kb in the pre-training stage to construct the ds data, and they introduce a contrastive learning-based method to pre-train bert on the generated ds data. given an anchor instance with a specific relation in the ds data, their contrastive learning method randomly selects one positive sample holding the same relation and maximizes the similarity between the anchor and positive sample. meanwhile, their method randomly selects multiple negative samples holding different relations from the anchor and minimizes the similarity between the anchor and negative samples. the results show that their re-oriented pre-training can effectively improve the final performance of the re task on various target datasets. however, in their pre-training stage, they ignore the intrinsic wrong labeling noise in the generated ds data. since their method relies on the dslabeled relation types to sample positive and negative instances, the noisy labeling problem leads to unreliable samples in figure 1 , potentially limiting the pre-training stage's effectiveness. to better utilize ds data, we propose a novel weighted contrastive learning framework to both use the abundant ds data and tackle the inevitable ds noise. first, we train a relation classifier on the ha dataset and leverage the classifier to predict the relation type of instances in the ds data. then for each ds instance, based on the output of the classifier, we can compute the confidence score to measure the reliability of its labeled relation type. finally, we introduce weights based on computed confidence scores into the contrastive learning loss to focus more on reliable instances while less on noisy ones. besides, distant supervision relies on the existing kbs to align raw corpora. to alleviate the need for kbs, we propose a new strategy to extract a triplet set from the ha dataset for generating ds data. we also include a kb-derived ds dataset in our experiments to show that our proposal can still work well for regular ds. in conclusion, we propose a weighted contrastive pre-training approach for supervised relation extraction and introduce its details in section 2. then we perform the experiments on three datasets to compare our proposed method with existing baselines in section 3. ## conclusions we introduce a weighted contrastive pre-training method by leveraging the ha dataset to estimate the reliability of instances in the abundant ds data. to alleviate the need for kbs, we also propose to construct ds data based on the triplets derived from the ha dataset for pre-training. experimental results demonstrate that our proposed method outperforms sota work on target ha datasets.