Continual Contrastive Finetuning Improves Low-Resource Relation Extraction (2023)

Abstract: Relation extraction (RE), which has relied on structurally annotated corpora for model training, has been particularly challenging in low-resource scenarios and domains. Recent literature has tackled low-resource RE by self-supervised learning, where the solution involves pretraining the entity pair embedding by RE-based objective and finetuning on labeled data by classification-based objective. However, a critical challenge to this approach is the gap in objectives, which prevents the RE model from fully utilizing the knowledge in pretrained representations. In this paper, we aim at bridging the gap and propose to pretrain and finetune the RE model using consistent objectives of contrastive learning. Since in this kind of representation learning paradigm, one relation may easily form multiple clusters in the representation space, we further propose a multi-center contrastive loss that allows one relation to form multiple clusters to better align with pretraining. Experiments on two document-level RE datasets, BioRED and Re-DocRED, demonstrate the effectiveness of our method. Particularly, when using 1% end-task training data, our method outperforms PLM-based RE classifier by 10.5% and 6.1% on the two datasets, respectively.

## introduction relation extraction (re) is a fundamental task in nlp. it aims to identify the relations among entities in a given text from a predefined set of relations. while much effort has been devoted to re in supervised settings @xcite @xcite , re is extremely challenging in high-stakes domains such as biology and medicine, where annotated data are comparatively scarce due to overly high annotation costs. therefore, there is a practical and urgent need for developing low-resource re models without the reliance on large-scale end-task annotations. to realize low-resource re, previous work has focused on pretraining entity pair embedding on large corpora using re-based pretraining objectives. @xcite propose a self-supervised matching-theblanks (mtb) objective that encourages embeddings of the same entity pairs in different sentences to be similar. later work @xcite extends this idea with distant supervision @xcite and improves representation learning using contrastive learning @xcite @xcite . to adapt to training on re annotations, these works finetune pretrained entity pair embedding on labeled data using classification-based objectives. although this paradigm produces better results compared to re models initialized with pretrained language models (plms), it creates a significant divergence between pretraining and finetuning objectives, thus preventing the model from fully exploiting knowledge in pretraining. in this paper, we aim to bridge this gap in re pretraining and finetuning. our key idea is to use similar objectives in pretraining and finetuning. first, we propose to continually finetune pretrained embedding by contrastive learning, which encourages the entity pair embeddings corresponding to the same relation to be similar. however, as pretraining and finetuning are conducted on different tasks, entity pairs of the same relation can form multiple different clusters in the pretrained embedding, where standard supervised contrastive loss @xcite may distort the representation because of its underlying onecluster assumption @xcite . therefore, we further propose a multi-center contrastive loss (mccl), which encourages an entity pair to be similar to only a subset of entity pairs of the same relation, allowing one relation to form multiple clusters. second, we propose to use classwise k-nearest neighbors (knn; @xcite in inference, where predictions are made based on most similar instances. we focus our work on document-level re @xcite , which consists of both intra-and cross-sentence relations. to the best of our knowledge, this work represents the first effort to explore self-supervised pretraining for document-level re. unlike prior studies @xcite , we do not use distant supervision. instead, we pretrain entity pair embedding with an improved mtb objective on unlabeled corpora, where we use contrastive learning to learn representations that suit downstream re. we then finetune the pretrained model on labeled data with mccl. experiments on two datasets, biored @xcite in the biomedical domain and re-docred @xcite in the general domain, demonstrate that our pretraining and finetuning objectives significantly outperform baseline methods in low-resource settings. particularly, in the low-resource setting of using 1% of labeled data, our method outperforms plm-based classifiers by 10.5% and 6.1% on biored and re-docred, respectively. based on our pretrained representations, mccl outperforms classification-based finetuning by 6.0% and 4.1%, respectively. we also find observe that as more data becomes available, the performance gap between mccl and classification-based finetuning diminishes. our technical contributions are three-fold. first, we propose to pretrain the plms based on our improved mtb objective and show that it significantly improves plm performance in lowresource document-level re. second, we present a technique that bridges the gap of learning objectives between re pretraining and finetuning with continual contrastive finetuning and knnbased inference, helping the re model leverage pretraining knowledge. third, we design a novel mccl finetuning objective, allowing one relation to form multiple different clusters, thus further reducing the distributional gap between pretraining and finetuning. ## method in this work, we study a self-supervised approach for document-level re. given a document d and a set of entities {e i } n @xmath0, where each entity e i has one or multiple entity mentions in the document, document-level re aims at predicting the relations of all entity pairs (e s , e o ) s,@xmath1,...,n } from a predefined set of relationships r (including an na class indicating no relation exists), where e s and e o are the subject and object entities, respectively. in the self-supervised re setting, we have a large unlabeled document corpus for pretraining and a labeled re dataset for finetuning. the document corpus has been annotated with entity mentions and the associated entity types but no relations. our goal is to train a document-level re classifier, especially in the low-resource setting. our training pipeline consists of two phases: pretraining and finetuning. in pretraining, we use the (unlabeled) document corpus to pretrain the entity pair embedding based on our improved matching-the-blanks training objective (mtb; baldini soares et al. 2019), where the lm learns to decide whether two entity pair embeddings correspond to the entity pairs or not, and the learning of representation is enhanced with contrastive learning. in finetuning, we continue to train the pretrained model on relation-labeled data using a multi-center contrastive loss (mccl), which achieves better performance than the traditional classifier paradigm due to its better-aligned learning objective with pretraining. after training, we use classwise k-nearest neighbor (knn) inference that suits well the contrastively finetuned model. the rest of this section is organized as follows: we introduce the model architecture used in both pretraining and finetuning in section 3.1, the pretraining process in section 3.2, finetuning in section 3.3, and inference in section 3.4. ## experiments we evaluate our proposed method with a focus on low-resource re (sections 4.1-4.3), and present detailed analyses (section 4.4) and visualization (section 4.5) to justify method design choices. ## conclusion in this paper, we study self-supervised learning for document-level re. our method conducts an improved mtb pretraining objective that acquires cheap supervision signals from large corpora without relation labels. to bridge the gap between pretraining and end-task finetuning, we propose a continual contrastive finetuning objective, in contrast to prior studies that typically use classification-based finetuning, and use knnbased inference. as pretrained representation may form multi-cluster representation, we further propose a multi-center contrastive loss that aligns well with the nature of the pretrained representation. extensive experiments on two documentlevel re datasets demonstrate the effectiveness of these key techniques in our method. future work is adapting our method to other tasks in information extraction, such as n-ary relation extraction, named entity recognition, typing, and linking.