SCL : Selective Contrastive Learning for Data-driven Zero-shot Relation Extraction (2024)

Abstract: Relation extraction has evolved from supervised relation extraction to zero-shot setting due to the continuous emergence of newly generated relations. Some pioneering works handle zero-shot relation extraction by reformulating it into proxy tasks, such as reading comprehension and textual entailment. Nonetheless, the divergence in proxy task formulations from relation extraction hinders the acquisition of informative semantic representations, leading to subpar performance. Therefore, in this paper, we take a data-driven view to handle zero-shot relation extraction under a three-step paradigm, including encoder training, relation clustering, and summarization. Specifically, to train a discriminative relational encoder, we propose a novel selective contrastive learning framework, namely, SCL, where selective importance scores are assigned to distinguish the importance of different negative contrastive instances. During testing, the prompt-based encoder is employed to map test samples into representation vectors, which are then clustered into several groups. Typical samples closest to the cluster centroid are selected for summarization to generate the predicted relation for all samples in the cluster. Moreover, we design a simple non-parametric threshold plugin to reduce false-positive errors in inference on unseen relation representations. Our experiments demonstrate that SCL outperforms the current state-of-the-art method by over 3% across all metrics.

## introduction relation extraction (re) aims to infer the relation between a pair of entities from text, which is a vital step for automatic knowledge graph construc-tion @xcite . recent supervised efforts define relation extraction @xcite @xcite as a multi-class classification task, and select the most appropriate relation label from a pre-defined set of relations for the target entity pair. however, in practice, the training data often fails to cover all relations, and thus supervised re methods may not be well suited for recognizing unobserved relations during training. to eliminate labor-intensive annotation for emergent relations, open relation extraction (openre) was first proposed to discover fresh relations by clustering without using any prior knowledge of scope and distribution @xcite . however, openre approaches @xcite fail to take advantage of relational knowledge in previously accumulated relations. confronted with this limitation, zero-shot relation extraction (zsre) has come as a remedy to train a model on historical relations with annotated instances and generalize it to extract relations that have never been seen during training @xcite . since training and testing relations are disjoint under the zero-shot setting, zsre models have no access to supervised signals for test relations. therefore, zero-shot learning represents a longstanding challenge in transferring knowledge from training data to test data @xcite . to meet the challenge of knowledge transfer, some previous research handles zsre by formulating it into a proxy task formulation, such as reading comprehension @xcite , textual entailment @xcite , and attribute representation matching @xcite . since they are required to define answering templates or relation descriptions, searching for suitable and effective options imposes a burden. additionally, these methods are based on the assumption that humans know what new relations need to be extracted in advance, which is easy to violate in real applications since new relations are always summarized from corresponding emergent instances. that is, we contend that zsre models are expected to be data-driven rather than human-guided. as a pioneering work, @xcite have adopted this idea, but fail to comprehensively define how to perform data-driven zsre. in this paper, we further formulate the datadriven paradigm of zsre, the workflow of which is depicted in figure 1 . in this formulation, we first train a relational encoder on instances of seen relations and employ it to map test samples into an embedding space. the embeddings of these test samples are clustered into several groups, and high-quality samples within each cluster are chosen for summarization by @xcite , generating a new relation name assigned to the corresponding cluster as the predicted label. therefore, the key to solving data-driven zsre lies in learning an effective projection function that can map instances of the same relation nearby and instances of different relations far apart in the embedding space. considering the success of contrastive learning in capturing discriminative representations @xcite , @xcite harness an instancelevel self-supervised contrastive method to train the relational encoder to better learn subtle differences between instances. however, there are still three drawbacks in existing methods when addressing the aforementioned paradigm of zsre: to address the drawbacks, we design a novel selective contrastive learning framework, scl, to handle data-driven zsre. (1) at the training stage, we employ prompt learning @xcite and adapt it to the task of zsre. by bridging the gap between pre-training objective and downstream task, prompt-tuning can take full advantage of knowledge in pre-trained language models. additionally, to learn better separation of relations, we propose to perform in-batch selective contrastive learning. for a batch of instances, we augment them with a different prompt template to construct another view as positive contrastive instances. it is noteworthy that we assign selective importance scores for various contrastive instances of an anchor, where we dynamically emphasize hard negatives since they are more useful for learning discriminative representations. (2) at the test stage, by following @xcite , we use the well-learned model to project all test samples into the representation space and separate them into several clusters by clustering algorithms @xcite . high-quality samples around the centroid of each cluster are selected for summarization as the prediction for all instances. based on the observation that in each cluster, false positives always have a greater distance to their centroid, we propose a post-processing method to detect these false-positive instances. specifically, we design a simple but effective threshold criterion to determine if a test instance is false positive or not. previous zsre works have never investigated this non-parametric method to improve performance, indicating that the seemingly simple idea is non-trivial. ## related work in this section, we review the related work from three perspective: zero-shot relation extraction, relational representation learning, and contrastive learning. ## preliminary this section defines the task of data-driven zsre, and then overviews our proposed scl method. ## methodology this section introduces our proposed method, namely, scl, to accomplish zero-shot relation extraction. in the following, we describe our prompt-based encoder in section 4.1, prompttuning module in section 4.2, contrastive learning module in section 4.3, and relation inference module in section 4.4. ## experiments in this section, we first describe the experimental setup and then present the experiment results with an in-depth analysis. ## conclusion in this paper, we re-investigate the task of zeroshot relation extraction (zsre) and propose a training method scl to transfer relational knowledge learned from seen relations to unseen relations. we formally define a three-step paradigm to perform data-driven relation extraction under a zero-shot setting, including encoder training, relation clustering, and summarization. specifically, to train a discriminative relational encoder, we propose a selective contrastive learning approach based on prompt-tuning, where false negative examples are removed and different importance scores are assigned to emphasize the importance of different negative samples. during the testing phase, we cluster the encoded test samples. to convert the virtual labels of the clustering results into relation labels, we select typical samples and summarize relation names from them. based on the distances between negative contrastive instances and the anchor instance, we can assess the difficulty of pushing each negative contrastive instance away from the anchor. the closer the distance, the more difficult the negative sample, indicating its higher importance in the contrastive loss. it is noteworthy that the importance score here is computed based on the relation representation of the instances rather than hyperparameters or learnable parameters. after evaluating the importance scores of different negative instances, we can then obtain the selective contrastive learning loss: substituting equation ( 2 ) into equation (3) will yield: in this view, the selective contrastive loss seems to be a combination of two similarities. we did not directly employ equation ( 19 ) to introduce the loss l scl because this form is less conducive to understanding its meaning. therefore, we compute the importance score and selective contrastive loss via equations ( 17 ) and ( 18 ), respectively, to better illustrate how we assess the importance score of a negative contrastive instance and how this importance score is reflected in the loss.