U - CORE : A Unified Deep Cluster-wise Contrastive Framework for Open Relation Extraction (2023)

Abstract: Within Open Relation Extraction (ORE) tasks, the Zero-shot ORE method is to generalize undefined relations from predefined relations, while the Unsupervised ORE method is to extract undefined relations without the need for annotations. However, despite the possibility of overlap between predefined and undefined relations in the training data, a unified framework for both Zero-shot and Unsupervised ORE has yet to be established. To address this gap, we propose U-CORE: A Unified Deep Cluster-wise Contrastive Framework for both Zero-shot and Unsupervised ORE, by leveraging techniques from Contrastive Learning (CL) and Clustering.1 U-CORE overcomes the limitations of CL-based Zero-shot ORE methods by employing Cluster-wise CL that preserves both local smoothness as well as global semantics. Additionally, we employ a deep-cluster-based updater that optimizes the cluster center, thus enhancing the accuracy and efficiency of the model. To increase the stability of the model, we adopt Adaptive Self-paced Learning that effectively addresses the data-shifting problems. Experimental results on three well-known datasets demonstrate that U-CORE significantly improves upon existing methods by showing an average improvement of 7.35% ARI on Zero-shot ORE tasks and 15.24% ARI on Unsupervised ORE tasks.

## introduction relation extraction (re) is a fundamental task in natural language processing (nlp) that aims to extract the relationships between pairs of entities * hongkui tu is the corresponding author. 1 the code can be found at: https://github.com /2kjiejie/u-core . mentioned in a given text, such as identifying the effect-cause relation between ''fire'' and ''fuel'' in the sentence ''the fire was caused by exploding fuel.'' re is an essential component of nlp systems that can facilitate diverse downstream tasks, including question answering (soares and parreiras, 2020), knowledge graphs @xcite , and dialogue systems @xcite . while supervised methods have demonstrated great success in extracting predefined relations, in reality, new relations frequently arise, and it can be time-consuming and labor-intensive to define them manually. as a result, open-domain relation extraction has become a popular research topic. based on prior studies, open relation extraction (ore) tasks can be categorized into two types, namely, zero-shot open relation extraction (zore) and unsupervised open relation extraction (uore). zore aims to extract novel relational facts where the target relation types are not observed in the training set @xcite . on the other hand, uore has the objective of extracting undefined relations without any annotation or prior knowledge @xcite . in recent years, significant attention has been devoted to ore tasks @xcite @xcite . despite variations in complexity and annotations, both zore and uore aim to develop an optimal encoder that can generate an appropriate relational representation using limited resources. additionally, practical scenarios involving zore and uore may overlap, such as when training data contain both predefined and undefined relations. consequently, the previous zore and uore methods, which concentrate on predefined or undefined relations, respectively, may yield suboptimal results due to their inability to account for all relations. given the similarities between ore techniques, as well as their potential overlap in practical applications, we propose a unified framework that addresses diverse ore tasks. recent studies have attempted to improve relation representation in zore by leveraging contrastive learning (cl) approaches @xcite . these methods typically rely on instance-wise cl, which aims to bring together relations from the same instances while separating those from different instances. however, @xcite have pointed out that instance-wise cl may treat instances with similar semantic information as negative pairs, leading to a drift in their representations and resulting in performance degradation. in the ore task, instances that belong to the same relation type share ''similar semantic information'' and should not be treated as negative paris. to address this limitation, we propose employing a cluster-wise contrastive learning approach, which facilitates the alignment of relations within the same clusters and the separation of relations across different clusters. contrarily, in uore tasks, the training process is unsupervised, and existing models tend to learn structural information from clustering techniques @xcite . however, the majority of existing uore methods depend on conventional clustering algorithms, such as k-means, for defining clustering centers. reclustering at the end of each epoch is typically required, which may be time-consuming and computationally intensive. consequently, we incorporate deep clustering into our ore framework to eliminate the need for frequent re-clustering and enhance the clustering performance. the combination of cluster-wise contrastive learning and deep clustering plays a crucial role in our unified ore framework. deep clustering enhances the performance of cluster-wise cl by providing more accurate clusters, while clusterwise cl improves deep clustering by generating better relation representations. however, during the training process, while the encoder is updated in each minibatch, the clustering assignment is only updated at the end of each epoch. this inconsistency between the relation representations and cluster centers in the feature space is known as the data-shifting problem, which has been identified in previous works @xcite . this problem requires careful attention in our unified framework to avoid the misalignment between the relation representations and the cluster centers. based on the above analysis, we propose u-core: a unified deep cluster-wise contrastive framework for open relation extraction in this article. the proposed framework aims to establish a unified approach to enhance the performance of both zore and uore methods. specifically, the cluster-wise contrastive learning approach is employed in our ore framework, which increases the inter-cluster spacing of clusters while minimizing intra-cluster spacing, enabling us to overcome the limitations of previous cl-based zore methods. to mitigate the need for regular re-clustering and enhance overall accuracy and efficiency, we introduce a deep cluster center updater. moreover, we propose the integration of adaptive self-paced learning in the proposed u-core to address issues regarding data-shifting and produce a more stable model. the framework is capable of obtaining an effective representation of relations and able to handle both zero-shot ore and unsupervised ore tasks. overall, our proposed u-core framework contributes towards the development of highly effective and versatile techniques for open relation extraction. the architecture of our framework is shown in figure 1 . we briefly summarize our contribution as follows: in this section, we survey the related work on open relation extraction, contrastive learning, and deep clustering. ## conclusion in this paper, we present a unified deep clusterwise contrastive framework, u-core, for open relation extraction tasks. our proposed framework can tackle various ore tasks and overcome the limitations of previous instance-wise cl-based methods. furthermore, we introduce the cluster center updater and adaptive self-paced learning to enhance the stability and efficiency of our model. the results of our experiments on three datasets provide evidence of the effectiveness of our framework, achieving new state-ofthe-art performance. recently, large language models (llms) like chatgpt 3 have demonstrated remarkable performance in various nlp tasks, but @xcite and @xcite indicate that llms exhibit subpar performance in ore tasks. from our aspect, we believe that llms have the potential to address ore tasks. in light of this, our feature work is to further explore the potential of llms in ore tasks.