A Relation-Oriented Clustering Method for Open Relation Extraction (2021)

Abstract: The clustering-based unsupervised relation discovery method has gradually become one of the important methods of open relation extraction (OpenRE). However, high-dimensional vectors can encode complex linguistic information which leads to the problem that the derived clusters cannot explicitly align with the relational semantic classes. In this work, we propose a relation-oriented clustering model and use it to identify the novel relations in the unlabeled data. Specifically, to enable the model to learn to cluster relational data, our method leverages the readily available labeled data of pre-defined relations to learn a relation-oriented representation. We minimize distance between the instance with same relation by gathering the instances towards their corresponding relation centroids to form a cluster structure, so that the learned representation is cluster-friendly. To reduce the clustering bias on predefined classes, we optimize the model by minimizing a joint objective on both labeled and unlabeled data. Experimental results show that our method reduces the error rate by 29.2% and 15.7%, on two datasets respectively, compared with current SOTA methods.

## introduction relation extraction (re), a crucial basic task in the field of information extraction, is of the utmost practical interest to various fields including web search @xcite , knowledge base completion @xcite , and question answering @xcite . however, conventional re paradigms such as supervision and distant supervision are generally designed for pre-defined relations, which cannot deal with new emerging relations in the real world. under this background, open relation extraction (openre) has been widely studied for its use figure 1 : although both instances s 2 and s 3 express founded relation while s 1 expresses ceo relation, the distance between s 1 and s 2 is still smaller than that between s 2 and s 3 . this is because there may be more similar surface information (e.g. word overlapping) or syntactic structure between s 1 and s 2 , thus the derived clusters cannot explicitly align with relations. in extracting new emerging relational types from open-domain corpora. the approaches used to handle open relations roughly fall into one of two groups. the first group is open information extraction (openie) @xcite @xcite , which directly extracts related phrases as representations of different relational types. however, if not properly canonicalized, the extracted relational facts can be redundant and ambiguous. the second group is unsupervised relation discovery @xcite @xcite . in this type of research, much attention has been focused on unsupervised clustering-based re methods, which cluster and recognize relations from high-dimensional representations @xcite . recently, the self-supervised signals in pretrained language model are further exploited for clustering optimization @xcite . however, many studies show that highdimensional embeddings can encode complex linguistic information such as morphological @xcite , local syntactic @xcite , and longer range semantic information @xcite . consequently, the distance of representation is not completely consistent with relational semantic similarity. although @xcite use self-supervised signals to optimize clustering, there is still no guarantee that the learned clusters will explicitly align with the desired relational semantic classes @xcite . as shown in figure 1 , we use the method proposed by @xcite to get the instance representations. although both instances s 2 and s 3 express the founded relation, the euclidean distance between them is larger than that between s 1 and s 2 , which express different relation. obviously, the clustering algorithm tends to group instances s 1 and s 2 together, rather than s 2 and s 3 which express the same relation. in this work, we propose a relation-oriented clustering method. to enable the model to learn to cluster relational data, pre-defined relations and their existing labeled instances are leveraged to optimize a non-linear mapping, which transforms high-dimensional entity pair representations into relation-oriented representations. specifically, we minimize distance between the instances with same relation by gathering the instances representation towards their corresponding relation centroids to form the cluster structure, so that the learned representation is cluster-friendly. in order to reduce the clustering bias on the predefined classes, we iteratively train the entity pair representations by optimizing a joint objective function on the labeled and unlabeled subsets of the data, improving both the supervised classification of the labeled data, and the clustering of the unlabeled data. in addition, the proposed method can be easily extended to incremental learning by classifying the pre-defined and novel relations with a unified classifier, which is often desirable in real-world applications. our experimental results show that our method outperforms current state-of-the-art methods for openre. our codes are publicly available at github * . to summarize, the main contributions of our work are as follows: (1) we propose a novel relation-oriented clustering method rocore to enable model to learn to cluster relational data; (2) the proposed method achieves the incremental learning of unlabeled novel relations, which is often desirable in real-world applications; (3) experimental results show that our method reduces * https://github.com/ac-zyx/rocore . the error rate by 29.2% and 15.7%, on two realworld datasets respectively, compared with current state-of-the-art openre methods. 2 related work open relation extraction. to meet the needs of extracting new emerging relation types, many efforts have been undertaken to exploring methods for open relation extraction (openre). the first line of research is open information extraction @xcite @xcite , in which relation phrases are extracted directly to represent different relation types. however, using surface forms to represent relations results in an associated lack of generality since many surface forms can express the same relation. recently, unsupervised clustering-based re methods is attracting lots of attentions. @xcite proposed to extract and cluster open relations by re-weighting word embeddings and using the types of named entities as additional features. @xcite proposed to exploit weak, self-supervised signals in pretrained language model for adaptive clustering on contextualized relational features. however, the self-supervised signals are sensitive to the initial representation @xcite and there is still no guarantee that the learned clusters will align with the relational semantic classes @xcite . @xcite proposed the relation similarity metrics from labeled data, and then transfers the relational knowledge to identify novel relations in unlabeled data. different from them, we propose a relation-oriented method explicitly clustering data based on relational information. knowledge in high-dimensional vector. pretrained static and contextual word representations can provide valuable prior knowledge for constructing relational representations @xcite . @xcite showed that different neural architectures (e.g., lstm, cnn, and transformers) can hierarchically structure linguistic information that varies with network depth. recently, many studies @xcite @xcite have shown that such hierarchy also exists in pretraining models like bert. these results suggest that high-dimensional embeddings, independent of model architecture, learn much about the structure of language. directly clustering on these highdimensional embeddings should hardly produce entity pair encoder (bert) [cls] bill gates was born in seattle in 1995 [sep] step 1: obtain entity pair representation relational semantic space step 2: optimize clustering and generate pseudo labels for novel relation optimized relational semantic space relational centroids step 3: train relation classifier and refine entity pair representation classifier for pre-defined relaion classifier for novel relation backprop non-linear mapping non-linear decoder ideal clusters in our desired way, which motivates us to extend current unsupervised clustering-based re methods to learn the representations tailored for clustering relational data. ## approach in this work, we propose a relation-oriented clustering method, which takes advantage of the relational information in the existing labeled data to enable model to learn to cluster relational data. in order to reduce the clustering bias on the predefined classes, we iteratively train the entity pair representations by optimizing a joint objective function on the labeled and unlabeled subsets of the data, improving both the supervised classification of the labeled data, and the clustering of the unlabeled data. the proposed method is shown in figure 2 . specifically, given an unlabeled dataset d @xmath0,...,m of relational instances s u i , our goal is to automatically cluster the relational instances into a number of classes c u , which we assume to be known a priori. to enable the model to learn to cluster data, we incorporate a second labeled dataset of pre-defined relations d @xmath1, y ℓ i )} @xmath2,...,n where y ℓ @xmath3, ..., c ℓ } is the relational label for instance s ℓ i . ## experimental setup in this section, we describe the datasets for training and evaluating the proposed method. we also detail the baseline models for comparison. finally, we clarify the implementation details and hyperparameter configuration of our method. ## results and analysis in this section, we present the experimental results of our model on two real-world datasets to demonstrate the effectiveness of our method. we also provide additional experimental results on hyperparameter analysis and relation representation visualization in appendix a and b. ## conclusions in this work, we introduce a relation-oriented clustering method that extends the current unsupervised clustering-based openre method. the proposed method leverages the labeled data of pre-defined relations to learn a relation-oriented representation from which the derived clusters explicitly align with relational classes. iterative joint training method effectively reduces the unwanted bias on labeled data. in addition, the proposed method can be easily extended to incremental learning of novel relations. experimental results show that our method outperforms sota methods for openre. improved neural relation detection for knowledge base question answering. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 571-581, vancouver, canada. association for computational linguistics. yuhao zhang, victor zhong, danqi chen, gabor angeli, and christopher d. manning. 2017. positionaware attention and supervised data improve slot filling. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 35-45, copenhagen, denmark. association for computational linguistics.