A Unified Representation Learning Strategy for Open Relation Extraction with Ranked List Loss (2021)

Abstract: Open Relation Extraction (OpenRE) aiming to extract relational facts from open-domain cor-pora is a sub-task of Relation Extraction and a crucial upstream process for many other NLPtasks. However various previous clustering-based OpenRE strategies either confine themselves to unsupervised paradigms or can not directly build a unified relational semantic space henceimpacting down-stream clustering. In this paper we propose a novel supervised learning frame-work named MORE-RLL (Metric learning-based Open Relation Extraction with Ranked ListLoss) to construct a semantic metric space by utilizing Ranked List Loss to discover new rela-tional facts. Experiments on real-world datasets show that MORE-RLL can achieve excellent performance compared with previous state-of-the-art methods demonstrating the capability of MORE-RLL in unified semantic representation learning and novel relational fact detection.

## introduction relation extraction (re) aims to extract pre-defined relational facts from plain text (e.g., "mary gave birth to keller in 1989s.", re can extract "gave birth to" between two named entities "mary" and "keller"). it is an important task that can structure a large amount of text data. therefore, it can benefit for unstructured text data storing and the procedure of many other down-stream nlp tasks or applications, such as knowledge graph construction @xcite , information retrieval @xcite , and logic reasoning @xcite . nevertheless, with the rapid development of social media and human civilization, novel relationships and new knowledge in open-domain text data are also increasing. accordingly, the relation types in the open-domain corpora may not be pre-defined, which is hard for re to handle. to meet the rapid emergence of such novel knowledge, openre emerged as the times required @xcite . the goal of openre is to detect novel relational facts from open-domain datasets. it is a crucial task for updating the human knowledge base and the study of human civilization. existing openre methods are divided into two main categories: tagging-based and clustering-based. the tagging-based strategies treat openre as a sequence labeling problem @xcite . still, these methods often extract surface forms that can not be utilized for downstream tasks (i.e., some sequences have the same relational semantic type, but their phrases generated from tagging-based methods are different because of overly-specific). comparatively, clustering-based methods aim to identify the rich semantic features in the text, then cluster them into certain relation types. recently, many efforts have been devoted to exploring clustering-based methods, such as @xcite . yet, those schemes are laborious and time-consuming because of the high dependence on well-designed features created by hand. profited from the substantial improvement of computing power in recent years, neural networks begin to be exploited in clustering-based openre tasks to alleviate the above issues, such as @xcite @xcite . even so, these strategies confine themselves to unsupervised or self-supervised paradigms and can not fully benefit from current high-quality human-labeled corpora. although several unconventional works have gained phenomenal performance, such as @xcite , the reliance on the extra knowledge for these strategies make it hard for us to compare with. besides, another supervised scheme learned the similarity metrics from labeled instances and further transferred the relational knowledge to the open-domain scene, namely relational siamese networks (rsns) @xcite . however, rsns target learning a similarity classifier rather than building relational representations directly. thereby, this may impact the efficiency and effect of down-stream clustering. in order to address these issues, we propose a novel supervised learning framework via a clusteringbased scheme driving neural encoder to build rich semantic representation directly. from our insight view, the essential target of the clustering-based openre algorithm is to construct a reasonable semantic space on the open-domain corpora, where all different relational facts can be distinguished clearly. therefore, the learning of semantic representation is a fundamental part of the whole task. it can not only extend the functionality of the neural encoder (i.e., the semantic space construction ability of the neural model can be used in other scenes, such as classification, etc.) but also bring benefits to downstream clustering. as a result, we pay attention to the unified semantic representation learning ability of neural encoders. specifically, we employ deep metric learning to drive the neural encoder to build a distinguishable semantic space on open-domain datasets. however, most prevailing deep metric learning methods, such as triplet loss @xcite , n-pair-mc @xcite , or proxy-nca @xcite , always bring low yield due to the poor supervision signals from the limited number of training data points. inspired by @xcite , we chose ranked list loss (rll) instead, which can capture set-based rich supervision signals. meanwhile, rll can preserve a better intraclass similarity structure within a hypersphere than other set-based schemes, hence constructing a more desirable semantic space. additionally, considering that the open scene corpora is usually full of noise, hence directly transferring knowledge may not be an ideal choice. to enhance the model's robustness, we also design virtual adversarial training for our semantic space construction algorithm. experiments demonstrate that more-rll can build more distinguishable semantic representations and obtain excellent performances on real-world datasets. to sum up, the main contributions of this work are as follows: ## methodology in this section, we will introduce our framework in detail. as shown in figure 1 , we exploit a neural encoder to extract relational representations from a batch of training samples. these sentence-level representations can be taken as relational semantic vectors, indicating the relative locations of facts in the semantic feature space. then, we use them to calculate the ranked list loss (rll) and gain rich c c l 2 0 2 1 london is the capital of the u.@xmath0. besides, we set virtual adversarial training to smooth the feature space to overcome noise in the open scenes. we repeat these steps until the encoder is well-trained and then transfer the prior knowledge from the training samples to the open-domain corpora. ## more-rll(bert) owing to the powerful extracting ability of the pre-trained language model, the performance of our framework has been greatly improved. in fact, there are multiple encoders that can be adopted in our framework. due to the limitation of the space, we do not extend it here. ## conclusion in this paper, we propose a novel supervised learning framework for open-domain relation extraction, namely more-rll. it aims to make the neural network gain a unified relational representation encoding ability and handle the open-domain relational instances. we utilize deep metric learning to drive the c c l 2 0 2 1 neural model to learn relational representations directly, thereby conducive to downstream clustering efficiency. moreover, we set virtual adversarial training to enhance the robustness of the neural encoder. our experiments show that more-rll achieves state-of-the-art performance on real-world re corpora comparing with previous methods and can build a more desirable semantic space. these all demonstrate the capability of our scheme on relational representation learning and novel relation detection.