Open-world Semi-supervised Generalized Relation Discovery Aligned in a Real-world Setting (2023)

Abstract: Open-world Relation Extraction (OpenRE) has recently garnered significant attention. However, existing approaches tend to oversimplify the problem by assuming that all instances of unlabeled data belong to novel classes, thereby limiting the practicality of these methods. We argue that the OpenRE setting should be more aligned with the characteristics of real-world data. Specifically, we propose two key improvements: (a) unlabeled data should encompass known and novel classes, including negative instances; and (b) the set of novel classes should represent long-tail relation types. Furthermore, we observe that popular relations can often be implicitly inferred through specific patterns, while long-tail relations tend to be explicitly expressed. Motivated by these insights, we present a method called KNoRD (Known and Novel Relation Discovery), which effectively classifies explicitly and implicitly expressed relations from known and novel classes within unlabeled data. Experimental evaluations on several Open-world RE benchmarks demonstrate that KNoRD consistently outperforms other existing methods, achieving significant performance gains.

## introduction relation extraction (re) is a fundamental task in natural language processing (nlp), aiming to extract fact triples in the format ⟨head entity, relation, tail entity⟩ from textual data. open-world re (openre) is a related research area that focuses on discovering novel relation classes from unlabeled data. recent advancements in openre have demonstrated impressive results by integrating prompting techniques with advanced clustering methods @xcite @xcite . however, current openre methods face limitations due to assumptions about unlabeled data that do not align with the characteristics of real-world datasets. these assumptions include: * corresponding author (1) the presumption that unlabeled data solely consists of novel classes or is pre-divided into sets of known and novel instances; (2) the absence of negative instances; (3) the random division of known and novel classes in a dataset; and (4) the availability of the ground-truth number of novel classes in unlabeled data. in this work, we critically examine these assumptions and align the task of openre within a realworld setting. we dispose of simplifying assumptions in favor of new assumptions that align with characteristics of real-world unlabeled data in hopes of increasing the practicality of these methods. we call our setting generalized relation discovery and make the following claims: our experimental results show that our proposed setting makes for a difficult task, ripe for advancements and future work. state-of-the-art approaches in relation discovery leverage a prompt-learning method to predict relation class names which are then embedded into a latent space and grouped via clustering @xcite @xcite . in addition to making simplifying assumptions about unlabeled data, past works use unconstrained predictions for relation class names-that is, the prompt model can predict any word in its vast vocabulary as a relation class name. we observe that relationships in text can be expressed either explicitly or implicitly. explicit instances contain class-indicative words, and implicit relationships are inferred via lexical patterns (see appendix a.1 for examples). in this work, we illustrate the effectiveness of designing a prompt method that optimizes for explicit and implicit relationships by predicting relation class names in two settings: (1) constrained to words found within an inputted instance; and, (2) unconstrained, where the model can predict any word within its vocabulary. constrained predictions optimize for explicitly expressed relationships while unconstrained predictions optimize for implicitly expressed relationships. this prompt method forms the backbone of our proposed method, known and novel relation discovery (knord), which can effectively classify explicitly and implicitly expressed relationships of known and novel classes from unlabeled data. another key aspect of our method is that it clusters labeled and unlabeled data within the same featurespace. each labeled instance serves as a "vote" for a cluster belonging to the set of known classes. we effectively bifurcate clusters into sets of known and novel classes by employing a majority-vote strategy. novel-identified clusters are then utilized as weak labels, in combination with gold labels, to train a model via cross-entropy (see figure 1 ). this methodology presents an innovative approach to relation discovery in open-world scenarios, offering potential applications across various nlp domains. the main contributions of this work are: traditionally, re methods have focused on a closed-world setting where the extracted relations are predefined during training (califf and @xcite @xcite @xcite . however, datasets used for training are rarely complete, and traditional re cannot capture new relation classes as additional data becomes available. researchers have proposed various approaches to discover emergent classes to address this issue. open-world re: open-world re seeks to discover new relation classes from unlabeled data. instances of relations are typically embedded into a latent space and then clustered via k-means. these works often simplify the task by assuming all instances of unlabeled data belong to the set of novel classes and that unlabeled data contains no negative instances @xcite @xcite . more recently, some openre methods have been proposed to predict known and novel classes from unlabeled data @xcite @xcite . however, these works assume unlabeled data comes pre-divided into sets of known and novel instances, that negative instances are removed, and that the number of novel classes is known. open-world semi-supervised learning: the setting we propose for relation discovery is inspired by open-world semi-supervised learning (open ssl) proposed by @xcite where the authors use a margin loss objective with adaptive uncertainty to predict known and novel classes from a set of unlabeled images. besides the difference in domains, their setting differs from ours in that they assume unlabeled data has an equal number of known and novel instances-an assumption we cannot make when working with relation extraction datasets which often have imbalanced, longtail class distributions @xcite @xcite . furthermore, @xcite assumes unlabeled data only contains positive instances, which does the setting proposed in @xcite is similar to ours however, negative instances are removed and their known/novel class splits are done randomly instead of by class frequency. furthermore, their method relies on active learning where human annotators annotate instances of novel classes. the annotations are then used to train a classifier. in contrast, our model and the baselines we evaluate do not require human annotation for novel classes. we formulate our task in the following way: each sentence x j contains an entity pair-a head entity e 1 and a tail entity e 2 -and the predicted relationship y j links the two entities, producing a fact triplet ⟨e 1 , y j , e 2 ⟩. ## method knord consists of four discrete stages: ## datasets we evaluate knord on three re datasets: ta-cred @xcite , retacred @xcite , and fewrel @xcite . for each dataset, we first construct splits of known and novel classes based on class frequency, assigning the top 50% most frequent relation classes to the set of known classes (c k ) and the lower 50% to the set of novel classes (c n ) (see figure 2 ). since fewrel is a balanced dataset with relationships defined from a subset of wikidata relationships, we obtain real-world class frequencies based on their frequency within wikidata. for more details on our to form the unlabeled dataset d u . the remaining 85% of known-class instances are used to create the labeled dataset d l . we include two versions of each dataset: with and without negative instances (e.g., the no relation class). the setting with negative instances best mirrors real-world data; however, as our experiments show, discovering novel classes in a sea of negative instances is difficult. we include results from both settings and the setting with negative instances will be ripe for advancement and future work. our focus is to evaluate methods on data with no distribution drift (i.e., known classes from training occur in the unlabeled data along with novel classes). we leave an evaluation on out-ofdistribution datasets @xcite to future work. ## experiments we compare knord to state-of-the-art openre baselines: (1) rocore @xcite , (2) matchprompt (wang et al., 2022a), 6 (3) tabs @xcite . since openre methods cannot naturally operate within the generalized relation discovery setting, we extend the openre baselines in the following ways: orca: we include ocra @xcite , a computer vision model developed for a similar generalized open-world setting. ocra is the only model architecture in our experiments that can predict known and novel classes from unlabeled data and, thus, requires no modification, beyond adaptation to the re task, to function within our proposed setting. for more details about adapting orca to predict relationships, see appendix a.4.5. gpt 3.5: given the zero-shot learning capabilities of large language models @xcite , we also include gpt 3.5 (openai, 2021) 7 as a baseline. to assess gpt 3.5, we leverage in-context learning-we provide examples of extracted relationships and a list of known relation classes. we instruct the model to predict the most appropriate relation class name or suggest a novel class name when an instance does not fit within the set of known classes (see appendix a.4.1 for details). gpt 3.5 +cos: since responses from gpt 3.5 may not align perfectly with ground-truth labels, we use deberta to map the responses (y i ) to ground-truth class names by embedding the predictions and the ground-truth class names (z y i and for all our baselines, we use identical settings for the number of known and novel classes and, save gpt 3.5, we use the same pre-trained model @xcite as a base model and map predictions to ground-truth classes via the hungarian algorithm. we use micro-f1 scores to assess relation classification performance and report overall performance as well as performance on known and novel classes to assess a model's ability to identify each class type from unlabeled data. ## results our proposed method, knord, outperforms the baseline models in all metrics ( we observe that gpt 3.5 underperforms in this setting. although mapping responses to ground-truth classes (gpt 3.5 +cos) yields a slight performance boost, the model still performs poorly relative to our other baselines. given the unsatisfactory results from gpt 3.5 in our simplified experiment setting without negative instances, we decide to exclude it from the more challenging setting where negative instances are present. we conclude that more advanced techniques are required to enable gpt 3.5 to accurately classify and discover relationships from textual data. a deeper examination of gpt 3.5's performance is provided in appendix a.4.3. in the setting with negative instances, all methods struggle to identify novel relation classes indicating the difficulty of discovering new classes among instances with no relation. we attribute the lower overall performance using tacred compared to retacred to tacred's wrong labeling problem @xcite . the relatively small drop in performance of all models between fewrel with and without negative instances can be attributed to fewrel lacking annotated negative instances, so we artificially augment the data with negatives from retacred. we posit that the models can exploit the slight difference in the distribution of the augmented negative instances, thus reducing the task's difficulty. these results emphasize the importance of future re dataset creation efforts in annotating negative instances. ## conclusion in this work, we address the limitations of existing approaches in openre and introduce the generalized relation discovery setting to align the task to characteristics of data found in the real-world. by expanding the scope of unlabeled data to include known and novel classes, as well as negative instances, and incorporating long-tail relation types in the set of novel classes, we aim to enhance the practicality of openre methods. furthermore, we propose knord, a novel method that effectively classifies explicitly and implicitly expressed relations from known and novel classes within unlabeled data. through comprehensive experimental evaluations on various open-world re benchmarks, we demonstrate that knord consistently outperforms existing methods, yielding significant performance gains. these results highlight the efficacy and potential of our proposed approach in advancing the field of openre and its applicability to real-world scenarios.