UOREX : Towards Uncertainty-Aware Open Relation Extraction (2025)

Abstract: Open relation extraction (OpenRE) aims to identify relational facts within open-domain corpora without relying on predefined relation types. A significant limitation of current state-of-the-art OpenRE approaches is their inability to accurately self-assess their performance. Which is caused by the reliance on pseudo-labels, that treats all points within a cluster equally, regardless of their actual relative position according to the cluster center. This leads to models that are often overconfident in their incorrect predictions , significantly undermining their reliability. In this paper, we introduce an approach that addresses this challenge by effectively modeling a part of the epistemic uncertainty within OpenRE. Instead of using pseudo labels that mask uncertainty, our approach is built to train a classifier directly with the clustering distribution. Our experimental results across various datasets demonstrate that the suggested approach improves reliability of OpenRE by preventing overconfident errors. Furthermore we show that by improving the reliability of the predictions, UOREX operates more efficiently in a generative active learning context where an LLM is the oracle, doubling the performance gain compared to the state-of-the-art.

## introduction the extraction of relations between entities from unstructured text is an essential component of knowledge graph construction @xcite . these graphs enable numerous downstream applications, such as web search @xcite , question-answering @xcite , and more recently, retrieval augmented generation (rag) based on llms @xcite @xcite . traditionally, relation extraction has been limited to identifying predefined relations, leading to either the misclassification of new relations-adding inaccuracies to the knowledge graph-or labeling them labeled data of known relations unlabeled data of new relations open relation extraction filtering low confidence predictions (a) -bad uncertainty modeling results in a noisy filtered kg (b) -good uncertainty modeling results in a reliable filtered kg extracted knowledge graph data with relations classes that are not contained in labeled data correct prediction incorrect prediction nodes are entities, and edges represent the discovered new relations, with each edge color referring to a different relation. as "unknown" (zhao et al., 2023), which avoids misinformation but still overlooks valuable new connections. to address this limitation, open relation extraction has emerged as an essential research area for discovering and extracting new relation types from open-domain text. most recent openre research has focused on unsupervised relation discovery (urd) @xcite @xcite , that aims to discover new relations classes in an open-domain corpora. initial breakthroughs in this area were made by @xcite , followed by weakly-supervised or semi-supervised approaches like @xcite , @xcite , @xcite and @xcite . these methods employ a set of known relations as a starting point to discover new, unknown relations from documents. a common theme across these approaches is that they use clustering in a smaller space to guide a classifier that will discover relations in a much larger space. the main issue with this method is its reliance on pseudo-labels, which treat all points within a cluster equally, regardless of their actual relative position according to the cluster center. this leads to three challenges: (1) overconfidence in openre models, complicating their integration into real systems due to the high risk of introducing noise into knowledge graphs as illustrated in figure 1. (2) the need for thorough posttraining evaluation of openre methods. given the nature of current openre approaches, they are not inherently transparent; therefore, after training, there is no clear indication of their performance because the confidence of their predictions is forced to be high. this often requires the labeling of a significant portion of the target documents, which contradicts the inherent self-supervised nature of openre. (3) inefficient use of an llm oracle in a generative active learning context. because these models are overly confident, it's challenging to use them effectively in generative active learning scenarios @xcite @xcite where an llm serves as the oracle. their inability to accurately model uncertainty means that even their least confident predictions include a large amount of correct answers, making the use of the oracle less efficient. in this paper, we propose uorex (uncertainty-aware open relation extraction) an approach that falls under the urd category and addresses the previously mentioned challenges. specifically, our approach is semi-supervised. we start with a set of labeled data containing known relations and our goal is to discover novel relations in the unlabeled data. we encode the contextual information of sentences and then project this encoded information into a smaller space. the adaptation of this space is achieved by optimizing a clustering loss that encourages labeled points to gather around their relational centers. as a result, unlabeled data points are separated based on their expressed relations. this separation is mainly related to the space learning the underlying concept behind relations. by taking advantage of this separation, we employ soft clustering to obtain a clustering distribution. this distribution captures an uncertainty that is epistemic in nature, arising from incomplete knowledge of relational patterns. we refer to this uncertainty as relational uncertainty and it is primarily linked to the number of known relations and is inversely proportional to the initial separation observed in the unlabeled data. more known relations result in less relational uncertainty and greater separation. on top of the encoded space, we train two clas-sifiers: one for discovering relations and another for refining these discoveries. the discoverer is directly guided by the reordered distribution resulting from soft clustering using kullback-leibler (kl) divergence, thus passing the relational uncertainty to the discoverer which mitigates overconfident predictions. the refinement of discoveries is then facilitated by the second classifier that is trained to classify both labeled and unlabeled data. this results in a model that understands the limits of its knowledge, thus enabling self-assessment of its performance using what we call overall confidence, a measure of average prediction confidence. additionally, by modeling relational uncertainty, our approach encourages the model to assign lower confidence to its incorrect predictions. this means that errors are more likely to appear among lowconfidence predictions, enhancing the efficiency of integrating our method into a generative active learning context. in summary, the contributions of this paper are summarized as follows: preliminary results demonstrate the effectiveness of our methodology in enhancing the overall reliability of openre. ## conclusion in this work, we introduce a novel and reliable openre approach uorex, designed to address the prevalent issue of overconfident errors in current state-of-the-art openre approaches caused by the use of pseudo-labels. uorex improves the reliability of openre, by modeling a relational uncertainty, that enables the model to understand the limits of its knowledge. as a result, uorex avoids confident mistakes, enables self-assessment of performance using overall confidence after training, and doubles the performance gain of openre models from oracles in a generative active learning context. experimental results on multiple datasets demonstrate that uorex not only enhances the overall reliability of openre but also achieves competitive performance. these advancements establish uorex as a solution for more dependable openre models. for future work, we intend to explore refining the uncertainty modeling using bayesian approaches for an even greater model reliability and extend the scope of the approach to work for low ressource languages. more importantly, we also plan to integrate the suggested approach within an integrated intelligent platform dedicated to support real-world applications for public sector agencies.