When Phrases Meet Probabilities: Enabling Open Relation Extraction with Cooperating Large Language Models (2024)

Abstract: Current clustering-based open relation extraction (OpenRE) methods usually apply clustering algorithms on top of pre-trained language models. However, this practice has three drawbacks. First, embeddings from language models are high-dimensional and anisotropic, so using simple metrics to calculate distances between these embeddings may not accurately reflect the relational similarity. Second, there exists a gap between the pre-trained language models and downstream clustering for their different objective forms. Third, clustering with embeddings deviates from the primary aim of relation extraction, as it does not directly obtain relations. In this work, we propose a new idea for OpenRE in the era of LLMs, that is, extracting relational phrases and directly exploiting the knowledge in LLMs to assess the semantic similarity between phrases without relying on any additional metrics. Based on this idea, we developed a framework, oreLLM, that makes two LLMs work collaboratively to achieve clustering and address the above issues. Experimental results on different datasets show that oreLLM outperforms current baselines by 1.4\%\sim 3.13\% in terms of clustering accuracy.

## introduction relation extraction (re) focuses on extracting relations between entity pairs from texts. however, conventional re is based on the closed-set hypothesis @xcite @xcite for handling pre-defined relations, thereby limiting its capability to deal with new emerging relations in the real world. therefore, open relation extraction (openre), which aims to extract new relations from unlabeled open-domain corpora, has received more attention. one paradigm of openre is open information extraction (openie) (etzioni et al., 2008; fader * corresponding author et al., 2011), which directly extracts relation-related phrases from sentences based on syntactic analysis, but often produces redundant or ambiguous results. therefore, another paradigm, namely unsupervised relation discovery @xcite , has been proposed to bridge the gap between closed-set re and openie by inducing new relations automatically. specifically, it commonly formulates openre as a clustering task, where the most critical challenge lies in how to measure the relational similarity between unlabeled instances effectively for clustering. for this purpose, a popular practice (as shown in figure 1(a) ) is to first obtain embeddings from pre-trained language models such as bert @xcite , and then apply clustering algorithms, e.g., k-means with euclidean, to measure the distances of these embeddings and form clusters @xcite . although this practice benefits from the knowledge in pre-trained language models, it has three drawbacks. first, the embeddings from pre-trained language models encode complex linguistic information @xcite and are anisotropic @xcite . consequently, using simple metrics, e.g., euclidean or cosine, to calculate distances between these embeddings may not accurately reflect the relational similarity. second, this practice exists a gap between the pre-trained language models and downstream clustering for their different objective forms @xcite . in other words, the pre-training tasks of language models (i.e., masked language modeling or next token prediction) are not consistent with the downstream clustering task (i.e., classification with pseudo labels), which restricts the transfer and adaptation of knowledge in pre-trained language models to unsupervised clustering. third, the essence of these methods is to perform clustering with embeddings, which does not meet the original intention of re since the relations in each sentence are not actually extracted. while some methods @xcite @xcite alleviate the first issue by using labeled instances with pre-defined relations to guide similarity learning in new types, they still fail to address the other two drawbacks. recently, a series of powerful generative large language models (llms) have been proposed and brought revolutionary changes to the field of nlp @xcite @xcite . we believe that llms have the potential to solve the above problems for two reasons. for one thing, llms are trained on a large amount of general text corpora, and have excellent generalization in understanding and generating text @xcite . this enables them to analyze and produce relational phrases for unlabeled instances. for another thing, their generation strategy follows "next token prediction" @xcite , which predicts the next token by calculating the likelihood probability of this token conditioned on its preceding sequence. this probability indicates the relatedness degree of the predicted token to its entire prefix and provides a new way to measure the similarity between phrases. in this work, we propose a novel framework, namely orellm, for openre. as illustrated in figure 1 (b), the key of orellm is to adopt llms in two aspects: generating relational phrases for relation extraction and calculating conditional probabilities between phrases for similarity learning. specifically, orellm consists of an api-based llm and an open-source llm. the former serves as a phrase extractor, generating relational phrases from instances. the latter operates as a probability estimator, computing the likelihood that one phrase generates another, thereby reflecting the semantic similarity among the generated phrases. as a result, the probability estimator directly exploits the knowledge in llms to assess the semantic relatedness between phrases, without relying on any additional metrics. moreover, we propose a cooperative cycle to make these two llms work collaboratively for clustering. in this cycle, we first select the initial centers and assign each data point to the corresponding centers according to probabilities. we then sort the data in each cluster by the magnitude of probability scores to obtain reliable and ambiguous data. after that, we design a "rethink" strategy to correct the potentially erroneous phrases in ambiguous data and discover new centers. in addition, we adopt the reliable data to fine-tune the probability estimator. this process is consistent with the next token prediction pattern, thus alleviating the gap between pre-trained language models and the downstream clustering task. finally, we use the fine-tuned probability estimator to reassign labels and update clustering for the next cycle. to summarize, the main contributions of our work are as follows: (1) we propose a new idea for openre in the era of llms, that is, first extract relational phrases and then directly exploit the knowledge in llms to assess the semantic similarity between phrases, without relying on any additional metrics. (2) based on this idea, we propose a framework orellm that makes two llms work collaboratively for clustering. (3) we conduct experiments on two datasets and orellm achieves better performance than the existing state-of-theart methods. (4) additionally, orellm can output relational phrases for each instance, which really meets the original intention of re. ## conclusion in this work, we propose a new idea for openre in the era of llms: extracting relational phrases and directly exploiting the knowledge in llms to assess the semantic similarity between phrases without relying on any additional metrics. based on this idea, we develop a framework named orellm, which consists of two llms. one serves as a phrase extractor, while the other operates as a similarity probability estimator. these two llms work together through a cooperative cycle for clustering. experimental results show that orellm outperforms current baselines by 1.4% ∼ 3.13% in terms of clustering accuracy.