Relation-Aware Prompting Makes Large Language Models Effective Zero-shot Relation Extractors (2025)

Abstract: While supervised relation extraction (RE) models have considerably advanced the state-of-the-art, they often perform poorly in low-resource settings. Zero-shot RE is vital when annotations are not available either due to costs or time constraints. As a result, zero-shot RE has garnered interest in the research community. With the advent of large language models (LLMs) many approaches have been proposed for prompting LLMs for RE, but these methods often either rely on an accompanying small language model (e.g., for finetuning on synthetic data generated by LLMs) or require complex post-prompt processing. In this paper, we propose an effective prompt-based method that does not require any additional resources. Instead, we use an LLM to perform a two-step process. In the first step, we perform a targeted summarization of the text with respect to the underlying relation, reduce the applicable label space, and synthesize examples. Then, we combine the products of these processes with other elements into a final prompt. We evaluate our approach with various LLMs on four real-world RE datasets. Our evaluation shows that our method outperforms the previous state-of-the-art zero-shot methods by a large margin. This work can also be considered as a new strong baseline for zero-shot RE that is compatible with any LLM.

## introduction relation extraction (re) aims to identify semantic relations between two entities from unstructured text. with the recent advances in large language models (llms), studies show that llms perform well in various downstream tasks without any training or fine-tuning. but it is unclear whether they are effective for zero-shot re. a recent line of research shows that such zero-shot approaches for relation extraction are ineffective or continue to lag behind supervised methods @xcite @xcite @xcite @xcite . however, other line of llm-based research reports results comparable or outperforming state-of-the-art. these methods fall into three groups. in a first group, these methods use llms to some extent, but eventually rely on fine-tuning a small language model @xcite @xcite . for instance, an llm is used to generate/augment synthetic data and then a model from bert family is fine-tuned on the generated data. the second group fine-tunes a large language model @xcite @xcite . fine-tuning llms requires specialized hardware, significant compute resources, and is expensive. the third group does not require fine-tuning but requires complex postprompt computations, e.g., @xcite performs a complex computation on the llm answers using an uncertainty-based active learning method to estimate output probabilities of the llm. a few other methods, e.g., @xcite , that do not belong to the above groups are evaluated on limited benchmarks. therefore, it is unclear whether prompt based zero-shot re is effective without any finetuning or complex post-prompt computations. 2 in this work, we present an effective promptbased method to re that does not require either fine-tuning or complex computations. our approach only requires an api access to an llm. this simplifies the zero-shot re process and makes it more accessible and faster to deploy which is important for developing zero-shot systems. we achieve this by a novel prompt-based method we call relation-aware prompting. formally: we evaluate our approach using various llms on four real-world and challenging relation extraction datasets. the evaluation shows that our method outperforms the previous state-of-the-art zero-shot methods by a large margin. we also perform an ablation study where we investigate the effectiveness and usefulness of our prompt elements that will demonstrate the effectiveness of the proposed method. this work can also be considered as a new strong baseline for relation extraction. any llm-based work in re (such as finetuning llms, or other methods) can use our method as a strong baseline for evaluating their respective approach. ## problem statement in the re task, the goal is to classify a sentence containing two marked entities (a head and a tail) into a set of predefined relations, or determine that none of the relations apply (referred to as none-ofthe-above or nota). this work focuses on zeroshot re where no re training data is provided to models prior to inference time. ## methodology our method is a two-step process. in the first step, (a) relations between the head and tail entities in instances are summarized; (b) the applicable label space is reduced; and (c) examples are synthesized. in the second step, the results of the first step are combined with other prompting elements into a final comprehensive prompt. the following subsections describe each step. figure 1 demonstrates an overview of our approach. ## conclusion in this work, we present relation-aware prompting, an effective prompt-based method for zeroshot relation extraction. we propose targeted summarization of instances with respect to the underlying relations to bring out the relations in the texts, reducing the applicable relations through a method inspired by the process of elimination, synthesizing examples using subject-verb-object structure, and other prompting elements. we evaluate our approach on four re datasets. our approach significantly outperforms current zero-shot llm promptbased methods. our approach can also be considered as a new strong baseline for zero-shot re that is compatible with any llm.