Revisiting Large Language Models as Zero-shot Relation Extractors (2023)

Abstract: Relation extraction (RE) consistently involves a certain degree of labeled or unlabeled data even if under zero-shot setting. Recent studies have shown that large language models (LLMs) transfer well to new tasks out-of-the-box simply given a natural language prompt, which provides the possibility of extracting relations from text without any data and parameter tuning. This work focuses on the study of exploring LLMs, such as ChatGPT, as zero-shot relation extractors. On the one hand, we analyze the drawbacks of existing RE prompts and attempt to incorporate recent prompt techniques such as chain-of-thought (CoT) to improve zero-shot RE. We propose the summarize-and-ask (SumAsk) prompting, a simple prompt recursively using LLMs to transform RE inputs to the effective question answering (QA) format. On the other hand, we conduct comprehensive experiments on various benchmarks and settings to investigate the capabilities of LLMs on zero-shot RE. Specifically, we have the following findings: (i) SumAsk consistently and significantly improves LLMs performance on different model sizes, benchmarks and settings; (ii) Zero-shot prompting with ChatGPT achieves competitive or superior results compared with zero-shot and fully supervised methods; (iii) LLMs deliver promising performance in extracting overlapping relations; (iv) The performance varies greatly regarding different relations. Different from small language models, LLMs are effective in handling challenge none-of-the-above (NoTA) relation.

## introduction relation extraction (re) aims to identify the relationships between entities in texts, and plays an important role in information extraction @xcite . most of existing re methods @xcite require large amounts of labeled training data which is labor-intensive and timeconsuming in practice. hence, extracting relations from texts using zero or few-shot methodologies has garnered significant scholarly attention @xcite . recent studies @xcite on large-scale pre-trained language models (llms), such as gpt-3 @xcite , demonstrate that llms perform well in various downstream tasks without any training or finetuning but only with a few examples as instructions, which is called in-context learning. however, there is currently no consensus on whether llms are good few-shot information extractors @xcite . different with some other tasks, re is more challenging for llms because the structured data containing multiple dependent elements are difficult to extract directly and accurately. although recent studies @xcite indicate that some conventional fine-tuning models still outperform llms in few-shot re tasks, we still want to explore whether llms can achieve competitive performance compared to fine-tuning models. similar to few-shot learning, llms also show promising performance on zero-shot settings @xcite . recent work for zero-shot re via prompting llms has achieved remarkable progress. qa4re @xcite is a multiplechoice question answering prompt format, in which each relation is transformed into a template and llms are expected to predict only a single letter. this prompt is simple but requires manually crafted templates and unable to deal with overlapping relations, which motivates us to find more general and effective prompts. chatie @xcite transforms the zero-shot ie task into a multiturn question answering problem with a two-stage framework, even surpasses some full shot models on several datasets. nonetheless, chatie still performs worse than the state-of-the-art and is only evaluated on limited benchmarks. thus it is still unclear how to improve extracting performance by designing effective prompts and whether llms are good zero-shot relation extractors. to this end, we revisit and investigate the potential of llms in zero-shot re on the following research questions: previous work @xcite fails to achieve promising results on re since black box llms such as chatgpt are difficult to ensure the reliability of outputs. to answer the first question, we investigate the feasibility of incorporating recent prompt techniques to improve the reliability of extracted results. for example, chain-of-thought (cot) prompting @xcite improves the reliability of model output by providing intermediate reasoning steps. active prompting @xcite is an uncertaintybased active learning method to quantify the uncertainty so as to select the most uncertain outputs. specifically, we propose the summarize-and-ask (sumask) prompting, which decomposed re into two subtasks: text summarization @xcite and question answering @xcite . we further introduce an uncertainty estimation method to approximately characterize output probabilities of llms, which yields substantial improvements compared to vanilla prompting. to answer the last two questions, we evaluate llms on both zero-shot relation classification and overlapping relation extraction. and six re benchmarks are used for evaluation: (1) fewrel @xcite and wiki-zsl @xcite for comparision with zero-shot re methods; (2) tacred @xcite , tacrev @xcite and re-tacred @xcite for comparision with fully supervised methods; (3) nyt @xcite for evaluating overlapping relation extraction. experimental results demonstrate that llms achieve promising results in all experimental settings. in summary, the contributions of this work are three-fold: llms achieve competitive or superior results compared with state-of-the-art zero-shot and fully supervised methods. notably, chatgpt with sumask prompting outperforms the state-of-the-art fully supervised method on tacred by an average of 2.8% micro-f1. few and zero-shot relation extraction fewshot re @xcite aims to predict novel relations by exploring a few labeled instances. prototypical networks @xcite are widely used and combined with pre-trained language models @xcite in few-shot settings to achieve impressive results. to be capable of extracting relations that were not specified in advance, zero-shot re @xcite is proposed to invent new models to predict new relations. however, existing zero-shot methods @xcite still requires much labeled data. recent studies @xcite leverage the llms with zero-shot prompting to extract relations from texts without any labeled samples in advance. but it is still unclear whether llms are good zero-shot relation extractors by carefully designed prompts. thus this work aims to investigate the capabilities of llms in zero-shot re. large language models and prompting besides the "pre-train and fine-tune" paradigm @xcite , pre-trained llms possess characteristics that are advantageous for few-shot @xcite and zero-shot @xcite learning, whereby appropriate prompts are used to effectively guide the model towards generating desired task outputs, thus beginning an era of "pre-train and prompt" @xcite . prior works @xcite note the sensitivity of prompting under slight modifications. empirical results demonstrate that answering the restrictive prompts is challenging due to biases acquired during pre-training @xcite . in this study, we evaluate different prompt formats tailored to the particularities of re and propose sumask prompting which outperforms vanilla prompting by a large margin. ## problem definition previous zero-shot re (chen and li, 2021) only involves single relation classification. we extend this zero-shot setting to multiple entities and relations. given the pre-defined relation set @xmath0, r 2 , ..., r n } and the sentence s containing the entity set @xmath1, e 2 , ..., e m }, we aim to extract all the relations between these entities composing the relational triples set @xmath2, r k , e j )}, where n denotes the number of relations, m represents the number of entities, and e i , e @xmath3, r @xmath4. ## conclusion this work provides a comprehensive study on zeroshot re with prompting-based llms. besides the vanilla prompting, we introduce a novel sumask prompting to fully explore the power of llms. our experiments on six benchmarks demonstrate the capability of llms in zero-shot re. furthermore, we are able to answer the three aforementioned questions. recent prompt techniques such as cot significantly improve zero-shot re prompting. properly instructed llms not only deliver competitive or superior results compared to state-of-the-art relation classification models, but also are promising for zero-shot overlapping re.