GPT - RE : In-context Learning for Relation Extraction using Large Language Models (2023)

Abstract: In spite of the potential for ground-breaking achievements offered by large language models (LLMs) (e.g., GPT-3) via in-context learning (ICL), they still lag significantly behind fully-supervised baselines (e.g., fine-tuned BERT) in relation extraction (RE). This is due to the two major shortcomings of ICL for RE: (1) low relevance regarding entity and relation in existing sentence-level demonstration retrieval approaches for ICL; and (2) the lack of explaining input-label mappings of demonstrations leading to poor ICL effectiveness. In this paper, we propose GPT-RE to successfully address the aforementioned issues by (1) incorporating task-aware representations in demonstration retrieval; and (2) enriching the demonstrations with gold label-induced reasoning logic. We evaluate GPT-RE on four widely-used RE datasets, and observe that GPT-RE achieves improvements over not only existing GPT-3 baselines, but also fully-supervised baselines as in Figure 1. Specifically, GPT-RE achieves SOTA performances on the Semeval and SciERC datasets, and competitive performances on the TACRED and ACE05 datasets. Additionally, a critical issue of LLMs revealed by previous work, the strong inclination to wrongly classify NULL examples into other pre-defined labels, is substantially alleviated by our method. We show an empirical analysis.

## introduction the emergence of large language models (llms) such as gpt-3 @xcite @xcite @xcite represents a significant advancement in natural language processing (nlp). instead of following a pretraining-and-finetuning pipeline @xcite @xcite @xcite , which finetunes a pre-trained model on 1 codes will be released after the anonymous period. a task-specific dataset in a fully-supervised manner, llms employ a new paradigm known as incontext learning (icl) @xcite which formulates an nlp task under the paradigm of language generation and makes predictions by learning from a few demonstrations. under the framework of icl, llms achieve remarkable performance rivaling previous fullysupervised methods even with only a limited number of demonstrations provided in various tasks such as solving math problems, commonsense reasoning, text classification, fact retrieval, natural language inference, and semantic parsing @xcite @xcite @xcite . despite the overall promising performance of llms, the utilization of icl for relation extraction (re) is still suboptimal. re is the central task for knowledge retrieval requiring a deep understanding of natural language, which seeks to identify a predefined relation between a specific entity pair mentioned in the input sentence or null if no relation is found. given a test input, icl for re prompts the input of llms with the task instruction, a few demonstrations retrieved from the training data, and the test input itself. then llms generate the corresponding relation. recent research (gutiérrez et al., 2022) has sought to apply gpt-3 icl to biomedical re, but the results are relatively negative and suggest that gpt-3 icl still significantly underperforms fine-tuned models. the reasons that cause the pitfall of gpt-3 icl in re are two folds: (1) the low relevance regarding entity and relation in the retrieved demonstrations for icl. demonstrations are selected randomly or via k-nearest neighbor (knn) search based on sentence embedding @xcite . regrettably, knn-retrieval based on sentence embedding is more concerned with the relevance of the overall sentence semantics and not as much with the specific entities and relations it contains, which leads to low-quality demonstrations. as shown in figure 2 , the test input retrieves a semantically similar sentence but is not desired in terms of entities and relations. to this end, we propose gpt-re for the re task. gpt-re employs two strategies to resolve the issues above: (1) task-aware retrieval and (2) gold label-induced reasoning. relations. both methods contain more re-specific information than sentence semantics, thus effectively addressing the problem of low relevance. for (2) gold label-induced reasoning, we propose to inject the reasoning logic into the demonstration to provide more evidence to align an input and the label, a strategy akin to the chain-of-thought (cot) research @xcite @xcite . but different from previous work, we allow llms to elicit the reasoning process to explain not only why a given sentence should be classified under a particular label but also why a null example should not be assigned to any of the pre-defined categories. this process significantly improves the ability of llms to align the relations with diverse expression forms. recent work reveals another crucial problem named "overpredicting" as shown in figure 3 : we observe that llms have the strong inclination to wrongly classify null examples into other predefined labels . a similar phenomenon has also been observed in other tasks such as ner @xcite . in this paper, we show that this issue can be alleviated if the representations for retrieval can be supervised with the whole set of null in the training data. we evaluate our proposed method on three popular general domain re datasets: semeval 2010 task 8, tacred and ace05, and one scientific domain dataset scierc. we observe that gpt-re achieves improvements over not only existing gpt-3 baselines, but also fully-supervised baselines. specifically, gpt-re achieves sota performances on the semeval and scierc datasets, and competitive performances on the tacred and ace05 datasets. ## related work in-context learning recent work shows that icl of gpt-3 @xcite can perform numerous tasks when provided a few examples in a natural language prompt. existing work focuses on various aspects to effectively utilize the advantages of gpt-3, from prompt design @xcite for proper input to coherence calibration @xcite for tackling the diverse generated output. another research path locates in the demonstration part, including ordered prompts @xcite and retrieval-based demonstrations @xcite @xcite . to the best of our knowledge, there is no previous work exploring the potential of gpt-3 on general domain re tasks. a recent work attempts to leverage gpt-3 in biomedical information extraction (ner and re), and reveals issues of icl that may be detrimental to ie tasks in general. our work succeeds in overcoming these issues to some extent and confirms the potential of gpt-3 in both general and the scientific domain re. retrieval-based demonstrations several studies have demonstrated that dynamically selecting few-shot demonstrations for each test example, instead of utilizing a fixed set, leads to significant improvement in gpt-3 icl @xcite @xcite . they also show that nearest neighbor in-context examples yield much better results than the farthest ones. this leads to the significance of better retrieval modules for demonstrations. existing attempts rely on sentence embedding in retrieval, including the sentence encoders of plms such as bert @xcite kate @xcite , simcse @xcite , sentence-bert (reimers and gurevych, 2019; @xcite . unlike these sentence embeddings, we propose to fine-tune plms on our target re tasks to produce more task-specific and robust representations for retrieval. ## conclusions this work explores the potential of gpt-3 icl on re for bridging the performance gap to the fine-tuning baselines via two strategies: (1) taskaware demonstration retrieval emphasizes entity and relation information for improving the accuracy of searching demonstrations; (2) gold labelinduced reasoning enriches the reasoning evidence of each demonstration. to the best of our knowledge, gpt-re is the first gpt-3 icl research that significantly outperforms the fine-tuning baseline on three datasets and achieves sota on semeval and scierc. we implement detailed studies to explore how gpt-3 overcomes the difficulties such as null example influence.