Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction (2023)

Abstract: Few-shot relation extraction involves identifying the type of relationship between two specific entities within a text, using a limited number of annotated samples. A variety of solutions to this problem have emerged by applying meta-learning and neural graph techniques which typically necessitate a training process for adaptation. Recently, the strategy of in-context learning has been demonstrating notable results without the need of training. Few studies have already utilized in-context learning for zero-shot information extraction. Unfortunately, the evidence for inference is either not considered or implicitly modeled during the construction of chain-of-thought prompts. In this paper, we propose a novel approach for few-shot relation extraction using large language models, named CoT-ER, chain-of-thought with explicit evidence reasoning. In particular, CoT-ER first induces large language models to generate evidences using task-specific and concept-level knowledge. Then these evidences are explicitly incorporated into chain-of-thought prompting for relation extraction. Experimental results demonstrate that our CoT-ER approach (with 0% training data) achieves competitive performance compared to the fully-supervised (with 100% training data) state-of-the-art approach on the FewRel1.0 and FewRel2.0 datasets.

## introduction relation extraction (re) aims at identifying the relation between two given entities based on contextual semantic information @xcite @xcite . however, the performance of re models often degrades significantly when the labeled data is insufficient. the few-shot relation extraction (fsre) task needs to be addressed with a limited amount of annotated training data @xcite @xcite . recently, numerous researchers have tackled this problem by employing meta-learning and neural graph techniques @xcite @xcite @xcite . these methods have achieved satisfying results by meta-training the model on a large dataset or incorporating external knowledge. more recently, pre-trained large language models (llms) such as gpt-series models, have exhibited significant in-context learning capabilities @xcite , achieving promising results across many nlp tasks. these findings suggest that llms can effectively perform various tasks without the need for parameter optimization, a concept known as in-context learning @xcite . within the paradigm of in-context learning, llms demonstrate competitive performance compared to standard fullysupervised methods across many nlp tasks, even with just a few examples provided as few-shot demonstrations in the prompt @xcite . furthermore, the chain-of-thought prompting method @xcite @xcite elicits an impressive reasoning capability from the llm in mathematical problems and commonsense reasoning. while in the re task, there may exist a reasoning process that guides the llm in determining the relation label. however, there is a lack of research to fill this gap. though gpt-re @xcite introduces a golden label-induced reasoning method by prompting the llm to generate a suitable reasoning process solely based on the given golden label. the performance improvement from the auto-generated reasoning process is marginal compared to a meticulously designed approach for few-shot demonstration retrieval. we argue that the one-step reasoning process generated by llm does not fully unleash the potential of llm: @xcite previous studies and our experiments indicate that the one-step auto-generated reasoning process by llm does not emphasize the higher-level abstraction of entity types, specifically the concept-level entities, which has been proven to be beneficial for fsre task @xcite . for instance, consider the following simple example: the relation between two location entities should not be categorized under the relation type between human beings. (2) due to the huge amount of pre-training data, the llm has already possessed a considerable knowledge base @xcite , which can be beneficial when the llm encounters an fsre task. (3) the quality of semantic representation of the relation label is not crucial in the fully-supervised setting, but in-context learning is sensitive to the relation label. for instance, given the relation @xmath0, child, sport} in fewrel 1.0 @xcite , relation labels "mother" and "child" would confuse the llm without appropriate prompt designing, as the primary distinction between these two relations is the positioning of the parent entity as either the head or tail entity. moreover, the word "sport" barely contains enough relation information for the llm to perform the re task. we call this issue the semantic ambiguity of relation labels. to this end, this paper presents a novel chainof-thought prompting method for the fsre task: chain-of-thought with explicit evidence reasoning, achieving competitive results compared to the state-of-the-art result on fewrel 1.0 and fewrel 2.0. our method employs a 3-step reasoning approach to address the aforementioned issues. in the first and second steps, cot-er requires the llm to output the concept-level entities corresponding to the head and tail entities, which serve as the foundation for re-specific reasoning. in the third step, cot-er prompts the llm to extract the relevant contextual spans as evidence that explicitly establishes a specific relationship between these two entities. by combining the head entity, tail entity, and relation label to form a coherent sentence, llms can determine the relation label between two given entities more semantically, addressing the issue of semantic ambiguity of relation labels in prompting methods. figure 1 demonstrates the difference between auto-cot and our cot-er. ## related work few-shot relation extraction. few-shot relation extraction aims at predicting semantic relations between head and tail entities indicated in a given instance based on a limited amount of annotated data. fewrel, a large-scale dataset introduced by @xcite , was the first to explore few-shot learning in relation extraction. many approaches @xcite @xcite incorporate external knowledge to improve performance given the scarcity of training data. another line of fsre research @xcite @xcite relies solely on the input text and provided relation description information, without incorporating external knowledge. most of the previous methods usually adopt complicated designs of neural networks or introduce external knowledge, which can be labor-intensive in realistic scenarios. in-context learning. gpt-3 in-context learning (icl) @xcite has emerged as a novel paradigm in nlp and demonstrates competitive performance across various tasks when compared to fine-tuned models. it's much easier to introduce prior knowledge into llms by incorporating relevant text information into prompt @xcite @xcite . furthermore, icl is a training-free approach by directly prompting the llms, which means it's a ready-to-use method and can be easily applied to various tasks with a few demonstrations in the prompt. recently, most researchers focus on the demonstration designing of icl to improve the performance in nlp tasks and gradually developed into two categories @xcite . the first line of demonstration designing tries to seek an optimal arrangement of the few-shot demonstrations in the prompt by selecting instances from the dataset @xcite @xcite and ordering the selected demonstration examples @xcite @xcite . another line of demonstration design aims to discover an effective prompting method to unleash the potential of llms. several studies @xcite find that the llms can generate the task instruction automatically. furthermore, @xcite revealed the reasoning ability of llm by adding intermediate reasoning steps manually before giving the answer, which is called chain-of-thought (cot). additionally, @xcite shows that by simply adding "let's think step by step" before each answer, llm can do the zero-shot reasoning without manually annotated data. based on this discovery, @xcite proposed auto-cot, replacing the manually written reasoning process in cot with the automatically generated reasoning process by llm. despite the cot prompting method achieving promising results in many nlp tasks, it still lacks relevant exploration for re. hence, in this paper, we propose a novel cot prompting method called cot-er to fill this gap. true few-shot learning. @xcite argues that prior research has achieved promising results by choosing prompts or tuning other hyperparameters using a large development set, such as selecting few-shot demonstrations from a large training set, which does not truly demonstrate the few-shot learning capability of llms. this setting has been adopted by many works @xcite @xcite to get a more accurate result of few-shot performance. we will also adopt the setting in this paper. ## conclusion in this paper, we explore the potential of llm incontext learning on few-shot relation extraction. to enhance the general performance caused by lowquality auto-generated reasoning processes, we introduce cot-er, a prompting method tailored to few-shot relation extraction. the core idea is to prompt llm to generate evidence using taskspecific and concept-level knowledge stored in its pre-training stage. these pieces of evidence will be utilized by llm during the re task, and facilitate the reasoning process. additionally, we devise a label verbalizing technique by integrating both entities and the relation label into a coherent expression. this technique addresses the semantic ambiguity of relation labels, which is a common challenge encountered during relation extraction when utilizing in-context learning. the experimental results on fewrel 1.0 and fewrel 2.0 outperform all training-free baselines, demonstrating the effectiveness of our proposed approach. moreover, achieving comparable results to the state-of-the-art fully-supervised method suggests that the paradigm of in-context learning holds promise as a novel solution for the few-shot relation extraction task.