Guideline Learning for In-Context Information Extraction (2023)

Abstract: Large language models (LLMs) can perform a new task by merely conditioning on task instructions and a few input-output examples, without optimizing any parameters. This is called In-Context Learning (ICL). In-context Information Extraction (IE) has recently garnered attention in the research community. However, the performance of In-context IE generally lags behind the state-of-the-art supervised expert models. We highlight a key reason for this shortfall: underspecified task description. The limited-length context struggles to thoroughly express the intricate IE task instructions and various edge cases, leading to misalignment in task comprehension with humans. In this paper, we propose a Guideline Learning (GL) framework for In-context IE which reflectively learns and follows guidelines. During the learning phrase, GL automatically synthesizes a set of guidelines based on a few error cases, and during inference, GL retrieves helpful guidelines for better ICL. Moreover, we propose a self-consistency-based active learning method to enhance the efficiency of GL. Experiments on event extraction and relation extraction show that GL can significantly improve the performance of in-context IE.

## introduction information extraction (ie), whose primary goal is to extract structured information from unstructured plain text, serves as a critical foundation for numerous downstream tasks such as question answering and knowledge base construction @xcite . ie tasks typically have complex task settings due to their requirement of translating diverse real-world facts into a few predefined classes. this often necessitates a large number of rules and examples to thoroughly and accurately define the target concept of the task. for example, the guidelines for ace relation extraction extend over 33 pages @xcite . in the past, the supervised learning paradigm has been applied to fine-tune numerous parameters on massive data to accurately learn the concept @xcite . this approach, while effective, is data-intensive, hard to train, and difficult to update. recently, however, the nlp community witnesses the rapid rise of large language models (llms), such as palm @xcite , @xcite and llama @xcite . these llms have achieved great performance on a wide range of nlp tasks with their superior language understanding power, but finetuning them faces closed-source and high-trainingcost issues. in-context learning (icl) @xcite , a characteristic feature of llms, offers a solution to harness the power of llms while sidestepping these issues. icl enables llms to perform new tasks without tuning any parameters. instead, they are given only the task instruction and a few input-output examples as the prompt. it achieves promising performance on many tasks like natural language inference and sentiment classification @xcite , demonstrating a new paradigm in the nlp community. several recent studies have explored the icl paradigm for ie @xcite . impressively, by merely providing task instructions and a handful of in-context examples, llms can achieve significant performance on many ie tasks. however, they still lag behind supervised sota models @xcite . we underline one primary reason for the suboptimal performance: underspecified task description. as discussed earlier, the target concept of ie is inherently complex. but the input context utilized for elucidating the target concept to llms is constrained by its limited length. consequently, the comprehended concept by llms might deviate from the target concept. an example of this is illustrated in figure 1 . in the sentence "the shipments have arrived into the stock", the pre-defined relation types content-container and entity-destination presents a grey area concerning the relation between the entities "shipments" and "stock". the target concept is embodied in a rule in the annotation guidelines 1 -"motion verbs prevailing over stative relations" -which is misaligned with the llm's comprehended concept. this paper attempts to mitigate this problem by introducing a guideline learning (gl) framework. this framework replicates the human annotation process, which first gathers annotation guidelines, and then annotates accordingly. specifically, it has two phrases. in the learning phase, a set of guidelines are iteratively learned from scratch based on a few labeled instances. a guideline here is a natural language rule derived by integrating the appropriate extrapolation of an error instance and its true label. this is different from previous supervised learning methods, which learn a set of model parameters. in the inference phase, given a new instance, it retrieves relevant rules from the guideline to compose a prompt, which includes the task instruction, the retrieved rules, a few examples, and the input instance. it then asks an llm agent to finish the task given the prompt. this failure-driven remind- furthermore, we incorporate a self-consistencybased active learning method to enhance the efficiency of label utilization. we also propose a "generalizer" to assist in the generation and retrieval of guidelines. finally, we conduct in-depth experiments on two representative ie tasks: (1) event extraction on financial documents, and (2) relation extraction on general domain resources, which both feature relatively complex target concepts. experimental results indicate that the use of 50 labeled samples per class can greatly boost the performance of icl in both tasks. ## task and implementation initially, we implement the guideline learning framework for two information extraction tasks: event extraction (sec 3.1) and relation extraction (sec 3.2). we choose these tasks because the target concepts of these tasks are relatively complex. ## experiments we conduct experiments foot_1 to demonstrate the effectiveness of the gl framework on event extraction (sec 4.1) and relation extraction (sec 4.2). in the last section, we analyze the quality of learned guidelines and conduct case studies (sec 4.3). metrics we use role-level micro precision, recall, and f1 for evaluation, following previous work @xcite . method though only working on etc, we also provide simple solutions for the other two subtasks for comparison with other methods. specifically, for eti, as all event types are related to equity transaction, we identify text spans with the format "{number} shares" as candidate triggers via string matching. for etc, we apply guideline learning framework and conduct binary classifications for each event type. as the documents in this dataset are long, we apply an extra llm agent to generate a description for each trigger about its meaning according to the document. we use the generated description as the input instance to conduct the classification. for eae, we apply an llm agent to generate an event table in the markdown format given predicted event triggers. compared models (1) redee @xcite and de-ppn @xcite : two supervised methods. we reproduce de-ppn on the entire dataset strictly following the official code. redee runs out of memory on 12g gpu so we do not reproduce it. (2) ee-icl: prompt the llm to directly output the event table without predicting event triggers. (3) ee-gl-b: baseline version of our guideline learning method with empty guidelines. (4) ee-gl-r: our guideline learning method. we randomly sample 50 documents from the training set and annotate event triggers. (5) ee-gl-a: we actively select 50 documents out of 400 we use the same human-annotated demonstrations for all ee methods. ## conclusion this paper explores the underspecified task description problem in in-context information extraction. we propose a guideline learning framework to alleviate the problem, which automatically learns guidelines from few labeled instances during the learning phrase, and retrieving helpful guidelines to assist in reasoning during inference. our experiments on event and relation extraction show that a straightforward implementation of guideline learning can enhance vanilla in-context learning by approximately 4%.