Classify First, and Then Extract: Prompt Chaining Technique for Information Extraction (2024)

Abstract: This work presents a new task-aware prompt design and example retrieval approach for information extraction (IE) using a prompt chaining technique. Our approach divides IE tasks into two steps: (1) text classification to understand what information (e.g., entity or event types) is contained in the underlying text and (2) information extraction for the identified types. Initially, we use a large language model (LLM) in a few-shot setting to classify the contained information. The classification output is used to select the relevant prompt and retrieve the examples relevant to the input text. Finally, we ask a LLM to do the information extraction with the generated prompt. By evaluating our approach on legal IE tasks with two different LLMs, we demonstrate that the prompt chaining technique improves the LLM’s overall performance in a few-shot setting when compared to the baseline in which examples from all possible classes are included in the prompt. Our approach can be used in a low-resource setting as it does not require a large amount of training data. Also, it can be easily adapted to many different IE tasks by simply adjusting the prompts. Lastly, it provides a cost benefit by reducing the number of tokens in the prompt.

## introduction this work introduces a new prompt chaining technique for information extraction (ie) in the incontext learning (icl) setting. since the large language model (llm)'s capability of handling various tasks in a few-shot setting has been demonstrated @xcite , many researchers have investigated using llms in the icl setting. a key challenge in this research area is example retrieval. retrieving good examples for the prompt improves the performance of llms in the icl setting @xcite . different approaches have been made to retrieve good examples, but they rely mostly on semantic similarity with the underlying text. however, semantic similarity-based example retrieval does not guarantee good example quality. as @xcite indicates, there are cases where task-aware example retrieval works better. for example, when working on a ie task from a domain-specific document in which many sentences share high semantic similarity yet contain different types of information, retrieving examples based on the information type contained in each sentence is a better option than using a semantic similarity-based approach. there are a few works that present task-aware example retrieval techniques @xcite . however, the techniques are not easily adaptable because they require training or finetuning a model. they also focus on addressing specific tasks rather than general ie tasks. our approach with the prompt chaining technique provides an alternative to these methods as it does not involve any training nor fine-tuning. also, it can be easily adapted to various ie tasks by simply adjusting prompts. the main idea of our approach is to split the ie tasks into two steps: (1) text classification and (2) information extraction. in the text classification step, an input text is classified based on the information contained in it. we prompt a llm to do the text classification in a few-shot setting. the output from this step is used to retrieve examples of the relevant type(s) that are relevant to the input text. with the retrieved examples, the prompt for the information extraction is generated. lastly, we ask a llm to do the information extraction using the generated prompt. the main contributions of this work are: in the out-of-domain dataset, it also outperforms the baseline for entity extraction (56.89% vs. 54.97%) and event extraction (39.44% vs. 38.08%). gpt-4o mini shows a similar trend, with the prompt chaining boosting entity extraction by 1.52 percentage points (77.05% vs. 75.53%) and event extraction by 2.37 percentage points (70.67% vs. 68.30%) in-domain. out-of-domain, it improves entity extraction by 0.71 percentage points (58.82% vs. 58.11%) and event extraction by 7.09 percentage points (42.13% vs. 35.04%). ## method we introduce a new information extraction approach using prompt chaining. prompt chaining is a concept introduced by @xcite . it is a method that divides a complex task into multiple smaller steps and prompts an llm in each step; the output from an earlier prompt becomes an input for the following prompt. @xcite suggests that using a dynamic prompt containing the reduced number of examples relevant to each input text improves extraction accuracy. their method for generating dynamic prompts involves training a bert classifier. unlike them, we investigate using the prompt chaining technique to generate a dynamic prompt without training or fine-tuning a model. we divide the ie task into two steps: (1) text classification, to understand which types are likely to be present in the underlying text, and (2) ie using prompts just for the likely types. both steps are implemented using icl and a vanilla llm (i.e., not fine-tuned for the task). figure 1 depicts our overall approach. ## task and evaluation we evaluate our approach on the entity and event extraction tasks from the legal will dataset introduced by @xcite 5 . we chose to do our evaluation on this legal task because wills contain highly diverse types of entities and events, and because the number of entity and event types is relatively large (see next subsection). extracting many types of entities and events using the llm in a standard few-shot setting can be challenging because the prompt should provide examples for all the entity and event types. these characteristics of the legal domain make it a good candidate for our approach, which focuses on (1) selecting examples with the same information type without depending on semantic similarity and (2) including only selected examples in the prompt. ## conclusion this work introduces a new prompt chaining technique for information extraction. the key idea of this approach is to split the information extraction into two steps: (1) text classification to understand which entity/event types are likely to be present, and (2) information extraction for the identified types. both steps are implemented using an llm with in-context learning. by classifying each input text based on the information type present in it first, we can complete the prompt for the information extraction task with the examples that are relevant to each input text. with the completed prompt, we ask a llm to conduct the information extraction task. we evaluate this technique on entity and event extraction tasks in the legal domain. the evaluation results demonstrate that the prompt chaining technique improves the model's overall performance. the prompt chaining approach also provides cost benefits by reducing the number of tokens in the prompt. the code used in this work can be found at: https://github.com/ml4ai/pc4wills/