APE : Active Learning-based Tooling for Finding Informative Few-shot Examples for LLM -based Entity Matching (2024)

Abstract: Prompt engineering is an iterative procedure that often requires extensive manual effort to formulate suitable instructions for effectively directing large language models (LLMs) in specific tasks. Incorporating few-shot examples is a vital and effective approach to provide LLMs with precise instructions, leading to improved LLM performance. Nonetheless, identifying the most informative demonstrations for LLMs is labor-intensive, frequently entailing sifting through an extensive search space. In this demonstration, we showcase a human-in-the-loop tool called ool (Active Prompt Engineering) designed for refining prompts through active learning. Drawing inspiration from active learning, ool iteratively selects the most ambiguous examples for human feedback, which will be transformed into few-shot examples within the prompt.

## introduction prompt engineering typically serves as the initial step when developing llm-based applications because it is a relatively fast process and requires fewer technical skills than fine-tuning. prompt engineering involves crafting and optimization of instructions provided to llms. these prompts need to be carefully designed to direct the behavior of llms towards performing specific tasks or generating desired outcomes @xcite . while llms (e.g., chatgpt and gpt-4) show impressive capabilities for zero-shot tasks without prior training, their performance can be further enhanced by integrating clear and informative few-shot demonstrations alongside the prompts @xcite . these demonstrations not only guide the llms but prompt engineering is a dynamic and iterative process that typically consists of the following stages: (1) task description: clearly outline the intended task for llms, (2) few-shot demonstration: provide a small number of concrete and helpful demonstrations to illustrate the precise semantics of the task, (3) task input and completion request: present the actual task input and request an llm completion. for all three steps, minor prompt rephrasing is typically needed, but this task is relatively light and does not require many iterations. however, choosing informative few-shot demonstrations can be a labor-intensive and timeconsuming process due to the large search space of the problem. for instance, to identify only 3 demonstration examples out of 100 examples, there are 970,200 (i.e., 100×99×98) different combinations, a daunting manual task. identifying representative and ambiguous examples to enhance the performance of machine learning models is a well-established subject within the active learning community. we can view the few-shot example identification as an active learning problem, where the goal is to find the most informative examples to be included in the prompt to help improve llms' performance. recently, @xcite proposed the idea of using various active learning sampling strategy to identify few-shot examples prompt engineering. our work follows the same direction with the main focus being building an interactive tool (with an intuitive user interface) that identifies the most informative fewshot examples through simple human interaction. in this paper, we present ape (active prompt engineering), an intuitive and intelligent prompt engineering tool that iteratively identifies the most informative and ambiguous examples for which a given llm will likely make a mistake, and then provide them in the prompt as few-shot examples after seeking human annotation. unlike @xcite , which focuses on the backend algorithm services, our goal is to hide the technical details by a carefully designed graph user interface so that we can have a usable tool that truly harnesses the power of active learning. ## methodology the main goal of ape is to identify a handful of informative few-shot examples that can boost an llm's performance. as an active learning tool, it follows the iterative procedure outlined in figure 1 , involving interaction with both a human user and the llm api for prompt engineering. the best way to understand ape end to end is to watch the video demo of the tool (see the link in the abstract). at a high level, in each iteration, we start with sampling informative examples based on the prompt of the current iteration, which includes applying a user-configured sampling strategy to let the llm choose the ambiguous examples from the user-provided sampling pool. next, users annotate the selected examples, potentially including explanations for chain-of-thought-style prompting. these newly annotated examples are then used to update the prompt. lastly, the new prompt is evaluated against evaluation data to report its performance. the core of the active learning process is the sampling strategy; for simplicity, we will use entity matching, a classic binary classification task, to illustrate the sampling methodology behind ape. given a set @xmath0, . . . , p m } of entity pairs, where p i consists of a pair ⟨e i ## concluding remarks due to limited space, we focus on the tooling aspect of ape in this demo paper, and are currently working on a research paper that will provide a comprehensive description of the system design, theoretical foundation underlying this optimization problem, and experimental evaluations.