RETAIN : Interactive Tool for Regression Testing Guided LLM Migration (2024)

Abstract: Large Language Models (LLMs) are increasingly integrated into diverse applications. The rapid evolution of LLMs presents opportunities for developers to enhance applications continuously. However, this constant adaptation can also lead to performance regressions during model migrations. While several interactive tools have been proposed to streamline the complexity of prompt engineering, few address the specific requirements of regression testing for LLM Migrations. To bridge this gap, we introduce RETAIN (REgression Testing guided LLM migrAtIoN), a tool designed explicitly for regression testing in LLM Migrations. RETAIN comprises two key components: an interactive interface tailored to regression testing needs during LLM migrations, and an error discovery module that facilitates understanding of differences in model behaviors. The error discovery module generates textual descriptions of various errors or differences between model outputs, providing actionable insights for prompt refinement. Our automatic evaluation and empirical user studies demonstrate that RETAIN, when compared to manual evaluation, enabled participants to identify twice as many errors, facilitated experimentation with 75% more prompts, and achieves 12% higher metric scores in a given time frame.

## introduction large language models (llms) have demonstrated proficiency in executing a wide array of complex tasks @xcite , which previously necessitated custom fine-tuned models. this capability has made the integration of llms into applications increasingly attractive, as it significantly reduces the costs associated with developing models from scratch. however, for llms to effectively perform these complex figure 1 : regression testing for prompting llms. the process involves: (1) input dataset, (2) initial prompt, (3) data slicing algorithm to identify behavioral differences (regressions) across models, and (4) prompt refinement to address identified regressions. the rapidly evolving landscape of llms, compels application developers to continually update to newer versions to maintain optimal performance. moreover, applications utilizing llm apis often face forced transitions as older models are deprecated and discontinued foot_0 . this creates a recurring challenge of re-engineering prompts for different llms to achieve the same task and maintain consistent model behavior, a process we define as llm migration. migrations to newer llms are difficult due to model regressions @xcite , necessitating the development of custom tools for analyzing discrepancies in model behaviors. such regression tests must focus both on pattern discovery for errors and systematic failure validation @xcite . these patterns can generally be encoded as a subgroup or "slice" of model outputs, with a corresponding metric that characterizes the observed behavior, and are often discovered in an iterative and manual manner by prompt developers @xcite . figure 1 illustrates a high-level regression testing process for prompting, drawing parallels with software engineering techniques. the main challenge in regression testing based prompting, is to design a systematic method of identifying regressions. while numerous tools and frameworks have been developed to assist in prompt engineering, ranging from interactive platforms @xcite @xcite to automated systems @xcite , few address the specific needs of regression testing in prompting. existing tools often lack support for data slicing (figure 1 ), which requires manual inspection to identify regressions and group data points into slices. furthermore, current tools provide insufficient support for analyzing model behaviors at various granularities. to bridge this gap, we propose retain (regression testing guided llm migration) -designed explicitly for regression testing in prompting and enables flexible analysis of model behaviors at various granularities. retain aims to reduce the effort required in identifying regressions by automatically detecting differences in model behaviors across different data subsets ( §4.4). our tool features an interactive interface supporting the analysis of various prompt iterations across multiple granularity levels: aggregate metric scores, distribution analysis of metric scores, and sideby-side comparisons at the instance level ( §4.3). furthermore, retain integrates prompt updating capabilities, making it a self-contained solution for the entire prompting process. through user studies, we demonstrate that retain, compared to manual prompting approaches, aids users in identifying twice as many errors, facilitates iteration over 75% more prompts, and achieves 12% higher metric scores in a given time frame. ## system in this section, we demonstrate retain using a scenario where a researcher or engineer utilizes our tool for llm migration in the task of prompt migration @xcite for a summarization task @xcite . the user is migrating a prompt optimized for gpt3.5-turbo-16k to llama-3-8b. it's important to note that retain is versatile and applicable to any prompt engineering setup. the user initiates the process by creating a simple declarative configuration file (detailed in appendix §a). this file contains essential information such as model names, access keys, initial prompts, metrics, and test data @xcite . with this configuration in place, the user can launch the retain tool. for implementation details, readers are directed to appendix a. ## evaluations to evaluate our system comprehensively, we employ two approaches: (1) an automatic evaluation ( §5.1) to assess the accuracy of our llm-based error discovery method in detecting distributional differences between model outputs, and (2) a user study ( §5.2) to compare retain's impact on the prompt migration process against current practices. ## conclusion we present retain-a tool for regression testing guided llm migration. retain comprises of an interactive prompting interface tailored to 306 regression testing needs, and an error discovery module that facilitates understanding differences in model outputs. the tools aims to help users in understanding where and why models score poorly on given metrics. our user study indicated that the tool enables users identify twice as many errors, iterate with more prompt versions and achieve a higher score on evaluation metrics within the same time frame. we hope that our easy to setup, self-contained tool will facilitate broader adoption among those involved in llm migration tasks.