Robust Prompt Optimization for Large Language Models Against Distribution Shifts (2023)

Abstract: Large Language Model (LLM) has demonstrated significant ability in various Natural Language Processing tasks. However, their effectiveness is highly dependent on the phrasing of the task prompt, leading to research on automatic prompt optimization using labeled task data. We reveal that these prompt optimization techniques are vulnerable to distribution shifts such as subpopulation shifts, which are common for LLMs in real-world scenarios such as customer reviews analysis. In this light, we propose a new problem of robust prompt optimization for LLMs against distribution shifts, which requires the prompt optimized over the labeled source group can simultaneously generalize to an unlabeled target group. To solve this problem, we propose Generalized Prompt Optimization framework , which incorporates the unlabeled data from the target group into prompt optimization. Extensive experimental results demonstrate the effectiveness of the proposed framework with significant performance improvement on the target group and comparable performance on the source group.

## introduction llms have gained significant attention for their remarkable performance in a broad range of natural language processing (nlp) tasks @xcite @xcite . this success has led to a shift in the paradigm of solving nlp tasks, moving away from training task-specific deep models towards developing task-specific strategies to effectively utilize llms @xcite @xcite . in the new paradigm, the prompt becomes a crucial factor in ensuring the effectiveness of llm on the nlp task, since even slight variations in prompt phrasing can largely affect llm output (reynolds and * * corresponding author. mcdonell, 2021; @xcite , making prompt optimization a promising research direction. existing research has explored automatic prompt optimization methods to eliminate manual effort in identifying effective prompts for a given task. these methods can be gradient-based or gradientfree, depending on the availability of model gradients. gradient-based methods optimize the prompt by calculating its gradients through the llm @xcite . gradient-free methods update prompts based on llm outputs using techniques such as an iterative search-and-select over the prompt space @xcite @xcite . this work focuses on gradient-free prompt optimization as llms are evolving into black-box api services @xcite . current gradient-free prompt optimization methods ignore distribution shifts @xcite , where the data an llm serves may differ from the labeled data for prompt optimization. realworld nlp applications often encounter distribution shifts, such as new user groups with distinct linguistic habits in customer review analysis. it is unclear if prompts hinder the robustness of llms against distribution shifts. to answer this question, we conduct experiments with the representative gpt-3.5-turbo-0301 model and prompts optimized by ape @xcite over paired data groups with distribution shifts. results on 30 pairs of data groups from six tasks show the risk of significant performance gaps under certain distribution shifts. based on this finding, we propose a new robust prompt optimization problem, which aims to optimize task-specific prompts with consideration of performance on both source and target groups under different distributions. given an nlp task such as sentiment analysis, our problem setting has a labeled source group similar as the conventional prompt optimization setting and a unlabeled target group. we keep the target group unlabeled for the consideration that distribution shifts happen along time in practice. labeling the newly coming target group will cause unnecessary labor cost and latency. accordingly, the main challenge for solving this robust prompt optimization problem is incorporating unlabeled data into prompt optimization. to this end, we propose the generalized prompt optimization (gpo) framework to obtain a taskspecific prompt for both source and target groups. to jointly considering the two groups in prompt optimization, the key lies in labeling the target group in an automatic and reliable manner by adapting knowledge from the labeled source group. towards this goal, we leverage the strong power of llm in zero-shot labeling, and prompt ensemble to enhance the labeling robustness. experimental results on three tasks demonstrate the effectiveness of our framework in improving the performance on the target group and simultaneously preserving a comparable performance on the source group. to sum up, our contributions are threefold: ## preliminary experiments prompt optimization aims to find the best prompt p that can instruct llms to predict the output y based on the concatenation of p and task input x, where x, y and p are all sequences of tokens. formally, given an nlp task with a dataset {(x, y)} following a distribution p , the goal is to obtain are prompts optimized by existing gradient-free methods robust to distribution shifts? ## robust prompt optimization in this section, we first formulate a robust prompt optimization problem and propose a gpo framework to enhance the robustness of the prompts. ## conclusion in this paper, we revealed the generalization issue of prompt optimization for llms under distribution shifts. we observed that the prompt optimized on the source data group may have a performance drop on the target group with distribution shifts. we performed an initial analysis aiming at identifying the factors that correlate to the varied generalization performance across groups, including label distribution shift and input distribution sim-ilarity. prompts by llm for further selection. for each task, we try the value of n as 9, 16, 25, 36, and k as n 's factors, to ensure obtaining effective prompts, where ape is not very parameter sensitive. moreover, we ablate the monte carlo search since it is optional and not significant for our tasks. given the randomness of the backbone llm, we set the temperature of the llm as 0, top p as 1.0. we set the max tokens for prompt generation as 100 to try to ensure no truncation, and keep other llm parameters the same as the official ape implementation. the parameters n and k are shown in