P ro SA : Assessing and Understanding the Prompt Sensitivity of LLM s (2024)

Abstract: Large language models (LLMs) have demonstrated impressive capabilities across various tasks, but their performance is highly sensitive to the prompts utilized. This variability poses challenges for accurate assessment and user satisfaction. Current research frequently overlooks instance-level prompt variations and their implications on subjective evaluations. To address these shortcomings, we introduce ProSA , a framework designed to evaluate and comprehend prompt sensitivity in LLMs. ProSA incorporates a novel sensitivity metric, PromptSensiScore, and leverages decoding confidence to elucidate underlying mechanisms. Our extensive study, spanning multiple tasks, uncovers that prompt sensitivity fluctuates across datasets and models, with larger models exhibiting enhanced robustness. We observe that few-shot examples can alleviate this sensitivity issue, and subjective evaluations are also susceptible to prompt sensitivities, particularly in complex, reasoning-oriented tasks. Furthermore, our findings indicate that higher model confidence correlates with increased prompt robustness. We believe this work will serve as a helpful tool in studying prompt sensitivity of LLMs. The project is released at: https://github.com/open-compass/ProSA .

## introduction in recent years, large language models (llms) have rapidly become the focus of the artificial intelligence field. by training on large-scale corpora, llms have shown impressive capabilities in multiple tasks @xcite . the input for llms, known as prompts, is crucial for their ability to complete a wide variety of tasks. the prompts for llms come in various forms. even for the same requirement, different individuals' varying expression habits can result in prompts of different styles. figure 1 illustrates four styles of prompts used by llms when completing mathematical tasks. the diversity of prompts elicits various responses from llms. recent studies @xcite @xcite investigate model performance across different prompt templates and demonstrate that llms are highly sensitive to the nuances of prompts. even minor alterations to prompts can lead to substantial declines in model performance. this sensitivity to prompts poses a challenge for researchers aiming to precisely evaluate the models' capabilities. moreover, users frequently have to iterate on their prompts numerous times to achieve higher quality outputs. however, current research on prompt sensitivity mainly centers on dataset-level analyses, focusing on performance variations across distinct prompt templates within identical datasets. also, existing works neglect examining how llms align with human expectations under different prompts in subjective evaluations, hindering the accuracy of insights reflecting real-world user experiences. moreover, it's remain changeling in elucidating the reasons behind llms' sensitivity to prompts. to tackle the aforementioned issues, we introduce the prosa, focusing on evaluation and understanding the prompt sensitivity of the current llms. it can serve as a proxy on monitoring the robustness and stability of the llms. in this work, we devise the prosa by focusing on the instancelevel analysis, develop a novel sensitivity metric, and leveraging decoding confidence for elucidating the underlying mechanisms. specifically, we emphasize instance-level measurements and analyses, comprising both objective evaluations against precise references and open-ended subjective evaluations. our experimental scope spans multiple tasks, ranging from understanding and logical reasoning to coding and general alignment capabilities. to achieve this, we introduce a novel metric designed to quantify prompt sensitivity. we define our metric, termed promptsensiscore or pss, as the average discrepancy in the llm's responses when confronted with different semantic variants of the same instruction (prompt). a comprehensive explanation of promptsensiscore (pss) is provided in sec. 2.2. furthermore, we investigate prompt sensitivity by measuring the instance-level pss of several popular open-source llms through the objective evaluation and subjective evaluation. in the objective evaluation, we assess 8 llms across 4 datasets, spanning diverse capabilities, with each model tested on 12 prompt variants. findings reveal variations in prompt sensitivity among models and datasets, with llama3-70b-instruct (ai@meta, 2024) demonstrating the highest robustness. the study further illustrates that incorporating few-shot examples alleviate prompt sensitivity, particularly evident in the transition from zero-to one-shot scenarios. larger llms especially benefit from increased few-shot instances, exhibiting greater robustness improvements. additionally, we analyze five advanced llms using two prominent subjective evaluation benchmarks: lc alpacaeval 2.0 @xcite and arena hard auto @xcite . our findings illustrate that these models demonstrate high resilience in answering straightforward queries but encounter heightened sensitivity with more complex ones. categorizing prompts reveals that llms are particularly robust when drawing upon established knowledge domains, such as troubleshooting it issues. conversely, in coding tasks or those requiring creativity, llms prove to be more susceptible to variations in prompts. we also delve into the underlying reasons for prompt sensitivity. leveraging the objective evaluation as a stand-in, we utilize the decoding confidence associated with the correct answer to scrutinize model behavior. findings suggest that prompt sensitivity is essentially a reflection of the model's confidence level: higher confidence in its outputs correlates with increased robustness against prompt semantic variations. our contributions can be summarized as follows: despite all instances in a dataset following the same topic, the differences in model performance under the same instances with different prompt templates are often overlooked. each instance can vary widely in complexity, context, and information type, ranging from straightforward factual questions to those requiring nuanced understanding. this diversity means the model's sensitivity to prompts can differ significantly between instances. as shown in figure 2 , llms may be robust to templates for some instances while being sensitive to templates for others. additionally, relying on aggregate metrics to assess prompt sensitivity can obscure important nuances. while overall trends may be useful, they can mask differences in individual instances where the model's responses drastically changes due to slight prompt modifications. therefore, by analyzing instance level prompt sensitivity, we can gain deeper insights into how various factors influence model responses, leading to more effective and reliable use of llms. ## prompt sensitivity on the objective evaluation the objective evaluation is a common form for evaluating llms. in this setup, the given tasks often have a specific ground truth, such as the correct options in multiple-choice questions or the answers to mathematical problems. the prompt sensitivity on the objective evaluation is analyzed by extracting the outcome of llm's different responses under various prompts for the same instance. ## prompt sensitivity on the subjective evaluation with the emergence of llms' capabilities and their large-scale deployment as assistances or tools serving humans, how to evaluate the quality of llms' responses is receiving increasing attention. in realworld scenarios, the vast majority of instances do not have a specific ground truth that is definitively better than others. thus, some previous work has attempted to evaluate the quality of generated responses using the subjective evaluation. this evaluation is often carried out by either using human raters or a powerful model, such as gpt-4, to score the generated text. this subjective evaluation, compared to the objective evaluation, better reflects the alignment of llms with human needs. however, to the best of our knowledge, existing research on prompt sensitivity has not involved an analysis of prompt sensitivity on the subjective evaluation. ## conclusion in summary, we propose an instance-level prompt sensitivity metric, pss, and conduct a comprehensive analysis on both objective and subjective evaluation. additionally, we explore the relationship between prompt sensitivity and model confidence. we believe our work can provide guidance for further sensitivity analysis and building robust llms.