What Did I Do Wrong? Quantifying LLM s’ Sensitivity and Consistency to Prompt Engineering (2025)

Abstract: Large Language Models (LLMs) changed the way we design and interact with software systems. Their ability to process and extract information from text has drastically improved productivity in a number of routine tasks. Developers that want to include these models in their software stack, however, face a dreadful challenge: debugging LLMs’ inconsistent behavior across minor variations of the prompt. We therefore introduce two metrics for classification tasks, namely *sensitivity* and *consistency*, which are complementary to task performance. First, sensitivity measures changes of predictions across rephrasings of the prompt, and does not require access to ground truth labels. Instead, consistency measures how predictions vary across rephrasings for elements of the same class. We perform an empirical comparison of these metrics on text classification tasks, using them as guideline for understanding failure modes of the LLM. Our hope is that sensitivity and consistency will be helpful to guide prompt engineering and obtain LLMs that balance robustness with performance.

## introduction there are only two hard things in computer science: cache invalidation and naming things. -phil karlton this famous quote refers to the innate tendency of computer scientists to choose poor names for a program's variables despite the existence of good coding practices @xcite . practically speaking, this is not a problem as long as the program does its job, but can we still argue the same in the era of large language models (llms)? llms @xcite have significantly changed how we process text by providing a straightforward interface, i.e., natural language, to define the problem to be solved @xcite . they provide software engineers with useful coding tips and can be used as part of larger and more complex software systems @xcite . it is common to set up an llm using a set of instructions called "prompt", and it soon became clear to both researchers and developers that the prompt itself can greatly influence an llm's performance @xcite @xcite . the process of writing a good prompt for the current task is called prompt engineering, and a great deal of different techniques have been proposed in this direction @xcite , ranging from a simple description of the problem to few-shot examples. from a practical standpoint, integrating llms in a software infrastructure introduces additional complexities, from choosing good prompt engineering strategies to parsing and controlling the output format of responses. in an effort to simplify the llm into a straightforward function call, new software libraries like instructor @xcite have emerged, reducing even further the entry barrier of llms for the ordinary programmer. instructor requires to define target labels as static fields of a python class, which is then automatically converted into a standardized json schema. the schema and the input sentence are then used to produce an llm response via a mechanism known as function calling @xcite . figure 1 provides an example of text classification using instructor. here, the assumption of the programmer should be that small changes to the labels' descriptions, as well as the labels' ordering in the code, should not affect the final result. unfortunately, reality is different: a change in the label definition, such as adding an article as shown in the figure, or in variables' naming can lead to minor prompt variations with drastic changes in the final prediction. as a result, developers remain unaware of the malfunction's cause and might later abandon the tool due to frustration. this simple yet troubling example might be generalized to other problems, for instance, code generation @xcite ; at the end of the day, llm this happens under the hood, making it very hard for a developer to debug the program. note that the same might happen, for instance, if the ordering or naming of variables is changed (hence the quote of section 1). we set out to address these problems by proposing two diagnostic metrics for black-box classifiers that are complementary to accuracy: the sensitivity to the input, which does not depend on the ground truth labels, and the consistency of predictions across examples of the same class. intuitively, it is desirable to have llms that are robust to semantically equivalent variations of the initial prompt, and their predicted labels' distributions should not vary much across samples of the same class. striving to improve these two metrics to- number location person description entity abbreviation 0.00 0.25 0.50 0.75 p τ (class) sample 1 sample 2 sample 3 wards low sensitivity and high consistency might significantly reduce the unpredictability of llms' behavior in complex software systems running in production. consider the illustrative example of figure 2 , where three samples of the same class person are classified (in probability) very differently when testing different variations of the same prompt. samples 1 and 3 have lower sensitivity than sample 2, as their distributions are quite stable on the correct class. in addition, sample 2 has low consistency compared to the other samples, which combined with sensitivity indicates "hardness" of classification. the behavior of sample 2 is highly undesirable, and the first step to address it is to formalize metrics that reflect these intuitions. we empirically test these metrics, finding that they indeed convey different information about the llms' behavior. we use this information to qualitatively analyze some of the datasets, mimicking what an llm engineer would do when debugging a real system. we also show how these metrics help to improve prompts by identifying different sets of problematic samples. ## related work this section positions our work in the context of three different but related research directions: influence of spurious features, uncertainty quantification, and prompt optimization. all these works fall under the broader umbrella of prompt engineering. ## methodology we formally introduce the metrics of interest, sensitivity and consistency, in a bottom-up fashion. let us consider a classification task τ with c possible classes and a data set of n i.i.d. samples @xmath0, y i ) @xmath1,...,n } , with x being the input text to classify and @xmath2, . . . , c} the ground truth label. moreover, we refer to the subset of samples of class y as d y . we make the typical conditional independence assumption about the data, that is, p(y, x) = p(y|x)p(x). without loss of generality, we introduce the distribution p ρ 0 (ρ) defined over all prompts that are semantically equivalent to a reference prompt ρ 0 , which is typically prepared by the user based on task τ ; when clear from the context, we will omit the subscript from the notation. because this distribution depends on the task but is independent of the input distribution, we shall write p(y, x, ρ) = p(y|x, ρ)p(x)p(ρ). please note that the distribution p(ρ) is not uniform. in practice, llms implement the distribution p(y|x, ρ), hence we can define the average probability that an llm predicts the class y under different variants of the same prompt as in this work, the sampler s is an llm that is tasked to rephrase prompts (appendix e shows examples), but it could also be an algorithm that generates text modifications as in @xcite . we then define the sensitivity of an llm to the prompt to reflect how much the llm prediction varies under the rephrasings of the original prompt. definition 1 (sensitivity). given a distribution p τ (y|x) defined as in equation 1, the sensitivity with respect to x is the normalized entropy it is important to note that the sensitivity does not require access to ground truth labels, which are often hard to acquire, and it does not necessarily correlate with the task's performance. the sensitivity should be used, for instance, as a guide to compare the "robustness" of different llms to variations of the prompt. a highly sensitive llm may require significant prompt optimization efforts, whereas a less sensitive llm tells us there might be no further room for improvement. the scope of the prompt variation is also important: one can measure sensitivity w.r.t. minor variations of the original prompt, as well as completely different prompting strategies as long as they convey a semantically equivalent instruction to the llm. the interpretation we attribute to sensitivity ultimately depends on the use case, but it is universal. the second metric is called consistency. it measures how much the distribution of equation 1 differs for two samples x, x ′ of the same class y using the total variation distance (tvd): definition 2 (consistency). given a categorical distribution p τ (y|x) defined as in equation 1 and two samples x, x ′ ∈ d y , the pair-wise consistency of a classifier is measured as intuitively, a consistent llm produces similar distributions p τ (•|x) regardless of the sample x of class y. when sensitivity is not 0, being consistent suggests that prompt rephrasings cause similar mistakes across all samples of class y, hence a careful tuning of the prompt is required. instead, an inconsistent llm behaves unpredictably among samples of the same class, where the same prompt rephrasings cause different mistakes; this might indicate that the problem is not the prompt, rather the classifier itself. note that, when sensitivity is 0, (in)consistency is uniquely determined by the inherent difficulty of the classification task, as it counts pair-wise class mismatches in the llm predictions. in this case, consistency will correlate more with classification metrics such as accuracy, thus we expect it to be most useful when sensitivity is high enough, e.g., 0.1. we argue that, in order to avoid bad surprises in production environments where new llms have to be tested and replaced quickly, it might be desirable to select llms with low sensitivity and high consistency, which means s τ → 0 and c @xmath3, ∀@xmath4, . . . , c}. finally, consider how probability-based output uncertainty is defined by @xcite for fixed input x and prompt ρ: output uncertainty is orthogonal to sensitivity, as it does not focus on variations of the prompt ρ. ## experiments the goal of the experiments is to assess how sensitivity and consistency can help in the context of english-based text classification datasets, analyzing the impact of prompt variations and prompting strategies from a different angle compared to previous works, but most importantly showing developers how to use these metrics foot_0 . therefore, it is outside the scope of this work to perform an extensive benchmark of these metrics across many datasets and llms, which would not be helpful to our purpose. large number of llms across we ran the llms on a 4 tesla v100 server with 32 gbs of memory and 252 gbs of ram. we consider llama-3-70b-instruct @xcite and mixtral-8x7b-instruct-v0.1 @xcite as the two open-source llms available on our servers, as well as gpt-3.5-turbo-0125 and gpt-4o-2024-08-06 @xcite as closedsource models. the temperature is set to zero and the seed is fixed to 42 to obtain quasi-deterministic behavior. following @xcite , five multiclass classification datasets are used for the comparison: trec, a 6-class (and 50 subclasses) question-answering task (voorhees and tice, 2000) with 500 test samples, committmentbank (cb) as a 3-way classification problem (de marneffe et al., 2019) with 250 test samples, a binary textual entailment problem (rte, @xcite ) with 2490 test samples, and the 14-class ontology extraction dataset dbpedia @xcite with 2000 balanced test samples. in addition, we also consider the 7-class web of science 46985 (wos) dataset with 2000 balanced test samples @xcite . in the event the llm cannot produce a valid class, we add an extra class label n/a. we analyze three different prompting strategies: a simple strategy, where the prompt consists of the task description and the list of classes; a detail strategy, where we provide a detailed description of each class; and a 1-shot strategy in which, compared to simple, we also provide one example for each class taken from samples that do not belong to the test set. to build different rephrasings of the task description only, we use the aforementioned llms. in particular, the prompt asks to rephrase the task description by changing the length or adding unnecessary words as long as the meaning remains the same. for the purpose of our work, which is only to show how sensitivity and consistency can be used, we fix @xmath5. also, the "right" value of q ultimately depends on the user's compromise between computational resources and stability of resulting statistics. in the quantitative analyses, we display the average sensitivity and average consistency, as well as the micro f1-score across all ρ. sensitivity and micro f1-score are averaged across samples, whereas pair-wise consistencies are averaged together. since the distributions of sensitivity and consistency values are far from being gaussian, looking at certain statistics such as the standard deviation can convey a misleading message of instability. therefore, we will perform qualitative analyses in later sections that show large deviations from the mean values, but in the interest of completeness we report standard deviations in appendix d. quantitative results to answer question i), as a result, developers should pay attention when switching llms in their applications: a prompt that worked well with an llm might cause instability (see the sensitivity gaps on cb) and significantly worse performance on another. for instance, on cb it might be preferable to choose the detail strategy with mixtral, since sensitivity is extremely low and the other metrics are highest compared to the alternative strategies. to further validate our results, appendix a, provides evidence of the significant deviation of our results compared to perturbed and random predictors. ## conclusions in this work, we have expanded the llm developer's toolbox with two diagnostic metrics, namely sensitivity and consistency, that complement performance metrics such as accuracy. in our experiment, we showed how different prompting strategies influence these metrics and how we can guide prompt engineering decisions based on new criteria that value intrinsic characteristics of llms' predictions. indeed, an llm that is very sensitive to variation of the prompt and has high test set accuracy might not be a good choice in a production environment, where multiple intermediate steps exist and each could lead to minor alterations in the prompt of the llm predictor. notably, sensitivity does not require access to the ground truth labels, and it would be interesting in future work to extend it to tasks different from classification, for instance, code generation. also, while this work mentioned prompt optimization, it is still an open question how to integrate these metrics in an automatic prompt engineering framework, leading to llms insensitive to nonsensical prompt variations and consistent in their (good) performances. our hope is that sensitivity and consistency to the input will become relevant metrics from both an academic and industrial perspective, helping to identify pain points of llms.