How are Prompts Different in Terms of Sensitivity? (2024)

Abstract: In-context learning (ICL) has become one of the most popular learning paradigms. While there is a growing body of literature focusing on prompt engineering, there is a lack of systematic analysis comparing the effects of prompt techniques across different models and tasks. To address this, we present a comprehensive prompt analysis based on sensitivity. Our analysis reveals that sensitivity is an unsupervised proxy for model performance, as it exhibits a strong negative correlation with accuracy. We use gradient-based saliency scores to empirically demonstrate how different prompts affect the relevance of input tokens to the output, resulting in different levels of sensitivity. Furthermore, we introduce sensitivity-aware decoding which incorporates sensitivity estimation as a penalty term in the standard greedy decoding. We show that this approach is particularly helpful when information in the input is scarce. Our work provides a fresh perspective on the analysis of prompts, and contributes to a better understanding of the mechanism of ICL.

## introduction in-context learning (icl) has become a popular learning paradigm in natural language processing (nlp) due to the rapid development of large language models (llms) @xcite @xcite . with carefully constructed prompts, icl achieves impressive performance on various tasks @xcite @xcite . as a result, prompt engineering, which aims to find prompts that lead to optimal performance, has emerged as a crucial research topic in icl @xcite . although effort has been made to understand the effectiveness of certain prompt techniques @xcite @xcite , there is no systematic analysis of them across various tasks and models @xcite . such an analysis is crucial for prompt engineering, prompt selection, and gaining a deeper understanding of the working mechanism of icl. in this paper, we present a systematic and comprehensive analysis of prompts based on the sensitivity of a function @xcite . we hypothesize that certain prompts are more effective for a given task because they decrease the level of sensitivity. based on the recent findings that icl implements gradient descent implicitly @xcite @xcite , an effective prompt can be seen as one that facilitates the learning of a new function with lower sensitivity compared to the original function learnt by the model. the sensitivity of a function enables a novel framework to analyze the effect of different prompts. see figure 1 for more details. we did extensive experiments to validate our hypothesis. we chose five widely used natural language understanding and common sense reasoning tasks. we selected models with varying sizes from three popular families: gpt, llama, and t5. we tested different prompts, including both human-designed prompts and prompts generated by an llm. the results strongly support our hypothesis that models exhibit different levels of sensitivity depending on the prompts used, and sensitivity is an unsupervised proxy of performance as it has a strong negative correlation between accuracy and sensitivity. with the help of gradient-based saliency scores, we find that tokens in the prompt (e.g., instructions) are more relevant to the output than tokens in the input where perturbations took place, which explains how different prompts lead to varying levels of sensitivity. furthermore, we introduce sensitivity-aware decoding, which incorporates sensitivity estimation as a penalty term in greedy decoding. we show that sensitivity-aware decoding is particularly effective when the prompt contains scarce information. our work provides a fresh perspective for comparing the effects of different prompts and enhances our understanding of the mechanism of icl. our contributions are summarized as follows: ## experiment settings data generation we use @xcite 's framework to generate perturbed data. 2 each of the synthetic data agrees on the original instance on all indices outside a subset. we notice that the synthetic data for one particular dataset are noisier (see ## results figure 2 shows the average accuracy and sensitivity of each model using various prompts across different datasets. 5 the pearson correlation coefficient shows a strong negative correlation between accuracy and sensitivity (@xmath0.8764, p-value ≪ 0.01). sensitivity can be viewed as an unsupervised proxy of accuracy given such a strong correlation. it is interesting to note that both flan-t5 models failed with zero_a and zero_b. we discuss this further in section 4.2. ## gradient-based saliency scores in light of recent studies that link icl and implicit gradient descent @xcite @xcite , we investigate the relationship between sensitivity and gradient. we use gradient-based saliency scores, which reveal the relevance of input tokens to the model prediction @xcite @xcite . the higher the score is, the more a token is supposed to contribute to the model output. we compute gradient-based saliency scores based on the norm of the gradient of the model output. the gradient g for a token x i in an input x is calculated as follows: the saliency score s(x i ) is obtained by taking the l1 norm of g(x i ): we calculate the saliency scores using gpt-jt-6b, gpt-j-6b, flan-t5-770m, and t5-770m. figure 5 gives an example of saliency scores over tokens, which shows that the tokens in the prompt are more relevant to the output than tokens in the input (see figure 10 in a.6 for more examples). to perform a quantitative analysis, we segment an instance into several parts. take figure 6 as an example. for a sentence x, tokens where perturbations happen are referred to as x input or input tokens (such as the input in figure 6 ), and the rest of the sentence is referred to as x prompt or prompt tokens. 6 we calculate the mean saliency score, denoted as s, for input tokens and prompt tokens respectively: the percentage of s input for base_b (27.2%) is higher than that for gkp (16.6%), which indicates that input tokens in base_b are relatively more relevant to the predictions than those in gkp. this is consistent with the observation in we also examine the mean saliency scores of ground truth tokens in zero_b (see prompt tokens are important in the sense that they provide information such as instructions and external knowledge, which are necessary for the model to produce outputs that the user expects. however, it is counter-intuitive that the mean saliency scores for prompt tokens are much higher than those for input tokens, as input tokens contain essential information as well. this observation may imply that memory plays an overwhelming role in icl @xcite @xcite @xcite )-llms were trained on similar instances, so they do not need to rely on input tokens in the test instances too much. in this sense, prompt tokens are more relevant because they trigger memories, and models implicitly infer task information from them @xcite @xcite . over-emphasis on prompt tokens may lead to hallucination as well. a recent study discovers that instruction tuning significantly increases sycophancy in llms, that they follow user's opinion or agree with user's claim even when they know it is false @xcite . 6 sensitivity-aware decoding we showed in section 4 that sensitivity can be viewed as an unsupervised proxy of accuracy. in this section, we further show that including sensitivity in decoding improves model performance. specifically, we add sensitivity as a penalty to greedy decoding: the results show that penalizing outputs with high sensitivity in decoding has an effect on model performance. we show that sensitivity-aware decoding works better with the most basic prompt, base_a, that contains plain input-target pairs. we believe this prompt represents those truly challenging problems in real life that have almost no clues or hints, and require very strong reasoning abilities to solve. apparently, none of the models we tested possess such abilities. therefore, sensitivity-aware decoding, which helps improve model performance under this "extreme condition," is highly meaningful in the present context. however, sensitivity-aware decoding is more computationally expensive compared to standard greedy decoding as it requires multiple inference passes. this makes it impractical for tasks that demand low latency. ## conclusion this work provides a novel perspective on prompt analysis, examining the effects of prompts in terms of the sensitivity of a function. we conducted a systematic and comprehensive analysis, and highlight how certain prompts are more effective due to their ability to reduce sensitivity levels. we show that sensitivity can serve as an unsupervised proxy of model performance, making it a valuable tool for evaluating model performance without using labeled data or ground truth. by introducing sensitivity-aware decoding, we show that incorporating sensitivity in greedy decoding is particularly helpful in cases where the input is less informative. since none of the models we tested performs well when there is limited information in the input, we believe sensitivity-aware decoding is highly practical in the current context. our work not only sheds light on prompt engineering, but also provides insight into the working mechanism of icl.