On the Relation between Sensitivity and Accuracy in In-Context Learning (2023)

Abstract: In-context learning (ICL) suffers from oversensitivity to the prompt, making it unreliable in real-world scenarios. We study the sensitivity of ICL with respect to multiple perturbation types. First, we find that label bias obscures the true sensitivity, and therefore prior work may have significantly underestimated ICL sensitivity. Second, we observe a strong negative correlation between ICL sensitivity and accuracy: predictions sensitive to perturbations are less likely to be correct. Motivated by these findings, we propose SenSel , a few-shot selective prediction method that abstains from sensitive predictions. Experiments on ten classification datasets show that SenSel consistently outperforms two commonly used confidence-based and entropy-based baselines on abstention decisions.

## introduction few-shot learning (fsl) refers to a system's ability to quickly learn a new task based on a few labeled examples. recently, in-context learning (icl) has made significant progress in fsl, where a language model (lm) is prompted with a few demonstrated examples that enable it to make predictions for new examples without any gradient update. however, a known issue of icl is that it is oversensitive to the prompt @xcite , making it less reliable in practice. despite a near-universal acknowledgment of this issue, when and how the prediction is sensitive remains unclear @xcite . this paper fills these gaps. we conduct a systematic study of the icl sensitivity to prompt perturbations. specifically, we perturb the task instruction (by paraphrasing and noise injection) and the in-context example orders. we then measure the prediction sensitivity by the magnitude of model output changes due to the prompt perturbation. our first observation is that the extent of sensitivity is significantly underestimated due to label bias in icl: lms tend to assign a higher probability to a specific label regardless of the prompt @xcite , thus appearing to make stable predictions. our study shows that the adjusted sensitivity after mitigating label bias is up to 2.8x of the raw sensitivity. after mitigating label bias, we observe a negative correlation between the adjusted sensitivity and the accuracy of icl: if a prediction is sensitive to prompt perturbations, then it is likely to be incorrect (figure 1 left). this finding aligns with our intuition that if the prediction is sensitive to the prompt that elicits the lm concept (e.g., sentiment) @xcite , then the example is likely not typical for that concept, and is thus more challenging. our experiments show a significant negative correlation of up to -0.40 (pearson) between icl sensitivity and accuracy. given the above findings, a natural idea is to use sensitivity as a signal to abstain from making predictions on error-prone examples-an important mechanism to increase user trust when deploying icl models to high-stakes domains such as healthcare @xcite and legal systems @xcite . our proposed method, sensitivity-based selective prediction (sensel), uses sensitivity to make abstention decisions: the lm abstains on examples where its prediction is sensitive to prompt perturbations (figure 1 right). compared to the common approach of training a separate model to make abstention decisions @xcite @xcite , our approach does not require large amounts of labeled data and thus is more suitable for the few-shot setting. our experiments show that sensitivity is a stronger signal than output probabilities for abstention. sensel consistently outperforms two baselines based on model probabilities (maxprob ## icl sensitivity study in this section, we study the interplay between label bias and prediction sensitivity in icl, as well as the relation between sensitivity and accuracy. ## sensitivity-based selective few-shot prediction motivated by the correlation between the sensitivity and accuracy of icl, we propose sensel-a selective few-shot prediction method based on sensitivity. problem statement the goal of selective prediction is to abstain on examples that the model is not confident about, to avoid presenting wrong predictions to users @xcite . selective prediction methods score model confidence c on each example, and abstain on examples with low prediction confidence (@xmath0, where γ is a threshold to control the trade-off between accuracy and coverage. sensitivity-based selective prediction sensel scores icl prediction confidence as the negative value of the prediction's sensitivity to prompt perturbations, and then abstains on examples whose confidence scores (i.e., negative sensitivity scores) are below a certain threshold γ. experiment setup for sensel, we always use the adjusted sensitivity computed after mitigating the label bias. as writing good task instructions can be hard @xcite , we experiment with two settings: inst (a task instruction is available), and no inst (no task instruction is available). we perturb the task instruction in the inst setting (sensel-insth, sensel-insta), and perturb the example ordering in the no inst setting (sensel-exord). we compare sensel to two simple yet strong baselines, maxprob, which uses the maximum output probability over the labels as the confidence score @xcite , and entropy, which uses the negative value of the entropy of the output probabilities over the labels as the confidence score @xcite . we evaluate the effectiveness of selective prediction methods with the area under the f1-coverage curve (auc), which measures the average f1-score at different coverage rates @xcite . for label bias mitigation, since the same conclusion holds for pc and cc, we report the results for pc in the main paper and the results for cc in appendix c.2. ## conclusion while icl sensitivity is a widely-known issue, its relation to other variables is not studied. this work first conducts a comprehensive study, and finds that icl sensitivity is negatively correlated with accuracy when label bias is mitigated. based on this observation, we develop a few-shot selective prediction method that abstains on highly sensitive predictions. our results show that icl sen-sitivity exhibits a useful pattern-it reflects how confidently an lm understands the task. there are many open questions for future work. first, our study of the sensitivity-accuracy relation is correlational but not causal. future work should explore causal experiments to study whether icl predictions are sensitive because they are uncertain. second, it remains unclear why sensitivity is negatively correlated with accuracy in icl, which requires a better understanding of the mechanism of icl. third, our work mainly focuses on the text classification tasks. future work can further explore other tasks such as text generation and question answering with structured output.