A Study on the Calibration of In-context Learning (2024)

Abstract: Accurate uncertainty quantification is crucial for the safe deployment of machine learning models, and prior research has demonstrated improvements in the calibration of modern language models (LMs). We study in-context learning (ICL), a prevalent method for adapting static LMs through tailored prompts, and examine the balance between performance and calibration across a broad spectrum of natural language understanding and reasoning tasks. Through comprehensive experiments, we observe that, with an increasing number of ICL examples, models initially exhibit increased miscalibration before achieving better calibration and miscalibration tends to arise in low-shot settings. Moreover, we find that methods aimed at improving usability, such as fine-tuning and chain-of-thought (CoT) prompting, can lead to miscalibration and unreliable natural language explanations. Furthermore, we explore recalibration techniques and find that a scaling-binning calibrator can reduce calibration errors consistently.

## introduction language models (lms) that encompass transformer-based architectures @xcite @xcite can generate coherent and contextually relevant texts for various use cases. despite their impressive performance, these models occasionally produce erroneous or overconfident outputs, leading to concerns about their calibration @xcite which measures how faithful a model's prediction uncertainty is. such a problem is pressing when users adapt them using a recent paradigm called in-context learning @xcite to construct performant predictors, especially for applications in safety-critical domains @xcite . we provide an in-depth evaluation and analysis of how well these models are calibrated -that is, the alignment between the model's confidence in its predictions and the actual correctness of those predictions. this token-level calibration assessment enables us to measure the discrepancy between the model's perceived and actual performance to assess its accuracy and reliability through a bayesian uncertainty lens. we find that lm such as llama @xcite is poorly calibrated in performant settings and there exists a calibration-accuracy trade-off (fig. 1 ) for low-shot settings (@xmath0: as we increase the amount of in-context samples, both prediction accuracy and calibration error increase. such a trade-off can be improved using more icl examples (@xmath1. crucially, this calibration degradation worsens when fine-tuning occurs using specialized data to improve usability, such as curated instructions @xcite , dialogues @xcite , or human preference data @xcite . though previous common practice suggests recalibrating models' logits via temperature scaling @xcite , we show that in contrast to classic regimes, the miscalibration issue in icl can not be easily addressed using such well-established scaling approaches @xcite . thus we propose to use scalingbinning @xcite , which fits a scaling function, bins its outputs, and then outputs the average of the function values in that bin, to reduce the expected calibration error below 0.1. furthermore, we study the trade-off in reasoning tasks that involve generation of explanations @xcite @xcite ) before the answer, showing that the model can produce confidently wrong answers (using confidence histograms and reliability plots) when prompted with explanations on strategy qa @xcite , commonsense qa @xcite , openbook qa @xcite , world tree @xcite . we carefully design our human evaluation and observe that, with the increase in model sizes and the quantity of icl examples, there is a corresponding rise in the proportion of confidently predicted examples among those incorrectly forecasted. moreover, we find that a high proportion of wrong predictions are of high confidence and showcase those typical confidently wrong examples of lms. moreover, we find that choosing icl samples from the validation set does not naturally lead to calibrated predictions, showing that icl learns in a fairly different way than stochastic gradient descent, a common prototype previous works hypothesize @xcite . motivated by this difficulty, we design controlled experiments to illustrate that when examples in the prompt are sampled from the same task instead of repeating a given example in various ways, the learning performance would be improved. ## related work calibration of language models. calibration is a safety property to measure the faithfulness of machine learning models' uncertainty, especially for error-prone tasks using lms. previous works find that pre-training (desai and durrett, 2020) and explanation @xcite improves calibration. models can be very poorly calibrated when we prompt lms @xcite , while calibration can also depend on model size @xcite . @xcite assesses the long-term dependencies in a language model's generations compared to those of the underlying language and finds that entropy drifts as models such as when gpt-2 generates text. the intricacy of explanations on complementary team performance poses additional challenges due to the overreliance on explanations of users regardless of their correctness @xcite . @xcite gives a framework for linguistic calibration, a concept that emphasizes the alignment of a model's expressed confidence or doubt with the actual accuracy of its responses. the process involves annotating generations @xmath2, <lo>, <@xmath3, then training the confidence-controlled model by appending the control @xmath4, followed by training a calibrator to predict these confidence levels, and finally predicting confidence when generating new examples. @xcite finds that asking lms for their probabilities can be better than using conditional probabilities in a traditional way. lhts @xcite is a simple amortized inference trick for temperaturescaled sampling from lms and diffusion models. to aggregate log probabilities across semantically equivalent outputs, @xcite utilize bidirectional entailment through a model to identify outputs that are semantically similar, thereby refining the uncertainty estimation process. @xcite identifies the calibration challenge in ambiguous qa and distinguishes uncertainty about the answer (epistemic uncertainty) from uncertainty about the meaning of the question (denotational uncertainty), proposing sampling and self-verification methods. @xcite trains a calibrator to identify inputs on which the qa model errs and abstains when it predicts an error is likely. @xcite proposes the pareto optimal learning assessed risk score for calibration and error correction but requires additional training. kalai and vempala (2023) show the trade-off between calibration and hallucination but they didn't study it in a realistic setting and how the predicted answer's accuracy would impact those two safety aspects. in-context learning. large models such as gpt-3 @xcite have demonstrated the potential of in-context learning, a method where the model infers the task at hand from the context provided in the input, without requiring explicit retraining or fine-tuning for each new task. some recent works attempt to understand icl through metalearning @xcite , bayesian inference @xcite , mechanistic interpretability @xcite , algorithm selection @xcite , synthetic data and simple function classes @xcite @xcite . notably, unlike previous works @xcite @xcite ) that focus on improving task accuracy using the same "calibration" terminology, we study the uncertainty of icl and measure its trade-off with accuracy. ## background setting. given a pre-trained language model p θ (w t |@xmath5, we seek to adapt it using the prompt w 0 = [x 1 , y 1 , x 2 , y 2 , . . . , x n-1 , y n-1 , x n ] to generate a predicted answer y @xmath6. in the context of reasoning, a popular approach is to hand-craft some explanations/rationales/chainof-thoughts e in the prompt w 0 = [x 1 , e 1 , y 1 , x 2 , e 2 , y 2 , . . . , x n-1 , e n-1 , y n-1 , x n ] to generate explanation e n and answer y n , for the test sample: en w 1 , w 2 , . . . , w k , y @xmath7. we extract answer token probabilities of lms, e.g. for binary classification tasks, we filter and extract probabilities p ("yes") and p ("no"), based on which we calculate the following statistics for studying the confidence and calibration of lms: confidence and feature norm. we record the maximum probability of the answer token as its confidence @xmath8. ## experiments we briefly summarize our results and findings before explaining the experimental settings. • for the base lms we considered, they are calibrated when prompting with a sufficient amount of icl examples to get non-trivial performance. ## discussion and concluding remarks in our investigation of the token-level calibration of in-context learning in contemporary language models, we illustrate the intricate trade-off between icl performance and calibration. our findings underscore the importance of being circumspect in model deployment, as maximizing icl performance does not invariably translate to improved calibration for low-shot and reasoning settings. as lms continue to evolve and gain more capabilities such as having long enough context windows that can include the whole training set as in-context examples for some downstream tasks, our result can be pedagogical when users would like to examine their uncertainty through prediction probabilities. moreover, the work suggests the following future directions: calibration beyond classification regimes. our findings indicate that in multi-choice or multi-class classification tasks, even though the calibration of answer tokens may deteriorate in high-performance settings, there may be a positive correlation between accuracy and the correctness of explanations in reasoning tasks. this suggests potential avenues for future research in exploring strategies such as the use of hedging words to express uncertainty and examining their relationship with predictive performance. implications in assessing beliefs of lms. previous works show that the expected calibration error would decrease monotonically as the number of icl examples increases @xcite when querying lms for answer probabilities. however, we find that zero-shot performance might be weak for models less than 30b, and in low-shot settings, calibration errors can sometimes be even worse than zero-shot. this implies that a close examination and careful control of epistemic uncertainty and aleatoric uncertainty can be needed before deriving conclusions in truthfulness @xcite for lowshot settings. limitations. we acknowledge the need to expand our evaluation, which is primarily focused on qa and classification tasks, beyond existing open-sourced language models and datasets. moreover, we didn't consider nuances such as inherent disagreement about labels @xcite and adaptive calibration error measures @xcite that might be important in certain use cases: it's worth noting that situations may arise where multiple labels share the highest predicted probability. in such instances, the definition (eq. ( 2 )) doesn't automatically become false; instead, we opt for the first maximal probability. these cases are less likely to occur in most of our experimental setups, where a substantial margin consistently exists between different labels.