Large Language Models are Miscalibrated In-Context Learners (2025)

Abstract: When adapting ICL with or without fine-tuning, we are curious about whether the instruction-tuned language model is able to achieve well-calibrated results without suffering from the problem of overconfidence (i.e., miscalibration) considering its strong instruction following ability, especially in such limited data setups. In this work, we deliver an in-depth analysis of the behavior across different choices of learning methods from the perspective of both performance and calibration. Through extensive controlled experiments, we observe that the miscalibration problem exists across all learning methods in low-resource setups. To achieve simultaneous gain for both in-task performance and calibration, we then study the potential of self-ensembling applied at different modeling stages (e.g., variations of in-context examples or variations in prompts or different ensembling strategies) to make the predictions more calibrated and have comparable or even better performance. We find that self-ensembling with max probability produces robust and calibrated predictions. Our work reveals the potential calibration problem of using ICL despite the improvements in task performance and sheds light on which learning paradigm to choose. We also provide practical guidelines for choosing learning paradigms depending on whether the data has been seen by the model before and a worthwhile solution via self-ensembling on how to enhance both task performance and calibration of LMs, which we hope could encourage further study.

## introduction machine learning and nlp have undergone a significant transformation recently, largely propelled by language models (lms) @xcite @xcite @xcite . among different learning paradigms, supervised fine-tuning (sft) and in-context learning (icl) have emerged as pre-dominant methodologies @xcite , demonstrating commendable efficacy across many tasks. sft tunes the model's parameter and effectively specializes a (general-purpose) model to specific tasks by learning the knowledge in the training data and optimizing the objective. icl, for each input, leverages the few-shot examples (i.e., the so-called demonstrations) to generate predictions without tuning model parameters and treating the model as a 'black box'. considering the different input format between training with sft and inference with icl, @xcite and @xcite introduce the in-context examples into training phrase, which we call supervised in-context learning (sicl). however, when the demonstrations, as a strong inductive bias, get combined with sft, it has been shown that lms become more likely to fall into the problem of overconfidence @xcite ; the predicted confidence distribution of icl may be miscalibrated due to the bias in in-context examples @xcite . through our extensive experiments, we observe that both paradigms, sft and icl, suffer from the problem of miscalibration in low-resource scenarios. the important challenges of overconfidence and miscalibration, particularly in scenarios marked by limited data availability, underscore the need for a nuanced understanding of these paradigms. these challenges could be more severe in instructiontuned models considering their strong instructionfollowing abilities. however, most of the previous work @xcite only focuses on comparing solely the performance of sft and icl on out-of-distribution (ood) data, targeting general-purpose lms. here, we instead focus on studying instruction-tuned taskspecialized language models, where the behavior of different paradigms' in-task performance along with their calibration remains an open research question. therefore, in this work, in addition to the is the sentence hate, neutral or support? <@xmath0 figure 1 : illustration of the self-ensembled learning methods. we introduce different types of variations to the input and feed them to a single language model. after having the predictions, we run self-ensembling to obtain final predictions and their confidence. based on the findings above, we propose practical guidelines for choosing different learning paradigms, which suggests supervised tuning methods are likely to achieve better performance on unseen datasets while on seen datasets icl combined with other 'tweaks' such as model calibration can be a better choice. in response to the possible miscalibration, we explore the application of selfensembling as a potential solution (shown in figure 1 ), inspired by the effect of ensembling of multiple independent models in improving the reliability of the predictions @xcite . we incorporate diverse variations in in-context examples and prompts, tailor them to different learning methods, and find that self-ensembling fortifies the model's calibration by 43% on average without compromising task performance. contributions. 1) we deliver a comprehensive empirical analysis with different choices of learning methods across a variety of tasks in limited data scenarios, demonstrating that icl doesn't improve the calibration of the model consistently ( §5.1). ## related work learning paradigms. fine-tuning (ft) pretrained lms has been used as an effective method to adapt them to specific tasks and datasets @xcite @xcite . with the generation of much larger and more powerful lms @xcite , parameterefficient fine-tuning (peft) @xcite has been proposed, where the central idea is to tune only a fraction of the model parameters to reduce computation and memory costs. without tuning the model, in-context learning (icl) has shown great potential, achieving promising performance on various tasks with demonstration examples @xcite . motivated by the positive results of icl by concatenating multiple in-context (ic) examples to the lm input at inference, @xcite and @xcite introduce labeled in-context examples to the supervised training process (sicl). this has been further improved and utilized with pretraining @xcite and other training strategies @xcite . @xcite study task performance and stability with different peft methods in addition to icl, including prompt tuning, and instruction tuning (it) @xcite . @xcite explore the relationship between icl and it, and interpret icl as implicit it. @xcite compare the generalization ability of icl and ft on out-of-distribution (ood) data. @xcite show the superior performance of hard prompt tuning over standard ft in lowdata scenarios. however, previous work has not explored all the learning methods systematically within low-data scenarios and has not investigated them through the joint optics of in-task performance, confidence, and their trade-offs. calibrating lms. a well-calibrated lm should be accurate in terms of performance while also producing reliable confidence estimates for their predictions @xcite . this is essential for ensuring the robustness and safety of language model deployment @xcite . when concatenating different in-context examples with various labels, tokens, and example ordering, icl would be influenced by the bias in the concatenated examples @xcite , and various calibration methods have been proposed to mitigate this issue @xcite @xcite . moreover, the model itself also contains certain 'implicit' label biases due to its (pre)training corpus, which would have an effect on the confidence estimation as well @xcite . furthermore, ft may suffer from miscalibration for both in-distribution and ood data due to overparameterization when adapting to the specific data @xcite , and the miscalibration effect, as we investigate in this paper, might be even more pronounced in limited data scenarios. ensembling model predictions has been used to mitigate the problem of overconfidence and improve the reliability of the final model predictions @xcite . a standard ensembling practice is to train the model with different hyperparameters or different initialization @xcite . concerning lms, @xcite ensemble fine-tuned lms to quantify the uncertainty with disagreement among different ensembling components. @xcite and @xcite ensemble partially tuned lms, considering the computation resources for training and the storage for saving different lm checkpoints. although the proposed method achieves better and more reli-able results, it takes a considerable amount of time, computation, and storage resources to train and save multiple models, which often makes them inapplicable to lms. in contrast to having several tuned models with supervised learning, @xcite demonstrate the feasibility of using selfensembling in icl without tuning the model. in this work, we explore self-ensembling in the novel context of diverse learning paradigms and low-data setups and show that it is possible to improve model calibration without compromising performance. ## background: learning paradigms in this paper, we analyze and compare four different learning paradigms relevant to icl in lowresource scenarios: zero-shot learning (zsl), 1 in-context learning (icl), supervised fine-tuning (sft) and supervised in-context learning (sicl). with classification tasks in focus, we briefly describe each paradigm in what follows. zero-shot learning (zsl). given the input x and the prompting template f p , the prediction ŷ from the lm can be represented as ŷ = arg max j p(y j |f p (x)), where the parameters of the underlying lm are fixed. the prompting template f p (x) includes the task instructions and special symbols which can be replaced by the input x. we attach the prompting templates for different classification tasks in appendix d. in-context learning (icl). similar to zsl, instead of only feeding the input x to the model, we first prepend m in-context examples (ic) (also called demonstrations) [f p (x 1 ), y 1 ; ...; f p (x m ), y m ] to the input x. the examples are retrieved from the pool of examples r following (random or non-random) selection strategy. the prediction is then defined as ŷ = arg max j p(y j |[f p (x ic ), y ic ], f p (x)). supervised fine-tuning (sft). as mentioned, zsl and icl are inference-only paradigms treating the lm as a black box. on the other hand, sft first trains the model on the training set following the input format f p (x) from zsl. note that here we use sft to refer to instruction-style fine-tuning with a prompting template, which we see as the context. inference with the tuned model p ′ is then conducted in the same way as with zsl. during training and inference, we can use different prompt-ing templates to create variations in the model input, which we further elaborate on in §6.1.1. supervised in-context learning (sicl). based on the propositions from @xcite and @xcite , we can also fine-tune the model to directly optimize the in-context learning objective. for each training step, m in-context examples (x 1 , y 1 ), ..., (x m , y m ) are selected from the pool r. we then prepend the selected in-context examples to the input x as before with icl and use the concatenation as the final model input, and train the model to generate y. inference proceeds in the same way as with icl, except that we now use the task-tuned model p ′ . ## experimental setup datasets and evaluation metrics. we consider 7 classification datasets that cover a range of label numbers and scenarios: sst-2, sst-5 @xcite , rte @xcite , anli @xcite , measuring hate speech corpus @xcite , intent detection from nlu++ @xcite and manifestos @xcite . in order to simulate low-data setups, we sub-sample smaller training data from the full data for each dataset. the details of the datasets along with their corresponding evaluation metrics are provided in appendix a.1. implementation details. unless noted otherwise, we use flan-t5 large @xcite as the main model in the experiments. detailed training environments and hyper-parameters are provided in appendix a.2 and a.3. further, we provide all the prompting templates in appendix d.1 and d.2. estimating calibration. beyond task performance of all the possible variants, we estimate the calibration of the model's predictions (as a proxy towards model confidence) by using expected calibration error (ece) @xcite . it divides the n predicted results based on their confidence into m bins b 1 to b m and then computes a weighted average over the absolute difference between the accuracy acc(b m ) and mean confidence conf(b m ) of the predictions within each bin. we set m to 10 in this work. ece measures the difference between the model's empirical accuracy and its confidence (predicted probability). the smaller the ece, the more confident the model prediction would be. we also report the negative log-likelihood (nll) @xcite ) -n @xmath1. we define well-calibrated results by an ece threshold below 0.2 to facilitate clearer discussion. ## rq1: how does icl impact model calibration? in this section, we are interested in how icl impacts the calibration of the language model. to answer this question, we experiment with zsl, icl, sft and sicl to find out 1) whether the model is calibrated and 2) the differences in calibration with different learning methods. we first present a comprehensive analysis of various learning methods in low-resource settings, detailing model performance and calibration errors in ## conclusion we have provided a comprehensive analysis of the intricate relationship between in-task performance and calibration across various learning methods in low-resource scenarios. our findings illuminate the nuances of in-task performance and calibration across different task families, meanwhile addressing the inherent miscalibration over all learning methods. we have also investigated effective strategies to enhance both aspects simultaneously, offering a viable solution through self-ensembling: it results in more calibrated predictions with comparable or superior task performance. we hope that this study will contribute valuable insights into the dynamic landscape of lms. these discoveries also offer practical guidance to practitioners, aiding them in choosing suitable learning paradigms and paving the way for the development of more reliable and high-performing lms across diverse applications.