On the Calibration of Large Language Models and Alignment (2023)

Abstract: As large language models attract increasing attention and find widespread application, concurrent challenges of reliability also arise at the same time. Confidence calibration, an effective analysis method for gauging the reliability of deep models, serves as a crucial tool for assessing and improving their reliability. However, such investigation has been comparatively underexplored. In this work, we conduct a systematic examination of the calibration of aligned language models throughout the entire construction process, including pretraining and alignment training. At each stage, we investigate how different training settings, such as parameter scales and training data, affect model calibration. To thoroughly assess model calibration, we evaluate models on three most concerned aspects: generation, factuality and understanding. Our work sheds light on whether popular LLMs are well-calibrated and how the training process influences model calibration.

## introduction large language models (llms) like gpt-3 @xcite , palm @xcite , and gpt4 @xcite , followed by many open-source replications including llama @xcite , pythia @xcite are revolutionizing the paradigm and re-shaping the expectation of modern natural language processing. when further trained with alignment treatment @xcite , these llms further exhibit impressive capability in responding to generalized human instructions, which implies their potential as generalpurpose intelligent assistants and this has since attract considerable attention in the field and around the world. as llms find more diverse applications and exert widespread influence, it becomes increasingly imperative to ensure their reliability and faithfulness, particularly in fields such as healthcare @xcite and law @xcite . these are domains where inaccurate predictions can lead to significant, potentially severe challenges. however, due to the intrinsic autoregressive mechanism and complex system structures, the behaviours of these models can not be easily attributed or interpreted. confidence calibration is an effective method to estimate a model's awareness of its uncertainty, and it helps enhance our understanding and assurance of the trustworthiness of deep models. generally, it associates model output confidence, i.e. probability, with ground truth correctness likelihood @xcite and informs the user to what extent the outputs should be trusted, even though they may not always be correct. intuitively, for example, given 100 predictions in a classification task which are produced by a classifier and each of them is assigned 0.8 confidence, we expect 80 of them to be correctly classified for a well-calibrated classifier. as a consequence, better calibration of llms could significantly extend their usability. in early meteorology, calibration was noted as validity @xcite or reliability @xcite , indicating the trustworthiness of forecasters. well calibrated probabilities can provide extra information for users to decide whether to trust the model's prediction, particularly for modern neural networks whose decisions are harder to interpret @xcite . studies have also pointed out that calibration is helpful to reduce hallucination in language models @xcite . previous works have shown that pre-trained language models can generate well-calibrated predictions @xcite . however, these works mainly concentrate on vanilla language models, while the aligned language models receive less focus. a newly proposed work evaluates calibration of some aligned models by prompting them to verbalize confidence in the token space @xcite , but it mainly studies black-box models, whose training process is not available, and thus can not provide insight into how model calibration is affected by different factors in the alignment training process. to conclude, a systematical study on the calibration of aligned language models is still missing, and our work aims to fill this gap. in this work, we study the calibration of aligned language models in the entire building cycle and provide evidence on how to achieve decent model calibration. an overview of the scheme of out study is at figure 1 . following the training process of aligned language models, we study model calibration in pre-training stage and alignment training stage respectively. in each stage, we reveal how model calibration changes when using different training settings. for pre-training stage, we examine the effect of parameter scale and training dynamics (steps). for alignment training stage, we study the effect of instruction tuning and rlhf, in which instruction tuning is further scrutinized by changing instruction datasets, training methods and also training dynamics. besides the understanding and generating ability, factual faithfulness and reasoning capability are two widely considered issues with large language models @xcite . we also follow this path to study models' calibration when applied to different tasks. for this purpose, we design three tasks for each of the stages above. (1) to evaluate model calibration on common text generation, we use causal language modeling (clm) task, which is also the objective of pre-training stage. through extensive experiments and analysis, we arrive at the following findings. for pretraining of llms: for alignment of llms: for different tasks: we believe these conclusions as well as detailed experiments can take us a step further towards understanding large language models, especially the intrinsic mechanism of their calibration behaviour. our experimental results also provide us with some possible solutions to improve calibration, including increasing model scales and employing parameter efficient tuning methods. besides, diversity guided instruction data construction may also be very promising. hopefully these findings can shed light on future works to construct more factual and trustworthy assistants. confidence calibration is a concerned problem for classification models. a large amount of works have studied the calibration of statistical machine learning systems and the methods to improve their calibration @xcite @xcite . later, calibration of neural networks have also been researched on @xcite @xcite . @xcite points out that modern neural networks are not as calibrated as their ancestors and proposes a temperature scaling methods to calibrate neural networks. in natural language processing field, calibration of transformer-based language models are evaluated among different tasks, including machine translation (kumar and sarawagi, 2019), qa @xcite and selective prediction @xcite . recently, large-scale generative language models are receiving growing attention, and some works have examined calibration of these models @xcite @xcite . there are also works improving calibration of large language models, for example, @xcite propose knn prompting to effectively mitigate calibration errors in in-context learning. however, as mentioned before, these works either concentrate on vanilla language models or study black-box models. we study models calibration in their whole life cycles from pre-training to alignment training, where our main contributions lie in. analysis of large language models. understanding various aspects of llms through theoretical or empirical approaches have long been an important interests for nlp scholars. many works have demonstrated the scaling law of llms in different scenarios w.r.t. model scales, data size and computational costs @xcite @xcite . @xcite defines and reveals the emergent abilities of large language models. @xcite proposes a holistic evaluation framework named helm to analyze large language models on their capabilities, limitations, and potential risks. hallucination and factuality also draw a lot of attention @xcite ), but they do not take a further step towards the intrinsic mechanism while merely explore the verification on the surface. differently, this paper provides a formal and systematical analysis on calibration behaviour of llms and their alignment treatment. ## definitions in this section we formally define some basic concepts in our work, including confidence calibration, reliability diagram and expected calibration error. confidence calibration is the main objective we are studying in this work and the other two are tools we use to evaluate model calibration. our selection of tools follows previous work @xcite . confidence calibration. given a supervised multi-class classification scenario, where we have input x, label @xmath0, 2...k}, model prediction y ′ ∈ y and confidence p ′ ∈ [0, 1]. a model is perfectly calibrated, if we can get for any input x @xcite . in another word, the more confident a model is, the chance of its prediction being the same as ground truth should be higher. it should be noted that p (y ′ = y|p ′ = p) can not be calculated with finite number of samples, so calibration is often evaluated by some statisitcal approximations. reliability diagram is a kind of visualized evaluation of confidence calibration (degroot and fienberg, 1983), which plots prediction accuracy as a function of confidence (e.g. figure 2 ). to evaluate the calibration with a model with a finite set of samples, we divide confidence interval [0, 1] into m bins with equal length (1/m ) and group model predictions into these bins according to their prediction confidence. let b m be the set of indices of samples which fall into the interval ( m-1 m , m m ], then for each interval bin, we can calculate corresponding accuracy and average confidence as follows: given acc(b m ) and conf (b m ), we can draw the reliability diagram for a model. for a perfectly calibrated model, we will have acc(b m ) = conf (b m ) for all m, so its reliability diagram will be @xmath1. obviously, the nearer a curve is to the diagonal, the better calibration it represents. though perfect calibration is impossible, we normally hope a model is well calibrated. note that since reliability diagram do not contain the number of samples, sometimes it could be insufficient to represent true calibration of a model when some bins only contain very few samples. ## calibration evaluation tasks and data as mentioned before, we evaluate model calibration on three tasks considering language models' ability of understanding, factuality and reasoning. in this section, we in detail introduce for each task how the evaluation is conducted and the datasets chosen for evaluation. causal language modeling is the task of predicting the next token for a given sequence, which is also the pre-training objective of causal language models. for a test sequence, we randomly sample a position in the sequence. then this sequence is fed into models to generate a token corresonding to the position. if the predicted token is the same as the one in original sentence, we count it as a true positive. in such way we can get generation accuracy and confidence of test dataset, and then evaluate model calibration with reliability diagram and ece metric. we use development and test set of the pile dataset @xcite in this task. the pile dataset is a large-scale english text corpus which is frequetly used in the pre-training of large language models. facts generation is the task aimed at evaluating models' memory on factual knowledge. the task is mostly the same as causal language modeling in form , except that we utilize entity linking data, in which texts are labeled with entity spans. we let models generate the first token of entities. we only take the first token of entities into account as when the first token is correctly generated, there is high chance that the whole entity can also be recovered. in facts generation task, we use an enitity linking dataset t-rex @xcite , which includes entity-labeled texts extracted from wikipedia pages. for t-rex and the pile dataset, we randomly draw 100k samples as our evaluation set. multi-task language understanding is a task where models are given questions across different fields with multiple answer options, which is designed for testing the understanding and reasoning ability of a language model. we mainly focus on questions with a single correct answer. following mmlu benchmark @xcite , we concatenate 5 in-context samples ahead of the questions and designed prompts to constrain models to respond with answer options (i.e. 'abcd'). we choose mmlu benchmark @xcite as our evaluation data, which covers singlechoice questions in 57 subjects across stem, the humanities, the social sciences and so on. ## calibration in pre-training stage in this section, we study the effect of parameter scales and training dynamics in pre-training stage to models' calibration. ## calibration in alignment stage alignment training is divided into two sub-stages, instruction tuning and rlhf. in this section we study how model calibration changes during these process. as the models are fine-tuned on instructions in this stage, we add instruction prompt for all three tasks (see ## discussion confidence calibration is helpful for building honest large language models in two ways. firstly, confidence calibration is closely related to the uncertainty of language models, which is leveraged in many approaches like self-consistency to improve model performance @xcite . studying and improving model calibration will provide further evidence to these methods and inspire more uncertainty-based techniques. ## conclusions in this work we systematically study the calibration of aligned large language models. we designed thorough experiments evaluating the model calibration with different training settings and reveal how model calibration is affected by pre-training and alignment training process. in pre-training, we find that model calibration improves as parameter scales and training dynamics increases. in alignment training stage, experimental results show that instruction tuning damages model calibration significantly and ill-distributed synthetic data does more harm. such harm will increase when finetuning process lasts longer while can be remediated by using parameter efficient training methods like lora. in the mean time, we surprisingly find that rlhf has little impact on calibration of instruction tuned models.