FIRST : Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation (2024)

Abstract: Large language models (LLMs) have become increasingly prevalent in our daily lives, leading to an expectation for LLMs to be trustworthy —- both accurate and well-calibrated (the prediction confidence should align with its ground truth correctness likelihood). Nowadays, fine-tuning has become the most popular method for adapting a model to practical usage by significantly increasing accuracy on downstream tasks. Despite the great accuracy it achieves, we found fine-tuning is still far away from satisfactory trustworthiness due to “tuning-induced mis-calibration”. In this paper, we delve deeply into why and how mis-calibration exists in fine-tuned models, and how distillation can alleviate the issue. Then we further propose a brand new method named Efficient Trustworthy Distillation (FIRST), which utilizes a small portion of teacher’s knowledge to obtain a reliable language model in a cost-efficient way. Specifically, we identify the “concentrated knowledge” phenomenon during distillation, which can significantly reduce the computational burden. Then we apply a “trustworthy maximization” process to optimize the utilization of this small portion of concentrated knowledge before transferring it to the student. Experimental results demonstrate the effectiveness of our method, where better accuracy (+2.3%) and less mis-calibration (-10%) are achieved on average across both in-domain and out-of-domain scenarios, indicating better trustworthiness.

## introduction with the rapid development of large language models (llms), many powerful models have been deployed into our daily lives for practical usage to help us make decisions @xcite @xcite . this makes it urgent for us to know to what extent we can trust the outputs of the models. calibration is one of the most important indicators beyond accuracy, which provides a confidence measure to the model's predictions @xcite . in llms, confidence is exactly the probability for each generated token. therefore, a well-calibrated model should align its prediction confidence with its ground-truth correctness likelihood as shown in figure 1 . as an example, recent hallucination detection methods rely on model prediction confidence as a significant indicator of potential hallucination @xcite . if the model is incapable of giving accurate confidence levels, people may fail to detect hallucinations due to the model's over-confidence, or people may falsely identify hallucinations due to the model's under-confidence. mis-calibration brings significant challenges for the deployment of llms in real-world applications. currently, there are two methods to obtain a language model for practical usage. first, fine-tuning, which fine-tunes pre-trained llms on specific datasets by matching each token entry with a target ground truth token. although fine-tuning can consistently improve performance on downstream tasks @xcite @xcite , we identify that the model obtained in this way exhibits a nature of "tuning-induced miscalibration". second, distillation-based methods transfer knowledge (e.g., soft labels) from larger llms to smaller models @xcite . although distillation shows better calibration than fine-tuning as it matches each token entry with a probability distribution instead of a hard label, we find it is still biased because of the mis-calibration nature of teacher models. in addition, distillation faces the challenge of determining the optimal amount of knowledge to transfer. transferring all the teacher's knowledge leads to high computational costs while transferring too little knowledge results in poor accuracy. therefore, it is crucial to balance between trustworthiness (accuracy and well-calibration) and efficiency for distillationbased methods. to address the challenge of obtaining a trustworthy model, we propose efficient trustworthy distillation (first), aiming to efficiently utilize a relatively small amount of the teacher's knowledge. specifically, we first identify the "concentrated knowledge" phenomenon, which shows that in the context of llms, the probability distribution of generated tokens is not uniform but rather concentrated on a few high-probability tokens. based on this finding, we propose to use the top-5 tokens as the knowledge to balance the trade-off between storage space and the amount of knowledge transferred, achieving efficient distillation. afterward, to eliminate the "tuning-induced mis-calibration" of the teacher model, we applied a "trustworthy maximization" to this portion of knowledge, ensuring that it maximizes the enhancement of the student model's accuracy while also guaranteeing its well-calibration. we first validate our method in in-domain scenarios, discovering that the models obtained by first achieve excellent accuracy, even with the use of a relatively small amount of top-5 knowledge and the "trustworthy maximization" process can significantly enhance these models' robustness to miscalibration. furthermore, we test our approach in out-of-domain settings, demonstrating that models obtained by first still exhibit the best trustworthiness and hold generalization ability. this indicates that first enables smaller models to genuinely learn the capability of being trustworthy, rather than being confined to in-domain scenarios. in summary, our key contributions include: ## efficient trustworthy distillation in this section, we introduce efficient trustworthy distillation (first), which can be divided into three parts. firstly, we select top-5 tokens as knowledge for transfer (efficient knowledge selection) in sec. §4.1. then, we adjust the knowledge for trustworthiness to ensure that the subsequent smaller models can maximize its utility (knowledge trustworthy maximization) in sec. §4.2. finally, we describe the learning process of the student model (knowledge matching) in sec. §4.3. the overall pipeline is shown in figrue 4. ## conclusion in conclusion, our proposed method, efficient trustworthy distillation (first), effectively enhances both accuracy and calibration in large language models. by applying "trustworthy maximization", first efficiently transfers the minimal yet most effective knowledge from teacher to student models. experimental results show that first consistently improves trustworthiness across various scenarios, demonstrating its potential to create reliable language models for practical applications.