R -Tuning: Instructing Large Language Models to Say ‘ I Don’t Know’ (2024)

Abstract: Large language models (LLMs) have revolutionized numerous domains with their impressive performance but still face their challenges. A predominant issue is the propensity for these models to generate non-existent facts, a concern termed hallucination. Our research is motivated by the observation that previous instruction tuning methods force the model to complete a sentence no matter whether the model knows the knowledge or not. When the question is out of the parametric knowledge, it will try to make up something and fail to indicate when it lacks knowledge. In this paper, we present a new approach called Refusal-Aware Instruction Tuning (R-Tuning). This approach is formalized by first identifying the disparity in knowledge encompassed by pre-trained parameters compared to that of instruction tuning data. Then, we construct the refusal-aware data based on the knowledge intersection, to tune LLMs to refrain from responding to questions beyond its parametric knowledge. Experimental results demonstrate R-Tuning effectively improves a model’s ability to answer known questions and refrain from answering unknown questions. Furthermore, when tested on out-of-domain datasets, the refusal ability was found to be a meta-skill that could be generalized to other tasks. Further analysis surprisingly finds that learning the uncertainty results in better calibration and an improved ability to estimate the uncertainty than uncertainty-based testing. Our code is available at https://github.com/shizhediao/R-Tuning

## introduction large language models (llms) have demonstrated remarkable performance across numerous tasks; however, they are also plagued by various issues, such as the propensity of large models to fabricate non-existent facts, a phenomenon commonly referred to as hallucination @xcite ). 1 our code is available at https://github.com/ shizhediao/r-tuning . pre-training embeds a large volume of parametric knowledge, while fine-tuning may involve knowledge that is not necessarily in the parametric knowledge. we explore the benefits of differentiating instruction tuning data based on parametric knowledge. ## refusal-aware instruction tuning in this section, we first introduce the refusal-aware instruction tuning method (r-tuning), the core idea of which is divided into two steps: the first step involves identifying and recognizing the uncertain data instances within the instruction tuning dataset, which are beyond the parametric knowledge boundary of the original model. the second step is to construct certain and uncertain dataset. then, we will detail the instruction tuning and inference extraction process. an illustration of r-tuning is shown in figure 2 . ## experimental settings in this section, we first provide an overview of the benchmark datasets and the corresponding evaluation settings. then the baseline models and the implementation details are presented in the following subsections, respectively. ## experimental results in the main experiments, we conduct single-task experiments to verify the model's refusal-aware answering ability and multi-task experiments to investigate the generalization of refusal ability. ## analysis in this section, we first introduce a variant, r-tuning-u, which adopts an unsupervised identification strategy for r-tuning. then we provide an interpretation from the uncertainty perspective for r-tuning. in addition, we verify the refusal ability on unanswerable questions, which should not receive answers from the model. more case studies are shown in ## conclusion in this paper, we propose a simple yet effective method, r-tuning, to teach llms to refuse unknown questions. it identifies the difference between instruction tuning data and parametric knowledge and splits the training data into certain and uncertain parts. then, r-tuning constructs the refusal-aware data by appending uncertainty expressions to the uncertain part. empirically, r-tuning outperforms the traditional finetuning baseline regarding ap score, illustrating a good tradeoff between prediction and confidence. r-tuning not only shows the refusal ability on in-domain data but also demonstrates such ability could be generalized to unseen tasks. it displays that refusal is a fundamental ability and could be abstracted via multi-task learning, so we call it meta-skill.