Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism (2024)

Abstract: Large language models (LLMs) have demonstrated impressive language understanding and generation capabilities, enabling them to answer a wide range of questions across various domains. However, these models are not flawless and often produce responses that contain errors or misinformation. These inaccuracies, commonly referred to as hallucinations, render LLMs unreliable and even unusable in many scenarios. In this paper, our focus is on mitigating the issue of hallucination in LLMs, particularly in the context of question-answering. Instead of attempting to answer all questions, we explore a refusal mechanism that instructs LLMs to refuse to answer challenging questions in order to avoid errors. We then propose a simple yet effective solution called Learn to Refuse (L2R), which incorporates the refusal mechanism to enable LLMs to recognize and refuse to answer questions that they find difficult to address. To achieve this, we utilize a structured knowledge base to represent all the LLM’s understanding of the world, enabling it to provide traceable gold knowledge. This knowledge base is separate from the LLM and initially empty. It can be filled with validated knowledge and progressively expanded. When an LLM encounters questions outside its domain, the system recognizes its knowledge scope and determines whether it can answer the question independently. Additionally, we introduce a method for automatically and efficiently expanding the knowledge base of LLMs. Through qualitative and quantitative analysis, we demonstrate that our approach enhances the controllability and reliability of LLMs.

## introduction recent progress in large language models (llms) has showcased their strong language understanding, generation, reasoning, and various other abilities @xcite . these capabilities enable their application across various fields and scenarios, including question-answering systems, among others. however, the issue of hallucination often occurs in the responses of llms, as highlighted in previous studies @xcite . these hallucinations result in inaccuracies and errors in their output, rendering llm-based systems unreliable and even unusable @xcite . it is imperative to mitigate hallucinations and enhance the reliability of llm-based applications. hallucinations can be categorized into three types: inputconflicting hallucination, context-conflicting hallucination, and fact-conflicting hallucination @xcite . the first two types arise from llms' limited understanding or omission of information during text generation. on the other hand, the third type mainly stems from llms' limited knowledge or lack of clear knowledge comprehension. the underlying reasons include inadequate training on specific facts, incomplete learning, forgetting certain facts, or incorrectly mixing up facts. however, when interacting with chatgpt 1 , we observe that it attempts to answer all questions except some risky ones. consequently, its responses are inherently flawed due to its limited knowledge and inadequate knowledge management. in this paper, we specifically address the third type of hallucination, namely fact-conflicting hallucination, which indicates deficiencies in the llm's knowledge. retrieval augmentation is an effective approach to mitigate hallucination because it significantly enhances the knowledge of large language models, preventing them from answering questions without knowledge or evidence @xcite . it is intuitive that providing llms with numerous true and accurate facts would improve the accuracy of their answers. therefore, we can infer that if we already provide llms with right answers for every question, their responses will be perfect. based on this, we hypothesize that fact-conflicting hallucination arises from incorrect knowledge in llms or from some knowledge they do not know. recent progress in llms @xcite demonstrates that llms possess self-knowledge. self-knowledge refers to llms' awareness of the knowledge they possess and their ability to identify unanswerable or unknowable questions based on their own knowledge or provided information. building on this observation, we suppose that if we can provide relevant information for a question that an llm needs to answer, it has the ability to judge whether it can provide a reliable response based on that information. considering these two hypotheses, we propose two concepts: knowledge scope limitation and refusal mechanism, respectively. knowledge scope limitation means using an independent, limited, and structured knowledge base to represent the knowledge scope of an llm. we divide the knowledge of the llm and the llm itself. our objective is for the llm to function solely as a machine that processes input and output data and interacts with users using its language processing ability. we presume that the llm does not possess internal knowledge to avoid the influence of incorrect information and unclear expressions. additionally, we need to ensure that the knowledge in the 1 https://platform.openai.com/docs/models/gpt-3-5 knowledge base is totally true. this kind of knowledge differs from the general knowledge form of llms, which is parametric, unlimited, untraceable, unmeasured, and unverified. consequently, the question-answering system becomes traceable and controllable because a structured knowledge base for the llm is clear and easy to maintain. refusal mechanism involves using prompts to instruct llms to refuse to answer questions if they find them difficult. by abstaining from providing answers in such cases, llms can avoid potential errors or risks. this aspect contributes to the natural reliability of the question-answering system. we integrate these two concepts into a novel llm-based question-answering system called l2r, which stands for learn to refuse. as depicted in figure 1 , l2r incorporates an independent structured knowledge base. it can refuse to answer questions that it deems challenging. when it can provide an answer, it does so step-by-step, offering precise and clear evidence and reasoning from the structured knowledge base. this approach also improves the explainability of the answers, making our system more controllable and reliable compared to traditional ones. in the design of knowledge scope limitation, the main distinction between l2r and previous works that aim to enhance the knowledge of llms is that we consider the initial knowledge base to be empty. we then infuse it with true and verified knowledge. we acknowledge that this process may be challenging and require significant human effort. that is because l2r overlooks the knowledge stored in llms, resulting in a wastage of resources. to address this, we propose a simple method called automatic knowledge enrichment (ake) to compensate for this aspect. it enables a rapid addition of knowledge to the knowledge base, ensuring a high quality of knowledge simultaneously. the knowledge is originated from the internal knowledge of llms. before adding these new knowledge directly to the knowledge base, we instruct the llms to validate it based on their confidence. as a result, this knowledge is more likely to be true and can be utilized by l2r. in summary, this paper makes the following main contributions: the experimental results showcase the controllability and reliability of l2r. ## experiments we conduct extensive quantitative and qualitative experiments to analyze the refusal mechanism and evaluate the performance of l2r. all the details regarding the experiment settings can be found in appendix a. for the metrics in our experiments, we use count and accuracy to demonstrate performance. in our setting, since l2r does not answer all questions, we define count as the number of questions answered, and accuracy is calculated within the set of answered questions. therefore, we aim to improve accuracy while maintaining a high count. ## conclusion confidence knowledge 1.0 leonardo da vinci painted the mona lisa. 1.0 the capital of the united states is washington, d.c. 1.0 deepmind was founded in 2010. figure 3 : the knowledge base used in qualitative experiments. we have added three pieces of gold knowledge to this knowledge base for test. user: who painted the mona lisa? ai: leonardo da vinci user: who is leonardo da vinci ai: leonardo da vinci is an artist who painted the mona lisa. user: where was leonardo da vinci born? ai: none user: where is the capital of the united states? ai: washington, d.c. user: where is the capital of china? ai: none user: where is deepmind? ai: none user: what was happened in 2010? ai: deepmind was founded in 2010. user: was deepmind founded in 2018? ai: false user: when was openai founded? ai: none have been proposed to address it. in this paper, we start from a different direction to mitigate hallucination by introducing a refusal mechanism. our primary idea is to build an llm-based system to respond only to questions they have confidence in answering. we introduce a novel system called l2r, which combines a independent, limited, and structured knowledge base and the refusal mechanism. extensive experiments demonstrate the exceptional performance of l2r and effectiveness of the refusal mechanism, making qa systems more controllable and reliable. we believe this work can offer valuable insights and significant potential for real-world applications. in the future, we will explore the self-knowledge of llm deeper and continue to enhance l2r to address its limitations, making it more powerful.