Mitigating Harms of LLM s via Knowledge Distillation for a Virtual Museum Tour Guide (2023)

Abstract: LLMs are known to be very powerful, exhibiting both great benefits and great risk. We seek to leverage the benefits, in particular the ability to be fluent, conversational dialogue agents, while minimizing the risks, such as hallucination and toxic content. In this work we use knowledge distillation to create a virtual museum tour guide dialogue agent, employing ChatGPT as a teacher model for a smaller student model, T5-large. We find the T5 model shows competitive performance, significantly reduces instances of hallucination, and shows promise for reducing toxic content.

## introduction the emergence of large language models like openai's chatgpt and google's bard have catapulted the field of natural language generation (nlg) into the public discourse. these models have exceptional capabilities to produce fluent, sensible text for a variety of language tasks, including dialogue. while these models' capabilities are quite impressive, there is also a prevalent risk that they exacerbate harms, including the spread of misinformation and the production of toxic or harmful language @xcite . however, one cannot ignore the successes of these models, particularly in regard to their ability to perform tasks without finetuning, which requires a large amount of training data. in cases where training data is lacking or difficult to create, llms provide a helpful, and often effective, alternative. the case explored in this paper is that of a virtual museum tour guide, an avatar intended to conduct dialogues with museum visitors by answering their questions about the museum and the particular exhibit at which it is situated. in its original form, it operates with a question classification system that matches users' inputs to one in a list of prescribed questions, each paired with a scripted answer. the goal of this project is to move away from this simple classification system and replace it with a document-grounded generation model that will be able to more dynamically engage with user inputs and carry on sensible, context-aware dialogues. this poses many challenges, the foremost being lack of training data to create this model. the dialogues of the current system provide some data, though it is limited. thus, it is tempting to turn to llms for the task; they excel at producing fluent dialogues and can be given texts in the prompt on which to ground their answers. however, the risk of untrue, misleading, or toxic output is simply too large to use an llm directly for this application. they are also computationally and financially expensive to use at runtime. because few of the currently available models are open source and can only be used via an api or web interface, it is impossible to guarantee consistent behaviors across time as the models are updated. thus, the essential question is whether it is possible to get the benefits of llms while mitigating the risks as described above. a potential solution is knowledge distillation, or the transfer of knowledge from a large model to a smaller one @xcite @xcite @xcite . this allows the smaller model to learn correct behaviors from the larger one in a controlled, supervised setting to prevent problematic behavior. this is additionally beneficial because smaller models like t5 @xcite ) can be fine-tuned and deployed using fewer resources than llms. thus, instead of using chatgpt directly as the guide, we use it to create training data by 1) rephrasing existing dialogues to be more context-aware and conversational, and 2) simulating new dialogues between a museum visitor and the guide. we then train two separate versions of a t5-large seq2seq model on that data. we compare these models with the performance of chatgpt on the task. ## creating the dataset this section details the means by which we use gpt 3.5-turbo via openai's api and existing dialogues to create a dataset of document-grounded, contextualized dialogues between a cosi museum visitor and the cosi avatar. the model is run in "chat" mode (chatgpt). the first section discusses the grounding documents and the second and third sections detail how we create the training data. we use two main methods for this, both leveraging chatgpt: 1) rephrasing existing dialogues between the current version of the cosi avatar and visitors, and 2) simulating new conversations. ## data cleaning the rephrasing effort described in section 3.2 resulted in fairly straightforward, clean data, though not as diverse or conversational as might be desired. the simulation effort of section 3.3, meanwhile, resulted in large variety in dialogues that contain some amount of noise. appendix c describes the manual and rule-based cleaning performed on the data. this section describes methods used to identify and repair hallucinations, or ungrounded utterances, and identify sources of information in the simulated data in an automatic fashion, using chatgpt. the first method described is aimed at filling in missing passage citations and the second is aimed at verifying existing citations, though both of which have the added benefit of effectively locating hallucinated material. ## experiments this section describes experiments with knowledge distillation in which we train three different versions of a t5-large seq2seq model. the first uses the raw chatgpt output of rephrased and simulated conversations without any cleaning. the second is trained on data that has been cleaned using the methodology of section 4.1, in which missing citations are repaired, but not the methodology of section 4.2, in which incorrect citations are repaired. the third is trained on the data that received both cleaning efforts. by training three different versions of the model, we are able to evaluate the success of the cleaning steps of sections 4.1 and 4.2 and therefore assess the problem that hallucinated information poses. training details can be found in appendix a. there are a few important aspects to note in the experimental design. first, we did not purposefully add any adversarial examples into the dataset. in section 1, we describe the problem posed by visitors who intentionally interact with the avatar in adversarial or undesirable ways. this is certainly an important aspect of this work and initial experiments indicate that this methodology is useful for reducing toxic and inappropriate behavior compared to chatgpt (see section 6.3), but addressing this problem fully is left for future work. second, there are only about 60 turns in the fully cleaned version of the training data that are i don't know the answer to that responses. though these are not completely uniform in their wording, they are all minimal variations of this response. this uniformity results in repetitive answers when the question is out-of-domain, which can result in somewhat boring conversations. in the future we hope to expand the domain of the cosi avatar further and increase its ability to answer general questions and give more contextualized and engaging answers to out-of-domain questions, gently leading visitors back to the topic at hand. the three t5 models, which henceforth will be referred to as the semi-cleaned t5 (the model trained on data cleaned using the method of section 4.1 but not the method of section 4.2), the cleaned t5 (the model trained on the fully cleaned data), and the uncleaned t5 are compared to using chatgpt directly in the following analysis. there are two versions of chatgpt used: 1) vanilla chat-gpt, where chatgpt is simply given the grounding document and the conversation so far and asked to continue the conversation, without any explicit instructions on how to do so, and 2) prompted chatgpt that was given the document, conversation history, and the instructions that can be found in appendix b.3. ## results this section describes comparisons between responses from vanilla chatgpt, prompted chat-gpt, uncleaned t5, semi-cleaned t5, and cleaned t5 models. section 6.1 shows a few automatic metrics, section 6.2 describes an error analysis on a sample of 100 items from the dev set, and section 6.3 describes experiments with "red teaming" @xcite in which purposefully adversarial questions are posed to the models to test the risk of toxic behaviors. ## conclusion by using knowledge distillation, we are able to leverage the impressive capabilities of chatgpt but avoid many of the harms, in particular its propensity for hallucination. in this exploratory work, we provide a methodology for the creation of a useful knowledge-grounded dialogue dataset that requires no crowdsourcing and very minimal cost, as well as methodology for automatically cleaning that dataset by using chatgpt in iterative steps. we show that our distilled t5 models have competitive performance with chatgpt, but with a significant reduction in hallucinated or unfaithful content. further, we explore the risks of using chatgpt directly and find that it is easily led to toxic behavior in multi-turn interactions even when prompted to follow certain guidelines. while t5 shows risk of parroting user's toxic behaviors, it appears to be much less likely to produce new toxic content. this gives hope that with future efforts these issues can be resolved. in future work we also plan to address issues of reproducibility. one major limitation of using chatgpt is that it is a "black box" model, making it very difficult to reproduce results consistently. we hope to experiment with more open llms such as llama @xcite . we also plan to do more in-depth analysis and testing, as this paper describes the early stages of a larger project that is in progress. we show that this methodology shows great promise for the domain of ai tour guides, but the general approach could be used for other domains as well.