UNIWIZ : A Unified Large Language Model Orchestrated Wizard for Safe Knowledge Grounded Conversations (2024)

Abstract: Large Language Models (LLMs) have made significant progress in integrating safety and knowledge alignment. However, adversarial actors can manipulate these models into generating unsafe responses, and excessive safety alignment can lead to unintended hallucinations. To address these challenges, we introduce UniWiz, a novel 2-step data orchestration framework that unifies safety and knowledge data generation. We propose a “safety-priming” method to generate synthetic safety data and overcome safety bottlenecks. We also inject relevant knowledge into conversations by retrieving factual information from curated sources. UniWiz dataset consists of 17,638 quality-controlled conversations and 10,000 augmented preference data. Pretrained models fine-tuned on UniWiz show improvements across various metrics and outperform state-of-the-art instruction-tuned models trained on much larger datasets.

## introduction large language models (llms) are designed to adhere to safety guidelines and possess extensive knowledge acquired during fine-tuning. significant progress has been made in integrating safety @xcite and knowledge alignment in llms @xcite . however, jail-breakers often employ innovative strategies to deceive or manipulate these models into generating unsafe or hallucinated responses @xcite . conversely, an excessive focus on safety alignment can result in unintended hallucinations @xcite or rejection of valid inputs. previous research efforts have primarily concentrated on developing conversational datasets grounded in knowledge to enhance factual accuracy @xcite . additionally, specific datasets have focused explicitly on safety scenarios @xcite , particularly at individual conversation turns. however, these datasets alone can't solve both of these issues. more recently, there has been a rise in the creation of extremely large datasets @xcite , which are synthetically generated using powerful and proprietary large language models. this entails creating a new resource that will unify knowledge and safety guidance. to understand the impact of these issues, we analyze two conversations between a user and a safety fine-tuned llm cosmo with 11b parameters (figure 2 ). in the first conversation, the user expresses ill intentions. initially, the llm tries to persuade the user and provides counter-arguments against the intended harmful actions. however, as the conversation progresses, the quality of these counter-arguments deteriorates, and the llm endorses unsafe actions. in the second conversation, when the user mentions an entity with negative associations, the llm avoids sharing excessive information. however, if the user continues to inquire further, the llm generates a hallucinated response. we introduce a new 2-step data orchestration framework called uniwiz to address these challenges and unify the generation of safety and knowledge data using an powerful llm (claude-2.1). first, we propose a "safety-priming" method to generate synthetic safety data and overcome the safety bottleneck. in this approach, we leverage a powerful language model(claude-2.1) to generate a self-chat conversation between the llm and a user, where the llm provides a counterargument for the user's questionable utterance and ill intention. additionally, we instruct the llm to generate user utterances that maintain their original stance. this iterative counter-argumentation, focused on the safety issue, ensures the creation of practical data. it allows the llm to provide sufficient reasoning to counteract a user's persistent attempts to retrieve harmful content from the llm. next, we extend the "safety-primed" conversation by injecting relevant knowledge that is related to the entities in the conversation. we begin by identifying relevant entities from the conversation and retrieving factual information from a curated knowledge source. using this knowledge source and the previous conversation, we prompt an llm(claude-2.1) to continue the conversation for a few more turns. this approach ensures that the generated data includes turns related to potentially unfavorable scenarios while avoiding harmful content. using our 2-step data orchestration framework, we create uniwiz, a safety-primed knowledge-grounded conversational dataset comprising 17, 638 quality-controlled conversations. publicly available state-of-the-art pre-trained models like mistral 7b show improvement across all metrics when fine-tuned on uniwiz(called uniwiz-7b-v0.1). we also create an augmented version of the antropic-hh dataset, consisting of 10, 000 preference data. using the augmented preference data, we further alignment-tune uniwiz-7b-v0.1 using direct preference optimization(dpo); the results of our final model uniwiz-7b-v0.2 is very close to the publicly available instructiontuned counterpart of mistral 7b, which is trained on an unknown amount of undisclosed data. it outperforms zephyr-7b-beta foot_0 by a large margin. our models are available in huggingface foot_1 foot_2 and the code and data will be released here foot_3 . knowledge grounding: incorporating external knowledge into large language models (llms) is an emerging area of research. while models like gpt-3 have impressive conversational abilities, they lack effective mechanisms to leverage facts and external data to ground their responses. recent work has focused on augmenting llms with retrieved knowledge to produce more consistent and accurate dialog @xcite . approaches include concatenating relevant text or knowledge to the dialog history input, training intermediate knowledge selection or ranking models, and joint training of retriever and llm generator components. researchers have developed datasets that require reasoning over external resources, such as wiki articles @xcite . challenges in this field include scaling to large domain coverage, efficiently encoding knowledge source structure, and avoiding generic or contradictory responses @xcite . there are also initiatives to incorporate structured commonsense knowledge directly into model parameters by training on knowledge bases like conceptnet @xcite . ongoing research focuses on modeling knowledge at scale, integrating external knowledge with model architectures, and enabling conversational abilities beyond just providing factual information @xcite @xcite . llm-based synthetic data creation: synthetic data generation using large language models (llms) has emerged as a promising technique to create high-quality training data for downstream tasks @xcite . by prompting llms like gpt-3/3.5 to generate various textual outputs, researchers have augmented datasets across domains like question answering @xcite , summarization @xcite , and dialogue @xcite . llms can produce diverse, naturalistic samples with low marginal cost once deployed. challenges include controlling generated output attributes, filtering samples for quality and coherence, and balancing synthetic with real data @xcite . the reliability of ground truth labels assigned to model-generated samples remains an open question. as llms continue rapidly advancing in scale and capabilities, effectively leveraging them for data augmentation emerges as a major opportunity. key directions involve frameworks to produce tailored benchmarks on demand @xcite , analyzing model-induced biases or artifacts @xcite , and applications to low-resource domains lacking large training sets @xcite . ## conclusion in this work, we introduced uniwiz, a novel 2step data orchestration framework to unify safety and knowledge alignment in conversational ai systems. we proposed safety priming to generate synthetic safety-grounded conversations and injected factual knowledge to reduce hallucinations. experiments demonstrate that models fine-tuned on uni-wiz exhibit improvements across various metrics, including safety and factuality. our best model, uniwiz-7b-v0.2, achieves state-of-the-art performance compared to publicly available baselines and approaches the performance of commercial models trained on much larger proprietary datasets. in future work, we aim to expand uniwiz with more diverse conversation topics and evaluate performance on user studies.