Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models (2024)

Abstract: Large Language Models (LLMs) exhibit impressive capabilities but also present risks such as biased content generation and privacy issues. One of the current alignment techniques includes principle-driven integration, but it faces challenges arising from the imprecision of manually crafted rules and inadequate risk perception in models without safety training. To address these, we introduce Guide-Align, a two-stage approach. Initially, a safety-trained model identifies potential risks and formulates specific guidelines for various inputs, establishing a comprehensive library of guidelines and a model for input-guidelines retrieval. Subsequently, the retrieval model correlates new inputs with relevant guidelines, which guide LLMs in response generation to ensure safe and high-quality outputs, thereby aligning with human values. An additional optional stage involves fine-tuning a model with well-aligned datasets generated through the process implemented in the second stage.Our method customizes guidelines to accommodate diverse inputs, thereby enhancing the fine-grainedness and comprehensiveness of the guideline library. Furthermore, it incorporates safety expertise from a safety-trained LLM through a lightweight retrieval model.We evaluate our approach on three benchmarks, demonstrating significant improvements in LLM security and quality. Notably, our fine-tuned model, Labrador, even at 13 billion parameters, outperforms GPT-3.5-turbo and surpasses GPT-4 in alignment capabilities.

## introduction large language models (llms) demonstrate impressive capabilities but also pose significant risks, such as generating biased, discriminatory, or privacy-infringing content. have used manually crafted rules to align model outputs with human values. despite some success, these rules have significant limitations: a. limited matching precision: these rules are typically formulated to prioritize generality over specificity to balance applicability and labor costs. consequently, they offer less precise guidance than those tailored for nuanced, fine-grained matching. b. limited comprehensiveness: predicting all potential deployment scenarios is challenging, making it hard to create rules that effectively cover diverse situations. moreover, inappropriate rules can introduce noise and unintended side effects, undermining the model's performance. insufficient risk perception models without safety training often lack risk perception, making them ineffective at identifying insecure information within a context. as a result, they are more likely to generate insecure outputs from diverse inputs. current research approaches, whether they involve prompting ai for self-critique and revision @xcite or enabling the model to select and adhere to rules(sun et al., 2023b), rely on a basic level of security risk awareness of the model. however, models without safety training face challenges in performing these tasks autonomously, particularly without external aids like in-context learning examples. to address the above issues, we propose a novel method called guide-align. this approach autonomously generates a comprehensive library of detailed guidelines tailored to diverse inputs. we also train a retrieval model to associate inputs with relevant guidelines. these resources are then used to guide llms during response generation, ensuring safe and high-quality outputs, thereby aligning with human values. as an optional step, we apply our method to open-source input data, creating an alignment dataset. this dataset is used to fine-tune a large model, which we call labrador. we foster a more fine-grained match between inputs and guidelines by tailoring specific guidelines for each input. additionally, we enhance the comprehensiveness of the guideline repository by introducing diversity in inputs while constructing the guideline library. these dual strategies enable our method to overcome limitations associated with manually written rules. furthermore, we address the safety awareness deficit in the model lacking safety training by integrating the security expertise derived from a more robust, safety-trained llm through a lightweight retrieval model. in summary, our contributions are as follows: ## our method: guide-align the guide-align framework encompasses two phases: guideline library construction and retrieval model training, followed by inference. additionally, we introduce fine-tuning as an optional phase. the framework of guide-align is shown as figure 1 , and we describe each step in detail. ## conclusion we propose guide-align, a guideline-oriented method to augment the safety and quality of llms. we construct a comprehensive and detailed guideline library complemented by a retrieval model that matches inputs with appropriate guidelines. these elements constitute a plug-and-play module that enables various llms to achieve alignment with minimal expenditure. experiments across diverse datasets consistently illustrate that our methodology bolsters model safety while preserving or enhancing performance in other dimensions. furthermore, we curate an alignment dataset and employ it to fine-tune a secure and dependable model, denoted as labrador.