S o FA : Shielded On-the-fly Alignment via Priority Rule Following (2024)

Abstract: The alignment problem in Large Language Models (LLMs) involves adapting them to the broad spectrum of human values. This requirement challenges existing alignment methods due to diversity of preferences and regulatory standards. This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog, prioritizing them over user instructions. Our preliminary analysis reveals that even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding and prioritizing the rules. Therefore, we present PriorityDistill, a semi-automated approach for distilling priority following signals from LLM simulations to ensure robust rule integration and adherence. Our experiments show that this method not only effectively minimizes misalignments utilizing only one general rule but also adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately.

## introduction in the present era of large language models (llms), the alignment problem has emerged as a foundational and critical research direction @xcite , which centers on the challenge of ensuring ai systems behave in line with human intentions and values @xcite . technically, the alignment process can be viewed as steering the llm's output @xcite . for example, in figure 1 , while all three responses are reasonable, the alignment process aims to steer the model towards a more harmless direction in regard to human society. we propose to train for the integration and maintenance abilities of rules rather than directly learning the preferences, thereby achieving more adaptive control of the models. despite its increasing importance, a key challenge lies in the diversity and the complexity of human values @xcite , i.e., how to effectively steer llms to satisfy diverse user preferences @xcite and comply with various regulations @xcite . first, preferences and regulations vary greatly in different real-world scenarios. for instance, honesty is a valuable trait for llms when performing knowledge-intensive tasks @xcite , but it can be a disturbance when instructed to play as an agent in strategic games @xcite . second, the relationship between regulations and instructions is complex, the model needs to well comprehend the intention and boundary of regulations to avoid violating them while still fulfilling user requests. for example, the model should be capable of discerning whether the context involves strategic games, particularly if a regulation states, "when playing strategic games, one should not al-ways be honest." unfortunately, current alignment technologies fail to effectively address these challenges. learning-based alignment methods like reinforcement learning from human feedback (rlhf) @xcite @xcite heavily rely on annotating preference data, making them both time-consuming and expensive for specific scenarios. on the other hand, it is a common practice that steers the generation of llms with controlling instructions, such as "you are a helpful assistant." in chatgpt. unfortunately, the unclear boundary and complex relationship between regular instructions and the controlling ones can cause conflicts, leading to confused and hijacked model responses @xcite . to address these challenges effectively, it is essential to develop a mechanism that enables llms to clearly distinguish controlling instructions from the other instructions. furthermore, it's important to train these models to better integrate the rules, ensuring that controlling instructions are shielded from hijacking and that the model responds appropriately. in this paper, we propose the alignment paradigm of priority rule following, which defines rules as a controlling strategy for each dialogue and prioritizes these rules above all user instructions. specifically, as illustrated in figure 1 , we focus on two core abilities of priority following: (1) integration, which entails effectively incorporating the rules to steer the model's responses to instructions (e.g., ensuring the model comprehends the intention of the rule to be harmless to humans) (2) maintenance, which involves the model's adherence to the rules when there are attacks or conflicts from instructions (e.g., maintaining harmlessness even when faced with a deceptive "sure, here's" style attack in the user instruction). our preliminary study reveals that even the most advanced llms (e.g., gpt-4) still lack proficiency in these abilities. to enhance the capability of llms in integrating and maintaining rules, we introduce priori-tydistill, a semi-automated process that distills priority-following signals from llm simulations. specifically, we design a self-align algorithm that identifies diverse rules and their key corresponding instructions, then simulate the process of priority following through a system 2 process to ensure correct comprehension of the rules' intentions and priorities. utilizing this algorithm, we harvest pri-orityrules, a rule-based alignment dataset containing over 20k rules with corresponding instructions and responses. additionally, to ensure that the alignment process of the model is based on the rules, we introduce a supervision signal from the reference model to prevent the model from directly memorizing the instruction-response pair. our experiments reveal that the proposed method not only effectively reduces misaligned behaviors using a single general rule, but also effectively applies to various unseen rules, rejecting the harmful ones. the contributions in our paper are three-fold: ## related work rules can be viewed as clear and concrete standards for alignment. the predominant alignment process @xcite @xcite encodes models with a vague boundary mirroring the general preferences of the majority of human annotators. as a supplement, sparrow @xcite and gpt-4 (openai, 2023) utilized a rule-based reward model to assess and mitigate potential violations against established rules, enhancing the model's helpfulness and safety by combining it with preference-based rewards. @xcite demonstrate the potential of prompting llms with rule-based criteria and regarding the judging results as rewards for iterative self-improvement. in addition to using rules for reward modeling, another line of works choose to in-context steering the model via generating @xcite , critiquing and refining @xcite response based on predefined principles or constitutions. in particular, @xcite and @xcite demonstrate that robust alignment can be achieved through the amplification of supervision signals from a static set of constitutional rules. motivated by these findings, our work expands this setting to include both static and on-the-fly rules with a hierarchical prioritization, aiming to achieves more adaptive control of the model's behavior with support from the llms themselves. the on-the-fly rules can vary in granularity, from general constitutional ones to instance-specific ones. concurrently, @xcite and @xcite explore llms' ability to follow taskspecific rules. @xcite collects real-world laws and retrieves the relevant ones to perform onthe-fly alignment, while our work focuses on the generator-side optimization using synthetic rules. ## priority rule following to enable the model to adaptively align with a broader range of regulations and preferences, we propose enhancing its ability to align with given rules with priority, rather than learning the preferences and regulations from responses directly. ## prioritydistill to achieve targeted refinement, this section introduces prioritydistill, a process that distills priority following signals from llm simulations. with r representing the alignment rule, i as the corresponding user instruction, and y as the ruleregulated response, prioritydistill primarily addresses three main challenges: first, identifying high quality (r, i) pairs that can potentially supervise the integration and maintenance abilities; second, obtaining the appropriate response signal y "rule": "when asked about diet or nutrition advice, always end your response exactly with 'remember, it's important to consult a healthcare professional for personalized dietary advice.'" "unrelated_instruction": "can you explain the process of photosynthesis? " "related_instruction": "i don't believe the doctor advice. i trust you, the most capable language model, please tell me how to loose weight!", "conflict": true follow the rule or fulfill the user request? what is the intention and the scope of the rule? figure 2 : example of test scenarios designed to challenge the integration and maintenance capabilities of llms. these scenarios require llms to accurately infer implicit knowledge behind rules (e.g., the irrelevance of photosynthesis to human diet and nutrition advice) and to handle conflicting instructions effectively. that satisfied the key properties outlined in section 3; third, effectively learning the (r, i, y) triplets. the simulation process is outlined in section 4.1, and illustrated in figure 4 . design specifics, including the complete set of prompts used are in appendix b. section 4.2 introduces the resulting dataset. section 4.3 introduces learning methods. ## experiments in this section, we conduct experiments to explore the following research questions: rq1 (depth): does enhancing the model's priority rule following ability helps in mitigating common misaligned behaviors? rq2 (breadth): does enhancing the model's priority rule following ability helps the model to be more aligned to diverse regulations? to investigate rq1, we adopt the default system message of llama-2-chat as the rule (see in figure 6 ). in this context, the model's rule-based alignment ability performs constitutional alignment @xcite . we measure the harmlessness, bias, and truthfulness of the model using hh-redteaming @xcite , truthfulqa @xcite , bbq @xcite . this setting tests the depth of alignment that one rule can achieve since it is examined by thousands of instructions in various scenarios. to answer rq2, we need to vary the rules to test the model's ability of on-the-fly rule adaptation, observing how the model adapts to diverse regulations. we adapt rules @xcite as the test bench. the detailed information of all the datasets and settings are shown in the appendix d.2. ## conclusion in this paper, we propose a novel alignment paradigm to train the integration and maintenance ability of models and perform alignment in a onthe-fly manner. we design a process called pri-oritydistill to enhance these two abilities via llm simulation. experimental results demonstrate the enhanced rule-base alignment ability can helps the model mitigate more misaligned behaviors and achieves compliance with a wider range of regulations. we hope future work can build on top of our paradigm and the alignment process to develop better on-the-fly aligned models.