I ter A lign: Iterative Constitutional Alignment of Large Language Models (2024)

Abstract: With the rapid development of large language models (LLMs), aligning LLMs with human values and societal norms to ensure their reliability and safety has become crucial. Reinforcement learning with human feedback (RLHF) and Constitutional AI (CAI) have been proposed for LLM alignment. However, these methods require either heavy human annotations or explicitly pre-defined constitutions, which are labor-intensive and resource-consuming. To overcome these drawbacks, we study constitution-based LLM alignment and propose a data-driven constitution discovery and self-alignment framework called IterAlign. IterAlign leverages red teaming to unveil the weaknesses of an LLM and automatically discovers new constitutions using a stronger LLM. These constitutions are then used to guide self-correction of the base LLM. Such a constitution discovery pipeline can be run iteratively and automatically to discover new constitutions that specifically target the alignment gaps in the current LLM. Empirical results on several safety benchmark datasets and multiple base LLMs show that IterAlign successfully improves truthfulness, helpfulness, harmlessness and honesty, improving the LLM alignment by up to 13.5% in harmlessness.

## introduction large language models (llms) have penetrated into a wide spectrum of applications such as psychology @xcite , education @xcite , social science @xcite and scientific understanding @xcite . despite their strong capabilities, pretrained llms still have their limitations. one of the notable challenges that arise is the alignment problem, where the llm's outputs may not consistently align with human ethical standards or preferences @xcite . this misalignment can lead to biased, inaccurate or harmful content, resulting in undesired outcomes. addressing this issue not only involves refining the model's training data and training process, but also integrating human ethical guidelines and feedback into the loop to make llms safe and reliable for diverse applications. to mitigate the misalignment issue, several llm alignment algorithms have been proposed @xcite . reinforcement learning with human feedback (rlhf) @xcite and constitutional ai (cai) @xcite stand out as the representatives. rlhf addresses alignment by integrating human feedback directly into the training process, thus guiding the base model using real human responses and preferences. on the other hand, cai uses a set of pre-defined guidelines called "constitutions" that encapsulate desired ethical standards and societal norms. these guidelines direct the training and behaviors of the llms, ensuring their outputs adhere to these predefined standards, thus addressing potential ethical and alignment issues. rlhf has achieved promising performance for llm alignment @xcite , but scalability poses a significant challenge for rlhf, given the elevated costs associated with collecting and processing human feedback. in contrast, cai @xcite obviates the reliance on human feedback labels and is thus more efficient. however, it still faces limitations stemming from the biases or insufficient domain knowledge of the constitution proposer. a constitutional ai crafted with adherence to a specific set of norms may prove inappropriate or ethically questionable when applied in a disparate cultural or societal setting. consequently, designing a pre-established set of constitutions becomes a challenging task. as a result, there is an urgent need for data-driven constitution-based alignment methods that can automatically and dy-namically produce constitutions to align the target llm. we propose iteralign, a data-driven constitution discovery and alignment framework for llms. unlike existing alignment techniques, iteralign has the following appealing features. first, it does not require massive human preference data or human composed constitutions, but only takes a base llm and a red teaming dataset as input. the red teaming data is much cheaper to obtain compared to crowd-sourced human preference data. second, it does not require handwritten constitutions to be provided a priori. instead, it leverages the red teaming instances and a strong llm to discover constitutions automatically, leading to a better aligned model and a set of valuable data-driven constitutions. iteralign consists of the following modules: (1) red teaming module: iteralign first identifies the weak spots of the base llm via red teaming. three widely used red teaming datasets combined with an advanced red teaming algorithm @xcite ) is used at this stage. then, iteralign uses an oracle model like gpt-3.5-turbo foot_0 for response evaluation, identifying responses needing improvement. (2) constitution proposal module: different from existing cai methods, iteralign generates specialized constitutions from the responses identified from the previous stage using a stronger llm as a proposer. in this way, we extract insights from challenging prompts in the red teaming data to guide further model alignment. (3) constitution-induced self reflection module: we use the constitution generated by iteralign to direct the base model using in-context learning (icl) to sample new responses that have addressed the issues mentioned in the constitutions. (4) supervised fine-tuning (sft): the inductive bias contained in the new responses is injected back into the base model via sft, optimizing the causal loss for language modeling. building upon these modules, iter-align iteratively executes the above steps for interactive, automatic constitution discovery, and selfimprovement. we summarize the key contributions of this paper as follows: llms, recognizing the imperative for introducing an automatic, data-driven framework for llm alignment. • we present iteralign, a data-driven framework for llms that utilizes red teaming data and a stronger llm to automatically discover constitutions, enabling iterative llm alignment. iteralign requires minimal human effort and also circumvents potential biases and inconsistencies that might exist in human feedback, making it a practical framework for use in real industry applications. • we present comprehensive experimental results that validate the effectiveness of iteralign. empirical results on various safety benchmark datasets and multiple base llms demonstrate that iteralign successfully enhances truthfulness, helpfulness, harmlessness, and honesty, improving llm alignment by up to 13.5% in harmlessness. ## preliminary we formally define the basic components of our iterative constitutional alignment framework as follows: base model: a base llm p θ (y|x) is characterized by its initial parameters θ. this model p θ (y|x) is generic and could be or not be pre-aligned with specific ethical or preferential guidelines. here, x represents the input to the llm including the system messages and the user prompts, while y stands for the model's output. constitution: constitution c is a series of guidelines and ethical principles that have been used to inform the alignment of the base model p θ (y|x). in the original constitutional ai method @xcite , c is specified by humans as an input to the constitutional alignment framework. however, in iteralign, the principles are proposed by oracle models in a data-driven manner. to distinguish from the original constitutional ai, we notate the derived principles as c ′ . the principles c ′ outputted from iteralign serve both as a record of the alignment process and as a potential template for future alignment tasks. aligned model: an aligned llm p θ ′ (y|x) should be transformed from p θ (y|x) where θ ′ represents the newly adjusted parameters reflecting alignment with human preferences and ethical standards. such a transformation involves the adjustment of the model's parameters θ through a learning process. this process is guided by the evolving set of constitutional principles c ′ , and it aims to minimize a loss l(p θ ′ (y|x), c ′ ) that represents the deviation of the model's outputs from desired ethical alignment criteria. ## conclusion in this paper, we present a novel data-driven constitution discovery and self-alignment framework for aligning large language models. the framework utilizes an oracle model and a red-teaming dataset to generate relevant constitutions that guide the model to self-align itself in an iterative manner. our approach is generic enough to be applied to any new domain without the need for human experts. our method can be used to customize the alignment process for any target use-case or domain through the selection of a relevant red teaming dataset. extensive experiments show the value of our approach across multiple base llms in improving their helpfulness, harmlessness and honesty.