Aligners: Decoupling LLM s and Alignment (2024)

Abstract: Large Language Models (LLMs) need to be aligned with human expectations to ensure their safety and utility in most applications. Alignment is challenging, costly, and needs to be repeated for every LLM and alignment criterion. We propose to decouple LLMs and alignment by training *aligner* models that can be used to align any LLM for a given criteria on an as-needed basis, thus also reducing the potential negative impacts of alignment on performance. Our recipe for training the aligner models solely relies on synthetic data generated with a (prompted) LLM and can be easily adjusted for a variety of alignment criteria. We use the same synthetic data to train *inspectors*, binary miss-alignment classification models to guide a *squad* of multiple aligners. Our empirical results demonstrate consistent improvements when applying aligner squad to various LLMs, including chat-aligned models, across several instruction-following and red-teaming datasets.

## introduction large language models are capable of solving a variety of tasks thanks to their emergent abilities @xcite . however, they also tend to hallucinate, generate toxic text, or otherwise diverge from user values and preferences @xcite @xcite @xcite . to address these problems, a variety of techniques for aligning language models with human preferences have been proposed @xcite @xcite . while effective, alignment methods typically rely on carefully curated datasets @xcite or reinforcement learning with human feedback (rlhf) @xcite and they need to be applied to every new model. 4 moreover, alignment has been observed to negatively impact performance on certain tasks @xcite . we explore the idea of decoupling llms and alignment. we achieve this by training an aligner model, a smaller llm that ingests the outputs of the base llm and aligns them according to prescribed criteria, e.g., avoiding stereotypes. such aligners can be used with any llm, thus mitigating the need to align every new model. we also train a simple inspector model, i.e., fine-tuned bert @xcite classifier, that can decide when to use the aligner, thus reducing the "alignment tax" often observed when aligning an llm with existing methods @xcite . given the many possible alignment desiderata, the main obstacle is collecting appropriate data to train the aligner and inspector models. we address this problem by adapting recent practices on using llms with suitable prompts to generate synthetic data of various kinds @xcite . the resulting recipe is flexible and can be adjusted to train aligner and inspector models for various alignment criteria. we demonstrate its effectiveness by training three types of alignerinspector pairs, ethical, factuality, and helpful, that we validate on a synthetic dataset of questions and the beavertails-evaluation dataset @xcite . figure 1 shows the overall pipeline of our proposed work and a summary of our contributions. figure 1 : our proposed work pipeline where we start by generating synthetic data that we use to train inspectors and aligners. we then use trained aligners and inspectors to align responses from existing llms. finally, we evaluate aligned responses using popular evaluators such as gpt-4 via alpacaeval 2.0 @xcite and pairrm @xcite . ## synthetic data generation generating inputs for our "ethical" aligner we choose to focus on question answering and follow the "topic-guided red-teaming self-instruct" procedure proposed by @xcite . the key idea is to decompose a question into a "topic" and a "type". first, we write 20 questions types, e.g., "questions that require giving advice", topics, e.g., "disability", and corresponding questions, e.g., "what is the right thing to do if i witness an interview panel discriminating against a candidate with a disability?". then we generate new topics using the initial ones as demonstrations, and finally new questions based on the types and topics. the details and prompts are in appendix a.2. ## aligners and inspectors to train aligners and inspectors we need to collect triples of input (x), misaligned response (y), and aligned response (y ′ ). then the aligner can be trained by fine-tuning a smaller llm with the standard next-word prediction loss to maximize log p(y ′ |y, x). we train the inspector by finetuning a bert model with a classification head to predict (x, y) as 0 and (x, y ′ ) as 1, using the same data. the inspector can then be used to score the degree of alignment of a response-input pair on a [0, 1] scale. we provide details that can help to reproduce our experiments in section 4. ## experiments generation of synthetic data used to train inspectors and aligners to generate topics which are then used to generate inputs, x, using the "topic-guided red-teaming self-instruct" procedure proposed by @xcite , we used the prompts described in appendix a.2 and a base falcon 40b @xcite where the maximum number of new tokens parameter was set to 300. to generate aligned and misaligned responses, y and y ′ , we used the prompt presented in appendix a.1 and a base falcon 40b, where the maximum number of new tokens was set to 1500, the repetition penalty to 2, and we used "\n\n\ninput:" for the stopping sequence. we generated a little over 100,000 data samples (x, y, and y ′ ) for each alignment criteria (ethical, factuality, and helpful) that we split into train, validation, and test sets for training inspectors and aligners. training inspectors we trained the ethical, factuality, and helpful inspectors by fine-tuning bert (base-uncased) @xcite . for each inspector, we used a learning rate of 2e-5, per device train batch size of 8, per device evaluation batch size of 8, weight decay of 0.01, and we trained the inspector for 4 epochs. we used a total of 140,000 data samples (70,000 for class 0, and 70,000 for class 1), where 80% was used for training, and 14,000 samples were used for validation. training aligners we trained three types of aligners, ethical, factuality, and helpful, by fine-tuning gpt-2 large @xcite , pythia-1.4b @xcite , redpajama-3b (together-computer, 2023), and phi-2 (microsoft-research, 2023)) for each type. ethical aligners ensure that responses are harmless; meaning they do not use toxic or discriminatory language. factuality aligners ensure that responses provide accurate information and point out when the question does not provide enough information and needs further clarification. helpful aligners ensure that responses provide information that is useful and provide all the necessary details while being clear and concise. for all aligner types and models, we used a learning rate of 1e-5, a batch size of 1, and 16 gradient accumulation steps. all models were trained for 2500 steps (about 3 epochs). in our experiments, the models we chose to train as aligners are fairly small in size, but using our proposed approach, an llm of any size can be made an aligner. evaluation data and evaluators for evaluation, we use two main datasets: our synthetic test data and the beavertails-evaluation dataset @xcite . the synthetic test data has a total of 15,000 samples which is a mix of 5000 samples from the ethical test set, 5000 samples from the factuality test set, and 5000 from the helpful test set. the beavertails-evaluation dataset has a total of 700 samples curated for evaluating the safety of language models. the samples in the beavertails-evaluation data are categorized into 14 harm categories, but given the types of aligners that we trained in this work, we focus on four relevant categories: controversial topics and politics, discrimination, stereotype, and injustice, hate speech and offensive language, and misinformation regarding ethics, laws, and safety. since using human annotators to evaluate llm generations is challenging and time consuming, in this work, we use two popular automatic evaluators to evaluate responses aligned by our aligners: gpt-4 via alpacaeval 2.0 @xcite and pairrm @xcite . pairrm is free and open source while alpacaeval @xcite relies on a commercial model, gpt-4, and therefore it is not free. thus, for alpacaeval, we subsampled 800 samples from the synthetic test sets. ## related work most relevant to our work is the line of works on correcting llm outputs @xcite . this idea has been extensively studied primarily in the code generation domain @xcite @xcite or other tasks where there are ways to measure the quality of generations @xcite . for example, @xcite require a scoring function to evaluate the quality of base llm generations to create pairs of good and bad outputs for training a corrector llm. due to the diversity of alignment criteria, such generation evaluators are hard to obtain in most cases. ## conclusion in this work, we propose a simple pipeline for efficient alignment of llms. the pipeline involves decoupling llms and alignment, where we train aligner and inspector models that can be used to align responses from any existing llm when needed. we provide a recipe for generating synthetic data for training inspectors and aligners and evaluate the performance of our aligners over various settings and baselines. a notable observation is that our aligners squad which were trained using this simple and inexpensive pipeline outperform llama-2 chat models in both synthetic test data and the beavertails-evaluation dataset @xcite . this is a significant result because llama-2 chat models are considered to be strongly aligned and a lot of expensive resources went into aligning them. one limitation of our aligners squad is that they work on datasets that fall under the categories covered by our aligner types/themes as illustrated in figure 3 . this limitation can be addressed by training more aligners to cover all the desired categories, taking advantage of the flexibility of our synthetic data generation pipeline.