A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques (2024)

Abstract: Large language models are first pre-trained on trillions of tokens and then instruction-tuned or aligned to specific preferences. While pre-training remains out of reach for most researchers due to the compute required, fine-tuning has become affordable thanks to parameter-efficient methods such as LoRA and QLoRA. Alignment is known to be sensitive to the many factors involved, including the quantity and quality of data, the alignment method, and the adapter rank. However, there has not yet been an extensive study of their effect on downstream performance. To address this gap, we conduct an in-depth investigation of the impact of popular choices for three crucial axes: (i) the alignment dataset (HH-RLHF and BeaverTails), (ii) the alignment technique (SFT and DPO), and (iii) the model (LLaMA-1, Vicuna-v1.3, Mistral-7b, and Mistral-7b-Instruct). Our extensive setup spanning over 300 experiments reveals consistent trends and unexpected findings. We observe how more informative data helps with preference alignment, cases where supervised fine-tuning outperforms preference optimization, and how aligning to a distinct preference boosts performance on downstream tasks. Through our in-depth analyses, we put forward key guidelines to help researchers perform more effective parameter-efficient LLM alignment.

## introduction large language models (llms) have achieved human-like performance across various tasks such as summarization, commonsense reasoning, and open-ended generation @xcite . these llms have billions of parameters and are pretrained on trillions of tokens scraped from the web. a lucrative utilization of llms is in the form of autonomous agents, to make them follow user instructions and adhere to specific preference requirements @xcite . however, the pre-trained models are often incapable of follow-ing instructions, and they need to be aligned using specially curated preference alignment datasets and methods for generalization @xcite . alignment methods either involve fine-tuning the pre-trained model using auto-regressive language modeling over the ground truth completions (supervised fine-tuning or sft) @xcite or using specialized alignment methods such as reinforcement learning from human feedback (rlhf) @xcite , direct preference optimization (dpo) @xcite , or prompt tuning @xcite . however, applying these methods to the full models is computationally expensive due to their large sizes. parameterefficient training (peft) methods such as low-rank adaptation (lora) @xcite and qlora @xcite have achieved comparable performance to full fine-tuning of llms at a much lower cost. this has enabled researchers to experiment with preference alignment datasets, methods, and models on systems with a single gpu. however, alignment is sensitive to numerous factors and design choices involved in the training @xcite . the design choices to align llms fit into one of the following three broad, crucial axes: (i) the quality and quantity of the alignment dataset, (ii) the preference alignment method, and (iii) the nature of the base model. given the increasing interest in preference alignment, an in-depth analysis of the effect of these axes on downstream performance is required. to the best of our knowledge, no extensive study has investigated them, especially in a peft setting. we fill this important gap by attempting to answer various key research questions across these three axes: alignment dataset how do the informativeness and quality, number of samples, and content of the preference dataset impact the downstream performance? alignment method how do different alignment ## background alignment methods reduce the mismatch between an llm's pre-training and preference requirements of users, as llms are not pre-trained on human preference objectives. we describe the two most widely used alignment methods. supervised fine-tuning (sft) uses a pair of input instructions and corresponding outputs to finetune the llm using autoregressive language modeling. sft is often used at the "instruction-tuning" stage of models such as alpaca @xcite and mistral-instruct @xcite . direct preference optimization (dpo) @xcite is a stable and optimized alternative to reinforcement learning algorithms such as rlhf @xcite . these methods have been used to align llms better using paired human preference data: data consisting of accepted and rejected outputs given an instruction. parameter-efficient training (peft) methods require updating only a fraction of the parameters of the full model while achieving performance close to that of full fine-tuning. examples of these methods include adapters @xcite , prefix-tuning @xcite , and prompttuning @xcite . low-rank adaptation (lora) @xcite is another popular method that inserts a smaller number of new weights into the model, the only trainable parameters. qlora @xcite improves the efficiency of lora by reducing the memory footprint and storage requirements for training while getting similar performance. ## experimental setup preferences and preference alignment datasets we focus on two distinct preferences commonly used to study performance trade-offs in alignment: harmlessness and helpfulness. when a model is aligned on a preference, there is often a performance trade-off on complementary preferences @xcite . hence, these preferences enable us to study these trade-offs in-depth across multiple axes. we use the two most popular datasets for these preferences: hh-rlhf @xcite and beavertails @xcite . hh-rlhf has explicit splits for harmlessness and helpfulness, and each dataset sample consists of a prompt, a chosen sample, and a rejected sample. however, many "harmless" responses in the dataset involve the model refraining from giving a response altogether @xcite . beavertails addresses this issue by providing more informative and elaborate responses. each dataset sample consists of a prompt, two responses, a safety label for each response, and a label for the preferred response. base models we investigate the preference alignment trade-offs using pre-trained and instructiontuned models. we use the two most popular 7 billion parameter pre-trained models, llama-1 @xcite and mistral-7b @xcite . for completeness, we use their instruction-tuned counterparts, vicuna-v1.3 @xcite and mistral-7b-instruct @xcite . the models trained on the harmless preferences are denoted by the suffix "-harmless" (e.g., mistral-7b-harmless), and those trained on the helpful preferences are denoted by "-helpful". evaluation tasks and metrics for evaluating helpfulness, we use (i) mmlu @xcite , a dataset consisting of 57 tasks to test the knowledge acquired by the models, (ii) big-bench hard (bbh) @xcite , a diverse evaluation suite of 23 tasks, and (iii) alpaca eval @xcite , a human-curated suite of 805 questions across different tasks evaluated against gpt-4. for harmlessness evaluation, we use (i) real-toxicity @xcite , a prompt dataset to measure toxic generation in language models. we choose the 1000 most severe prompts from the dataset, and (ii) red-instruct's @xcite dangerousqa with chain-of-utterance, a dataset comprising 200 harmful questions across six adjectives-racist, stereotypical, sexist, illegal, toxic, and harmful. following existing work, we evaluate mmlu using accuracy, bbh using exact match, and alpaca-eval using win rate against gpt-4. for realtoxicity, we use the score given by a reward model trained to classify toxic generations foot_0 , and 100-attack success rate for red-instruct. training setup we experiment with both qlora and lora for alignment training. for sft, we keep the rank of the lora/qlora matrix as 64 and lora/qlora alpha as 16. for dpo, we keep the rank of the lora/qlora matrix as 16 and lora/qlora alpha as 32. we use a batch size of 16, a learning rate of 2e -4 for sft and 5e -5 for dpo, and train the models for 700 steps. we perform all our experiments on a single 40gb a100 gpu and run 5 seeds for all experiments. ## effects of the alignment dataset we investigate the effects of the alignment dataset in terms of its informativeness and quality, the number of samples used, and the preference sets and mixtures used for alignment. ## 98 llama-1-helpful 10.5 4.5 models using a mixture of preferences, the model performs worse than models trained on individual preferences. this trend is consistent across both pre-trained and instruction-tuned models across the preferences. this might arise because the model encounters conflicting responses for prompts that might be similar to each other when orthogonal preferences are used for alignment, leading to nonoptimal training. we observe larger degradations when using dpo with a mixture of the preferences compared to sft, suggesting that dpo is more sensitive to the type of samples used for alignment and requires more uniform preference samples. overall, better curation of dataset mixtures when using alignment methods is necessary to achieve optimal performance. ## effects of the alignment method previous works have shown that preference optimization methods, such as rlhf and dpo, are better than methods like sft for full fine-tuning of models on standard preference datasets @xcite . we validate this claim using peft across the different preferences. setup we compare the performance attained when using sft and dpo to train our models across preferences. we use all four models in our study to align with the harmless and helpful preferences of the beavertails dataset. ## comparing qlora with lora we compare the performance trends observed when using qlora as the peft method with lora as the peft method in figure 6 . we observe similar patterns as observed when using qlora across the three axes of our study, and also observe similar performance as expected. we provide the extended results of using qlora and lora across the training setup in the appendix a. ## key takeaways consolidating our findings and conclusions, we present key takeaways across the three core axes of preference alignment in peft settings. ## 98 llama-1-helpful 10.5 4.5 figure 6 : comparing the performance when using qlora and lora as the peft method (section 7). we observe similar trends and performance across the two methods. ## conclusion and future work in this work, we perform an extensive analysis of the trade-offs of downstream performance of models over distinct preferences across three core alignment axes: the alignment dataset, the preference alignment method, and the nature of the base model, particularly when using peft. we use the two most widely-used 7 billion parameter models, llama-1 and mistral-7b, along with their instruction-tuned variants, two popular preference alignment datasets, hh-rlhf and beavertails, two commonly used alignment methods, sft and dpo, and evaluate on 5 benchmarks across help-fulness and harmlessness. by conducting over 300 experiments with qlora and lora, our findings reveal interesting trends that address the limitations of existing literature and often deviate from them. we consolidate our findings into key takeaways and hope that our guidelines will help researchers perform more effective parameter-efficient preference alignment. as future work, we aim to extend our study on trade-offs across multiple preferences, spanning various domains for alignment. we also plan to explore peft methods, such as model merging and mixture-of-experts, to mitigate these trade-offs.