Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment (2024)

Abstract: Large language models (LLMs) are still struggling in aligning with human preference in complex tasks and scenarios. They are prone to overfit into the unexpected patterns or superficial styles in the training data. We conduct an empirical study that only selects the top-10% most updated parameters in LLMs for alignment training, and see improvements in the convergence process and final performance. It indicates the existence of redundant neurons in LLMs for alignment training. To reduce its influence, we propose a low-redundant alignment method named **ALLO**, focusing on optimizing the most related neurons with the most useful supervised signals. Concretely, we first identify the neurons that are related to the human preference data by a gradient-based strategy, then identify the alignment-related key tokens by reward models for computing loss. Besides, we also decompose the alignment process into the forgetting and learning stages, where we first forget the tokens with unaligned knowledge and then learn aligned knowledge, by updating different ratios of neurons, respectively. Experimental results on 10 datasets have shown the effectiveness of ALLO. Our code and data will be publicly released.

## introduction alignment with human preferences has become a desired property of llms @xcite @xcite , e.g., helpfulness, honesty, and harmlessness, and reinforcement learning from human feedback (rlhf) @xcite is a widely utilized technique for achieving it. typically, rlhf aims to fine-tune llms on human preference data, to maximize and minimize the likelihood of generating the positive and negative responses, respectively. after figure 1 : training loss curve and benchmark performance of qa tasks using different trainable neurons in llm. we perform alignment training using dpo @xcite on ecqa and qasc. the top/last-10% related neurons are selected based on the accumulated gradients during dpo training. rlhf training on corresponding datasets, llms can better follow user instructions @xcite , solve complex problems @xcite , and generate unbiased responses @xcite . however, it is hard to train a well-aligned llm for complex tasks and scenarios @xcite . the key issue is that llms might overfit into the unexpected patterns or superficial styles in the human preference data (i.e., pairs of positive-negative response) @xcite . it is the side effect of their powerful learning capability derived from the large-scale trainable parameters @xcite . recently, a surge of work @xcite has found that each neuron (i.e., one of the trainable values of the parameter matrixes in llms) is relevant with special knowledge, and the neurons in llms are generally sparsely activated. inspired by it, we consider whether the irrelevant neurons exist in llms during the alignment process and the full-parameter trained llms might lead to redundant updates on alignment-irrelevant neurons. thus, we conduct the empirical experiment using dpo algorithm, where we only update the top/last-10% neurons according to their accumulated gradient values, which indicate the relevance between the current neuron and the downstream scenario @xcite . as shown in figure 1 , with the top-10% trainable neurons, llms can converge faster and achieve better performance than optimizing all the neurons. it indicates the existence of alignment-irrelevant neurons and redundant updates in dpo training, affecting the convergence and final performance. besides the redundant neurons, previous work @xcite has shown the existence of redundant tokens in training data. to reduce the influence of redundant updates, we aim to prune the alignment training, to focus on optimizing the most related neurons with the most useful supervised signals. concretely, we first identify the neurons that are related to the human preference data, based on the accumulation values of gradients. second, we identify the key tokens about human preference, and only compute loss on them for optimizing the alignment-related neurons. in this way, we perform a low-redundant optimization for aligning llms with humans, which reduces the redundancy of learning irrelevant tokens and training irrelevant neurons, avoiding their effect on alignment performance. however, the involved tokens and neurons are not always consistent in the alignment objectives, since the alignment training focuses on both removing the unaligned knowledge and learning the aligned one. therefore, we decompose the alignment process into the forgetting and learning stages, and adapt the low-redundant optimization strategy on them. for the forgetting stage, relatively fewer neurons are trained by unlearning algorithm @xcite to forget the unaligned knowledge, and we leverage a token-level reward model @xcite to identify the unaligned tokens required to be focused. for the learning stage, we train more neurons using the dpo algorithm, and also utilize its reward score to select the key tokens. in this work, we proposed an alignment method with low-redundant optimization (allo) to finetune llms. in allo, we first identify the most important neurons by training a reference model. then, for the forgetting and learning stages, we utilize npo and dpo algorithms with unlearning and learning losses on corresponding key tokens respectively, to optimize the identified important neurons. to comprehensively assess the effectiveness of allo, we conduct extensive experiments on three downstream scenarios, i.e., question answering, mathematical reasoning, and instruction following, totally 10 datasets. experiment results show that allo mostly outperforms competitive human alignment methods (e.g., sft @xcite , dpo @xcite , ppo @xcite ), achieving a 9.7% maximum relative improvement over vanilla dpo. ## related work we introduce the related work from the perspectives of large language models, llms alignment, and unlearning of llms. ## preliminary llms alignment refers to aligning the behaviors of llms to human preference, e.g., helpfulness, honesty, and harmlessness @xcite . existing work typically utilizes rlhf methods @xcite to fine-tune llms using human preference data, for improving alignment. formally, the human preference data is composed by input prompts, positive responses, and negative responses, denoted as @xmath0, 𝑦 + 𝑖 , 𝑦 - 𝑖 ⟩} 𝑛 𝑖=1 . the input prompt or response consists of a series of natural language tokens {𝑡 1 , 𝑡 2 , . . . , 𝑡 𝑙 }. given the input prompt 𝑥, we aim to train llms that tend to generate the well-aligned positive response 𝑦 + , while avoiding generating the unaligned negative one 𝑦 -. in this work, we focus on devising an effective training algorithm to improve the alignment of llms, which can be utilized to satisfy the diverse requirements in real world (e.g., instruction following and question answering). according to our empirical study in figure 1 , updating only top-10% trainable neurons would achieve better performance than full-parameter tuning for alignment training. it indicates that there are redundant updates in the training process of llms, which may affect the alignment performance. to address it, in this work, we aim to perform parameter-efficient fine-tuning for reducing the redundant updates on unrelated neurons, to improve the alignment of llms. given the training data, we first identify the highly-relevant neurons @xmath1, . . . , 𝜃 𝑖 𝑘 } in the parameter matrices of llms, and perform low-redundant optimization on the llm as: ## approach in this section, we introduce our proposed method allo, a low-redundant alignment method for finetuning llms. in allo, we compute loss on selected key tokens, and only optimize the selected important neurons. concretely, we first train a reference model to locate the important neurons through gradient. then, we identify the key tokens related to unaligned knowledge, and utilize the unlearning algorithm to update few neurons for forgetting them. next, we leverage dpo algorithm to improve the alignment of the llm, where the dpo reward is used for selecting the key tokens. the framework of allo is presented in figure 2 . ## conclusion in this paper, we proposed allo, an alignment method with low-redundant optimization, to train the most related neurons with the most useful supervised signals. in allo, we first estimated the importance of neurons in the llm based on the weight changes of a reference model, and located the most related neurons for optimization. then, we decomposed the alignment process into the forgetting and learning stages, where we leveraged token-level reward and dpo reward scores to identify the key tokens, and computing loss on them for training. experimental results on questionanswering tasks, mathematical reasoning tasks, and instruction following tasks have shown the effectiveness of allo. as future work, we will consider leveraging allo on other important scenarios, e.g., reducing hallucination. besides, we will also implement allo in larger llms and multimodal llms to validate its effectiveness.