How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective (2024)

Abstract: This paper introduces a novel generalized self-imitation learning GSIL framework, which effectively and efficiently aligns large language models with offline demonstration data. We develop GSIL by deriving a surrogate objective of imitation learning with density ratio estimates, facilitating the use of self-generated data and optimizing the imitation learning objective with simple classification losses. GSIL eliminates the need for complex adversarial training in standard imitation learning, achieving lightweight and efficient fine-tuning for large language models. In addition, GSIL encompasses a family of offline losses parameterized by a general class of convex functions for density ratio estimation and enables a unified view for alignment with demonstration data. Extensive experiments show that GSIL consistently and significantly outperforms baselines in many challenging benchmarks, such as coding (HuamnEval), mathematical reasoning (GSM8K) and instruction-following benchmark (MT-Bench). Code is public available at https://github.com/tengxiao1/GSIL.

## introduction pre-training endows large language models (llms) with extensive knowledge about the world. however, it does not behave in accordance with some task-dependent requirements. to achieve the desired performance on certain tasks, a post-training process known as alignment or fine-tuning is essential. alignment has emerged as a pivotal approach to improve the following performance of pre-trained language models, especially in complex instruction-following tasks: commonsense reasoning, coding, summarization, and math problemsolving @xcite @xcite . the current alignment methods can be broadly categorized into groups: (i) supervised fine-tuning (sft) based on demonstration data, aligning an input prompt and a human response. (ii) preference fine-tuning @xcite with reinforcement learning from human feedback (rlhf) @xcite or direct preference optimization (dpo) @xcite @xcite @xcite based on preference data containing preferred and dis-preferred responses to prompts. although rlhf and dpo have achieved promising results @xcite , they require expensive human preference labels on several candidate demonstrations of a query to be used as feedback, limiting their applicability to language model alignment in settings where there is a lack of preference feedback. furthermore, preference fine-tuning may suffer from reward overoptimization (also known as reward hacking), as shown by @xcite . recent work @xcite also shows that simply performing sft on demonstrations can result in a better model than preference fine-tuning with ai feedback. therefore, while preference fine-tuning has garnered more increasing attention in the literature for llm alignment @xcite @xcite @xcite @xcite , this work poses the following critical questions that remain unanswered. what is the best use of human demonstration data? is sft the most effective method for leveraging this data, or are there other approaches that could yield better results in specific contexts? to answer this question, we investigate the expressive power of imitation learning (il). we begin by presenting gsil, a simple, effective, and general framework for alignment with human demonstration data. instead of using demonstration data directly via sft (the simplest approach to imitation learning, which corresponds to behavior cloning), we propose a generalized self-imitation learning framework to learn a better policy. imitation learning treats the task of learning a policy from a set of expert demonstrations, proving particularly promising in domains such as robot control, autonomous driving, where manually specifying reward functions is challenging but historical human demonstrations are readily accessible @xcite @xcite . while the motivation is straightforward, we are faced with important challenges in applying imitation learning for alignment of large language models. state-of-the-art imitation learning frameworks in the literature of reinforcement learning (rl) are considerably more complex and computationally intensive than sft, involving inefficient and unstable adversarial or iterative training on separate discriminator and policy networks @xcite @xcite . this challenge largely prevents us from aligning large language models in real-world applications. contributions. we propose a principled imitation learning framework, namely gsil, which not only can learn an effective policy from human demonstration data, but also achieves simple and fast fine-tuning as sft. specifically, we first deduce an equivalent surrogate objective for standard imitation learning, allowing for the utilization of demonstration data. then, we make use of the connection between imitation learning and density ratio estimation that can be solved with simple classification in an entirely offline fashion. importantly, this connection enables us to represent the policy and discriminator via the same language model, achieving simple fine-tuning without adversarial training. intuitively, gsil tries to increase the relative log probability of demonstrations to self-generated synthetic responses. gsil is a generalized framework, and we show that essentially any density ratio estimation loss (e.g., logistic, hinge, brier) can be used in gsil as shown in section 3.4. ## background let the text sequence @xmath0, x 2 , . . .] denote the input prompt, and @xmath1, y 2 , . . .] denote the response. we use notation π θ (y | x) to denote the policy parameterized by parameters θ, which outputs the response y given the input x. we assume that we have access to demonstrations @xmath2, y i )} |d| @xmath3, where (x i , y i ) is collected from an unknown policy π data . given demonstrations d, the goal is to fine-tune the language model policy π θ (y | x) to achieve effective alignment. sft. given a demonstration dataset, the objective of sft is minimizing the negative log-likelihood over the demonstration data as follows: it is worth noting that sft is equal to behavior cloning (bc) @xcite which is a classical and commonly used imitation learning method. bc is typically cast as kl divergence minimization between the learning policy and expert policy as: for each prompt x sampled from dataset d. it is easy to see that the bc problem above shares the same optimal solutions as sft in expectation. rlhf. typically, given the reward function r(x, y), which indicates the human preferences, rlhf optimizes policy π θ for x to maximize reward with the following rl objective: although rlhf with ppo has achieved remarkable success, the training process of ppo is unstable because of the high variance of the estimates of the policy gradients @xcite . dpo. dpo assumes an additional offline preference dataset @xmath4, y w , y l )} |p| @xmath5, where y w and y l denote preferred and dispreferred responses, obtained typically by sampling from π ref , respectively. the response pairs are typically presented to humans (or an oracle) who express preferences for responses given the prompt, denoted as y w ≻ y l | x. to optimize π θ directly using the preference data, dpo @xcite @xcite uses the log-likelihood of the policy to implicitly denote the preference (reward) function: with preference expressed in terms of policy, dpo optimizes π θ based on the bradley-terry (bt) preference model @xcite : dpo and many of its extensions @xcite @xcite @xcite follow a similar intuition: increasing the margin between the likelihood of preferred responses and the likelihood of dispreferred responses. however, as noted earlier, these preference fine-tuning objectives require expensive human preference labels on candidate demonstrations of a query to be used as feedback and may suffer from reward overoptimization, as shown by @xcite . recent work @xcite also shows that simply performing sft on demonstrations can result in a better model than preference fine-tuning with ai feedback. contrastive fine-tuning with preference has recently attracted increasing attention @xcite @xcite @xcite @xcite @xcite . nevertheless, the question of the best use of human demonstration data remains largely unexplored. is sft the best way to align large language models from demonstration data? in this paper, we make the first attempt to answer this question by proposing gsil, a family of imitation learning losses, which enables a unified view of alignment from offline demonstrations. ## the proposed method in this section, we begin by formally introducing the imitation learning formulation for language model alignment and transforming it into an equivalent form, which can effectively use the demonstration data and conduct alignment efficiently. ## experiments in this section, we present the main experimental results, highlighting the superior performance of gsil on various benchmarks and ablation studies. ## conclusions in this paper, we propose gsil, a simple and generalized framework for language model alignment with offline demonstration data from an imitation learning perspective. by aligning the auxiliary reward function with the optimal policy, gsil eliminates the need for adversarial training, achieving simple and fast fine-tuning. importantly, gsil enables a unified view on imitation learning on demonstration data and sheds light on connecting a rich literature on density ratio estimation to the designs of offline alignment with only demonstration data. extensive experiments show that our algorithms consistently outperform existing approaches such as sft and spin across various benchmarks, including mt-bench and open llm leaderboard.