Inference-Time Language Model Alignment via Integrated Value Guidance (2024)

Abstract: Large language models are typically fine-tuned to align with human preferences, but tuning large models is computationally intensive and complex. In this work, we introduce **Integrated Value Guidance (IVG)**, a method that uses implicit and explicit value functions to guide language model decoding at token and chunk-level respectively, efficiently aligning large language models purely at inference time.This approach circumvents the complexities of direct fine-tuning and outperforms traditional methods.Empirically, we demonstrate the versatility of IVG across various tasks. In controlled sentiment generation and summarization tasks, our method significantly improves the alignment of large models using inference-time guidance from **gpt2**-based value functions. Moreover, in a more challenging instruction-following benchmark AlpacaEval 2.0, we show that both specifically tuned and off-the-shelf value functions greatly improve the length-controlled win rates of large models against gpt-4-turbo (e.g., 19.51 % \rightarrow 26.51\% for **Mistral-7B-Instruct-v0.2** and 25.58 % \rightarrow 33.75 \% for **Mixtral-8x7B-Instruct-v0.1** with Tulu guidance).

## introduction learning-based algorithms have become the standard for aligning large language models (llms) with human preferences, as evidenced by numerous studies @xcite @xcite @xcite . despite their success, fine-tuning llms is notably resource-intensive and poses implementation challenges @xcite . these challenges have catalyzed the development of inference-time alignment methods that maintain llms in a frozen state and guide their decoding during testing @xcite ## related work large unsupervised language models, trained on vast internet-scale datasets, have demonstrated remarkable capabilities @xcite @xcite . nonetheless, aligning these models with human values remains challenging. traditionally, alignment is achieved through fine-tuning based on human evaluations of model-generated responses @xcite @xcite @xcite @xcite . while effective, this method demands significant computational and engineering resources. moreover, the diversity of human values complicates the creation of universally aligned models @xcite @xcite @xcite . in response to these challenges, we propose an inference-time alignment approach that freezes pre-trained models while modulating their outputs through a decoding phase managed by smaller, specialized models. this strategy minimizes the need for extensive retraining and adapts more readily to individual preferences. the conceptual framework for aligning language models at inference time is rooted in the use of value functions. implicit value functions, as described by @xcite , proposed a token-level markov decision process to adjust language model outputs based on the log-likelihood differences between fine-tuned and base models. @xcite introduced a method that leverages explicit value functions trained through a kl-regularized reinforcement learning objective, which acts as a prefix scorer. our empirical findings, detailed in section 5, underscore that while implicit value functions excel at refining tokenlevel nuances, explicit value functions provide superior sequence-level contextual understanding. these insights motivate our method, which integrates these value functions to enhance model alignment with human preferences during the decoding phase of model output generation. ## preliminaries in this section, we introduce the mathematical formulation of aligning large language models (llms) with human preferences and then describe the implicit and explicit value functions used in our approach. ## method in this section, we introduce our proposed method, integrated value guidance (ivg). first, we discuss how to utilize the value function in two distinct ways: token-wise sampling @xcite and chunk-level beam search @xcite . next, we explain how to train the implicit and explicit value functions on the preference dataset. finally, we present the overall inference-time alignment process and analyze the computational efficiency of our method. ## experiments in this section, we empirically evaluate the ability of the proposed ivg to align large language models with human preferences using only inferencetime guidance from small language models. first, in controlled-sentiment generation @xcite and summarization (stiennon et al., 2020), we tune gpt2 to model the desired behaviors in each task to get the implicit value function and train the explicit value function based on the same base model. then, we use the trained implicit and explicit value functions to steer larger models of various scales (section 5.1). next, in a more difficult instruction-following benchmark, alpacaeval 2.0 @xcite , in addition to tuning small models, we reuse the off-the-shelf open-source 7b models and their untuned versions as the implicit value function and train the explicit value function by one of the bestperformance sequence-wise reward models evaluated by rewardbench @xcite . we then use them to steer a series of large models. note that many existing methods can be represented by the framework: emulator fine-tuning (eft) @xcite can be viewed as applying only the implicit value function in tokenwise sampling. weak-to-strong search can be viewed as applying only the implicit value function in chunk-level beam search. we use the efti, efte, cbsi and cbse to represent the corresponding combination. for example, efti denotes applying implicit value function for token-wise sampling, and cbse denotes applying explicit value function for chunk-level beam search. ## discussion we have presented integrated value guidance (ivg), a method that combines implicit and explicit value functions, applied to token-wise sampling and chunk-level beam search. we conducted experiments on synthetic tasks and instruction-following task and found that ivg achieved the best performance in both tasks. we also explored the performance of different combinations of implicit and explicit value functions in the two tasks and found that the explicit value function applied to chunklevel beam search can significantly improve the results, while the implicit value function applied to token-wise sampling can improve the results. ivg combines the advantages of both, so it achieves the best performance in both tasks. limitations. our work primarily focuses on enhancing the alignment capabilities of large language models through the integration of implicit and explicit value functions. this approach introduces several complex questions that extend beyond the current scope of our research: 1. our methodologies have been limited to using the dpo @xcite and fudge @xcite algorithms for training and deriving the implicit and explicit value functions. it remains unclear whether incorporating other large model alignment strategies or offline reinforcement learning algorithms for tokenwise sampling and chunk-level beam search might influence our findings. this aspect warrants additional experimental investigation. ## acknowledgement this work is supported in part by the national key r&d program of china (no.2022zd0160102).