Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding (2025)

Abstract: Large Language Models (LLMs) require alignment with human preferences to avoid generating offensive, false, or meaningless content. Recently, low-resource methods for LLM alignment have been popular, while still facing challenges in obtaining both high-quality and aligned content. Motivated by the observation that the difficulty of generating aligned responses is concentrated at the beginning of decoding, we propose a novel framework, Weak-to-Strong Decoding (WSD), to enhance the alignment ability of base models by the guidance of a small aligned model. The small model first drafts well-aligned beginnings, followed by the large base model to continue the rest, controlled by a well-designed auto-switch mechanism. We also collect a new dataset, GenerAlign, to fine-tune a small-sized Pilot-3B as the draft model, which effectively enhances different base models under the WSD framework to outperform all baseline methods, while avoiding degradation on downstream tasks, termed as the alignment tax. Extensive experiments are further conducted to examine the impact of different settings and time efficiency, as well as analyses on the intrinsic mechanisms of WSD in depth.

## introduction scaling law has boosted llms to handle more complex tasks with increasing model size, as well as the risk of generating offensive, false, or meaningless content, leaving the necessity to align the model with human preferences. such alignment is often achieved through fine-tuning, which may lead to two issues: the alignment tax, i.e. a degradation in model performance on downstream tasks @xcite , and the huge computational overhead spent on fine-tuning. recently, low-resource llm alignment has experienced fast development, which can be divided into two trends. some methods directly interfere with next-token decisions with external scoring during llm decoding @xcite . despite certain effectiveness, they avoid improving the model's inner alignment performance but reduce the coherence and fluency of generated text. the other trend is to influence the token distribution through in-context learning (icl). for example, @xcite propose a series of well-crafted contextual demonstrations to guide the llms to be more helpful. however, using the context to influence response generation corresponding to the current query is indirect, leaving much room for further improvement toward direct query-oriented solutions. with a preliminary experiment (see section 2.1), we observe that the alignment of large models is mainly hindered by its modeling the entire text space, where the well-aligned responses for one query are often not the highest-ranked one among all valid candidate decoding paths. however, once the large model has moved along the path of aligned response, the difficulty of generating the following content will be dramatically reduced. as the saying goes, "well begun is half done." these inspire us to focus on how to enable llms to overcome the obstacle of path selection for a given query at the beginning of decoding, which has a totally different motivation from the above two trends. in this paper, we propose a novel framework for low-resource preference alignment, where the decoding is completed with the coordination of a small draft model aligned with preference and a large base model with strong capabilities, named weak-to-strong decoding (wsd). specifically, the draft model will provide well-aligned responses at the beginning of decoding, then switch to the base model to generate the rest based on a welldesigned auto-switch mechanism, as illustrated in figure 1 . on the other hand, fine-tuning small models is feasible in low-resource settings, and can quickly improve its performance of preference alignment, which is opposite to the high computational cost of training large models. it also avoids increasing the time overhead, as the draft model can generate the same length of the beginning more quickly than the large model. we also collect a new dataset, named gener-align, to train a better draft model pilot-3b, then test its collaborative performance with multiple large base models under the proposed wsd framework. extensive experiments show that wsd is effective in enhancing the performance of the base llms under low-resource settings, outperforming all baseline methods. we also examine the impact of different hyperparameter settings and their intrinsic mechanisms. moreover, unlike fine-tuning leading to the alignment tax, we find that wsd does not even impair the performance of the base model on downstream tasks. in summary, our contributions are as follows: 1) we propose a novel framework, wsd, for lowresource preference alignment, which is able to enhance the alignment ability of base models without incurring a large cost. ## methodology in this section, we propose our method, wsd, which aims to enhance the performance of a base llm in a low-resource manner. ## related works most existing methods for preference alignment are based on fine-tuning, ranging from reinforcement learning to supervised methods @xcite , such as dpo @xcite , simpo @xcite , macpo @xcite , etc. however, as the model size increases, the cost of such methods also increases, which is hard to afford. recently, many works have attempted to achieve this goal in low-resource scenarios, from different perspectives. for example, @xcite and @xcite take external scoring models to guide llm decoding, while @xcite split it into multiple segments, each of which comes from rejection sampling with scoring models. another line is to indirectly influence the token distribution, such as in-context learning @xcite @xcite . differently, @xcite design a value model to edit each hidden representation before token projection in llms, to shift it towards the given preference. aligner @xcite goes further by directly editing the decoded content to acquire highquality responses. the proposed wsd framework differs from these methods by first drafting a wellaligned start directly for the query, then leveraging the autoregressive nature to influence the llm token distribution to approach user preference. ## conclusion we propose wsd, a novel framework improving llm in preference alignment through a lowresource way, where a small aligned model m first drafts well-aligned beginnings, followed by the base model to complete the rest, controlled by a well-designed auto-switch mechanism. we then acquire pilot-3b as m with the proposed gener-align, boosting different large base models to outperform baselines, while avoiding the alignment tax on downstream tasks. further experiments provide detailed evidence of the effect and intrinsic mechanisms of wsd. we hope all these efforts can provide comprehensive insights into wsd and inspire future research in this direction foot_2 .