Enhancing Language Model Alignment: A Confidence-Based Approach to Label Smoothing (2024)

Abstract: In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains. Within the training pipeline of LLMs, the Reinforcement Learning with Human Feedback (RLHF) phase is crucial for aligning LLMs with human preferences and values. Label smoothing, a technique that replaces hard labels with soft labels, emerges as promising techniques to enhance RLHF training. Despite the benefits, the choice of label smoothing parameters often relies on heuristic approaches and lack theoretical understanding. This paper addresses the challenge of selecting the label smoothing parameter in a principled manner. We introduce Confidence Aware Label Smoothing (CALS), a method that iteratively updates the label smoothing parameter based on preference labels and model forecasts. Our theoretical analysis characterizes the optimal label smoothing parameter, demonstrates its dependence on the confidence level, and reveals its influence on training dynamics and equilibrium. Empirical evaluations on state-of-the-art alignment tasks show that CALS achieves competitive performance, highlighting its potential for improving alignment.

## introduction large language models (llms), empowered by deep transformer architecture @xcite and huge text datasets, have acquired remarkable capabilities in various domains @xcite @xcite , achieving tremendous success on diverse tasks. the full training pipeline of llms can be divided into three stages: pretraining phase @xcite where the llm is trained with unsupervised learning on huge corpus of text data, supervised fine-tuning (sft) phase @xcite where the llm is fine-tuned with supervised learning on datasets for the downstream task, and reinforcement learning with human feedback (rlhf) phase @xcite where the llm is trained on pairwise (or listwise) comparison dataset to improve the alignment with human preferences. in particular, the rlhf phase plays a critical role in controlling the model behavior and aligning llms with human values @xcite @xcite . proximal policy optimization (ppo) @xcite is widely adopted as the rl optimizer in rlhf phase. recently, direct preference optimization (dpo) @xcite emerges as a promising alternative to reinforcement learning methods, as it directly optimizes the model with a classification loss and bypasses the need of fitting a separate reward model. in dpo and more generally mle training, label smoothing @xcite is a standard technique to mitigate the noise in preference labels and overfitting issues. however, the choice of the label smoothing parameter is often set heuristically and theoretical understanding remains scarce. therefore in this work, we ask the question: how to select the label smoothing parameter in a more principled way? to answer this question, we first analyze the tradeoff between bias and variance of the gradient estimate and then characterize the optimal choice of the label smoothing parameter that minimize the expected error of gradient estimate under different distance measures. our theorem suggests that the label smoothing parameter should depend on the confidence level of the preference label. motivated by this result, we propose confidence aware label smoothing (cals), a method that iteratively updates the label smoothing parameter to reflect the confidence of preference label. our method ensures that the gradient is weighted by both the confidence level of preference label and the correctness of model forecast based on preference label and model forecasts, illustrated in figure 1 . we analyze the equilibrium and dynamics of dpo with cals, showing that our proposed method has a more stable gradient and conservative equilibrium. finally, we empirically evaluate our method in state-of-the-art alignment tasks, which shows competitive performance over the baseline. label smoothing. label smoothing has been used successfully in improving deep learning models in various tasks @xcite @xcite . in knowledge distillation, using soft labels help the training of student networks @xcite . various methods propose adaptive or instancedependent label smoothing techniques (krothapalli and abbott, 2020; @xcite @xcite to strengthen the method. it has been shown that appropriate label smoothing can improve generalization @xcite , mitigate label noises @xcite , and prevent reward overfitting @xcite . using the learned reward function r ϕ , rlhf then applies reinforcment learning (rl) to train the policy model π θ , maximizing the objective direct preference learning. besides rlhf, directly learn from human preference may also be directly learned @xcite @xcite without the need of first learning a reward function. among direct preference optimization @xcite (dpo) directly optimizes the following objective ## gradient estimation in this section, we analyze the influence of label smoothing parameters on gradient estimation. consider the bce loss in eq. ( 3 ) . for each l i , its population version l * i := e z i [l i ] represents the average loss on w i under the ground truth distribution the parameter α i controls the trade-off between the bias and variance: when α = 0, the gradient estimate is unbiased, in the sense that therefore, it is crucial to determine the α * i that minimizes the distance between ∇l α i i and ∇l * i . this is answered by the following theorem. theorem 3.1. we abbreviate q := p * (w i ). consider the expected distance e z i [d(∇l α i i , ∇l * i )] between the gradient estimate ∇l α i i and the expected gradient ∇l * i under distance d, then: 21343 1. if d(u, v) = ∥u -v∥ β β where ∥ • ∥ β is the ℓ β metric (β > 1), the expected distance the proof is found in appendix a. the above result characterizes the optimal choice of the labelsmoothing parameter under a range of distance measures. specifically, for α 2 , the optimal α * has the closed form expression ## confidence aware label smoothing (cals) motivated by the previous section, we develop confidence-aware label smoothing where the label smoothing parameter is set to reflect the confidence level of the label. ideally, we hope the parameter for data (w, z) to be certain function of the confidence per data: p * (w) = p * (@xmath0. however, p * (w) is usually not accessible and difficult to esti-mate as each individual prompt only appears once or a few times in the rlhf dataset. to deal with this issue, we define a function α : [0, 1] → [0, 0.5) such that for any @xmath1, 1], ## conclusions in this paper, we present confidence aware label smoothing (cals) as a principled approach to select the label smoothing parameter in direct preference optimization (dpo) for the rlhf phase of llm training. by analyzing the trade-off between bias and variance in gradient estimation, we found that the optimal label smoothing parameter that minimizes the expected error under different distance measures should align with the confidence levels of the label. the proposed method ensures that the gradient is weighted based on both the confidence level of preference label and the correctness of model forecast, leading to more accurate alignment with human preferences. empirical results on alignment tasks using state-of-the-art models demonstrated that cals improves performance over baseline methods. limitations. while our proposed method shows promising results in enhancing the alignment of large language models (llms) with human preferences, it comes with some limitations. first, cals relies on the accurate estimation of confidence levels for preference labels, which can be challenging in scenarios where the data is sparse or of low quality. second, our analysis on gradient estimation in theorem 3.1 is difficult to generalize to larger batch sizes. furthermore, while our empirical evaluations demonstrate competitive performance, the generalizability of cals across different tasks and model architectures requires further investigation. order optimality gives p(w) = p * (w) + α( p(w)) -2p * (w)α( p(w)). notice that this gives a injective map from p * (w) to p(w). it follows that p(w) is the unique solution of the above equation. when