Weak-to-Strong Honesty Alignment via Learning-to-Rank Supervision (2025)

Abstract: Honest alignment refers to the ability of a language model to truthfully convey its knowledge limitations by appropriately refusing to answer questions when it lacks sufficient information. Existing solutions, such as prompt engineering and fine-tuning, face limitations: the former provides only marginal improvements, while the latter struggles to enhance honesty when annotated data is scarce.To overcome the above limitations, we propose , a novel framework that enhances honesty through weak-to-strong generalization. Specifically, we train the strong LLMs under weak model supervision to improve their honesty. For the weak model, we employ a learning-to-rank strategy to train a “honest head”, which learns to select the most honest response among model’s outputs generated through beam search. For the strong LLM, we leverage the self-labeled dataset to update its parameters. Our proposal requires only minimal training data to train the weak honest model, yet achieve decent performance for labeling data. In addition, it enables the strong LLMs to have the capabilities to generalize even facing with the flawed label data. Extensive experiments show significantly boosts honest alignment in large models even with limited labeled data. Our code is available at https://github.com/zewanfaan/WHAT_Honesty .

## introduction in recent years, large language models (llms) have achieved significant progress in tasks like code generation @xcite @xcite , mathematical reasoning @xcite , and scientific research @xcite , driven by scaling paradigm @xcite @xcite . however, llms still frequently produce outputs that are factually inconsistent or lack * corresponding author. however, achieving honesty in llms remains challenging, as current alignment practices often prioritize helpfulness and harmlessness over honesty @xcite . this imbalance stems from the difficulty of creating model-specific alignment data, tailored examples that teach models to distinguish between known and unknown domains. without such data, it is difficult to define precise knowledge boundaries of a model; thus, models risk either overconfidence in uncertain contexts or excessive refusal of valid queries. efforts to enhance model honesty can be divided into two main approaches: prompt engineering and model fine-tuning. when provided with well-designed prompts, llms can improve its honesty @xcite . some studies suggest using carefully crafted prompts to encourage models to be more cautious when answering questions about unknown knowledge @xcite . in addition, other research focuses on improving llm's honesty via fine-tuning. these approaches first determine whether the model possesses knowledge about a given question. when the model lacks sufficient knowledge, it is trained to express uncertainty. this training can be carried out through supervised fine-tuning @xcite @xcite , direct preference optimization @xcite , or proximal policy optimization @xcite and their variants @xcite . while these methods improve honesty to some extent, they face significant limitations. on the one hand, prompt engineering yields only modest improvements since the training process does not explicitly align with honesty objectives. on the other hand, fine-tuning methods achieve better results but depend heavily on high-quality, diverse training data, which is often limited, particularly for complex questions where answers may be diverse or subjective. evaluating such answers is inherently challenging, further complicating data curation. moreover, each target model architecture may exhibit unique knowledge gaps; thus it requires tailored dataset construction. this customization imposes significant computational and human resource demands, making large-scale finetuning less scalable and harder to generalize across varied scenarios. to address these challenges, we introduce what (weak-to-strong honesty alignment via learning-to-rank supervision), a framework inspired by weak-to-strong generalization @xcite . this approach leverages a lightweight "honest head" network to guide llms toward more truthful outputs, even under limited supervision. specifically, the key mechanisms of our proposal consist of three parts: (1) honest head training: an honest head is a lightweight model that is trained on available labeled data using a learningto-rank framework. it is then used to identify the most honest responses among those generated by llms through beam search. this design mitigates data scarcity by requiring minimal labeled data and emphasizing comparative honesty over absolute ground truth. (2) large-scale self-labeling: for questions lacking reference answers, we combine the honest head's predicted scores with the llm's prediction probabilities to generate pseudo label. this mechanism extends honesty alignment to unlabeled data, significantly reducing reliance on human annotations and mitigating data scarcity. our contributions are summarized as follows: ## preliminary this section presents preliminary knowledge regarding language model decoding and honesty measurement, which is helpful for understanding the subsequent methodology. ## weak-to-strong honest generation this section describes our method to generate pseudo-labels using a lightweight, easily trainable weak model, thereby reducing dependence on manual data annotation. figure 1 provides an overview of our approach, what. our method is composed of three phrase: (1) weak honest model training: we train a lightweight "honest head" model on limited labeled data within a learning-to-rank framework. this model is used to identify the most honest responses among those generated by llms through beam search. ## conclusion in this study, we introduce what, a method designed to enhance model honesty through weakto-strong generalization. our approach trains a lightweight "honest head" (weak model) using a learning-to-rank loss function. this head reranks beam search candidates from the models output, enabling the self-labeling of unannotated data. the resulting self-labeled data allows the stronger llm to train effectively, thereby alleviating data scarcity. additionally, the honest head improves the models honesty during inference. extensive experiments demonstrate that what effectively mitigates labeled data scarcity and achieves stateof-the-art results in honesty alignment.