Transferable Post-training via Inverse Value Learning (2025)

Abstract: As post-training processes utilize increasingly large datasets and base models continue to grow in size, the computational demands and implementation challenges of existing algorithms are escalating significantly. In this paper, we propose modeling the changes at the logits level during post-training using a separate neural network (i.e., the value network). After training this network on a small base model using demonstrations, this network can be seamlessly integrated with another pre-trained models during inference, enabling them to achieve similar capability enhancements. We systematically investigate the best practices for this paradigm in terms of pre-training weights and connection schemes. We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes within the same family, models undergoing continuous pre-training within the same family, and models with different vocabularies across families. In certain cases, it can achieve performance comparable to full-parameter fine-tuning. Furthermore, we explore training methods to enhance transferability, which effectively improve the transfer performance of the value model across models of various parameter scales and prevent overfitting to the base model used during training.

## introduction post-training of large languge models (llms), which can be referred to as fine-tuning a pretrained model for helpfulness, honesty, and harmlessness @xcite , is becoming increasingly important @xcite . a welldesigned post-training process can maximize the potential injected during pre-training while remaining aligned with human intentions and values. however, post-training has become increasingly intricate @xcite @xcite figure 1 : illustration of inverse value learning. v.m. denotes vocabulary mapping. @xcite . a well-designed post-training procedure typically consists of two main stages: supervised fine-tuning (sft) and reinforcement learning from human feedback (rlhf) @xcite . each stage involves iterative data synthesis, quality control, and training pipelines @xcite , which collectively introduce significant complexity to the model training workflow. furthermore, as model sizes continue to grow, the processes of both sampling and training largerscale models are increasingly resource-intensive. to address these challenges, we propose a novel framework for transferable post-training that efficiently adapts models of varying sizes and families with minimal reconfiguration. this framework distills the post-training process to another network by leveraging the logits space as a shared interface for model adaptation and capability transfer. in contrast to parameter or representation spaces, the logits space is more consistent across different models and can serve as a universal channel for communication @xcite . by adjusting the base model's logits during decoding, we can adapt any base models to new tasks without modifying their original parameters. this allows for efficient model adaptation, avoiding the resource-intensive process of full fine-tuning. specifically, we propose transferable posttraining via inverse value learning, a framework that captures changes at the logits level using a separate neural network, referred to as the value network. this network is trained on a lightweight base model using demonstration data to capture the necessary adaptations of the logits in the post-training process. during inference, the value network can be seamlessly integrated with various pre-trained base models, enabling them to adopt the learned adjustments without requiring further training, thus facilitating efficient model transfer of model capabilities. we systematically investigate practical implementations for this framework. specifically, we first study the importance of pre-trained weights in modeling the residual logits. then, we examine two schemes for integrating the value network with pre-trained models: cascade and residual connections. our experiments reveal that the residual connection scheme, where the value network predicts the delta logits based solely on previous text inputs, demonstrates superior transferability and efficiency. to improve the generalization ability of the value network, we incorporate regularization techniques, such as norm constraints, to mitigate overfitting to the base model during training. additionally, we introduce a vocabulary mapping algorithm to facilitate effective cross-vocabulary transfer. we assess our approach on multiple datasets and tasks, including general instruction following, zeroshot capabilities, and few-shot learning scenarios. our results show that the value network exhibits broad transferability across pre-trained models of varying parameter sizes within the same model family, models undergoing continual pre-training, and even across different model families. in certain cases, our framework reaches close performance with full-parameter finetuning. this highlights the potential of our method for efficient and practical application. our main contributions are as follows: ## problem statement and background the soaring computational costs of full-parameter updates with growing model sizes have driven researchers to explore efficient alternatives like lowrank parameter updates @xcite or representation interventions @xcite while freezing pre-trained weights. however, inconsistencies between the parameter spaces and representation spaces across different models stand as a key challenge for transferable post-training. in contrast, the logits space has better sharing properties and can serve as a communication channel for model interaction, enabling the transfer of capabilities between models @xcite . formally, the impact of post-training on the logits of a pretrained model can be formally expressed as: another line of research using logits-based steering focuses on controlling model outputs to satisfy predefined attributes (e.g., style, toxicity, length). a key characteristic of these works is their reliance on trajectory-level rewards, such as scores from toxicity or topic classifiers. these studies investigate the conversion of trajectory-based rewards into tokenlevel dense guidance @xcite @xcite to achieve attribute-compliant text generation. compared to the logits arithmetic works, we provide a more practical training-based solution that reduces the number of models needed during inference from three to two. furthermore, in contrast to controllable text generation methods that rely on predefined outcome-based reward signals, we learn the reward function implicitly by modeling log p ∆ using a separate neural network trained with demonstrations, thus efficiently achieving the transfer of post-training capabilities. ## conclusions in this paper, we introduced a novel method for transferable post-training called inverse value learning, which models residual logits adjustments using a separate value network. we systematically explored various aspects of this paradigm, including network architectures, pre-training weights, and connection schemes. additionally, we addressed challenges related to overfitting and crossvocabulary transfer by incorporating regularization techniques and a vocabulary mapping algorithm. our results confirm that the value network exhibits broad transferability across models of different sizes and families, highlighting the potential of logits-space adjustments for efficient model adaptation.