EMORL : Ensemble Multi-Objective Reinforcement Learning for Efficient and Flexible LLM Fine-Tuning (2025)

Abstract: Recent advances in reinforcement learning (RL) for large language model (LLM) fine-tuning show promise in addressing multi-objective tasks but still face significant challenges, including competing objective balancing, low training efficiency, poor scalability, and limited explainability. Leveraging ensemble learning principles, we introduce an Ensemble Multi-Objective RL (EMORL) framework that fine-tunes multiple models with individual objectives while optimizing their aggregation after the fine-tuning to improve efficiency and flexibility. Our method is the first to aggregate the hidden states of individual models, incorporating contextual information from multiple objectives. This approach is supported by a hierarchical grid search algorithm that identifies optimal weighted combinations. We evaluate EMORL on counselor reflection generation tasks, using text classification models to score the generations and provide rewards during RL fine-tuning. Through comprehensive experiments on the PAIR and Psych8k datasets, we demonstrate the advantages of EMORL against existing baselines: significantly lower and more stable training consumption (17,529 ± 1,650 data points and 6,573 ± 147.43 seconds), improved scalability and explainability, and comparable performance across multiple objectives.

## introduction multi-objective optimization (moo) for large language models (llms) is a crucial research direction in natural language processing (nlp) tasks that require fulfilling diverse and often competing requirements @xcite @xcite . such tasks require models to simultaneously optimize for multiple evaluation criteria, balancing trade-offs between different objectives. this paper focuses on the counselor reflection generation task, where llm reflects on users' prompts, building the first step of therapeutic communication @xcite . as exemplified in figure 1 , this task requires generations that optimize competing objectives, such as reflection, empathy, and fluency in high-quality counseling interactions. reinforcement learning (rl) offers a promising approach for fine-tuning pre-trained llms in moo @xcite . conventional rl fine-tuning approaches combine multiple objectives into one reward function, enabling optimization of objectives during training @xcite @xcite . however, these approaches face significant challenges: optimizing appropriate weights for each objective, maintaining training efficiency, and ensuring scalability while preserving result explainability @xcite . from another perspective, compared to rl with human feedback (rlhf) @xcite , moo explicitly defines objectives, advantaging user-specified llm characteristics. human feedback typically expresses preferences rather than specific objective values, requiring exhaustive human annotations, while moo directly targets explicit objectives, transcending implicit instruction limitations. challenges also exist in ensuring objectives align with human perception and in incorporating many objectives simultaneously. these challenges highlight the need for more efficient and flexible fine-tuning methods. ensemble learning offers a potential solution for this growing need by aggregating multiple trained models into a global model @xcite . we propose a novel ensemble multi-objective rl (emorl) framework for llm fine-tuning that distributes objectives to respective models, enabling training for individual objectives independently. our experiments demonstrate that models trained with single-objective reward functions converge significantly faster than those optimizing multiple objectives simultaneously. the framework incorporates a hidden states aggregation method coupled with a hierarchical grid search algorithm during aggregation, which efficiently identifies optimal weights for combining these single-objective models. our results demonstrate that the aggregated output achieves higher training efficiency while achieving performance comparable to models trained using single-policy methods, as shown in figure 2 . the performance is represented by an objective reward, which uses a utility function that weights objectives according to user-specific preferences. our framework is highly scalable, accommodating different user preferences and new objectives through aggregation. additionally, the framework enhances explainability by providing insights into the importance of different objectives easily through evaluating different weighted combinations of objectives on test samples. the code for our framework and experiments is publicly available and can be found at https://github.com/engineerkong/emorl . succinctly, our main contributions are as follows: (1) we introduce emorl framework that separately trains and aggregates models in the counselor reflection generation task. (2) we develop an effective hidden-state level aggregation method and a hierarchical grid search algorithm for optimizing the weighted combination. (3) we demonstrate our framework's effectiveness through a comprehensive evaluation against moo baselines, achieving comparable performance across multiple objectives while offering lower and more stable training resources, improved scalability and explainability. ## related work prior work on rl for llm fine-tuning has explored various approaches to balance multiple objectives. ## challenges in this section, we analyze the challenges of moo by comparing the training processes of models with individual objectives against multi-objective models using conventional single-policy methods. we tested five fine-tuning setups for counselor reflection generation to demonstrate these challenges, focusing on the objectives: reflection, empathy, and fluency. three setups optimized single objectives separately, while two single-policy approaches fine-tuned for all three objectives: (1) uniform weighted, assigning equal weights ( 1 3 ) to each objective, and (2) dynaopt, dynamically adjusting weights using a multi-armed bandit algorithm. experiments used 5 random seeds with 3 generation runs each, evaluating objective rewards, data consumption, and training time. progress was tracked via weights & biases, plotting objective reward against data consumption. these observations suggest that integrating multiple objectives in training inherently presents convergence, stability, and performance challenges. a promising research direction emerges from this insight: ensembling single-objective models to achieve efficient and flexible moo. ## methodology as shown in figure 4 , the emorl framework consists of three key stages. first, multiple models are independently fine-tuned for distinct individual objectives. second, we employ hierarchical grid search to find optimal linear combinations according to our utility function, which defaults to average objective preferences. finally, we use the optimal weights to aggregate models at the hiddenstate level during inference, generating outputs that effectively integrate all objectives. this ensemble learning approach simplifies the complex multiobjective fine-tuning problem into an optimization problem for aggregation weights. ## results and discussion hierarchical grid visualization demonstrates explainable results: figure 5 illustrates the hierarchical grid search process: in iteration 1 (left subplot), we evaluated 3 × 3 × 3 weighted combinations for reflection, empathy, and fluency, with color mapping indicating objective reward values. the 2 × 2 × 2 area with best total performance was then used to refine the search space for subsequent iterations. the right subplot shows iteration 5, where the search converged to a more precise and refined space: (0.75, 0.8125), (0.4375, 0.5) and (0.0, 0.0625) for the aggregation weights of reflection, empathy, and fluency models respectively. this progressive refinement allows for the precise identification of the optimal weighted combination. the visualization reveals significant insights regarding each objective's contribution to overall performance. unlike defining weight coefficients in the reward function during training, which doesn't correspond linearly to performance, our aggregation phase's contribution distribution accurately reflects each objective-specific model's actual contribution. as shown in figure 5 , the aggregation benefits (orange points) from higher weights of the reflection model. empathy delivers optimal overall performance with moderate weights, while extreme weights reduce the objective reward. the fluency model demonstrates a negative effect (cyan points) on other objectives when assigned high weights, with lower weights facilitating better integration with the other objectives. this visualization can be readily acquired during aggregation by evaluating weighted combinations on batches of test data. although results vary across experiments with different sampled models, as shown in ## conclusion to conclude, our study addresses the challenges of moo in llm fine-tuning. we identify the key limitations of convergence speed, process stability, and performance metrics in conventional singlepolicy approaches. to address these limitations, we propose emorl, a novel meta-policy frame-work using ensemble learning to aggregate diverse models trained on individual objectives. the results demonstrate that emorl achieves greater diversity, efficiency, scalability, and explainability while maintaining performance comparable to existing methods in counselor generation tasks. our approach is the first to aggregate hidden states to incorporate multiple objectives in nlp. unlike classification or regression models, which can simply concatenate results from each model to collaboratively decide on the final output, llms produce complete sequential generations where fluency cannot be separated or neglected. the hidden states represent rich contextual information, which proves more valuable than incorporating parameters or logits. by using the hidden-state level aggregation, emorl exceeds the performance limitations of meta-policy methods. emorl also offers unique advantages over conventional singlepolicy methods, including improved training efficiency, extensive scalability, and enhanced explainability due to the parallel training properties, making it more efficient and flexible. we investigated the nature of the moo problem and discovered that optima consistently appear within higherperformance regions. this insight led us to design a simple yet effective hierarchical grid search algorithm that requires fewer evaluations to find the globally optimal weights. our approach is both task-and model-agnostic. it's compatible with all transformer-based llms, as these architectures maintain hidden states that represent contextual information encompassing multiple objectives in nlp. this study demonstrates the potential of ensemble learning to advance current rl training paradigms and points to a promising novel direction for efficient and flexible moo in future applications.