DRBO : Mitigating the Bottleneck Effect via Dynamic Reward Balancing in Multi-reward LLM Optimization (2025)

Abstract: In the current landscape of large language models (LLMs), many evaluation metrics have been developed and used as rewards during training to improve specific metrics. However, balancing these metrics and dynamically adjusting reward weights remains challenging, as current approaches often fail to enhance weaker metrics. To address this, we empirically propose a D ynamic R eward B alancing O ptimization framework DRBO to mitigate the “bottleneck effect” by measuring performance, adjusting reward weights to prioritize weaker metrics, and optimizing the model via reinforcement learning. We apply DRBO to both single-task and multi-type task scenarios, validating its effectiveness in generation with citations and online shopping conversation tasks. The results demonstrate improved overall performance and balanced optimization across multiple metrics, effectively overcoming the diversity and complexity inherent in LLMs. Our codes are available at https://github.com/NuoJohnChen/DRBO .

## introduction evaluation is crucial for llm applications, which often necessitates adherence to diverse evaluation criteria @xcite , even for a single task. thanks to their generalization, llms are also expected to excel across multiple types of tasks simultaneously @xcite , and each type of task might have its own metrics. in reinforcement learning, one could structure these metrics (previously for evaluation) as rewards that could be boosted during training @xcite @xcite @xcite , to optimize complex objective functions even at testing time @xcite . however, when reward weights remain static, the weakest figure 1 : drbo adjusts rewards and weights dynamically through iterations of three main stages: performance metric evaluation, reward weight balance, and model parameter optimization. elements of this schema are from the binding of isaac. metric (the "bottleneck") becomes a bottleneck that restricts overall llm effectiveness, which introduces the "bottleneck effect" in multi-reward optimization. for example, in figure 2 , when the scaled reward itself (or its growth trend) has not yet reached saturation, its update magnitude should accordingly be increased. the mitigation of such effect is crucial because, for the optimization of llms' comprehensive capabilities, we typically aim for the model to avoid weaknesses in any individual metric, ensuring that its performance reaches a state of dynamic equilibrium across all metrics. to mitigate the bottleneck effect, we propose a framework called drbo to balance multiple rewards (depicted in figure 1 ). drbo interleaves between evaluation, balance, and optimization stages, to alternately update the rewards and model parameters. after performance evaluation, we dynamically reduce the weight of saturated rewards, enabling the model to prioritize weaker areas and mitigate the bottleneck effect in balance stage. during optimization, we update the model through the weight of drboaverage is averaged and static. the weight of the bottleneck (accuracy) correspondingly increases. ## methodology of drbo to address this challenge, we propose the algorithm drbo to mitigate the bottleneck effect, which means improving the model's performance should focus on the weaker metrics. additionally, the performance across all metrics should be balanced according to pre-set weights. drbo algorithm achieves this goal through an iterative cycle involving three key phases detailed in the subsections, as colored in algorithm 1: ## applications in this section, we introduce the two configurations of drbo : the single task setting and the multi-type task setting, along with their respective application scenarios. single-task setting the model is evaluated using multiple metrics on the same dataset, similar to how a species adapts to different environments. by dynamically adjusting weights, the model can effectively allocate resources and optimize its performance across all metrics. multi-type task setting the model handles different types of tasks, each with its own specific metrics and datasets. dynamic reward weight adjustment helps the model balance its performance across all tasks, ensuring comprehensive optimization, similar to how a species adapts to different ecological niches. the adaptability makes it an ideal choice for multi-task competitions or benchmark challenges. single-task application on long-context generation with citation single-task drbo can effectively address the challenges faced by longcontext retrievers or citation-based text generation @xcite , such as handling complex queries, providing accurate results, and ensuring credible references. by dynamically adjusting the weights of metrics on fluency, correctness, and citation quality, drbo can achieve balanced optimization across these aspects, thereby improving the model's ability to synthesize information from multiple sources. multi-type task application on online shopping conversation skills multi-type task drbo can handle challenges presented by online shopping interactive conversations @xcite , which includes shopping skills such as concept understanding, knowledge reasoning, user behavior alignment, and multi-lingual abilities. by dynamically adjusting the weights of criterias including multiple choice, geneation, retrieval, and so on, drbo ensures balanced optimization across these tasks, improving the model's ability to serve as general shop assistants. ## experiments to demonstrate drbo 's methodological innovation and its immediate applicability, the experiments is conducted in both multi-evaluation and multi-task settings. we systematically evaluate the effectiveness of drbo through a series of experiments designed to test its performance across the scenarios detailed in section 5. ## conclusion we proposed drbo framework to address the bottleneck effect in multi-reward optimization for llms. by dynamically adjusting reward weights, drbo effectively prioritizes weaker metrics to achieve a balanced and improved overall performance. experimental results across single-task and multi-type task scenarios demonstrate that drbo significantly enhances model performance, ensuring that no single metric dominates optimization. furthermore, drbo provides a flexible and adaptive mechanism that can be integrated into various tasks, highlighting its versatility and scalability. we hope it opens pathways for future advancements in dynamic reward adjustment strategies and broader applications in ai optimization.