MT - R eward T ree: A Comprehensive Framework for Advancing LLM -Based Machine Translation via Reward Modeling (2025)

Abstract: Process reward models (PRMs) have shown success in complex reasoning tasks for large language models (LLMs). However, their application to machine translation (MT) remains underexplored due to the lack of systematic methodologies and evaluation benchmarks. To address this gap, we introduce MT-RewardTree, a comprehensive framework for constructing, evaluating, and deploying process reward models in MT. Unlike traditional vanilla preference pair construction, we propose a novel method for automatically generating token-level preference pairs using approximate Monte Carlo Tree Search (MCTS), which mitigates the prohibitive cost of human annotation for fine-grained steps. Then, we establish the first MT-specific reward model benchmark and provide a systematic comparison of different reward modeling architectures, revealing that token-level supervision effectively captures fine-grained preferences. Experimental results demonstrate that our MT-PRM-Qwen-2.5-3B achieves state-of-the-art performance in both token-level and sequence-level evaluation given the same input prefix. Furthermore, we showcase practical applications where MT-PRMs successfully identify token-level translation differences and enable test-time alignment for LLMs without additional alignment training. Our work provides valuable insights into the role of reward models in MT research. Our code and data are released in https://sabijun.github.io/MT_RewardTreePage.

## introduction the next-token prediction process in large language models (llms) is often modeled as a markov decision process (mdp) and has achieved remarkable success across various domains, largely attributed ## mt-rewardtree in this section, we introduce the components of the mt-rewardtree. we first describe how we construct token-level preference pairs using an mctsbased method. next, we review several approaches employed for reward modeling. ## conclusion in this work, we propose mt-rewardtree, a comprehensive framework for constructing, evaluating, and deploying process reward models in mt. our framework leverages an automatic token-level preference pair generation approach inspired by approximate monte carlo tree search, effectively addressing the challenge of large-scale fine-grained supervision annotation. extensive experiments on both sequence-level and token-level benchmarks demonstrate that our mt-prm achieves advanced performance in reward modeling in mt, surpassing traditional sequence-level preference pairs. our exploration of token credit assignment and test-time alignment provide valuable insights for the application of reward models in mt.