MT -R1-Zero: Advancing LLM -based Machine Translation via R1-Zero-like Reinforcement Learning (2025)

Abstract: Large-scale reinforcement learning (RL) methods have proven highly effective in enhancing the reasoning abilities of large language models (LLMs), particularly for tasks with verifiable solutions such as mathematics and coding. However, applying this idea to machine translation (MT), where outputs are flexibly formatted and difficult to automatically evaluate with explicit rules, remains underexplored. In this work, we introduce MT-R1-Zero , the first open-source adaptation of the R1-Zero RL framework for MT without supervised fine-tuning or cold-start. We propose a rule-metric mixed reward mechanism to guide LLMs towards improved translation quality via emergent reasoning. On the WMT 24 English-Chinese benchmark, our MT-R1-Zero-3B-Mix achieves competitive performance, surpassing TowerInstruct-7B-v0.2 by an average of 1.26 points. Meanwhile, our MT-R1-Zero-7B-Mix attains a high average score of 62.25 across all metrics, placing it on par with advanced proprietary models such as GPT-4o and Claude-3.5-Sonnet, while the MT-R1-Zero-7B-Sem variant achieves state-of-the-art scores on semantic metrics. Moreover, our work exhibits strong generalization capabilities on out-of-distribution MT tasks, robustly supporting multilingual and low-resource settings. Extensive analysis of model behavior across different initializations and reward metrics offers pioneering insight into the critical role of reward design, LLM adaptability, training dynamics, and emergent reasoning patterns within the R1-Zero paradigm for MT. Our code is available at https://github.com/fzp0424/MT-R1-Zero.

## introduction large-scale reinforcement learning (rl) has empowered large language models (llms) with strong reasoning capabilities (openai, 2024; team, † corresponding author. 2025a,b), demonstrating significant success in tasks such as mathematical reasoning or coding in which answers can be clearly verified. in particular, deepseek-r1-zero @xcite introduced a pure rule-based rl approach that directly fosters emergent reasoning ability without requirements on structured chain-of-thought (cot) data @xcite or sophisticated techniques such as monte carlo tree search (mcts) @xcite @xcite . however, the applicability of these methods to machine translation (mt) remains challenging and underexplored, as mt outputs are flexibly generated and hard to evaluate automatically with explicit rules. recent work has launched attempts to empower llms for mt with reasoning capabilities @xcite . early studies investigate explicit reasoning methods for improved translation, such as finetuning with cot @xcite or mcts @xcite , where advanced multi-step pipelines with self-correction or long-thought agentic mechanisms are further ex-plored @xcite . another line of work leverages rl to empower llms for mt through process reward models or supervised finetuning (sft) with manually annotated cot data @xcite . however, these methods often depend on manually designed or synthetically generated structured cot data, rely on complex search algorithms, or require explicit multi-stage prompting, leaving the potential of pure rl-based approaches largely unexplored. furthermore, the performance reported in these studies often lags behind state-of-the-art (sota) open-source or proprietary models. developing pure rl methods to directly enhance the reasoning ability of llms for better translation requires answering three key questions: 1) feasibility: how to design r1-zero-like rl pipelines with effective reward signals to directly solve mt tasks without binary rule-based rewards; 2) reasoning capability: could pure rl training cultivate emergent reasoning abilities and induce models to generate explicit thinking patterns for mt, such as multi-step cot or verification/reflection; 3) generalizability: could the training paradigm generalize across different models (e.g., pre-trained base models, instruction-tuned models, or models pretrained on translation data) or diverse downstream settings (e.g., out-of-distribution, multilingual or low-resource scenarios). in this work, we introduce mt-r1-zero, the first open-source implementation that extends the r1-zero-like rl training paradigm to mt. we propose a rule-metric mixed reward mechanism that adapts the original rule-based reward concept to effectively guide training in mt scenarios. we explore different rewards optimizing over lexical (lex), semantic (sem), and lex-sem mixed (mix) objectives to guide llms towards improved translation quality via emergent reasoning. our experiments demonstrate the efficacy of this approach: as rl training progresses, our mt-r1-zero-3b-mix achieves competitive performance, surpassing towerinstruct-7b-v0.2 by an average of 1.26 points across all metrics (bleu, cometkiwi, xcomet) on the wmt 24 english-chinese (en-zh) benchmark. meanwhile, our mt-r1-zero-7b-mix surpasses llama-3.1-70b by an average of 1.24 points and qwen2.5-72b by 0.48 points, even on par with top proprietary models such as gpt-4o and claude-3.5-sonnet. the mt-r1-zero further demonstrates promising generalizability across multilingual and low-resource settings. extensive experiments further provide key findings and insight into the adaptation of r1-zero paradigm to mt. first, we empirically demonstrate that the choice of metric reward plays a pivotal role in steering rl optimization and translation style (semantic or lexical) (finding 1). further analysis reveals that mt-r1-zero induces diverse emergent reasoning patterns, including dynamic language-ofthought transition during translation (finding 2). we also identify distinct rl adaptability of different llms @xcite . ablation studies suggest that pure rl process alone can lead to substantial translation improvements, independent of thinking verbosity. our core contributions are as follows: ## method in this section, we present our method that trains a translation model with pure rl using a hybrid reward system. unlike tasks with fixed correct answers, translation allows for multiple valid outputs, making the evaluation more complicated. in this work, we introduce a rule-metric mixed reward that integrates reasoning format checking with multiple translation quality assessment metrics, which is used within the group relative policy optimization (grpo) @xcite algorithm to ensure stable and efficient rl training. ## conclusion in this work, we introduced mt-r1-zero, the first successful adaptation of r1-zero rl framework to mt using a novel rule-metric mixed reward mechanism that combines format enforcement with quality metrics. our mt-r1-zero significantly im-proves translation quality, achieving leading results on multiple benchmarks, i.e., our 3b models compete with much larger open-source models, while our 7b models are on par with advanced proprietary models. the mt-r1-zero also demonstrates strong ood generalization and multilingual applicability. through extensive experiments and analysis, we highlight the significant impact of reward metric choice for optimization, showcase distinct adaptability across different llms, and reveal that performance gains are principally from the rl process itself rather than reasoning steps or verbosity, establishing r1-zero as a viable and potent paradigm for advancing mt. more broadly, our work highlights the great potential of rl for diverse language processing tasks beyond translation.