RIVAL : Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation (2025)

Abstract: Large language models (LLMs) possess strong multilingual capabilities, and combining Reinforcement Learning from Human Feedback (RLHF) with translation tasks has shown great potential. However, we observe that this paradigm performs unexpectedly poorly when applied to colloquial subtitle translation tasks. In this work, we investigate this issue and find that the offline reward model (RM) gradually diverges from the online LLM due to distributional shift, ultimately leading to undesirable training outcomes. To address this, we propose RIVAL, an adversarial training framework that formulates the process as a min–max game between the RM and the LLM. RIVAL iteratively updates the both models, with the RM trained to distinguish strong from weak translations (qualitative preference reward), and the LLM trained to enhance its translation for closing this gap. To stabilize training and improve generalizability, we also incorporate quantitative preference reward (e.g., BLEU) into the RM, enabling reference-free quality modeling aligned with human evaluation. Through extensive experiments, we demonstrate that the proposed training framework significantly improves upon translation baselines.

## introduction recent advances in pre-trained large language models (llms) have yielded state-of-the-art results across a wide range of benchmarks @xcite @xcite . in the field of neural machine translation (nmt), researchers have also been exploring ways to leverage the powerful capabilities of llms to improve translation quality. although most existing methods @xcite figure 1 : the illustration highlights the issue arising when directly applying vanilla rlhf to a colloquial subtitle dataset. the weak translator tends to generate content that is not present in the source text but receives a favorable score from the rm. currently, most nmt systems @xcite focus on formal and written language, with limited research addressing loosely structured and colloquial subtitles. to bridge this gap, we first construct a dedicated dataset and apply rlhf to optimize the translation quality for this task. this dataset is characterized by its diversity in both linguistic styles and domains. however, we find that the performance of vanilla rlhf is suboptimal. as shown in figure 1 , the model tends to generate content that does not exist in the source text for cheating the rm to obtain a high score, which is usually called reward hacking. through analysis, we identify that this issue arises from the continuous distributional shift @xcite of the llm during rl training, indicating that offline rm is increasingly ineffective in providing reliable supervision signals, particularly in the context of online rl training. to address the issue of distribution shift, we propose rival, (reinforcement learning with iterative and adversarial optimization) an approach inspired by the adversarial training paradigm of generative adversarial networks (gans) @xcite . we reformulate the conventional rlhf training scheme as an explicit min-max game between the rm and the llm. the rm is trained to maximize the score gap between the translation pairs produced by strong translators and weak translators. we name this signal the qualitative preference reward. the llm training objective is to minimize the gap in translation quality between itself and strong translators. by iteratively optimizing both models, the rm is continuously updated to adapt to the distributional shifts introduced during llm training. in experiments, we further observe that as training progresses, the rm may learn a distribution divergent from the true reward signal. we attribute this issue to the large exploration space inherent in translation tasks, where purely qualitative preference reward may not consistently ensure beneficial optimization. to mitigate this, we introduce quantitative preference rewards (e.g., bleu @xcite ) into our adversarial framework to stabilize the iterative training process. however, directly using these signals suffers from poor generalizability @xcite and makes them unsuitable as dynamic rewards. we extend the rm to simultaneously predict this type of quantitative preference reward for incorporating them into our adversarial framework. this approach enables more generalized reference-free modeling of translation quality while maintaining alignment with human evaluation criteria through indirect reference guidance. our core contributions are as follows: ## background in the bradley-terry model (bradley and @xcite , a pairwise choice between items i and j depends on their latent utilities. the probability of selecting the item i increases monotonically with its utility relative to that of j. formally, these probabilities are obtained by applying a softmax function to the log-utilities r(•) of the items. while this formulation elegantly captures static preference judgments, real-world applications-particularly in natural-language generation often require aligning model behavior with nuanced human values. rlhf addresses this gap by leveraging human evaluators to provide comparative or scalar feedback on model outputs. first, a rm is trained to predict these human judgments from pairs or ratings of model candidates @xcite . next, the llm is fine-tuned via policy optimization methods such as proximal policy optimization (ppo) to maximize the learned reward signal @xcite . this two-stage process has proven effective in producing more helpful, truthful, and harmless generations in large-scale language systems @xcite . in this paper, rlhf is not exclusively limited to human feedback, but also encompasses model-generated feedback. ## pilot experiment and analysis in this section, we first explore the application of vanilla rlhf to nmt and conduct a preliminary analysis of the experimental results. ## methods based on the experimental findings in the previous section, we observe that vanilla rlhf is not well-suited for colloquial subtitle translation tasks. therefore, inspired by the principles of gans @xcite , we propose an adversarial rl approach, rival, that enables both the rm and the llm to progressively improve from weak to strong. specifically, we formulate the two-stage training process of rlhf as a min-max adversarial game between llm π θ (generator) and rm r ϕ (discriminator) , as expressed by the following objective: ## experiment we will introduce experimental setup, main results and analysis in this section. ## related works here we present a concise overview on rl-based method for the mt(details in appendix g). reinforcement learning for machine translation early rl studies alleviated the training-evaluation gap by directly optimizing corpus-level bleu and alleviating exposure bias @xcite @xcite . later work used human-in-the-loop feedback for domain or user adaptation @xcite . reward-shaping variants inject intermediate signals, yet even token-wise bleu remains too coarse for fine-grained rewards @xcite @xcite . modern rl-enhanced llms such as openai o1 and deepseek-r1 achieve state-of-the-art performance accross diverse benchmarks @xcite . motivated by these, r1-translator introduces chain-of-thought alignment for mt @xcite ; mt-r1-zero adds zero-shot hybrid rewards @xcite . remedy frames mt evaluation as pairwise reward modeling @xcite , while severity-weighted token rewards from xcomet provide richer signals @xcite . reward hacking in reinforcement learn-ing reward hacking happens when an rl agent finds loopholes in the reward and chases the proxy instead of the real goal @xcite . it has appeared in robotics, atari-style games, and rlhf language models, where agents loop waypoints, rewrite tests, or game metrics to boost scores without real gains @xcite @xcite . to prevent the above issue, recent work suggests the following methods of defence: (1) fix the reward with ensembles @xcite , adversarial training, caps, or trip-wires @xcite ; (2) harden the policy through look-ahead planning, adversarial blinding, or indifference methods ( uesato et al., 2020) (3) add human checks, anomaly detection, and data audits after training @xcite . ## conclusion using a real-world colloquial subtitle dataset, we find that vanilla rlhf struggles to improve translation quality due to distributional shifts from offline rm. to address this, we propose rival-an adversarial framework enabling iterative optimization between the rm and the llm, incorporating both qualitative and quantitative preference rewards. through extensive experiments and analysis, we demonstrate the effectiveness and generalizability of this framework. our work offers valuable insights for applying rl to nmt, and more broadly, serves as a reference for rl-based approaches in general post-training of llms.