Demystifying Multilingual Reasoning in Process Reward Modeling (2025)

Abstract: Large language models (LLMs) are designed to perform a wide range of tasks. To improve their ability to solve complex problems requiring multi-step reasoning, recent research leverages process reward modeling to provide fine-grained feedback at each step of the reasoning process for reinforcement learning (RL), but it predominantly focuses on English. In this paper, we tackle the critical challenge of extending process reward models (PRMs) to multilingual settings. To achieve this, we train multilingual PRMs on a dataset spanning seven languages, which is translated from English. Through comprehensive evaluations on two widely used reasoning benchmarks across 11 languages, we demonstrate that multilingual PRMs not only improve average accuracy but also reduce early-stage reasoning errors. Furthermore, our results highlight the sensitivity of multilingual PRMs to both the number of training languages and the volume of English data, while also uncovering the benefits arising from more candidate responses and trainable parameters. This work opens promising avenues for robust multilingual applications in complex, multi-step reasoning tasks.

## introduction aligning large language models (llms) with human preferences can significantly improve the model performance across various downstream tasks @xcite . this requires a reward model that is trained on human preference data @xcite @xcite . typically, reward models are trained based on the final outcome of the llms' response, and we refer to these as outcome reward models (orms) (cobbe 1 https://github.com/weixuan-wang123/ multilingual-prm et al., 2021a; @xcite . however, most of recent work demonstrates that orms fall short on complex multi-step reasoning tasks @xcite . to overcome this limitation, process reward models (prms) are introduced, providing fine-grained rewards at each step of the llms' reasoning @xcite @xcite . previous research has shown that llms supervised by prms can effectively produce better responses @xcite . despite these significant advances, recent research on orms and prms has predominantly focused on monolingual settings, particularly english @xcite . however, the exploration of multilingual prms remains relatively limited. therefore, with the advent of multilingual llms, a natural research question arises: how can we effectively train multilingual prms for complex, multi-step reasoning tasks? to address this research question, we translate the existing prm datasets, prm800k @xcite and math-shepherd @xcite , from english into six additional languages, resulting in a total of seven seen languages for training. we then train multilingual prms using the collection of these translated datasets. we define three prm setups: prm-mono, prm-cross, and prm-multi. the prm-mono setup is trained and evaluated solely on a single language, the prm-cross setup is trained on one language but evaluated on all test languages, and the prm-multi setup is trained on seven seen languages and evaluated on all test languages. finally, we conduct a comprehensive evaluation on two popular reasoning tasks (math500 and mgsm) across 11 languages (seven seen languages and four unseen languages) using three llms (metamath-mistral-7b, llama-3.1-8b-math, and deepseekmath-7b-instruct). our main takeaways are summarized as follows: ## related work reward model in mathematical reasoning to advance the accuracy of mathematical reasoning, reward models (rms) have emerged as powerful tools for evaluating and guiding solution generation. in particular, two principal rm paradigms have garnered significant attention: the outcome reward models (orms) @xcite and the process reward models (prms) @xcite @xcite @xcite @xcite . orms assign a single score to an entire solution and thereby focuses on final correctness, whereas prms score each individual step of the reasoning process, offering more finer-grained evaluations. as a result, prms provide more detailed guidance and have demonstrated greater potential in enhancing reasoning capabilities compared to orms @xcite . multilingual reward model beyond englishlanguage tasks, the integration of rms into multilingual scenarios is still under-explored. reinforcement learning approaches often rely on rms predominantly trained on english data @xcite . this over-representation introduces biases, as these rms may overfit to english-specific syntactic and semantic patterns, limiting their effectiveness in cross-lingual tasks and motivating the development of multilingual rms @xcite . while there is growing evidence that cross-lingual transfer is feasible @xcite , existing research often overlooks the unique challenges of multilingual reasoning. after the release of the openai-o1 model (openai, 2024), prms, with their capability for fine-grained feedback, have attracted even greater interest. yet, the performance of multilingual prms in diverse linguistic contexts remains insufficiently investigated @xcite . to bridge this gap, we investigate how multilingual prms contribute to solving mathematical tasks across different languages, aiming to provide insights into how fine-grained process supervision can enhance reasoning capabilities beyond english, thereby contributing to the development of more universally applicable reasoning models. ## experimental setups training datasets we combine the prm800k @xcite and math-shepherd @xcite as training data to finetune prms, and translate the combined dataset from english (en) to six languages: german (de), spanish (es), french (fr), russian (ru), swahili (sw), and chinese (zh) with using nllb 3.3b @xcite . the reasoning step statistics are presented in test dataset we evaluate the performance of llms using two widely used math reasoning datasets, mgsm @xcite and math500 @xcite . for the math500 datset, we translate it from english to ten languages: bengali (bn), german (de), spanish (es), french (fr), japanese (ja), russian (ru), swahili (sw), telugu (te), thai (th), and chinese (zh) with google translate, which is consistent with the languages included in the mgsm dataset. furthermore, we also categorize the languages involved in the downstream tasks into two groups based on the training data of prm: seen languages (en, de, es, fr, ru, sw, and zh) and unseen languages (bn, ja, te, and th). to ensure the quality of our testset, we employ two human translators to post-edit the translated examples for each high-resource language (de, es, fr, ru, zh, and ja) and leverage gpt-4o to revise the translations in low-resource languages (bn, sw, te, and th). more details are shown in appendix b. ## recipes for multilingual prm training in this section, we conduct a series of experiments to investigate the performance of multilingual prm. we examine how prm-multi compares to prm-mono and prm-cross (section 5.1), the impact of the number of training languages (section 5.2), and the effect of varying the proportion of english in the training data (section 5.3). ## analysis in this section, we present a comprehensive analysis of our multilingual prm, focusing on five critical aspects: error positions (section 6.1), number of solutions (section 6.2), integration of lora with prm (section 6.3), comparative evaluation with multilingual orm (see section 6.4), and implement ppo with multilingual prm (see section 6.5). ## conclusion through comprehensive evaluations spanning 11 languages, our work demonstrates that multilingual prms significantly enhance the ability to perform complex, multi-step reasoning tasks in various languages, consistently outperforming both monolingual and cross-lingual counterparts. furthermore, our findings highlight that prm performance is sensitive to the number of languages and the volume of english training data. the multilingal prms also benefit from more candidate responses and model parameters. these results underscore the importance of diverse language training in providing fine-grained rewards and open up promising avenues for multilingual reasoning.