MAPO : Advancing Multilingual Reasoning through Multilingual-Alignment-as-Preference Optimization (2024)

Abstract: Intuitively, reasoning abilities are considered language-agnostic. However, existing LLMs exhibit inconsistent reasoning abilities across different languages, e.g., reasoning in the dominant language like English is superior to other languages due to the imbalance of multilingual training data. To enhance reasoning abilities in non-dominant languages, we propose a Multilingual-Alignment-as-Preference Optimization framework (MAPO) to align the reasoning processes in other languages with the dominant language. Specifically, we harness an off-the-shelf translation model for the consistency between answers in non-dominant and dominant languages, which we adopt as the preference for optimization, e.g., Direct Preference Optimization(DPO) or Proximal Policy Optimization (PPO). Experiments show that MAPO stably achieves significant improvements in the multilingual reasoning of various models on all three benchmarks (MSVAMP +16.2%, MGSM +6.1%, and MNumGLUESub +13.3%), with improved reasoning consistency across languages. The project is available at https://github.com/NJUNLP/MAPO.

## introduction the reasoning ability of large-scale models has attracted much attention @xcite @xcite . though we consider reasoning to be languageagnostic, existing studies @xcite show that due to the imbalance of pre-training and finetuning data across languages, the mathematical reasoning ability of existing large-scale models varies across languages, e.g., english, is far superior to that of the other languages. to improve the reasoning ability in other languages, @xcite translated the english reasoning processes into other languages for supervised fine-tuning @xcite . although sft brings in preliminary capability for multilingual reasoning, we argue that two problems hinder further improvement. firstly, the annotation for the reasoning process is expensive to obtain even for the dominant language, and the reasoning processes involve complex mathematical reasoning, which may result in translation errors @xcite . as a result, the translated reasoning annotation for nondominant languages is limited in both scale and quality. without sufficient and diverse data, the results of supervised training are limited. it may also suffer from generalization issues in versatile task scenarios @xcite , indicating potential difficulty in adapting to the out-of-domain test set. more importantly, although supervised training with translated reasoning processes improves the reasoning ability for almost all trained languages, this strategy only fills in the missing reasoning annotation for non-dominant languages that originate from the dominant language. thus the inherent gap between dominant and non-dominant languages is hardly narrowed. different from existing attempts, we propose to use the reasoning ability of the dominant language as the director for improving non-dominant languages. as the reasoning process is critical for obtaining the correct result, the multilingual reasoning ability may be improved by encouraging the reasoning in non-dominant languages to be similar to that in the dominant language. therefore, we propose a multilingual-alignment-as-preference optimization (mapo) framework by aligning the reasoning process of non-dominant languages to the dominant. notably, mapo exploits the strong reasoning ability in the dominant language and requires no annotation for the reasoning process. more specifically, mapo consists of two stages: preference estimation via multilingual alignment and preference optimization. during preference estimation, the reasoning processes to the same question are sampled from the llm in both dominant and non-dominant languages. a well-trained, off-the-shelf translation model is employed to yield the translation probability between the reasoning in dominant and non-dominant languages. since higher translation probability indicates a more consistent reasoning aligned with the dominant language, the corresponding reasoning in the nondominant language is considered better and shall be promoted. during preference optimization, proximal policy optimization @xcite and direct preference optimization @xcite are adopted to optimize the previously estimated preference. so the llms are trained to reason in the non-dominant languages as they do in the dominant language. we also conduct iterative dpo for further preference optimization. experiments are conducted on three challenging multilingual reasoning test sets, namely msvamp @xcite , mgsm @xcite , and mnumgluesub constructed from numglue @xcite , each covering 10 languages. mapo achieves accuracy improvements of up to 16.2%, 6.1%, and 13.3% on the three benchmarks, respectively, reaching state-of-the-art performance on 7b models, thereby demonstrating the effectiveness of our method. among them, the improvement on the out-of-domain dataset msvamp (+16.2%) shows that, by enhancing multilingual reasoning consistency, mapo improves the multilingual reasoning capabilities of the model in a generalizable manner. ## analysis 6.1 preference estimating on a dataset different from sft we conduct experiments with preference optimization using the mgsm8kinstruct dataset instead of the mnumgluesub. ## conclusion in this paper, we propose mapo, a novel multilingual alignment-as-preference optimization framework, enhancing reasoning ability in non-dominant languages by aligning them with dominant languages. experimental results demonstrate that our framework achieves significant improvements on various base models across all three benchmarks, especially with a no