Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations (2024)

Abstract: Existing research predominantly focuses on developing powerful large language models (LLMs) for mathematical reasoning within monolingual languages, with few explorations in preserving efficacy in a multilingual context. To bridge this gap, this paper pioneers exploring and training powerful Multilingual Math Reasoning (xMR) LLMs. Firstly, by utilizing translation, we construct the first multilingual math reasoning instruction dataset, **MGSM8KInstruct**, encompassing ten distinct languages, thus addressing the issue of training data scarcity in xMR tasks. Based on the collected dataset, we propose different training strategies to build powerful xMR LLMs, named MathOctopus, notably outperform conventional open-source LLMs and exhibit superiority over ChatGPT in few-shot scenarios. Notably, MathOctopus-13B reaches 47.6% accuracy which exceeds ChatGPT 46.3% on MGSM testset. Beyond remarkable results, we unearth several pivotal observations and insights: (1) When extending the rejection sampling strategy to the multilingual context, it proves effective for model performances, albeit limited. (2) Employing parallel corpora for math Supervised Fine-Tuning (SFT) across multiple languages not only significantly enhances model performance multilingually but also elevates their monolingual performance. This indicates that crafting multilingual corpora can be regarded as a vital strategy for enhancing model performance in a specific language, especially in mathematical reasoning tasks. For instance, MathOctopus-7B improves its counterparts that trained on English from 42.4% to 50.8% on the GSM8K test set.

## introduction large language models (llms) @xcite @xcite @xcite ) such * corresponding author. as @xcite and gpt4 (openai, 2023) have exhibited remarkable performances across a wide array of downstream tasks. notwithstanding these significant advancements, even the most extensive llms are confronted with challenges when faced with mathematical reasoning tasks that necessitate multiple reasoning steps @xcite . ## methodology in this section, we aim to illustrate our method in detail. we first review the problem formulation of multilingual mathematical reasoning. then we describe the collection process of mgsm8kinstruct. subsequently, we present training strategies of our mathoctopus and multilingual rejection sampling methods, sequentially. ## experiments in this section, we first review our in-domain evaluation dataset: mgsm, and the collection of the out-of-domain testset: msvamp. subsequently, we present the main results and findings of our experiments. we illustrate our experimental setup for training and testing in appendix b. ## discussion in this section, we delve deeper through a series of meticulous experiments designed to address the following inquiries: ## conclusion in this paper, we pioneer the exploration of training multilingual mathematical llms. to address data scarcity in low-resource languages, we first collect the first multilingual math reasoning instruction dataset, named mgsm8kinstruct, consisting of ten various languages. the models, trained on mgsm8kinstruct with different training strategies, named mathoctopus, show superior performances compared to other open-source llms. we prove that mathoctopus with paralleltraining could achieve better in-domain test results while mathoctopus with cross-training presents better robustness in the collected out-of-domain test set, msvamp. we also investigate the impact of the multilingual rejection sampling strategy, finding it has a limited effect on xmr tasks. our extensive experiments reveal that creating aligned bilingual question-answer corpora significantly improves the model's mathematical capabilities in its native language. in future work, we will explore additional methodologies and diverse parallel corpora for training xmr llms, potentially involving rlhf.