Question Translation Training for Better Multilingual Reasoning (2024)

Abstract: Large language models show compelling performance on reasoning tasks but they tend to perform much worse in languages other than English. This is unsurprising given that their training data largely consists of English text and instructions. A typical solution is to translate instruction data into all languages of interest, and then train on the resulting multilingual data, which is called translate-training. This approach not only incurs high cost, but also results in poorly translated data due to the non-standard formatting of mathematical chain-of-thought. In this paper, we explore the benefits of question alignment, where we train the model to translate reasoning questions into English by finetuning on X-English parallel question data. In this way we perform targeted, in-domain language alignment which makes best use of English instruction data to unlock the LLMs’ multilingual reasoning abilities. Experimental results on LLaMA2-13B show that question alignment leads to consistent improvements over the translate-training approach: an average improvement of 11.3% and 16.1% accuracy across ten languages on the MGSM and MSVAMP multilingual reasoning benchmarks.

## introduction large language models have recently shown a strong ability to reason in english, but performance in other languages, especially more distant languages, still trails far behind @xcite . it is unsurprising, considering that their training data is predominantly composed of english text and instructions @xcite @xcite . to elicit llm's multilingual performance, previous approach typically follows the translatetraining paradigm @xcite , which first translates english instruction data into non-english with a translation engine and then uses the multilingual data for instruction-tuning. however, the translate-training has the following drawbacks: (1) translating english training data to numerous non-english languages incurs significant translation cost, especially considering the constant addition of large and complex instruction tuning sets @xcite . (2) additionaly, it is hard for the translation engine to accurately translate lengthy, logical texts containing mathematical symbols in chain-of-thought (cot) responses, which can compromise the quality of translated data (evidence are shown in appendix a). consequently, we explore the following research question in this paper: can we unlock the llm's multilingual reasoning ability by teaching it to translate reasoning questions into english? in this paper, we focus on the multilingual mathematical reasoning task and explore the benefits of question alignment (qalign), where we fine-tune the pre-trained llm to translate reasoning questions into english with x-english parallel question data. this targeted, in-domain language alignment enables the subsequent effective utilization of english instruction data to unlock llms' multilingual reasoning abilities. following question alignment, we implement response alignment by further fine-tuning the language-aligned llm with cuttingedge english instruction data. even when only english supervised data is available, our alignmentenhanced llm can achieve superior performance on non-english tasks with its transferable english expertise. to demonstrate the advantages of question alignment, we conduct experiments on challenging multilingual mathematical reasoning benchmarks, mgsm @xcite and msvamp @xcite . we use two of the most advanced open-source llms, llama2-7b and llama2-13b @xcite , as base models. exper-iment results show that the inclusion of the question alignment stage brings an average improvement of up to 13.2% in multilingual performance. the performance improvement on low-recourse languages, e.g. thai and swahili, can be 30%-40%. compared to the translate-training baseline, math-octopus @xcite , which tuned with a multilingual version of gsm8k dataset , our alignment-enhanced llms achieves average performance improvement of 9.6% (7b) and 11.3% (13b) on mgsm. on the out-of-domain test set msvamp, our fine-tuned llms achieve 13.1% (7b) and 16.1% (13b) average accuracy improvement, also demonstrating our approach is robust to domain shift. in general, we observe that although incorporating translated instruction data does benefit multilingual performance, our question alignment strategy provides a more efficient and effective choice. in our analysis, we also present the effects of other implementations for performing language alignment and illustrate the importance of choosing the appropriate translation direction and domain during this phase of training. the main contributions of this paper can be summarized as: subsequently, many efforts are put into creating effective instruction data to further unlock llm's potential @xcite @xcite . however, since the proposed instruction datasets consist mainly of english, the directly fine-tuned llms struggle on non-english languages, especially on those languages that are dissimilar to english @xcite @xcite . multilingual mathematical reasoning mathematical reasoning is a challenging and representative task for evaluating the intelligence of llms @xcite , where llms need to understand the given math question and produce a numerical answer through step-by-step reasoning. @xcite expanded the scope to a multilingual context by translating english math questions from the gsm8k test set @xcite into non-english languages, thereby creating a multilingual benchmark called mgsm. subsequently, many efforts are put into enhancing llm's multilingual reasoning capabilities, which can be categorized into two approaches: prompting close-source llms and instructiontuning open-source llms. in the first approach, @xcite and @xcite carefully craft prompts for close-source llms like chatgpt @xcite . their strategy involves first prompting the llm to explicitly translate non-english questions into english, then ask the model to solve the translated problem instead. however, the effectiveness of these prompting methods are not well-examined on open-source llms. and it remains an open challenge to equip opensourced llms with strong multilingual mathematical problem-solving skills. in the second approach, @xcite follow the translate-training method @xcite . initially, they translate english instruction data in gsm8k into non-english with chat-gpt, followed by employing multilingual data for instruction-tuning. moreover, @xcite investigate cross-lingual training strategies such as mixing questions and cot responses in different languages, but fail to achieve consistent improvement. although the translate-training approach is effective, it incurs high translation cost and is errorprone foot_1 . it also becomes increasingly impractical training stage i: question alignment training stage ii: response alignment [german question] randy hat 60 mangobäume auf seiner farm. er hat auch 5 weniger als die hälfte so viele kokosnussbäume wie mangobäume. wie viele bäume hat randy insgesamt auf seiner farm? [japanese question] ランディーさんは農場にマンゴーの⽊を60本 持っています。また、彼はマンゴーの⽊の半分から5本少ないコ コナッツの⽊を持っています。彼の農場には合計で何本の⽊があ りますか？ [chinese question] 兰迪在他的农场上有60棵芒果树。他还有比芒 果树数量的⼀半少5棵椰⼦树。兰迪⼀共有多少棵树？ [english question] randy has 60 mango trees on his farm. he also has 5 less than half as many coconut trees as mango trees. how many trees does randy have in all on his farm? [question] what is the total amount that james paid when he purchased 5 packs of beef, each weighing 4 pounds, at a price of $5.50 per pound? to translate vast quantities of augmented data into numerous languages, especially considering recent findings that augmented training data, e.g., meta-mathqa @xcite -which is 50 times larger than gsm8k-greatly enhances llm's reasoning skills. without relying on translated cot responses, in this paper, we present a novel question alignment technique to utilize cutting-edge englishonly supervised data to boost open-source llm's performance on multilingual reasoning tasks. ## methodology an illustration of our devised method is shown in figure 1 . the key idea of our approach is strengthening language alignment within llm before exposing it to english instruction-response pairs. by doing so, we can utilize llm's expertise in english to enhance its performance on non-english tasks. below we introduce the two training stages of our framework: question alignment ( §3.1) and response alignment ( §3.2). ## experiment setting base llm in our experiments, we use two of the most advanced open-source llms, llama2-7b and llama2-13b as the base model. training dataset in the question alignment stage, we utilize multilingual questions from gsm8kinstruct foot_2 @xcite . dur- dataset usage # lang # sample metamathqa training 1 395,000 gsm8kinstruct training 10 73,559 mgsm evaluation 10 2,500 msvamp evaluation 10 10,000 training details we use stanford_alpaca foot_3 as our code base. we use consistent training hyperparameters across two stages of training. at each stage, we fine-tune llm's full parameters for 3 epoch on eight nvidia a100 gpus. the learning rate is set to 2e-5, with a batch size of 128. baseline systems for comparison, we consider following systems which are instruction-tuned from llama2 with diverse training recipes: we also reproduce this model in our experiments, denoted as multireason. • metamath, which is instruction-tuned with metamathqa @xcite . it is currently the most powerful english instruction data for mathematical reasoning. we also reproduce this model in our experiments, denoted as monoreason. among these baseline systems, most models are tuned with english data and only mathoctopus and multireason are tuned with multilingual data. ## main results in this section, we report our experiment results and introduce our main findings. ## analysis 6.1 ablation study impact of using different translation training data during the question alignment stage, we implement the translation task by training llms on translating questions from non-english to english. now we present the ablation study to show the effects of alternative implementations ( for instance, training llms on reverse translation tasks greatly degenerates non-english performance (question:@xmath0, response:@xmath1. the data quality. training the llm with translation data from commonly-used corpora, such as flo-res 8 , does not work as well, indicating that the domain of the translation data is another crucial factor in establishing language alignment. ## conclusion in this paper, we introduce a novel question alignment method to empower llms on multilingual mathematical reasoning tasks without requiring multilingual instruction data. experiment results on mgsm and msvamp benchmarks show that our proposed question alignment stage brings an average improvement of up to 13.2% in multilingual performance. our alignment-enhanced llm outperforms the unaligned counterpart and the translate-training baseline by a large margin and shows a more robust performance. generally, our devised method successfully narrows the gap between llm's performance between english and non-english, showing a new possibility to unlock llm's capabilities to solve multilingual tasks.