Eliciting Better Multilingual Structured Reasoning from LLM s through Code (2024)

Abstract: The development of large language models (LLM) has shown progress on reasoning, though studies have largely considered either English or simple reasoning tasks. To address this, we introduce a multilingual structured reasoning and explanation dataset, termed xSTREET, that covers four tasks across six languages. xSTREET exposes a gap in base LLM performance between English and non-English reasoning tasks.We then propose two methods to remedy this gap, building on the insight that LLMs trained on code are better reasoners. First, at training time, we augment a code dataset with multilingual comments using machine translation while keeping program code as-is. Second, at inference time, we bridge the gap between training and inference by employing a prompt structure that incorporates step-by-step code primitives to derive new facts and find a solution. Our methods show improved multilingual performance on xSTREET, most notably on the scientific commonsense reasoning subtask. Furthermore, the models show no regression on non-reasoning tasks, thus demonstrating our techniques maintain general-purpose abilities.

## introduction the ability to perform complex reasoning tasks is fundamental to human intelligence, where multiple steps of thought are required. complex reasoning remains an open-problem for large language models (llms), despite some recent progress. prior works consider complex reasoning tasks specified only in english. such an english-centric perspective provides a limited assessment of the underlying reasoning capabilities of llms, given any specific language is largely a surface-form representation. base llm reasoningenhanced llm finetune original train data finetune infer tcc data reasoning tasks domain shift between training and inference! predictions ru es en zh ar ja un conejo blanco puede saltar 15 metros en un minuto. el conejo marrón salta 12 metros por minuto. ¿cuál es la distancia total que saltarán los dos conejos en 5 minutos? @xmath0, '她在五月份卖出 的片段只有原来的一半。'] @xmath1.append(question) @xmath2 figure 1 : an overview of our methods to improve multilingual structured reasoning. first (top), we create the translated code comments (tcc) dataset, and use it in a fine-tuning setup. second (bottom), we use the resulting llm for inference on reasoning tasks. we find the most success with a code prompt format that bridges the representations between training and inference. this motivates our first inquiry into the multilingual complex reasoning capabilities of llms. we introduce the xstreet reasoning and explanation dataset (as shown in figure 2 ). xstreet covers 4 tasks, and extends the english street benchmark @xcite to 5 additional diverse languages, inheriting the source's expert annotations and structured graphs for reasoning steps (7.8 average steps/answer). the tasks cover arithmetic, logic and science commonsense problems. we perform machine translation for the training and development data splits, and also perform human post-editing to the test sets, to ensure a high quality multilingual benchmark. we use xstreet to evaluate several llms, identifying the multilingual setting as significantly challenging. to remedy the non-english reasoning gap, we turn to the widely accepted hypothesis that llms trained on code are better at reasoning than those trained only on text. this code and reasoning hy-a white rabbit can hop 15 meters in one minute. the brown rabbit hops 12 meters per minute. what is the total distance the two rabbits will hop in 5 minutes? white rabbit + brown @xmath3 the two rabbits will hop 135 meters in 5 minutes. un conejo blanco puede saltar 15 metros en un minuto. el conejo marrón salta 12 metros por minuto. ## related work we adopt the scope of complex reasoning from @xcite , and use complex reasoning and structured reasoning interchangeably. the goal is to study the reasoning process itself, and how an llm can structure its output in steps to improve the final performance. the tasks are selected so that the knowledge to answer a question is contained with the input question and context. for ease of evaluation, the answers are multiple-choice selections, or numbers for arithmetic reasoning. further details are given in §2.4. ## multilingual complex reasoning benchmark: xstreet we create the xstreet dataset by translating street into 5 languages: arabic (ar), spanish (es), russian (ru), chinese (zh), and japanese (ja). these languages have linguistic and script diversity; furthermore, they are the languages used in many online programming help websites. to create the xstreet test split, we hire expert human translators for all 5 languages through an internal team (detailed in §9). translators are tasked with post-editing the machine translation of one sentence at a time; for context, they can refer to the entire street entry the sentence comes from. after receiving the translations, we re-use the reasoning graph edges, and replace english nodes with the translations to create xstreet. this process is shown in figure 2 . we therefore extend the 914 english entries in street to 5484 examples in xstreet (914 * 6 languages). to create the xstreet train and development splits, we use machine translation. 5 we then asked native speakers to evaluate the quality of 10 random sampled translations of each language. annotators gave feedback that, despite some errors, the translations were of reasonable enough quality to use for training purposes. dataset statistics for the xstreet test benchmark are given in ## code with multilingual comments as indirect supervision for reasoning taking the idea of using code for reasoning, and comments for multilinguality a step further, we address the question: can multilingual code serve as indirect supervision for multilingual reasoning? in other words, we investigate whether the code & reasoning hypothesis holds multilingually. we therefore propose a lightweight fine-tuning recipe, which consists of creating a multilingually commented code dataset, then fine-tuning on it, which serves as indirect supervision for downstream reasoning tasks. ## multilingual complex reasoning as a downstream task we hypothesize that structure, when applied to reasoning problems formulated in different languages, can abstract away some of the language-specific details, better surfacing the reasoning steps needed for a model. we thus propose the sim (selectand-infer multilingual comments) code prompts for complex reasoning tasks. sim code prompts utilize several functions. we do not provide the api definitions, instead, we expect the model to learn to use them from the in-context examples. the functions are: a key difference, though, is that we use a single prompt, instead of iterative prompts. we therefore include is_solved(fact, question) as a signal for the llm to stop generation. each function is annotated with its return value in an inline code comment. this is inspired by prior work @xcite . infer_new_fact has a string return value, i.e., the text of the new fact. we experiment with two versions of the return value of select_facts. the first, termed simindexed, uses variables facts[i] to reference the facts array (similar to the indices used in linearized format). the second, termed sim-text, directly uses each fact's text, dereferenced from facts @xcite . we find that sim-text works best for smaller models, while sim-indexed does for larger ones, and hence apply this going forward. we write a rule-based python script that converts existing structured graph annotations to sim code prompts. sim prompts express the exact same information as the linearized format. this property is unlike code prompts for prior work, wherein the conversion is done through in-context learning with an llm, which can introduce errors as discussed in §2.2. the different prompting formats for llms are shown in figure 3 . ## experimental setup models used we primarily study open-source models, which allows for application of both train and inference-time techniques. we use bloomz @xcite as our base llm. this model is instruction-finetuned on prompts in 46 natural languages and 10 programming languages. for our experiments, we consider the 7.1b-parameter bloomz, as well as bloomz-tcc which is further finetuned on tcc. for inference-time only, we consider two larger llms. we use the instruction-finetuned version of falcon @xcite (40b ), another open-source llm trained on text+code. compared to bloomz, falcon is more performant on english tasks; however, it has limited multilingual abilities. 9 we also use gpt-3 (175b) foot_8 , a closedsource model that is popularly-used and powerful. prompting setup we use few-shot prompting, and random sample up to 5 exemplars from the train split (up to a model's context length). 11 for each inference example, the same exemplars are used for all models and prompt types. we use greedy decoding, and task the model with generating up to 682 tokens. (max context length of bloomz 2048 // 3). in the code prompts, we color code facts which are aligned. ## results we report results on the xstreet benchmark. we use the answer accuracy metric, adapting evaluation from ribeiro et al. (2022). 13 given the extensive nature of the xstreet benchmark and our model experimentation, we highlight our findings iteratively. we first consider only bloomz and bloomz-tcc, with a particular focus on arc, where our methods are the most impactful. we then consider gpt-3 and falcon. the full results are given in appendix d. here we provide numbers for all tasks, languages, models, and prompt formats (this also includes the direct prompting format). ## analysis to further understand wherein our techniques help, or fail to help, model reasoning, we perform some manual analysis. for brevity, we focus on 2 languages, en and ar, and 2 tasks, arc and gsm8k. we first perform error analysis on bloomz, then perform a case study for each task. we further perform 3 additional experiments, which are detailed in appendix §e. to highlight one interesting finding, we show that training on diverse code comments, such as from the multilingual tcc, boosts xstreet performance in all languages including english. error analysis for bloomz english for this task, and with base bloomz, sim achieves 61.5 arc accuracy, while linearized achieves 35.3. our manual analysis of outputs reveals that the performance discrepancy is largely due to poor instruction-following when using linearized vs. using sim. @xcite find that for linearized (and their model), 62% of generations fail to generate a parsable answer (i.e., reasoning graph is incomplete). our findings concur, in that linearized has 66% (223/340) invalid generations. in contrast, sim has only ˜19% invalid. bloomz-tcc with sim further reduces invalid rate to 9%, and increases accuracy to 76.2. we observe that in cases where all formats output successfully, the reasoning graph and answers are nearly identical. the difference is that sim prompts allows the model to generate a complete reasoning graph far more often. we reiterate that this behavior is a novel finding given bloomz-tcc was indirectly supervised on code, rather than directly on reasoning tasks. further discussion is found in appendix a. we summarize this section with the following view: our techniques elicit better instruction-following of the proscribed reasoning format from a base llm, leading to improved benchmark performance. within a reasoning step, the models are making similar decisions, but at the reasoninggraph level, our methods assist in harder cases. ## conclusion we introduced xstreet, a multilingual structured reasoning benchmark which covers 5 diverse languages , spans science commonsense, arithmetic and logical reasoning tasks, and includes highquality intermediate reasoning steps. we found that current multilingual llms underperform in the non-english setting, then proposed two methods to remedy this, based on the popular hypothesis that llms trained on code are better reasoners. at training, we propose translating the comments of a source code dataset, to use as indirect supervision data for parameter-efficient fine-tuning. during inference, we leverage code structure to represent reasoning graphs. we perform extensive experimentation, and both of our methods better elicit underlying reasoning abilities of llms. our work brings together two areas of challenge for llms -multilinguality, and complex reasoning. in particular, our fine-tuning recipe shows that the code & reasoning hypothesis can apply multilingually. we suspect that improvements can be amplified if multilingual comments are included at the pre-training, instead of the fine-tuning stage. we hope our findings underscore the key role that code should play in the development of llms with better reasoning capabilities across languages.