Python is Not Always the Best Choice: Embracing Multilingual Program of Thoughts (2024)

Abstract: Program of Thoughts (PoT) is an approach characterized by its executable intermediate steps, which ensure the accuracy of the logical calculations in the reasoning process. Currently, PoT primarily uses Python. However, relying solely on a single language may result in suboptimal solutions and overlook the potential benefits of other programming languages. In this paper, we conduct comprehensive experiments on the programming languages used in PoT and find that no single language consistently delivers optimal performance across all tasks and models. The effectiveness of each language varies depending on the specific scenarios. Inspired by this, we propose a task and model agnostic approach called MultiPoT, which harnesses strength and diversity from various languages. Experimental results reveal that it significantly outperforms Python Self-Consistency. Furthermore, it achieves comparable or superior performance compared to the best monolingual PoT in almost all tasks across all models. In particular, MultiPoT achieves more than 4.6% improvement on average on ChatGPT (gpt-3.5-turbo-0701).

## introduction program of thoughts (pot) aims to prompt code large language models (code llms) to decompose complex problems into successive executable codes @xcite . through execution by an external interpreter, the final results are accurately obtained, decoupling the computational process from the llms. pot significantly reduces computation errors and improves reasoning performance @xcite . subsequently, benefiting from its flexibility and scalability, it is gradually applied to a broader spectrum of fields like image reasoning @xcite ; sent pot in different forms. as shown in figure 1 , the representation and calculation of dates in r is more concise than that in python, which can reduce the complexity when llms generate pots. from the perspective of models, their multilingual ability is inconsistent. for instance, c++ of deepseek coder outperforms python on the code generation task @xcite . it is natural to wonder whether this phenomenon also occurs on reasoning tasks. therefore, a crucial question is raised with these perspectives: is python truly the optimal language for all tasks and models for pot? relying on python may lead to a local optimum. in figure 1 , python's 'timedelta' does not support 'year', resulting in a miscalculation for the leap year. in contrast, r and javascript yield the correct answer. motivated by this, we conduct comprehensive experiments for multilingual pots. beyond python, we select four pls: three widely used general languages (javascript, java, and c++) and a niche but comprehensive language (r). for a comprehensive comparison, we identify five distinct sub-tasks within reasoning tasks: math applications @xcite @xcite , math @xcite , tabular, date, and spatial @xcite . we select four backbone llms: chatgpt (gpt-3.5-turbo-0701) and three strongest code llms @xcite , code llama @xcite ). under both greedy decoding and self-consistency @xcite settings, we answer that "python is not always the optimal choice, as the best language depends on the specific task and model being used." in addition to the analysis contribution, to leverage the strength of multiple pls, we further introduce a simple yet effective approach, called mul-tipot (multilingual program of thoughts). it is a task and model agnostic approach, which uses llms to synchronously generate pots with various pls and subsequently integrates their results via a voting mechanism. the use of multiple pls also provides greater diversity and reduces the probability of repeating the same errors compared to single-language sampling. experimental results demonstrate that multipot outperforms python self-consistency significantly. furthermore, mul-tipot effectively matches or even surpasses the top-performing languages across nearly all tasks and models, and outperforms on averages. especially on both chatgpt and starcoder, multipot performs the best on four out of five tasks, with only a slight underperformance on the remaining task, and shows an improvement of over 4.6% compared to the best monolingual pot on average. our contributions are summarized below: ## methodology figure 2 provides an overview of multipot and self-consistency to highlight their differences. concretely, multipot consists of two main steps. first, a dedicated prompt is designed for each pl to sufficiently leverage the capability of the model with regard to the pl (section 3.1). second, pots in various pl are respectively generated by prompting the llm with the prompts. the final answer is obtained by executing the pots and integrating their results via a voting mechanism (section 3.2). distinct from self-consistency, which relies on a single pl, multipot integrates various pls to utilize their strength and diversity. ## results in this section, we first discover that python is not the best language for all tasks and all models from the results of greedy decoding. there is no such perfect language. the performance of each pl varies greatly depending on the task and model (section 5.1). after self-consistency, the perfor-mance discrepancy still exists. finally, by integrating multiple languages, multipot significantly outperforms python. furthermore, its performance matches or exceeds the best monolingual pots in almost all scenarios and achieves improvement on task and model averages (section 5.2). ## conclusion regarding the reliance on python in pot, we conducted extensive experiments across various models and tasks using multiple pls. our findings show that python is not always the best choice; the optimal language depends on the specific task and model. building on this insight, we introduce mul-tipot, a simple yet effective multilingual integrated method that leverages the strengths and diversity of different pls. multipot significantly outperforms python and achieves matches or exceeds performance to the best monolingual outcomes in nearly all scenarios. with its high stability, multipot offers a promising avenue for future research.