The Impact of Language on Arithmetic Proficiency: A Multilingual Investigation with Cross-Agent Checking Computation (2024)

Abstract: This paper critically examines the arithmetic capabilities of Large Language Models (LLMs), uncovering significant limitations in their performance. Our research reveals a notable decline in accuracy for complex calculations involving large numbers, with addition and subtraction tasks showing varying degrees of proficiency. Additionally, we challenge the notion that arithmetic is language-independent, finding up to a 10% difference in performance across twenty languages. The study also compares self-verification methods with cross-agent collaborations, showing that a single model often outperforms collaborative approaches in basic arithmetic tasks. These findings suggest a need to reassess the effectiveness of LLMs in tasks requiring numerical accuracy and precision.

## introduction large language models (llms) have garnered significant attention over the past year. several studies have re-evaluated various tasks to assess the capabilities of general-purpose llms @xcite @xcite . a topic of particular interest is mathematical and numerical reasoning @xcite @xcite . figure 1 illustrates an instance where llms generate step-by-step operational expressions while solving a math word problem, named chain-of-thought prompting @xcite . while previous research indicates improved performance by llms in solving math word problems, there is a scarcity of discussion on whether llms truly comprehend the operations they generate. this paper delves into this issue through extensive experimentation and reveals a notable limitation of llms in arithmetic. unlike other semantic tasks such as humor estimation @xcite or emotion prediction @xcite , where different labels may emerge due to language and cultural variations, arithmetic is typically considered languagefree and culture-free, as the same expression should yield a consistent answer regardless of these factors. in this study, we investigate twenty languages and demonstrate that this assumption does not hold in practice. our findings reveal that the overall performance can vary by up to 10% in accuracy simply by altering the language when utilizing llms for arithmetic tasks. conversely, addition and subtraction are fundamental yet critical tasks in arithmetic. as depicted in figure 1 , it is commonly assumed in prior research that llms are capable of solving such elementary calculations. contrary to this belief, our study reveals a significant decline in performance for calculations involving more than five digits in addition and more than four digits in subtraction. furthermore, we observe a 20% discrepancy in accuracy between addition and subtraction tasks. these findings underscore the need to reassess the extent to which llms genuinely comprehend the principles of basic arithmetic. finally, checking computation is a crucial step in human arithmetic processing. we initially investigate different prompts to examine the extent to which performance alters with different approaches. besides self-verification by the same model, our study also delves into cross-agent checking. contrary to prior research, which indicates that multiagent communication can enhance performance in contexts such as software development @xcite and generated-text evaluation @xcite , our findings suggest that a single model surpasses cross-agent collaboration in simple arithmetic tasks. this challenges the prevailing notion that collaborative approaches always yield superior results in nlp tasks. ## related work and preliminary arithmetic computation forms the cornerstone of mathematical capability. earlier studies @xcite classify arithmetic tasks into two groups: learnable and unlearnable, and @xcite demonstrated that llms fail at multi-digit multiplication. tasks categorized as learnable include copying, splitting, comparison, ordering, addition, subtraction, and n-digit versus 1-digit multiplication/division. it is anticipated that model performance would be robust when trained specifically on these learnable tasks. supporting this, @xcite provides evidence for the comparison task, where models achieve a 99% accuracy rate after straightforward fine-tuning with artificially generated datasets. however, this falls outside the purview of our paper, as our focus is on the capabilities of general-purpose llms trained with commonly available resources. in this study, we specifically investigate addition and subtraction within a multilingual context, a subject seldom addressed in previous research. on the other hand, checking computation is another seldom-explored area of prior studies. drawing inspiration from @xcite , which demonstrated that llms trained on the premise "a is b" struggle to comprehend "b is a" (reversal curse), our research investigates the validity of these findings in arithmetic tasks. advancing this inquiry, we observe that communicative agents exhibit superior performance compared to the use of a single llm in various tasks, as noted in many recent studies @xcite @xcite . building upon this trend, our study delves into the realm of cross-agent checking computation. our study demonstrates that llms currently lack the capability for self-correction in basic arithmetic scenarios, even through llm interaction. ## conclusion this study aimed to demonstrate negative results and uncover shortcomings of llms in basic arithmetic tasks. our findings reveal that (1) numeracy is intertwined with linguistic elements, (2) llms exhibit suboptimal performance in computation verification tasks, and (3) the concept of carrying/borrowing is not effectively mastered by llms, especially borrowing. these results provide a foundation for future research to (1) investigate the robustness of numeracy in language models, (2) enhance computational verification capabilities in number-aware fact-checking tasks, and (3) improve the fundamental arithmetic proficiency of llms. additionally, our observation that language would enhance numeracy is another promising topic that future studies can pay attention to. for example, researchers could investigate how incorporating language-based strategies into mathematics problem-solving improves models' understanding and retention of numerical concepts.