GSM -Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLM s as Mathematical Problem Solvers (2024)

Abstract: Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. However, there are increasing debates regarding whether these models truly understand and apply mathematical knowledge or merely rely on shortcuts for mathematical reasoning. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs’ math reasoning capability by testing a wide range of question variations. We introduce the adversarial grade school math (GSM-Plus) dataset, an extension of GSM8K augmented with various mathematical perturbations. Our experiments on 25 LLMs and 4 prompting techniques show that while LLMs exhibit different levels of math reasoning abilities, their performances are far from robust. In particular, even for problems that have been solved in GSM8K, LLMs can make mistakes when new statements are added or the question targets are altered. We also explore whether more robust performance can be achieved by composing existing prompting methods, in which we try an iterative method that generates and verifies each intermediate thought based on its reasoning goal and calculation result.

## introduction mathematical reasoning stands as a crucial testament to the development of artificial intelligence @xcite . it requires rigorous problem understanding, strategy formulating, and calculation performing abilities @xcite . large language models (llms) @xcite have demonstrated strong performance on various mathematical benchmarks including grade school * work done during an internship at tencent ai lab. ## related work numerous datasets have been curated to assess the mathematical reasoning abilities of ai systems. early math datasets @xcite focused on basic math problems with equation-based solutions. subsequently, more difficult datasets have been introduced, spanning grade-school level @xcite , high-school level @xcite , and college-level datasets @xcite . amid this progress, there has been a surge in the development of llms towards solving those math benchmarks @xcite . despite the substantial difficulties posed by advanced-level math for llms, recent llms has shown huge potential for solving grade school math @xcite . supervised fine-tuning (sft) is a line of work to effectively adapt language models to mathematics domains @xcite @xcite seed question: janet's ducks lay 16 eggs per day. she eats three for breakfast every morning and bakes muffins for her friends every day with four. she sells the remainder at the farmers' market daily for $2 per fresh duck egg. how much in dollars does she make every day at the farmers' market? solution: janet sells 16 -3 -4 = 9 duck eggs a day. she makes 9 * 2 = 18 every day at the farmer's market. answer: 18 ## the gsm-plus dataset to comprehensively evaluate the robustness of llms in utilizing math-related skills, we build an adversarial dataset gsm-plus using the gsm8k dataset as a foundation. inspired by polya's principles, we design eight types of perturbations from five different perspectives to test the robustness of llms in math reasoning, as depicted in ## experimental setup we compare the accuracy of representative llms on the gsm8k and gsm-plus datasets, and use the prd and asp metrics to evaluate the robustness in diverse math reasoning. we consider closed-source foundation models, i.e., gpt-4 (openai, 2023) and gpt-3.5-turbo (openai, 2022), open-source foundation models, i.e., @xcite , llama-2 @xcite , as well as open-source sft models specifically designed for math reasoning, i.e., metamath @xcite , abel @xcite , tora @xcite , mammoth @xcite , and sego @xcite . the decoding temperature is set to 0 for deterministic predictions. open-source base models are inferred using 8 demonstrations. our results on the gsm8k of the listed llms match their previously reported accuracy. we also establish a human performance baseline by engaging qualified human annotators who have successfully passed a qualification exam. all annotators possess at least a bachelor's degree. further details can be found in appendix c.1. ## conclusions and discussions in this work, we introduce gsm-plus, a benchmark designed to systematically analyze the robustness of llms in solving math word problems. we examine a variety of perturbation types to evaluate the performance stability of llms in understanding and utilizing math-related knowledge, our evaluation of 25 prominent models found that compared to their performance on the standard benchmark, significant declines are observed in performance when perturbations are introduced in math questions that were successfully solved. this disparity set a clear direction for future research: (1) the systematic evaluation of models across diverse math-related skills; (2) the development of models capable of consistently and flexibly performing math reasoning while remaining resilient to minor variations. although compositional prompting can enhance the performance of llms, its impact is limited in terms of both performance and robustness. it is crucial to acknowledge that most llms, particularly for open-source models, still fall significantly short of human performance, particularly in math domains where even small errors can lead to task failure. overall, gsm-plus aims to facilitate detailed evaluation and understanding of llms on math reasoning. dataset and evaluation suits will be released.