Mathador- LM : A Dynamic Benchmark for Mathematical Reasoning on Large Language Models (2024)

Abstract: We introduce Mathador-LM, a new benchmark for evaluating the mathematical reasoning on large language models (LLMs), combining ruleset interpretation, planning, and problem-solving. This benchmark is inspired by the Mathador game, where the objective is to reach a target number using basic arithmetic operations on a given set of base numbers, following a simple set of rules. We show that, across leading LLMs, we obtain stable average performance while generating benchmark instances dynamically, following a target difficulty level. Thus, our benchmark alleviates concerns about test-set leakage into training data, an issue that often undermines popular benchmarks. Additionally, we conduct a comprehensive evaluation of both open and closed-source state-of-the-art LLMs on Mathador-LM. Our findings reveal that contemporary models struggle with Mathador-LM, scoring significantly lower than average 3rd graders. This stands in stark contrast to their strong performance on popular mathematical reasoning benchmarks. The implementation of Mathador-LM benchmark is available at https://github.com/IST-DASLab/Mathador-LM.

## introduction the ability of large language models (llms) to approach non-trivial tasks involving both information retrieval and mathematical reasoning has led to significant research interest in evaluating these properties. yet, the popularity of reasoning benchmarks, such as the often-used grade-school math (gsm) @xcite or math @xcite datasets, is leading to performance saturation (see figure 1 ), and can potentially lead to training set contamination. thus, there is a stringent need to develop new strong benchmarks to evaluate llm reasoning. ## the mathador-lm benchmark the informal definition of the mathador-lm game we use is provided in figure 2 , which coincides with the prompt we provide to the llm in the default version of the game. in ## model evaluations evaluation setup. a dataset of mathador-lm problems is generated for each model evaluation by sampling the operand dataset a based on the official rules @xcite and then sampling from possible targets {t|∃@xmath5.t. val(e) = t} based on the desired difficulty distribution. the restrictions for base numbers are closely following the official rules of the human game, so that results we obtain with llms are directly comparable to human results. this means that base numbers are sampled as integers, uniformly at random from the following ranges: n 1 ∈ [1, 4], n 2 ∈ [1, 6], n 3 ∈ [1, 8], n 4 ∈ [1, 12], and n 5 ∈ [1, 20]. the prompt in figure 2 is populated based on a newly generated problem set to get the final prompt. the model's generated answer to the prompt is parsed to get the solution block which is then scored. in addition to the prompt shown in figure 2 we have experimented with everything from concise descriptions to extensive, detailed explanations. in terms of few-shot prompts, we have also tried quite a few approaches to present the model with step-by-step solutions. some prompts were designed and tested following an error analysis that helped identify the most common mistakes made by the models. for instance, we developed specific prompts after observing that the majority of errors involved the use of illegal operands. these prompts explicitly specify the set of permissible operands at each step. unfortunately, despite these additional instructions, we have not seen any noticeable accuracy gains. figure 4 presents evaluations on several popular open and closed models. we observe that small models (≤ 3b) and mistral-7b tend to perform @xmath6.36 points per instance, on average), meaning that they reach a correct solution (@xmath7. surprisingly, well-performing medium models such as qwen2-7b, llama-3-8b, and phi-3-medium perform on par with gpt 3.5 and gpt4, as well as claude-haiku (5 to 7%), at a level corresponding to reaching a correct solution less than 20% of the time. further, we observe a higher tier for 70b models and claude-opus, which reach similar ∼ 12% performance. in appendix a we expand our analysis, and detail the score distribution across models. in appendix b we investigate how allowing multiple attempts to solve each question affects the accuracy of llms. stability. a reliable benchmark must be reproducible, which is why most benchmarks are static. errors analysis. in