Solving for X and Beyond: Can Large Language Models Solve Complex Math Problems with More-Than-Two Unknowns? (2024)

Abstract: Large Language Models have demonstrates remarkable performance in solving math problems, a hallmark of human intelligence.Despite high success rates on current benchmarks, however, these often feature simple problems with only one or two unknowns, which do not sufficiently challenge their reasoning capacities. This paper introduces a novel benchmark, BeyondX, designed to address these limitations by incorporating problems with multiple unknowns. Recognizing the challenges in proposing multi-unknown problems from scratch, we developed BeyondX using an innovative automated pipeline that progressively increases complexity by expanding the number of unknowns in simpler problems. Empirical study on BeyondX reveals that the performance of existing LLMs, even those fine-tuned specifically on math tasks, significantly decreases as the number of unknowns increases - with a performance drop of up to 70% observed in GPT-4. To tackle these challenges, we propose the Formulate-and-Solve strategy, a generalized prompting approach that effectively handles problems with an arbitrary number of unknowns. Our findings reveal that this strategy not only enhances LLM performance on the BeyondX benchmark but also provides deeper insights into the computational limits of LLMs when faced with more complex mathematical challenges.

## introduction mathematical problem-solving is a fundamental aspect of human intelligence, necessitating both language comprehension and reasoning skills. recently, llms pretrained on extensive web-scale datasets, have exhibited exceptional abilities in addressing a variety of complex tasks. consequently, mathematical challenges are frequently employed to benchmark the reasoning abilities of llms. studies have shown that these models demonstrate human-level efficacy in solving math problems, aided by diverse prompting techniques include incontext learning @xcite @xcite and the integration of external computational tools @xcite @xcite . existing math datasets (see while quantifying the complexity of these math problems is multi-dimensional, one common measure is the number of unknowns required to solve the problem. problems with more unknowns involve larger systems of equations, reflecting more complex relationships between the quantities, and thus demanding more sophisticated solving methods. however, creating datasets that include problems with multiple unknowns presents significant challenges, as it is difficult for humans to manually develop a sufficient number of these complex problems from scratch. as a result, existing math datasets are dominated by problems with at most two unknowns @xcite @xcite . this paper tackles the aforementioned challenge systematically, by presenting three key contributions: @xcite the development of a multi-unknown math benchmark, (2) an empirical study assessing the performance of current llms on this new benchmark, and (3) the introduction of a specialized prompting strategy designed to enhance the ability of llms to solve multi-unknown problems. ## automatic solver of algebra problems to investigate whether the observed performance drop is primarily due to inadequate prompting strategies or the limitation of llms. we develop formulate-and-solve, an automated prompting method designed for llms to solve math problems with an arbitrary number of unknowns. we also show that our method performs competitively to state-of-the-art algorithms even for non-algebra problems in appendix a.1. a major challenge in applying the prompting method to multi-unknown problems is the scarcity of hand-crafted demonstrations. traditional examples with a single unknown do not scale well to more complex, multi-unknown scenarios, necessitating automated demo generation. furthermore, while llms can be guided by prompts to solve these systems of equations, they often require external tools due to their limited ability to independently solve and explicitly formulate these problems into a system of equations. to overcome these limitations, we propose formulate-and-solve, a framework that incorpo-rates a set of principles to instruct llms in generating demonstrations automatically. this framework empowers llms to translate problems into equations and subsequently utilize external tools to solve them. the overall pipeline is illustrated in figure 3 and we include the actual prompts used in each step in appendix 17. ## discussion and analysis ablation study. we analyze the significance of each component within in formulate-and-solve through an ablation study. we assess five variations: (1) use only a system of equations as a rationale for reasoning. (2) remove the instruction before demonstrations. (3) remove demonstrations after the instruction. (4) use an llm instead of a symbolic solver to solve a system of equations. (5) formulate-and-solve. we randomly select 60 problems for each unknown from alg514, draw-1k, and beyondx to evaluate each variation. the results are provided in we observe that performance decreases significantly when instruction or demonstration is removed, highlighting its role in guiding the llm. interestingly, instruction has a greater impact than demonstration. replacing the symbolic solver with an llm also leads to a large decrease in accuracy. these findings confirm that all elements in formulate-and-solve contribute significantly to solving multi-unknown problems. error reason percentage (%) (e1) fewer or more equations 17.4% (12/69) (e2) incorrect equation 44.9% (31/69) (e3) incorrect extraction form 37.7% (26/69) ## conclusion we introduce beyondx, the first benchmark for evaluating llms on multi-unknown problems. our analysis reveals a significant performance drop in llms and existing prompting methods when faced with such problems. to address this, we propose formulate-and-solve, a novel prompting method that leverages instruction, automatic demonstrations and a system of equations. experiments demonstrate the effectiveness of formulate-and-solve in tackling multi-unknown problems.