Interpretable Math Word Problem Solution Generation via Step-by-step Planning (2023)

Abstract: Solutions to math word problems (MWPs) with step-by-step explanations are valuable, especially in education, to help students better comprehend problem-solving strategies. Most existing approaches only focus on obtaining the final correct answer. A few recent approaches leverage intermediate solution steps to improve final answer correctness but often cannot generate coherent steps with a clear solution strategy. Contrary to existing work, we focus on improving the correctness and coherence of the intermediate solutions steps. We propose a step-by-step planning approach for intermediate solution generation, which strategically plans the generation of the next solution step based on the MWP and the previous solution steps. Our approach first plans the next step by predicting the necessary math operation needed to proceed, given history steps, then generates the next step, token-by-token, by prompting a language model with the predicted math operation. Experiments on the GSM8K dataset demonstrate that our approach improves the accuracy and interpretability of the solution on both automatic metrics and human evaluation.

## introduction arithmetic math word problems (mwps) consist of natural language statements describing real-world scenarios that involve numerical quantities, followed by a question asking for an unknown value. solving mwps require parsing the textual statements and carrying out the corresponding calculations @xcite . mwps are an important educational tool that helps assess and improve student knowledge in basic mathematical concepts and skills @xcite . they also represent a long-standing interest in artificial intelligence (ai) research since correctly solving them serves as a key benchmark task for testing and improving the mathematical reasoning skills of ai models @xcite @xcite . there is a large body of literature that focuses on automatically solving mwp. earlier works took a modular approach that first analyzes unconstrained natural language and then maps intricate text patterns onto mathematical vocabulary @xcite . as a result, this approach relies heavily on hand-crafted rules to fill the gap between natural language and symbolic mathematical vocabulary @xcite . recent works leverage advances in natural language processing and take a neural network-based, end-to-end approach, where a neural network encodes a numerical representation of the mwp (and the underlying equation), from which a decoder generates the final answer @xcite @xcite @xcite @xcite . unfortunately, the vast majority of these works focus on generating and predicting a single final answer, since answer correctness is often the only evaluation metric. therefore, these works do not provide any insights or explanations into how the models arrive at the answer. as a result, it is often difficult, if not entirely impossible, to explain the model's behavior, especially when it produces a wrong answer. the lack of interpretability of these methods makes it challenging to analyze them and unsafe to use them in real-world applications. this interpretability issue has attracted increasing interest in mwp solving research. recent works have shifted to designing models that not only generate the final answer for an mwp, but also the intermediate steps. the ability to generate intermediate steps not only enables researchers to investigate model behavior but also new applications. for example, in personalized education and intelligent tutoring systems, these models have the potential to generate detailed, personalized solution steps as feedback to improve stu-dent understanding of the mathematical concepts and resolve misconceptions @xcite @xcite . the recent gsm8k @xcite dataset contains mwps that come with 2 to 8 intermediate steps described in natural language, which provides us a good resource to study step-by-step solution generation. many works apply (large) language models (lms) on this dataset and achieve high accuracy in final answer generation, without studying the quality of intermediate steps @xcite @xcite @xcite @xcite . these works use verifiers, self-consistency decoding strategy (majority votes), chain-of-thought prompting, or calculators; see section 4 for a detailed discussion. however, existing lms are still prone to generating incorrect intermediate steps despite yielding the correct final answer. the models are not competent at numerical reasoning, possibly because they generate intermediate steps word by word (or token by token) and cannot look far ahead. as a result, they only use shallow heuristics @xcite in word occurrence and lack multi-step mathematical reasoning capabilities, which solving an mwp requires. a recent study that experiments on gpt-4 also points out that the architecture of next-word prediction precludes any "inner dialog" and cannot really plan ahead @xcite . ## methodology we now define our mwp solution generation task and detail the specifics of our approach. our task is that given a question q, we need to generate a stepby-step solution @xmath0, s 2 , . . ., with each step consisting of a combination of textual and mathematical tokens, to reach the final answer. we formulate the problem as a step-wise controllable generation task using prompts-based lm fine-tuning. figure 1 shows an overview of our approach 1 in-cluding its two main components: first, we utilize the mwp and the solution history to plan and predict next mathematical operation to apply in the next step. second, we use the predicted operation prompt with instruction prompt to guide the next step generation process. our key technical challenges are (i) how to learn a solution planning strategy to transition from step to step and (ii) once we have the next operation, how to apply and design prompts to guide the generative lm to generate the next step to follow the plan. ## experiments we now detail a series of experiments that we conducted to validate the effectiveness of our proposed planning-lm approach on step-by-step mwp solution generation. since our focus is on mwp solution generation with explanations, gsm8k @xcite is a good fit for our purpose. this dataset contains 8.5k high-quality and linguistically diverse mwps, where each mwp has 2-8 solution steps. see section c for details on data preprocessing. ## conclusions in this paper, we addressed the new problem of performing fine-grained, step-by-step controllable solution generation for math word problems. we proposed an approach combining planning and language models to generate interpretable solution steps. our approach leverages pre-trained language models in two ways: at each step, plan the mathematical operation to be applied, followed by using these plans as prompts to control the tokenby-token generation of each step. we demonstrated that with minimal additional parameters introduced, our approach significantly improves math word problem-solving performance over simply fine-tuning language models. we also showed that due to the interpretability and high controllability of operation prompts, we can use our approach to generate solutions with alternative strategies by giving it different solution plans. future work can further explore generating an entire solution path by predicting math operators for each step and revising the plan after each step is generated. we can also explore the application of our approach in real-world educational settings, e.g., for openended answer scoring @xcite .