Improving Mathematics Tutoring With A Code Scratchpad (2023)

Abstract: Large language models can solve reasoning tasks (like math problems) more effectively when they are allowed to generate rationales. However, a good tutoring system should not just generate solutions, but should also generate explanations and should be able to correct and guide students. We show that providing a code scratchpad improves performance on each tutoring step with a gradeschool mathematics dataset. On these tutoring tasks, GPT-3 models provided with a code scratchpad significantly outperform those given only a language scratchpad (77.7% vs 48.7% cumulative accuracy).

## introduction intelligent tutoring systems (its) are known to be effective aids to learning, but are currently difficult and time consuming to create. such systems can aid learning significantly despite limitations, improving student performance with a median improvement of 0.66 standard deviations @xcite . however, many notable its (for example @xcite ) have been limited due to the time-intensive and costly processes required to create them. previous work on its has typically focused on rule-based methods. to the degree that large language models (llms) are used, it has been to generate additional rules for such systems. recently, advances in natural language processing have pointed at the possibility of using llms as tutoring systems, most notably 1) the success of large language models in math world problem solving due to rationale generation @xcite @xcite and 2) the improved alignment of dialogue agents such as chatgpt and sparrow @xcite . we conduct a feasibility study on the application of llms to tutoring in the context of mathematics at an elementary school level by investigating their performance on the tasks required by an its (see figure 1 ). figure 1 : we evaluate the performance of two gpt-3 models on the sub-tasks present in an intelligent tutoring system, providing one with a text-only scratchpad and the other with a code scratchpad. our contributions are the following: • we evaluate llms on the tasks present in an its by proving a mapping between the subtasks in an its and tasks which can be done by an llm. using this, we show that gpt-3 with a text-only scratchpad has a significant error rate when acting as a domain model and tutoring model. ## related work & background early uses of nlp in its involved the use of knowledge-based and rule-based systems @xcite . such systems have shown to be pedagogically effective @xcite , and as such they continue to constitute the majority of its today. teaching and interacting with the student in an its takes place through some fixed set of interactions, often mediated by extracting keywords from user utterances or as goal-oriented dialogue systems. this tends to be the case in both knowledge-based its @xcite , and in rule-based systems @xcite . for openended domains, named entity recognition (ner) has been used to determine whether a student's open-ended response meets a set of constraints @xcite . techniques from nlp have also been used more selectively to implement features in these systems, such as machine translation for language learning @xcite and automatic speech recognition (asr) for audiobased tutors @xcite . however, newer techniques such as llms have not found extensive use in implementing tutoring systems. this is despite the success of generative models such as @xcite and palm @xcite across a wide variety of tasks, the improvement in dialogue systems stemming from alignment as seen in models like chatgpt and sparrow @xcite , and the success of llms (especially those that generate code) in the related domain of math word problem solving @xcite . much of the work on llms in education has focused on question generation as opposed to intelligent tutoring systems, for example @xcite for flashcard generation or @xcite for programming exercises. this may be the result of the difficulty in evaluating the quality of generations from llms, especially explanations for the answers that they give, as noted in @xcite . in this paper, we evaluate the ability of llms to serve as tutors, focusing on the evaluation of generated explanations and corrections. intelligent tutoring systems are typically composed of four components @xcite : the domain model, student model, tutoring model, and user interface model. the domain model consists of the actions and correct steps required to solve a problem. for example, in an its for mathematics the domain model might consist of all the relevant operations and the correction method of solving problems. the student model consists of the actions taken by the student (for example, the scratchpad the student is using to do their work). when the student deviates from the domain model, the tutoring model provides feedback (for example, telling a student what step they should take next or what a student did wrong in their scratchpad). finally, the user interface model facilitates interaction between the user and the tutoring model (this might be the system which parses the scratchpad and then parlays feedback to the student). we can instantiate a tutor using an llm by creating each of the following parts. the user interface model is simply natural language. the domain model consists of problems with correct solutions (generated by the model), the student model consists of the language produced by the student, and the tutoring model consists of comparing domain and student models in text and producing feedback. we illustrate each of the parts of an its and how they can be performed by an llm in figure 4 . ## experiments our first experiment evaluates the difference in performance between text and code scratchpads in math problem solving. we evaluate, as is typical for math word problem solving, by measuring the percentage of numerically correct answers produced by the model. this is a necessary, but not sufficient, part of generating the domain model. the llm should produce not only a correct answer, but should also provide a correct explanation to produce that answer. therefore, our second experiment evaluates whether the model provides an acceptable explanation for its answer. because we generate answers with gpt-3 by using cot prompting, an explanation is automatically produced. for the code scratchpad, we generate an explanation by asking the model to convert the code used to produce an answer into plain english. these two experiments evaluate the ability of the llms to serve as a domain model. our third experiment evaluates the ability of the llms to serve as tutoring models. we start with the correct answers and explanations provided by the model. for each question answered correctly, we prompt the models using poorly formed prompts in order to generate plausible incorrect answers (i.e. using the model to simulate the output of a student). then, we provide the model with the incorrect answer and the correct answer, and prompt it to explain why the incorrect answer is wrong and to accordingly provide feedback to the student. the first experiment is evaluated automatically, while the second and third experiments are evaluated by human annotators. ## evaluation we tasked 208 annotators to evaluate the quality of explanations and corrections. each annotator was shown 20 examples of explanations and later shown 20 examples of corrections. a total of 213 explanations and 190 corrections were evaluated in this way. we modify the question evaluation procedure in @xcite for evaluating explanations and asked the following yes/no questions: if an annotator answered yes to all of the above questions, the explanation/correction was considered "acceptable"; otherwise, it was considered "unacceptable". using fleiss' κ, we observe moderate inter-annotator agreement (κ = 0.21). in in figure 2 we report the detailed results of our evaluation for explanations. we can see that language and code scratchpads achieve similar performance in generating explanations. this is notable because of the difference in how the two models can create explanations. text generation, by virtue of generating a chain of thought, comes with an explanation. code generation requires an additional step of transforming code into text, which introduces an opportunity for more errors. this is reflected in the fact that explanations generated in text are more likely to be correct. however, code generation is much more likely to result in a complete explanation. this makes sense, as the model must explicitly list steps in code in order for the code to compile, while text is more prone to logical leaps or implicit steps. in figure 3 we report the detailed results of our evaluation for corrections. in contrast with explanation generation, when generating corrections, code scratchpads encounter fewer errors of all kinds than text ones. ## conclusion & future work in this work we show that large language models can perform the tasks associated with traditional intelligent tutoring systems (its). we show that models which use text scratchpads suffer from sbustantial errors in solving and correcting mathematical questions, and that these errors can be ameliorated through the use of code scratchpads. nonetheless, code generation (while accurate enough to potentially useful as tool for authoring its) still suffers from significant errors. future work should seek to further explore the applicability of llms to tutoring. this includes developing both new evaluation methods and new methods of reducing errors.