Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learning (2023)

Abstract: Due to the remarkable language understanding and generation abilities of large language models (LLMs), their use in educational applications has been explored. However, little work has been done on investigating the pedagogical ability of LLMs in helping students to learn mathematics. In this position paper, we discuss the challenges associated with employing LLMs to enhance students’ mathematical problem-solving skills by providing adaptive feedback. Apart from generating the wrong reasoning processes, LLMs can misinterpret the meaning of the question, and also exhibit difficulty in understanding the given questions’ rationales when attempting to correct students’ answers. Three research questions are formulated.

## introduction after the pandemic, e-learning has become part of mainstream education @xcite . however, for students, online learning is not without its problems @xcite . apart from the difficulty in maintaining focus in online classes, the lack of real-time communication and timely feedback are also serious problems. in particular, in online assessments, students may fail to understand their mistakes, even after reviewing the provided answers; such failure to immediately clarify points of confusion yields poor learning outcomes. synchronous communication between teachers and students in an e-learning setting is necessary, but teachers find that promptly responding to students' questions is a significant challenge. as large language models (llms) offer a wide range of applications, several studies @xcite @xcite address the use of llms in education. in this work, we seek to investigate the integration of llms into education. since each subject has its own issues that must be addressed, we explore the scenario of llm use in mathematics education and the challenges thereof. many studies address math word problem solving @xcite @xcite , but the human utility of the mathematics reasoning processes and the natural language explanations generated by models within educational settings is rarely discussed. given the extraordinary capability of llms to generate free-text rationalization, we investigate their mathematical problem-solving competence and assess whether the generated step-by-step explanations constitute a useful educational resource. in particular, we analyze the ability of llms to provide adaptive feedback by identifying and explaining errors within a student's math problem-solving process. we address these issues using the mathqa dataset @xcite , which consists of gre-level questions that require advanced mathematical knowledge. the questions cover arithmetic, algebra, geometry, and data analysis. in appendix a we explain in detail our reasoning behind the dataset selection. for all experiments, we use gpt-3.5 @xcite . we raise and discuss a total of three research questions. in this pilot study, we conduct both quantitative and qualitative evaluations to answer the research questions. the contributions of this work are threefold. ## utility of llms in mathematics learning to explore the utility of llms in mathematics learning, we raise the first research question (rq1): how can llms be utilized to assist students in learning mathematical problem-solving skills? since exams are a common way for teachers to evaluate student learning progress, in this study, we focus on leveraging llms to equip students with the knowledge skills needed to solve math word problems. in terms of question types, in addition to true/false and multiple-choice questions, shortanswer questions are also included in math exams. students answer the questions using problemsolving processes. automated assessment @xcite of the student's problem-solving process remains to be investigated. in education, feedback is also crucial @xcite . simply scoring the student answer is often insufficient, as scores do not reflect the reasons for incorrect answers. to learn from their mistakes, students need the corresponding explanation. figure 1 illustrates the scenario of an llm applied in mathematics education. after a student responds to a question, the model determines whether the student's answer is correct or not. if the answer is correct, the system informs the student of the test objective for that question, enhancing their understanding of the examined skills. if the answer is incorrect, the system provides adaptive feedback @xcite by indicating the location of the error and offering an explanation, assisting the student to clarify any misunderstanding. given that there are multiple methods for solving a math word problem, a model must be able to understand and correct a student's thought process. if the overall strategy is sound apart from minor concept errors or obstacles, the model should guide students to the next step following their chosen ap- ## llm ability to solve math problems many textbook math exercises provide the calculation process but lack a natural language explanation. accompanying each step with an explanation would greatly enhance the student's understanding of each equation. this leads to the second research question (rq2): what is the problem-solving capability of llms in mathematical problems and their ability to explain the computational process? automatic evaluation: the dataset used in this work is a modified version of mathqa in which unsolvable questions were removed by @xcite . we utilized openai's api, in particular the "gpt-3.5-turbo-0301" model. the temperature was set to 0. with incorrect reasoning, or give an incorrect answer despite correct reasoning @xcite . however, detailed analyses and statistical evaluations of model errors have been less extensively studied. to further analyze whether llms are able to reason through complex mathematical problems, we invited an expert who majored in mathematics to evaluate the answers generated by the gpt-3.5 model. a total of 120 questions-20 from each of the six question typeswere selected by the expert. each selected question involved a process of reasoning and calculation of more than four steps. the error types are shown in ## pedagogical ability of llms to rectify students' answers the most important thing when helping students to learn mathematical problem-solving is providing immediate adaptive feedback. in this section, we measure the pedagogical ability of llms in mathematics. pedagogical ability refers to the ability to understand and help the student @xcite . this leads to the third research question (rq3): are llms able to identify errors in students' answers and provide corresponding explanations? teacher-student framework for quantitative evaluation: due to the difficulty in obtaining real-world student answers, we simulate a scenario in which students answer questions and teachers correct the students' responses. based on the experimental results from ## conclusion in this work we propose a research agenda for leveraging llms in mathematics learning. first, we explore the use of llms to assist students in learning math word problem-solving skills. then, we analyze the mathematical reasoning ability of llms. finally, we investigate the pedagogical ability of llms in terms of rectifying model-generated or human answers and offering adaptive feedback. we conduct experiments with the gpt-3.5 model, and conclude that there remains room for improve-ment in the llm's performance in solving complex mathematical problems. in addition, although it generates comprehensive explanations, the llm is limited in accurately identifying model's and human's errors due to its poor ability to interpret mathematical equations. in the future, we plan to develop an advanced method by which to improve the utility of llms in mathematics education.