Automatic Generation of Socratic Subquestions for Teaching Math Word Problems (2022)

Abstract: Socratic questioning is an educational method that allows students to discover answers to complex problems by asking them a series of thoughtful questions. Generation of didactically sound questions is challenging, requiring understanding of the reasoning process involved in the problem. We hypothesize that such questioning strategy can not only enhance the human performance, but also assist the math word problem (MWP) solvers.In this work, we explore the ability of large language models (LMs) in generating sequential questions for guiding math word problem-solving. We propose various guided question generation schemes based on input conditioning and reinforcement learning.On both automatic and human quality evaluations, we find that LMs constrained with desirable question properties generate superior questions and improve the overall performance of a math word problem solver. We conduct a preliminary user study to examine the potential value of such question generation models in the education domain. Results suggest that the difficulty level of problems plays an important role in determining whether questioning improves or hinders human performance. We discuss the future of using such questioning strategies in education.

## introduction questioning can be a valuable way of supporting student thinking. it can be conceived as a scaffold @xcite , where a more knowledgeable tutor helps a student in solving problems otherwise too difficult. one approach well-suited for mathematics is funneling @xcite , which uses prompting questions to guide students towards a solution. figure 1 : math word problems can be precedurally solved in multiple reasoning steps. one operationalization of socratic questioning is to map each step in the procedure to a question. asking (machines/humans) the right set of questions in a certain sequence (shown in green) can be an effective way to do so. in order to be effective, the socratic questioning should be focused and goal-driven. figure 1 shows an example of a math word problem where this questioning strategy might be beneficial. we hypothesize that these questions can not only help humans in understanding the problem better and improve their performance but can also assist mwp solvers. even though question generation (qg) models have been studied for factual squad-like questions @xcite , these models fail to generate sequentially-coherent questions @xcite . furthermore, domain-specific questioning is challenging as the qg model needs to understand the reasoning process required to provide fine-grained responses. moreover, the role of a teacher using questioning is to interject questions that focus on the most critical points in an explanation and take the understanding forward @xcite . as seen in bold in the figure 1 , we refer later to these properties of questioning as focused and goaldriven. in this work, we explore the use of large language models @xcite to generate guiding sub-questions for math word problems. in particular, we use reinforcement learn-ing (rl) with rewards from various sources including math question answering (math qa) models and various forms of input conditioning for generating these questions. we train and evaluate our models on the recently released gsm8k mathqa dataset @xcite of multi-step reasoning mwps. we illustrate the benefit of our rl-based generation strategy using both automatic and human evaluation metrics. our evaluation shows that our guided approach makes the generation model ask more logically relevant and structurally correct questions, which follow the appropriate sequencing of questioning at the right granularity level. we further show that our generated questions, when provided as additional context, can aid a math question answering model, thereby providing further empirical justification of the value of questioning for math qa model training. questioning could facilitate reasoning of mwp solvers by making intermediate reasoning steps explicit. finally, we explore the didactic usefulness of our questioning strategy by conducting a preliminary user study and use it to show that the generated sequence of questions may have the potential to improve students' problem-solving. however, we cautiously note that achieving this would require further progress on many fronts in ai and education. in what follows, we begin by discussing related work and introducing our research questions in section 2 and section 3. we propose ways to induce these properties in lms using planning and reinforcement learning in section 4; section 5 empirically demonstrates the effectiveness of inducing questioning strategy in lms and the quality of generated questions evaluated using automatic metrics and by humans. finally, we evaluate the potential of using such questions as an educational tool for helping students solve mwps in section 6. ## related work socratic questioning approaches have evolved within the learning sciences community into the theory of scaffolding @xcite , which broadly refers to assisting students in problem-solving beyond their zone of proximal development @xcite . computerbased scaffolds (e.g., in the form of hints, prompts, feedback) have moderate effects on student learning outcomes @xcite , and our work can be used to automatically generate such scaffolds in the form of questioning prompts. for mathematics, @xcite analyzed interactions in math classrooms and proposed two distinct interaction patterns -funneling, which functions by guiding students using leading/prompting questions to a predetermined solution procedure, and focusing, which functions by drawing student attention to the critical aspects of the problem. we draw inspiration from this strand of work. our overall question generation approach can be conceived to be similar to funneling, with specific sub-questions focusing on the important domain concepts. @xcite , generation of questions for student assessment @xcite , generation of factual questions based on wikipedia articles @xcite or generation of sequential informationseeking questions in dialogue-based scenarios @xcite . other work has also explored similar ideas of improving answerability by question-asking @xcite @xcite and ranking them @xcite . however, factual questions do not usually require much reasoning and mostly boil down to information retrieval from text. in this work, we focus on question generation for reasoning problems. ## research questions we now discuss the usefulness of questions in solving a math word problem and then study the different properties of a good questioning strategy. rq1: does sub-questioning help in understanding a math word problem better? question prompts as a teaching strategy act as instructions that guide the students throughout a problemsolving process @xcite . such questioning, as a valid scaffolding strategy @xcite , is valuable in supporting student thinking and is commonplace in high-quality math instruction @xcite . we explored the sub-questioning strategy with our trained nlp model and found that sub-questioning helps answer the mwps more effectively ( rq2: what are the properties of a good questioning strategy? once we established that subquestioning is helpful, we performed the same subquestioning experiment as rq1 with nlp models but with the permuted ordering of sub-questions, change in granularity of sub-questions or changed content ( based on our results and inspired by prior work @xcite , we hypothesize the most important components of a socratic questioning strategy as: ## methodology we discuss our approach to modeling socratic questioning using large lms. we begin by defining our mwp dataset d as a collection of mwps. each mwp p in the dataset is accompanied by its solution s and the numerical answer a. we do not always assume the existence of problem solutions s and answers a as they can be automatically derived from various mathqa models. each mwp @xmath0, q) consists of the story context c and the question q. the problem solution s consists of n solution steps @xmath1, ..., s n ). we define socratic questioning such that each solution step s i can be mapped to a sub-question q i . we refer to q as a collection of all socratic questions q 1 , ..., q n for a given mwp p in our work. an example mwp is present in figure 2 . our main module is the question generator (qg) module, which is a transformer @xcite based encoder-decoder model. the qg model takes the reference math word problem p and generates the socratic questions q * as close to the true sub-questions q as possible. the learning objective of the qg module is as: next, we propose to inject the two socratic questioning properties in our qg model as follows: ## empirical analysis we now demonstrate the effectiveness of inducing the defined questioning properties in large lms. ## a preliminary user study with learners finally, we designed a preliminary user study to evaluate whether our generated questions, when presented as further problem-solving exercises (as typically used in educational settings) can help learners on the way to solving the overall problem. given our research question, we hypothesized that guidance with questions can increase the overall problem-solving success rate for users in the questions (treatment) group compared to the noquestions control group. our study uses socratic questions as the main pedagogical intervention. we focus on participants who cannot solve a problem on the first attempt to clearly distinguish the impact of automated sub-questioning. the key metric we measure is the success rate, which is defined as the percentage of correctly solved problems. for our study, we built a simple user interface which allowed participants to solve math word problems (see figure 5 and figure 6 in the appendix). the interface contained a calculator which the users could use if needed. the study comprises 5 pre-test problems and 8 problem-solving exercises. these problems were randomly selected from the gsm8k test set. our user study with this interface was then deployed on mechanical turk and participants were hired using the platform and were paid 10-12$ per hour. we selected participants with moderate levels of prior knowledge using the pre-test scores as the selection criteria, and only those scoring in the range of 40-80% were selected for the study. this way, we excluded both low-prior knowledge participants and experts in our study to ensure there was a learning possibility. we randomly split the participants into two groups -no-questions group (@xmath2, and questions group (@xmath3. both groups used the same interface for solving math word problems and had the opportunity to resolve their answers after the first incorrect submission. the only difference was that after incorrectly solving a problem on the first submission, participants in the questions group saw sub-questions, while those in the noquestions group were only prompted to try again. the sub-questions were generated using the bestperforming model with planning and rewards. the results of the user study are shown in the first attempt success rate is 58.4% for the control group and 66.0% for the treatment group, which might be the result of a slightly skewed prior knowledge distribution of 0.68 and 0.65 for treatment and control groups respectively. even though participants in the treatment group (@xmath4.9, @xmath5.1) spend significantly more time (@xmath6.01) solving problems during the second attempt relative to the control group (@xmath7.5, @xmath8.4), we did not find any statistically significant difference between the groups in the second submission success rate (@xmath9.659, bf 01 = 2.755, cohen's @xmath10.157), indicating weak odds favouring the null hypothesis and rather a small effect size. as our study was unable to establish overall performance improvements, we further analysed the second submission success rate per problem (see figure 4 ), and correlated it with the difficulty of the question. this analysis indicated that subquestioning seems to improve the success of simpler problems and degrade the accuracy for relatively more complex problems. prior work has suggested that the effectiveness of question prompts varies according to an individual's prior knowledge @xcite , and with insufficient prior knowledge, performance for complex problems may suffer. a posthoc inspection of the generated sub-questions for more complex problems shows that they also scored lower in the human quality evaluation. thus, we hypothesize that for more complex questions, the generated sub-questions are not good enough, and so they may make the task more challenging for participants. while we were not able to establish any direct benefits of automatic socratic questioning in a real learning scenario, we leave a more complete user study for future work. deployment of socratic questioning systems in real educational scenarios would require a better assessment of question generation quality as well as a better understanding of learners. we believe this is an interesting avenue for future research and encourage future work to attempt to address these issues. ## conclusion we study the importance of sub-questioning for learning a mathematical concept and explore how lms may generate these sub-questions. we demonstrate the usefulness of socratic questioning strategies and propose ways to induce these properties group 1st success 2nd success m sd m sd no-questions 58.4 23.0 35.8 32.5 questions 66.0 21.1 31.0 27.9