M ath D ial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems (2023)

Abstract: While automatic dialogue tutors hold great potential in making education personalized and more accessible, research on such systems has been hampered by a lack of sufficiently large and high-quality datasets. Collecting such datasets remains challenging, as recording tutoring sessions raises privacy concerns and crowdsourcing leads to insufficient data quality. To address this, we propose a framework to generate such dialogues by pairing human teachers with a Large Language Model (LLM) prompted to represent common student errors. We describe how we use this framework to collect MathDial, a dataset of 3k one-to-one teacher-student tutoring dialogues grounded in multi-step math reasoning problems. While models like GPT-3 are good problem solvers, they fail at tutoring because they generate factually incorrect feedback or are prone to revealing solutions to students too early. To overcome this, we let teachers provide learning opportunities to students by guiding them using various scaffolding questions according to a taxonomy of teacher moves. We demonstrate MathDial and its extensive annotations can be used to finetune models to be more effective tutors (and not just solvers). We confirm this by automatic and human evaluation, notably in an interactive setting that measures the trade-off between student solving success and telling solutions. The dataset is released publicly.

## introduction dialogue tutoring systems have demonstrated significant potential in augmenting learning outcomes across various domains @xcite teacher: hi winnie, could you please walk me through your solution? student: sure! i first calculated the number of letters written in a week, which is 3 pages x 2 @xmath0. then i multiplied that number by 52 weeks to get the number of pages written in a year, which is 312 pages per year. ## mathdial collection pipeline this section introduces a framework for collecting high-quality tutoring conversations, highlighted in figure 2 . the core idea behind it is to connect category intent example seek strategy so what should you do next? guiding student focus can you calculate . . . ? focus recall relevant information can you reread the question and tell me what is . . . ? asking for explanation why do you think you need to add these numbers? seeking self correction are you sure you need to add here? perturbing the question how would things change if they had . . . items instead? probing seeking world knowledge how do you calculate the perimeter of a square? revealing strategy you need to add . . . to . . . to get your answer. telling revealing answer no, he had . . . items. greeting/fairwell hi . . . , how are you doing with the word problem? good job! is there anything else i can help with? generic general inquiry can you go walk me through your solution? an expert annotator, who roleplays a teacher, with an llm that simulates the student. 1 we use this methodology to collect dialogues based on gsm8k @xcite , a diverse collection of grade school multi-step math word problems (mwps). first, we estimate student confusion for a given mwp by using temperature sampling to obtain diverse solutions from an llm. we then select the most frequent incorrect solution. therefore, each tutoring dialogue deals with the solution of exactly one mwp and one confusion. as a next step, we pair a human teacher with the llm to create a dialogue that should resolve the confusion. we ground the llm in one of six student profiles. these student profiles consist of common misconceptions of students learning algebra, such as struggling to recognize the problem type, and are taken from @xcite . a detailed description of these profiles is found in section c. the teacher has access to the mwp and its correct step-by-step solution, as well as the initial student confusion (cf. figure 7 ). then, the teacher is tasked to guide the student to solve the problem by employing a sequence of scaffolding moves, which we refer to as a teaching strategy. the teachers themselves can use their expertise to determine the strategy but are required to select the current move before writing a response, as we have found this to lead to more diverse pedagogical patterns. we describe these moves in section 3.4. the dialogue ends when the teacher marks the problem as solved or a certain time limit is reached. in addition to the collected dialogues, we obtain metadata that future work can explore for building more effective tutor models. in particular, for each dialogue mathdial contains the mwp, step-by-step solution, the exact step that led to student confusion, and annotations indicating if it was resolved over the course of the dialogue. stepby-step and student solutions are also provided as equations. ## mathdial analysis we quantitatively evaluate the collected tutoring dialogues to assess their quality. for this, we outline descriptive statistics in ## modeling tutors with mathdial we focus our initial studies on mathdial on the task of tutor response generation. tutor response generation aims to model the teacher in a dialogue by generating follow-up turns to guide the student towards learning and solving the problem. in the following subsections, we compare different finetuned and prompted language models on the task and evaluate how much detailed information that can be given to the model, such as step-by-step solutions of the mwp, influence performance. ## conclusion we introduce a new framework for semi-synthetic dialogue dataset collection. we use it to collect a pedagogically rich dataset for tutoring math word problems that follow equitable tutoring practices and learning sciences research on scaffolding student understanding, called mathdial . our dataset consists of ca. 3k tutoring conversations grounded in math word problems from gsm8k. model success @5 telling @5 success @10 telling @10 problems with 2 step solutions flan-t5 780m 65% 0% 68% 2% chatgpt 57% 14% 77% 20% problems with 3 step solutions flan-t5 780m 18% 0% 30% 7% chatgpt 17% 19% 43% 38% problems with 4 step solutions flan-t5 780m 27% 1% 34% 2% chatgpt 29% 20% 52% 32% we benchmark open-source models on the task of tutor response generation and show that smaller models finetuned on our mathdial can significantly surpass the performance of much larger prompted llms. moreover, in our proposed interactive tutoring simulation, the finetuned model achieves similar student-solving success as prompted llm while keeping the direct telling rate lower. nevertheless, models still require better reasoning over student solutions and better generalization to unseen problems. our dataset fills a crucial gap towards studying effective dialogue tutors at scale by providing a significantly larger amount of dialogues than other available corpora in one-on-one tutoring and provides a tough testbed towards better tutoring models. we hope that it can spark more research in this meaningful but understudied area of nlp.