Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes (2024)

Abstract: Scaling high-quality tutoring remains a major challenge in education. Due to growing demand, many platforms employ novice tutors who, unlike experienced educators, struggle to address student mistakes and thus fail to seize prime learning opportunities. Our work explores the potential of large language models (LLMs) to close the novice-expert knowledge gap in remediating math mistakes. We contribute Bridge, a method that uses cognitive task analysis to translate an expert’s latent thought process into a decision-making model for remediation. This involves an expert identifying (A) the student’s error, (B) a remediation strategy, and (C) their intention before generating a response. We construct a dataset of 700 real tutoring conversations, annotated by experts with their decisions. We evaluate state-of-the-art LLMs on our dataset and find that the expert’s decision-making model is critical for LLMs to close the gap: responses from GPT4 with expert decisions (e.g., “simplify the problem”) are +76% more preferred than without. Additionally, context-sensitive decisions are critical to closing pedagogical gaps: random decisions decrease GPT4’s response quality by -97% than expert decisions. Our work shows the potential of embedding expert thought processes in LLM generations to enhance their capability to bridge novice-expert knowledge gaps. Our dataset and code can be found at: https://github.com/rosewang2008/bridge.

## introduction human tutoring plays a critical role in accelerating student learning, and is one of the primary ways to combat pandemic-related learning losses @xcite @xcite @xcite . to accommodate the growing demand for tutoring, many tutoring providers engage novice tutors. while novice tutors may exercise the domain knowledge, they often lack the specialized training of professional educators in interacting with students. however, research suggests that novices with proper training can be effective tutors @xcite . responding to student mistakes in real-time is a critical area where novice tutors tend to struggle. mistakes are prime learning opportunities to address misconceptions @xcite , but effective responses involve pedagogical expertise in engaging with student's thinking and building positive rapport @xcite @xcite . novices typically learn from experts to understand the expert's thought process however hiring experienced educators to provide timely feedback is resourceintensive @xcite . one potential solution is the use of automated tutors @xcite . with recent advances in large language models (llms), this approach has gained even more interest @xcite . however their ability to remediate is yet to be evaluated. prior work suggests several shortcomings with llms, including lacking reliable subject and pedagogical knowledge @xcite @xcite , that can be mitigated using explicitly thought processes such as through chain-of-thought prompting @xcite . to address these challenges, our work makes several key contributions. first, we build bridge, a method that leverages cognitive task analysis to elicit the latent thought processes of experts. we apply bridge to remediation where we collaborate extensively with experienced math educators to translate their thought process into a decisionmaking model. bridge breaks down the experts' thought process: illustrated in figure 1 , @xcite a is to infer the student's error (e.g., the student guessed); step b is to determine the remediation strategy (e.g., provide a solution approach); and step c is ⃝ how do we model the expert's thought process? our work builds bridge which leverage cognitive task analysis to translate the latent thought process of experts into a decision-making model. 3 ⃝ applying bridge with llms. to bridge the knowledge gap, we scale the expert's knowledge with llms using the expert-guided decision-making model. we construct a dataset of real-world tutoring conversations, annotated with expert decisions and responses. our open-source dataset consists of 700 real tutoring sessions conducted with 1st-5th grade students in title i schools, predominantly serving low-income students of color. following ferpa guidelines, our study is irb-approved and conducts secondary data analysis based on our data use agreement with the tutoring provider and school district. we conduct a thorough human evaluation to compare the expert, novice and llms in remediation. to our knowledge, our work is the first to assess the performance of llms such as gpt4 and instruct-tuned llama-2-70b on remediating student mistakes. we find that the response quality of llms significantly improve with the expert's decision-making process: response from gpt4 with expert-and self-generated decisions are 76-88% more preferred than gpt4 without. context-sensitive decisions are also critical to closing the knowledge gap: random decisions decrease gpt4's response quality -67% than expert decisions. complementing our quantitative analy-sis, our lexical analysis reveals that novices and llms without the expert's decision-making process engage superficially with student's problemsolving process: they give away the answer or prompt the student to re-attempt without further guidance ("double check", "try again"). ## data sources tutoring transcripts. our data is sourced from a tutoring provider that offers end-to-end services for school districts, including the tutoring platform, instructional materials, and tutors. the research team executed data use agreements with the tutoring provider and southern u.s. school district serving over 30k that outlined the allowable usage of the data to improve instruction in collaboration with an educational agency. following ferpa guidelines, we were eligible to engage in secondary data analysis with student data, which is what we did for this study. the students in these tutoring sessions are in the first to fifth grade, learning a variety of math topics. the majority of schools are classified as title i and three-quarters of students identify as hispanic/latinx. this district focused on addressing existing achievement gaps among their students, as well as responding to the learning disruptions caused by the pandemic. the tutoring interactions are text-based, integrated on the providers' online platform. the platform has several features, including a whiteboard. the tutor communicates primarily through text message in a chat box, while the student uses either voice recording or the chat. preprocessing. the chat transcripts are deidentified by the tutoring provider. the student's name is replaced with [student] and the tutor's name is replaced with @xcite . our data uses excerpts from the original tutoring chat sessions, where the tutor responds to a mistake. tutors on this platform use templated responses to flag mistakes, such as "that is incorrect" or "good try." we leverage these templates to create a set of signalling expressions used by the tutor to identify excerpts. specifically, we search for a three turn conversation pattern where (1) the tutor sends a message containing a question mark "?", (2) the student responds via text, then (3) the tutor uses a signalling expression. the set of signalling expressions were validated on a random sample of 100 conversations to ensure complete coverage. appendix c includes the full set of signalling expressions we use. ## the bridge method for expert-guided decision-making we introduce bridge which uses cognitive task analysis (cta) to analyze the experts' latent thought process ( §4.1). we translate it into a decision-making process ( §4.3), where each step is associated with a set of decision options ( §4.2). ## evaluation we measure the extent to which the generated responses improve over the original tutors' responses. we recruit teachers through prolific (identified through prolific's screening criteria) to perform pairwise comparisons between the tutor response and a response generated by the expert or one of the 12 models. a random set of 40 pairs per model is evaluated by 3 annotators each, who are blind to the source of the responses. raters evaluate the pairs along four dimensions. the first two are usefulness and care, as these have been identified as key qualities of effective remediation in prior work @xcite @xcite . the third is human-soundingness; our preliminary analysis indicated that low learning outcomes strongly correlated with whether the student was distracted by whether their tutor was human during their tutoring session. given that the tutoring is chat-based, we include this as another dimension for measuring effectiveness. finally, we ask the raters which responses they prefer using, if they were the tutor. ## discussion & conclusion our work presents several contributions for bridging the expert-novice gap and improving the learning experience at scale. first, we develop bridge, which leverages cognitive task analysis to translate an expert's latent thought process into a decisionmaking framework. we apply this to the task of remediating mistakes because they are prime learning opportunities to correct misunderstandings hindering learning. second, we contribute a rich dataset with expert annotations on their decisions and responses. the dataset comes from a tutoring program that works with a majority of title i schools, and is a valuable resource for providing equitable, high-quality learning experiences. finally, we perform a thorough evaluation and lexical analysis of experts, novices and llms. we demonstrate that expert-guided decision-making and strategic decision selection are critical to improving remediation quality. novices and llms alone use passive remediation language and do not engage with the student's error traces. our findings indicate promising avenues for scaling high-quality tutoring with expert-guided decision-making. for example, the tutor can make the decisions and the llm generates an initial response that is further edited by tutor. altogether, our work shows promising results of an expert-guided human-llm approach that makes strides towards bridging the knowledge gap.