Instruct, Not Assist: LLM -based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging (2024)

Abstract: Socratic questioning is an effective teaching strategy, encouraging critical thinking and problem-solving. The conversational capabilities of large language models (LLMs) show great potential for providing scalable, real-time student guidance. However, current LLMs often give away solutions directly, making them ineffective instructors. We tackle this issue in the code debugging domain with TreeInstruct, an Instructor agent guided by a novel state space-based planning algorithm. TreeInstruct asks probing questions to help students independently identify and resolve errors. It estimates a student’s conceptual and syntactical knowledge to dynamically construct a question tree based on their responses and current knowledge state, effectively addressing both independent and dependent mistakes concurrently in a multi-turn interaction setting. In addition to using an existing single-bug debugging benchmark, we construct a more challenging multi-bug dataset of 150 coding problems, incorrect solutions, and bug fixes– all carefully constructed and annotated by experts. Extensive evaluation shows TreeInstruct’s state-of-the-art performance on both datasets, proving it to be a more effective instructor than baselines. Furthermore, a real-world case study with five students of varying skill levels further demonstrates TreeInstruct’s ability to guide students to debug their code efficiently with minimal turns and highly Socratic questioning.

## introduction with the rapidly expanding conversational and reasoning abilities of large language models (llms), there has been a substantial rise in demand for exploiting their capabilities within a multitude of educational applications @xcite in order to widen accessibility via personalized feedback. specifically, several recent works explore * these authors contributed equally to this work. single-turn assistant: i'll be happy to help you. you made an error in the recursive function. here is the corrected code: multi-turn instructor: i'll be happy to help you. how is the fibonacci sequence defined and computed? student: the fibonacci sequence is a series of numbers in which each number is the sum of the two preceding ones. instructor: that is correct! let's take a step further. how are you utilizing recursion for computing the sequence? the use of llms for providing feedback and guidance to students @xcite @xcite . however, llms are typically optimized to generate customer-serving, assistant-like responses, which also translates into the types of questions asked. especially for educational domains, this style of questioning can be suboptimal @xcite @xcite . for instance, if a student is seeking help from an instructor for correcting their mistakes (e.g., debugging their buggy code), we consider two forms of potential responses: assistant-like and instructor-like. as shown in figure 1 , an assistant-like response would not be a successful educational interaction, as it leads to the assistant directly providing an answer. on the other hand, an instructor-like response reflects the educational philosophy of socratic questioning. socratic questioning is a teaching strategy where the student independently solves their problem by answering guiding questions, instead of being given the solution directly @xcite . this is a more effective learning strategy because the weight of learning falls on the student as they must put in effort to answer a question as opposed to solely relying on the model @xcite . therefore, we aim to re-orient an llm to be an instructor, not an assistant, by asking socratic questions that (1) help the student understand their mistakes, and (2) do not directly provide the answer. to tackle these challenges, we propose treeinstruct based on the following principles: while these principles can apply to many educational domains, this paper focuses on code debugging, which presents unique challenges. realworld code debugging often involves multiple, potentially interdependent conceptual and syntactical bugs. for instance, figure 1 shows that first resolving the student's conceptual misunderstanding of recursion in fibonacci helps them identify their recursive syntactical bug (figure 1 ). however, existing work fails to account for such nuances and assumes single-turn feedback @xcite @xcite . this ignores the sub-steps required for the student to understand each bug. in contrast, treeinstruct constructs a multi-turn debugging plan (state representation), defined as the set of student misunderstandings and mistakes (state variables) to be resolved in order to comprehend and correct their bug(s). we define all potential paths to complete these tasks as the state space. we traverse the space using socratic questions and trace which variables have been resolved, grounded based on the student's responses. while existing llm-based tutors are effective in fixing the student's code with high success, they are either prone to directly revealing code answers or cannot be adapted to new student responses. for example, codeaid @xcite (specifically, the "help fix code" and "question from code" modules, as these are most similar to our setting) directly provides code or pseudocode 57% of the time, and achieves a mere 55% rate of helpfulness. on the other hand, treeinstruct exploits the state space to dynamically construct a tree of questions based on (1) incorrect student responses, or (2) gaps in the student's knowledge. the sibling and parent-child relationships between questions reflect the manner in which they traverse the state space. finally, it exploits both the student's knowledge state and any proposed bug fixes to serve as the dynamic stopping condition. overall, treeinstruct takes a more structured approach to multi-turn conversational feedback, as (1) grounding the conversation on the state space representation ensures that all bugs are sufficiently addressed, and (2) constructing a tree based on the student's current level of understanding allows for more relevant and personalized question generation. we summarize our contributions below: reproducibility: we release our data and source code foot_0 to facilitate further studies. algorithm 1 treeinstruct require: p (problem description), b (buggy code, bug descriptions), c (corrected b code, bug fixes) 1: @xmath0, τ2, . . . , τ k } ← generatestate(p, b, c) ▷ section 3.2: state representation: (resolved?, task) 2: l ← 0, q ← {l : []}, h ← [], f ← {} ▷ tree level, question list/level, conv. history, student bug fixes 3: q ← generatequestion(p, b, c, τ1) ▷ section 3.3: generate initial question 4: while ∃ τ ∈ s s.t. ¬isresolved (s, f, c) do ▷ section 3.4: process while tasks or bugs remain 5: r ← studentresponse(q) 6: v, w ← verifyresponse(q, r) ▷ section 3.3: is r to q correct (v); why or why not (w)? 7: h.add(q, r) 8: q[l].add(q) 9: if @xmath1: q ← generatesiblingquestion(τ, q[l], h, w) ▷ section 3.3: factor in why the student was incorrect 11: else ▷ correct student response 12: s, w ← updateunderstanding(s, q, r) ▷ section 3.3: tasks τi . . . τ k resolved? if ¬s[τ ], why (w)? 13: if ¬s[τ ] then 14: q ← generatechildquestion(τ, q[l], h, w) ▷ section 3.3: factor in why τ was unresolved 15: l ← l + 1 ▷ advance to next tree level 16: else ▷ task τ resolved 17: f ← getstudentbugfixes(h) ▷ section 3.4: ask student for bug fixes (if any) 18: l ← 0, q ← {l : []} ▷ τ @xmath2 ## methodology as shown in figure 2 , treeinstruct aims to dynamically guide the multi-turn conversation based on its estimated state space. section 3.1.2 provides an overview of the three different agents we use and their respective roles during the state space generation/update and tree construction processes (outlined in figure 3 ). this allows treeinstruct to respond to the student's current level of understanding adequately. algorithm 1 contains the pseudocode for all components in our method. ## conclusion this paper proposes a novel method, treeinstruct, for state space estimation and dynamic tree-based questioning for multi-turn socratic instruction. we construct a novel multi-bug debugging dataset, multi-debug, with 150 expert-annotated conceptual and syntactical problems and buggy solutions/fixes. extensive experiments on an existing benchmark and multi-debug demonstrate that treeinstruct can be universally applied to both open and closed source models. we also showcase that treeinstruct's strong socratic questioning abilities widely outperform all baselines through both (1) rigorous quantitative and qualitative expert evaluation (preferred over 77.94% of the time), and (2) real-world interactions with students of varying coding abilities.