Training Turn-by-Turn Verifiers for Dialogue Tutoring Agents: The Curious Case of LLM s as Your Coding Tutors (2025)

Abstract: Intelligent tutoring agents powered by large language models (LLMs) have been increasingly explored to deliver personalized knowledge in areas such as language learning and science education. However, their capabilities in guiding users to solve complex real-world tasks remain underexplored. To address this limitation, in this work, we focus on coding tutoring, a challenging problem that requires tutors to proactively guide students towards completing predefined coding tasks. We propose a novel agent workflow, Trace-and-Verify (TRAVER), which combines knowledge tracing to estimate a student’s knowledge state and turn-by-turn verification to ensure effective guidance toward task completion. We introduce DICT, an automatic evaluation protocol that assesses tutor agents using controlled student simulation and code generation tests. Extensive experiments reveal the challenges of coding tutoring and demonstrate that TRAVER achieves a significantly higher success rate. Although we use code tutoring as an example in this paper, our approach can be extended beyond coding, providing valuable insights into advancing tutoring agents for human task learning.

## introduction tutoring has long been recognized as one of the most effective methods for enhancing human learning outcomes and addressing educational disparities @xcite . by providing personalized guidance to students, intelligent tutoring systems (its) have proven to be nearly as effective as human tutors in fostering deep understanding and skill acquisition, with research showing comparable learning gains @xcite . more recently, the advancement of large language models (llms) has offered unprecedented 1. define a func,on `serialize` that takes an `item` of type `image` and returns a tuple containing bytes. 2. check if the `item` is an instance of `jpegimagefile`. 3. … from abc import abc, abstractmethod … class pilserializer(serializer): … (more 100 lines) figure 1 : an illustration of coding tutoring, where a tutor aims to proactively guide students toward completing a target coding task while adapting to students' varying levels of background knowledge. opportunities to replicate these benefits in tutoring agents @xcite @xcite , unlocking the enormous potential to solve knowledge-intensive tasks such as answering complex questions or clarifying concepts. previous research has extensively explored tutoring in knowledge delivery, including language learning @xcite , mathematical reasoning @xcite , and scientific concept education @xcite . most aim to enhance students' understanding of target knowledge by employing pedagogical strategies such as recommending exercises @xcite or selecting teaching examples @xcite . however, these approaches fall short in broader task tutoring situations demanding both understanding and practical application of specific pieces of knowledge to solve real-world, goal-driven problems. such scenarios require tutors to proactively guide people toward completing targeted tasks (e.g., coding). furthermore, the tutoring outcomes are challenging to assess since targeted tasks can often be completed with openended solutions. to bridge this gap, we introduce coding tutoring, a promising yet underexplored task for llm agents. as illustrated in figure 1 , the tutor is provided with a target coding task and task-specific knowledge (e.g., cross-file dependencies and reference solutions), while the student is given only the coding task. the tutor does not know the student's prior knowledge about the task. coding tutoring requires the tutor to proactively guide the student toward completing the target task through dialogue. this is inherently a goal-oriented process where tutors guide students using task-specific knowledge to achieve predefined objectives. effective tutoring requires personalization, as tutors must adapt their guidance and communication style to students with varying levels of prior knowledge. developing effective tutoring agents is challenging because off-the-shelf llms lack grounding to task-specific knowledge and interaction context. specifically, tutoring requires epistemic grounding @xcite , where domain expertise and assessment can vary significantly, and communicative grounding @xcite , necessary for proactively adapting communications to students' current knowledge. to address these challenges, we propose the trace-and-verify (traver) agent workflow for building effective llm-powered coding tutors. leveraging knowledge tracing (kt) @xcite , traver explicitly estimates a student's knowledge state at each turn, which drives the tutor agents to adapt their language to fill the gaps in task-specific knowledge during utterance generation. drawing inspiration from value-guided search mechanisms @xcite @xcite , traver incorporates a turn-by-turn reward model as a verifier to rank candidate utterances. by sampling more candidate tutor utterances during inference (see figure 2 ), traver ensures the selection of optimal utterances that prioritize goal-driven guidance and advance the tutoring progression effectively. furthermore, we present dialogue for coding tutoring (dict), an automatic protocol designed to assess the performance of tutoring agents. dict employs code generation tests and simulated students with varying levels of programming expertise for evaluation. while human evaluation remains the gold standard for assessing tutoring agents, its reliance on time-intensive and costly processes often hinders rapid iteration during development. by leveraging simulated students, dict serves as an efficient and scalable proxy, enabling reproducible assessments and accelerated agent improvement prior to final human validation. through extensive experiments, we show that agents developed by traver consistently demonstrate higher success rates in guiding students to complete target coding tasks compared to baseline methods. we present detailed ablation studies, human evaluations, and an inference time scaling analysis, highlighting the transferability and scalability of our tutoring agent workflow. ## problem definition we formulate coding tutoring as an interactive dialogue process between a tutor and a student, where the goal is to help the student implement a working solution that passes predefined unit tests for a target coding task. formally, the tutor is assigned a coding task t that consists of a function signature and a requirement description outlining the desired functionality. the tasks are repository-level, which require an understanding of multiple interdependent files within the codebase to implement a correct solution. the tutor has access to task-specific knowledge k, which includes (i) code contexts: contextual code snippets surrounding the desired code, which help the tutor show examples when necessary; (ii) reference dependencies: cross-referenced elements such as intra-class, intra-file, and cross-file dependencies, along with their corresponding descriptions (e.g., docstrings), which involve key knowledge for completing the desired code; and (iii) reference solution steps: key steps required to complete the target task, describing using natural languages. the student is given the task t and possesses some subset of k as their prior knowledge, but the tutor remains unaware of which specific concepts or dependencies the student has already mastered. the goal of the tutor is to guide the student, regardless of his or her background, toward solving the task t through multi-turn interactions. ## trace-and-verify agent workflow we propose trace-and-verify (traver), an effective workflow for developing tutor agents (see the middle part in figure 3 ). traver integrates two key components: (i) explicit tracing of a student's knowledge state and (ii) utterance decoding guided by a verifier model for turn-by-turn verification. ## the dict evaluation protocol one key challenge in developing tutoring agents is the lack of robust evaluation methods. while human evaluation is essential, its high cost, time requirements, and complexity make it impractical for scalable benchmarking. to address this limitation, we present dialogue for coding tutoring (dict), an automatic protocol for evaluating tutor agents. the overview of dict is shown in figure 3 . our protocol employs llms to simulate students with varying levels of programming knowledge. first, tutors engage in multi-turn dialogues with students to tutor the task. then, students demonstrate their learning outcome by implementing the solution code. we evaluate tutoring effectiveness through automated unit tests of the student-generated code. this automated approach enables controlled, reproducible, and scalable evaluations of tutoring agents. ## related work interactive tutoring. as an advanced form of tutoring systems, interactive intelligent tutoring systems (itss) @xcite @xcite can provide personalized feedback and adaptive learning experiences. they have been extensively explored across various educational domains for knowledge delivery, such as language learning @xcite @xcite , mathematical reasoning @xcite @xcite , and scientific concept education @xcite . these studies mainly focus on enhancing students' understanding of specific pieces of knowledge, using pedagogical strategies such as designing exercises @xcite @xcite , selecting teaching examples @xcite , and remediating student reasoning errors @xcite . furthermore, the effectiveness of these approaches is often measured using closed-form assessments, such as questionanswering @xcite or multiple-choice tests @xcite . instead of focusing on specific knowledge delivery, we explore task-level tutoring, using coding tutoring as a representative example. this domain requires students to engage in open-ended code generation to evaluate tutoring effectiveness. ## conclusion and future work this work explores the potential of llms as tasktutoring agents, using coding tutoring as a representative example. we propose traver, an effective workflow that incorporates knowledge tracing and turn-by-turn verification, to tackle key challenges in coding tutoring. while this work focuses on coding tutoring as an example, the proposed method extends beyond coding to other task-tutoring scenarios. we further introduce dict, a novel evaluation protocol combining student simulation and coding tests to assess tutor performance. such automated evaluation is critical for developing task-tutoring agents as it supports a systematic development and evaluation cycle. although it's outside the scope of this paper, the best-performing agent from the automated evaluation can be further assessed through studies with real human students in the future.