A Survey of LLM -Based Applications in Programming Education: Balancing Automation and Human Oversight (2025)

Abstract: Novice programmers benefit from timely, personalized support that addresses individual learning gaps, yet the availability of instructors and teaching assistants is inherently limited. Large language models (LLMs) present opportunities to scale such support, though their effectiveness depends on how well technical capabilities are aligned with pedagogical goals. This survey synthesizes recent work on LLM applications in programming education across three focal areas: formative code feedback, assessment, and knowledge modeling. We identify recurring design patterns in how these tools are applied and find that interventions are most effective when educator expertise complements model output through human-in-the-loop oversight, scaffolding, and evaluation. Fully automated approaches are often constrained in capturing the pedagogical nuances of programming education, although human-in-the-loop designs and course-specific adaptation offer promising directions for future improvement. Future research should focus on improving transparency, strengthening alignment with pedagogy, and developing systems that flexibly adapt to the needs of varied learning contexts.

## introduction introductory programming courses serve as critical gateways to computer science and related stem careers @xcite ), yet they present unique pedagogical challenges that contribute to high attrition rates @xcite . students must simultaneously master syntax, develop computational thinking skills, and learn to debug complex logical errors, creating cognitive demands that often overwhelm novices. success in these courses often depends on access to timely, targeted interventions, e.g., feedback, explanations, and guided problem-solving, that address individual learning * both authors contributed equally to this work. large language models (llms) have emerged as promising tools to automate aspects of programming education support by addressing the core challenges novice programmers face, including debugging code errors @xcite , repairing faulty code @xcite , obtaining timely feedback, and mastering foundational concepts to work with programming problems (ahmad and ghazali, 2020; @xcite . recent advances in natural language processing now make it possible to generate context-aware feedback, offer real-time debugging support, and adapt explanations to a student's skill level @xcite @xcite . early work also shows the potential of human-ai collaboration, where llm outputs are refined or guided by educators to better align with pedagogical goals @xcite . though promising, it is still unclear what best practices should guide the integration of llms into programming education and how responsibilities should be balanced between automation and human expertise. the presented paper addresses this gap by reviewing recent applications of llms in programming education, focusing on how they are applied, the challenges that arise in practice, and the opportunities to align technical advances more closely with pedagogy. ## methodology and paper selection to survey research on the use of llms in programming education, we reviewed publications across leading venues in computing education, hci, and nlp, such as sigcse, iticse, icer, lak, edm, chi, emnlp, and related workshops. we focused on the 2021-2025 period, when llm-driven systems first began to appear in educational settings. from the surveyed literature, we identified three focal areas in which llms are being applied: formative code feedback, assessment, and knowledge modeling. each aligns with a central pedagogical challenge in introductory programming courses, supporting student learning at scale in contexts where one-on-one guidance is difficult to provide. within formative code feedback, we distinguish between approaches that generate hints and explanations to help students identify and understand their errors, and those that produce corrected code as examples or candidate repairs. assessment focuses on grading and providing evaluative feedback at scale, while knowledge modeling seeks to represent what students know and how they progress in order to give instructors actionable insights through learning analytics. the distribution of reviewed papers across these areas is shown in ## llm usage in formative code feedback debugging assistance is one of the most common reasons to seek help in programming courses, and improvements in code correctness after such support can substantially boost short-term performance @xcite . however, the scalability limits of human-led help sessions have prompted researchers to explore how llms can extend this support in programming education. llm-based feedback on student code is typically delivered in two complementary forms. in some cases, models generate explanations, hints, or scaffolding that help students locate and reason about their own programming mistakes. in others, models produce corrected code directly, offering candidate repairs or examples that students can study and compare against their own solutions. both approaches aim to reduce the bottleneck of human-provided feedback, although they differ in the amount of agency they leave with the learner. a large body of work has explored how llms can generate hints and explanations to guide student reasoning. early evaluations of open-source models suggest that they can assist with syntax and minor semantic issues but continue to struggle with more complex bug localization and multi-line logic errors @xcite . to address these challenges, researchers have developed systems that focus on scaffolding student reasoning. bugspotter, for instance, combines static analysis with llm reasoning to create interactive debugging exercises for low-level programming languages @xcite . iterative selfdebugging loops have been proposed, where models run their generated code, collect execution feedback, and refine patches in multiple passes @xcite . adaptive scaffolding systems further extend these capabilities by monitoring learner progress and providing timely hints or explanations to break complex reasoning into manageable steps @xcite . although such advances improve the accuracy and relevance of llm-supported feedback, human oversight remains necessary to ensure that outputs align with pedagogical goals @xcite . for example, @xcite found that while llms can accelerate support for students, direct answers without educator scaffolding risk undermining learning, highlighting the role of instructors in contextualizing automated feedback. another cluster of research investigates improving the clarity of error explanations to support student comprehension. fine-tuned llms have demonstrated the ability to produce clearer, context-sensitive error messages, improving novice problem-solving @xcite . comparative analyses of human and model debugging strategies reveal differences in reasoning patterns, pointing to opportunities for de-signing ai-assisted tools that nudge learners toward more expert-like approaches @xcite . related work on prompt engineering and explainability techniques, such as step-by-step runtime verification, has also shown promise for improving the readability of error messages and fostering trust in human-ai collaboration @xcite @xcite . expanding further, researchers have also applied llms to code quality feedback, detecting issues such as misleading variable and function identifiers in novice code @xcite . beyond generating hints and explanations, a growing body of work explores producing corrected code and worked examples that students can study alongside their own solutions. for instance, llms have been applied to generate codetracing questions for introductory courses, producing diverse and pedagogically aligned items @xcite . recent work demonstrates how llms can generate worked examples that help students learn strategies and better understand solution approaches @xcite . research also shows that students learn from the process of spotting and fixing code errors @xcite , and that these skills strongly predict learning outcomes and course success @xcite . to support this process, automated program repair (apr) systems have targeted syntactic and semantic errors in student submissions, with llms broadening the scope of repairs to be more contextsensitive and benchmarked transparently @xcite . examples include pydex @xcite , which generates accurate leverages llms to automatically generate accurate fixes for common novice errors in python assignments @xcite ; coast, a multiagent framework that coordinates detection, repair, and verification while synthesizing debugging datasets @xcite ; and repairl-lama, which incorporates repair-aware representations and parameter-efficient fine-tuning to outperform vanilla prompting on standard apr benchmarks @xcite . while these advances demonstrate the technical potential of llms for formative programming support, their educational value depends on when and how the feedback is delivered. automated fixes that come too early, solve too much of the problem, or present complete answers can short-circuit the learning process by removing opportunities for students to reason through their own errors. in con-trast, tools that generate hints, scaffold reasoning, and explain errors without directly supplying solutions are better aligned with pedagogical goals. the challenge is therefore not only improving accuracy on complex bugs but also designing systems that adapt the level of support to the learner's needs. emerging best practices point toward hybrid approaches, where llms address routine or surface-level issues and generate scalable practice materials, while human educators provide context, address deeper misconceptions, and guide students toward lasting debugging strategies. ## llm usage in assessment with the increasing availability of llms in education, there are now provisions for the use of automated teaching assistants (tas) in assessments, particularly in programming courses where grading is frequent and labor-intensive @xcite . early evaluations benchmarked the ability of llms to provide such feedback, demonstrating that even in zero-shot configurations, they can produce rubric-aligned evaluations with moderate agreement to human graders @xcite . these findings position llms as viable tools for scalable deployment, reducing the need for extensive rule-based assessment design. for example, abscribe @xcite demonstrates how llms can support human-ai co-writing by generating and organizing multiple text variations, easing ta workload and improving revision efficiency. however, meta-analytic perspectives caution that such systems inherit biases from training data and require prompt and rubric alignment to meet coursespecific standards @xcite . in classroom settings, results have been mixed. in a study involving more than 1,000 students, gpt-4 reliably evaluated straightforward and clear-cut submissions but required human arbitration for nuanced cases @xcite . similarly, automated grading with llms in a bioinformatics course reduced ta workload and accelerated grading speed, but raised concerns about transparency, reproducibility, and student trust in ai-generated assessments @xcite . to address this, frameworks like begrading @xcite have integrated llms into a multi-stage feedback pipeline, combining initial automated grading with targeted suggestions for improvement, while codev @xcite applied chain-of-thought prompting, ensemble reasoning, and consistency checks to produce accurate and constructive feedback. other work, such as the ai-augmented ta feasibility study @xcite , examined how llms can fit into human ta workflows in cs1 courses, focusing on providing timely, individualized support while preserving grading quality. beyond grading accuracy, the specificity and pedagogical usefulness of llm-generated feedback vary considerably @xcite . recent studies have examined how models can generate formative, actionable feedback that supports skill development in introductory programming courses @xcite . one line of work uses program repair tasks as a proxy for feedback quality, showing that automated grading outputs can contribute to improvements in students' problem-solving and code comprehension skills @xcite . others highlight the need for careful prompt engineering, rubric alignment, and iterative evaluation to ensure that feedback remains contextually relevant and educationally valuable. therefore, human oversight remains a central design principle in this space, with llms serving as collaborators that augment ta capacity rather than replacements. ## llm usage in knowledge modeling just as llms have been applied to formative feedback and assessment, they are increasingly being explored for a broader challenge in programming education: modeling what students know and how their understanding develops over time. knowledge modeling supports instructional design by making student learning more visible, helping educators monitor progress, identify misconceptions, and create targeted interventions at scale. one common approach is knowledge component (kc) extraction, where student work is mapped to the concepts they need to master, such as variables, loops, and conditionals. while this process helps educators monitor progress and create targeted interventions, performing it manually is time-consuming and limits scalability. recent advances have demonstrated how llms can automate this extraction process with promising results. researchers used gpt-4 to generate and tag kcs from multiple-choice questions, with human evaluators preferring the llm-generated tags over instructor-assigned ones in about two-thirds of cases @xcite . kcluster @xcite is another approach to combine llmgenerated question similarity metrics with clustering algorithms to automatically group related problems and discover their underlying kcs, producing models that outperform expert-designed baselines. additionally, others @xcite @xcite have presented early successful results on the use of llms toward kc extraction. researchers have also explored how llms can perform kc extraction during real-time learning interactions. llms can annotate student-tutor dialogues with kc tags during conversations, achieving close to human-level accuracy @xcite . to gain more granular insights into student understanding, test case-informed knowledge tracing is another approach where individual test case pass/fail results serve as indicators for llms to better distinguish which concepts students have mastered versus those they struggle with @xcite . additionally, incorporating student self-reflection prompts can significantly improve kc tagging performance by llms @xcite . while these advances highlight the potential of llms to scale knowledge modeling, their value depends on expert validation of concept mappings and alignment with course objectives. if not carefully validated, knowledge models that are inaccurate or poorly contextualized can misguide instructors and weaken their ability to design effective interventions. when integrated responsibly, however, llm-generated models can strengthen learning analytics by giving educators actionable insights into student progress, revealing common misconceptions, and informing the design of targeted supports at scale. future research can focus on improving the accuracy and reliability of these approaches across varied datasets and on developing methods that ensure valid and useful representations of student knowledge. ## discussion our review of recent llm applications in programming education indicates that systems that successfully address students' pedagogical needs tend to retain human involvement throughout the workflow. across formative code feedback, assessment, and knowledge modeling, successful applications frequently incorporate educators in the workflow to interpret results, refine automated outputs, or make topic best practices formative code feedback use llms to generate hints, explanations, and error messages that scaffold in pedagogically-sound ways; balance automation with formative scaffolding to prevent over-complete fixes and overreliance. assessment ensure llm grading aligns with rubrics and course standards; use human arbitration for nuanced cases. knowledge modeling when using llms for kc extraction and clustering, validate outputs against expert review of topics and subtopics. altogether, the systems surveyed demonstrate several shared strengths. they address scalability by automating tasks that would otherwise demand substantial instructor time, such as grading large cohorts or generating individualized debugging hints. open-source language models have also been incorporated into apr pipelines, where evaluation frameworks use gpt-4-as-a-judge to approximate expert review at scale. this approach highlights the benefits of mixed human-and-automated evaluation in balancing accuracy, scalability, and cost in this domain @xcite . research also incorporates mechanisms that improve consistency and transparency in instructional support, as seen in begrading's variance analysis and criteria-aligned feedback generation @xcite . increasingly, these tools integrate pedagogical considerations into their design, from scaffolding strategies in debugging systems to feedback phrased in ways that guide student reflection and self-correction. however, at the same time, performance varies considerably depending on factors such as prompt design, availability of course-specific training data, and evaluation practices. many llm-based systems are not explicitly tuned to instructional objectives, which can lead to technically correct but educationally unhelpful feedback @xcite . performance often drops when moving from controlled benchmarks to authentic, noisy student code, and the opacity of model reasoning can reduce trust among both students and instructors. there is also the risk that students may overrely on incorrect model outputs, undermining opportunities for productive learning interactions @xcite . ## limitations and future work the current body of research on llms in programming education is still in its early stages, with limitations that can guide future research. while the focal areas covered in this survey reflect active areas of research, other domains, such as collaborative coding and accessibility support, remain underexplored. addressing these gaps can take place in a larger-scale systematic literature review. additionally, relating to the maturity of the current work reviewed, many of the systems surveyed are early-stage prototypes or evaluated only in smallscale settings, with limited evidence on scalability or long-term learning outcomes. technically, while early work has investigated fine-tuning and reinforcement learning with human feedback (rlhf) @xcite , there remains significant scope for advancing model development and designing workflows explicitly aligned with pedagogical goals through course-specific fine-tuning. research could also investigate adaptive collaboration frameworks where the degree of automation varies according to task complexity, user proficiency, and the model's own confidence in its output. further priorities include identifying and mitigating biases in model outputs, especially in grading and feedback, and expanding the use of multi-modal, context-aware interaction that can adapt feedback to the learner's current state. while llms can offer an improved learning experience for programming education, their greatest potential lies in augmenting rather than replacing human expertise. systems that remain adaptable, transparent, and closely aligned with pedagogical best practices are most likely to deliver meaningful and sustainable benefits for learners.