Error Tracing in Programming: A Path to Personalised Feedback (2024)

Abstract: Knowledge tracing, the process of estimating students’ mastery over concepts from their past performance and predicting future outcomes, often relies on binary pass/fail predictions. This hinders the provision of specific feedback by failing to diagnose precise errors. We present an error-tracing model for learning programming that advances traditional knowledge tracing by employing multi-label classification to forecast exact errors students may generate. Through experiments on a real student dataset, we validate our approach and compare it to two baseline knowledge-tracing methods. We demonstrate an improved ability to predict specific errors, for first attempts and for subsequent attempts at individual problems.

## introduction the increasing importance of digital technologies has made programming a critical skill. the teaching of programming has long been recognised as difficult, and novice programmers often struggle with syntax, and with conceptual and problemsolving skills @xcite . practical assignments, designed to enhance understanding, often become stumbling blocks due to compiler errors that are not informative for beginners, leading to confusion or discouragement @xcite . further, given large class sizes, providing personalised feedback from instructors is difficult @xcite . recent research has explored automatic feedback generation, including test-case analysis @xcite and ai-driven automatic program repair systems @xcite @xcite . but many of these system fail to trace the individual learner's profile or unique learning trajectory, thus reducing the effectiveness of the feedback provided @xcite . in contrast, knowledge tracing (kt), an educational data mining technique, has the potential to create personalised learning experiences by predicting student performance based on their mastery of concepts @xcite @xcite . in programming education, kt is useful for recommending exercises, predicting assignment outcomes, and identifying students at risk of underperforming @xcite . but traditional kt models often overlook the granularity of student responses, treating all correct or incorrect attempts uniformly @xcite . programming errors, though, vary widely, from simple syntax mistakes like a missing semicolon, to more complex issues such as failing to implement a loop correctly. deep knowledge tracing (dkt) @xcite uses neural networks to identify specific patterns, and thus allows more specific feedback. this paper propose a refined application of dkt to identify precise compiler errors. by analysing the error patterns in students' historical performance, we aim to identify the specific concepts or syntax elements that a student has not yet mastered. this then enables the delivery of targeted feedback focused on those elements. in addition, by analysing the patterns of multiple students in a class, we can highlight common error patterns, for further action by educators. our contributions are as follows. ## related work knowledge tracing (kt) is designed to predict students' future performance by analysing their past interactions with learning materials. initially, kt relied on probabilistic models such as bayesian knowledge tracing (bkt) @xcite , which estimates students' mastery using a bayesian network and a set of fixed parameters (guess, slip, learn, and sometimes forget). bkt's extended by @xcite through the introduction of dynamic bkt, to account for interactions between different knowledge components. deep knowledge tracing (dkt) leverages recurrent neural networks to harness the sequential patterns in student interaction data, effectively capturing not only correctness of responses but also the order and context of these interactions @xcite . recent advances includes techniques such as attention mechanisms (e.g., akt-contextaware attentive knowledge tracing @xcite ), external memory modules (e.g., dkvmn-dynamic key-value memory networks for knowledge tracing @xcite ), and gkt-graphbased kt @xcite , each aiming to better understand the learning process's complexities. dkt has outperformed recent deep learning models @xcite . @xcite ; @xcite gives a comprehensive review of kt models. traditional dkt models primarily rely on sequences of question numbers and the correctness of attempts for prediction, often overlooking detailed information about students' approaches to solving questions @xcite @xcite . this omission restricts their predictive power across different domains. however, incorporating domain-specific features has been shown to enhance performance. for example, in the mathematical domain, @xcite enhanced predictions by including question-concept relationships derived from pretraining embeddings via bipartite graph (pebg), while in the programming domain, @xcite introduced code features using code2vec. there has been a push to extend dkt's application beyond mere correctness prediction. @xcite adapts dkt to forecast the specific options students select in multiple-choice questions. inspired by this, our work aims to tackle the more complex scenario of open-ended programming questions, which creates the challenge of interpreting diverse compiler errors. @xcite develops open-ended knowledge tracing (okt), which integrates an enhanced dkt model with code features from an abstract syntax tree neural network-astnn @xcite and textual question features from gpt-2, aiming to predict student performance. they then employ a gpt-2based text-to-code generator, guided by the dkt model's hidden state as a knowledge estimate, to generate diverse code solutions that mirror the student's comprehension. ## methodology our work introduces an alternative approach for dkt to predict directly the specific errors students are likely to encounter. we assess how different domain features like student code submissions, reference solutions, and question-concept relationships affects error prediction. after pinpointing individual errors, we employ a bottom-up approach, aggregating these error predictions to assess overall student performance as pass (error-free) or fail (submission with errors). this is to assess whether focusing on granular error predictions can enhance the accuracy of student outcome forecasts compared to traditional dkt predictions. figure 1 illustrates our proposed model structure. ## experimental setup our experimental setup, detailed below, outlines the data collection methods, model training protocols, and evaluation metrics used to rigorously test the efficacy of our proposed models. ## results results are shown in ## conclusion and future works in our study, we enhanced traditional knowledge tracing methods by developing a framework capable of predicting overall correctness and specific student errors. our error-dkt models demonstrated significant effectiveness, substantially outperforming baseline okt models in overall attempts prediction with improvements of +15.8% and +23.2% using single-step and ensemble strategies (holistic error prediction), respectively. the ensemble approach significantly enhances accuracy by initially distinguishing error-free submissions from erroneous ones, and then specifically pinpointing the errors in submissions forecasted to fail. predictions for initial attempts generally exhibit higher accuracy, likely due to the uniformity of these submissions and the rich historical data available. however, as students revise their submissions in response to compiler feedback, the complexity of prediction increases, particularly for subsequent attempts. this issue is compounded in problems with high error rates and frequent submissions, like problems 5 and 6, where performance notably declines. despite the advantages in error prediction, our method showed less effectiveness in integrating individual errors into holistic pass/fail assessments compared to direct evaluations by traditional dkt methods. nonetheless, the integration of diverse input features enhances the dkt model's ability to predict correctness. these findings underscore the potential of error-dkt to improve the precision of error predictions and affirm the ongoing need for models that can adapt to complex error patterns and improve feedback mechanisms in educational settings. for future work, several promising directions emerge. first, experimenting with advanced dkt architectures like akt and dkvmn and refined ensemble methods may improve error prediction and accommodate a wider array of error types with more extensive datasets. secondly, optimising the okt model to extend its predictive competence to logical as well as compiler errors could yield more comprehensive error detection. thirdly, there's a significant opportunity to enhance knowledge tracing models to interpret learned patterns, correlating them to specific knowledge areas, such as debugging skills reflected in students' coding progression. finally, integrating our framework with an automated feedback mechanism will be vital in evaluating its effectiveness in delivering personalised, actionable feedback to students.