Using Program Repair as a Proxy for Language Models’ Feedback Ability in Programming Education (2024)

Abstract: One of the key challenges in programming education is being able to provide high-quality feedback to learners. Such feedback often includes explanations of the issues in students’ programs coupled with suggestions on how to fix these issues. Large language models (LLMs) have recently emerged as valuable tools that can help in this effort. In this article, we explore the relationship between the program repair ability of LLMs and their proficiency in providing natural language explanations of coding mistakes. We outline a benchmarking study that evaluates leading LLMs (including open-source ones) on program repair and explanation tasks. Our experiments study the capabilities of LLMs both on a course level and on a programming concept level, allowing us to assess whether the programming concepts practised in exercises with faulty student programs relate to the performance of the models. Our results highlight that LLMs proficient in repairing student programs tend to provide more complete and accurate natural language explanations of code issues. Overall, these results enhance our understanding of the role and capabilities of LLMs in programming education. Using program repair as a proxy for explanation evaluation opens the door for cost-effective assessment methods.

## introduction large language models (llms) and applications leveraging them such as chatgpt have been embraced by both the general public and academia. the adoption is also visible in the domain of computing and programming education, where researchers have highlighted a variety of learning tasks that llms can tackle @xcite , including their performance in providing help and feedback to students @xcite . feedback is a crucial part of learning @xcite . while various forms of feed-0.2 0.4 0.6 pass@1 0.2 0.4 0.6 0.8 1.0 completeness model tinyllama gemma-2b codellama zephyr-beta mistral gemma-7b gpt-3.5-turbo gpt-4-turbo figure 1: summary benchmarking results. the quality of llms' natural language descriptions of issues in students' code (completeness) tends to increase with llms' ability to fix the student programs (pass@1). with the increasing number of llms proficient at providing feedback @xcite to some degree, selecting the best one before deploying it in classrooms @xcite ) can be challenging. human evaluation can take time, as it requires either manual assessment or annotated datasets. while research in the automated evaluation of llm generation is on the rise @xcite , also in educational areas @xcite , the developed methods often rely on other language models (e.g., utilizing powerful yet expensive llms such as gpt-4), which can induce computational or financial costs. a more cost-effective approach is needed. before the advent of llms, a stream of work in programming education has focused on educational program repair @xcite @xcite , where the goal is to produce fixes for students' incorrect programs. although repairs to student programs are not always directly provided to students, they serve as a fundamental step in generating different types of support, including next-step hints for intelligent tutoring systems @xcite . while direct evaluation of feedback with natural language explanations can be challenging, evaluating whether llms can fix programs is much more straightforward. with this in mind, we hypothesize that the student program repair capability of an llm may relate to its capability to provide natural language explanations of code issues. if this would hold, program repair capability -which is easier to assess -could serve as a proxy for evaluating feedback quality. our intuition is supported by prior work that has found relationships between llms' abilities in related domains. for instance, llms that are proficient in solving specific problems are effective judges of the quality of explanations in those domains @xcite . similarly, there is some evidence that instruction-tuned llms trained on specific tasks can generalize to unseen parallel or close tasks @xcite . in this article, we investigate whether there effectively exists a relationship between the ability of llms to repair students' programs and their ability to explain code issues in natural language. if our hypothesis holds, researchers could more easily benchmark llms for other educational purposes, allowing educators to streamline the selection of llms. our evaluation focuses on several leading and popular open-source language models, as well as proprietary models. the main contributions of this article are (1) the benchmarking of several leading language models' abilities for program repair and (2) natural language explanation of code issues, as well as (3) the analysis and identification of the relationship between the two tasks. ## methodology we (1) evaluate how llms perform in generating repairs to incorrect programs, (2) evaluate how llms perform in explaining the issues in programs, and (3) study the potential relationship between the ability to generate repairs and the ability to generate natural language explanations. to ensure a comprehensive assessment, our study encompasses zero-shot evaluations @xcite of proprietary and state-of-the-art open-source llms having less than 7 billion (7b) parameters. our experiments leverage a publicly available dataset comprising real-life students' submissions to python programming problems. next, we describe the programming dataset, outline our evaluation methodology, and list the language models included in this evaluation. we release the code used to perform our experiments as an additional contribution foot_0 . ## results first, we describe our general results, then, we outline an ablation analysis detailing the performance of the selected models over a set of programming concepts. ## discussion repair as a proxy for feedback. our results suggest that language models' relative ability to fix students' programs (which is easy to evaluate) tells us how these language models will compare in finding all issues in students' code while avoiding hallucination (for big enough language models). based on our discovery, one can devise more efficient llm selection pipelines. for instance, a simple strategy consists of filtering out language models for which repair performance does not reach a certain threshold, a threshold set based on a few evaluations of llm natural language generation performance. as an illustrative scenario, only evaluating the mistral model on our dataset allows us to reasonably assume that language models performing worse than 0.32 in pass rate (pass@1) are unlikely to generate complete explanations for more than 73.8 % of programs. using this pass rate value can thus act as a selection lower bound. as llms are becoming more widely adopted in education @xcite , and as the number of available models is increasing, these insights can help in the adoption process as institutions evaluating llms for their context can potentially reduce the number of llms to consider or limit the number of tasks conducted during the evaluation. open-source language models strike back. another important finding emerging from our results is that while high-performance program repair must rely on proprietary models, recent 7b parameters models such as gemma-7b can generate highquality feedback competitive with sota models @xcite . this has positive implications for educators interested primarily in giving students feedback rather than repairing solutions, as such feedback can also be generated via privacypreserving open-source models. however, it's important to acknowledge that running such models requires custom computational resources. in the literature, 7b parameter models are sometimes termed "small" due to their relative size compared to many large language models (e.g. falcon-180b model @xcite ). yet, a 7b parameter is not small in terms of computational resources as it requires a large gpu to fit entirely into memory (without quantization). there is currently a trend in developing small language models (less than 3b parameters) such as tinyllama and gemma which can run on more modest hardware (e.g., consumer laptop gpu, or accelerated hardware). however, the performance of such llms, as our results suggest is still lagging behind their 7b parameters counterparts. identifying specific knowledge gaps. unfortunately, our results do not yet allow us to identify which programming concepts llms will struggle to explain in natural language from their program repair performance. while individual repair performance depends on the concept being practised, a language model's performance in explaining issues does not (i.e., the performance is constant across all concepts). we hypothesize that the per-concept performance gap is only revealed for the harder task of fixing students' programs. uncovering llm knowledge gaps with automated measures might require us to rely on harder automatically evaluable tasks (e.g. qlcs @xcite ). interplay of programming feedback types. our primary research objective is to deepen our understanding of llms' feedback capabilities in educational contexts. specifically, we seek to explore the relationship between different forms of feedback and program repair. while we treated feedback (identifying and explaining issues in programs) and program repair as distinct tasks in this study, we acknowledge their inherent interdependence. previous research suggests that high-quality repairs can induce high-quality feedback when provided in context @xcite . however, generating high-quality repairs is inherently challenging, as our results suggest, requiring the language model to comprehend what is wrong in a program and how to address the issues. in contrast, we believe explanations of issues in students' programs could serve as reasoning steps @xcite , enhancing the subsequent generation of repairs @xcite . these refined repairs, in turn, could facilitate the generation of high-quality next-step hints @xcite . research investigating the interplay between different types of feedback is thus pivotal in unlocking the full potential of language models to support programming education. by studying the performance of generating repairs without conditioning on feedback, nor generating feedback based on repairs, our work establishes a foundational understanding that will allow the research community to assess the extent to which various prompting techniques enhance feedback performance. ## conclusions in this article, we have uncovered an intriguing relationship between llm performance in program repair and the capability to explain issues in code. our evaluations encompassed both opensource and proprietary models, examining their generic performance as well as concept-specific proficiency. while selecting and deploying a specific language model may not be challenging, identifying the most suitable one for a particular purpose can be complex, particularly when considering financial, hardware, or other limitations. at a time when there are calls to rethink how programming is taught @xcite , the insights gleaned from our work can provide valuable guidance for educators in choosing llms that align with their instructional contexts. future work. our future work will involve two specific directions. first, we'll continue our investigation of the relationships between various types of programming feedback and program repair. all these efforts remain an attempt to streamline the selection process of language models based on automated evaluation measures. besides studying llm performance, our second objective is to leverage our computational resources to improve these llms' ability to provide feedback. in particular, small language models' poor explaining performance suggests that these models will benefit from alignment procedures designed specifically to improve feedback abilities @xcite .