Évaluation pédagogique du code à l’aide de grands modèles de langage. Une étude comparative à grande échelle contre les tests unitaires (2025)

Abstract: L’évaluation automatisée en éducation par projet pour l’apprentissage de la programmation s’appuie traditionnellement sur les tests unitaires pour juger les soumissions de code des étudiants, mettant l’accent sur la correction fonctionnelle. Cependant, ces tests négligent souvent des aspects qualitatifs du code, comme la lisibilité ou la modularité. Cette étude examine le potentiel des grands modèles de langage (LLM) pour évaluer les soumissions de programmation, en comparant leurs résultats à ceux des tests unitaires. À partir d’un grand ensemble de données de rendus d’étudiants à une collection de projets de développement logiciel, nous appliquons des analyses statistiques, modélisations prédictives, ainsi que plusieurs comparaisons pour évaluer l’efficacité des LLMs. Nos résultats mettent en évidence une corrélation significative entre les évaluations des LLMs, pour des prompts donnés, et les tests unitaires. Les modèles prédictifs montrent que les scores des LLMs peuvent être approximés à partir des résultats des tests unitaires, et les classements d’étudiants issus des deux approches sont fortement corrélés. Ces constats restent robustes même en présence de bruit injecté dans les rendus étudiants. Ces résultats suggèrent que les LLM, en capturant des dimensions supplémentaires de la performance, peuvent enrichir les cadres d’évaluation éducative, offrant une approche totale plus nuancée et complète.

## introduction in recent years, large language models (llms) such as meta llama, gpt-4, and openai codex have demonstrated remarkable capabilities in understanding, generating, and analyzing source code. these models are increasingly employed for tasks like code completion, refactoring, documentation generation, and even bug detection. their strength lies in their ability to capture not only the syntactic structure of code, but also deeper semantic patterns, architectural styles, and developer intentcapacities that go far beyond traditional static analysis tools. despite these advancements, most automated code evaluation systems in educational and industrial contexts continue to rely heavily on unit testing. unit tests are powerful for verifying that a program behaves correctly on a predefined set of inputs, but they often fall short in assessing broader dimensions of code quality such as readability, maintainability, modularity, or security. furthermore, unit tests can be brittle, easy to bypass, or incomplete, particularly in the case of student-written code where test coverage is limited and implementation diversity is high. this raises a central question : can llms serve as effective evaluators of code quality, and how do their assessments relate to more traditional, test-based evaluation methods ? in other words, to what extent do llm-based evaluations agree with unit test results, and where do they diverge ? understanding this relationship could open the door to hybrid evaluation systems that combine the precision of testing with the semantic insight of machine learning models. in this paper, we propose a systematic study of this question based on a large-scaled dataset of computer science evaluation corpus of code assessment. our contributions are threefold. first, we build a curated dataset of over 700 real-world student programming submissions in c and c++, each paired with execution traces used for dynamic unit testing. second, we introduce an llm-based evaluation framework that produce structured numerical scores and descriptive feedback across seven code quality dimensions : complexity, readability, maintainability, efficiency, modularity, documentation, and security. third, we conduct a detailed comparative statistical analysis of llm outputs and unit test results to reveal correlations between the two approaches. our findings highlight the potential of llms to serve as complementary tools to unit testing, especially for evaluating aspects of code that are hard to express or verify through behavioral tests alone. this work contributes to a broader effort toward more holistic, intelligent, and nuanced code evaluation systems. ## related work research on automated code evaluation spans several decades and has primarily focused on two complementary approaches : static analysis and dynamic testing. static analysis tools-such as pylint, eslint, clang-tidy or sonarqube-perform rule-based inspections of source code to detect syntax errors, code smells, style deviations, and certain classes of bugs without executing the program @xcite . while these tools are light-weight and provide immediate feedback on code structure and style, they cannot verify runtime behavior or catch errors that only manifest under specific inputs. dynamic testing frameworks, including junit, pytest, and google test, execute test suites on compiled or interpreted code to ensure that functions produce expected outputs for a variety of inputs @xcite . unit tests excel at validating functional correctness but depend heavily on the quality and coverage of test cases : poorly designed or incomplete tests leave latent bugs undetected, and writing exhaustive test suites is labor-intensive. in educational contexts, instructors often provide limited test harnesses, which may not fully capture the diversity of student implementations @xcite . machine learning techniques have also been explored for code evaluation. early work applied classical supervised models and feature engineering to predict code quality metrics (e.g., cyclomatic complexity, churn rate) from static code features @xcite . more recently, deep learning models trained on large code repositories enabled tasks such as code summarization, bug detection, and automatic grading of simple programming assignments @xcite . however, these models often required significant task-specific training data and lacked the flexibility to generalize across languages and domains. the advent of llms-such as openai codex, meta llama, and gpt-4-has transformed the field by providing pretrained representations capable of zero-or few-shot performance on a broad range of code tasks. benchmark studies show that these models achieve state-of-the-art results in code completion, bug fixing, and semantic code search @xcite . prompt engineering techniques have been developed to elicit structured outputs-scores, json reports, even junit xml-from llms, enabling "analysis-as-a-service" for code quality metrics @xcite . despite this progress, relatively few studies have systematically compared llm-based semantic assessments with traditional dynamic testing in educational grading contexts. prior work has measured the correlation between llm-generated grades and unit test pass rates on python assignments, reporting moderate alignment but limited interpretability of the model feedback @xcite . other approaches have proposed hybrid pipelines that combine language models for style feedback with traditional unit testing frameworks for functionality, though these efforts often lack rigorous statistical analysis across multiple quality dimensions @xcite . this work fills this gap by statistically quantifying agreement with dynamic tests. in doing so, we provide the first comprehensive framework for juxtaposing semantic llm evaluations with execution-based assessments in programming education. ## dataset the dataset utilized in this study consists of both test-unit and llm-based analysis of student submissions generated in response to a diverse set of prompts, each designed to evaluate different facets of the subject material. these prompts were constructed to encompass a range of cognitive and technical skills, including algorithmic problem solving, code correctness, style, and conceptual understanding. to evaluate the submissions, several language models were employed. each model provided distinct assessments, leveraging varying underlying architectures and training paradigms. for each submission, two primary forms of data were collected : ## methods we systematically investigate the relationship between language model evaluations and unit test results. the approach encompassed classical statistical analysis, predictive modeling, and ranking analysis. each method provide complementary insights into the extent and nature of the alignment between the two evaluation systems. ## conclusion this study provides empirical evidence that large language models (llms) can effectively analyze student submissions, complementing traditional unit testing. statistical analysis reveals a significant relationship between llm scores and unit test outcomes, suggesting that llms capture meaningful patterns of correctness and understanding. predictive modeling confirms that while unit-test results explain much of the llm scoring, the models also highlight qualitative aspects beyond binary correctness. these findings support the integration of llms into educational assessment frameworks to enrich evaluation with dimensions like partial correctness, reasoning, and code quality. future work will explore broader evaluation criteria, alternative model architectures, and hybrid approaches that combine the strengths of both assessment methods across varied contexts. the results of the noise-mixing probe, reported in