Impeding LLM -assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation (2024)

Abstract: While Large language model (LLM)-based programming assistants such as CoPilot and ChatGPT can help improve the productivity of professional software developers, they can also facilitate cheating in introductory computer programming courses. Assuming instructors have limited control over the industrial-strength models, this paper investigates the baseline performance of 5 widely used LLMs on a collection of introductory programming problems, examines adversarial perturbations to degrade their performance, and describes the results of a user study aimed at measuring the efficacy of such perturbations in hindering actual code generation for introductory programming assignments. The user study suggests that i) perturbations combinedly reduced the average correctness score by 77%, ii) the drop in correctness caused by these perturbations was affected based on their detectability.

## introduction large language model (llm)-based tools such as chatgpt @xcite have demonstrated an impressive ability to create high-quality code given simple prompts and have the potential for significant impact on software development @xcite . while there are ongoing efforts to incorporate such tools into computer science (cs) education @xcite , integrating new technologies into educational curricula can take a long time @xcite . meanwhile, existing cs curricula are under the threat of llm-assisted cheating and require immediate attention @xcite . given that educators have little direct control over the capabilities of industrial-strength llms, two possible directions towards addressing this (a) original prompt (b) perturbed prompt in a file grid_adjacent.py , you will define one function. you are not expected to implement any class. in all … … grid_get_height(grid) this function returns … which are sensible. [omitted for brevity] threat are (i) to detect and penalize llm-assisted cheating; and (ii) to modify problem statements to impede llm-assisted cheating. the first approach is problematic because it can be difficult to determine reliably whether some given content is llmgenerated or not @xcite , and both false positives and false negatives are possible. in this paper, we explore the second option and ask the following question: how can instructors modify assignment prompts to make them less amenable to llm-based solutions without impacting their understandability to students? while there has been some work on the impact of adversarial prompts on llms @xcite , we are not aware of any research investigating adversarial strategies for impeding llm-assisted cheating in a blackbox setting in an academic context. to systematically study the problem, we break it into the following three steps: step 1. measure the accuracy of llms on introductory cs programming assignments, as introductory assignments are at imminent risk @xcite . step 2. develop adversarial techniques to perturb programming assignment prompts and ana-lyze their impact on the quality of llmgenerated solutions to those problems. step 3. run a user study to understand the potential for such perturbation techniques in impeding actual llm-assisted cheating, focusing in particular on whether students can detect and reverse such perturbations. an overview of these steps is presented in figure 2. to measure the accuracy of llm-generated code, we use the same test inputs used to evaluate student submissions. to modify problem statements in a blackbox setting, we design a set of perturbation techniques that are informed by existing literature on adversarial perturbation @xcite @xcite . we use shap @xcite with a surrogate model to guide the perturbation for better efficacy vs. modification tradeoff. we define efficacy (definition 1) for a perturbation technique to quantify the portion of lowering the llm accuracy. to ethically conduct the user study in step 3, we select the study group from students who have already taken the courses corresponding to the assignments used for the study. our findings suggest that existing llms generally struggle to solve assignments requiring interactions across multiple functions and classes. our evaluation of different perturbation techniques shows a high overall success rate, causing degradation of more than 85% of the assignments for all five models (example in figure 1 ). we find that high variations in solution generations strongly correlate with high success rates. our user study with undergraduates shows that the average efficacy dropped from 15.43% to 15% when perturbations were noticed. it also suggests that subtle perturbations, i.e., substituting tokens or removing/replacing characters, when unnoticed, are likely to retain high efficacy in impeding actual solution generation. additionally, the detectability of a high-change perturbation might not imply reversion. the implication is that under perturbations, students have to check and modify llm solutions rather than adopt them unchanged -instructors can use these perturbations when preparing homework problems to reduce cases where students do not learn but use chatgpt as is. the goal of this evaluation is to answer the following question: how do llms perform on our corpus of programming assignment problems? what problems are more amenable to llm-assisted cheating? ## exploring perturbations (step 2) in this section, we explore the following research question: how can we leverage black-box adversarial perturbation techniques to impede llmassisted solution generation? towards that end, following existing literature, we design several perturbation techniques and measure their efficacy on the assignments that llms solved with non-zero correctness scores. for a given perturbation technique, we define its efficacy as follows. definition 1 (efficacy) the efficacy of a perturba-tion technique for a given assignment is the reduction of the llm's correctness score from the base correctness score on the assignment. ## field experiment (step 3) in this step, we aim to understand how students would detect and reverse our perturbations. this would provide valuable insights into the potential of the perturbation techniques for impeding actual llm-assisted cheating. ## conclusion high-performant llms pose a significant threat to enable cheating on introductory programming assignments. it investigates the potential of adversarial perturbation techniques to impede llm-assisted cheating by designing several such methods and evaluating their efficacy in a user study. the result suggests that the combination of the perturbation indeed caused a 77% reduction in the correctness of the generated solutions, showing early promises. our perturbations show positive results, but they might only be effective temporarily. future techniques, including rigorous training data and protective layers in the prompting pipeline of llms, could counter these results. we hope our study will inspire ongoing efforts to prevent the misuse of llms in academic settings.