Substance Beats Style: Why Beginning Students Fail to Code with LLM s (2025)

Abstract: Although LLMs are increasing the productivity of professional programmers, existing work shows that beginners struggle to prompt LLMs to solve text-to-code tasks (Nguyen et al., 2024; Prather et al., 2024b; Mordechai et al., 2024). Why is this the case? This paper explores two competing hypotheses about the cause of student-LLM miscommunication: (1) students simply lack the technical vocabulary needed to write good prompts, and (2) students do not understand the extent of information that LLMs need to solve code generation tasks. We study (1) with a causal intervention experiment on technical vocabulary and (2) by analyzing graphs that abstract how students edit prompts and the different failures that they encounter. We find that substance beats style: a poor grasp of technical vocabulary is merely correlated with prompt failure; that the information content of prompts predicts success; that students get stuck making trivial edits; and more. Our findings have implications for the use of LLMs in programming education, and for efforts to make computing more accessible with LLMs.

## introduction there is a growing body of evidence that large language models (llms) are increasing the productivity of professional programmers @xcite . at the same time, previous work shows that students struggle to leverage llms in programming across a variety of tasks and models @xcite @xcite . but why is this the case? prior work has reported on students' and instructors' perception of why student-llm interactions go wrong, positing many explanations including unfamiliarity with technical vocabulary @xcite @xcite , model non-determinism @xcite , and trouble understanding llm output @xcite . however, there is little quantitative evidence about these potential sources of miscommunication. in this paper, we test two competing hypotheses about the cause of student-llm miscommunication. one possibility is that students provide all of the information that the model needs, but use language that models cannot understand. non-expert programmers talk about code differently than experts, leading to problems for models trained largely on expert code. a second possibility is that students do not understand what information a model needs to solve a given problem. writing prompts involves decisions about what information the model may be able to infer from pretraining versus what information must be stated directly in the prompt. these decisions may be more challenging for students to make, since they do not yet have a strong sense of what information code typically contains. this paper tests the impact of these potential error sources in two sets of experiments on a dataset of 1,749 prompts authored by 80 students @xcite . to isolate the effect of linguistic variation, we conduct a causal analysis of lexical choices for technical terminology by replacing them with near-synonyms used by students. to study information selection, we annotate series of prompts in student problem-solving attempts with problemspecific "clues," or information that describes the intended behavior of generated code. overall, our findings reveal that student-llm coding difficulties spring from challenges in selecting relevant information rather than challenges with technical vocabulary. our study of the information content of prompts shows that prompts with missing clues almost always fail. moreover, students typically get "stuck" in cycles because they make trivial edits to prompts instead of changing their information content. our causal analysis of prompt wording finds relatively weak effects of modifying technical terminology. although certain substitutions can hurt prompt success rates, correcting non-standard terminology rarely improves them. this suggests that the relationship between technical vocabulary and prompt success is more correlational than causal. taken together, our results provide empirical evidence that the information content of student prompts is what matters, rather than their (mis)use of technical vocabulary. these findings have strong implications for the use of llms in programming education and, more broadly, for efforts to broaden the accessibility of computing with llms. 1 ## related work as the use of llms for programming has become widespread, the question of prompt wording has become increasingly important. early work revealed high sensitivity to prompt wording on programs @xcite , which has efficiency implications @xcite . several techniques that address prompt wording @xcite @xcite . @xcite take a user-centered approach to teaching strategies for prompting. @xcite novice programmers and llms. llms have have sparked much discussion in computing education @xcite . there is a growing body of work studying how students use llms in computing classes @xcite @xcite @xcite . a convergent finding is that students struggle to leverage llms. many potential explanations have been advanced: lau and guo (2023)'s study of cs educators discusses model non-determinism as a barrier; @xcite explores the cognitive load imposed by code suggestions; and @xcite describe over-reliance on the model. finally, multiple studies posit that technical language is a barrier between students and llms @xcite @xcite . this paper uses the dataset by @xcite , which contains 1,749 prompts from students who have completed one college programming course. @xcite turn their dataset into a benchmark to measure llm performance on novicewritten prompts. they report some correlations between technical terms and prompt success. @xcite study student experiences during the experiment, including students' self-perceptions of why the task is challenging: they highlight prompt wording as a key student-perceived barrier. this is reaffirmed in feldman and anderson (2024)'s replication with students with no coding experience. ## dataset our goal is to understand what it is about studentwritten prompts that makes them less effective for llm code generation. we use the studente-val dataset released by @xcite , who use a subset of their data to benchmark llms for code generation. unlike many datasets of programming prompts, it contains many different prompts per task, including multiple submissions by the same author, allowing us to explore both wording choices and how the information content of total_bill ( @xcite ] , 0.5) # 10.5 docstring attempt 1 (generated code fails some tests) you will have two inputs a list of lists and the tax rate. for every list in the list of lists multiply the second and third item and add all of them and then multiply that by the sales tax plus 1 docstring attempt 2 (generated code passes all tests) you will have two inputs a list of lists and the tax rate. for every list in the list of lists multiply the second and third item and add all of them and then multiply that by the sales tax plus 1. if the resulting number has more than two decimal places shorten it to two decimal places. figure 1 : an example problem that a student solves in two attempts. given the function signature and tests, they write the first docstring. the platform prompts the model to generate the function body from the function signature and docstring (not the tests), and then tests the generated code. from the failed tests, the student realizes that the model needs to be told to round to two decimal places. they add this clue in the second prompt, which succeeds. we use different subsets of the studenteval dataset to explore our research questions. to study the effect of information content on prompt success, we consider problems where at least five students submitted multiple times (33 tasks). we exclude tasks that were trivial (all students succeeded at first try) or that were attempted by few students, since these are uninformative. to study the effect of prompt wording, we select a lexically diverse subset by taking each student's first and last prompts per problem (953 prompts). we study only the first and last prompts because there is often little lexical variation in intermediate prompts (as shown in our information content experiment), which would skew the results of our lexical analysis. 2 the dataset contains sequences of prompt-edits, but their benchmark uses only the first/last prompt by each student. ## methods our work explores the impact of two potential causes of student-llm miscommunication: how students word their prompts, and how students select information to include in their prompts. ## results: style rarely matters we measure the effect of prompt wording through a causal intervention experiment in which we explore a range of lexical substitutions for terms referring to 12 key programming concepts. if what hinders students is their lack of fluency with technical vocabulary, we should be able to improve the pass rate of their prompts by substituting more precise technical vocabulary for their unconventional ways of referring to these concepts. we also measure the effect of word choice on high-quality prompts: by including substitution terms that are commonly ## results: substance matters an alternative hypothesis about student-llm miscommunication is that students struggle to select the right information for models. we explore this using prompt trajectory graphs ( §4.2) for codedavinci-002 to understand prompt editing. what kinds of edits to information content do students make, and how do they effect the success of their prompts? we focus our discussion on high-level trends; appendix f contains graphs for each studied task. ## conclusion by investigating two commonly espoused concrete hypotheses about why students struggle to effectively prompt llms for code, our work sheds light on what it means for students to write "good prompts." our results suggest that it is the (lack of) information in prompts, rather than how the information is communicated, that causes student-llm miscommunication. although these findings imply that attempts to help student prompters by suggesting alternative wording are unlikely to be very useful, by providing the first empirical evidence of the source of student struggles, we hope our findings will guide future work on teaching prompting towards more impactful interventions.