S tudent E val: A Benchmark of Student-Written Prompts for Large Language Models of Code (2024)

Abstract: Code LLMs have the potential to make it easier for non-experts to understand and write code. However, current CodeLLM benchmarks rely on a single expert-written prompt per problem, making it hard to generalize their success to non-expert users. In this paper, we present a new natural-language-to-code benchmark of prompts written by a key population of non-experts: beginning programmers. StudentEval contains 1,749 prompts written by 80 students who have only completed one introductory Python course. StudentEval contains numerous non-expert prompts describing the same problem, enabling exploration of key factors in prompt success. We use StudentEval to evaluate 12 Code LLMs and find that StudentEval is a better discriminator of model performance than existing benchmarks. Our analysis of student prompting strategies reveals that nondeterministic LLM sampling can mislead students about the quality of their descriptions, a finding with key implications for Code LLMs in education.

## introduction large language models trained on code (code llms) have the potential to democratize programming by enabling less-experienced programmers to write code in natural language. a growing body of work shows their utility to professional programmers @xcite @xcite , but to broaden the accessibility of programming, models must also work well for non-experts. code llm benchmarks @xcite @xcite @xcite are largely modelled after experts, both in the choice of task, and by having professionals write benchmark prompts. achieving good performance on these benchmarks indicates that a model will perform well if the user can write prompts equally as well as the expert. our goal is to facilitate research on how to better align code llms with non-expert programmers, who may talk about code differently than experts. towards this goal, we present studen-teval, a dataset with 1,749 prompts written by beginning cs students and validated with expertwritten test cases. prompts were collected using 48 beginner-appropriate problems, with numerous different prompts for each problem. our prompts exhibit the variation in technical vocabulary and lack of familiarity with how to describe code that are common with non-experts. while other work explores code llm use in classrooms @xcite @xcite , studenteval is the first benchmark based on student interactions. it differs from existing benchmarks in three key ways: 1) other benchmarks have prompts authored by experienced programmers, whereas stu-denteval has prompts authored by students who have only completed one computer science course. our key contributions are: ## background although code llms allow both code and natural language prompting, we focus on the naturallanguage-to-code task. @xcite report that this is a popular and effective strategy for code llm use among experts; it is also more accessible to non-experts than code prompting. existing benchmarks pair natural language descriptions of code with test cases to check the validity of generated programs. the two most widely used benchmarks, humaneval @xcite and mbpp @xcite , are in python. there are also multi-language benchmarks that translate problems from one language to another @xcite . finally, there are alternate formats, including multi-turn evaluation @xcite and docstring generation @xcite . general-purpose benchmarks most existing benchmarks have a single natural language description of a problem, typically written by an expert programmer. there are a few exceptions that scrape the web or crowdsource @xcite @xcite , but expertwritten benchmarks predominate. these benchmarks provide wide coverage, but come with limitations. first, they have a single prompt per problem. consider this humaneval prompt: imagine a road that's a perfectly straight infinitely long line. n cars are driving left to right; simultaneously, a different set of n cars are driving right to left. the two sets of cars start out being very far from each other. all cars move in the same speed. two cars are said to collide when a car that's moving left to right hits a car that's moving right to left. however, the cars are infinitely sturdy and strong; as a result, they continue moving in their trajectory as if they did not collide. this function outputs the number of such collisions. while the correct solution is simply n 2 , the prompt is designed to be confusing. models succeed or fail based on this specific phrasing. having a single prompt precludes explorations of how crucial word choice, grammar, etc. is to model success. studenteval's non-expert, multi-prompt construction enables us to analyze what makes a successful prompt: each problem has at least 14 prompts that describe the task in a different way. second, existing benchmarks contain problems at widely varying difficulty levels. compare the problem above, which requires mathematical reasoning that may challenge many programmers, with a trivial problem from the same benchmark @xcite : return length of given string. although these benchmarks cover a wide range of programming tasks, it is difficult to interpret their results as evidence that a model will or won't suit a particular group of programmers, since they aggregate over very different skill levels. ## the studenteval dataset in this section we describe studenteval, a many-prompt-per-problem benchmark that targets a specific programmer skill level. the dataset consists of 1,749 english-language prompts for 48 programming problems, with at least 14 prompts per problem. all prompts were written by university students who had completed a single semester of computer science in python (cs1). these students represent a population of programmers with a uniform knowledge base, allowing us to choose problems that they all should be able to solve. ## results we evaluate 12 code llms. we focus our comparison on gpt-3.5-turbo, the three "python specialist" code llama models @xcite , the four starcoderbase models @xcite , and phi-1 @xcite . appendix h presents results for several other models. we confirm that none of the studenteval prompts appear in the stack, the open training dataset for starcoderbase and other models. as with other benchmarks, we use hidden unit tests to evaluate the correctness of model-generated code. to account for their nondeterminism, we use the standard pass@1 metric @xcite , which estimates the probability that the code llm produces a solution that passes all hidden unit tests in one shot, calculated over 200 samples. ## conclusion we present studenteval, a large code llm benchmark of prompts written by students who have taken a single cs1 course. a key feature of studenteval is the numerous natural language descriptions per problem, written by beginning programmers with varying levels of prompting skill. we show that larger models are more capable of following student-written instructions than smaller models. we also find that many student-written prompts are unreliable (have low pass@1): students get lucky (or unlucky) when using code llms, an issue for educational use of code llms. finally, we investigate the question of what makes a good prompt from several angles, finding that models struggle to understand some valid strategies, such as giving examples of complex data. we hope that studenteval will make it easier to evaluate how well code llms work for natural language instructions written by non-experts, leading to the development of models that are better aligned with this key group of users. finally, this research was only possible due to model access and funding. to obtain the benchmark results from the 12 llms, we used around 2 weeks of gpu time on an h100 gpu. there are ongoing ethical concerns about access to models and infrastructure. the evaluation of the dataset in this paper centers both open-source and small-scale models, but fully addressing these issues should be a priority for the broader community.