Aligning Offline Metrics and Human Judgments of Value for Code Generation Models (2023)

Abstract: Large language models have demonstrated great potential to assist programmers in generating code. For such human-AI pair programming scenarios, we empirically demonstrate that while generated code are most often evaluated in terms of their functional correctness (i.e., whether generations pass available unit tests), correctness does not fully capture (e.g., may underestimate) the productivity gains these models may provide. Through a user study with N=49 experienced programmers, we show that while correctness captures high-value generations, programmers still rate code that fails unit tests as valuable if it reduces the overall effort needed to complete a coding task. Finally, we propose a hybrid metric that combines functional correctness and syntactic similarity and show that it achieves a 14% stronger correlation with value and can therefore better represent real-world gains when evaluating and comparing models.

## introduction large language models trained on code (e.g., codex @xcite , alphacode @xcite , codegen @xcite , incoder @xcite ) have shown impressive capabilities on code generation tasks. one important application for such models is human-ai pair programming, where a model suggests in-line code completions (e.g., within an ide) that programmers can choose to ignore, accept, or edit as needed. early studies suggest that this paradigm may dramatically boost productivity and transform the practice of software development @xcite . as is common with model development more generally, code-generation advances are largely driven by comparing model performance on offline metrics (i.e., metrics computed automatically over held out evaluation data) that can be easily tracked on leaderboards. functional correctness metrics such as pass@k @xcite currently represent the state-of-best-practice @xcite @xcite @xcite @xcite . these metrics evaluate generations by executing a set of unit tests and assessing whether the generations pass or fail. while functional correctness is clearly important, it does not fully capture the productivity gains programmers may value about code generation assistance. for example, a generation that fails unit tests might yet provide critical hints to solve a task (see example in fig 1 ), or serves as boilerplate that can be adapted with minimal effort. likewise, functionally correct code might be difficult to read or maintain, or may contain other vulnerabilities. with developer productivity in mind @xcite , we investigate syntactic similaritybased offline performance metrics (e.g., @xcite @xcite ) as proxies of programmer effort needed to modify or correct automatic code generations. similarity-based metrics compute how similar a generation is to reference or ground truth code, typically available in the offline setting. we then conducted a user study with @xmath0. our work answers the following key research questions: ## related work offline performance evaluation of ai models typically consists of running models as isolated components over benchmark datasets and then computing aggregate metrics (e.g., accuracy, auc, and precision/recall) that can be easily compared and tracked on leaderboards. while these evaluation practices have led to rapid advancements in ai by enabling efficient apples-to-apples model comparison, a growing body of work has raised concerns about the mismatch between popular metrics and what people need and value in the real world @xcite @xcite @xcite @xcite . using metrics that fail to appropriately capture what people value can result in deploying models that are at best less effective than they could be, and at worst harmful to people and society @xcite @xcite . in this work, we investigate the extent to which common offline code generation metrics capture what professional programmers value about code generation models. in particular, we examine how well existing code generation metrics capture notions of developer effort and productivity @xcite . the current most popular family of code generation metrics is based on measuring functional correctness. functional correctness metrics seek to evaluate generated code against known objective properties such as passing unit tests @xcite @xcite . following the release of codex and the humaneval dataset @xcite -which is a dataset of 164 hand-written problems in python with associated unit tests-the functional correctness metric of pass@k (where k code samples are generated per problem and a problem is considered solved if any of the k generations passes the corresponding unit tests) has emerged as the dominant method for evaluating code generation models (e.g., @xcite @xcite ). advocates of functional correctness metrics argue for their resemblance to programming best practices (e.g, test-driven development) and fidelity to capturing functional behaviour @xcite . however, in this work we demonstrate that functional correctness does not fully capture what people value about code generation models. similarity-based metrics compare tokens from generated code to tokens of known solutions, with code that is more similar to given solution(s) being considered better. multiple similarity-based metrics have been proposed for evaluating code generation models including exact match @xcite , edit distance @xcite , bleu @xcite , @xcite , and rogue @xcite . analyses of similarity-based metrics and other measures of code quality have been mixed (e.g., @xcite vs @xcite ). however, in most of these cases, similarity was considered a proxy for functional correctness. in this work, we revisit similarity-based metrics as proxies for effort saved in coding tasks @xcite and demonstrate how they can be used to better capture value. in this work we focus on pass@k as a proxy for functional correctness and we experiment with two similarity-based metrics, namely, normalized edit similarity @xcite @xcite (which measures how many single-character edits-including insertion, substitution, or deletion-are required to convert generated code to some reference code) and bleu (which measures the token overlap between the generated and reference text) to investigate how these metrics approximate different facets of what programmers value in practice. ## user study we designed a user study to evaluate how well functional correctness-and similarity-based offline metrics approximate value of code generations for programmers. the study showed experienced programmers various programming tasks, together with code generations and reference solutions. programmers then rated the generations on perceived accuracy, effort, and value. ## study results at the end of the study period, we obtained responses from 49 participants. we then applied the following criteria to evaluate the quality of responses: first, we computed the median response time per task for all participants and also computed a performance rating on the code equivalence task in the same way we determined top performers in our study. data from three participants who fell within the bottom 10th percentile of the median task completion times and their performance was worse than the probability of random chance (given the questions they responded to) was excluded from the data analysis. the final dataset ## conclusion we studied how well two types of offline metrics for evaluating code generation models (i.e., functional correctness such as pass@k based on unit tests and similarity-based metrics such as edit similarity) align with human judgements of value when used for human-ai pair programming. our user study with 49 experienced programmers suggests that while programmers find functionally correct code generations valuable, the effort to edit and adapt generations also matters. existing offline metrics show high correlation with human judgements of value, but there is room for improvement. one reason is that while code that passes unit tests is very likely to be rated high-value, code that fails unit tests is often still considered valuable by programmers. based on this observation, we propose a combined offline metric inspired by hinge-loss in support vector machines that allows for partial credit by combining strengths of functional correctness and similarity-based metrics. our analysis shows that this combined metric aligns better with human judgements of value in code generations than functional correctness or similarity alone. overall our work highlights the importance of validating that offline metrics in ai capture what people value and that human-centered metrics, inspired by what people value, can provide better estimates of what people want from their ai-pair programmers.