A Static Evaluation of Code Completion by Large Language Models (2023)

Abstract: Large language models trained on code have shown great potential to increase productivity of software developers. Several execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. Nevertheless, it is expensive to perform the same evaluation on complex real-world projects considering the execution cost. On the other hand, static analysis tools such as linters, which can detect errors without running the program, haven’t been well explored for evaluating code generation models. In this work, we propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees. Compared with execution-based evaluation, our method is not only more efficient, but also applicable to code in the wild. For experiments, we collect code context from open source repos to generate one million function bodies using public models. Our static analysis reveals that Undefined Name and Unused Variable are the most common errors among others made by language models. Through extensive studies, we also show the impact of sampling temperature, model size, and context on static errors in code completions.

## introduction automatic code completion by large language models trained on numerous code repositories has demonstrated great potential in accelerating software development. code assistant services powered by these models provide developers with code suggestions following the current context in realtime. however, it has been shown that about 70% of the suggestions are discarded by users in a recent study @xcite . even worse, misleading recommendations can lead to failure in completing programming tasks @xcite . therefore, it is important to understand the weakness of current code generation models through comprehensive evaluation and analysis. recently, execution-based evaluation has become increasingly popular, where model-generated code is executed with unit tests to check functional correctness. several benchmarks have been proposed along this direction, such as humaneval @xcite , mbpp @xcite , mbxp @xcite , codecontests @xcite , and ds-1000 @xcite . although these benchmarks are highly reliable and accurate, they only focus on well-defined algorithmic and data science problems, which do not reflect the need in general software development. running execution-based evaluation with real-world codebases is, however, prohibitively expensive because each project requires a different setup and the computation cost is potentially unbounded. in contrast to the execution-based approach, static program analysis (or static analysis) can analyze programs without executing them. although static analysis is usually unable to determine functional correctness, it covers a large collection of static error types, such as undefined names or unused variables that are illustrated in figure 1 . more importantly, the analysis can be very fast and does not require any project specific environment setup, which allows us to evaluate model completions for complex real-world code at large scale. static analysis tools such as linters have been widely used, for example in code editors, to examine human-written code, but their value in evaluating code generation models has not been well explored yet. in this work, we propose a static evaluation framework for python language. code snippets are first parsed into abstract syntax trees (asts) and then analyzed by pyflakes 1 , a popular static analysis tool for python. to simulate real-world use cases of auto completion, we collect code from public github repositories to build a function completion dataset of 100k problems. in each problem, we randomly mask out a function body in a python file and ask the model to complete it given the preceding context up until the function header. we then evaluate public models by sampling 10 completions for each problem, resulting in one million generations for each model and sampling temperature, which will be examined by our static evaluation pipeline. during ast parsing, we find most of the errors arise from incomplete generations that hit the max length limit. otherwise, models of all sizes perform quite well in producing parsable codes. moving forward, pyflakes analysis reveals that undefined name and unused variable are the most prominent static errors in model-generated code. we also observe higher temperatures consistently lead to more errors. scaling up the model, while able to reduce errors of many types, do not show a clear benefit for preventing undefined names. through a more fine-grained classification, we find larger models generate fewer undefined variables but more undefined methods, which add up to a mixed result. finally, we demonstrate that errors in context can lead to errors of the same type in generation, which is likely a consequence of large language models' in context learning capability. in summary, our main contributions include the following. (1) we propose a static evaluation framework for code completion. (2) our evaluation on public models reveals common static errors and how they are impacted by various factors such as temperature, model size, and context. abstract syntax tree an abstract syntax tree (a.k.a., ast) is used to represent a source code in a concise tree form. by discarding unnecessary details of the underlying code and its corresponding parsed tree, ast only presents the main structural content of the source code following the language grammar @xcite . static analysis static analysis is a common way to detect software bugs without executing the program @xcite @xcite . static analyzers tend to detect bugs by analyzing the static code text, its ast, documentation, etc. the users usually need to specify the error patterns and static analyzers use different ast, graph, and path analysis to find those patterns in the code. there are a plethora of static analysis tools and they can detect a wide range of errors depending on the specified patterns @xcite . for example, linter is a popular tool that checks for coding style errors and thus, tries to enforce a coding standard @xcite . left: we parse [context] and [context + generation] into asts. if [context] is not parsable, we stop without reporting any error on generation. if [context] is parsable, but [context + generation] is not, we report the ast error in generation. right: if both are parsable, we run pyflakes on the trees, which reports errors in [context] and errors in [context + generation]. taking the difference gives us errors in generation. software developers use code generation models as black-box services on a diverse set of coding projects. to better simulate the real-world scenario, we build an evaluation set by sampling from public github repositories. specifically we collected permissively licensed python code in repositories that were created between april, 2022 and august, 2022. the selection criterion precludes any chronological overlap between our evaluation data and the training data of models to be tested in this work. 2 the collected python codes are reformatted as function completion problems. we first use treesitter foot_2 to parse the whole file to identify all the functions. then a function that contains a docstring is randomly selected. the code from the beginning of the file up until the end of the docstring is used as the context, and the function body is considered as the groundtruth. the rest of the file is discarded. at test time, we prompt the model with the context part as input, and let the model generate the function body. we choose only functions with docstrings so that context is well-defined and model can generate meaningful code completions. we further select test samples whose context length is between 64 and 768 tokens, and groundtruth length is shorter than 256 tokens, to match our model generation setting. our final evaluation set consists of 100k function completion problems. ## static error analysis we propose an evaluation pipeline to detect errors in function completions generated by models, illustrated in figure 2 . suppose the model generates a completion x given the input context c. we cannot directly analyze x which is partial code without context. meanwhile, c may also contain errors especially in real-world cases. therefore, we perform our analysis in two passes. we first check c for any errors in the input that need to be excluded, and then do another pass on the full code (c, x), the concatenation of the context and model completion. any error that is identified in (c, x) but not in c must arise from x, or in other words, be generated by the model. more specifically, we conduct the following two steps of analysis for python code. ## experiments with the proposed pipeline we conduct error analysis for codegen models @xcite on the test set described in section 3, and present the analysis results. ## discussion we present a static evaluation framework for code completions generated by large language models. by utilizing the proposed framework, we conduct error analysis of codegen models on a large scale real-world python evaluation set. our experiment reveals common static errors made by pretrained models, as well as their frequency trend across model sizes and sampling temperatures. by pointing out weaknesses of existing models, we hope our study also sheds light on future directions towards more accurate code generation. there are a few limitations of this study. first, we focus on left-to-right code generation without considering right-side and cross-file context, which can be used to determine broader categories of errors with improved precision. second, each static analysis tool has its own limitations. thus, the presented analysis is limited by pyflakes's accuracy and coverage to detect certain code issues.