Testing the Effect of Code Documentation on Large Language Model Code Understanding (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive abilities in recent years with regards to code generation and understanding. However, little work has investigated how documentation and other code properties affect an LLM’s ability to understand and generate code or documentation. We present an empirical analysis of how underlying properties of code or documentation can affect an LLM’s capabilities. We show that providing an LLM with “incorrect” documentation can greatly hinder code understanding, while incomplete or missing documentation does not seem to significantly affect an LLM’s ability to understand code.

## introduction recently, large language models (llms) have approached or pushed the state of the art for multiple natural language processing (nlp) tasks and benchmarks such as machine translation (mt) @xcite , human evaluation of mt @xcite , sentence completion, and question answering @xcite . the same can be said for some programming language processing (plp) tasks, such as code generation (openai, 2023) and code translation @xcite , but other tasks, such as code summarization, have proven to be difficult for llms @xcite . these tasks requiring code understanding pose challenges distinct from those faced in natural language. as an example, a unique challenge in plp is the rigidity of syntax and semantic precision required in generation or translation that is not required to the same degree in nlp. solving these problems, or at least improving their solutions, has the potential to greatly increase productivity and satisfaction in software development, as * approved for public release; distribution unlimited. public release case number 23-4132. copyright ©2024 the mitre corporation. all rights reserved. while several works have shown the importance of applying effective prompting strategies for plp with llms @xcite @xcite , very little work has investigated how the reliability of the documentation at input can affect an llm's performance. in this work, we provide a preliminary empirical analysis of how documentation quality can influence this ability. we hypothesize that correct code documentation will improve an llm's code understanding, and that its understanding will decrease as the prevalence and accuracy of the documentation is decreased. to the authors' knowledge this is the first work to consider code documentation reliability for this problem. the rest of the paper is organized as follows: first we discuss other works and how they relate to our study. then we present experimental analysis and results. our experiments provide a basic analysis of how an llm's code understanding performance degrades along with documentation quality and quantity, and we end with a discussion and analysis of the results. ## related works code analysis is a very well-studied problem. traditional compilers and syntax-tree parsers are commonly used to perform code analysis @xcite or extract code metrics @xcite @xcite . however, these tools lack an llm's ability to process natural language, and therefore are incapable of extracting semantic understanding or considering code documentation. since llms have become widely available, many works have also attempted to leverage language models to analyze code. several works have sought to create llms designed for code understanding tasks such as code generation, code completion, program repair, and code translation @xcite @xcite . unlike these works, instead of developing new tools we provide a rigorous analysis of where and how existing tools can best be leveraged. our work also introduces other means of determining code understanding than the benchmarks developed in @xcite . other works provided empirical analyses of existing models' abilities to analyze code. xu et al. (2022) present an analysis of the performance of various llms on different code benchmarks, and introduce a new llm. ma et al. (2023) present an empirical analysis of using gpt to generate syntax trees, call graph and other syntactic and semantic representations of code. they conclude that gpt has approximately the same abilities as an abstract syntax tree (ast) parser. similarly palacio et al. (2023) introduce a framework called astxplainer that uses questions about an ast as an evaluation criteria for llms. leinonen et al. (2023) provide an empirical study comparing human generated explanations with those generated by gpt. all of these works are focused on evaluating an llms performance in general and, unlike this work, none investigate how code content or documentation can effect an llm's understanding. ## llm code understanding it has been argued that understanding is merely the "knowledge of causes" @xcite , but one can see that this is an incomplete definition of what it means to truly understand something and that "a proper explanatory grip on how cause and effect are related" @xcite is required for true understanding. nevertheless, the knowledge of causes is at least a crucial early step in developing a complete understanding of a topic. if this is the basic definition we are using for our basis of determining understanding, then to truly assess whether a piece of code is understood by a person or a language model, then the subject must display knowledge of the cause and effect relationship of the code. or, to put more simply, the subject should be able to answer the question, "what does this piece of code do?" past work has attempted to test an llm's ability to answer this question through code translation by measuring errors in the target language @xcite ; through code summarization by measur-ing differences in bleu, meteor, or rouge-l scores between an llm-generated summarization and the reference summarization @xcite ; and through the generation of unit tests by measuring statement and branch coverage across various javascript libraries @xcite . our approach for testing an llm's understanding of source code is most closely related to the lattermost method, but differs by not improving upon the initial attempt at unit test generation iteratively if the tests fail. in many cases, a program can be represented as a mapping from inputs into outputs @xcite . for example, the program square(n) can be described with square(1) = 1, square(2) = 4,... etc. unsurprisingly then, how well an individual can predict input-output pairs can serve as a very useful surrogate metric for how well an individual understands a piece of code. after all, someone cannot truly understand software without being able to predict the software's behavior. module documentation, function docstrings, inline code comments, and variable names provide hints or detailed descriptions of what the user can expect of these input-output pairs, and often allow human users to better predict that behavior. modern unit test frameworks provide both a formal language for describing input-output pairs and an easily automated method of checking an llm's output. in order to measure an llm's ability to generate these input-output pairs, we therefore task the llm with generating unit tests for a piece of software while varying the quality or quantity of the documentation at input, as well as other code properties. we then measure the percent of unit tests that pass vs the number that fail or produce an error as the percent of correctly predicted inputoutput pairs. ## experimental setup in the following section we describe our experimental process for testing an llm's ability to understand code well enough to produce successful unit tests. in particular, we present an experimental setup for testing the generation of unit tests under varying initial documentation conditions. to test an llm's ability to understand a piece of software, we task the language model to generate unit tests. we then run the unit tests and record each test as 1 of 3 results: runtime error, where a unit test crashed before it could finish running; failure, where a unit test ran but failed to pass; and success, where a unit test ran and successfully passed. all llms used their default parameters during experiments. we use all 164 humaneval (chen et al., 2021) ground truth solutions as our basis for generating unit tests. 1 we chose humaneval both for its common use as an llm benchmark for code understanding @xcite @xcite (although intended for text-to-code generation), and for its wide variety of self encapsulated functions that can be easily run within unit test frameworks. we perform several post processing steps on the llm generated code to ensure it can run automatically. first, we remove all import statements that result in an error (the llm will commonly import hallucinated modules as it was not provided the original module names at input). second, we append the llm's code to a file with the humaneval function to ensure the unit test has access to all code pieces required to run. we generate several automatic variations of each function in the humaneval dataset. base file, which includes only the ground truth solution for a humaneval function with no docstring comments. comments, which is the base file but includes any comments and the docstring provided for the hu-maneval function. the docstrings for humaneval functions typically contain a description of the function along with a few execution examples. random comments which has the base file but with a random comment or docstring from a different humaneval function, animal variable names, which is the base file but with all variable and (non-external) function names replaced with animals (e.g. bird, cat, dog etc.), and random variable names which is the base file but with all variable and function names replaced with random strings. for instance, a variable format_str might be changed to embafsd. in addition, we also investigate how removing random portions of the docstrings, line-by-line, can effect performance. each line was kept with uniform random probability. we evaluate unit tests generated from 10%, 25%, 50% and 75% of the original docstrings remaining. to avoid any infinite loops generated by the llm, all tests would automatically fail after taking more than 10 seconds to run. figure 1 shows an example of a function from the comments category. ## results figure 2 shows the results of gpt-3.5 (gpt-3.5turbo) and gpt-4 generating unit tests on the variations of the humaneval dataset discussed above. 2 there are several key things to note from these results. first the number of runtime errors produced by gpt-3.5 is much greater than those produced by gpt-4, as is consistent with many results for similar code understanding tasks (openai, 2023). a manual examination of the generated test cases shows that gpt-3.5 often generated simple assert statements rather than using pytest, and many of the runtime errors were caused by assert statements failing. second, we note that the random comments scenario performs worse than any other scenario on both gpt-3.5 and 4 (only 22.1% and 68.1% successes respectively) as shown in figure 2 , confirming our hypothesis that incorrect documentation can hurt an llm's understanding of a piece of code. a statistical bootstrap shows that random comments have a higher proportion of runtime errors and failed tests than any other version of the code by a statistically significant margin with α = 0.05. thirdly, we note that changing the code content (via variable names), had a relatively minor effect on the llm's code understanding when compared with the base file. a bootstrap shows that changing variables to random names did not impact the proportion of errors and failed tests by a statistically significant margin, but changing the variables to animals did cause a statistically significant change (albeit a small one of 44.7% to 40.6% successes for gpt-3.5 and 78.5% to 76.6% for gpt-4). finally, having comments did not significantly increase the llm's ability to understand the code. again a bootstrap confirms the proportion of errors and failed tests did not change between the base file with/without comments by a statistically significant margin. as mentioned above, we also conducted experiments to determine how partial comments might affect an llm's ability to understand code. figure 3 shows these results. gpt-3.5 shows that providing partial comments causes an initial spike in the number of errors, before declining as the percentage of comments kept increases. gpt-4 does not show the same pattern, however a bootstrap shows that having the full amount of comments gave fewer errors over when the proportion of kept comments was 10%, 25% and 50% by statistically significant margin. as shown in the figure, these changes were relatively minor. no other comparisons were statistically significant for gpt-4. we therefore cannot currently conclude whether or not partial comments can significantly hurt an llm's code understanding based on the current evidence. finally, we conducted an analysis of line coverage by unit tests to prevent the possibility of the llm producing trivial tests to gain a higher success rate. this is shown in figures 4 and 5 . first, it is worth noting that the results with random comments had lower average code coverage than all other methods by a statistically significant margin. second, we note that for both gpt-3.5 and gpt-4, code with comments generated a higher average line coverage than all other methods by a statistically significant margin. thirdly, we note that modifying the variables names of the code largely did not have a statistically significant effect. the one exception was the animal names category generated by gpt-3.5, which had an average amount of line coverage that was significantly lower than the line coverage of unit tests generated from the baseline, as seen in figure 4 . finally, partial docstrings mostly did not have a statistically significant effect on the average amount of line coverage. there was one exception when 10% of the docstring was kept for gpt-3.5, which did worse than the baseline by a statistically significant margin. 0 0.1 0.25 0.5 0.75 1 proportion docstring kept 0.0 0.2 0.4 0.6 gpt-3.5 0 0.1 0.25 0.5 0.75 1 proportion docstring kept gpt-4 test failures runtime errors figure 3: proportion of runtime errors or failed tests that happen with gpt-3.5 (left) and gpt-4 (right) generating unit tests on different proportions of docstring lines kept on humaneval code. we also provide the number of unit tests generated by models for various conditions. ## conclusions and discussion in this paper, we introduce the effect of code documentation on llm code understanding and show in our initial experiments that the relative prevalence of documentation has little to no significant effect on an llm's understanding as we have defined it. this is a little complicated by the fact that we found no significant difference in the amount of successful unit tests but did see a significant difference in the code coverage of those unit tests. even when comparing commented code to code without comments and all of the variable names changed to random characters, we find little to no significant difference in the llm's ability to generate successful unit tests. so, although more of the code with comments is being covered by the unit tests than the code without, this does not improve the model basefile comments random comments random names animal names gpt-3.5 1323 883 892 1464 1482 gpt-4 3832 3295 2324 3725 3589