Can docstring reformulation with an LLM improve code generation? (2024)

Abstract: Generating code is an important application of Large Language Models (LLMs) and the task of function completion is one of the core open challenges in this context. Existing approaches focus on either training, fine-tuning or prompting LLMs to generate better outputs given the same input. We propose a novel and complementary approach: to optimize part of the input, the docstring (summary of a function’s purpose and usage), via reformulation with an LLM, in order to improve code generation. We develop two baseline methods for optimizing code generation via docstring reformulation and test them on the original HumanEval benchmark and multiple curated variants which are made more challenging by realistically worsening the docstrings. Our results show that, when operating on docstrings reformulated by an LLM instead of the original (or worsened) inputs, the performance of a number of open-source LLMs does not change significantlyThis finding demonstrates an unexpected robustness of current open-source LLMs to the details of the docstrings. We conclude by examining a series of questions, accompanied by in-depth analyses, pertaining to the sensitivity of current open-source LLMs to the details in the docstrings, the potential for improvement via docstring reformulation and the limitations of the methods employed in this work.

## introduction large language models for coding (code llms) emerged in recent years as the dominant approach to code generation @xcite @xcite . the research community proposed various benchmarks to systematically evaluate the code generation abilities of llms @xcite @xcite . one of the grand challenges of code generation is the task of function completion: given its definition and an explanation of the desired behaviour, write the body of the desired function. current methods in code generation focus on multiple directions: better pre-training datasets and procedures for more performant base models @xcite , specific finetuning methods for coding models @xcite @xcite , test-time improvements via self-testing @xcite , solutions reranking @xcite , self-repair @xcite or other techniques @xcite @xcite . in this work we explore the idea of input transformations as a new, complementary approach to improving code generation abilities of llms. in the context of function completion tasks, the input can be formulated as a function declaration followed by a docstring, which explains in words what the function is supposed to do. as illustrated in figure 1 , we consider a specific input transformation for this task: docstring reformulation. the motivation for this approach is the following: first, optimizing the docstrings could provide benefits that are model-agnostic, that is, benefit all code llms. we hypothesize that this could be the case, drawing inspiration from the success of prompting techniques, such as chain-of-thought @xcite and similar works @xcite , that demonstrate efficacy across a broad class of models. second, optimizing docstrings, while maintaining their readability, can provide interpretable insights on the influence of docstrings' properties on code llms in the context of code generation. this work makes the following contributions: evolutionary methods evolutionary methods involving language models have also been explored. @xcite expand the self-instruct method by @xcite with instruction evolution, employed in the creation of high-quality instructiontuning datasets. @xcite investigate the use of language models as variation operators in evolutionary algorithms, enabling tasks such as symbolic regression and sentiment modification. @xcite combine evolution through large models with map-elites @xcite to generate functional examples of python programs in the sodarace domain, a task unseen during pre-training. chen et al. (2023a) focus on producing valid neural network architectures for neural architecture search using llms. ## methodology in this section, we first introduce the function completion task as formulated in benchmarks such as humaneval, then we formalise the docstring reformulation task and finally we present our baseline methods for optimising docstring reformulations. ## discussion and conclusions in the following section we raise a series of questions about the docstring reformulation framework, the sft and opro methods and the experimental setup. we address these questions with ulterior argumentations and analyses, before drawing the conclusions from this study. a more extended discussion is presented in appendix a. q1. limitations of docstring optimization for code generation: how capable are the code llms considered in this work to leverage docstrings' improvements? we use the following working definition of docstring improvement: an increase in the information that the docstring contains about the body of the function to be completed. this definition is model-agnostic, as it does not make reference to the performance of any model; rather, we expect that llms can leverage the increased information in the docstrings for better code generation. first, we show in then, we compare the performance of the various llms when evaluated on the faulty variants of humaneval versus the original dataset, which can be considered an improved version of them. we find that five out of six models surprisingly increased performance on at least one of the four faulty variants of humaneval, indicating that a docstring improvement does not necessarily benefit code generation and it can even hurt performance. finally we evaluate the coder models on two new sets of strongly improved docstrings for the hu-maneval problems, produced while having access to the ground-truth function completions (oracle docstrings). the first set of docstrings is produced by gpt-4 foot_0 with access to the ground-truth solution ('oracle hints'), asking the model to give detailed hints on how to implement the function. the second set contains the true body of the function to be completed ('oracle solutions'), so that the task of the coder models simplifies to copy-pasting the solution. our results, presented in figure 3 : performance of different docstrings. we report the pass@1 averaged across the six coder models for different kinds of docstrings: no docstrings at all ('no docs'), faulty docstrings ('faulty (avg.)', average across misspelling, ambiguity and distractor variants), original docstrings ('original'), oracle docstrings with hints ('oracle hints') and with solutions ('oracle solutions'). figure 3 summarises our findings on the ability of the considered llms in leveraging docstrings for code generation: coder models may not reliably leverage small improvements in the docstrings, but clearly benefit from the overall information included in them. in particular, the performance with oracle hints can be considered as a good estimate of the potential of optimizing code generation via docstring reformulation; how to achieve such performance without access to the ground-truth solution to generate hints remains an open question.