Exploring Language Model’s Code Generation Ability with Auxiliary Functions (2024)

Abstract: Auxiliary function is a helpful component to improve language model’s code generation ability. However, a systematic exploration of how they affect has yet to be done. In this work, we comprehensively evaluate the ability to utilize auxiliary functions encoded in recent code-pretrained language models. First, we construct a human-crafted evaluation set, called HumanExtension, which contains examples of two functions where one function assists the other.With HumanExtension, we design several experiments to examine their ability in a multifaceted way. Our evaluation processes enable a comprehensive understanding of including auxiliary functions in the prompt in terms of effectiveness and robustness. An additional implementation style analysis captures the models’ various implementation patterns when they access the auxiliary function. Through this analysis, we discover the models’ promising ability to utilize auxiliary functions including their self-improving behavior by implementing the two functions step-by-step. However, our analysis also reveals the model’s underutilized behavior to call the auxiliary function, suggesting the future direction to enhance their implementation by eliciting the auxiliary function call ability encoded in the models. We release our code and dataset to facilitate this research direction.

## introduction program synthesis, i.e., writing function code by taking natural language descriptions as inputs, has garnered attention in the research community @xcite @xcite . with the help of language modeling, several code-pretrained large language models (llms) implement functions with prompts that contain target function signatures @xcite @xcite @xcite . additional code components, e.g., comment lines @xcite , documents @xcite , and other function and class definitions across files @xcite , have been attached to the prompts to boost up their implementation ability. auxiliary function is one promising component to improve their code synthesis ability. we define the auxiliary function as a function that handles a subroutine for the target one or performs an easier version of the actual requirements. when this function is included in the prompt, llms could call the function to delegate their subroutine or refer to their implementation while synthesizing the target function. however, due to the lack of an evaluation dataset that enables a systematic examination of how these auxiliary functions are utilized, no structured analysis has yet to be conducted. in this work, we investigate several llms' abil-ity to utilize auxiliary functions. to do this, we first construct an evaluation dataset, called humanextension, which contains human-crafted examples of two functions that are closely related to each other (figure 1 ). specifically, we guided labelers to extend functions in the humaneval dataset @xcite . we offer software design concepts related to function extension such as subtyping @xcite to promote labelers to create realistic function relationships. additionally, the curated examples are parsed into several components to enable robustness evaluation similar to @xcite . with the humanextension dataset, we conduct systematic analyses to understand how llms leverage auxiliary functions. first, we investigate if appending a single auxiliary function to the prompt enhances the likelihood of accurately implementing the target function. specifically, we design several prompts with auxiliary functions while considering their existence, their functional relevance, and the availability to access auxiliary function implementations. with these prompts, we generate implementations with llms and analyze the model behavior focusing on the auxiliary function's effectiveness, robustness, and the models' implementation style. second, we examine the cases where llms can access multiple auxiliary functions for synthesizing target functions. the randomly sampled auxiliary functions are additionally included in the prompts to verify whether llms can selectively use the appropriate one. similar to @xcite , we inspect whether the position of a relevant function affects their code generation ability. this investigation is combined with the implementation style analysis to permit an in-depth analysis through the lens of the auxiliary function call. our experimental results show current llms' capabilities to utilize auxiliary function and their limitations. first, most llms exhibit large performance improvement with proper relevant auxiliary functions. also, for some advanced llms, our evaluation process sheds light on their selfimproving behavior by implementing the two functions in a step-by-step manner. however, the ability to utilize auxiliary functions is varied depending on the factors that do not change their functionality, which raises a question about their robustness. in addition, our implementation style analysis results reveal that the models prefer repeating the internal logic in the auxiliary function even when the logic can be easily handled by simply calling them. finally, our human preference evaluation of their style shows this disparity between model-generated implementation and that of humans, suggesting the future direction of enhancing the ability to delegate their subroutine to the auxiliary functions by calling them. several studies have been conducted to evaluate code generation ability (xu et al., 2022). neelakantan et al. (2016); iyer et al. (2018) first introduce neural networks into code completion tasks and evaluate them on traditional metrics, e.g., bleu. chen et al. (2021) propose the humaneval dataset and show llms can generate functionally correct implementations by introducing a functional correctness evaluation process. concurrently, austin et al. (2021) propose the mbpp dataset for python basic programs and hendrycks et al. (2021) release the apps dataset related to coding contest problems. consecutive studies have proposed datasets targeted for realistic purposes. lai et al. (2023) focused on data science problems and wang et al. (2023b) paid attention to realistic coding queries from stackoverflow and yu et al. (2024) aimed at python and java code generation tasks from real-world open-source projects, and babe et al. (2023) concentrated on beginning programmers. these work are combined and included in several coding benchmarks @xcite @xcite . for the metrics, @xcite propose codescore to estimate functional correctness and @xcite propose code-bertscore that utilizes bertscore @xcite . there exists research work that extends the hu-maneval dataset to support other features. @xcite ; @xcite extend the dataset to support multiple programming languages and @xcite propose the humaneval+ dataset that extends their test case to enable rigorous evaluation of functional correctness. @xcite focused on prompt robustness by extending the humaneval dataset. however, an evaluation procedure that enables systematic analysis of how llms leverage auxiliary functions has yet to be released in code generation tasks. ## dataset we manually construct a variety of coding examples with corresponding auxiliary functions. to do this, we treat the python examples in the hu-maneval dataset as our base auxiliary functions and employ human experts to create an extended function for each example. we guide them to produce functions that have additional functionalities compared to the given functions. the following aspects are considered to remove the ambiguity inside the concept of extension and enhance their quality. extension type there exist two different types of extension, i.e., black-box extension and whitebox extension. the black-box extension extends a function by calling the auxiliary function. it does not consider the internal mechanism of the auxiliary function. however, the white-box extension extends them by rewriting the improved internal mechanism. we allow any type of extension, but recommend the black-box one as calling the existing functions if possible is mostly better than rewriting the whole mechanism @xcite . ## experiments we comprehensively evaluate llms' ability to harness auxiliary functions using our humanextension dataset. to do this, we designed research questions as follows. we first examine the effectiveness and robustness of including a single auxiliary function in the prompt and extend this setting into multiple auxiliary functions. also, we explore their implementation styles and analyze them based on human preference. ## conclusion we have explored the ability to utilize auxiliary functions encoded in the llms through our newly proposed humanextension dataset. the humanextension dataset is constructed to contain function relationships while considering the software engi-neering concepts. our multi-faceted experiments with the humanextension dataset comprehensively show the current llms' ability to harness auxiliary functions. our auxiliary function experiments demonstrate that the llms have the ability to utilize auxiliary functions even when the function is implemented by themselves. however, our indepth analysis discovered that their ability varies depending on the factors that humans might not, i.e., the position of relevant functions, function name, and docstring. furthermore, our implementation style analysis reveals that, in some cases, the llms repeat the mechanism of the given auxiliary function while humans simply call the auxiliary functions, suggesting the future research direction of current code llms for auxiliary function calls.