Eliciting Instruction-tuned Code Language Models’ Capabilities to Utilize Auxiliary Function for Code Generation (2024)

Abstract: We study the code generation behavior of instruction-tuned models built on top of code pre-trained language models when they could access an auxiliary function to implement a function. We design several ways to provide auxiliary functions to the models by adding them to the query or providing a response prefix to incorporate the ability to utilize auxiliary functions with the instruction-following capability. Our experimental results show the effectiveness of combining the base models’ auxiliary function utilization ability with the instruction following ability. In particular, the performance of adopting our approaches with the open-sourced language models surpasses that of the recent powerful language models, i.e., gpt-4o.

## introduction generating codes based on natural language requirements, i.e., code generation, becomes an appealing application for natural language processing community due to the recent advance of code pre-trained language models @xcite @xcite . pre-training on large-scale code corpora enables a language model to implement correct functions based on their requirements written in the docstrings. also, tuning a code pre-trained language model to follow instructions has been released due to the effectiveness of instruction-tuned language models on natural language tasks @xcite @xcite . 1 these instruction-tuned models boost up the code generation ability. in code generation tasks, leveraging an auxiliary function reduces the implementation difficulty of a target function compared to that of implementing them from scratch. the auxiliary function is a function that helps implement a target function by inspiring novel mechanism or handling complicated subroutines for the target function through function calls @xcite . therefore, properly utilizing the given auxiliary function becomes important for the instruction-tuned models. however, limited research has been conducted on providing auxiliary functions to make the instruction-tuned models utilize the auxiliary functions effectively. @xcite initially included the auxiliary function in the prompt, but it showed inferior results compared to just prompting the corresponding base pre-trained models. also, the instruction-tuned models' ability to incorporate the code content with their natural language text has not been fully explored, except that the model providers showcase some qualitative examples in their appendix @xcite . in this work, we comprehensively explore the instruction-tuned models' code generation behavior when they can access an auxiliary function. to do this, we design several prompts that are likely to elicit the ability to utilize auxiliary functions by leveraging the query-response structure employed in the instruction-tuned models. to be specific, we provide detailed information about the auxiliary function in the query and provide an incomplete codeblock to the prefix in the response to complete the remaining response. then, we evaluate their effectiveness across several competitive instructiontuned models. our evaluation results show that our proposed prompts perform efficaciously on the instruction-tuned models compared to the corresponding base models, and even surpass gpt-4o, which is purportedly known as the most powerful language model. finally, we perform an in-depth analysis to demonstrate that incorporating auxiliary function utilization ability already encoded in their base model with instruction-following capability through the response prefix is the main cause of the superior performance. ## related work several language models @xcite @xcite are pretrained on massive code corpora and further finetuned to follow user instructions using a pre-defined chat format. they show progress on various representative coding benchmarks with a prompt template @xcite @xcite @xcite . the widely used template during the evaluation forms a function signature with the docstring containing the requirement. however, considering that the instruction-following models are trained with chat format, their effectiveness could be overlooked when we evaluating them without considering their formats. @xcite initially design a prompt template for instruction-following models, but the effectiveness on their models are marginal under their evaluation process. along with the prompt engineering research line (sondos mahmoud bsharat, 2023) demonstrating that even a simple prompt adjustment brings enormous gain by eliciting the power of language models, this work explores a better prompt that effectively combines the power of base model and the instruction following capabilities. ## methods we design several prompts for pre-trained models and instruction-tuned models to show their ability to generate code with and without auxiliary functions. to do this, we build up two different approaches to naturally fuse the information inside the auxiliary functions into the instruction format used during instruction tuning. ## experiments we explore the effectiveness of our proposed strategy with recent competitive instruction-tuned models and analyze their behavior from various angles. ## conclusion in this work, we study the instruction-tuned models' behavior when they can access the auxiliary function. through various prompting approaches, i.e., providing an auxiliary function in the query or response, we discover effective prompting approaches that enhance the probability of implementing correct codes using open-sourced models and surpass the recent powerful proprietary models, i.e., gpt-4o. our further investigation identifies that providing docstring or function signature to the response code-block is the major reason to boost performance. we believe that incorporating the ability to utilize other functions with the instructionfollowing capability is indispensable for generating complex code, and our work becomes a cornerstone towards this research direction.