I nstruct C oder: Instruction Tuning Large Language Models for Code Editing (2024)

Abstract: Code editing encompasses a variety of pragmatic tasks that developers deal with daily. Despite its relevance and practical usefulness, automatic code editing remains an underexplored area in the evolution of deep learning models, partly due to data scarcity. In this work, we explore the use of Large Language Models (LLMs) to edit code based on user instructions. Evaluated on a novel human-written execution-based benchmark dubbed EditEval, we found current models often struggle to fulfill the instructions. In light of this, we contribute InstructCoder, the first instruction-tuning dataset designed to adapt LLMs for general-purpose code editing, containing high-diversity code-editing tasks such as comment insertion, code optimization, and code refactoring. It consists of over 114,000 instruction-input-output triplets and covers multiple distinct code editing scenarios. The collection process starts with filtered commit data sourced from GitHub Python repositories as seeds. Subsequently, the dataset is systematically expanded through an iterative process, where both seed and generated tasks are used to prompt ChatGPT for more data. Our findings reveal that open-source LLMs fine-tuned on InstructCoder can significantly enhance the accuracy of code edits, exhibiting superior code-editing performance matching advanced proprietary LLMs. The datasets and the source code are publicly available.

## introduction developers typically engage in a cyclic routine of writing and revising code. as a crucial element, code editing takes up a great portion of this process, encapsulating diverse sub-tasks such as code optimization, refactoring, and bug fixing, each posing distinct challenges. automated code editing tools could substantially boost developer productivity by alleviating the burden of monotonous tasks. however, it remains an under-explored area, partly due to the lack of relevant data, hampering substantial progress by deep learning models. inspired by the recent advancements in llms @xcite @xcite @xcite and code llms @xcite @xcite , we explore the proficiency of llms in code editing tasks based on user instructions, for instance, "add a docstring to the function for clarity", "remove redundant code", or "refactor it into reusable functions". these tasks are distinctly different from code completion, which involves generating code to complete given code snippets or comments. code editing requires the model to not only understand the existing code but also execute modifications that are in line with the given instructions, while seamlessly integrating with the context. for example, removing redundant code or refactoring a function should not affect the return value. to systematically evaluate llms for code editing, we created a novel benchmark named editeval. it contains various types of code edits adapted from github commits and existing datasets. intriguingly, we found that open-source models yield unsatisfactory results, and even the most advanced proprietary llms struggle to solve these tasks. in addressing this challenge, we present instruct-coder, a diverse dataset for instruction finetuning, particularly designed to improve the code editing figure 1 : data collection pipeline of instructcoder (left) and a qualitative example from the dataset (right, best viewed with zoom). initial seed tasks are selected from github commits, and inspire chatgpt to generate new instructions. plausible scenarios where the filtered instructions may be used are then generated. finally, corresponding code input and output are obtained conditioned on both the instruction and scenario. high-quality samples are manually selected and recurrently added to the task pool for further generation. our empirical studies reveal that llms display notable gains in code editing abilities after finetuning with instructcoder. code llama achieves the best results through fine-tuning, attaining an accuracy of 57.22%, closely matching chatgpt. further studies also signify that while the pre-training of the models is fundamental, the code editing performance is highly influenced by the quality and volume of the instruction-tuning data. in summary, the contributions of this work are (1) instructcoder, the first instruction-tuning dataset featuring a wide range of diverse code editing tasks, and demonstrate the effectiveness of instruction-finetuning with instructcoder; (2) editeval, a novel human-written execution-based benchmark for the rigorous evaluation of generalpurpose code editing; (3) we find that opensource models instruction-tuned with instruct-coder can demonstrate strong code editing performance matching chatgpt. ## editeval: evaluating code editing models as aforementioned, code editing is significantly different from code completion. consequently, widely utilized datasets in the realm of code completion, such as mbpp @xcite and humaneval @xcite , fall short in evaluating code editing capabilities. to rigorously evaluate the code editing capabilities, we curated a test set of 194 code editing tasks, derived from three key sources: github commit data, mbpp, and hu-maneval. we harness the input code from these sources and create plausible edit instructions. for github sources, we manually create execution contexts so that the code is runnable. each sample is accompanied by a canonical solution written by humans to ensure the instruction is viable. the generated code edits are strictly assessed using automated test cases to evaluate the correctness of the edits. an edit is considered correct only if it passes all the test cases. this automated method provides a robust and objective evaluation framework, essential for benchmarking the model's performance in diverse code editing situations. appendix a showcases an example of the test set. we benchmarked several instruction-tuned models on editeval, and the results are listed in ## instructcoder: instruction-tuning empowers code editing in this section, we introduce how we create in-structcoder to boost the code editing abilities of llms via instruction finetuning. we employed a method based on self-instruct @xcite , which expands instruction finetuning data by bootstrapping off language model generation. the methodology of generating data with llms requires minimal human-labeled data as seed tasks while maintaining the quality and relevance of the tasks in the dataset. through an iterative process of generating instructions and refining them with deduplication, we create a dataset of a wide range of code-editing tasks. figure 1 illustrates the data collection pipeline of instructcoder. ## data analysis we analyze instructcoder in terms of 1) diversity, 2) complexity, and 3) correctness. we provide distribution and complexity analyses of the task instances. finally, we demonstrate through human investigation that our data is highly reliable. ## conclusion we introduce instructcoder, the first instructiontuning dataset for general-purpose code-editing tasks. it comprises generations of llms, where real github commits serve as seed tasks to guide the generation process. a scenario-conditional approach is introduced to ensure both diversity 8 and high quality of the data. our experiments on the novel editeval benchmark show that opensource models can gain huge improvements and even yield performance matching proprietary models through computationally lightweight parameterefficient fine-tuning with instructcoder. we also reveal that the llm base model and the scale of fine-tuning data are both profound factors of codeediting ability. we hope the dataset can benefit and inspire more research in this area towards building more powerful coding models.