W ave C oder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning (2024)

Abstract: Recent work demonstrates that, after instruction tuning, Code Large Language Models (Code LLMs) can obtain impressive capabilities to address a wide range of code-related tasks. However, current instruction tuning methods for Code LLMs mainly focus on the traditional code generation task, resulting in poor performance in complex multi-task scenarios. In this paper, we concentrate on multiple code-related tasks and present WaveCoder, a series of Code LLMs trained with Widespread And Versatile Enhanced instruction data. To enable the models to tackle complex code-related tasks, we propose a method to stably generate diverse, high-quality instruction data from open source code dataset in multi-task scenarios and obtain CodeOcean, a dataset comprising 19,915 instruction instances across 4 code-related tasks, which is aimed at improving the generalization ability of Code LLM. Our experiments demonstrate that WaveCoder models significantly outperform other open-source models in terms of the generalization ability across different code-related tasks. Moreover, WaveCoder-Ultra-6.7B presents the state-of-the-art generalization abilities on a wide range of code-related tasks.

## introduction recently, large language models (llms) such as chatgpt, @xcite , and gemini 1 have attained unprecedented performance levels in a broad array of nlp tasks. these models utilize a self-supervised pre-training process, and subsequent supervised fine-tuning to demonstrate exceptional zero/few-shot capabilities, effectively following human instructions across various tasks. for code-related tasks, several previous works, including codex @xcite , starcoder @xcite , @xcite and @xcite , have successfully demonstrated that pre-training on code corpus can significantly improve the model's capability to tackle code-related problems. after the process of pre-training, instruction tuning @xcite @xcite has shown its effectiveness in the aspect of improving the quality of llm responses. to specifically enhance the performance of code llms on code-related tasks through instruction tuning, many existing methods for instruction data generation have been designed. for example, code alpaca @xcite utilizes the method of self-instruct @xcite within the coding domain, leveraging the few-shot capabilities of teacher llm to generate instruction data. similarly, wizardcoder @xcite applies the evol-instruct @xcite approach based on code alpaca, demonstrating a novel and effective method for the generation of instruction data. these applications underscore the potential of utilizing teacher llms to produce instructional content effectively, thereby offering an avenue for the creation of instruction data in the code domain. however, the quality of the data they generate heavily relies on the performance of the teacher llm and the limited initial seeds, which often produces a large amount of duplicate instruction instances and reduce the effectiveness of instruction tuning @xcite @xcite . to break away from dependence on teacher llms, octopack @xcite constructs a code instruction dataset leveraging the natural structure of git commits. nonetheless, ensuring the quality of data in git messages presents a considerable challenge, and the comprehensive screening of data through artificial filtering rules is often a complex task. additionally, these endeavors are predominantly centered on traditional code generation tasks and lack the capability to produce detailed, taskspecific instructions in multi-task scenarios. in this paper, we primarily focus on multiple code-related tasks, aiming to generate high-quality and diverse instructional data tailored to specific task requirements. addressing the aforementioned challenges, we refine the instruction data by classifying the instruction instances to four universal code-related tasks in codexglue @xcite : 1) code summarization, 2) code generation, 3) code translation, 4) code repair and propose a widespread and versatile enhanced instruction generation method that could make full use of open source code data and stably generate high quality and diverse instruction data in multitask scenarios. by this generation strategy, we obtain a dataset of 19,915 instruction instances across four code-related tasks, termed codeseaxdataset. to validate our approach, we train starcoder @xcite , codellama (roziere et al., 2023), and deepseekcoder (guo et al., 2024) with our initial codeseaxdataset dataset and get wave-coder. following a thorough assessment on hu-maneval @xcite , mbpp @xcite , humanevalpack @xcite benchmarks, experimental results show that our wavecoder exhibits outstanding generalization ability based on widespread and versatile enhanced instruction tuning. moreover, to further explore the improvements brought by data quality, we use gpt-4 (openai, 2023) to regenerate response for the instruction in codeseaxdataset. finetuned with the enhanced 20k codeseaxdataset dataset, we obtain wavecoder-pro-6.7b which achieve 72.0% pass@1 on humaneval @xcite and surpass open source code llms but still behind sota code llm. combining enhanced codeseaxdataset with wavecoder-evolcodealpaca, the decontaminated magicoder-evolcodealpaca foot_0 dataset, we present wavecoder-ultra-6.7b, with sota generalization capabilities on multiple code-related tasks. ## conclusion this paper presents wavecoder, a code llm finetuned with widespread and versatile enhanced instruction data. by enabling language models to effectively tackle complex code-related tasks, our approach demonstrates the potential of integrating multiple code-related tasks into instruction tuning for code llm and generating high-quality and diverse instruction data for specific task requirements in multi-task scenarios. wavecoder achieves stateof-the-art generalization performance on different code-related tasks surpassing existing open source code llms. furthermore, our analysis of the relationship of different tasks provides valuable insights for future research, paving the way for more extensive code-related tasks and larger dataset.