C ode M : Less Data Yields More Versatility via Ability Matrix (2024)

Abstract: In the era of code large language models (code LLMs), data engineering plays a pivotal role during the instruction fine-tuning phase. To train a versatile model, previous efforts devote tremendous efforts into crafting instruction data covering all the downstream scenarios. Nonetheless, this will incur significant expenses in constructing data and training model. Therefore, this paper introduces CodeM, a novel data construction strategy, which can efficiently train a versatile model using less data via our newly proposed ability matrix. CodeM uses ability matrix to decouple code LLMs’ abilities into two dimensions, constructing a lightweight training corpus that only covers a subset of target scenarios. Extensive experiments on HumanEvalPack and MultiPL-E imply that code LLMs can combine the single-dimensional abilities to master composed abilities, validating the effectiveness of CodeM.

## introduction code large language models (code llms) have been booming recently @xcite ). an abundance of code llms are released in succession, e.g., codex @xcite , @xcite , @xcite , and codellama @xcite . recent trends have witnessed the versatility of code llms, aiming to train a multilingual multitasking model. to meet this need, some efforts @xcite typically created the corresponding instruction training data for every downstream language and task, to fine-tune the model. however, this will entail significant costs in constructing data and training models, considering the countless downstream scenarios @xcite . for instance, if we enable the model to support 3 tasks and 6 languages, we need to laboriously craft training data for a total of 18 diverse scenarios. moreover, our expectations for the versatile abilities of code llms are continually growing @xcite @xcite . to more efficiently fine-tune code llms, we propose a novel data construction strategy, namely codem, which can empower code llms with powerful multilingual multitasking abilities using less training data via our newly proposed ability matrix. this matrix assists codem in decoupling the versatile abilities of code llms into two ability dimensions (understanding languages and completing tasks). then, codem only requires constructing the training datasets that cover each single-dimensional ability (e.g., python, cg, or ce), rather than that of all composed abilities (e.g., python+cg, java+ce, or go+cr). we suppose that those composed abilities not covered by the constructed dataset can be implicitly learnable based on two conjectures: (1) code llms can generalize based on explicitly learnable abilities; (2) code llms can imitate based on the relationships between multiple explicitly learnable abilities. as shown in figure 1 , compared to its baseline sft-mixed, codem only needs to cover a subset of 18 composed abilities by picking out one row and one column. codem aims to master all com-posed abilities (e.g., java+ce) by combining the explicitly learned single-dimensional abilities (e.g., java and ce), and showcases uncompromising performance in these implicitly learned abilities. to evaluate codem, we first craft the corresponding instruction training data for each of these 18 scenarios. then, we conduct experiments to validate codem's effectiveness on humanevalpack. extensive results demonstrate that codem can rival sft-mixed, while utilizing less than half the data. to more comprehensively verify codem, we explore the proposed two conjectures for codem. we first verify the generalization of code llms across 15 languages, 7 tasks, and even 3 domains on multipl-e. our findings reveal that the model trained on one scenario training data can produce a universal generalization. secondly, to verify whether the code model can imitate, we deliberately remove key data from codem's training corpus to disrupt the preconditions for imitation, and observe performance changes. the results prove the existence of imitation behavior in code llms. in a nutshell, our contributions can be listed as follows: (1) we introduce codem, a simple yet effective data construction strategy, which can yield more versatility of code llms using less training data via our newly proposed ability matrix. this matrix is capable of decoupling multiple scenarios into two ability dimensions, thereby aiding codem in achieving efficient training. (2) we carry out extensive experiments on humaneval-pack and multipl-e to validate codem's effectiveness, as well as offer some valuable analyses. ## training corpus construction regarding the 3 tasks (cg, ce, and cr) across 6 languages (python, javascript, java, go, c++, and rust) mentioned in section 2.1, totaling 18 scenarios, we craft corresponding instruction dataset for each of these by harnessing the capability of openai's gpt-3.5-turbo. in detail, we start from codealpaca foot_0 , an instruction dataset of code generation, from which we extract those python-related instances as our seed data. based on these seed data, we meticulously design prompts for each of the 18 scenarios to request openai's gpt-3.5-turbo, deriving the corresponding instruction datasets. more details about prompt engineering can be seen in appendix a. in our experiments, each training dataset contains 9.6k data pairs, where a concrete data pair is displayed in appendix figure 6 . we use humanevalpack @xcite as our primary benchmark to evaluate the versatility of code llms brought about by codem. it provides a series of evaluations for 3 tasks (code generation, code explanation, and code repair) across 6 programming languages (python, javascript, java, go, c++, and rust). besides that, we also conduct experiments based on multipl-e @xcite , which focuses on the code generation evaluation across 11 programming languages, including python, c, c++, javascript, typescript, php, go, rust, bash, java, and racket. the two benchmarks are created by adapting humaneval @xcite , which is a hand-written benchmark for python code generation comprising 164 programming problems with comprehensive test cases. ## conclusion and future work in this paper, we propose codem, which aims to empower code llms with powerful multilingual multitasking capabilities using less training data by leveraging ability matrix. the matrix assisting codem divides the model's intrinsic abilities into two dimensions: languages and tasks, then guides the construction of instruction training data. extensive experiments on humanevalpack and multipl-e demonstrate the effectiveness and superiority of codem. furthermore, we validate two conjectures of codem: generalization and imitation, and obtain many insightful findings. in our future work, we would like to delve into code llms to uncover more effective and intriguing data construction strategies.