Curriculum Learning for Small Code Language Models (2024)

Abstract: Code language models have emerged as useful tools for various programming tasks, yet they often struggle when it comes to complex ones. In this paper, we explore the potential of curriculum learning in enhancing the performance of these models. While prior research has suggested that curriculum learning does not necessarily help in improving the performance of language models, our results surprisingly show that this may not be the case for code language models. We demonstrate that a well-designed curriculum learning approach significantly improves the accuracy of small decoder-only code language models on the task of code execution, while its effect on code completion is less significant. To explore the potential of curriculum learning, we train multiple GPT models with 1 million parameters each to predict the next token and evaluate them on code completion and execution tasks. Our contributions include proposing a novel code difficulty assessment metric by combining software code measures, investigating the effectiveness of Curriculum Learning for code language models, and introducing a Novel Curriculum Learning schedule that enhances the performance of small decoder-only language models in code execution tasks. The results of this paper open the door for more research on the use of curriculum learning for code language models.

## introduction with the advent of large language models (llms) like gpt-3 @xcite , auto-regressive decoder-only architectures have become dominant in language modeling. these models have shown significant improvement over state-of-the-art performance on a wide range of natural language tasks. accordingly, previous work @xcite @xcite has introduced such architectures for code * both authors contributed equally to this work and share first authorship. modeling, motivated by the software naturalness hypothesis @xcite , which suggests that programming languages can be understood and generated like natural languages @xcite . however, these models often struggle with complex tasks such as understanding code and reasoning about it, which remains a challenge for them. @xcite evaluated the ability of large language models to predict the output of groundtruth programs. the authors found that the few-shot execution performance of their largest model, with 137 billion parameters, never exceeded 29% accuracy across various prompt configurations. finetuning on an code execution dataset resulted in only modest improvements, with the best configuration achieving 28.2% accuracy. in this context, we investigate whether curriculum learning (cl) -training models on simpler examples first before gradually increasing difficulty -can improve the performance of decoder-only language models' trained on source code. we assume that training language models using cl will lead to better performance compared to conventional training. we focus on small-scale models, which allows us to experiment with different setups and iterate quickly. prior research has investigated the use of curriculum learning for language model pre-training, finding no substantial evidence to support its effectiveness @xcite . however, the potential benefits of this approach in the context of code intelligence @xcite , remain relatively unexplored. in contrast to these earlier findings, our investigation indicates that the advantages of cl may be more task-dependent. particularly, we show that while cl exhibits potential in enhancing code execution capabilities, its influence on code completion tasks is less significant. more specifically, we follow an incremental study where we generate a python code dataset, design a code difficulty assessment metric which enables us to categorize our dataset into three levels: "easy", "medium", and "hard". based on these levels, we propose three-stage curriculum learning (cl) schedules to train our code language models. to further illustrate the challenges posed by complex code examples, we evaluated the code llama 13b model @xcite on our "hard level" test set, where it achieved only 39.06% accuracy. this evaluation highlights the limitations of current llm-based code modeling approaches, which still struggle to effectively capture the semantics of source code. our results indicate that performance on code execution can indeed be improved when we design a good curriculum schedule and use a robust code difficulty metric. however, when it comes to code completion tasks, the impact of cl is less pronounced. this suggests that the benefits of cl may not be present across all tasks, but rather, depend on the specific nature of the task. believing this can advance the research on code language models, we have open-sourced our datasets 1 , models and source code 2 . our main contributions can be summarized as follows: ## overview in order to explore whether using curriculum learning can improve the performance of decoderonly language models trained on code, we adopt the following methodology (presented in figure 1 ) : we first generate data (consisting of code snippets followed by their outputs) focusing on a subset of the python programming language, which allows us to reduce the vocabulary size (section 4). we then assess the difficulty of the generated code 1 https://tinyurl.com/tinypyd 2 https://tinyurl.com/cl4sclm ## code difficulty metric determining the difficulty of code is not straightforward. it requires a quantitative measure, which can be provided by commonly used software engineering metrics like cyclomatic complexity (cc) and halstead difficulty (hd). cc, proposed by @xcite , quantifies the number of linearly independent paths through a program's source code. on the other hand, hd, introduced by @xcite , is calculated using the number of operators and operands present in the code. these established metrics allow for the numerical evaluation of code difficulty. however, their independent use may not fully capture the overall difficulty of the code. therefore, we have designed a new metric, referred to as the overall metric (om), which is the average of cc and hd (see equation 1 ). the idea behind creating om is to have a more comprehensive measure of difficulty that takes into account both structural complexity via cc and operational complexity via hd. ## curriculum learning schedules curriculum learning (cl) is a training strategy that presents easier or simpler examples earlier in training and gradually increases the difficulty of examples over time. this section details the curriculum learning schedules we propose, namely: sequential, incremental, and hybrid, illustrated in figure 4. each schedule is divided into three stages. after completing a stage, we reset the learning rate and optimizer before continuing training on the data for the next stage. the three schedules are defined as follows: ## experimental setup in this section, we describe our experimental setup for evaluating the effectiveness of curriculum learning for code language models, including the model architecture we used, our training process, and the evaluation tasks and metrics we employed. ## discussion we designed a code difficulty metric combining software measures, referred to as om, to categorize generated programs into easy, medium and hard levels. the inverse correlation between the om scores and the model accuracies validates its effectiveness for program difficulty assessment. an interesting observation is that conditionals posed more difficulty for models than loops, contrary to expectations. this suggests certain language features are inherently harder to learn for models. this categorization allowed us to explore various three-stage curriculum schedules for model training. our experiments revealed that the hybrid technique achieves much higher output accuracy compared to the conventional training baseline, especially on complex code, indicating its effectiveness in incrementally developing model capabilities. however, the sequential strategy, while helping models learn hard concepts, suffers a loss in overall accuracy. this highlights the importance of curriculum design : simply progressing from easy to hard tasks does not guarantee gains. in the context of code completion tasks, the influence of cl is not as significant as expected. this implies that the advantages of cl may not be applicable to all tasks, but instead, they may vary based on the particular characteristics of the task. furthermore, our fine-tuning experiments with the code llama 7b model further validated the effectiveness of curriculum learning. while the gains in code completion tasks were minor, the hybrid cl approach significantly improved code execution performance. these findings reinforce our findings that a well-designed curriculum can enhance model capabilities, especially for complex tasks, even when scaling to larger models. 9 related works ## conclusion in this paper, we explored the potential of curriculum learning in enhancing the performance of code language models, given their struggle with complex tasks. first, we generated a dataset of python code using the tinypy generator. second, we designed a code difficulty metric (om) combining software complexity measures, and validated its efficacy in assessing program difficulty. third, we used the om to categorize programs into easy, medium, and hard levels and explored various curriculum schedules. finally, we evaluated our models on code completion and execution tasks and compared them to a baseline trained on all the data shuffled. our results show that certain curriculum learning strategies can significantly improve language models' performance on code execution, compared to conventional training. nonetheless, for code completion, the gains from cl were not as significant as expected. additionally, our fine-tuning experiments with the code llama 7b model reinforced these findings, demonstrating that cl can lead to significant improvements in code execution tasks even for larger models. in conclusion, our investigation shows that thoughtfully implemented curriculum learning can improve generative code language models' perfor-mance on code execution tasks. yet, its impact is less noticeable in code completion tasks. this suggests that curriculum learning's effectiveness may vary depending on the task's specific characteristics. overall, our work highlights the potential of curriculum learning to enhance language models for complex code reasoning.