A Comparative Study of PEFT Methods for Python Code Generation (2025)

Abstract: Fine-tuning language models incurs high costs in training, inference and storage. Parameter-efficient fine-tuning (PEFT) methods have emerged as a more cost-effective alternative to full fine-tuning. However, limited work has compared different PEFT approaches for tasks like code generation. In this study, we examine the effect of various PEFT training methods on model performance in the task of Python code generation. We fine-tune four model families, ranging from 124M to 7B parameters, using three PEFT approaches alongside standard full fine-tuning. Our findings reveal that the effectiveness of each PEFT method varies with the model size and the corpus used.

## introduction language models (lms) have shown great capabilities across a variety of natural language processing (nlp) downstream tasks, including code generation tasks @xcite @xcite @xcite . generally, larger lms tend to perform better on downstream tasks @xcite , as evidenced by codellama, which exhibits improved code completion and generation abilities as its size increases from 7 billion to 70 billion parameters @xcite . however, the training of these larger models is resource-intensive, requiring substantial computational power and high storage costs. to address these challenges, parameter-efficient fine-tuning (peft) methods have emerged @xcite @xcite @xcite . these approaches update a small subset of the model parameters during fine-tuning, while the rest remain frozen, significantly reducing both computational and storage costs for each downstream task. while significant research has been conducted on both peft methods and code lms individually, at the time of this study, there is only limited research evaluating peft approaches applied to code lms for code generation tasks @xcite @xcite . existing studies on this topic have notable shortcomings: many focus only on smaller models, ignoring those with 1b parameters or more @xcite , while others concentrate solely on tasks like code understanding or clone detection, which often outperform code generation tasks under similar peft training conditions @xcite @xcite . these limitations highlight a significant research gap, particularly as state-of-the-art models increasingly feature billions of parameters and are predominantly generative. we aim to fill existing research gaps through two key questions: 1) which peft method delivers the best performance across various model sizes for python generation tasks? 2) how do these methods compare to full fine-tuning? ## experimental approach in this section, we describe the methodology used to investigate how models of different sizes adapt to the python code generation task using peft. the experimental approach is illustrated in figure 2 , which outlines the models, dataset, data processing methods, training setup, and evaluation strategy. these elements will be described in detail in the following section. figure 2 : diagram describing the experimental approach adopted in this study. ## discussion best peft approach we observe that smaller models tend to achieve higher codebleu scores when utilizing prompt-based techniques, while larger models show improved performance with lora. prompt-tuning, which tunes the fewest parameters, demonstrates enhanced effectiveness as model size increases, consistent with the findings of @xcite . in terms of bleu scores, lora consistently outperforms other peft techniques. it seems that lora tries to learn the exact n-gram matches from the python solution, succeeding to do so for larger models. conversely, prefix-tuning appears to degrade performance across all models, aligning with the results reported by @xcite . full fine-tuning versus peft