PLEX : Adaptive Parameter-Efficient Fine-Tuning for Code LLM s using Lottery-Tickets (2025)

Abstract: Fine-tuning large language models (LLMs) for code generation is challenging due to computational costs and the underrepresentation of some programming languages (PLs) in pre-training. We propose PLEX, a lottery-ticket based parameter-efficient fine-tuning (PEFT) method that adapts LLMs to either well-supported and underrepresented PLs.During lottery ticket selection, PLEX employs a dual strategy: for well-represented PLs, it leverages the LLM’s full parametric knowledge by selecting from full layers, while for underrepresented PLs, it narrows the selection scope to dense layers, prioritizing the most influential parameters.Additionally, PLEX-E, a low-rank extension of PLEX, further reduces computational costs by limiting the scope of fine-tuning. On MultiPL-E benchmarks, PLEX achieves state-of-the-art performance among PEFT methods, while PLEX-E maintains competitive results with reduced computational overhead. Both variants demonstrate effective adaptation across diverse programming languages, particularly for those underrepresented in pre-training.

## introduction code generation is a critical task in software development, and large language models (llms) have shown great promise in this domain @xcite . in industrial settings, serving code llms often requires optimized generation for a target pl. moreover, the target language can be proprietary, such as those used in chip design, which are typically absent from pretraining corpora. fine-tuning llms for specific pls faces a computational bottleneck: while scaling laws suggest that larger models yield better performance, fully adapting the model for each target pl is prohibitively resource-intensive. this underscores ˚equal contribution : corresponding author. an additional challenge arises when the target pl is underrepresented. we specifically use the term 'underrepresented', distinct from 'lowresourced', as low-resourced is defined with respect to an amount of public training resources per language. meanwhile, underrepresentation is defined with respect to a specific pl-model pair. for instance, one language that is abundantly observed in pretraining one model, can be underrepresented in another. needs for adapting to an underrepresented pl are common in industrial setting, when supporting a rare pl or a private or proprietary language unavailable during pre-training. however, in our preliminary experiments, we observe that no existing peft method is one-size-fits-all for both well-supported and underrepresented pls. to address these issues, we propose plex, a novel peft method designed to efficiently adapt llms to both well-supported major pls and underrepresented or private pls. our method employs an adaptive parameter selection using lotteryticket @xcite @xcite , adjusting the parame-ter groups based on whether the target pl is supported by the llm. for well-supported pls, plex leverages the full parameter space for ticket selection, maximizing the use of the model's pretrained knowledge. for underrepresented or private pls, plex narrows the focus to dense layers, ensuring that the most influential parameters are prioritized during fine-tuning @xcite . to further improve computational efficiency, we introduce plex-e, an extension of plex that replaces full fine-tuning with low-rank lora tuning @xcite for parameter selection. this reduces the computational burden while minimizing performance drops, making it feasible to apply to larger models. our experimental results on the multipl-e hu-maneval and mbpp benchmarks @xcite demonstrate the effectiveness of plex. the method performs well not only on wellsupported languages like java, php, c++, and swift for starcoder-7b @xcite and java for santacoder-1.1b @xcite , but also on underrepresented languages like php and c++ for santacoder-1.1b. these results validate the adaptability of plex across a diverse range of programming languages, including those underrepresented during pre-training. additionally, in the starcoder-7b experiments, plex-e, the computationally efficient version of plex, generally outperforms existing peft baselines while remaining competitive with plex. the code and dataset are publicly available. foot_0 2 related work ## experiments in this section, our goal is answering to the following research questions: