M ulti PL - M o E : Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts (2025)

Abstract: Despite LLMs’ excellent code creation capabilities, multilingual code generation remains extremely challenging. To address this, we intent to improve the multi-programming-lingual (MultiPL) performance of the base LLMs while retaining the most popular ones using restricted computational resources. We consider MultiPL to be a special case of multiple natural languages and propose a MultiPL extension of LLMs utilizing a hybrid mixture of experts (MoE), called MultiPL-MoE. Specifically, MultiPL-MoE combines two paired MoEs to optimize expert selection at both the token and segment levels. The **token-level MoE** is a standard upcycling MoE structure with a shared expert and a novel gate weight normalization approach that aids in the final fusion with the segment-level MoE. The **segment-level MoE** incorporates two innovative designs to better capture the syntactic structure and contextual patterns of programming languages: First, using a sliding window to partition the input token sequence into multiple segments; Then, adopting an expert-choice routing strategy that allows experts to select the top-k segments. The results of the experiment proved the effectiveness of MultiPL-MoE.

## introduction large language models (llms) such as gpt4 @xcite and qwen2.5-coder @xcite have shown remarkable capabilities in code generation, facilitating software engineers to boost their efficiency. however, many studies have highlighted a significant discrepancy between performance on high-resource programming languages such as python and low-resource ones like rust @xcite , which refers to those with limited availability both online and in training datasets. figure 1 : multipl-moe achieves dual success on hu-maneval by preserving proficiency in high-resource programming languages and advancing performance on low-resource languages. specially, it attains a pass@1 score surpassing the second-best model by 1.97 points on low-resource languages while sacrificing only 0.9 points below qwen1.5 on high-resource languages. the growing demand for multilingual code generation capabilities has driven efforts to inject extensive programming language knowledge into llms. to enhance the base llms' multiprogramming-lingual (multipl) capabilities, some existing methods @xcite @xcite @xcite @xcite continue train a base llm with data from multiple programming languages but are highly computationally intensive. others @xcite @xcite @xcite improve specific programming languages by fine-tuning a base llm with specific high-quality low-resource data. how-ever, they suffer from severe catastrophic forgetting for the base model's existing code capabilities. facing these challenges, we intend to improve llms' multipl performance, especially for lowresource programming languages, while maintaining the performance of existing popular ones with a relatively small additional computational budget. recent advances in programming language translation have demonstrated that they share common semantics @xcite and can be converted from one to another in the same way that natural languages do @xcite . consequently, we may enhance the multipl performance of llms by leveraging recent advancements in natural language expansion technology, which adopt mixtureof-experts (moe) architecture by upcycling methods and prove to achieve good language expansion effectively while preventing catastrophic forgetting @xcite . in this paper, we take multipl as a special case of multiple natural language expansion and propose multipl-moe, a multi-programming-lingual (mul-tipl) extension of llms through hybrid mixtureof-experts (moe). unlike prior moe methods for multiple natural language extension @xcite that focus on internal language consistency by predicting subsequent tokens but ignore the internal syntax structures of programming languages @xcite , multipl-moe employs novel hybrid moe architecture for fine-grained learning of both the token-level semantic features and segment-level syntax features, enabling the model to reason about code structure by identifying and categorizing different syntactic elements. the token-level moe combines traditional tokenchoice routing with a shared expert @xcite and a novel routing weight normalization mechanism to handle scale mismatch during the later fusion with the segment-level moe. in particular, multipl-moe combines two paired moes to optimize expert selection at both the token and segment levels. the token-level moe is a typical upcycling moe structure with a shared expert @xcite @xcite and a novel gate weight normalization approach that aids in the final fusion with the segmentlevel moe. to better capture the syntactic structure and contextual patterns of programming languages, the segment-level moe incorporates two innovative designs: 1) using a sliding window to partition the input token sequence into multiple segments. 2) adopting a different routing strategy with segment-level moe that allows experts to select the top-k segments instead of segments selecting experts. as illustrated in figure 1 , experiment results show that multipl-moe achieves balanced mul-tipl proficiency with 1.97% higher average accuracy than baselines on low-resource languages (rust, go, ruby) in the humaneval benchmark, while matching state-of-the-art performance within a 1.80% gap for high-resource languages like python, java, and c++. ## methodology figure 2 illustrates the overall framework of multipl-moe, which is constructed through upcycling the original ffn layer of a base llm. the token-level moe is a conventional token-choice routing combined with a shared expert @xcite with a novel routing weight normalization method to address scale mismatch during the later fusion with the segment-level moe. for the segment-level moe, we adopt the expertchoice routing mechanism with the input as contextually coherent segments, enabling experts to capture the syntax structures and discourse-level features. the outputs of the two moes are finally fused together. ## conclusion this paper proposes multipl-moe, a hybrid moe with both token-level and segment-level. multipl-moe introduces shared experts to capture the commonality of knowledge at the token level, and obtains semantic and logical information between segments at the segment level. extensive empirical results on two different benchmarks demonstrate the effectiveness of multipl-moe. in summary, multipl-moe is an effective method for extending low-source programming languages in the postpretraining phase.