CLIP - UP : A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling (2025)

Abstract: Mixture-of-Experts (MoE) models are crucial for scaling model capacity while controlling inference costs. While integrating MoE into multimodal models like CLIP improves performance, training these models is notoriously challenging and expensive. We propose CLIP-Upcycling (CLIP-UP), an efficient alternative training strategy that converts a pre-trained dense CLIP model into a sparse MoE architecture. Through extensive experimentation with various settings and auxiliary losses, we demonstrate that CLIP-UP significantly reduces training complexity and cost. Remarkably, our sparse CLIP B/16 model, trained with CLIP-UP, outperforms its dense counterpart by 7.2% and 6.6% on COCO and Flickr30k text-to-image Recall@1 benchmarks respectively. It even surpasses the larger CLIP L/14 model on this task while using only 30% of the inference FLOPs. We further demonstrate the generalizability of our training recipe across different scales, establishing sparse upcycling as a practical and scalable approach for building efficient, high-performance CLIP models.

## introduction clip @xcite has become foundational across domains such as image classification, multimodal retrieval, and aidriven multimodality content generation @xcite @xcite @xcite . as applications grow, scaling clip becomes essential. most efforts focus on enlarging dense models @xcite , which improves performance but incurs high computational and inference costs. an efficient alternative is sparse modeling with mixture-of-experts (moe) @xcite . however, training moebased clip models like limoe @xcite from scratch remains expensive and often requires auxiliary losses for stability. for instance, limoe outperforms dense clip but demands 1.35× more training flops @xcite . to address this, we explore sparse upcycling @xcite , which initializes moe layers from a pre-trained dense model. as shown in figure 1 , our extensive experiments demonstrate that sparse upcycling with a separated backbone achieves the best performance while reducing training zflops from 4.2 to 3.7 compared to training from scratch. although limoe's entropy losses @xcite improve sharedbackbone models trained from scratch, they still underperform other setups. section 3.1 details these strategies and the effects of auxiliary losses. in contrast, we propose clip-up, a single-stage sparse upcycling method for clip. by leveraging pre-trained weights, clip-up provides a warm start that boosts efficiency and surpasses both dense continued training and sparse-from-scratch methods across model scales. 1 our main contributions are: 1. we introduce clip-up, a simple and effective training recipe for moe clip models m text embed image embed selfatten dense text layer mlp 1 router (from scratch) mlp 2 mlp e … weighted sum moe dense text layer moe text layer m selfatten dense image layer mlp 1 router (from scratch) mlp 2 mlp e … weighted sum moe dense image layer moe image layer contrastive loss + auxiliary loss selfatten mlp pretrained text layer copy weights make e mlp copies text encoder image encoder make e mlp copies copy weights selfatten mlp pretrained image layer sparse upcycling figure 2: clip-up overview with sparse upcycling initialization. selected mlp layers are replaced with moe layers, initialized from the dense checkpoint, while routers are randomly initialized. via sparse upcycling, avoiding complex auxiliary losses and outperforming existing methods across shared and separated backbones. 2. clip-up significantly improves performance on text-image retrieval, surpassing dense clip by 7.2% and 5.5% (recall@1) on coco and flickr30k, respectively, with a b/16 backbone. 3. we demonstrate clip-up's scalability from b/32 to l/14 and provide insights into key factors and challenges to inform future design. ## clip-up figure 2 illustrates the clip-up architecture and training strategy. we detail both in this section. ## experiments datasets. we trained both the initial dense clip checkpoint, clip-up, and the baseline model on the same paired image-text datasets-wit-300m @xcite and dfn-5b @xcite . evaluation was performed on im-agenet @xcite for classification and on coco @xcite and flickr30k @xcite for image-text retrieval, with additional benchmarks provided in the appendix c.2 and appendix c.5. the input image resolution is 224 for all of the datasets. setup. we train a dense clip model for 440k steps, then upcycle it into an moe version with 350k additional steps. both use adamw with a 32k batch size; the dense model uses a learning rate of 5 × 10 -4 and weight decay of 0.2, reduced to 5 × 10 -5 and 0.05 for upcycling. in the moe model, half of the transformer's mlp layers follow an alternating [dense, sparse] pattern @xcite , each sparse layer using 8 experts with top-2 routing. router loss coefficients α = 0.01 and β = 0.001 balance expert usage without dominating training @xcite . for fair comparison, we also train a dense clip for 790k steps using the same settings. ## conclusions in this work, we introduce clip-up, a simple yet efficient training strategy for clip that integrates moe with sparse upcycling. extensive experiments on various configurations and auxiliary losses show it reduces training costs while consistently improving retrieval and classification performance across model scales. notably, clip-up outperforms larger dense models in some tasks while drastically cutting inference flops, highlighting sparse upcycling as a practical and scalable approach for building efficient, high-performance clip models.