Qwen3.5-2B-MoE-2x (sparse-upcycled, experimental)

A 2-expert sparse-upcycled Mixture-of-Experts variant of Qwen/Qwen3.5-2B-Base, created via the Mistral-style sparse upcycling recipe. Each layer's FFN was duplicated into N=2 experts; a learned router sends each token through one expert (top-1). The model was then continued-trained with KD against the dense teacher to let the experts diverge.

This is an experimental demonstration, trained on a small budget (~2.6M tokens on a single RTX 3090, 10K optimizer steps). Quality is degraded vs the dense base — see "Caveats" below. The point is to validate the upcycling pipeline at small scale; production-quality versions would need substantially more compute.

How to load (vLLM)

from vllm import LLM, SamplingParams
llm = LLM(
    model="loktar/Qwen3.5-2B-MoE-2x",
    dtype="bfloat16",
    max_model_len=2048,
    gpu_memory_utilization=0.85,
)
sp = SamplingParams(temperature=0.0, max_tokens=64)
print(llm.generate(["The capital of France is"], sp)[0].outputs[0].text)

Requirements:

• vLLM ≥ 0.23.0
• transformers ≥ 5.12 (for native Qwen3.5-MoE architecture support)
• An NVIDIA GPU with enough VRAM (~7GB at bf16)

Sample outputs (greedy decode)

"The capital of France is Paris, and the capital of Germany is Berlin.
 The capital of Italy is Rome..."

def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci(n-2)

Architecture

• Base: Qwen3.5-2B (hybrid GDN + full-attention, 24 layers, 2048 hidden, 6144 intermediate)
• Per layer: 2 expert FFNs (each intermediate = 6144), 1 small shared expert (intermediate = 64, zero-initialized → no contribution), 1 router
• Per token: top-1 routing (each token → exactly 1 of 2 experts)
• Inference architecture: Qwen3_5MoeForConditionalGeneration (multimodal wrapper; text-only at runtime)
• Total params: ~2.9B (vs 2B dense base)

Evaluation

Metric	This model	Dense Qwen3.5-2B
WikiText-2 PPL	33.5	9.02
Inference speed in vLLM (batch=1, decode, RTX 3090)	42.0 tok/s	44.6 tok/s
Expert divergence (avg cos sim between experts)	0.86 (from init = 1.0)	n/a

Training details

• Recipe: Sparse upcycling per Komatsuzaki 2022 / Mixtral technical report.
• Initialization: Each expert = identical copy of dense Qwen3.5-2B's FFN, plus tiny noise (std ≈ 0.5% of weight std) to break symmetry. Router randomly initialized.
• Objective: KD (forward KL on logits against frozen dense teacher) + load-balance auxiliary loss (Switch Transformer style, coefficient 0.01).
• Optimizer: AdamW 8-bit (bitsandbytes), peak LR 1e-4, cosine decay to 0, 100-step linear warmup, grad clip 1.0.
• Data: same mixed corpus used by ToMoE (alpaca + wikitext + others, ~mix).
• Steps: 10,000 with block_size=256, batch_size=1 → ~2.6M tokens total.
• Hardware: single RTX 3090, ~4 hours.

Caveats

1. Tiny training budget. Mistral's original sparse upcycling used hundreds of billions of tokens. This model used ~2.6M — about 40,000× less. WikiText PPL of 33 (vs dense 9) reflects that the experts only partially specialized in the time available.

2. Visual tower is the dense Qwen3.5-2B's copied verbatim. The model is text-only at runtime, but the visual weights are included so vLLM's Qwen3_5MoeForConditionalGeneration loads without complaint. Don't expect vision inputs to do anything useful (or anything at all).

3. N=2 is a demonstration, not a real production design. Inference speed is within ~6% of dense because the routing overhead nearly offsets the FFN compute savings at small N. Real-world MoE designs use N=8 (Mixtral) or N=128 (Qwen3-30B-A3B) for meaningful per-token compute reduction.

4. The shared expert is disabled (initialized to zero, intermediate_size=64). The Qwen3.5-MoE architecture has a shared expert that's always active in addition to routed experts; ours has the structure but contributes nothing. This is intentional for cleaner experiment design.

Reproducing the upcycle + training

Code: [https://github.com/](add your repo if you publish it).

The pipeline:

1. upcycle.py — load dense Qwen3.5-2B, duplicate FFNs N=2 times, add router, save.
2. train_sparse_upcycle.py — KD continued training with teacher = frozen dense.
3. convert_to_qwen35moe.py + fix_vllm_keys.py + add_vision_weights.py — pack into the Qwen3.5-MoE format vLLM expects.

License

Inherits Apache 2.0 from the Qwen3.5-2B-Base base model.

Citation

If you build on this work:

@misc{loktar2026qwen35moe2x,
  author = {loktar},
  title  = {Qwen3.5-2B-MoE-2x: Sparse-Upcycled Demonstration},
  year   = {2026},
  url    = {https://huggingface.co/loktar/Qwen3.5-2B-MoE-2x}
}