MobileMoE
Collection
a family of on-device MoE language models with sub-billion
active parameters (0.3-0.9B active and 1.3-5.3B total) that establish a new Pareto frontier • 10 items • Updated • 4
You need to agree to share your contact information to access this model
The information you provide will be collected, stored, processed and shared in accordance with the Meta Privacy Policy.
Log in or Sign Up to review the conditions and access this model content.
MobileMoE-S (Base) Model Card
MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a Base model (pre-training + mid-training), an SFT model (supervised fine-tuning), and a QAT model (quantization-aware training). You are currently in the MobileMoE-S-Base repository — the pre-trained 0.3B-active base model.
| S | M | L | |
|---|---|---|---|
| Active / total params | 272M / 1.3B | 528M / 2.8B | 922M / 5.3B |
| Layers | 20 | 26 | 32 |
| Model dimension | 768 | 1024 | 1280 |
| Heads (Q / KV) | 12 / 4 | 16 / 4 | 20 / 4 |
| Routed experts | 60 | 60 | 60 |
| Top-k | 4 | 4 | 4 |
| INT4 weight memory | 0.68 GB | 1.48 GB | 2.75 GB |
| Base | MobileMoE-S-Base | MobileMoE-M-Base | MobileMoE-L-Base |
| SFT | MobileMoE-S-SFT | MobileMoE-M-SFT | MobileMoE-L-SFT |
| QAT (INT4) | MobileMoE-S-QAT | MobileMoE-M-QAT | MobileMoE-L-QAT |
For the detailed technical report: 📝 MobileMoE: Scaling On-Device Mixture of Experts
For more versions, check out the 🤗 MobileMoE Collection
MobileMoE establishes a new Pareto frontier for on-device LLMs. Average benchmark accuracy, computed over 14 benchmarks spanning commonsense, knowledge, science, comprehension, and reasoning, is plotted against (a) per-token inference compute Finf = 2Nact (GFLOPs) and (b) total parameters Ntotal (B); in (b), x-axis tick labels show total params (B) | projected INT4 memory (GB).
Key Features
- A new Pareto frontier for on-device LLMs. Across 14 foundational benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs at 2–4× fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters.
- Scaling-law-derived architecture. The architecture is derived from an on-device MoE scaling law that jointly optimizes under mobile memory and compute constraints, identifying an on-device sweet spot: moderate sparsity, with fine-grained experts and shared expert.
- Four-stage recipe. Pre-training → mid-training → instruction fine-tuning → INT4 quantization-aware training, all on open-source datasets.
Model Information
Model: MobileMoE-S-Base (pre-trained + mid-trained)
Active Parameters: 272M
Total Parameters: 1.3B
Layers: 20
Model Dimension: 768
Attention Heads: 12
KV Heads: 4 (GQA)
Head Dimension: 64
Routed Experts: 60 (fine-grained, FFN hidden dim 384 each)
Active Experts per Token: 4 (top-k sigmoid routing, with normalization)
Shared Expert: 1, always on (FFN hidden dim 1536)
Vocabulary Size: 128,256
Other Features: QK-Norm, tied input/output embeddings, RoPE (θ = 500,000)
Input Modality: Text
Output Modality: Text
Languages: English
Training Stages: Pre-training → mid-training
Context Length: 8,192 tokens
Precision: BF16
Model Developer: Meta
Model Release Date: Aug 2026
License: MobileMoE is FAIR NC licensed
Results
All numbers below are for the base (pre-trained) models, re-evaluated under identical settings with greedy decoding using lm-eval; few-shot counts are given in parentheses after the benchmark name, and benchmarks shown without one are evaluated 0-shot.
Foundational benchmarks
| Capability | Benchmark | Gemma 3 270M | SmolLM2 360M | MobileMoE-S |
|---|---|---|---|---|
| Active / total params | 270M | 362M | 272M / 1.3B | |
| Commonsense Reasoning | HellaSwag | 41.4 | 56.5 | 58.9 |
| PIQA | 68.3 | 71.7 | 75.4 | |
| SIQA | 40.2 | 40.7 | 46.8 | |
| WinoGrande | 53.7 | 59.0 | 58.6 | |
| Knowledge | MMLU (5-shot) | 26.7 | 25.2 | 43.7 |
| NaturalQuestions (5-shot) | 4.1 | 7.4 | 12.6 | |
| TriviaQA (5-shot) | 14.3 | 26.8 | 33.2 | |
| Science | ARC-Challenge (25-shot) | 29.4 | 40.5 | 46.5 |
| ARC-Easy | 56.8 | 68.1 | 73.9 | |
| OpenBookQA | 30.4 | 37.6 | 34.6 | |
| Reading | BoolQ | 58.3 | 61.8 | 60.2 |
| DROP (3-shot) | 14.2 | 17.9 | 39.0 | |
| Reasoning | BIG-Bench Hard (3-shot) | 29.5 | 31.7 | 31.8 |
| GSM8K (8-shot) | 1.8 | 5.3 | 36.2 | |
| Average | 33.5 | 39.3 | 46.5 |
Training
MobileMoE uses a four-stage recipe. This checkpoint is the output of stage 2 (mid-training).
| Pre-training | Mid-training | SFT | QAT | |
|---|---|---|---|---|
| Context length | 2,048 | 8,192 | 8,192 | 8,192 |
| Total tokens | ~6T | ~500B | ~126B | ~21B |
| Peak learning rate | 4×10-4 | 4×10-5 | 4×10-6 | 4×10-6 |
| LR schedule | Cosine | Linear | Cosine | Cosine |
| Token dispatch | drop-and-pad | drop-and-pad | dropless | dropless |
How to use
MobileMoE uses a custom architecture (model_type: mobilemoe) that is not yet part of upstream transformers, so trust_remote_code=True is required. The modeling code ships in this repo (configuration_mobilemoe.py, modeling_mobilemoe.py).
Requirements
pip install "torch>=2.1" "transformers>=4.57" "safetensors>=0.4" "accelerate>=1.0"
Verified with the following versions:
| Package | Version |
|---|---|
torch |
2.8.0 (cu128) |
transformers |
4.57.6 |
tokenizers |
0.22.2 |
safetensors |
0.7.0 |
accelerate |
1.13.0 |
For batch evaluation we recommend vLLM (≥ 0.10.2) with enforce_eager=True.
Text generation
This is a base model — it has no chat template and is not instruction-tuned. Prompt it with plain text continuation:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL_ID = "facebook/MobileMoE-S-Base"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
trust_remote_code=True,
dtype=torch.bfloat16,
)
model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()
prompt = "Why are open-source on-device language models great?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
input_ids=inputs["input_ids"],
attention_mask=inputs["attention_mask"],
max_new_tokens=64,
do_sample=False,
temperature=None,
top_p=None,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Known issues. Loading the tokenizer on transformers 4.57.6 prints a fix_mistral_regex=True warning. Please ignore it and do not set the flag, as MobileMoE uses the Llama-3 tokenizer whose default tokenization is already correct.
Citation
@article{chen2026mobilemoe,
title={MobileMoE: Scaling On-Device Mixture of Experts},
author={Chen, Yanbei and Huang, Hanxian and Chang, Ernie and Szwejbka, Jacob and Desai, Digant and Liu, Zechun and Chandra, Vikas and Krishnamoorthi, Raghuraman},
journal={arXiv preprint arXiv:2605.27358},
year={2026}
}
License
MobileMoE is distributed under the FAIR Noncommercial Research License.
- Downloads last month
- -