🐋 nanowhale-100m-DIME

Pure-NumPy reference implementation of the DeepSeek-V4-Tiny architecture (MLA + Hyper-Connections + MoE).

Maintained by eulogik — building DIME, the disk-streaming Mixture Engine for running 284B-class MoE models on 16 GB Macs.

No PyTorch. No GPU. No CUDA. Just numpy + safetensors. And it matches the official PyTorch reference exactly — cosine similarity = 1.000 on logits, verified across 1–9 tokens.

This is the reference engine behind DIME (Disk-streaming Mixture Engine) — the runtime that runs 284B-class MoE models on a 16 GB Mac Mini by streaming experts from disk.

⚡ Run a 110M-parameter MoE model in ~10 lines, in the browser, on a phone, or on a toaster. The weights are ~441 MB and load with zero ML frameworks.

---

🚀 Quickstart

pip install numpy safetensors

from huggingface_hub import snapshot_download
from nanowhale_dime import NanowhaleDIME

model_dir = snapshot_download("cmpatino/nanowhale-100m")
model = NanowhaleDIME(model_dir)

print(model.generate("The future of efficient inference is", max_new_tokens=32))

Or just open the Colab notebook — zero setup.

---

🧠 What's inside

nanowhale-100m is a 110M-parameter DeepSeek-V4-Tiny that distills the key ideas of DeepSeek-V4 into a tiny, hackable package:

Component	Detail
Layers	8
Attention	Multi-head Latent Attention (MLA) with RoPE
Experts	4 routed + 1 shared, top-2 routing
Hyper-Connections	hc_mult=4 with Sinkhorn-iterating mixer
MTP	Multi-Token Prediction head
Params	~110M (float32, ~441 MB safetensors)

All of it is reimplemented in readable NumPy in a single file (nanowhale_dime.py) that you can read end-to-end in one sitting. Great for learning how modern MoE + MLA models actually work under the hood.

---

✅ Why "bit-exact"?

We compared our NumPy forward pass against the official PyTorch reference model on logits for sequences of length 1, 4, and 9 tokens:

tokens=1  cos=1.000
tokens=4  cos=1.000
tokens=9  cos=1.000

The implementation faithfully reproduces:

• RMSNorm with the exact variance epsilon
• MLA query/key de-rotation (inverse RoPE on attention output)
• The Softplus + gate-bias router normalization
• Hyper-Connection pre/post mixing with Sinkhorn-balanced combination matrix
• Shared + routed expert summation with top-2 scaling

---

🌊 What is DIME?

DIME is an MoE inference runtime for Apple Silicon that breaks the "must fit in RAM" rule. It lazily loads expert weights from local or networked NVMe and keeps a frequency-promoted hot tier of the experts you actually use (the same trick ktransformers uses to run 236B models on consumer hardware).

nanowhale-100m-DIME is the fully self-contained, verified reference. Once the math is proven at 110M params, the same engine scales to DeepSeek-V4-Flash-class (284B) models — streamed, not loaded.

Follow eulogik for the 284B-on-a-Mac-Mini release. 🍎

---

📎 Original model

Weights and tokenizer are from cmpatino/nanowhale-100m. This repo contains only the NumPy inference engine + Colab + this card.

---

🌊 Maintained by eulogik

We build DIME (Disk-streaming Mixture Engine) — MoE inference for Apple Silicon that runs models far larger than RAM by streaming experts from disk. Follow @eulogik and star eulogik/dime for the 284B-on-a-Mac-Mini release. 🍎

📄 License

Apache-2.0 (weights inherit their original license).