VelaVec: 9.8M encoder, 53.9× faster than 33M teacher on CPU (pure Rust)
VelaVec: 9.8M retrieval encoder, 53.9× faster than its 33M teacher on CPU (pure Rust)
We've been working on a hybrid-distillation retrieval encoder and wanted to share the results — both the model and the deployment story behind it.
The model (xagent2025/VelaVec, MIT): a 33M bge-small teacher distilled into a 9.8M trunk — frozen 30,522×256 static table (distilled token embeddings) + 3-layer bidirectional attention + dual pooling heads (head 0 = symmetric STS/clustering, head 1 = retrieval). MTEB-ish numbers hold up: STS12 0.712 (91.9% of teacher), Banking77 0.837 (102.3% of teacher), SciFact 0.631, ArguAna 0.456.
The speed part is what we're most excited about. We rebuilt inference in pure Rust (BNNS/CoreML pre-packed GEMM graphs + NEON kernels) and on an Apple M4 CPU:
| VelaVec | bge-small teacher | |
|---|---|---|
| Query latency | 103 µs | 5.5 ms (53.9× faster) |
| Doc latency (128 tok) | ~580 µs (2-thread) | 13 ms (~22×) |
| Cold start | 5.4 ms | ~29 s (PyTorch) |
Along the way we hit three hardware lessons worth sharing:
- fp16 is a trap on Apple CPU — no AMX fp16 GEMM path for small shapes; fp16 was slower than fp32 (0.74–0.90×).
- int8 doesn't speed anything up — BNNS rejects Int8 outright; via CoreML int8 weights ≈ fp32, int8 w+a ≈ 2× slower. No public AMX int8 GEMM path.
- The real win: BNNSGraph with weight pre-packing — 2.5–3.1× over BLAS at the model's actual GEMM shapes (l=15), hitting ~2.4–3.0 TF/s single-thread AMX.
We also added a vision projection head (VelaVec-T2I): frozen CLIP b32+l14 → residual MLP into VelaVec space, only +1.5M params — text→img r@1 0.560 on flickr30k.
🤗 Model: huggingface.co/xagent2025/VelaVec
🖼️ Multimodal: huggingface.co/xagent2025/VelaVec-T2I
Happy to answer questions on the Rust/CoreML optimization details!