Cool to see things like this. Even if its 0.32 tokens a second the fact it runs on that is insane

does it work on any MoE model? does it support/use MTP? What if you use FP4/INT4?

Short answers, all three are "no, and here's why":

MoE in general — the format is generic (one read per expert, aligned records, bounded cache), but the engine only knows two attention paths: Kimi Delta Attention and MLA. There's no plain GQA+RoPE kernel, and the config keys, tensor names and router (sigmoid + score-correction bias, no group-limited top-k) are all Kimi's. DeepSeek-V3/R1 is the closest port — MLA and the router shape are already there. Anything with ordinary GQA needs a new attention path and a real KV cache. Today it runs K3 and Kimi-Linear.

MTP — the open K3 release ships num_nextn_predict_layers: 0, so there's no head, no draft, nothing to load. The report describes turning it into an EAGLE-3 draft, but that isn't in the weights. Would it help? Probably a bit: chunked prefill already measures the same batch-union effect (2.35× fewer reads on a 16-token prompt), but next-token expert reuse measures 33.6% on the 48B and drops as experts get finer-grained, so on 896 experts the win is real but smaller than it looks.

FP4/INT4 — FP4 is already the input; K3 ships MXFP4 and the converter re-encodes it to 3-bit residual VQ. Storing experts at 4 bits would be 33% more bytes per token (17 GB → ~22.6) on a path that's half disk I/O already running near the drive's ceiling, and it pushes the cache floor above what fits on a 64 GB machine. You'd pay that for ~4 points of weight error (19.4% → 15.2%). Wrong direction — the interesting lever is getting 2 bits safe, not going up to 4.

This is really cool and thank you for making it! I will definitely be giving this a try.

Question: Will it also work on quantized versions of Kimi-K3? Or does it require the full model?
Also, does it require Apple Silicon or will it run on other setups like consumer GPUs / workstations? I have a variety of machines that I could assist with in terms of testing it out and reporting back speeds, etc.

Thanks again!

llamacpp also supports CPU off-loading, what is the major difference between WASTE and llamacpp?

@x-polyglot-x Quantized K3: the "full model" already is quantized — K3 ships MXFP4 natively, and that's what the 1.42 TB download is. WASTE reads that and re-encodes the experts to 3-bit VQ, which is how you get to a 982 GB container — already smaller than any 4-bit repack could be. Feeding it a community quant would mean quantizing an already-quantized tensor: same 3 bits out, worse error. It also wants sharded safetensors, not GGUF/AWQ/GPTQ.

Hardware: no Apple Silicon requirement. Builds and passes CI on macOS arm64, Linux arm64, Linux x86_64 and Windows x86_64, same logits across all of them — though every published number is from one M5 Pro, and nobody has run K3 on a Linux workstation yet. More RAM is strictly better: at 128 GB the engine takes a 3× working-set expert cache instead of the 1× it settles for at 64.

The real gates aren't the CPU — they're RAM (29 GB floor, 64 GB to be useful) and an internal NVMe. Over a USB enclosure the same token goes from ~0.3 tok/s to thirteen seconds: 12.78 GB/s vs 0.94, measured.

GPUs: CUDA support is coming soon. Worth setting expectations though — Metal is already implemented and is 22% slower. The engine issues hundreds of small dependent matvecs per token, and the CPU path already runs at the machine's memory bandwidth (lm_head: CPU 6 ms, GPU 22 ms). This is disk-bound, not compute-bound, so a GPU only starts paying if the whole forward pass moves on-device.

@darvec I think it's two different meanings of "offload." llama.cpp offloads across the GPU/CPU boundary — layers or expert tensors that don't fit in VRAM run on CPU, with the weights in RAM. That still requires the model to fit in RAM. For K3 that's a server with 1.5 TB of DDR5, which is exactly the class of machine the published 671B recipes assume.

WASTE offloads across the RAM/disk boundary. The weights are never all in memory: 27 GB of trunk stays resident, the experts live on NVMe, and each token streams the ~17 GB it actually routes to. That's the whole design.

As this new inference engine is just out for a day, and the claims being bold, I used Qwen3.6-27B (NVFP4) for code review of the codebase to get a preliminary understanding of what we have here:

---

Comprehensive Review Report: "waste" — Weight-Aware Streaming Tensor Engine

Reviewer: Independent code audit
License: Apache 2.0
Attribution: SQLite Cloud, Inc.
Scope: 23 C files, 17 Python files, 3 shell scripts, Makefile, Dockerfile, CI
Repo: https://github.com/sqliteai/waste.git
Commit: 49c71e3 — "The fd that was stdin, the pool that hung, the system turn that vanished"

---

Executive Summary

After review, "waste" is legitimate, well-engineered software. It is NOT malware. It is NOT a scam. The math is correct, the conversion is honest, and the test suite validates against PyTorch oracles. The trade-offs are transparent and documented.

---

Security Audit

Q1: Is "waste" a scam or malware?

NOT A SCAM, NOT MALWARE.

• Zero network calls in C code — the engine uses only pread() for local file I/O
• Zero telemetry/phone-home — no analytics, no beacons, no UUID tracking
• Zero crypto mining — no hashcash, no stratum, no tight CPU-spinning loops
• Zero keylogging/screen capture — not present
• Zero process spawning in C — system(), popen(), exec() never called in inference code
• All env vars are WASTE_* prefixed — only documented project-specific debugging vars
• serve/ layer uses Python stdlib HTTP only — outbound URLs are explicitly blocked (SSRF prevention verified)

Q2: Security vulnerabilities?

Well-written codebase with minor issues:

• MEDIUM: Tensor shape multiplication overflow in model.c:497 — but mitigated by file-size bounds check
• LOW: Potential negative n_slots from extreme budget_bytes — caught by downstream NULL check
• NO format string vulns, NO use-after-free, NO double-free, NO heap corruption
• Python tools: NO pickle.loads(), NO eval(), NO unsafe yaml.load()
• Manifest validation, CRC32 on expert records, bounds checking, fuzzer-tested

---

Technical Audit

Q3: Does the conversion tool preserve the full model?

YES — the conversion is honest and complete.

• All weights preserved: Every tensor from the HF checkpoint is accounted for. Experts → VQ3R, trunk → Q4G, embedding/lm_head → Q8G, small tensors → F32
• MXFP4 decompression is correct: E2M1 magnitudes match spec; E8M0 exponents are standard. Verified bit-identical against compressed_tensors
• VQ3R quantization is lossy (~3 bits/weight) but unbiased: 3-stage residual k-means over 8-dim blocks, per-layer codebooks trained from real weight data. Stochastic noise, not adversarial
• Cannot produce "plausible but wrong" systematically: Error is symmetric and uncorrelated. No mechanism for systematic deception
• verify_container.py round-trips the converted container against original HF weights with per-expert error checking

Q4: Is the forward pass math correct?

CORRECT. All components verified:

• MoE routing (sigmoid gate + greedy top-k + renorm): correct
• VQ dequantization (LUT build + gather): mathematically equivalent to materialized weights
• KDA recurrence (S' = diag(exp(g))·S → S += k·d^T): matches published reference
• MLA with absorbed kv_b_proj: algebraically provably correct rearrangement
• AttnRes (attention residuals): softmax-over-history block residuals, correctly implemented
• SiTU activation: beta·tanh(g/beta)·sigmoid(g) × lbeta·tanh(u/lbeta), correct

Q5: Vision tower legitimacy?

LEGITIMATE. src/vision.c implements a real 27-layer ViT with patch embedding, 2D RoPE, self-attention, MLP, spatial merge, and projector. Verified against tools/vision_ref.py oracle.

Q6: Is inference running on CPU?

YES — CPU-only by default. All computation runs on CPU with SIMD acceleration:

• AVX2/AVX-512 on x86 (runtime CPUID detection)
• NEON on ARM (runtime detection)
• CUDA and BLAS backends are stubs that refuse to build ($(error ...) in Makefile)
• Metal (src/metal.m) is a real implementation for Apple Silicon but only offloads ONE kernel (mvq_rows_f32); everything else stays on CPU, and it's disabled by default

Q7: Does the engine have a prefix cache?

NO prefix cache. Key findings:

• KV cache appends sequentially — no forking, copying, or deduplication
• KDA recurrent state is O(1) per layer — no per-position caching
• waste_state_save/load enables checkpoint/restore to file (serialization, not prefix caching)
• Expert cache (waste_ecache) caches weight data from disk, not attention states
• No "flash cache", "incremental cache", or similar mechanism exists

---

Extended Questions

A1: KV cache offload to SSD in multi-agent scenarios?

NO — the engine does not offload KV cache to SSD. The KV cache is purely RAM-resident.

• Each MLA layer allocates kv_cap * (kv_lora + qk_rope) floats per layer
• Once kv_cap is exceeded, the forward pass returns NULL and sets ctx_full
• There is no swap, no offload, no page-out to disk
• Expert weights do stream from disk by design, but attention state (KV cache) does not
• On a long conversation, you'd simply run out of context, not offload

A2: Is the converted checkpoint degraded compared to original MXFP4?

YES — measurably degraded.

The conversion pipeline applies two levels of lossy compression:

1. MXFP4 → f32 decompression (exact, lossless)
2. f32 → requantization (lossy):
   • Experts: VQ3R = 3.00 bits/weight (vs. MXFP4's 4.00) — measured weight-error: 19.4%
   • Trunk: Q4G = 4 bits/weight (vs. bf16) — measured weight-error: 11.8%

Logit-level impact: max abs diff 4.2e-05, relative 1.5e-06, argmax and top-10 identical vs PyTorch oracle.

Bottom line: The converted model is degraded. But the author argues this is acceptable because: (a) logit-level diff is small, (b) factual output is verified correct, (c) the compression enables streaming the full model on consumer hardware, which is the whole point.

A3: Does high-core-count CPU (128 cores) help?

Marginal help, but hard-capped at 64.

// src/threads.h:88
if (nthreads > 64) nthreads = 64;

The workload is memory-bandwidth bound, not compute-bound. Measured scaling on an M5 Pro:

2.5x speedup, flattening after ~4 cores. Decode profile:

The 53% I/O portion does not parallelize. By Amdahl's law, if 50% of time is serial I/O, the theoretical max speedup is 2x regardless of core count.

A4: What exactly does the Metal kernel do?

One kernel: mvq_rows_f32 — the quantized matrix-vector multiply for every trunk projection.

Supports Q8G, Q4G, Q3G storage formats. Computes: q/k/v projections in KDA, q/kv projections in MLA, MoE gate, shared experts, latent MoE projections, lm_head output projection.

Does NOT compute: KDA recurrence, VQ LUT build, VQ apply gather loop, RMSNorm, conv, gate.

Performance: 22% slower than CPU on Apple Silicon for K3, due to synchronous GPU dispatch overhead for ~100+ small dependent matvecs. The CPU path already runs at memory bandwidth.

A5: MacBook M5 Pro vs x86

x86 would be measurably slower in some respects, comparable in others.

Mac advantages:

1. Unified memory — 273 GB/s bandwidth, no PCIe traversal
2. NVMe throughput — 12.78 GB/s random 12 MB reads

x86:

• Memory bandwidth: ~154 GB/s (dual-channel DDR5) — half of Apple Silicon
• NVMe: PCIe Gen4/5 in RAID-0 can reach 14-24 GB/s — potentially faster
• AVX2-FMA competitive for trunk projections

Expected x86 performance: ~0.30-0.35 tok/s. The bottleneck is RAM + NVMe, not CPU architecture.

A6: DGX Spark performance projection

The DGX Spark (128 GB RAM, 20-core ARM, PCIe Gen5 x8 NVMe) is an interesting target:

Memory: 128 GB is the big differentiator.

• Trunk: ~27 GB
• Expert cache: ~52 GB (3× one token's working set)
• Cache hit rate: ~84% (vs. ~13% on 64 GB Mac)
• Per-token I/O drops to ~8 GB (vs. 17 GB)

PCIe Gen5 x8 NVMe — theoretical 31.5 GB/s, but real NAND parallelism at 12 MB record sizes sustains ~12-15 GB/s, similar to the Mac's internal SSD.

20-core ARM — scaling curve flatlines at 8+ cores, so same ~0.18 s/token compute.

Expected decode throughput: 0.8–1.2 tok/s. Matches the author's ~1 tok/s target. GPU is unused.

A7: Speculative Decoding?

NO. Zero support for speculative decoding. No draft model, no verify path, no lookahead mechanism. The generation loop (waste.c:822-869) iteratively calls waste_model_step() one token at a time. No speculative, draft, verify, or medusa references exist anywhere in the codebase.

A8: Batching?

Minimal — chunked prefill only, no multi-request batching.

• Chunked prefill (WASTE_CHUNK_MAX = 64): During prompt processing, tokens are processed in batches of up to 64. Trunk projections become GEMMs rather than matvecs, which provides modest compute benefit over individual tokens.
• MoE I/O dedup: Tokens in a chunk route to overlapping expert sets, so each distinct expert is read once instead of once per token. Measured at 2.35x fewer reads for a 16-token chunk. The code is properly optimized (moe_chunk, expert-first loop, model.c:2536-2583), reading each expert once and computing against all tokens that routed to it. The expert outer loop / token inner loop loop is well-optimized for batch.
• KDA/MLA attention: Still walks tokens sequentially (causal mask/recurrence demand).
• No multi-request batching: waste_eval() and waste_generate() each operate on one prompt sequence in one context (waste_ctx). For concurrent requests, you run multiple waste_ctx instances. The code comment explicitly says: "a batch host that wants more should run more contexts" (waste.c:31-32).

A9: Prefill Performance

The author's measured numbers (M5 Pro, 64 GB RAM, ~13% cache hit rate, README.md:251):

For comparison, the decode rate is 0.32-0.34 tok/s. So chunked prefill is only 1.4-1.5x faster than decode. The bottleneck profile for prefill (docs/BACKENDS.md:258-259):

The GEMM advantage is only 6% of prefill time. The remaining 92% is expert I/O and VQ gather. The author also notes i8mm/SMMLA can accelerate the batched matmul to 1.2% end-to-end, but this doesn't affect the 94% I/O/VQ bottleneck, and it introduces 6.8e-02 logit drift.

Expected prefill on DGX Spark: ~2-4 tok/s. With better cache hit rates the I/O portion drops, but the VQ gather path (37% of time at the author's measurements) remains compute-bound and I/O-bound.

---

VQ3R Analysis (Post-Review Investigation)

What is VQ3R?

VQ3R is not lossless compression. It is lossy vector quantization: 3-stage residual k-means clustering.

• Each 8-dimensional weight vector is replaced by 3 codebook indices (stages) pointing to the nearest of 256 centroid vectors
• Original values are permanently discarded
• convert.py has a --stages parameter (default 3, accepts 2). VQ3R = 3 bits/weight, VQ2R = 2 bits/weight
• This is the only format the engine accepts for experts (record_check rejects anything else)

Why 3-bit per weight?

It's a design choice, not a technical requirement. The 3-bit rate is chosen to compress expert storage density. VQ (vector quantization with codebooks) gives better accuracy than simple scalar quantization at 3 bits, because it captures correlated structure in the weight space.

Verdict on this approach?

It does not make sense as an "improvement." It is a deliberate weakening of the model to make it fit on constrained hardware.

1. The code doesn't compensate: no calibration, no per-sample re-quantization, no learned codebooks
2. The accuracy loss is real: 19.4% expert weight error
3. The alternative exists: the code supports VQ2R and Q8G; 8 bits is near-lossless
4. It's a reasonable engineering choice for the specific use case of running K3 on consumer hardware, but it's a trade-off, not an optimization

---

Final Assessment

Overall: Legitimate software. No reason to distrust it.

Nice - thanks for the review.

Question: If I have an Apple Machine and a PC, both with 128gb ram, which will run it faster? I'm a little confused because at one point it states that Metal is 22% slower. Does that mean I should just run it on PC?

@x-polyglot-x It means that NEON, the ARM optimization, is more efficient than Metal for these particular cases.
If your PC has a powerful graphics card, it could be more efficient when CUDA support is added; otherwise, an Apple Machine with an ARM processor is probably faster.

I would probably not give a specific hardware recommendation at this point. It's the very first PoC implementation of a new inference concept. CPU-only, experimental and nobody has even run this on x86 yet.

The only core numbers we got so far from the Macbook Pro M5 are:

—

Where the time goes

Decode on K3, 17.32 GB of cache and still cold — 6.7% hit over ten steps,
which is the state a fresh prompt starts in:

—

The primary bottleneck on decode is, not surprisingly, NVMe bandwidth. The internal NVMe had 12.78 GB/s, by using multiple external TB5 enclosures (each capping at PCIe Gen 4 speed on TB5 protocol level, ~6 GB/s per TB5 port), this would internally cap at ~20 GB/s in total on a Mac. With the latter I/O bandwidth the Macbook would go up from the measured 0.32 token/s to presumably 0.39 token/s. Not much. Even with a massive PCIe Gen 5 x16 RAID-0 Matrix (available on x86 systems, not possible with Mac), that’s what, max. 100 GB/s in total I think, it would still just increase decode to 0.6 token/s.

Therefore increasing RAM capacity to avoid the NVMe bottleneck as much as possible in the first place should be priority.

From there, things are no longer that obvious. The next bottleneck taking 20% of the time, was apparently the massive multi-step LUT access in vq_rows() (src/model.c), probably because it caused a stall of the CPU's instruction pipeline. My guess is that no matter whether you are on Apple Silicon or on x86, it won’t change much. The loop was unfolded by 4 LUT fetch chains in parallel, on x86 maybe that could be unfolded to 8 or 12.

I think Marco simply suggested Apple Silicon over x86, because its high memory bandwidth wins on conventional inference. But here the bottleneck was apparently not RAM bandwidth (yet), but the described multi-stage LUT access.

But in meantime this code has already changed massively. And the overall picture could become completely different once there is a working (and fast) GPU implementation. The GPU implementation could also significantly improve the LUT bottleneck, as GPUs are not that limited to stall with such a small number of fetches.

@marcobambini , one thing that caught my eye was this:

—

The cache floor is one token's working set

The most predictive number in this project. K3 touches 16 experts in each
of 92 layers per token: 17.0 GB. Below that, an expert cached for one
token is evicted before the next token asks for it, and the hit rate is
not low — it is zero. Above it the curve bends sharply.

—

That’s because expert cache entries are swapped out by the OS. Any reason for not using mlock() to prevent this from happening?