kingjones777/Macaron-V1-Tall-NVFP4

Macaron-V1-Tall — NVFP4 (with MTP, LoRA-compatible)

This is an NVFP4 quant of mindlab-research/Macaron-V1-Tall, built and tested on an NVIDIA GB10 (ASUS Ascent GX10, 121 GiB unified memory, CUDA 13.0) running vLLM.

Macaron-V1-Tall is a Mixture-of-LoRA model — a Qwen3.6-35B-A3B MoE base plus four LoRA specialists. That combination made it the interesting one to quantize: it isn't enough to compress the base, the specialists have to still work on top of it. They do, and I verified that rather than assuming it.

Short version: 103 GB of BF16 becomes 23.3 GiB, runs at 58.7 tok/s with MTP (1.41× over no-MTP on the same box), and all four LoRA specialists load and apply correctly on the quantized base.

This is the base — the LoRA specialists are NOT in this repo

Macaron's four specialists live in the original repo under loras/L0 … loras/L3 and are not part of this quant. You need both:

hf download kingjones777/Macaron-V1-Tall-NVFP4  --local-dir ./macaron-nvfp4
hf download mindlab-research/Macaron-V1-Tall --include "loras/*" --local-dir ./macaron-base

adapter	role	size
L0	Chat / instruction following (also the router upstream)	7.03 GiB
L1	Agent — tool use, personal-agent work, long-running workflows	7.03 GiB
L2	Coding — repositories, terminal, software engineering	14.07 GiB
L3	GenUI — dashboards, interfaces, generative UI	7.03 GiB

All four are rank 64 / alpha 128, targeting q/k/v/o_proj, gate/up/down_proj, linear_attn.in_proj_* and shared_expert_gate.

⚠️ The adapters are BF16 and stay BF16 — they are not quantized and cannot be. vLLM's lora/ package ships no quantized-LoRA kernel and the fused-MoE LoRA path expects unquantized adapters. FP8 or INT4 adapters will not load. Budget the full 35 GiB on disk (28 GiB if you take the BF16-cast L2, see below) on top of the 23.3 GiB base.

Do the specialists actually work on a 4-bit base? Yes — measured

This was the real question. The adapters put 160 tensors onto fused MoE experts and 240 onto Qwen3.5's linear_attn.in_proj_* — exactly the weights compressed to 4 bits. vLLM handles it. The startup log shows:

MoE model detected. Using fused MoE LoRA implementation.
Loaded new LoRA adapter: name 'L2'
Triton kernel JIT: _fused_moe_lora_one_shot_kernel

and, more importantly, base / L0 / L2 produce different output on identical greedy prompts — which is only possible if the adapters are genuinely modifying the forward pass. A LoRA that loads but silently no-ops would give byte-identical text.

⚠️ vLLM serves the adapters, it does not route between them. Upstream's L0 router picking a specialist per request is a property of their MoL serving harness, not of vLLM. Here you select the adapter by passing its name as the model field.

⚠️ Running with an adapter costs about a third of the throughput — ~37.7 tok/s with a LoRA active versus 58.7 on the bare base. The fused-MoE LoRA kernel runs on every forward pass. That's the price of the specialists and it's worth knowing before you size anything.

Running it

docker run -d --name vllm-macaron --gpus all --network host \
  --ipc=host --shm-size=16g -v /path/to/models:/models \
  -e CUTE_DSL_ARCH=sm_121a \
  vllm/vllm-openai:nightly-aarch64 \
    /models/Macaron-V1-Tall-NVFP4 \
    --served-model-name macaron --port 8000 --trust-remote-code \
    --max-model-len 65536 --max-num-seqs 2 \
    --gpu-memory-utilization 0.55 --kv-cache-dtype fp8 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --enable-lora --max-lora-rank 64 --max-loras 1 \
    --lora-modules L0=/models/loras/L0 L1=/models/loras/L1 \
                   L2=/models/loras/L2 L3=/models/loras/L3

Then call "model": "L2" for coding, "L1" for agent work, and so on; "macaron" hits the bare base. Raise --max-loras to keep more than one specialist resident — at 4 you need headroom for ~35 GiB of adapters on top of the base.

Three things cost me time:

The image entrypoint is already ["vllm","serve"]. The first argument is the model path. Passing the word serve produces vllm serve serve /models/... and fails in a way that looks like a broken checkpoint.

Use "method":"mtp". You'll see Resolved architecture: Qwen3_5MoeMTP in the log when the MTP block is found — that's your confirmation.

Set --kv-cache-dtype fp8 at serve time. I could not bake kv_cache_scheme into the checkpoint: this model nests its text settings under text_config, so there's no top-level num_attention_heads and llm-compressor's KV path fails looking for it. The flag does the same job at runtime.

The MTP layers are in here, and that took an extra step

The base ships 19 mtp.* tensors (spread across shards 25–26). llm-compressor drops them — it loads the model through a class that never instantiates the MTP layers, so they never reach the output. A first pass produced a checkpoint with zero MTP tensors and no speculative decoding.

I grafted them back at BF16 and added them to quantization_config.ignore. They are deliberately not quantized: a drafter with degraded numerics produces bad drafts and tanks acceptance, which costs more than the 1.6 GiB it saves.

Benchmarks

One GB10, vLLM 0.26.1rc1, 64K context, FP8 KV, max_num_seqs 2, 256-token generations, nonce-prefixed so nothing came from the prefix cache. Warmed before measuring. Medians of 3–4 runs.

	tok/s
NVFP4 + MTP	58.71
NVFP4, no MTP	41.63
Speedup	1.41×
MTP acceptance rate	77.4% (466 accepted / 602 drafted)
With a LoRA adapter active	~37.7

Same checkpoint, same box, same flags — the only difference is whether --speculative-config is passed. The no-MTP runs were tight (41.5, 41.6, 41.7, 41.7).

Warm up before measuring. A cold first request underreports badly — my initial reading on the sibling model was 20% low and I nearly published it.

Correctness: 17*23 → 391, capital of Japan → Tokyo, coherent 256-token freeform prose. A badly quantized MoE loads fine and then emits repetition or token soup, so "it loaded" proves nothing.

No BF16 A/B. I did not benchmark the 103 GB original on the same box, so I'm not quoting a quality delta against it.

Calibration disclosure

This model was calibrated with 128 samples at 1024 sequence length. My Frontis-MA1-35B NVFP4 used 256 @ 2048. Both pass the same correctness gates, but they are not identically calibrated and I'd rather say so than have the two look equivalent. If you want a heavier calibration run, recipe.yaml in the repo has the exact parameters to change.

Optional: BF16-cast L2

L2 ships as FP32 (14.07 GiB) while L0/L1/L3 are already BF16. vLLM loads adapters as bfloat16 regardless, so I cast L2 down and tested whether it costs anything.

I ran both through 15 hard problems — 10 algorithmic tasks where the generated code was executed against assertions (LRU cache, edit distance, median of two sorted arrays, hand-rolled regex with ./*, N-queens, trapping rain water, word break, merge-k-sorted, LIS, coin change) plus 5 exact-answer math problems.

	FP32 L2	BF16 L2
Code, executed and passed	8/10	8/10
Math, exact	3/5	3/5
Total	11/15	11/15
Tokens	31,632	31,747

The same problems passed and the same ones failed. (Three of those four failures were my harness, not the models — one math key was wrong, one answer was truncated at the token cap, and two "failures" were my code extractor mangling indentation. They hit both precisions identically.)

Output is not bit-identical — greedy decoding diverges on wording — but capability is equivalent on an execution-scored test, which was the bar.

⚠️ This saves disk, not VRAM. vLLM loads adapters as BF16 either way, so serving footprint is unchanged. The win is 35.16 → 28.13 GiB on disk, and only L2 is affected.

The model

Macaron-V1-Tall is MindLab Research's Mixture-of-LoRA model built on Qwen3.6-35B-A3B — 35B total / ~3B active, 40 layers, 256 experts with 8 active per token, 262K context. Four LoRA specialists cover chat, personal-agent work, coding and code-native Generative UI, with an L0 router selecting per request in their own harness.

Everything about behaviour, prompt format and capability comes from upstream. See the base model card and the blog post. All I changed is the numeric format of the base.

MIT, same as the base model.

Credit

The model is MindLab Research's; the NVFP4 format and kernels are NVIDIA's; the quantization tooling is llm-compressor from the vLLM project. I built the quant, grafted the MTP block, verified the LoRA specialists on the quantized base, ran the benchmarks and wrote down what I found.