Inkling-mini-nvfp4

This is a broken test fixture, not a usable model.

It is a deliberately crippled cut-down of thinkingmachines/Inkling-NVFP4, built only to exercise inference-engine plumbing. Most of the network has been deleted: 12 of 66 layers and 16 of 256 experts remain. It produces gibberish. Do not use it for generation, do not evaluate it, and do not treat any output as representative of Inkling. If you want the real model, use thinkingmachines/Inkling-NVFP4.

Attribution

All weights here are the work of Thinking Machines Lab, sliced without modification from thinkingmachines/Inkling-NVFP4 (revision main) and redistributed under that model's Apache-2.0 licence.

To be precise about what "minimized" means: the retained tensors are the real trained values, bit-for-bit — nothing is randomly initialized and no weight is altered. The reduction is purely deletion, keeping a prefix of the layers and the first 16 experts of each MoE layer. That is what makes the fixture useful for shape and dataflow testing, and it is also why upstream attribution and licensing apply in full.

No claim is made about this artifact's quality, and no endorsement by Thinking Machines Lab is implied.

Routed experts are NVFP4 (packed uint8 weights with F8_E4M3 block scales at block size 16, per-expert F32 scale2, and a BF16 input_amax), except layer 2, whose experts are BF16 upstream and stay BF16 here. Attention, gates, shared experts, embeddings and norms are all BF16, matching the 479-entry exclude_modules list in hf_quant_config.json. Requires Blackwell FP4 tensor cores.

What was reduced

	upstream	here
text layers	66	12
global attention layers	11	2 (at 5, 11)
dense MLP layers	0-1	0-1
routed experts	256	16
experts per token	6	6
shared expert sinks	2	2
MTP layers	8	2
hidden size / heads / head_dim	6144 / 64 / 128	unchanged
vocab	201024 (200058 unpadded)	unchanged

19.88 GiB, 370 tensors, 6 shards.

The retained layers are a prefix of the upstream schedule, so layer indices, local_layer_ids and dense_mlp_idx carry over unchanged and the real 5:1 sliding-window / full-attention alternation is preserved rather than hand-written. Hidden size, head counts and head_dim are untouched so kernel shapes match production. The vision and audio towers are complete.

Why the output is gibberish

This is expected and by design, not a bug to report. At 12 of 66 layers and 16 of 256 experts the output distribution is broad and near-uniform. Greedy decoding collapses onto a single special token; temperature 1.0 returns unrelated vocabulary tokens. Logprobs are finite and greedy decoding is deterministic, which is all the fixture needs to be.

This fixture validates plumbing — tensor shapes, the four short-convolution sites per layer, per-layer-type KV head counts and relative-bias widths, gate-plus-sink routing, embedding norm, muP logit scaling, the sampler — and it is a suitable input for a logit-level comparison against a reference implementation. It does not validate correctness on its own, and it should not be evaluated for quality.

Serving

Verified with vLLM 0.26.0 on a single B200:

export VLLM_USE_V2_MODEL_RUNNER=1
export MAX_JOBS=16                # FlashInfer's ninja JIT otherwise fans out
                                  # to the host CPU count and can OOM the box
vllm serve Inkling-mini-nvfp4 \
  --tokenizer-mode inkling \
  --tensor-parallel-size 1 \
  --kernel-config.enable_flashinfer_autotune=False \
  --max-model-len 4096 \
  --max-num-seqs 16

--max-num-seqs matters: vLLM allocates max_num_seqs * max_model_len * 4 B of pinned host memory, and the default can stall cudaHostAlloc for minutes. scipy is an undeclared runtime dependency of the vision tower. At TP=1 the fused Lamport collective logs a ValueError: TP world size must be 2, 4, or 8 traceback at ERROR level; it is a benign fallback to NCCL, not a failure.

Licence

Apache-2.0, inherited from thinkingmachines/Inkling-NVFP4. The weights are Thinking Machines Lab's; this repository only removes layers and experts from them.