seekr-org/Inkling-2-Small-FP8-ROCm

Inkling 2 Small FP8 for AMD MI300X

A row-scaled E4M3FNUZ conversion of Thinking Machines' Inkling 2 Small MXFP8, revision 3febc62259ee92ea36e2fb6e8303371c4eb3922c. The original model was developed by Thinking Machines Lab. This repository contains a precision conversion, not a new trained model. The upstream model card is preserved in UPSTREAM_README.md.

Precision and files

The 96 expert projection tensors use native AMD E4M3FNUZ FP8, with one FP32 scale per expert output channel. Activations use dynamic per-token E4M3FNUZ quantization. Weights are dequantized from the original MXFP8 block scales and requantized with scale = max(abs(row)) / 240. Attention, dense projections, vision, and the DSpark draft retain the upstream precision. This is mixed precision inference.

All 48 target weight shards, tokenizer/configuration files, vision weights, and mtp.safetensors are included. The checkpoint is approximately 605.14 GB. conversion_manifest.json records the source revision, per-shard tensor sizes, and serialization verification fingerprints. The original checkpoint is unchanged.

hf_quant_config.json marks converted experts with compute_dtype: fp8; config.json selects expert_fp8_scale_mode: row. Expert scale tensors retain the _weight_scale suffix and have shape [384, output_channels], dtype FP32. The loader reads the converted weights directly without requantizing them.

Serving

Use the ROCm vLLM image built by the Seekr GitLab pipeline from the source tree containing Inkling 2 and prequantized row-FP8 support. The Docker build validates that the wheel and installed package contain the current Inkling implementation, renderer, and model registrations. No separate patch is distributed here. This checkpoint requires that implementation; an unmodified vLLM installation is not a supported loader for this format.

Validated hardware: 8 AMD MI300X GPUs (gfx942), 192 GiB per GPU. Other GPU architectures have not been validated.

Inside the deployment image:

export OMP_NUM_THREADS=1
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_CUSTOM_AR=1

vllm serve seekr-org/Inkling-2-Small-FP8-ROCm \
  --quantization inkling2 --tensor-parallel-size 8 \
  --max-model-len 262144 --max-num-batched-tokens 2048 --max-num-seqs 8 \
  --no-enable-prefix-caching \
  --reasoning-parser inkling --limit-mm-per-prompt '{"image":2}' \
  --gpu-memory-utilization 0.9 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","decode_cudagraph_capture_request_sizes":[1,2,4,8],"use_breakable_cudagraph":false}' \
  --kernel-config '{"enable_jit_warmup":false}'

The image's entrypoint is vllm serve. With Docker, expose /dev/kfd and /dev/dri, use host IPC, mount a persistent Hugging Face cache, and pass the model name and flags above as container arguments. Set HF_TOKEN in the container environment if required; no credentials are included in this repo.

Images use standard image_url chat content parts. To request direct answers, set chat_template_kwargs.reasoning_effort to 0. The default permits reasoning.

Measurements and accuracy

Prior end-to-end measurements of this row-FP8 quantization used 24 distinct math/programming/prose tasks, about 2K input and 512 completion tokens per request, temperature 1, default reasoning, prefix caching disabled, and the seven-token probabilistic DSpark configuration above:

Metric	Row FP8	Original MXFP8 scales
C1 output tokens/s	247.47	141.20
C8 aggregate output tokens/s	800.03	348.82
GSM8K, fixed 50-problem sample	48/50	48/50
HumanEval, fixed 20-problem sample	19/20	20/20

Throughput includes prefill, the final partially occupied batch, and reasoning tokens. These are short-prompt results, not throughput at 256K input. The C8 1,000-token/s goal was not reached. Prose alone averaged 198.89 tokens/s at C1.

The paired accuracy check used temperature 0, reasoning effort 0, and a 2,048-token generation limit for each mode. The row-FP8 coding miss exhausted its output budget. One GSM8K reference answer is arithmetically incorrect; raw scores preserve that label. These small samples do not establish full benchmark parity, and greedy continuations can vary with batching.

Both quantization paths passed a 261,040-input-token recall check and text, image colors, OCR, multiple-image, and streaming checks before serialization. The exported checkpoint's additional load validation is recorded in validation_results.json.

License and attribution

Apache License 2.0, inherited from the upstream checkpoint. See LICENSE. Original model: Thinking Machines Lab. This repository converts the expert precision for AMD MI300X deployment and retains the upstream model files and attribution. Model purpose and upstream information are in UPSTREAM_README.md.