Ling-3.0-Flash-CIRU-int4-Strix-native

Ling-3.0-Flash-CIRU-int4-Strix-native is CIRU's native vLLM/ROCm runtime distribution for InclusionAI's official packed-INT4 Ling 3.0 Flash checkpoint on AMD Strix Halo (gfx1151).

The model weights were not changed, requantized, merged, fine-tuned, or renamed internally. This release combines the original checkpoint at revision ca3ea63b0255d212c4fe6020db9e0a51ce136006 with a pinned vLLM fork, ROCm 7.15 build recipe, and launch profiles validated on an AMD Radeon 8060S with 128 GB unified memory.

A 97.5x native ROCm serving campaign: the first working upstream-compatible build decoded at 0.269 tok/s; this distribution reaches 26.23 tok/s with native MTP K1, while retaining the official vendor checkpoint, OpenAI-compatible serving, continuous batching, and Ling's structured tool parser.

What this release changes

The runtime is based on vLLM commit d35eb6c44071ea806018841c490f0d2f3219c485 plus a 15-commit CIRU branch ending at 388d82de2abe418e73380ba250270d3abc62ac12. Its net source delta is six files, 365 insertions, and 26 deletions.

• Includes liminfei-amd's Wave32 LDS fix for the ROCm top-k merge launch, preserving the AMD author's commit credit.
• Adds an opt-in safe PyTorch attention-state merge for a gfx1151 Triton/HSA fault, including both supported MLA LSE layouts.
• Routes causal uniform multi-token MLA verification blocks through Triton decode with correct per-token sequence lengths.
• Adds narrowly guarded W4A16 MoE decode/verification fast paths for Ling's exact small-token geometry.
• Adds a gfx1151 Triton SiLU-and-multiply kernel for Ling's exact eager W4A16 shapes.

See CHANGELOG.md for the public file-level runtime diff.

Performance: 97.5x over the public starting point

The closest runnable public/upstream starting point was frozen vLLM commit d35eb6c44071ea806018841c490f0d2f3219c485 plus the publicly proposed five-line ROCm sampler fix required to launch on gfx1151. With the unchanged official model, that path produced 0.268944 tok/s in a warmed deterministic target-only decode test. This release reaches 26.2343 effective tok/s with the checkpoint's native K1 MTP: 97.55x the starting throughput, or approximately 100x.

Public-to-release progression	Decode throughput	Multiple vs. starting point
Upstream-compatible starting path, target only	0.268944 tok/s	1.00x
This release, target only	21.4407 tok/s	79.72x
This release, native MTP K1	26.2343 effective tok/s	97.55x (~100x)

The starting row is a warmed deterministic eight-token test; the final rows are medians from deterministic 32-token tests. The multiplier therefore describes the full optimization campaign on the same model and host, not a single-variable matched A/B. The final exact-shape kernel step was also tested separately against the already-optimized CIRU parent: target-only decode rose from 19.8993 to 21.4407 tok/s (+7.7457%), and K1 rose from 25.1348 to 26.2343 tok/s (+4.3743%). That smaller percentage is the final incremental step, not the overall release gain.

MTP acceptance remained 82.35% with 1.8235 mean tokens per step. Retained controls had exact token IDs/text and zero faults. These are local single-host measurements, not claims for other AMD architectures.

How 0.269 became 26.23 tok/s

Stage	Decode throughput	Incremental result
First working upstream-compatible build	0.269 tok/s	starting point
Disable pathological ROCm skinny-GEMM dispatch on Wave32	7.57 tok/s	28.16x
vLLM compilation mode 3, decode size 1, graphs off	11.21 tok/s	+48.1%
Coherent ROCm 7.15 / Torch 2.13 / Triton 3.8 stack	14.84 tok/s	+32.4%
W4A16 MoE expert assignment and finalize reduction	19.71 tok/s	+42.4%
Repair optimized multi-token verifier routing; native MTP K1	25.13 tok/s	+27.3%
Exact-shape gfx1151 SiLU-and-multiply kernel	26.23 tok/s	+4.37% K1; 97.55x overall

The largest single problem was silent selection of vLLM's ROCm skinny-GEMM path on Wave32: disabling it alone improved identical-output decode by 28.16x. The remaining gains came from compilation, a coherent AMD software stack, Ling-shaped W4A16 MoE paths, repaired MTP verification routing, and the final exact-shape activation kernel.

Serving results

On exact 2K prompts with 256 forced output tokens, K1 MTP, and simultaneous bursts:

Concurrent requests	Aggregate prefill	Aggregate decode	Per-request decode
1	521.06 tok/s	26.33 tok/s	26.33 tok/s
2	503.31 tok/s	38.90 tok/s	19.73 tok/s
4	508.15 tok/s	59.03 tok/s	15.34 tok/s
5	468.30 tok/s	62.16 tok/s	14.40 tok/s

The C4 row reproduced at 59.15 and 59.03 aggregate decode tok/s across independent loads. For context, CIRU's local llama.cpp Vulkan GGUF reference reached 51.08 tok/s single-stream with MTP: this native vLLM stack is slower single-stream, but exceeds that single-stream reference at C4 and C5 while using the official checkpoint and vLLM serving stack.

The runtime also sustained 540.19, 550.20, and 536.13 prefill tok/s at 2K, 8K, and 16K in clean long-prefill tests on an earlier retained build; treat those rows as a floor rather than a claim measured again on the final commit. The final stack completed CIRU's 20-scenario structured tool-call suite using Ling's native parser.

60K PP/TG comparison

This retained comparison uses one frozen 60,000-token prompt followed by 512 forced output tokens:

Stack	PP	TG	Request wall
CIRU native K1 MTP	419.33 tok/s	19.92 tok/s	168.75 s
AtomicChat AD-IQ4_XXS, llama.cpp Vulkan	265.63 tok/s	25.11 tok/s	246.28 s
ROCmFP4 STRIX MTP, llama.cpp Vulkan	249.23 tok/s	19.00 tok/s	267.84 s

Native vLLM gives up 20.66% TG to Atomic at 60K, but its 57.87% PP lead completes the combined request 31.48% sooner. It leads ROCmFP4 by 68.25% PP, 4.88% TG, and 37.00% in combined wall time.

The native row predates the release's final SiLU kernel commit, so this table is a retained baseline rather than a final-commit maximum. See docs/BENCHMARKS.md for the exact protocol, source identities, and refresh rule.

Files in this package

• runtime/Ling-3.0-Flash-CIRU-int4-Strix-native-vllm.bundle: exact Git objects for the CIRU branch, based on the pinned upstream commit.
• runtime/Ling-3.0-Flash-CIRU-int4-Strix-native-vllm.patch: reviewable net source diff.
• install.sh: guided human installer plus a flag-driven interface for agents and automation.
• scripts/download-model.sh: downloads only the unchanged checkpoint files from this jcbtc repository, pinned to the verified model revision, into a parent directory or exact --model-path.
• scripts/install-host-deps.sh: distro-aware Linux/WSL host dependency installer.
• scripts/build-vllm-gfx1151.sh: reproducible build entry point for the pinned AMD wheel stack.
• scripts/run-256k.sh: native 256K, native-MTP K1 server profile with five active sequence slots.
• scripts/run-1m-yarn-experimental.sh: experimental 4x YaRN 1M profile with two active sequence slots.
• MANIFEST.json: machine-readable identities and scope.

The named release archive is Ling-3.0-Flash-CIRU-int4-Strix-native.tar.gz. It contains the runtime distribution, image, and documentation—not a second 77 GB copy of unchanged weights.

Quick start

The exact validated environment is Python 3.12, Torch 2.13.0+rocm7.15.0a20260728, HIP 7.15.0, Triton 3.8.0+git4cff872c.rocm7.15.0a20260728, FlashAttention 2.8.3, and the CIRU vLLM source at 388d82de2.

For a guided install, run this from the repository root:

bash install.sh --interactive

The runtime path defaults to the current directory. The installer asks for the exact model directory, but the 77 GB checkpoint download is strictly optional and defaults to no. Declining it still installs and validates the runtime, writes a sourceable ling3-runtime.env, and prints a command to download or provide the model later.

Agents and automation can avoid prompts:

bash install.sh \
  --install-root "$PWD" \
  --model-path "/path/to/Ling-checkpoint"

source "$PWD/ling3-runtime.env"
bash scripts/run-256k.sh

Add --install-host-deps when the supported distro packages are not already installed. Add --download-model only when you want this installer to fetch the checkpoint; omit it for a runtime-only install. Use bash install.sh --help for the complete agent interface.

The API advertises model ID Ling-3.0-Flash-CIRU-int4-Strix-native on http://127.0.0.1:18081/v1.

The default launcher provides the checkpoint's native 256K context (262144 tokens), five active sequence slots, and gpu_memory_utilization=0.72. Requests beyond five wait in vLLM's scheduler.

If you do not need that much simultaneous KV-cache capacity, lower the memory fraction at launch to leave more unified memory for the operating system and other applications. For example:

GPU_MEMORY_UTILIZATION=0.70 \
VENV="$HOME/ciru-ling-runtime/.venv" \
MODEL_PATH="$HOME/models/Ling-3.0-Flash-CIRU-int4-Strix-native" \
bash scripts/run-256k.sh

Lower values reduce KV-cache capacity and may prevent five requests from simultaneously reaching the full 256K context. Verify the reported KV-cache capacity at startup for your workload and host.

For the experimental 1M profile instead:

VENV="$HOME/ciru-ling-runtime/.venv" \
MODEL_PATH="$HOME/models/Ling-3.0-Flash-CIRU-int4-Strix-native" \
bash scripts/run-1m-yarn-experimental.sh

That launcher applies a 4x YaRN extension to 1,048,576 tokens, uses gpu_memory_utilization=0.82, and admits two active sequences. It is not native 1M training; requests beyond two wait.

Recommended live sampling

Follow InclusionAI's official Ling 3.0 Flash model card for live generation: temperature=0.6, top_p=0.95, top_k=20, with enable_thinking=true for better performance. The CIRU launcher does not replace these request-level sampling choices with a temperature-zero preset.

curl -fsS http://127.0.0.1:18081/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Ling-3.0-Flash-CIRU-int4-Strix-native","messages":[{"role":"user","content":"Explain in one sentence why the sky is blue."}],"chat_template_kwargs":{"enable_thinking":true},"temperature":0.6,"top_p":0.95,"top_k":20,"max_tokens":256}'

Temperature 0 appears only in the retained benchmark documentation as a deterministic measurement control. It is not the recommended live-generation preset.

Choose the platform guide before building:

• Native Linux: Ubuntu/Debian, Fedora, Arch, and NixOS notes
• Windows 11 through WSL2 and AMD ROCDXG

The exact runtime has been validated on CIRU's native-Linux NixOS Strix Halo host. Mainstream native-Linux and Windows/WSL instructions are designed to be reproducible but are not yet CIRU clean-host validations. Native Windows vLLM is not supported; Windows users must use WSL2.

Context and concurrency

The official checkpoint is natively configured and trained for 256K context. run-256k.sh uses max_model_len=262144, max_num_seqs=5, and gpu_memory_utilization=0.72. On CIRU's 128 GB Strix Halo host, this exact profile reported a 1,552,168-token KV pool and 5.92x capacity at 262,144 tokens per request, enough for five full native-context sequences.

Five is also the validated serving safety cap. A six-request 2K+256 simultaneous burst terminated the engine in the chunked-prefill MLA accumulation path with an LSE extent mismatch. With max_num_seqs=5, additional requests wait in the scheduler instead of entering that unvalidated C6 path.

The 1M profile is an experimental 4x YaRN extension, not native 1M training. It uses max_model_len=1048576, max_num_seqs=2, and gpu_memory_utilization=0.82; further requests wait in the vLLM scheduler.

Does this work for other W4A16 models?

The fork remains a vLLM runtime and safely falls back to upstream paths when its guards do not match, but the measured speedups are mostly Ling-specific:

• The Wave32 top-k fix is broadly relevant to ROCm Wave32 GPUs.
• Safe attention merging and uniform verifier routing are relevant to compatible ROCm MLA workloads, independent of W4A16.
• The W4A16 fast assignment path currently requires 512 experts, top-k 8, block size 32, and one to three tokens.
• The fast reduction requires hidden size 2560 and top-k 8.
• The SiLU path requires exact (8,1536)->(8,768) or (16,1536)->(16,768) BF16 shapes on gfx1151.

Other W4A16 models will run only if already supported by this vLLM base; they should not be expected to receive these speedups unless their geometry matches. Generalizing those kernels is part of the upstream plan.

Credits and license

• Model and official INT4 weights: InclusionAI/Ling-3.0-flash-int4, MIT license.
• Inference engine: vLLM, Apache-2.0.
• Wave32 top-k merge fix: liminfei-amd, vLLM PR #46012. The release history retains the original AMD authorship.
• ROCm, Torch, and Triton packages: AMD and their respective upstream projects.
• Strix Halo kernel, TTM/GTT, and WSL/ROCDXG guidance: AMD ROCm documentation and ROCDXG.
• Runtime integration, profiling, kernels, and validation: CIRU.

The original model license continues to govern the weights. The vLLM patch and bundle retain vLLM's upstream licensing and file headers.