neroued/GLM-5.3-Flash-NInfer

GLM-5.3-Flash for NInfer

A mixed-precision NInfer v3 artifact of GLM-5.3-Flash, converted from NVIDIA's NVFP4 checkpoint. It includes Text, Vision, native MTP and an additional full-vocabulary Q4 draft head.

The native NInfer architecture is GLM5Next, combining KDA linear attention, Kpool sparse MLA, mHC and MoE. The artifact includes the source tokenizer, GLM chat template and processor resources. Use it directly with NInfer for CLI generation or OpenAI-/Anthropic-compatible serving on two DGX Sparks.

Download

hf download neroued/GLM-5.3-Flash-NInfer \
  --include 'glm5_3_flash.ninfer*' \
  --local-dir models/GLM-5.3-Flash-NInfer

The complete artifact is 178.33 GiB across six files:

glm5_3_flash.ninfer
glm5_3_flash.ninfer.part-0001
glm5_3_flash.ninfer.part-0002
glm5_3_flash.ninfer.part-0003
glm5_3_flash.ninfer.part-0004
glm5_3_flash.ninfer.part-0005

Keep all parts beside the entry file and place the complete set on each node. Pass only glm5_3_flash.ninfer to NInfer; rank-specific weight slicing happens during loading. File sizes and the artifact identity are in artifact-manifest.json.

Representation

Weights	Stored representation
Main routed experts and first three dense MLP layers	Imported NVIDIA NVFP4, with calibrated scales
KDA Q/K/V/O, MLA output and shared-expert projections	Q8 A16
MTP routed experts	Block-FP8 converted from source BF16
Main head, embedding, mHC, Vision and other retained parameters	BF16; source FP32 KDA/control fields remain FP32
Optimized MTP head	Additional full-vocabulary Q4 A16; Main head stays BF16

The glm5next recipe retains imported Main NVFP4 codes/scales and performs the additional quantization above. The optimized draft head does not crop the vocabulary.

Run on two Sparks

Build the current NInfer revision on both nodes with NCCL development dependencies installed:

cmake --preset dgx-spark-release -DNINFER_ENABLE_DISTRIBUTED=ON
cmake --build build/dgx-spark-release -j

Full dependency and checkout instructions are in the build guide. From the repository root, prepare the deployment config:

cp examples/configs/spark2-glm5_3-flash.toml ninfer.local.toml
# Edit coordinator/worker artifact paths, worker executable paths and network addresses.
./build/dgx-spark-release/apps/ninfer-serve --config ninfer.local.toml

The example uses TP2 with attention DP2, FP8 KV, Vision, MTP3, the optimized head, CUDA Graph decode and up to eight active requests. Its logical context limit is 1,048,576 tokens. The coordinator launches both workers, using SSH for the remote node. Clients connect to its single HTTP endpoint.

Experts use TP by default (ep = 1); set ep = 2 to select whole-expert EP while retaining TP2 and attention DP2 elsewhere. Keep all six artifact files on each node. The complete artifact does not fit on one Spark. Runtime allocations also include workspace and the shared KV/State pool; read the startup capacity estimate for the selected configuration.

For CLI use, run apps/ninfer --config ninfer.local.toml --prompt "..." --max-tokens 2048 on the coordinator. Detailed startup and network instructions are in the deployment guide.

Model options

• Thinking stays enabled. Supported reasoning efforts are low, high and max; the template supplies the default. --no-thinking and effort none are unsupported.
• Native MTP supports 1..5 drafts. --proposal-head optimized selects the Q4 draft head; the target head stays BF16. Draft count can be adjusted for workload and concurrency.
• Latent KV uses --kv-dtype bf16 or fp8. The Kpool indexer uses FP8 by default; NINFER_KPOOL_INDEXER_BF16=1 selects BF16 indexer storage independently.
• Vision, tool calls, constrained output, streaming and prefix reuse use the same public Engine and protocol endpoints as text generation. Enable Vision at startup for image/video input.
• GLM uses its own chat template and tool framing; retain the included GLM frontend resources.

DFlash and DFlash2 are not used by this architecture. NInfer returns generated tool calls to the client; it does not execute them.

Sources and license

The primary conversion source is nvidia/GLM-5.3-Flash-NVFP4, revision da920bb0b9f4a06727223a349e55468e38352348, based on zai-org/GLM-5.3-Flash. The model weights use the upstream MIT license. NInfer is Apache-2.0-licensed software.