zhiyaowang/MiniCPM-SALA-NVFP4-GPTQ

MiniCPM-SALA NVFP4 GPTQ

NVFP4-quantized target checkpoint derived from OpenBMB MiniCPM-SALA, for SGLang inference.

This repository publishes the deployment checkpoint named MiniCPM-SALA-NVFP4-GPTQ46-96k. The weight shards, configuration, tokenizer, chat template, and model source files are preserved unchanged. The two safetensors shards total 6,196,512,952 bytes (6.197 GB / 5.771 GiB). File hashes are listed in SHA256SUMS.

Format

• GPTQ quantization with the NVFP4 scheme; the saved recipe is included as recipe.yaml.
• SGLang loader: modelopt_fp4; NVFP4 GEMM backend: flashinfer_cutlass.
• The quantization recipe excludes lm_head, which remains BF16.
• Original model configuration and tokenizer assets are included.

SGLang usage

Use a SGLang build with MiniCPM-SALA model support, the minicpm_flashinfer attention backend, and compatible InfLLM kernels. This checkpoint has been served on NVIDIA RTX 5090 GPUs with CUDA 13.0.

hf download zhiyaowang/MiniCPM-SALA-NVFP4-GPTQ \
  --local-dir ./MiniCPM-SALA-NVFP4-GPTQ

python -m sglang.launch_server \
  --model-path ./MiniCPM-SALA-NVFP4-GPTQ \
  --trust-remote-code \
  --quantization modelopt_fp4 \
  --fp4-gemm-backend flashinfer_cutlass \
  --attention-backend minicpm_flashinfer \
  --context-length 16384 \
  --chunked-prefill-size 4096 \
  --disable-radix-cache \
  --disable-prefill-cuda-graph

The matching EAGLE3 draft is zhiyaowang/MiniCPM-SALA-EAGLE3-Draft-NVFP4.

中文说明

本仓库发布 MiniCPM-SALA 的 NVFP4 GPTQ target 权重，对应本地部署版本 MiniCPM-SALA-NVFP4-GPTQ46-96k。两个权重分片及配置、tokenizer、chat template 和模型源码保持原样，附带量化 recipe 和 SHA-256 校验清单。

使用支持 MiniCPM-SALA 的 SGLang 与相应 InfLLM 内核加载，量化参数为 --quantization modelopt_fp4 --fp4-gemm-backend flashinfer_cutlass。上面的命令给出 16K context 的启动示例。

Attribution

Base model: openbmb/MiniCPM-SALA, by the OpenBMB team. The upstream model is licensed under Apache-2.0; original copyright and license headers are retained in the included model source files.