voves/gemma-4-26B-A4B-it-NVFP4

sglang workaround for vision_tower

TL;DR: vLLM nightly runs this model out of the box. For sglang, a small monkey-patch is needed until official support lands.

Problem

sglang crashes on the first multimodal request:

RuntimeError: shape '1, 3456' is invalid for input of size 8709120 at compressed_tensors_w4a4_nvfp4.py:169 → out.view(*output_shape)

Root cause

In config.json → quantization_config.ignore, the vision_tower layers are listed under their unfused names (q_proj.linear, k_proj.linear, v_proj.linear, gate_proj.linear, up_proj.linear, down_proj.linear). However, sglang's gemma4_vision implementation fuses them into qkv_proj / gate_up_proj.

As a result, the ignore-filter in CompressedTensorsConfig.get_quant_method does not match the fused layer names — an empty NVFP4 scheme gets created for them (no weight_packed / weight_scale in the checkpoint, since vision_tower is stored as bf16), and out.view(*output_shape) with [1, w_n] crashes on a 3D input [1, seq, hidden].

Patch

Two monkey-patches (no sglang source edits):

1. CompressedTensorsConfig.get_quant_method — return UnquantizedLinearMethod() for any layer whose prefix contains vision_tower..
2. CompressedTensorsW4A4Fp4.apply_weights — correct reshape for arbitrary input dimensions: out_shape = list(x.shape[:-1]) + [w_n] instead of the hardcoded [x.shape[0], w_n].

Usage

Drop gemma4.py next to your model and launch sglang through it (the patch is applied in-process before the server starts):

#!/usr/bin/env python
import sys
import sglang
import sglang.srt.layers.quantization.compressed_tensors.compressed_tensors as ct_mod
from sglang.srt.layers.quantization.compressed_tensors.schemes.compressed_tensors_w4a4_nvfp4 import (
    CompressedTensorsW4A4Fp4 as _Nvfp4,
)
from sglang.srt.layers.quantization.modelopt_quant import (
    enable_flashinfer_fp4_gemm, fp4_gemm, fp4_quantize,
)
from sglang.srt.layers.quantization.unquant import UnquantizedLinearMethod

# Patch 1: force UnquantizedLinearMethod for vision_tower layers
_orig_get = ct_mod.CompressedTensorsConfig.get_quant_method
def _patched_get(self, layer, prefix):
    if "vision_tower." in prefix:
        return UnquantizedLinearMethod()
    return _orig_get(self, layer, prefix)
ct_mod.CompressedTensorsConfig.get_quant_method = _patched_get

# Patch 2: handle 3D inputs in apply_weights
def _safe_apply(self, layer, x, bias=None):
    out_dtype = x.dtype
    w_n, _ = layer.weight_packed.shape
    out_shape = list(x.shape[:-1]) + [w_n]
    x_fp4, x_bs = fp4_quantize(x, layer.input_global_scale)
    w = layer.weight_packed.T if enable_flashinfer_fp4_gemm else layer.weight_packed
    w_bs = layer.weight_scale.T if enable_flashinfer_fp4_gemm else layer.weight_scale
    out = fp4_gemm(x_fp4, w, x_bs, w_bs, layer.alpha, out_dtype, w_n)
    if bias is not None:
        out = out + bias
    return out.view(*out_shape)
_Nvfp4.apply_weights = _safe_apply

# Launch sglang serve
from sglang.launch_server import run_server
from sglang.srt.server_args import prepare_server_args
from sglang.srt.plugins import load_plugins

if __name__ == "__main__":
    load_plugins()
    server_args = prepare_server_args(sys.argv[1:])
    run_server(server_args)

Launch:

python gemma4.py \
  --model-path /models/gemma-4-26B-A4B-it-NVFP4 \
  --served-model-name Gemma-4-26B-A4B \
  --host 0.0.0.0 --port 8000 \
  --trust-remote-code