rockerBOO/qwen3-4b-nvfp4-convrot

Qwen3-4B text encoder — quantized (Flux.2 Klein)

Quantized variants of the Qwen3-4B text encoder used by Flux.2 Klein (Comfy-Org/vae-text-encorder-for-flux-klein-4b, split_files/text_encoders/qwen_3_4b.safetensors), derived from the base language model Qwen/Qwen3-4B.

Files

File	Size	Format	Derived from
qwen3_4b_int8_convrot.safetensors	4.49 GiB	ConvRot INT8 (row-scaled)	Comfy-Org/vae-text-encorder-for-flux-klein-4b:split_files/text_encoders/qwen_3_4b.safetensors (BF16, 7.49 GiB)
qwen3_4b_nvfp4_convrot_int8.safetensors	3.51 GiB	NVFP4 (bulk MLP) + ConvRot INT8 (row-scaled, attention projections)	same source as above
qwen3_4b_mixed_int4_int8_convrot.safetensors	3.38 GiB	INT4 W4A4 ConvRot (bulk MLP) + ConvRot INT8 (row-scaled, attention projections)	same source as above

embed_tokens and the first/last 2 of the model's 36 transformer blocks are kept at source BF16 precision in all three files.

• qwen3_4b_int8_convrot.safetensors: the remaining 224 attention/MLP weight tensors are quantized to INT8 with ConvRot (group-wise Hadamard rotation, group size 256) and row-wise scaling.
• qwen3_4b_nvfp4_convrot_int8.safetensors: the remaining 224 tensors are split by role — the 128 attention-projection tensors (self_attn.{q,k,v,o}_proj in the non-excluded blocks) are INT8 with ConvRot (group size 256, row-scaled), and the 96 MLP tensors (mlp.{gate,up,down}_proj) are NVFP4.
• qwen3_4b_mixed_int4_int8_convrot.safetensors: same role-based split as above, but the 96 MLP tensors are INT4 W4A4 ConvRot (group size 256) instead of NVFP4, and the 128 attention-projection tensors are promoted to INT8 ConvRot (row-scaled) to recover quality INT4 alone tends to lose on those layers. Quality caveat: per this project's quantization procedure, INT4 ConvRot output quality depends heavily on layer-selection method and rounding mode — this promotion criterion (attention projections promoted, bulk MLP left at INT4) is the same role-based split used for the NVFP4 variant above, not a result from a per-layer sensitivity sweep, and hasn't been separately evaluated for quality beyond the functional check below.

Hardware requirements

File	Hardware
qwen3_4b_int8_convrot.safetensors	Any modern GPU with usable INT8 tensor-core throughput — no Blackwell requirement.
qwen3_4b_nvfp4_convrot_int8.safetensors	Requires a Blackwell GPU (SM ≥ 10.0) for NVFP4 inference.
qwen3_4b_mixed_int4_int8_convrot.safetensors	No Blackwell dependency, but INT4 tensor-core throughput varies significantly by GPU generation — verify actual runtime behavior on target hardware.

Quantization method

Produced with convert_to_quant (ctq):

ConvRot INT8 (qwen3_4b_int8_convrot.safetensors):

ctq -i qwen_3_4b.safetensors -o qwen3_4b_int8_convrot.safetensors \
    --int8 --scaling-mode row --dynamic-convrot --convrot-group-size 256 \
    --layer-config qwen3_4b_layer_config_int8_convrot.json \
    --comfy_quant --save-quant-metadata

Where the layer-config (qwen3_4b_layer_config_int8_convrot.json, included in this repo) excludes model.embed_tokens and model.layers.{0,1,34,35} from quantization (kept at source BF16 precision); every other 2D weight tensor falls through to the --int8 primary format.

NVFP4 + ConvRot INT8 (qwen3_4b_nvfp4_convrot_int8.safetensors):

ctq -i qwen_3_4b.safetensors -o qwen3_4b_nvfp4_convrot_int8.safetensors \
    --nvfp4 --low-memory \
    --custom-layers 'model\.layers\.(2|3|...|33)\.self_attn\.(q|k|v|o)_proj' \
    --custom-type int8 --custom-scaling-mode row --custom-convrot --custom-convrot-group-size 256 \
    --layer-config qwen3_4b_layer_config_nvfp4_skip_only.json \
    --comfy_quant --save-quant-metadata

Where the layer-config (qwen3_4b_layer_config_nvfp4_skip_only.json, included in this repo) only holds skip: true entries for model.embed_tokens and model.layers.{0,1,34,35}; --custom-layers/--custom-type route the attention-projection layers in the remaining blocks to INT8 ConvRot; everything else falls through to the --nvfp4 primary format. (--custom-type/--fallback must be given alongside --nvfp4 on this ctq build for --layer-config to be consulted at all — see this repo's quantization tooling notes for why a bare --nvfp4 --layer-config ... invocation silently ignores the layer-config.)

Mixed INT4/INT8 ConvRot (qwen3_4b_mixed_int4_int8_convrot.safetensors):

ctq -i qwen_3_4b.safetensors -o qwen3_4b_mixed_int4_int8_convrot.safetensors \
    --int4 --dynamic-convrot --convrot-group-size 256 --low-memory \
    --layer-config qwen3_4b_layer_config_mixed_int4_int8.json \
    --comfy_quant --save-quant-metadata

Where the layer-config (qwen3_4b_layer_config_mixed_int4_int8.json, included in this repo) holds skip: true entries for model.embed_tokens and model.layers.{0,1,34,35}, plus "format": "int8_tensorwise" (ConvRot, row-scaled) entries for the attention-projection layers in the remaining blocks; every other 2D weight tensor falls through to the --int4 primary format.

Learned-rounding optimization (not RTN) was used for all three files — the default when --simple is omitted.

All three files were verified to load and generate end-to-end in ComfyUI as the text encoder in a Flux.2 Klein 4B txt2img workflow. Output quality (vs. the BF16 source) has not been separately assessed beyond this functional check.

License

This is a derivative of Qwen/Qwen3-4B, licensed under the Apache License 2.0. A copy of the license is included below per the license's redistribution terms; this file has been modified (quantized) from the original.

                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

Copyright 2024 Alibaba Cloud

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Full terms: https://www.apache.org/licenses/LICENSE-2.0