jajmangold/Hy3-FP8-fni8

Hy3-FP8 — fni8 (int8)

An int8/int4 quantization of tencent/Hy3-FP8 to the .fni8 format, for the fni8 DP4A kernels on NVIDIA Volta (sm_70) GPUs (Tesla V100 and CMP 100-210). It is a derivative of the parent model; its license and acceptable uses follow the parent, linked above.

Files

file	precision	scheme	size
tencent__Hy3-FP8.b8.fni8	int8	int8 per-row W8A8	299.82 GB

.b8. files are int8, .b4. files are int4. Download the one you want. The bytes are the resident dp4a VRAM layout, so loading is a memory-map and copy with no dequantize or repack step.

How to use

Serve with fni8-serve: load_fni8_state_dict(<file>) into an LLMEngine. The architecture is read from the file; no separate config is needed.

What was quantized

• Linear and attention weights go to int8 (per-row) or int4 (per-group), with fp32 scales. Norms, embeddings, and the MoE router are kept in fp16.
• Target hardware is sm_70, where the fp16 tensor cores are firmware-limited, so the integer __dp4a path is used for the matmuls.

Intended use and scope

• For inference with the fni8 runtimes above, on Volta (sm_70) GPUs.
• Out of scope: other GPU architectures (the kernels require sm_70), and anything the parent model's license does not permit. It is a derivative, not a new model.

Limitations

• Quantization is lossy. int8 and especially int4 outputs differ from the fp16/bf16 parent, and the difference varies by model and task.
• This repository does not include per-model accuracy or benchmark measurements. Evaluate on your own task before relying on it.
• Any capabilities, biases, and risks of the parent model carry over. See the parent model card for those.

License

Follows the parent model (apache-2.0). This is a derivative quantization, not a relicense.

---

Part of the fni8 stack: kernels · LLM serving · ComfyUI DiTs.