Qwen3.5-0.8B-fp8

FP8 (W8A8 dynamic) quantization of Qwen/Qwen3.5-0.8B, produced with llm-compressor. lm_head is kept in the original precision.

Usage

vLLM:

vllm serve mahadev9/Qwen3.5-0.8B-fp8

transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mahadev9/Qwen3.5-0.8B-fp8", torch_dtype="auto", device_map="auto"
)
tok = AutoTokenizer.from_pretrained("mahadev9/Qwen3.5-0.8B-fp8")