Instructions to use local-inference-lab/Qwen3.8-27B-NVFP4-QAD with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use local-inference-lab/Qwen3.8-27B-NVFP4-QAD with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="local-inference-lab/Qwen3.8-27B-NVFP4-QAD")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("local-inference-lab/Qwen3.8-27B-NVFP4-QAD")
model = AutoModelForMultimodalLM.from_pretrained("local-inference-lab/Qwen3.8-27B-NVFP4-QAD", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use local-inference-lab/Qwen3.8-27B-NVFP4-QAD with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "local-inference-lab/Qwen3.8-27B-NVFP4-QAD"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/Qwen3.8-27B-NVFP4-QAD",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/local-inference-lab/Qwen3.8-27B-NVFP4-QAD

SGLang

How to use local-inference-lab/Qwen3.8-27B-NVFP4-QAD with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "local-inference-lab/Qwen3.8-27B-NVFP4-QAD" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/Qwen3.8-27B-NVFP4-QAD",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "local-inference-lab/Qwen3.8-27B-NVFP4-QAD" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/Qwen3.8-27B-NVFP4-QAD",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use local-inference-lab/Qwen3.8-27B-NVFP4-QAD with Docker Model Runner:
```
docker model run hf.co/local-inference-lab/Qwen3.8-27B-NVFP4-QAD
```

*** WORK IN PROGRESS ***

Qwen3.8-27B-NVFP4-QAD

A mixed NVFP4/MXFP8 quantization-aware distillation of Qwen3.8-27B, exported at training step 7,000.

The student learns from the original BF16 teacher with quantized MLP weights in its forward pass. Distillation updates the MLP weights, text normalization weights and BF16 student LM head. This is a trained distillation checkpoint, not a post-training conversion of the original weights.

Precision

Component	Representation
MLP gate, up and down projections	NVFP4, 16-element blocks
Gated delta network projections	MXFP8, 32-element blocks; frozen
Full-attention query, key, value and output projections	Original BF16; frozen
Student LM head	Trained FP32 master rounded to BF16
Text normalization weights	Trained FP32 masters
Token embeddings, GDN convolutions and dynamics	Original BF16; frozen
Vision encoder and remaining source tensors	Unchanged

Packed NVFP4 and MXFP8 weights reconstruct to the BF16 weight values used during training. The tokenizer, chat template, generation configuration and multimodal processors are retained from the base model.

Activation scales

The 192 MLP input scales are copied exactly from Qwen3.8-27B-QAD-E1, whose step-4,779 weights were calibrated on 390,497,191 raw-text and chat tokens. That calibration selected the pooled p99.999 token-row maximum from exact BF16 histograms. These scales were not recalibrated at step 7,000.

Each dense layer has equal gate/up scales in separate tensors and an independent down-projection scale. Serving uses calibrated NVFP4 MLP activations and dynamic MXFP8 GDN activations, adding activation quantization beyond the BF16 training forward.

Format

Hugging Face safetensors with ModelOpt mixed-precision metadata. The runtime must support the base architecture, NVFP4 dense linears, MXFP8 linears and the BF16 full-attention/head exclusions. Packed reconstruction, serialized tensors and copied input scales are checked. Serving quality and performance have not been evaluated for this export.

License

Apache 2.0, following the base model.

Downloads last month: 218

Safetensors

Model size

19B params

Tensor type

F32

BF16

F8_E4M3

Model tree for local-inference-lab/Qwen3.8-27B-NVFP4-QAD

Base model

Qwen/Qwen3.8-27B

Quantized

(1217)

this model