Instructions to use klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP")
model = AutoModelForMultimodalLM.from_pretrained("klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP

SGLang

How to use klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP with Docker Model Runner:
```
docker model run hf.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP
```

Qwen3.8-Flash-Next AutoRound 3 bpw + MTP for Ampere

This is a public, mixed-precision AutoRound checkpoint of Qwen/Qwen3.8-Flash-Next, pinned to revision de4b8e4d43b917e7706784d8bb445c9af86a3540. It is intended for NVIDIA Ampere GPUs with a 64 GiB GPU-resident weight budget.

Qwen3.8-Flash-Next is an experimental preview model. Runtime support is also experimental; read the compatibility section before downloading.

Quantization

48 routed-expert banks: depth-stratified 2.998 effective bpw (scale/zero-point overhead included): W3A16G128 on layers 0-11 and 36-47, W2A16G64 on layers 12-35.
Backbone QSA/GDN major linear projections: W8A16G128.
Routers, shared experts, hyperconnection/control paths, PLE projections, vision tower, embeddings, and LM head: BF16.
MTP experts: W4A16G128 symmetric RTN; MTP QSA and dense projections: W8A16G128 symmetric RTN.
Backbone optimization: calibrated AutoRound SignRoundV2, 50 iterations, using 128 sequences of 512 tokens from NeelNanda/pile-10k (seed 42).
Reproducibility: dataset selection and sampling were seeded, but PyTorch reported nondeterministic CUDA memory-efficient attention backward, so the calibration is not claimed to be bit-for-bit reproducible.
Packing: native AutoRound auto_round:auto_gptq mixed-bit format.

The effective packed qweight inventory by bit width is: {"2": 36864, "3": 36864, "4": 1536, "8": 175}.

Measured storage topology

GPU-resident weight tensors: 47.11 GiB.
Host-offloaded PLE n-gram table: 95.37 GiB (128 tensors).
MTP tensors are included in the checkpoint and counted in the GPU-resident figure.
Safetensor shards: 13; indexed tensors: 227702.

The 64 GiB figure is a weight budget, not a claim that every context length fits. KV cache and runtime workspaces require additional HBM. The 95 GiB n-gram table must remain in host memory; provision ample system RAM.

Runtime compatibility

At publication time, stock vLLM cannot serve this exact checkpoint topology. Two required changes are still open:

In addition, vLLM issue #53908 tracks auxiliary-device offload for the PLE table; PR #53896 currently allocates that embedding through the normal GPU embedding path. Combining the two PRs alone is therefore insufficient for this BF16 PLE checkpoint on one GPU.

The intended fallback is Transformers AutoModelForMultimodalLM with the layer-1 PLE embedding lookup on CPU and its projections plus the packed language backbone on CUDA:0. This pinned Transformers fallback performs ordinary autoregressive decoding and does not instantiate the included MTP speculative head. Provision at least 200 GiB of host RAM.

Validation

Validation details will be added.

License

Qwen Community License 1.0, inherited from the base model. Review the included LICENSE file and the base model card for its conditions and limitations.

Downloads last month: 32

Safetensors

Model size

54B params

Tensor type

I64

I32

BF16

F16

Model tree for klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP

Base model

Qwen/Qwen3.8-Flash-Next

Quantized

(131)

this model