Instructions to use z-lab/Qwen3.6-27B-DFlash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use z-lab/Qwen3.6-27B-DFlash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="z-lab/Qwen3.6-27B-DFlash", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("z-lab/Qwen3.6-27B-DFlash", trust_remote_code=True)
model = AutoModel.from_pretrained("z-lab/Qwen3.6-27B-DFlash", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use z-lab/Qwen3.6-27B-DFlash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "z-lab/Qwen3.6-27B-DFlash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3.6-27B-DFlash",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/z-lab/Qwen3.6-27B-DFlash

SGLang

How to use z-lab/Qwen3.6-27B-DFlash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "z-lab/Qwen3.6-27B-DFlash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3.6-27B-DFlash",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "z-lab/Qwen3.6-27B-DFlash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3.6-27B-DFlash",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use z-lab/Qwen3.6-27B-DFlash with Docker Model Runner:
```
docker model run hf.co/z-lab/Qwen3.6-27B-DFlash
```

DGX Spark - 66.7% token acceptance rate - let's push it higher together

#19

by MavisFace - opened 16 days ago

Discussion

MavisFace

16 days ago

•

edited 16 days ago

My Qwen3.6-27B is fucking flying now. I don't regret selling the Mac Studio M4 Max 128gb.........not one bit!!!

It's inconsistent, but regularly hitting above 40% acceptance rate with lows of 15% and my highest was 55%. Here's my launch code

Basics:
--enable-prefix-caching
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--trust-remote-code
--attention-backend flash_attn (needed for this config I think)

Things to tweak:
--speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-27B-DFlash","num_speculative_tokens":5}' (4 and 6 were not as good as 5 speculative tokens...for my setup)
--max-num-batched-tokens 12288
--max-model-len 65536
--max-num-seqs 4
--gpu-memory-utilization 0.75-.76

Maximum concurrency for 65,536 tokens per request: 3.98x to 4.47x

Lowering speculative tokens to 5 got me much higher acceptance rate...I'm just using Chat GPT for advice. If anyone else has some tips or launch settings please contribute!!

(APIServer pid=162547) INFO 07-03 22:03:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 62.6%
(APIServer pid=162547) INFO 07-03 22:03:59 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.41, Accepted throughput: 4.10 tokens/s, Drafted throughput: 14.50 tokens/s, Accepted: 41 tokens, Drafted: 145 tokens, Per-position acceptance rate: 0.655, 0.345, 0.172, 0.138, 0.103, Avg Draft acceptance rate: 28.3%
(APIServer pid=162547) INFO 07-03 22:04:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 62.6%
(APIServer pid=162547) INFO 07-03 22:04:09 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.76, Accepted throughput: 2.20 tokens/s, Drafted throughput: 14.50 tokens/s, Accepted: 22 tokens, Drafted: 145 tokens, Per-position acceptance rate: 0.414, 0.207, 0.069, 0.034, 0.034, Avg Draft acceptance rate: 15.2%
(APIServer pid=162547) INFO 07-03 22:04:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 62.6%
(APIServer pid=162547) INFO 07-03 22:04:19 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.73, Accepted throughput: 2.20 tokens/s, Drafted throughput: 15.00 tokens/s, Accepted: 22 tokens, Drafted: 150 tokens, Per-position acceptance rate: 0.500, 0.200, 0.033, 0.000, 0.000, Avg Draft acceptance rate: 14.7%
(APIServer pid=162547) INFO 07-03 22:04:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 62.6%
(APIServer pid=162547) INFO 07-03 22:04:29 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.35, Accepted throughput: 7.30 tokens/s, Drafted throughput: 15.50 tokens/s, Accepted: 73 tokens, Drafted: 155 tokens, Per-position acceptance rate: 0.742, 0.677, 0.484, 0.290, 0.161, Avg Draft acceptance rate: 47.1%
(APIServer pid=162547) INFO 07-03 22:04:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 62.6%
(APIServer pid=162547) INFO 07-03 22:04:39 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.26, Accepted throughput: 7.00 tokens/s, Drafted throughput: 15.50 tokens/s, Accepted: 70 tokens, Drafted: 155 tokens, Per-position acceptance rate: 0.677, 0.581, 0.452, 0.355, 0.194, Avg Draft acceptance rate: 45.2%
(APIServer pid=162547) INFO 07-03 22:04:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 62.6%
(APIServer pid=162547) INFO 07-03 22:04:49 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.27, Accepted throughput: 6.80 tokens/s, Drafted throughput: 15.00 tokens/s, Accepted: 68 tokens, Drafted: 150 tokens, Per-position acceptance rate: 0.800, 0.533, 0.433, 0.300, 0.200, Avg Draft acceptance rate: 45.3%
(APIServer pid=162547) INFO 07-03 22:04:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 62.6%
(APIServer pid=162547) INFO 07-03 22:04:59 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.77, Accepted throughput: 8.60 tokens/s, Drafted throughput: 15.50 tokens/s, Accepted: 86 tokens, Drafted: 155 tokens, Per-position acceptance rate: 0.871, 0.742, 0.484, 0.419, 0.258, Avg Draft acceptance rate: 55.5%
(APIServer pid=162547) INFO 07-03 22:05:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 62.6%
(APIServer pid=162547) INFO 07-03 22:05:09 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.95, Accepted throughput: 3.70 tokens/s, Drafted throughput: 9.50 tokens/s, Accepted: 37 tokens, Drafted: 95 tokens, Per-position acceptance rate: 0.737, 0.632, 0.316, 0.211, 0.053, Avg Draft acceptance rate: 38.9%
(APIServer pid=162547) INFO 07-03 22:05:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 62.6%

MavisFace

15 days ago

66.7% token acceptance rate now. 5 speculative tokens is the sweet spot for me.

--max-num-batched-tokens 16384
--max-model-len 65536
--max-num-seqs 4
--gpu-memory-utilization 0.751 (it's a little too low for concurrency of 4)

(APIServer pid=977753) INFO 07-04 16:50:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.82, Accepted throughput: 6.20 tokens/s, Drafted throughput: 17.00 tokens/s, Accepted: 62 tokens, Drafted: 170 tokens, Per-position acceptance rate: 0.618, 0.500, 0.324, 0.235, 0.147, Avg Draft acceptance rate: 36.5%
(APIServer pid=977753) INFO 07-04 16:50:11 [loggers.py:273] Engine 000: Avg prompt throughput: 154.4 tokens/s, Avg generation throughput: 12.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 57.4%
(APIServer pid=977753) INFO 07-04 16:50:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.75, Accepted throughput: 8.80 tokens/s, Drafted throughput: 16.00 tokens/s, Accepted: 88 tokens, Drafted: 160 tokens, Per-position acceptance rate: 0.938, 0.594, 0.469, 0.406, 0.344, Avg Draft acceptance rate: 55.0%
(APIServer pid=977753) INFO 07-04 16:50:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 57.4%
(APIServer pid=977753) INFO 07-04 16:50:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.97, Accepted throughput: 10.70 tokens/s, Drafted throughput: 18.00 tokens/s, Accepted: 107 tokens, Drafted: 180 tokens, Per-position acceptance rate: 0.944, 0.722, 0.583, 0.389, 0.333, Avg Draft acceptance rate: 59.4%
(APIServer pid=977753) INFO 07-04 16:50:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 57.4%
(APIServer pid=977753) INFO 07-04 16:50:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.06, Accepted throughput: 10.10 tokens/s, Drafted throughput: 16.50 tokens/s, Accepted: 101 tokens, Drafted: 165 tokens, Per-position acceptance rate: 0.818, 0.758, 0.636, 0.485, 0.364, Avg Draft acceptance rate: 61.2%
(APIServer pid=977753) INFO 07-04 16:50:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 57.4%
(APIServer pid=977753) INFO 07-04 16:50:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.33, Accepted throughput: 9.00 tokens/s, Drafted throughput: 13.50 tokens/s, Accepted: 90 tokens, Drafted: 135 tokens, Per-position acceptance rate: 0.889, 0.778, 0.667, 0.556, 0.444, Avg Draft acceptance rate: 66.7%

MavisFace changed discussion title from DGX Spark - 55% token acceptance rate - let's push it higher together to DGX Spark - 66.7% token acceptance rate - let's push it higher together 15 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment