Instructions to use deepsweet/Qwen3.6-27B-DFlash-FP16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepsweet/Qwen3.6-27B-DFlash-FP16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepsweet/Qwen3.6-27B-DFlash-FP16", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("deepsweet/Qwen3.6-27B-DFlash-FP16", trust_remote_code=True)
model = AutoModel.from_pretrained("deepsweet/Qwen3.6-27B-DFlash-FP16", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepsweet/Qwen3.6-27B-DFlash-FP16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepsweet/Qwen3.6-27B-DFlash-FP16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepsweet/Qwen3.6-27B-DFlash-FP16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/deepsweet/Qwen3.6-27B-DFlash-FP16

SGLang

How to use deepsweet/Qwen3.6-27B-DFlash-FP16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepsweet/Qwen3.6-27B-DFlash-FP16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepsweet/Qwen3.6-27B-DFlash-FP16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepsweet/Qwen3.6-27B-DFlash-FP16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepsweet/Qwen3.6-27B-DFlash-FP16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use deepsweet/Qwen3.6-27B-DFlash-FP16 with Docker Model Runner:
```
docker model run hf.co/deepsweet/Qwen3.6-27B-DFlash-FP16
```

How can I convert z-lab/Qwen3.8-27B-DFlash2 to fp16?

by crazyi - opened 26 days ago

Discussion

crazyi

26 days ago

The following is the error message:

mlx_lm.convert --hf-path /Volumes/TiPlus7100/llm/models/z-lab/Qwen3.8-27B-DFlash2 --mlx-path /Volumes/TiPlus7100/llm/models/z-lab/Qwen3.8-27B-DFlash2-fp16
--dtype float16

[INFO] Loading
Traceback (most recent call last):
File "/opt/homebrew/bin/mlx_lm.convert", line 6, in
sys.exit(main())
~~~~^^
File "/opt/homebrew/Cellar/mlx-lm/0.31.3_2/libexec/lib/python3.14/site-packages/mlx_lm/convert.py", line 259, in main
convert(**vars(args))
~~~~~~~^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/mlx-lm/0.31.3_2/libexec/lib/python3.14/site-packages/mlx_lm/convert.py", line 112, in convert
model, tokenizer, config = load(
~~~~^
hf_path,
^^^^^^^^
...<3 lines>...
lazy=True,
^^^^^^^^^^
)
^
File "/opt/homebrew/Cellar/mlx-lm/0.31.3_2/libexec/lib/python3.14/site-packages/mlx_lm/utils.py", line 491, in load
model, config = load_model(model_path, lazy, model_config=model_config)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/mlx-lm/0.31.3_2/libexec/lib/python3.14/site-packages/mlx_lm/utils.py", line 341, in load_model
model_args = model_args_class.from_dict(config)
File "/opt/homebrew/Cellar/mlx-lm/0.31.3_2/libexec/lib/python3.14/site-packages/mlx_lm/models/base.py", line 15, in from_dict
return cls(
**{
...<3 lines>...
}
)
TypeError: ModelArgs.init() missing 1 required positional argument: 'rope_theta'

deepsweet

Owner 24 days ago

Hi.

Here you go but please keep in mind that I haven't tried it by myself:

https://huggingface.co/deepsweet/Qwen3.8-27B-DFlash2-FP16

crazyi

23 days ago

•

edited 23 days ago

Thank you for your help, but I think the DFlash2-FP16 maybe have some issue.
Here are the benchmarks:
Benchmark Model: Qwen3.8-27B-Uncensored-oQ8e-fp16-mtp
Engine: Auto
Context: 代码（Python）

Test	Config	TTFT(ms)	TPOT(ms)	pp TPS	tg TPS	E2E(s)	Throughput	Peak Mem(GB)
pp1024/tg128	Dflash2‑fp16	5225.5	153.38	196.0	6.6	24.726	46.6	32.14
pp1024/tg128	Dflash2‑BF16	5276.3	61.08	194.1	16.5	13.051	88.3	32.25
pp1024/tg128	MTP	5153.4	51.30	198.7	19.6	11.691	98.5	35.32

pp4096/tg128	Dflash2‑fp16	20480.0	157.38	200.0	6.4	40.467	104.4	33.86
pp4096/tg128	Dflash2‑BF16	20885.3	68.05	196.1	14.8	29.550	142.9	33.91
pp4096/tg128	MTP	20242.7	54.56	202.3	18.5	27.196	155.3	36.74

pp8192/tg128	Dflash2‑fp16	41776.0	161.47	196.1	6.2	62.302	133.5	34.91
pp8192/tg128	Dflash2‑BF16	42115.4	47.48	194.5	21.2	48.168	172.7	35.04
pp8192/tg128	MTP	41449.8	51.85	197.6	19.4	48.056	173.1	37.10

pp16384/tg128	Dflash2‑fp16	87513.6	110.27	187.2	9.1	101.548	162.6	36.69
pp16384/tg128	Dflash2‑BF16	88289.0	48.97	185.6	20.6	94.543	174.6	36.98
pp16384/tg128	MTP	19565.5	35.35	837.4	28.5	24.072	685.9	36.47

pp32768/tg128	Dflash2‑fp16	192590.3	165.90	170.1	6.1	213.664	154.0	39.80
pp32768/tg128	Dflash2‑BF16	192931.8	49.27	169.8	20.5	199.189	165.1	40.40
pp32768/tg128	MTP	41194.8	35.87	795.4	28.1	45.767	718.8	36.86

pp65536/tg128	Dflash2‑fp16	453229.7	188.47	144.6	5.3	477.231	137.6	45.28
pp65536/tg128	Dflash2‑BF16	454595.1	64.03	144.2	15.7	462.818	141.9	46.50
pp65536/tg128	MTP	92047.0	40.37	712.0	25.0	97.191	675.6	37.54

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment