Instructions to use fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash")
model = AutoModelForMultimodalLM.from_pretrained("fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash

SGLang

How to use fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash with Docker Model Runner:
```
docker model run hf.co/fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash
```

Ornith-1.0-397B W4A16 + DFlash (vLLM, RTX Pro 6000)

Pilcothink's AutoRound W4A16 quant of Ornith-1.0-397B, packaged with the z-lab DFlash draft for speculative decoding. Tuned on 4× RTX Pro 6000 Blackwell (SM120, 4×96GB).

NOTE: This can likely be improved a lot further.

Usage

hf download fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash --local-dir Ornith-1.0-397B
cd Ornith-1.0-397B
./serve-rtx-pro.sh

OpenAI-compatible server on port 8000: TP4, 262144 context, reasoning parser qwen3, tool parser qwen3_xml. Runs on voipmonitor/vllm:eldritch-enlightenment-v2226f26-b12x15cd38c-cu132-20260629 (vLLM 0.11.2.dev + native support for the DFlash sliding-window drafter). First run pulls the image and downloads the draft weights. Override via PORT, TP, MODEL_DIR, DRAFT_DIR, IMAGE.

Config notes

--attention-backend FLASHINFER: split-KV decode keeps the multi-token verify step at ~20 ms per step regardless of context length.
num_speculative_tokens=12: ~40% faster decode than k=8 at concurrency 1, equal at 2. k=16 (the draft's native block) is slower.
Keep the KV cache bf16 and --async-scheduling off; both alternatives lose speed and/or acceptance on this hybrid-GDN architecture.
NCCL_P2P_LEVEL=SYS + VLLM_MARLIN_USE_ATOMIC_ADD=1: +13% decode, +40% prefill on 4× PCIe GPUs.

Benchmarks

4× RTX Pro 6000 Blackwell, TP4, streaming OpenAI requests, real coding prompts generated to natural EOS (≤2048 tokens), 3 repeats. Vanilla = same image and quant, default vLLM serving per the base model card, no speculative decoding. Accept = mean accepted tokens per verify step.

prompt	conc	decode tok/s	vanilla tok/s	uplift	accept	TTFT p50	ITL p50
1,000	1	197.7	121.2	1.63×	4.20	152 ms	21.0 ms
10,000	1	139.5	119.9	1.16×	3.71	1.28 s	21.1 ms
100,000	1	148.5	107.9	1.38×	3.41	14.2 s	22.6 ms
1,000	3	149.1	80.1	1.86×	4.20	392 ms	29.2 ms
10,000	3	104.7	75.6	1.38×	3.45	2.86 s	29.7 ms
100,000	3	69.8	47.1	1.48×	3.61	28.1 s	33.6 ms

Prefill throughput 6.7–8.7k tok/s.

Credits

Model: DeepReinforce Team (MIT)
Quant: Pilcothink, rehosted unmodified
DFlash: z-lab (arXiv 2602.06036)
vLLM image: voipmonitor eldritch builds

Downloads last month: 813

Model tree for fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash

Base model

deepreinforce-ai/Ornith-1.0-397B

Quantized

(19)

this model

Paper for fraserprice/Ornith-1.0-397B-W4A16-AutoRound-DFlash

DFlash: Block Diffusion for Flash Speculative Decoding

Paper • 2602.06036 • Published Feb 5 • 89