gratex/Ornith-1.5-397B-W4A16-g32-sym

Ornith-1.5-397B — W4A16 Quantization (g32, Symmetric)

This is a W4A16 (4-bit weight-only) quantization of ornith-ai/Ornith-1.5-397B, a 397B-parameter mixture-of-experts model with a hybrid linear-attention / full-attention backbone, native tool calling, and chain-of-thought reasoning.

Quantization was performed with AutoRound (0.16.0-main) using SignRoundV2 and exported in the compressed-tensors pack-quantized format. Total quantization time was 33.2 h on 4× RTX PRO 6000 (Blackwell, 96 GB each).

Model Details

Property	Value
Base Model	ornith-ai/Ornith-1.5-397B
Quantization Method	AutoRound SignRoundV2 (symmetric, group_size=32)
Weight Precision	INT4 (4-bit, symmetric)
Activation Precision	BF16 (weight-only quantization)
Group Size	32 (fine-grained, fp16 scales)
Quantization Library	Intel AutoRound 0.16.0-main → llm-compressor
Format	compressed-tensors, pack-quantized
Architecture	Qwen3_5MoeForConditionalGeneration
LM Layers	60 (45 linear_attention + 15 full_attention, hybrid)
Hidden Size	4,096
MoE	512 routed experts, top-10, moe_intermediate_size 1,024
Attention Heads	32 (query), 2 (key/value, GQA)
Head Dimension	256
Vocabulary Size	248,320
Context Window	262,144 tokens (262K)
Vision Encoder	Qwen3 VisionTransformer (27 layers, hidden_size=1152)
Quantized Components	Routed-expert FFN projections only (gate_proj / up_proj / down_proj, 60 layers × 512 experts)
Preserved in BF16	Vision encoder + projector, full-attention q/k/v/o projections (15 layers), linear-attention projections (in_proj_qkv/z/b/a, conv1d, out_proj), shared experts + MoE gates, lm_head, embed_tokens
Model Size	~251 GB (61 shards) vs ~752 GB BF16 base (≈3.0× smaller) — 279,284 tensors
Calibration	512 task samples, seqlen 4,096, 1,000 iterations, batch 8, seed 42

Quantization Details

Recipe

from auto_round import AutoRound, SignRoundV2Config

ar = AutoRound(
    model="ornith-ai/Ornith-1.5-397B",
    scheme="W4A16G32",                       # bits=4, group_size=32, symmetric
    alg_configs=[SignRoundV2Config(iters=1000)],
    dataset="calibration_data.jsonl",
    nsamples=512,
    seqlen=4096,
    device_map="0,1,2,3",                    # 4× GPUs
    ignore_layers=IGNORE_LAYERS,             # 832 layer patterns preserved in BF16
    scale_dtype="fp16",
    enable_torch_compile=True,
    low_cpu_mem_usage=True,
    low_gpu_mem_usage=True,
    batch_size=8,
    gradient_accumulate_steps=1,
    seed=42,
)
model, _ = ar.quantize_and_save(format="llm_compressor", max_shard_size="5GB")

Calibration Dataset

512 task samples formatted for deployment distribution alignment, padded to 4,096-token sequences.

Quality Benchmarks

All shown numbers are measured on this checkpoint.

PPL (Perplexity) — WikiText-2

Teacher-forced chunked evaluation on wikitext-2-raw-v1 (test split), non-overlapping 512-token chunks, scoring the second half of each chunk. Full test split: 639 chunks, 296,434 scored tokens, zero failed chunks.

Model	PPL ↓
This model (W4A16 g32 sym)	4.6611
ornith-ai/Ornith-1.5-397B-NVFP4 (official NVFP4 quantization published by ornith-ai)	4.8522

Throughput Benchmarks

Benchmarked with a stdlib OpenAI client harness: 4× RTX PRO 6000 (96 GB) TP=4 + expert parallel, vLLM 0.26.1, kv_cache_dtype: fp8_e4m3, context 262,144, CUDA graphs enabled. Unique random prompts per request to prevent prefix-cache hits.

Single Request (concurrency=1)

20 requests, max_tokens=1024, temperature=0.9.

Metric	Value
Aggregate throughput	118.7 tok/s
Per-request throughput	min 111.6 / avg 117.7 / max 119.2 tok/s
Average latency	6.81 s
Min / Max latency	0.35 s / 8.68 s
Completion tokens	min 40 / avg 808 / max 1024
Success rate	20/20 (100%)
Wall time	136.2 s

16 Concurrent Requests

160 total requests, max_tokens=1024, temperature=0.9 (max_num_seqs = 16).

Metric	Value
Aggregate throughput	787.5 tok/s
Per-request throughput	min 8.0 / avg 50.2 / max 70.9 tok/s
Average latency	14.51 s
Min / Max latency	0.65 s / 23.16 s
Completion tokens	min 30 / avg 746 / max 1024
Success rate	160/160 (100%)
Wall time	151.6 s

Hardware Requirements

The compressed-tensors W4A16 scheme requires NVIDIA GPUs with compute capability ≥ 7.5 (Turing); vLLM dispatches its fused INT4 GEMM kernels accordingly (Marlin on most architectures, Machete additionally available on Hopper sm_90). The served configuration has been verified on Blackwell (sm_120).

Weights occupy 62.8 GB per GPU at TP=4 (251 GB total) plus activation/reserve headroom — VRAM, not tensor-core generation, is the binding constraint at this model size. Tensor-parallel size must divide the model cleanly: TP ∈ {1, 2, 4, 8}.

GPU layout	Total VRAM	Weights/GPU	Status
4× RTX PRO 6000 (96 GB, Blackwell)	384 GB	62.8 GB	✅ verified — full 262K context, fp8 KV, config below
4× H100 (80 GB, Hopper)	320 GB	62.8 GB	weights fit; smaller KV pool, not measured
4× H200 (141 GB, Hopper)	564 GB	62.8 GB	weights fit; large KV headroom, not measured
8× RTX PRO 6000 (96 GB)	768 GB	31.4 GB	weights fit; maximum KV headroom, not measured
2× H200 (141 GB)	282 GB	125.6 GB	weights fit; KV pool very tight at 262K context, experimental

Usage with vLLM

Docker Deployment (verified production configuration)

docker run -d --name vllm-ornith15-w4a16 \
  --runtime=nvidia --gpus '"device=0,1,2,3"' \
  -p 8000:8000 \
  -v /path/to/model:/workspace/model \
  --ipc=host --shm-size=16g \
  --restart unless-stopped \
  vllm/vllm-openai:latest \
  /workspace/model \
  --host 0.0.0.0 --port 8000 \
  --served-model-name ornith-1.5-397b-w4a16 \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.90 \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --limit-mm-per-prompt '{"image": 4}'

Inference Test

# Text completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"ornith-1.5-397b-w4a16","messages":[{"role":"user","content":"What is 2+2? One word."}],"max_tokens":10}'

# Tool calling test
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ornith-1.5-397b-w4a16",
    "messages": [{"role":"user","content":"What is the weather in Bratislava?"}],
    "tools": [{"type":"function","function":{"name":"get_weather","description":"Get weather","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
    "tool_choice": "auto",
    "max_tokens": 256
  }'

Transformers / Python

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_id = "gratex/Ornith-1.5-397B-W4A16-g32-sym"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Explain INT4 weight quantization in two sentences."}]
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Files in This Repository

File	Description
model-00001..00061-of-00061.safetensors	Quantized model weights (INT4 routed experts + BF16 everything else), ~5 GB per shard
model.safetensors.index.json	Shard index (279,284 tensors)
model_extra_tensors.safetensors	BF16 tensors outside the main sharded set
config.json	Model configuration with quantization_config (compressed-tensors, pack-quantized)
quantization_config.json	Standalone copy of the quantization parameters
tokenizer.json / tokenizer_config.json	Tokenizer (identical to base model, vocab 248,320)
chat_template.jinja	Native chat template (inherited from base)
generation_config.json	Default generation parameters
preprocessor_config.json / processor_config.json	Vision preprocessor / multimodal processor (inherited from base)

License

This quantization is released under the MIT License, following the base model's license.

The base model ornith-ai/Ornith-1.5-397B is licensed under MIT.

Citation

If you use this model in your research or production systems, please cite:

@misc{ornith15-397b-w4a16-g32-sym,
  title = {Ornith-1.5-397B AutoRound W4A16 g32 Symmetric Quantization},
  author = {Gratex International},
  year = {2026},
  howpublished = {\url{https://huggingface.co/gratex/Ornith-1.5-397B-W4A16-g32-sym}},
  note = {Quantized with Intel AutoRound 0.16 (SignRoundV2), compressed-tensors pack-quantized format}
}

Acknowledgments

This quantization was produced using hardware and infrastructure provided by Gratex International, a.s.

Original Model: ornith-ai/Ornith-1.5-397B Quantization Tool: AutoRound Quantization Format: compressed-tensors (pack-quantized) Deployment Engine: vLLM