Ornith-1.0-35B-int4-Autoround

This repository contains 4-bit quantized weights (W4A16) for deepreinforce-ai/Ornith-1.0-35B, generated using Intel Auto-Round.

Auto-Round is an advanced weight-only quantization algorithm using sign-gradient descent to fine-tune quantization scales and clipping thresholds, delivering high accuracy preservation for low-bit LLMs.

---

Quantization Configuration

Parameter	Value
Base Model	deepreinforce-ai/Ornith-1.0-35B
Quantization Scheme	W4A16 (4-bit weights, 16-bit activations)
Quantization Engine	Intel Auto-Round
Group Size	64
Calibration Sequence Length	2048
Calibration Samples (nsamples)	64
Optimization Iterations (iters)	300
Format	auto_round

---

Quickstart & Serving

Ornith-1.0-35B is a reasoning model: by default, the assistant turn opens with a <think> ... </think> block before the final answer. The serving commands below configure the reasoning and tool-call parsers so the chain-of-thought is returned in a separate reasoning_content field.

1. Launch with vLLM

Install vLLM (version >= 0.25.1 recommended):

pip install auto-round vllm

Start the vLLM OpenAI-compatible server:

vllm serve urakozz/Ornith-1.0-35B-int4-Autoround \
    --served-model-name Ornith-1.0-35B-int4-Autoround \
    --quantization auto-round \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.90 \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3 \
    --trust-remote-code

---

2. Querying the OpenAI-Compatible API

Once the vLLM server is running, connect via any OpenAI client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="Ornith-1.0-35B-int4-Autoround",
    messages=[
        {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
    temperature=0.6,
    top_p=0.95,
    max_tokens=2048,
)

message = response.choices[0].message

# reasoning_content holds the <think> trace; content holds the final answer
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Answer:", message.content)

---

Benefits & Memory Footprint

• VRAM Savings: Quantizing the 35B model with W4A16 and group_size=64 reduces memory consumption from 70GB+ down to **20–22GB VRAM**, enabling execution on single consumer/workstation GPUs.
• Accuracy: Auto-Round optimizes quantization parameters over 300 iterations per block, significantly mitigating quantization loss compared to standard round-to-nearest (RTN) methods.

---

Acknowledgments & Citation

• Base Model: deepreinforce-ai/Ornith-1.0-35B
• Quantization Engine: Intel Auto-Round

@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {[https://deep-reinforce.com/ornith_1_0.html](https://deep-reinforce.com/ornith_1_0.html)},
    author = {{DeepReinforce Team}},
    year = {2026}
}