Ornith-1.0-35B-Q8

This is a W8A8 INT8 quantized checkpoint of deepreinforce-ai/Ornith-1.0-35B, saved in the compressed-tensors format for vLLM.

Checkpoint details

• Base revision: 5df2ed3f675c7beaa490328cc70bb573b65fb660
• Architecture: Qwen3_5MoeForConditionalGeneration
• Quantization: per-channel INT8 weights and dynamic per-token INT8 activations
• Calibration: 256 samples from HuggingFaceH4/ultrachat_200k
• Calibration sequence length: 4096
• All MoE experts were calibrated
• Weight shards: 8
• Quantized weight size: 37,675,700,456 bytes (35.09 GiB)

The initial hybrid-MoE SmoothQuant fold was reverted after validation found that the mapped normalization layer also fans out to routed experts. The affected normalization tensors were restored from the base checkpoint, and the 110 affected projection tensors were re-quantized directly with per-channel INT8. The repaired tensors have a minimum cosine similarity of 0.99976 to their base weights. Details are included in quantization_report.json and smoothquant_fold_repair.json.

Tested vLLM launch

The following configuration was validated on one NVIDIA A100-PCIE-40GB using vllm/vllm-openai:v0.21.0-cu129-ubuntu2404:

docker run --rm \
  --gpus '"device=0"' \
  --ipc=host \
  -p 8000:8000 \
  -v /absolute/path/Ornith-1.0-35B-Q8:/model:ro \
  vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 \
  /model \
  --served-model-name Ornith-1.0-35B-Q8 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --language-model-only \
  --dtype auto \
  --quantization compressed-tensors \
  --gpu-memory-utilization 0.95 \
  --cpu-offload-gb 8 \
  --max-model-len 4096 \
  --max-num-seqs 1 \
  --enforce-eager

The validated run selected CompressedTensorsW8A8Int8, used the INT8 MoE backend, and consumed approximately 38.7 GB of A100 memory after KV-cache initialization.

Context length / max tokens

• The model configuration declares max_position_embeddings: 262144.
• The configuration tested in vLLM is --max-model-len 4096.
• 4096 is therefore the verified combined prompt-plus-generation token limit for the command above.
• Larger values, up to the architecture-declared limit, require additional KV-cache memory and must be validated for the target GPU, concurrency, and offload settings. Do not assume 262144 tokens will fit on a single 40 GB GPU.

For an H100 80GB, start with the validated command and increase --max-model-len gradually while watching available KV-cache memory. Lowering --max-num-seqs and using tensor parallelism are options for longer contexts.

API checks

List the served model:

curl http://127.0.0.1:8000/v1/models

Chat:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Ornith-1.0-35B-Q8",
    "messages": [{"role": "user", "content": "What is 7 multiplied by 8?"}],
    "temperature": 0,
    "max_tokens": 512
  }'

The validation run returned HTTP 200 from both /v1/models and /v1/chat/completions, served Ornith-1.0-35B-Q8, and correctly produced 56.

Notes

• This repository contains a quantized derivative. Review the base model card and its limitations before production use.
• Output quality and long-context behavior should be evaluated on your own workload.