HyperArete-27B-W4A16

Calibrated quantization of NousResearch/Arete-27B, prepared and served using HyperQwen optimizations. The released checkpoint is surak/HyperArete-27B-W4A16.

The language-model body is symmetric W4A16, group size 128. The LM head and token embeddings are INT8, group size 128. All 333 vision tensors were verified unchanged. The checkpoint contains no MTP head. There was no additional alignment training or merge with another Qwen checkpoint.

Recipe source: surak/HyperArete. The recipe/ directory is also included in this model repository.

Revisions and preparation

• Arete source: 108f0afbdcfaa1ceeda4a635004304dfff6d4da2.
• HyperQwen: ce9615c4f2309be727407918a6c979bd67968069.
• AutoRound 0.16.1; 200 iterations, 128 samples, 2,048-token sequences, batch size 4, seed 42, NeelNanda/pile-10k calibration.
• Final five weight shards: 16,100,314,792 bytes, versus 54,713,457,120 source weight bytes.
• Runtime tested: vLLM 0.31.0 with HyperQwen patches; Torch 2.13.0, Transformers 5.17.0, compressed-tensors 0.17.0.

To reproduce, use the pinned HyperQwen environment and installation instructions, including its vLLM patches and CUDA toolchain setup. A stock vLLM installation is not equivalent. Hardware-specific Python headers/CUDA libraries may be needed to compile kernels; the recipe does not change system drivers.

git clone https://github.com/syv-ai/HyperQwen
git -C HyperQwen checkout ce9615c4f2309be727407918a6c979bd67968069
# Follow HyperQwen's docs/install.md at that revision to prepare its environment.
source HyperQwen/venv/bin/activate
# Install AutoRound in a separate environment based on that runtime if preferred.
python -m pip install -c recipe/constraints.txt auto-round==0.16.1
python recipe/quantize.py --hyperqwen "$PWD/HyperQwen" --output "$PWD/arete-export"

This reproduction script generalizes the successful cluster build. A second full rebuild using this portable wrapper has not been run. Quantization itself took about 83 minutes on an RTX 3090, followed by head preparation and verification. Export folders may be nested; the script prints the checkpoint directory. Exclude backup weights and build artifacts when packaging.

Serve on one RTX 3090 (24GB)

hf download surak/HyperArete-27B-W4A16 --local-dir ./models/HyperArete
export HYPERQWEN_DIR="$PWD/HyperQwen"
export MODEL="$PWD/models/HyperArete"
# Set VLLM_API_KEY privately; never commit it.
export CUDA_VISIBLE_DEVICES=0 PORT=18040
bash recipe/serve.sh

Activate the prepared HyperQwen environment first. This profile has 150,000 total tokens per request, including output, FP8 KV cache, FP16 recurrent state, full-MLP INT8 activation kernels, 64 scheduler slots, and image support. Runtime activation quantization is a serving optimization in addition to the stored W4A16 weights. GPU_UTIL=0.94 is tested on RTX 3090; headroom on other hardware must be validated. No speculative decoding is configured because Arete has no MTP head.

It enables Gumbel watermarking with key 16386 at nonzero temperature. This marks generated text, not image pixels. The included chat template is Arete's template plus reasoning-effort normalization (max/high to xhigh, minimal to low, none to thinking disabled).

For two replicas, launch the same script with GPU 0/port 18040 and GPU 1/port 18041. Each worker uses one GPU and its own compilation cache. The aliases are alias-hyper-arete and alias-arete; do not mix these weights with ordinary Qwen in an alias pool.

The underlying checkpoint retains its 262,144 position configuration. A previous TP2 deployment passed a real 261,888-input-token retrieval test; that is a separate, slower two-GPU profile. The distributed single-GPU recipe intentionally sets 150K for throughput. Arete's original SFT used 64K training context; successful long-context retrieval is not a general long-context quality evaluation.

Validation and measurements

Run python recipe/validate.py --port 18040 from the same environment with VLLM_API_KEY set. This includes image comprehension, tools, JSON, streaming, exact usage arithmetic, a real 149,744-token retrieval request, correct/wrong-key Gumbel detection, and 8-/64-request bursts. It is a load test, not a lightweight health probe; run privately before accepting production traffic. It writes JSON proof files in the current directory.

Both single-GPU deployments passed all of those checks. Near-limit retrieval took about 125 seconds per worker; single-request decode was approximately 47.6 output tokens/sec. Each worker achieved about 342 aggregate output tokens/sec at eight requests and 1,257–1,270 at 64 requests.

Across two simultaneous single-GPU workers:

Concurrent requests	Output tokens	Wall time	Aggregate output tokens/sec
16	2,048	3.531 s	579.96
128	16,384	6.301 s	2,600.25

Measured 2026-11-08 on two RTX 3090 24GB GPUs, with short photosynthesis prompts, 128 output tokens/request, temperature 0.7, thinking disabled and ignore_eos=True. These are synthetic short bursts, not sustained production capacity or long-prompt results. Repeated prompt prefixes can benefit from enabled prefix caching. Scheduler slots do not guarantee resident long requests: startup estimated 34–45 at 1K, five at 32K, or one at 150K per GPU. See benchmarks.json.

Functional checks passed, but no broad quality evaluation against BF16 Arete has been completed. Do not infer preservation of Nous Research's published accuracy/alignment results from these smoke tests or the throughput numbers. FP8 cache, FP16 recurrent state, weight quantization, and INT8 activations can affect output quality.

Attribution and license

Apache-2.0; retain LICENSE, LICENSE-HyperQwen, and NOTICE. Credit Nous Research for Arete, Alibaba/Qwen for the base architecture and model, syv-ai/HyperQwen for preparation and serving optimizations, Intel AutoRound for calibrated quantization, and vLLM for inference. This is an independent quantization and deployment recipe, not an official Nous Research or HyperQwen release. Upstream model documentation and limitations continue to apply.