topabaem/Haverbex-Stingray-35B-A3B

Haverbex Stingray 35B-A3B · IQ3_XXS-mixed

A per-tensor mixed-precision GGUF of Qwen/Qwen3.5-35B-A3B. It occupies 15.27 GB instead of 71.07 GB, a 4.65× reduction to 3.52 bits per weight, and its MMLU is statistically indistinguishable from the BF16 original.

It fits a 24 GB consumer card. Only 3B of the 35B parameters are active per token, so decoding is fast for the size.

File: haverbex-stingray-35b-a3b-IQ3_XXS-mixed.gguf

The name says what the build is: the expert tensors — the overwhelming majority of the bytes in a 35B mixture-of-experts — are IQ3_XXS, and everything around them is held higher. mixed marks it as a per-tensor allocation rather than one of llama.cpp's uniform presets.

Measured benchmarks

benchmark	setting	score
GSM8K	5-shot, flexible-extract	96.44% ± 0.51
GSM8K	5-shot, strict-match	96.36% ± 0.52
IFEval	0-shot, inst-level loose	93.76%
IFEval	0-shot, inst-level strict	90.77%
IFEval	0-shot, prompt-level loose	90.94% ± 1.24
IFEval	0-shot, prompt-level strict	86.88% ± 1.45
MATH Lvl 5	4-shot, exact_match	77.79% ± 1.08
MMLU	5-shot, 1,396-item screen	85.60%

Run with lm-evaluation-harness 0.4.12 against llama-server at llama.cpp 32e41fa5b, chat template applied, few-shot as multi-turn, max_gen_toks=2048, 16 concurrent slots, on one RTX 3090. Raw logs and per-task result JSON are in measurements/.

MATH Lvl 5 by subject: number theory 89.61%, prealgebra 85.49%, precalculus 66.67%, intermediate algebra 57.86%.

These are non-thinking numbers

The model runs in thinking mode by default and this evaluation disabled it, via --chat-template-kwargs '{"enable_thinking":false}'. That was not a stylistic choice. With thinking on and a 2,048-token generation budget, 75% of GSM8K responses came back with an empty content field: the reasoning consumed the budget and nothing was left for the answer. Disabling thinking took the loss rate to 0% and GSM8K from 24% to 97% on the same 100-item probe.

If you serve this model with thinking on, give it a generation budget in the 5,000–10,000 token range, or you will silently lose answers.

What the compression cost

Measured separately, paired per item against the BF16 GGUF through one runtime:

build	expert tensors	size	bpw	MMLU (1,396 paired)	difference	McNemar	95% CI
BF16	—	71.07 GB	16.40	85.74%	—	—	—
IQ3_XXS-mixed (this)	IQ3_XXS	15.27 GB	3.52	85.60%	−0.14 pp	0.02	[−0.85, +1.14]
IQ2_XXS variant	IQ2_XXS	11.24 GB	2.59	84.03%	−1.72 pp	6.30	[+0.43, +3.01]
IQ1_M variant	IQ1_M	9.98 GB	2.30	81.23%	−4.51 pp	31.25	[+2.96, +6.07]

The two lower variants are reported but not published. Cost per bit surrendered makes the shape clear: 16.40 → 3.52 bits costs about 0.01 pp of MMLU per bit; 3.52 → 2.59 costs 1.70; 2.59 → 2.30 costs 9.62. The curve is flat until roughly 3.5 bits and then turns hard, so this build sits just above the knee.

Also in this repository: haverbex-stingray-35b-a3b-5.00bpw.pacific, a 21.66 GB archive in the Pacific engine's own format at 5.00 bpw. It is not a GGUF and llama.cpp cannot load it. The pacific/ package here is what reads it — load_archive, verify_archive, decode and the packed runtime modules. The write path that produces such archives is not published, so this package can open and run an archive but not create one.

How it compares to other extreme-compression work

Two recent builds target the same problem from much lower bit-widths. Comparing them honestly requires stating what cannot be compared.

The three models have three different parents. This build compresses Qwen3.5-35B-A3B. Mach-1 compresses Qwen3.6-35B-A3B, a later generation. Ternary Bonsai 27B compresses Qwen3.6-27B, a dense model of different size. Their absolute scores therefore measure different underlying models as much as they measure different compression methods.

What is comparable is how much of its own parent each build keeps, and at what size:

build	parent	params	file on disk	bits per weight, on disk	retention	over how many benchmarks	measured by
Haverbex Stingray 35B-A3B · IQ3_XXS-mixed	Qwen3.5-35B-A3B	34.7 B	15.27 GB	3.52	99.8%	1 (MMLU screen)	us, paired per item
Mach-1-Additive-35B	Qwen3.6-35B-A3B	35 B	7.0 GB ᵃ	1.60 ᵃ	95.1%	12	vendor
Ternary Bonsai 27B	Qwen3.6-27B	27.3 B	7.17 GB	2.10 ᵇ	94.6%	15	vendor

ᵃ Neither figure is in Mach-1's model card; both come from the vendor's announcement post, which states 1.7 bits per weight and 7 GB. Those two do not quite agree with each other — 7 GB over 35 B parameters is 1.60 bits — and the table reports the value implied by the file size.

ᵇ Bonsai's card states 1.71 bits per weight, which is the information content of a ternary weight (1.585 bits) plus its share of the FP16 group scale, and gives 5.9 GB as the corresponding "ideal size". The shipped GGUF packs each ternary weight into a 2-bit slot, so the file you actually download is 7.17 GB, which is 2.10 bits per weight on disk. Both numbers are honest; they answer different questions, and only the second is comparable to the other rows.

Why their files are less than half the size

File size is just parameters x bits-per-weight / 8, and every row above checks out against it. Two things separate them, and only one is about compression:

• Bit-width. At 3.52 bits this build is roughly twice as dense as the other two. The same model at 1.71 bits would be 7.42 GB — essentially Mach-1's footprint. There is nothing else to find; the size gap is the bit-width gap.
• Parameter count. Bonsai also starts from a smaller parent, 27.3 B against 34.7 B here, which accounts for about a fifth of its advantage on its own.

The interesting difference is not the bit-width but how it is bought. This build is post-training quantization: no gradient step is taken, and the ladder measured what that method costs as it descends — 2.59 bits gave up 1.72 pp of MMLU and 2.30 bits gave up 4.51 pp. Around 2.10 bits, Bonsai's on-disk density, pure post-training quantization on this model would be worse still. Both of the other builds get their bit-width by training: Mach-1 reports under 15 GPU-hours of retraining, and Bonsai's ternary weights come from a training procedure rather than a rounding rule. That is what lets them hold ~95% where rounding alone cannot.

Our 99.8% is not comparable to their 95.1% and 94.6%, and reading it as "better" would be wrong. Ours is retention on a single knowledge benchmark. Theirs are means over twelve and fifteen benchmarks spanning math, code, instruction following, agentic tool use and (for Bonsai) vision. A mean over a suite that wide would almost certainly be lower than a single MMLU figure, and we have not measured it: our generative benchmarks were run on the quantized build only, never against its BF16 parent, so no retention figure exists for them.

The other honest reading is that these are different operating points, not competitors. At 3.52 bits this build gives up nothing measurable and accepts 4.65×. At ~1.7 bits the other two accept a real and stated ~5% loss for 10× and 9.4×. Our own ladder measured what the intermediate territory costs: dropping to 2.59 bits cost 1.72 pp of MMLU and 2.30 bits cost 4.51 pp, which is the same order of loss those builds report — reached by post-training quantization alone, where both of theirs involve retraining or a redesigned representation.

Where the published benchmarks overlap

Three benchmarks appear in all three cards. The columns are not like-for-like — different parents, different modes, different harnesses — so this is a reference table, not a ranking.

benchmark	Haverbex Stingray 35B-A3B	Mach-1-Additive-35B	Ternary Bonsai 27B
GSM8K	96.44	95.22	96.06
IFEval	90.77 ᵇ	85.90	85.03
MMLU	85.60 ᶜ	88.93 ᵈ	88.05 ᵈ
MATH	77.79 ᵉ	96.80 ᶠ	99.20 ᶠ

ᵇ instruction-level strict; the vendors do not state which IFEval variant they report, and the four variants of this metric differ by up to 7 points on this model alone. ᶜ our 1,396-item MMLU screen. ᵈ MMLU-Redux, a different and generally higher-scoring benchmark. ᵉ MATH Level 5, the hardest subset only. ᶠ MATH-500, which spans all difficulty levels and is not comparable to Level 5. Our mode is non-thinking; Bonsai's figures are thinking-mode; Mach-1's card does not say.

The MATH row is the clearest illustration of why these columns should not be read across: 77.79 on Level 5 and 99.20 on MATH-500 are not evidence about which model is better at mathematics.

Precision map

Experts are the whole story in a 35B mixture-of-experts with 3B active. This build puts them at IQ3_XXS and keeps everything else high: attention, shared experts, the router gate, embeddings and the output head stay at their protected types, and all normalisation tensors are left to the quantizer's defaults.

An importance matrix was used, built over a calibration mix that excludes both the MMLU items and the perplexity corpus used for evaluation. IQ3_XXS requires one.

The full per-tensor recipe is e3/recipe-e3.txt.

Usage

llama-server -m haverbex-stingray-35b-a3b-IQ3_XXS-mixed.gguf -ngl 999 -c 65536 \
  --host 127.0.0.1 --port 8080 --jinja

To reproduce the numbers above, add --chat-template-kwargs '{"enable_thinking":false}'. To use the model as intended, leave thinking on and raise the client's max_tokens.

import openai
client = openai.OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="-")
reply = client.chat.completions.create(
    model="haverbex-stingray-35b-a3b",
    messages=[{"role": "user", "content": "Explain mixture-of-experts routing briefly."}],
    max_tokens=8192,
)
print(reply.choices[0].message.content)

Note that -c is the total context divided across --parallel slots, not the context per slot.

Provenance

• base Qwen/Qwen3.5-35B-A3B at revision 59d61f3c
• llama.cpp 32e41fa5b for both quantization and evaluation
• MMLU pairing on an A100 80GB; the benchmark suite on an RTX 3090

Limits

The MMLU figure is a 1,396-item screen out of 14,042. GSM8K, IFEval and MATH Lvl 5 are complete task sets.

The benchmark suite was run on the quantized build only. The MMLU pairing is what establishes that quantization cost nothing measurable; it does not follow automatically that generative and instruction-following behaviour is equally preserved, because those were not measured against BF16.

Loglikelihood-scored benchmarks — ARC, HellaSwag, WinoGrande, TruthfulQA and the leaderboard MMLU-Pro and BBH variants — could not be run: llama.cpp returns logprobs in a shape lm-eval's loglikelihood path does not read, and does not honour echo, so prompt-token probabilities are unavailable. Agentic and coding behaviour, where this model family is strongest, is not evaluated here.

Citation

@misc{haverbexstingray35ba3b2026,
  title        = {Haverbex Stingray 35B-A3B: quality-first mixed-precision quantization
                  of Qwen3.5-35B-A3B at 3.52 bits per weight},
  author       = {topabaem},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/topabaem/Haverbex-Stingray-35B-A3B}},
  note         = {GGUF build \texttt{haverbex-stingray-35b-a3b-IQ3\_XXS-mixed.gguf};
                  base model Qwen/Qwen3.5-35B-A3B at revision 59d61f3c;
                  quantized and evaluated with llama.cpp 32e41fa5b}
}

Please also cite the base model and the tools this build depends on:

@misc{qwen35,
  title        = {Qwen3.5-35B-A3B},
  author       = {{Qwen Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Qwen/Qwen3.5-35B-A3B}}
}

@software{llamacpp,
  title  = {llama.cpp},
  author = {Gerganov, Georgi and {llama.cpp contributors}},
  url    = {https://github.com/ggml-org/llama.cpp}
}

@software{lmeval,
  title   = {A framework for few-shot language model evaluation},
  author  = {Gao, Leo and others},
  version = {0.4.12},
  url     = {https://github.com/EleutherAI/lm-evaluation-harness}
}

Models referenced in the comparison: Mach-1-Additive-35B (Syzygy Research) and Ternary-Bonsai-27B-gguf (Prism ML). All figures attributed to them are taken from their own model cards, except Mach-1's bits-per-weight and file size as noted above.