tiyuvta/DictaLM-3.0-24B-NVFP4

DictaLM-3.0-24B-Thinking NVFP4 (GPTQ)

NVFP4 (4-bit e2m1 weights with an FP8-e4m3 per-16 scale plane, 4.5 bits/element) weight-only mint of dicta-il/DictaLM-3.0-24B-Thinking, quantized with GPTQ via llm-compressor. Calibration is Hebrew-led, because this checkpoint exists to serve Hebrew.

Built with memra, a from-scratch Rust and CUDA inference engine for RTX Blackwell, by tiyuvta (inference.tiyuvta.ai).

• 15 GB, down from 24 GB for the vendor FP8 and 48 GB for BF16. Fits one 32 GB card with room for a real context window.
• 280 quantized tensors, 83 kept (norms, embeddings, lm_head).
• No g_idx: actorder is off, so there is no column permutation to reconstruct.

Quality

Measured on Global-MMLU at n=2000 per language, 5-shot, greedy single-token (the instrument, not the serving shape), against the vendor FP8 release, which is the format this model is normally served in.

arm	Hebrew	English	size
vendor FP8 (baseline)	0.6735	0.7735	24 GB
this mint (GPTQ-NVFP4)	0.6590	0.7635	15 GB

Paired McNemar on the 2000 shared sample_ids: Hebrew -1.45 points (p=0.0293), English -1.00 points. A per-tensor similarity screen cannot see a gap this size, and cannot see a much worse one either: an earlier round-to-nearest NVFP4 mint of this model scored 0.996-0.999 row cosine on every tensor family while answering Hebrew prompts with a space. Language-level evaluation is the only screen that works here.

Decode: 99.6 tok/s on one RTX 5090, three reps, vendor-default sampled (no sampling parameters in the request, so the model's own generation_config governs).

What more calibration does not buy

The shipped mint calibrates on 128 sequences of 512 tokens. Running the same recipe at 512 x 1024 (8x the tokens) and scoring it paired on the same host moved nothing: Hebrew 0.6590 -> 0.6630 (McNemar p=0.5045, 51 vs 59 disagreements) and English 0.7635 -> 0.7615 (p=0.7463). The gap to the vendor FP8 is not a calibration-volume artifact, so this artifact stays at the smaller, cheaper calibration.

Where the remaining Hebrew gap is not

Every weight family was held out of NVFP4 in turn on this exact recipe and scored paired against this artifact on the same box (Hebrew, n=2000): down_proj+o_proj p=0.5758, q,k,v p=0.5655, gate,up +0.95 pt at p=0.1296 while costing 27% of decode and 6 GB. A second mint from a different calibration seed moved one sample out of 2000 (p=1.0), so those nulls are measured against a floor that does not move. The residual gap to the vendor FP8 is not in any one family at a price worth paying, which is why this is the shipped shape.

What did not help

Weight-only per-channel FP8 on the reduction projections (down_proj, o_proj) is what makes a round-to-nearest NVFP4 mint of this model usable at all. Under GPTQ it buys nothing: 0.6625 vs 0.6590 Hebrew is McNemar p=0.5758 on paired samples, for 20% less decode throughput (80.5 vs 99.6 tok/s) and 3 GB more. GPTQ's per-column error compensation already does that work, so this mint is NVFP4 everywhere.

Serving

from openai import OpenAI
client = OpenAI(base_url="https://api.tiyuvta.ai/v1", api_key="...")

Or run it locally with memra. The checkpoint is standard compressed-tensors nvfp4-pack-quantized, so any runtime that reads that format reads this.

Reproducibility

Scored from scratch on two different hosts, weeks apart in build and driver, with identical correct-counts, not merely identical accuracies:

he  1318/2000 = 0.6590
en  1527/2000 = 0.7635

The 5-shot greedy instrument is host-independent, which is what makes the paired comparisons above mean anything. Decode throughput is NOT host-independent and is quoted for one 5090 only.

Provenance

Quantized 2026-09-06 with llm-compressor 0.13.0 plus the patch in vllm-project/llm-compressor#3144, which this mint required: IntermediatesCache rebuilt offloaded dataclasses through their own __init__, and every transformers>=5 config is a huggingface_hub @strict dataclass that rejects the internal wrapper, so the sequential pipeline (the only one a 24B fits in) could not run at all.

Calibration: dicta's own quantization calibration set, then Hebrew Wikipedia and English Wikipedia at roughly 4:1, 128 sequences of 512 tokens, seed 20260905. Global-MMLU is never touched by calibration.