Agnes-3.0-Flash-oQ4e (Mixed-Precision Quantized)

Agnes-3.0-Flash converted to qwen3_5 format and quantized with oMLX's enhanced oQ4e streaming mode.

Model Details

Property	Value
Base model	Agnes-AI/Agnes-3.0-Flash
Conversion	mlx_lm with custom Agnes→qwen3_5 adapter
Quantization	oQ4e (mixed-precision affine, group_size=64)
Total size	~19 GB
Architecture	72 layers, 5120 hidden, 19456 intermediate

Bit Distribution

Bit Width	Layers
4-bit (base)	428
5-bit	128
6-bit	1
8-bit	1 (embed_tokens)

Usage

from mlx_lm import load, generate

model, tokenizer = load("hermitdave/Agnes-3.0-Flash-oQ4e")

response = generate(
    model,
    tokenizer,
    prompt="The capital of France is",
    max_tokens=128
)
print(response)

Or with oMLX:

# Place in ~/.omlx/models/hermitdave/Agnes-3.0-Flash-oQ4e/
# Restart oMLX → model appears in dropdown

Sensitivity Analysis

The oq_imatrix_report.json in this repo contains 558 per-layer sensitivity entries collected with oqe_code_multilingual (128 samples, seq_length=512). Sensitivity was measured on a uniform 4-bit proxy to determine per-layer bit assignments.

MTP Speculative Decoding

This quant does not include MTP weights (oQ normalizes mtp_num_hidden_layers to 0). For speculative decoding, use the standalone MTP drafter with mlx_vlm.server.

mlx_vlm.server serve \
    --model hermitdave/Agnes-3.0-Flash-oQ4e \
    --draft-model hermitdave/Agnes-3.0-Flash-MTP-drafter

Converion Notes

• One-centered RMSNorm: mlx_weight = hf_weight + 1.0
• Parallel FFN folding: intermediate_size: 19456 (main + parallel concatenated)
• Attention renaming: delta_attn → linear_attn, global_attn → self_attn
• Vision: Not included (Agnes vision tower dimensions incompatible with Qwen3-VL)
• MTP: Extracted separately for drafter use

Attribution

• Original model: Agnes-AI
• Converted with Hermes Agent + mlx_lm
• Quantized with oMLX