jollenchen/Nivora

Qwen3.5-0.8B-NG4

Qwen3.5-0.8B-NG4 is an experimental grouped 4-bit conversion of Qwen3.5 0.8B for the Nivora on-device inference runtime.

The model is stored in the experimental NGUF container and uses the NG4 weight layout. NG4 is designed specifically for regular, predictable T=1 matrix-vector decoding on Apple GPUs.

This repository is intended for runtime research, Metal kernel development, numerical parity testing, and performance evaluation. It is not yet a general-purpose replacement for GGUF, MLX, or Transformers model files.

Model Summary

Property	Value
Base architecture	Qwen3.5
Base model	Qwen3.5 0.8B
Parameters	Approximately 752M
Container	NGUF v1
Quantization	NG4 affine grouped 4-bit
Group size	64
Standard row tile	32
Embedding/head row tile	64
Scale format	FP16
Bias format	FP16
Packed weights	Unsigned 4-bit nibbles
Primary target	Apple Metal
Decode target	Autoregressive T=1
Runtime	Nivora experimental NG4 runtime
Status	Experimental

Important Notice

The initial release of this model was converted from an already quantized GGUF model containing:

• Q4_0 tensors
• Q8_0 token embeddings
• FP32 normalization and recurrent-state parameters

The conversion path was therefore:

Original model
→ Q4_0 / Q8_0 GGUF
→ dequantized approximation
→ NG4 affine grouped 4-bit

This is a requantized model.

Its manifest reports:

{
  "precision": "requantized",
  "quantized_types": [
    "Q4_0",
    "Q8_0"
  ],
  "quality_status": "experimental-requantized"
}

Requantization may introduce additional quality loss beyond ordinary 4-bit quantization. For quality-sensitive use, a future NG4 release converted directly from BF16 or FP16 source weights is recommended.

What Is NG4?

NG4 is a regular grouped 4-bit format developed for the Nivora runtime.

Each weight group contains 64 values and uses affine quantization:

dequantized_weight = scale × quantized_value + bias

Each group stores:

64 × 4-bit weights
1 × FP16 scale
1 × FP16 bias

The approximate storage cost is:

32 bytes packed weights
+ 2 bytes scale
+ 2 bytes bias
= 36 bytes per 64 weights
≈ 4.5 bits per weight

Additional tensor alignment and higher-precision tensors increase the effective model-wide bits per weight.

Physical Layout

NG4 does not use a conventional row-only tensor layout.

Large linear tensors are packed using a tile-oriented layout:

output row tile
→ input group
→ local output row
→ packed 4-bit words
→ per-row scale
→ per-row bias

The standard layout is:

row tile: 32
group size: 64
packing: 8 weights per UInt32

Embedding and output-head-compatible tensors use:

row tile: 64
group size: 64

This layout is intended to provide:

• predictable Metal memory access
• coalesced packed-weight reads
• regular scale and bias access
• SIMD-group cooperative matrix-vector execution
• a dedicated path for output-head reduction
• future fused GPU top-k and argmax operations

Tensor Policy

Large linear tensors are converted to NG4, including tensors such as:

blk.*.attn_qkv.weight
blk.*.attn_q.weight
blk.*.attn_k.weight
blk.*.attn_v.weight
blk.*.attn_output.weight
blk.*.attn_gate.weight
blk.*.ffn_gate.weight
blk.*.ffn_up.weight
blk.*.ffn_down.weight
blk.*.ssm_beta.weight
blk.*.ssm_out.weight
token_embd.weight

Small or numerically sensitive tensors are retained in FP16, including:

normalization weights
SSM decay parameters
SSM time-step biases
SSM convolution weights
small recurrent parameters

This mixed storage policy should not be confused with mixed-bit quantization. Large matrix weights remain consistently grouped 4-bit; selected small tensors remain FP16 for numerical stability and because quantizing them offers little memory benefit.

Model Structure

The model contains 24 transformer blocks:

• 18 Gated DeltaNet / recurrent blocks
• 6 full-attention blocks
• full-attention interval: every fourth block

Relevant architecture properties include:

Property	Value
Hidden size	1,024
Feed-forward size	3,584
Layers	24
Attention heads	8
KV heads	2 on attention layers
Vocabulary size	248,320
SSM state size	128
SSM inner size	2,048
SSM groups	16

The large vocabulary makes the embedding and output-head-compatible tensor particularly important for decode performance research.