Gander-heretic-ARA-NVFP4

The Thinker of justsumguy/Gander-heretic-ARA, quantised with NVIDIA ModelOpt 0.46.1, plus the unmodified BF16 Talker. The Thinker is 7.2 GB and the Talker 0.7 GB. The abliteration details and numbers are on the BF16 page. In short: heretic ARA branch, ARA-LoRA rank 128, trial 4, refusals 98 → 7 of 100, KL 0.128 on recheck, no healing applied.

Quantisation (thinker/)

Component	Format	Notes
llm (252 linears)	NVFP4 W4A4	block 16, FP8 E4M3 block scales, FP32 global scale, static activation scale; lm_head and embed_tokens BF16; KV cache not quantised
vpm (162 linears)	FP8 E4M3 per tensor, W8A8	patch_embedding BF16
apm, resampler, audio_projection_layer	BF16	unchanged

talker/ is BF16 and identical to the BF16 repository's.

• Recipes: ModelOpt NVFP4_DEFAULT_CFG for llm; FP8_DEFAULT_CFG for vpm, with patch_embedding excluded.

• llm calibration: 256 samples × 512 tokens of tatsu-lab/alpaca (shuffled, seed 0), run through the extracted text tower. Mean NLL on 8 calibration chats: BF16 4.335, NVFP4 fake-quant 4.408 (+0.073).

• vpm calibration: 80 images from johnowhitaker/imagenette2-320. Three quarters were unsliced; one quarter used max_slice_nums=4.

• vpm held-out check (16 images):

  Output	Mean cosine	Min cosine	Mean rel. L2
  vision tower, last hidden state	0.972	0.957	0.234
  resampler	0.989	0.977	0.145

• Hardware:

  • One RTX 5060 Ti 16 GB (SM120) and one RTX 3050 Ti Laptop 4 GB (SM86), with lm_head on CPU during calibration. The host had 15 GB of RAM.
  • The BF16 tower was streamed from the shards across both GPUs.

Loading: not verified

This is a ModelOpt export: llm weights are packed uint8 with weight_scale, weight_scale_2 and input_scale tensors, and hf_quant_config.json says MIXED_PRECISION. Neither from_pretrained with the remote code's pinned transformers nor the Gander runtime loads it as-is. A ModelOpt-aware loader is needed.

The Gander runtime's NVFP4 path has not been confirmed. Realtime duplex behaviour and performance have not been measured on this build.

License

Apache-2.0, as upstream. See thinker/LICENSE and thinker/NOTICE. The modifications are the abliteration of the Thinker's language tower and the quantisation described above.