nbeerbower/Inkling-Gutenberg-DPO-colibri-int4

Inkling-Gutenberg-DPO — sabrewing int4

nbeerbower/Inkling-Gutenberg-DPO (a Gutenberg-DPO fine-tune of Thinking Machines Inkling, 975B total / 41B active) converted to the sabrewing / colibrì expert-streaming int4 container, for CPU-first inference on a single machine.

Quantized directly from the merged BF16 fine-tune (not from NVFP4), so the routed experts take a single quantization step: bf16 → int4.

Format

• Routed experts (~95% of parameters): int4, symmetric per-row scales, packed U8 + .qs f32 sidecars — the standard sabrewing/colibrì container.
• Attention, dense MLP, shared experts, embeddings, lm_head: bf16 passthrough.
• Norms, short-conv weights, relative-position banks, router (+bias/scales): f32.
• MTP head (out-mtp.safetensors) included for future speculative decode.
• Vision/audio encoders not included (text-only inference).

Total ≈ 479 GiB. Runtime RSS ≈ 90 GB (bf16 residents) + expert cache.

Usage

git clone -b main https://github.com/Schneewolf-Labs/sabrewing
cd sabrewing/c && make inkling                 # or: make inkling CUDA=1
SWING_MODEL=/path/to/this/repo ./swing api     # OpenAI API + web UI on :8000
# or one-shot:
SNAP=/path/to/this/repo ./inkling -p "Write the opening of a novel." -n 256

Recommended inference settings

This is a preference-tuned creative-writing model and it runs peaked — it commits hard to a voice (a large DPO reward margin). Two practical consequences, both validated on this int4 snapshot via the sabrewing OpenAI server:

• Do not use greedy / temperature = 0. It falls into repetition loops.
• Sample, through the chat template. With temperature it writes cleanly; raw completion is off-distribution for a chat DPO and degenerates.

Setting	Recommended
temperature	0.7 – 0.9 (0.8 is the sweet spot)
top_p	0.9 – 0.95
repetition_penalty	1.15 – 1.25

• 0.7 — most controlled and coherent, still distinctly literary.
• 0.9 — richer imagery; occasional long run-on sentences.
• ≥1.0 — vivid but tends toward breathless, unpunctuated cascades.

Example (temp 0.7, top_p 0.9, rep 1.2) — "Write the opening paragraph of a novel about an old lighthouse keeper on his final night of duty."

The lantern had burned for forty-seven years without fail, and Silas knew—without checking the clock or consulting any log—that tonight it would not be lit again by his hand. He climbed to its crown with legs that remembered every rung more clearly than they did stairs in town, carrying only one match he never intended to strike.

At the recommended settings the prose is sensory and literary and noticeably lighter on AI clichés than the base model. For tighter sentence control favor temperature 0.7 + repetition_penalty ~1.2.

Notes

• Quantization faithfulness: on a tiny oracle the sabrewing converter reproduces the transformers reference to 0.00% (f32) / within noise (int4).
• This is a creative-writing / anti-slop tune; evaluate on prose, not factual QA.
• First int4 conversion of the first public Inkling fine-tune, runnable on a single workstation.