forkjoin-ai/lfm3-1.7b-knotchain

LFM3-1.7B KNOT (knotchain)

LFM3-1.7B converted to the dense KNOT container for fat-station, the sovereign inference engine in open-source/gnosis/distributed-inference. KNOT is a single-file format: magic KNOT, u16 version, u32 manifest length, JSON manifest (config + tensor table), then a contiguous payload.

Files

file	bytes	content
LFM3-1.7B-Q8_0.knot	1,838,020,045	all 154 large tensors Q8_0, 133 small tensors F32
lfm3-1.7b-hybrid.knot	1,201,881,065	136 Q4_K + 18 Q8_0 + 133 F32
lfm3-1.7b-q4_k_m.knot	1,091,409,177	136 Q4_K + 18 Q6_K + 133 F32

Why a hybrid exists

Q4_K_M ships token_embd (which is also the tied LM head) and ffn_down at Q6_K, and the Q6_K kernel reaches only ~11.7 GB/s at large row counts where the Q8_0 kernel reaches ~22.2 GB/s (measured on an Apple M-series host). Since the LM head runs once per generated token, Q6_K there is the latency floor. The hybrid keeps every Q4_K matrix and re-sources those 18 tensors at Q8_0 from the Q8_0 knot — the same weights at higher precision, 65% of the Q8_0 footprint.

Admission evidence

All three knots are the same model and produce the same next token:

prompt: "The capital of France is"   (token ids [597, 5205, 302, 3980, 355])
Q8_0     top-1 4741 " Paris" 15.856   top-2 278 " the" 13.263
hybrid   top-1 4741 " Paris" 16.087   top-2 278 " the" 13.862
Q4_K_M   top-1 4741 " Paris" 16.071   top-2 278 " the" 14.013

Top-1 agreement across all three; logits within 1.5%. Reproduce with:

lfm3-probe --knot <file>.knot --gguf tok.gguf \
  --text 'The capital of France is' --gen 0 --top-k 3

Measured performance (and its conditions)

Kernel streaming throughput, matvec-stream (512 MB walk, RAM-bound), 7680x2048, n=20, Apple M-series, single process:

q8_0  15.9 GB/s    q4_k  12.1 GB/s    q6_k  12.3 GB/s

Whole-model matvec time (LFM3_MATVEC_PROFILE=1, 17 generated tokens): Q8_0 2173.5 ms vs hybrid 2124.4 ms — parity on 65% of the bytes. Per-call the hybrid is faster (q4_k 0.56 ms vs q8_0 0.64 ms); its 396 q8_0 calls are the LM-head-sized ones.

Honest caveat: end-to-end tok/s could NOT be measured stably on the reference host — three serving engines plus a 1.8 GB knot exceeded free RAM and the same knot swung 3.3-10 tok/s across consecutive runs. The matvec totals above are the trustworthy number. A hot-cache microbenchmark is the wrong instrument for this question and reported Q4_K as faster while the model showed parity.

Config notes

LFM3 needs its tri-hybrid plan in the manifest. Two encoder bugs are worth knowing about because both fail silently:

1. lfm3.attention.head_count_kv is a 24-entry per-layer ARRAY [0,0,4,0,4,...] (conv layers carry no KV heads). Reading element 0 yields num_kv_heads: 0.
2. GGUF tensor names (shortconv.in_proj, post_attention_norm, post_ffw_norm) differ from the names the engine loads (conv_in_proj, post_attn_norm, post_ffn_norm). A missing name returns ZEROS rather than erroring, so a knot with unrenamed tensors runs with 87 of 287 tensors silently zeroed.

Both are handled by gguf-to-knot.py at encode time.

Provenance

Base model: LiquidAI/LFM3-1.7B (LFM Open License). Quantized upstream as LiquidAI/LFM3-1.7B-GGUF (Q8_0, Q4_K_M); KNOT container produced by open-source/gnosis/distributed-inference/gguf-to-knot.py, hybrid by station/build-hybrid-knot.py. Tensor payloads are verbatim from the upstream GGUF — no re-quantization is performed by the converter.