Been doing this a long time and It takes a lot to impress me
So keep a cheaper approximation of the machinery, then learn a compact way to repair what comes out of it.
My bet is progressive activation-residual distillation: freeze the low-bit backbone and train per-block gated, causal-convolution bottleneck correctors on the progressively corrected student’s own hidden-state trajectories to mMinimize deviation from FP16 teacher activations?
Changes to llama.cpp impressive. Good work.
Thanks, that means something coming from someone who's been at this a while.
The runtime side is all in the patch, so you've read the architecture right off the source. The training side is the part I'm keeping to myself for now, so I'll neither confirm nor correct the guess, and I'd rather not play warmer/colder with it!
What I can say is that the file and the runtime are there to be measured: KL to the FP16 logits and Q4_K_M's MMLU-Pro on the same scorer land on the card today. What are you working on? If there's a model or a deployment you'd want this on, that conversation I'm happy to have.
LOL fair enough, and I just noticed the patent pending, so I won’t push.
For this kind of work I came at it from the opposite side: no teacher, no retraining, no corrective network. Transform the frozen tensors into a representation where the error you’d otherwise have to learn largely disappears.
I got annoyed by Bonsai’s ternary buzzword-heavy launch about six weeks ago, said “hold my beer,” and about six hours later, mostly from my phone, had the same Qwen 27B family at 1.055× reference perplexity directly from the original tensors. Mine is still at a larger whole-model byte budget than Opti, though, so you’ve got me cleanly on the published size/quality point right now.
What caught my attention is that we’re apparently asking the same deeper question from opposite directions: how little of the original representation do you actually need to preserve to retain the useful computation?
And, genuinely, the way you approached it shows a mind that solves problems with thought and care. That’s usually the mark of someone doing the work for the right reasons, and it makes you easy to cheer for.
As for what I work on beyond quantization: creatively shit-disturbing cognition and theory of mind, machine memory and associative recall, attention and awareness, consequence modelling, high-dimensional geometry and algebraic algorithm optimization, quantum-compute simulation and compression, distributed/GPU compute and novel networking protocols, cognitive architectures, and whatever other interesting problem wanders close enough for me to poke it with a stick.
So, you know. Normal hobbies still quieter than trains. Cheaper than Warhammer. About as addictive as MTG? 🤷♂️