uncrayon/kvpark-hub6

kvpark hub6: switch local models mid-conversation without re-reading it

When a local agent hands a long conversation from one model to another, the new model normally has to re-read (prefill) the whole context first: tens of seconds on a consumer GPU. This bundle lets it skip most of that. It translates the old model's attention cache (keys and values) into the new model's cache through one shared latent, and the new model only re-reads the last ~2,048 tokens itself.

• 15 open models, 7 cache families, 12 tokenizers, one set of closed-form linear adapters (2 per model, no pairwise maps, no gradient training).
• A median 2.2–2.9× faster to resume after a switch than a native prefill on coding-agent sessions (range 1.4–3.9×, thermal pauses removed), on one AMD Ryzen AI Max+ 395 (Radeon 8060S iGPU). The translation itself takes 0.4–1.8 s.
• The prompt stays exact. llama.cpp reuses only cells whose tokens match. Translation only changes the cached values of old tokens, which are approximate, so it shifts what the model writes.

Used by kvpark bridge in uncrayon/kvpark (MIT). Method and full measurements: docs/bridge.md and experiments/universal_kv/ there.

How much time it saves

These are 36 live hand-offs of coding-agent sessions between two running llama-servers, Qwen3.8-27B and Gemma-4-31B. Each point is one session; timings exclude the machine's thermal-guard pauses. A switch costs about the same whatever the length (translation plus a fixed 2,048-token native tail). A native prefill grows with the session, so the saving grows with length. On 12–15K-token chats into Qwen3.8, a switch took 9.7–12.5 s of wall time against 48–52 s of native prefill. Thermal pauses make that ratio uncertain: the conservative bound is 1.9–3.9×.

How good the translated cache is

We score the new model teacher-forced on the next message after the switch, against its own full prefill. The baseline is an information-free cache: the same switch, but every translated cell holds the adapter's mean, so nothing comes from the old model. The bars show how much of that baseline's extra loss (KL divergence) the translation removes: 0% means no better than the baseline, and 100% means as good as re-reading everything.

Dense, sliding-window and MLA models gain the most. Mamba hybrids with few attention layers gain little, so for them a native prefill is the honest choice.

On coding-agent sessions

Agent sessions are harder than chat. The next message draws on file contents and test output far back in the context. A hub fitted on chat removes a smaller share there; adding 11 agent sessions to the fit raises it from 19% to 27% (two-model hubs, better on 13 of 18 sessions, p = .013). That same agent-fitted hub costs on chat (35% → 28%). About half of what is left lives in tokens the hub does not cover, such as template markup:

Does the agent still do the right thing?

After a switch from Qwen3.8, Gemma-4-31B chose the same tool as after a native prefill in 15 of 18 sessions, the same file or command in 8 and an identical call in 5. The baseline did about as well (14, 10 and 4). On these sessions the next step is carried by the native tail and Gemma's own system prompt, not by the translation. What translation buys is a lower divergence from a full re-read; whether that matters over later steps is unmeasured.

What to expect

Time to resume after a switch	5–18 s, median 7–9 s, on 5–14K-token agent sessions (native prefill: 10–37 s); the saving grows with length
Translation itself	0.4–1.8 s on an iGPU (Vulkan or ROCm); 2–14 s on the CPU
Quality, chat	removes 16–83% of the information-free baseline's loss for 9 of 11 targets
Quality, coding-agent sessions	removes 16–38% (Qwen3.8 ↔ Gemma-4-31B)
Next tool call	same tool as native in 12–15 of 18 sessions; no measurable gain over the baseline yet
Correctness	token ids stay exact and only old cached values are approximate; a translated cache can be undone (kvpark bridge undo)

Example

Run each model's llama-server with a slot directory, register the models, and point the bridge at this bundle:

hf download uncrayon/kvpark-hub6 --local-dir hub6
kvpark bridge add qwen38 --url http://127.0.0.1:18080 --gguf /models/Qwen3.8-27B-Q8_0.gguf --slot-dir ~/.local/share/kvpark/slots/qwen38 --native-prefix
kvpark bridge add gemma-4-31b --url http://127.0.0.1:18081 --gguf /models/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf --slot-dir ~/.local/share/kvpark/slots/gemma
kvpark bridge engine --python /path/to/python --script experiments/universal_kv/engine.py --hub hub6 --device auto
kvpark bridge switch --from qwen38 --to gemma-4-31b --conversation conversation.json

Per-target settings (--native-prefix, --recurrent-mean for hybrid targets, --no-source) are in docs/bridge.md. conversation.json is the conversation in the OpenAI chat format, ending with the message the new model should answer. Here is a real switch of a 9,105-token coding-agent session from Qwen3.8 to Gemma-4-31B (abridged):

{"P": 9105, "t0": 7057, "restored": 7057, "tail": 2048, "backend": "ggml", "device": "Vulkan0",
 "timings": {"source_save_s": 0.22, "engine_s": 1.48, "restore_s": 0.35, "total_s": 2.23}}

That's 7,057 tokens of Gemma's cache written from Qwen's in 2.2 s. Gemma then reads only the last 2,048 tokens itself, and the server reuses exactly the 7,057 translated cells.

Models

Fitted jointly: Qwen3.8-27B, Muse-Glimmer-30B, Qwen3-4B, Ornith-1.5-9B. Onboarded (each model fits only its own adapters): LFM2-2.6B, DeepSeek-V2-Lite, Qwen3.6-35B-A3B, GLM-4.7-Flash, Granite-4.0-H-Tiny, Falcon-H1-3B, Nemotron-3.5-Lightning-30B-A3B, Kimi-Linear-48B-A3B, Ling-3.0-Tiny, Gemma-4-26B-A4B, Gemma-4-31B.

The adapters were fitted on caches of specific GGUF files: Q8_0, except Gemma 4 (QAT UD-Q4_K_XL), GLM-4.7-Flash (Q5_K_M) and Kimi-Linear (Q4_K_M). Other quantizations of the same model will usually work, less well. Nemotron-3.5 translates out worse than no translation, so register it as a target only (kvpark bridge add --no-source).

Files

• manifest.json: settings, models, and the sha256 of every file (checked on load).
• <model>.<k|v>.encoder.f16.npy and <model>.<k|v>.decoder.f16.npy: packed adapters. The GPU engines read f16, and the float32 readers widen it.
• <model>.<k|v>.view.f32.npy: per-model view projections (PCA per layer, then whitening).

This is the f16 copy of hub6-20261009-v2 (published.of_manifest_sha256 in the manifest); about 17.7 GB.

Limits

All measurements come from one machine. Scores are teacher-forced on the first 256 tokens of one message per conversation, and some intervals rest on few conversations. The agent results cover one model pair and the next tool call, not task completion.

Data

Fitted on the models' own caches of the train split of multi-turn SmolTalk2 conversations (chat, reasoning and tool-calling subsets), each rendered with the model's own chat template. No coding-agent sessions and no competition data went into this bundle. Each model's weights are under their own license; this bundle contains only linear maps between their activations.