lancejames221b/muse-glimmer-30b-longctx-patch

Muse-Glimmer-30B long-context patch

A 4-byte header edit that unlocks 262k (and beyond) context on Muse-Glimmer-30B GGUF quants, plus the measured reason not to reach for YaRN when you do it.

No weights here. This repo is a patch script, a llama.cpp wrapper, and findings. Bring your own quant (e.g. unsloth/Muse-Glimmer-30B-GGUF).

TL;DR

• The GGUF header ships context_length = 131072. It is a uint32 you can edit in place; tensor data is untouched.
• Serving past that needs the header raised. Runtimes clamp to it.
• Do not add RoPE/YaRN scaling. On this architecture it costs prefill time and buys nothing. See Why YaRN backfires.
• Verified working on a single RTX 4090 (24 GiB) at 262k, 8 parallel slots, with quantized KV.

Quick start

# inspect
python scripts/patch_context_length.py model.gguf --show

# patch to 262k
python scripts/patch_context_length.py model.gguf 262144

# put it back
python scripts/patch_context_length.py model.gguf --restore

The script locates the key by name rather than trusting a hardcoded offset, validates the GGUF magic and the value type before writing, and reads the value back to confirm the write stuck. It refuses to touch a file whose header layout does not match.

Self-check, no model file needed:

python scripts/patch_context_length.py --selftest

Why YaRN backfires here

The obvious move after raising the header is to add --rope-scaling yarn. On Muse-Glimmer that makes things worse. From config.json:

• 39 of 52 layers are sliding_attention, hard-capped at a 2048-token window.
• 13 of 52 layers are full attention with rope_theta = 0 -- NoPE, no positional encoding at all.

RoPE scaling only affects layers that use RoPE. Here those are the sliding layers, and their window does not grow no matter what you scale. The global layers that actually see the full sequence have no RoPE to scale.

So YaRN distorts local attention, changes nothing at long range, and you still pay full prefill cost. Measured outcome at 1M ctx: prefill slow enough to be unusable in an interactive session. Removing the YaRN flags and relying on the header alone was strictly better.

This is why the config-only path is the one to take. It is also what upstream long-context evaluations of this model used.

Tested configurations

Context	Method	Result
262,144	header only	Works. Current daily driver on a 4090.
524,288	header only	Loads and serves, 8 slots, ~6 GiB VRAM free.
1,048,576	header + YaRN	Loads, but prefill unusable. Not recommended.

Reference points from the wider community: config-only 512k passing needle-in-haystack / multi-hop / semantic retrieval, with counting-and-aggregation tasks degrading first; ~832k retrieval verified on multi-GPU rigs. Long-context retrieval and long-context reasoning are not the same axis -- retrieval holds up considerably further than aggregation does.

KV cache cost

Only the 13 global layers hold full-length KV. With GQA 16:1 (2 KV heads, head_dim 128) that is far cheaper than the parameter count suggests:

Context	F16	q8_0 K / q4_0 V
262k	3.25 GiB	~1.2 GiB
512k	6.50 GiB	~2.4 GiB
1M	13.0 GiB	~4.9 GiB

KV was never the binding constraint. Prefill compute is.

LM Studio users: the KV-quant trap

LM Studio's per-model config exposes kCacheQuantizationType / vCacheQuantizationType, and silently drops them. Config says q4_0, the live process runs --cache-type-k f16. At long context that is the difference between fitting in 24 GiB and not.

scripts/llama-server-wrapper.sh is a shim that sits in front of the real llama-server binary and forces the flags through. It rewrites argv only when --model matches your target file; every other model passes through untouched. It also strips any rope/YaRN flags so a stale config cannot resurrect them.

Install it beside the backend binary:

cd <lm-studio>/extensions/backends/llama.cpp-<platform>/
mv llama-server llama-server.real
cp /path/to/llama-server-wrapper.sh llama-server
chmod +x llama-server

Point MUSE_TARGET_MODEL / MUSE_CTX_SIZE at your file and context.

LM Studio backend upgrades overwrite this. A fresh llama.cpp-<platform> extension ships its own llama-server. Reinstall after updating.

Verify what is actually running

lms ps reports LM Studio's intent, not the served configuration. With a wrapper in play the two can disagree. Read the live process:

PID=$(pgrep -f 'llama-server.real' | head -1)
tr '\0' '\n' < /proc/$PID/cmdline | grep -A1 -E 'ctx-size|parallel|cache-type'
tr '\0' '\n' < /proc/$PID/cmdline | grep -iE 'yarn|rope' || echo "no rope flags - correct"

Gotchas

• Re-downloading the model reverts the patch. A fresh pull ships 131072. Re-run the script.
• Header, runtime flag, and any launcher config must agree. llama.cpp clamps to the smallest of them.
• Endpoint quirks on recent llama.cpp builds: /slots works, /v1/slots 404s, /metrics needs --metrics.
• With unified KV, all slots report the full n_ctx -- they share one pool rather than each owning a private copy.

Files

scripts/patch_context_length.py   # the patch, with a runnable selftest
scripts/llama-server-wrapper.sh   # forces KV-quant flags, strips rope/yarn

License

Apache-2.0, matching the base model. Patch tooling only -- no weights are redistributed here.