huzey/Qwen3.5-9B-MLP-Cache

MLP Cache for Qwen3.5-9B

Trained Routers and predictor tables for MLP Cache on Qwen3.5-9B. Download them and run the model with a cache attached, with no training.

What this is

MLP Cache stores small local predictors for the MLPs of a frozen Transformer. When an MLP input falls in a region the cache has seen, a predictor answers instead of the MLP. Otherwise, the original MLP runs. The model's weights do not change.

Tables

Each table set has all 32 layers and shares the same 64 Routers.

--budget	Table	Error budget τ	Enabled predictor memory	Download
1	cache_1gib	0.001	17.77 GiB	21.5 GiB
4	cache_4gib	0.003	96.36 GiB	111.6 GiB

Quick start

Requires Linux, Python 3.11 or newer and a GPU.

pip install 'mlp-cache[model] @ https://huggingface.co/huzey/Qwen3.5-9B-MLP-Cache/resolve/v2/code/mlp-cache-0.3.0.zip'
mlp-cache download --budget 1 --output cache
mlp-cache run --cache cache --table cache_1gib --image photo.jpg --question "What is in this image?"

The output has the answer and hit rates. Layers 0, 1, 3 and 31 stay uncached by default, as in the paper. Use --layers to choose others.

Tested with PyTorch 2.13 and Transformers 5.12. Newer versions also work, but numerics can differ slightly: on 30 questions with PyTorch 2.15 and Transformers 5.22, the total hit count changed by 0.2%.

From Python:

from mlp_cache import attach
from mlp_cache.models import generate, load_model

model, processor = load_model("Qwen/Qwen3.5-9B")
with attach(model, "cache", table="cache_1gib") as handle:
    tokens = generate(model, processor, "What is in this image?", image="photo.jpg")
    print(processor.decode(tokens, skip_special_tokens=True))
    print(handle.stats()["combined"])

The code bundle in code/ also builds new caches, and its README explains how.

Files

Path	Contents
routers/layer_XX/	The Vision and Other Routers of a layer, and their scales
cache_1gib/layer_XX/, cache_4gib/layer_XX/	payload.bin (INT8 predictors), control.npz (addresses, ranks, coordinates, support ranges) and manifest.json
configs/paper.json	Enabled layers and the base model revision
code/	The code, as an installable bundle
SHA256SUMS	Checksums of all files

The table format is described in docs/FORMAT.md in the code bundle. These tables were built for base model revision c202236235762e1c871ad0ccb60c8ee5ba337b9a.

Results reported in the paper

These numbers come from the paper. They use 28 cached layers, on MME-Coarse and MMStar-Coarse.

Measure	τ = 0.001, 1 GiB/layer	τ = 0.003, 4 GiB/layer
Share of MLP calls replaced	8.48%, 7.39%	27.67%, 24.09%
Accuracy change (points)	0.00, −1.87	−2.22, −0.93
Predictor memory	17.77 GiB	96.36 GiB
Decode speed on A100	1.04×	1.10×
Prefill speed on A100	0.91×	0.89×

Limits

• Speed: the paper's timings used a separate native backend and optimized kernels, which are not in the code bundle. This runtime is for checking outputs and hit rates and may be slower than the original model.
• Layer choice: the paper selects layers with a separate search, which is not included.

License

The code, Routers and tables use the MIT license. The base model and datasets keep their own licenses.