phase-sensitive-kv — reproduction checkpoints

Model collection for an independent, from-scratch reproduction of "Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression" (ByteDance Seed, arXiv:2609.36322).

This is an independent reproduction, not the official implementation. Code, training details and evaluation results live in the companion repo: ShengenWu/phase-sensitive-kv.

All models share the same Qwen3-0.6B backbone, FineWeb-Edu data (data_seed 777), optimizer/schedule, and evaluation protocol. They differ only in attention: c0-* uses the paper-reference chunked KV compression (W=S=8, 1 KV head, scalar gates, separate K/V branches); fullattn-* is the data-matched uncompressed control (full causal attention, unbounded cache).

Checkpoints

Path	Variant	Training tokens	Seed
c0-1p75b/seed-42.pt	compressed (C0)	1.75B	42
c0-1p75b/seed-43.pt	compressed (C0)	1.75B	43
fullattn-1p75b/seed-42.pt	full attention (control)	1.75B	42
fullattn-1p75b/seed-43.pt	full attention (control)	1.75B	43
c0-3p5b/seed-42.pt	compressed (C0)	3.5B	42
fullattn-3p5b/seed-42.pt	full attention (control)	3.5B	42

Each file is a torch.save checkpoint dict (bf16 weights, ~2.2 GB) with keys config (full ResCSAConfig) and model (state dict), stripped of optimizer state. Load with the companion repo's package:

import torch
from rescsa.model.modeling import ResCSAForCausalLM

ck = torch.load("c0-3p5b/seed-42.pt", map_location="cpu", weights_only=False)
model = ResCSAForCausalLM(ck["config"])   # variant/layout are carried in the config
model.load_state_dict(ck["model"])

Headline result

Phase gap (best minus worst source-phase accuracy, prefix_local protocol) grows with budget in the compressed model while the uncompressed control stays flat: 5.0 → 27.4 → 46.6 pp at 500M → 1.75B → 3.5B tokens for C0 (seed 42), versus ≤ 3.5 pp for the full-attention control at matched budgets. See RESULTS.md for full numbers and confidence intervals.

Citation

If you use these checkpoints, please cite the original paper:

@article{zhu2026periodic,
  title   = {Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression},
  author  = {Zhu, Xingyu and Yi, Pu (Luke) and Cheng, Ziheng and Lv, Ang and Liu, Jing and Ying, Lexing and Ma, Yiyuan and Dong, Xin},
  journal = {arXiv preprint arXiv:2609.36322},
  year    = {2026}
}