Notes You Can't Delete — model checkpoints
Small from-scratch symbolic-music transformers from the study Notes You Can't Delete: take a musical idea out of everything a model is trained on, and it rebuilds the idea anyway.
Each model is a ~25–26M-parameter decoder-only GPT (nanoGPT shape) trained on REMI-tokenized MIDI. Every checkpoint is a matched control or edited twin — the whole point of the study is comparing pairs that differ in exactly one thing.
Full write-up, audio, and the numbers: https://notes-you-cant-delete.vercel.app
Run one
pip install torch numpy miditok symusic huggingface_hub
python generate_example.py --model g_metal_s1 # writes out.mid
generate_example.py and model.py (the ~120-line GPT) are in this repo. Each checkpoint is checkpoints/<name>/ckpt.pt (a state dict + config) with its config.json; tokenizers are in tokenizers/.
The checkpoints
Metal (guitar & full band)
| model | what it is |
|---|---|
g_metal_s1 |
Guitar base — 22,037 metal guitar tracks, 163M tokens, val 0.23 |
g_metal_band_s1 |
Full band — guitar + bass + drums, ctx 2048, 316M tokens, val 0.18 |
g_metal_deb5 |
The b5-deleted twin of g_metal_s1 — never saw a tritone, still expects it at 99% |
g_metal_drums_s1 |
Drums-only baseline (uses the metal_band tokenizer) |
g_metal_td_s2 |
Tech-death fine-tune |
g_metal_bm_s3 |
Black-metal fine-tune |
g_metal_minus_bm · g_metal_minus_dj · g_metal_minus_rand |
Lineage-cut models — a genre (or random data) removed |
Tonal (classical & jazz)
| model | what it is |
|---|---|
blues_control_s0 · blues_deblue_s0 |
Blue-note deletion — matched pair over the Weimar Jazz Database |
inject_control_s0 · inject_loose_s0 · inject_clean_s0 |
Blue-note injection into classical, two doses |
cond_control_s0 · cond_synthall_s0 · cond_condv2_s0 |
The invented rule — uniform habit vs chord-conditional |
remap_permuted_s0 |
Scrambled tonal function (un-scrambles to coherent tonality) |
chordmel_s0 |
Chord-melody representation |
p1_control · p1_dom7_drop |
Dominant-seventh deletion pilot |
g1_control_s1 |
Classical control |
The one result
Delete a concept from the training data — a note, a chord, a whole genre — and a model that never sees it once still expects it, reconstructed from the grammar around the hole. The blue note comes back at 97–98%, the metal tritone at 99%, the dominant seventh at ~90%, black metal fully. The only place deletion finally bites is a genuinely novel trait (djent's sub-bass register) that nothing else in the data implies.
Corpora: Weimar Jazz Database, MAESTRO v3, and a public Kaggle metal-MIDI dump (PDDL). Models are small-scale and from-scratch; caveats are stated throughout the site.