malerlab/unirqvae3-ytsv

U-MusT score-image codec (RQ-VAE, 16x compression)

The score-image tokenizer used by U-MusT, and the one the paper's reported results use. A Residual-Quantized VAE retrained from scratch on grayscale sheet music.

• Single-channel input, channel multipliers [1, 1, 2, 2, 4] for 16x compression rather than the original 32x, so each token covers a 16x16 pixel patch — roughly one or two note heads
• Attention blocks removed, specializing the model for local features
• 4 unshared codebooks of 1024 entries, model dimension 256
• Trained with a resolution-adaptive crop schedule to avoid staff-line artifacts at unseen input heights

Usage

Place both files where the code expects the image tokenizer:

vq_models/unirqvae3_f16_c1024_k4/config.yaml
vq_models/unirqvae3_f16_c1024_k4/epoch61_model.pt

The loader resolves vq_models/<vq_model> from the vq_model field of a training recipe and finds config.yaml and the checkpoint recursively, so the nesting under that directory does not matter.

Licensing

This release is licensed CC BY-NC-SA 4.0.

It is derived from a training mixture whose corpora carry differing terms: MAESTRO, ASAP and MusicNetEM (CC BY-NC-SA 4.0), MusicNet and SLakh (CC BY 4.0), BPSD (CC BY 3.0), OLiMPiC (CC BY-SA 4.0), and YTSV metadata (CC BY-NC-SA 4.0). The non-commercial and share-alike terms are applied here because they are the most restrictive verified terms in that chain.

One conflict is irreducible and is disclosed rather than resolved: CC BY-SA 4.0 and CC BY-NC-SA 4.0 are mutually incompatible, since the former permits commercial use and forbids adding restrictions while the latter requires them. Any work derived from both — including these weights — cannot satisfy both simultaneously. See the code repository for the full per-corpus license table and the conflicts.

This is not legal advice. Verify the current terms with each corpus publisher before redistributing anything derived from this release or using it commercially.

Citation

@article{jung2026umust,
  title   = {U-MusT: A Unified Framework for Cross-Modal Translation of Score Images, Symbolic Music, and Performance Audio},
  author  = {Jung, Jongmin and Kim, Dongmin and Lee, Sihun and Cho, Seola and Soh, Hyungjoon and Bukey, Irmak and Donahue, Chris and Jeong, Dasaem},
  journal = {IEEE Transactions on Audio, Speech and Language Processing},
  volume  = {34},
  pages   = {1876--1891},
  year    = {2026},
  doi     = {10.1109/TASLPRO.2025.3648794}
}