You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Access to the U-MusT translation model weights is granted for non-commercial research use. These weights are trained on a mixture that includes corpora licensed CC BY-NC-SA 4.0 and CC BY-SA 4.0, so the non-commercial and share-alike terms travel to the weights and to anything generated with them. Requests are reviewed manually; please describe your intended use so it can be assessed.

U-MusT translation models

Translation model weights for U-MusT, a unified framework for cross-modal translation between score images, symbolic notation, MIDI and performance audio.

Both directions use an identical architecture — 12 encoder and 12 decoder layers, model dimension 1024, feed-forward 4096, 16 attention heads, plus a one-layer sub-decoder with 8 heads for codebook-wise prediction. The two directions do not share weights.

Run directory	Direction	Instrumentation	Image codec
`run-20250225_062905-9n1554as`	Image-to-Audio	piano	`unirqvae3`
`run-20250130_150202-x9znhap2`	Image-to-Audio	multi-instrument	`unirqvae`
`run-20250128_025927-ks0ibl4v`	Audio-to-Image	multi-instrument	`unirqvae`

Usage

Download into the code repository's models/ directory, keeping the run-directory names:

hf download malerlab/u-must --local-dir models/

Then point the inference script at a run directory:

python3 infer.py input.mxl --run_path models/run-20250225_062905-9n1554as -o output/

Mind which codec you infer with. These three runs do not all use the same score-image codec — the piano run expects unirqvae3, the strings and Audio-to-Image runs expect unirqvae. infer.py reads the right one from each run's own files/config.yaml, so download both image codecs and let it choose. If you drive the model yourself, or pair a checkpoint with a token dataset by hand, check the table under Codec compatibility first — a mismatch produces plausible-looking garbage rather than an error.

You also need the codecs — unirqvae3-ytsv or unirqvae-ytsv for score images, and unidac4-ytsv for audio. Each run's tokenizer generation is recorded in its own files/config.yaml and is read from there automatically.

Intended use

Research use. These weights are trained on a mixture that includes non-commercial corpora; see Licensing below.

Codec compatibility — read before pairing with a checkpoint

Image tokens are not interchangeable between codec generations. The unirqvae and unirqvae3 codebooks differ, so the same token index means different things. Pairing a checkpoint with tokens from the other generation produces meaningless output, silently.

Released run	Direction	Image codec it expects
`run-20250225_062905-9n1554as`	Image-to-Audio, piano	`unirqvae3`
`run-20250130_150202-x9znhap2`	Image-to-Audio, multi-instrument	`unirqvae`
`run-20250128_025927-ks0ibl4v`	Audio-to-Image, multi-instrument	`unirqvae`

Every published image-token dataset in this collection is unirqvae3, so only the piano Image-to-Audio run pairs with them directly. For the strings and Audio-to-Image runs, image tokens must be re-baked with unirqvae-ytsv using scripts/bake_image_tokens.py from the code repository. This matters most for the Audio-to-Image direction, where image tokens are the output target — evaluating it against unirqvae3 tokens measures nothing.

infer.py reads the tokenizer from each run's own files/config.yaml, so inference on a downloaded checkpoint selects the right codec automatically. The mismatch only arises when pairing a checkpoint with a token dataset by hand.

Licensing

This release is licensed CC BY-NC-SA 4.0.

It is derived from a training mixture whose corpora carry differing terms: MAESTRO, ASAP and MusicNetEM (CC BY-NC-SA 4.0), MusicNet and SLakh (CC BY 4.0), BPSD (CC BY 3.0), OLiMPiC (CC BY-SA 4.0), and YTSV metadata (CC BY-NC-SA 4.0). The non-commercial and share-alike terms are applied here because they are the most restrictive verified terms in that chain.

One conflict is irreducible and is disclosed rather than resolved: CC BY-SA 4.0 and CC BY-NC-SA 4.0 are mutually incompatible, since the former permits commercial use and forbids adding restrictions while the latter requires them. Any work derived from both — including these weights — cannot satisfy both simultaneously. See the code repository for the full per-corpus license table and the conflicts.

This is not legal advice. Verify the current terms with each corpus publisher before redistributing anything derived from this release or using it commercially.

Citation

@article{jung2026umust,
  title   = {U-MusT: A Unified Framework for Cross-Modal Translation of Score Images, Symbolic Music, and Performance Audio},
  author  = {Jung, Jongmin and Kim, Dongmin and Lee, Sihun and Cho, Seola and Soh, Hyungjoon and Bukey, Irmak and Donahue, Chris and Jeong, Dasaem},
  journal = {IEEE Transactions on Audio, Speech and Language Processing},
  volume  = {34},
  pages   = {1876--1891},
  year    = {2026},
  doi     = {10.1109/TASLPRO.2025.3648794}
}

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

malerlab
/

u-must