DiffRhythm 2 ONNX
ONNX builds of DiffRhythm 2, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each
precision is one folder holding the graphs and a small Python pipeline.py that does the rest (features, prompt,
sampling loop, codec windows); what the graphs take and return is under How to run.
Files
The last column is the benchmark against the original model; how each model is judged (sampled songs differ from run to run) is under How to run.
| Precision | File | Size | sha256 | Benchmark |
|---|---|---|---|---|
| fp16 | fp16/decoder.onnx |
263.8 MB | 438d9b1d455a |
MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA) |
| fp16 | fp16/dit.onnx |
0.9 MB | b5d0ef30eb35 |
MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA) |
| fp16 | fp16/dit.onnx.data |
2.16 GB | 7c0e5dddbfda |
MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA) |
| fp16 | fp16/latent_embed.onnx |
0.6 MB | 0ba86c6a5db7 |
MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA) |
| fp16 | fp16/pipeline.py |
8.4 kB | 4699de206c4f |
MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA) |
| fp16 | fp16/style_text.onnx |
657.0 MB | 0e7783ccd72a |
MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA) |
| fp16 | fp16/style_tokenizer/special_tokens_map.json |
0.3 kB | 8574792bfefa |
MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA) |
| fp16 | fp16/style_tokenizer/tokenizer.json |
17.1 MB | 794764476bf5 |
MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA) |
| fp16 | fp16/style_tokenizer/tokenizer_config.json |
1.2 kB | 185f40a89596 |
MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA) |
| fp16 | fp16/text_embed.onnx |
1.0 MB | c7459c7e58f1 |
MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA) |
| int8 | int8/decoder.onnx |
132.9 MB | 7256482d8ba7 |
MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA) |
| int8 | int8/dit.onnx |
1.08 GB | 973af7eddcc6 |
MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA) |
| int8 | int8/latent_embed.onnx |
0.3 MB | eea13e15f306 |
MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA) |
| int8 | int8/pipeline.py |
8.4 kB | 4699de206c4f |
MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA) |
| int8 | int8/style_text.onnx |
907.0 MB | c728ba88789c |
MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA) |
| int8 | int8/style_tokenizer/special_tokens_map.json |
0.3 kB | 8574792bfefa |
MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA) |
| int8 | int8/style_tokenizer/tokenizer.json |
17.1 MB | 794764476bf5 |
MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA) |
| int8 | int8/style_tokenizer/tokenizer_config.json |
1.2 kB | 185f40a89596 |
MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA) |
| int8 | int8/text_embed.onnx |
2.1 MB | 69c7641737c3 |
MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA) |
| int4 | int4/decoder.onnx |
132.9 MB | 7256482d8ba7 |
DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA) |
| int4 | int4/dit.onnx |
691.4 MB | ffed02e52cbc |
DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA) |
| int4 | int4/latent_embed.onnx |
0.3 MB | eea13e15f306 |
DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA) |
| int4 | int4/pipeline.py |
8.4 kB | 4699de206c4f |
DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA) |
| int4 | int4/style_text.onnx |
907.0 MB | c728ba88789c |
DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA) |
| int4 | int4/style_tokenizer/special_tokens_map.json |
0.3 kB | 8574792bfefa |
DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA) |
| int4 | int4/style_tokenizer/tokenizer.json |
17.1 MB | 794764476bf5 |
DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA) |
| int4 | int4/style_tokenizer/tokenizer_config.json |
1.2 kB | 185f40a89596 |
DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA) |
| int4 | int4/text_embed.onnx |
2.1 MB | 69c7641737c3 |
DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA) |
How to run
License note. DiffRhythm 2's own weights (the DiT, the embeddings, the decoder) are Apache-2.0. style_text.onnx is
converted from OpenMuQ/MuQ-MuLan-large, which is CC-BY-NC-4.0: non-commercial use only. It holds MuQ-MuLan's text
tower: XLM-RoBERTa-base (MIT) plus MuLan's own layers. DiffRhythm 2 needs a MuQ-MuLan style embedding (512-d) to
generate, so commercial use needs another source of that embedding.
DiffRhythm 2 (ASLP-lab / Xiaomi: a 1.1B block flow-matching model for full songs from lyrics and a style prompt;
48 kHz) as ONNX graphs plus pipeline.py, a numpy runtime that reproduces the official CFM.sample_block_cache +
decode_audio (github.com/xiaomi-research/diffrhythm2 @13a7b09) around them.
dit.onnx: the 16-layer DiT:x[B, N, 512] (embedded lyrics or latents),time[B, N],position_ids[B, N],style[B, 512],past_key_values.{0..15}.key / .value[B, 16, P, 128] ->velocity[B, N, 64] and the call's ownpresent.*[B, 16, N, 128].text_embed.onnx(lyrics tokens [B, L] -> [B, L, 512]) andlatent_embed.onnx(latents [B, N, 64] -> [B, N, 512]).decoder.onnx: the BigVGAN decoder, one 20-frame chunk: latent [B, 64, 20] -> 48 kHz audio [B, 1, 192000].style_text.onnx: MuQ-MuLan's text embedding of the style prompt:input_ids,attention_mask[1, T] ->style[1, 512] (style_tokenizer/is its XLM-RoBERTa tokenizer).
pipeline.py does the rest in numpy:
- the lyrics, then 2-second blocks of 10 latent frames;
- per block, 16 Euler steps over linspace(0, 1, 16) with classifier-free guidance 2.0 against a zero style and zero lyrics, both kept in the batch;
- the KV cache: the lyrics' keys and values (time -1) and every finished block's (time 1);
- the end of the song where a latent frame reaches the all-ones end frame;
- the decoder's 20-frame chunks with 5 frames of overlap.
Lyrics come in as the official frontend's phoneme ids: diffrhythm2's g2p (pypinyin, a polyphone BERT, espeak). It
isn't ported here; pipeline.lyrics_to_tokens(lyrics, repo) calls it from a checkout of the official repository.
On Windows, espeakng-loader supplies espeak, and pyopenjtalk-plus replaces pyopenjtalk. Output is mono; the
official script's fake stereo isn't included.
import sys, soundfile as sf
sys.path.insert(0, "fp16")
import pipeline
m = pipeline.DiffRhythm2("fp16", "cuda")
tokens = pipeline.lyrics_to_tokens("[verse]\nwalking down the street in the morning light", "diffrhythm2")
song, sr = m(tokens, "pop, upbeat, female vocal, piano, drums", seconds=60, seed=0) # 48 kHz mono
sf.write("song.wav", song, sr)
Sampling uses numpy's generator, so a seed gives the same song every run here, not the same song as PyTorch with that
seed. On Windows, give CUDA sessions the run option memory.enable_memory_arena_shrinkage = gpu:0, as pipeline.py
does. Every input length differs, so ONNX Runtime's GPU arena otherwise keeps growing.
Precisions. fp16/: every graph fp16 weights with fp32 inputs / outputs. int8/: weight-only int8 (per output
channel; the lyrics embedding table per row). int4/: the DiT's MatMuls 4-bit MatMulNBits (block 32), the rest as
int8/. No bf16 build: fp16 passes.
Benchmark. The block sampler is chaotic: even the original's own fp16 run drifts from its float32 run block by block, with the same noise. In the first block it is 56 dB from float32; in the tenth, 3 dB. So songs aren't compared sample by sample. Each build is judged the way the original is, at generation level. Two songs (style "pop, upbeat, female vocal, piano, drums" with English lyrics, and "folk, acoustic guitar, male vocal, calm" with Chinese lyrics) are generated with seeds 0-3, 20 s each. Each song is scored by MuQ-MuLan's audio-text similarity to its style prompt; MuQ-MuLan is the model's own style encoder. The original is the official sampler and decoder in fp16 on CUDA, as inference.py runs on a GPU. Pass: per song, mean similarity at most 0.03 below the original's. Real-time factor (processing time / audio time) on an RTX 3080 Laptop GPU (16 GB), sampling and decoding.
| MuLan similarity, song 1 / song 2 (4 seeds) | first block vs float32, same noise | Real-time factor | |
|---|---|---|---|
| original (PyTorch fp16, CUDA) | 0.513 / 0.547 | 56 / 54 dB | 0.70 |
| ONNX graphs in float32 (not published) | - | 64 / 62 dB | - |
fp16/ (CUDA) |
0.532 / 0.561 | 37 / 35 dB | 0.38 |
int8/ (CUDA) |
0.582 / 0.551 | 41 / 36 dB | 0.95 |
int4/ (CUDA) |
0.612 / 0.548 | 23 / 8 dB | 0.66 |
Each similarity is the mean over 4 seeds; the seeds alone move it by several hundredths, so a build above the original
isn't better, just within the spread. Graphs checked against the original modules before the benchmark (largest
absolute difference, float32): DiT 1.7e-6 of 2.1 (past 113 positions), lyrics embedding 0, latent embedding 7e-7,
decoder 1.3e-6, style text 1.2e-7 against MuQMuLan(texts=...).
How these were made
From DiffRhythm 2 at revision 9aa15742e4. The graphs were exported from the original PyTorch modules with
torch.onnx (opset 17 / 18) and checked against them; language-model decoders were built with onnxruntime-genai's
model builder (inputs_embeds in, KV cache in and out). Weight-only quantization: int8 is a per-output-channel int8
weight behind DequantizeLinear (or 8-bit MatMulNBits in the LM), int4 is ONNX Runtime's MatMulNBits (block 32). Each
card above says which precision every graph uses. Every precision was benchmarked against the original before upload
(the pass criteria are under How to run), and the sha256 column above is what the files must hash to.
License
Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.
Original model card
Di♪♪Rhythm 2: Efficient And High Fidelity Song Generation Via Block Flow Matching
Yuepeng Jiang, Huakang Chen, Ziqian Ning, Jixun Yao, Zerui Han, Di Wu, Meng Meng, Jian Luan, Zhonghua Fu, Lei Xie†
DiffRhythm 2 (Chinese: 谛韵, Dì Yùn) is the next-generation open-sourced music generation framework that advances the original DiffRhythm with a semi-autoregressive diffusion architecture. It is capable of generating full-length songs with precise lyric alignment and coherent musical structures. The name inherits the essence of DiffRhythm — “Diff” reflects its diffusion-based generative backbone, while “Rhythm” emphasizes its dedication to musicality and temporal flow. The Chinese name 谛韵 (Dì Yùn) continues this dual symbolism: “谛” (attentive listening) represents perceptual awareness, and “韵” (melodic charm) captures the expressive beauty of music.
📢 News and Updates
- 2025.10.30 🚀 We released the DiffRhythm2 paper, demo code, and model weights.
📋 TODOs
- Support Colab.
- Gradio support.
- Song extension.
- Instrumental music generation.
- Release code and weights.
- Release paper to Arxiv.
🔨 Inference
Following the steps below to clone the repository and install the environment.
# clone and enter the repositry
git clone https://github.com/ASLP-lab/DiffRhythm2.git
cd DiffRhythm2
# install the environment
## espeak-ng
# For Debian-like distribution (e.g. Ubuntu, Mint, etc.)
sudo apt-get install espeak-ng
# For RedHat-like distribution (e.g. CentOS, Fedora, etc.)
sudo yum install espeak-ng
# For MacOS
brew install espeak-ng
# For Windows
# Please visit https://github.com/espeak-ng/espeak-ng/releases to download .msi installer
## install requirements
pip install -r requirements.txt
On Linux you can now simply use the inference script:
# For inference using a reference WAV file
bash inference.sh
Weights will be automatically downloaded from Hugging Face upon the first run.
Example files of lyrics and reference audio can be found in example.
📜 License & Disclaimer
DiffRhythm 2 (code and weights) is released under the Apache License 2.0. This open-source license allows you to freely use, modify, and distribute the model, as long as you include the appropriate copyright notice and disclaimer.
We do not make any profit from this model. Our goal is to provide a high-quality base model for music generation, fostering innovation in AI music and contributing to the advancement of human creativity. We hope that DiffRhythm 2 will serve as a foundation for further research and development in the field of AI-generated music.
DiffRhythm 2 enables the creation of original music across diverse genres, supporting applications in artistic creation, education, and entertainment. While designed for positive use cases, potential risks include unintentional copyright infringement through stylistic similarities, inappropriate blending of cultural musical elements, and misuse for generating harmful content. To ensure responsible deployment, users must implement verification mechanisms to confirm musical originality, disclose AI involvement in generated works, and obtain permissions when adapting protected styles.
Citation
@article{diffrhythm2,
title={DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching},
author={Jiang, Yuepeng and Chen, Huakang and Ning, Ziqian and Yao, Jixun and Han, Zerui and Wu, Di and Meng, Meng and Luan, Jian and Fu, Zhonghua and Xie, Lei},
journal={arXiv preprint arXiv:2510.22950},
year={2025}
}
Model tree for moonbox-dev/DiffRhythm2-ONNX
Base model
ASLP-lab/DiffRhythm2