DiffRhythm 2 ONNX

ONNX builds of DiffRhythm 2, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each precision is one folder holding the graphs and a small Python pipeline.py that does the rest (features, prompt, sampling loop, codec windows); what the graphs take and return is under How to run.

Files

The last column is the benchmark against the original model; how each model is judged (sampled songs differ from run to run) is under How to run.

Precision File Size sha256 Benchmark
fp16 fp16/decoder.onnx 263.8 MB 438d9b1d455a MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA)
fp16 fp16/dit.onnx 0.9 MB b5d0ef30eb35 MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA)
fp16 fp16/dit.onnx.data 2.16 GB 7c0e5dddbfda MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA)
fp16 fp16/latent_embed.onnx 0.6 MB 0ba86c6a5db7 MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA)
fp16 fp16/pipeline.py 8.4 kB 4699de206c4f MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA)
fp16 fp16/style_text.onnx 657.0 MB 0e7783ccd72a MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA)
fp16 fp16/style_tokenizer/special_tokens_map.json 0.3 kB 8574792bfefa MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA)
fp16 fp16/style_tokenizer/tokenizer.json 17.1 MB 794764476bf5 MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA)
fp16 fp16/style_tokenizer/tokenizer_config.json 1.2 kB 185f40a89596 MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA)
fp16 fp16/text_embed.onnx 1.0 MB c7459c7e58f1 MuLan similarity 0.532 / 0.561 (original 0.513 / 0.547, fp16); first block 37 / 35 dB vs float32 (original fp16 56 / 54); RTF 0.38 (CUDA)
int8 int8/decoder.onnx 132.9 MB 7256482d8ba7 MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA)
int8 int8/dit.onnx 1.08 GB 973af7eddcc6 MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA)
int8 int8/latent_embed.onnx 0.3 MB eea13e15f306 MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA)
int8 int8/pipeline.py 8.4 kB 4699de206c4f MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA)
int8 int8/style_text.onnx 907.0 MB c728ba88789c MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA)
int8 int8/style_tokenizer/special_tokens_map.json 0.3 kB 8574792bfefa MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA)
int8 int8/style_tokenizer/tokenizer.json 17.1 MB 794764476bf5 MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA)
int8 int8/style_tokenizer/tokenizer_config.json 1.2 kB 185f40a89596 MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA)
int8 int8/text_embed.onnx 2.1 MB 69c7641737c3 MuLan similarity 0.582 / 0.551; first block 41 / 36 dB; RTF 0.95 (CUDA)
int4 int4/decoder.onnx 132.9 MB 7256482d8ba7 DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA)
int4 int4/dit.onnx 691.4 MB ffed02e52cbc DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA)
int4 int4/latent_embed.onnx 0.3 MB eea13e15f306 DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA)
int4 int4/pipeline.py 8.4 kB 4699de206c4f DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA)
int4 int4/style_text.onnx 907.0 MB c728ba88789c DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA)
int4 int4/style_tokenizer/special_tokens_map.json 0.3 kB 8574792bfefa DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA)
int4 int4/style_tokenizer/tokenizer.json 17.1 MB 794764476bf5 DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA)
int4 int4/style_tokenizer/tokenizer_config.json 1.2 kB 185f40a89596 DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA)
int4 int4/text_embed.onnx 2.1 MB 69c7641737c3 DiT 4-bit: MuLan similarity 0.612 / 0.548; first block 23 / 8 dB; RTF 0.66 (CUDA)

How to run

License note. DiffRhythm 2's own weights (the DiT, the embeddings, the decoder) are Apache-2.0. style_text.onnx is converted from OpenMuQ/MuQ-MuLan-large, which is CC-BY-NC-4.0: non-commercial use only. It holds MuQ-MuLan's text tower: XLM-RoBERTa-base (MIT) plus MuLan's own layers. DiffRhythm 2 needs a MuQ-MuLan style embedding (512-d) to generate, so commercial use needs another source of that embedding.

DiffRhythm 2 (ASLP-lab / Xiaomi: a 1.1B block flow-matching model for full songs from lyrics and a style prompt; 48 kHz) as ONNX graphs plus pipeline.py, a numpy runtime that reproduces the official CFM.sample_block_cache + decode_audio (github.com/xiaomi-research/diffrhythm2 @13a7b09) around them.

  • dit.onnx: the 16-layer DiT: x [B, N, 512] (embedded lyrics or latents), time [B, N], position_ids [B, N], style [B, 512], past_key_values.{0..15}.key / .value [B, 16, P, 128] -> velocity [B, N, 64] and the call's own present.* [B, 16, N, 128].
  • text_embed.onnx (lyrics tokens [B, L] -> [B, L, 512]) and latent_embed.onnx (latents [B, N, 64] -> [B, N, 512]).
  • decoder.onnx: the BigVGAN decoder, one 20-frame chunk: latent [B, 64, 20] -> 48 kHz audio [B, 1, 192000].
  • style_text.onnx: MuQ-MuLan's text embedding of the style prompt: input_ids, attention_mask [1, T] -> style [1, 512] (style_tokenizer/ is its XLM-RoBERTa tokenizer).

pipeline.py does the rest in numpy:

  • the lyrics, then 2-second blocks of 10 latent frames;
  • per block, 16 Euler steps over linspace(0, 1, 16) with classifier-free guidance 2.0 against a zero style and zero lyrics, both kept in the batch;
  • the KV cache: the lyrics' keys and values (time -1) and every finished block's (time 1);
  • the end of the song where a latent frame reaches the all-ones end frame;
  • the decoder's 20-frame chunks with 5 frames of overlap.

Lyrics come in as the official frontend's phoneme ids: diffrhythm2's g2p (pypinyin, a polyphone BERT, espeak). It isn't ported here; pipeline.lyrics_to_tokens(lyrics, repo) calls it from a checkout of the official repository. On Windows, espeakng-loader supplies espeak, and pyopenjtalk-plus replaces pyopenjtalk. Output is mono; the official script's fake stereo isn't included.

import sys, soundfile as sf
sys.path.insert(0, "fp16")
import pipeline
m = pipeline.DiffRhythm2("fp16", "cuda")
tokens = pipeline.lyrics_to_tokens("[verse]\nwalking down the street in the morning light", "diffrhythm2")
song, sr = m(tokens, "pop, upbeat, female vocal, piano, drums", seconds=60, seed=0)     # 48 kHz mono
sf.write("song.wav", song, sr)

Sampling uses numpy's generator, so a seed gives the same song every run here, not the same song as PyTorch with that seed. On Windows, give CUDA sessions the run option memory.enable_memory_arena_shrinkage = gpu:0, as pipeline.py does. Every input length differs, so ONNX Runtime's GPU arena otherwise keeps growing.

Precisions. fp16/: every graph fp16 weights with fp32 inputs / outputs. int8/: weight-only int8 (per output channel; the lyrics embedding table per row). int4/: the DiT's MatMuls 4-bit MatMulNBits (block 32), the rest as int8/. No bf16 build: fp16 passes.

Benchmark. The block sampler is chaotic: even the original's own fp16 run drifts from its float32 run block by block, with the same noise. In the first block it is 56 dB from float32; in the tenth, 3 dB. So songs aren't compared sample by sample. Each build is judged the way the original is, at generation level. Two songs (style "pop, upbeat, female vocal, piano, drums" with English lyrics, and "folk, acoustic guitar, male vocal, calm" with Chinese lyrics) are generated with seeds 0-3, 20 s each. Each song is scored by MuQ-MuLan's audio-text similarity to its style prompt; MuQ-MuLan is the model's own style encoder. The original is the official sampler and decoder in fp16 on CUDA, as inference.py runs on a GPU. Pass: per song, mean similarity at most 0.03 below the original's. Real-time factor (processing time / audio time) on an RTX 3080 Laptop GPU (16 GB), sampling and decoding.

MuLan similarity, song 1 / song 2 (4 seeds) first block vs float32, same noise Real-time factor
original (PyTorch fp16, CUDA) 0.513 / 0.547 56 / 54 dB 0.70
ONNX graphs in float32 (not published) - 64 / 62 dB -
fp16/ (CUDA) 0.532 / 0.561 37 / 35 dB 0.38
int8/ (CUDA) 0.582 / 0.551 41 / 36 dB 0.95
int4/ (CUDA) 0.612 / 0.548 23 / 8 dB 0.66

Each similarity is the mean over 4 seeds; the seeds alone move it by several hundredths, so a build above the original isn't better, just within the spread. Graphs checked against the original modules before the benchmark (largest absolute difference, float32): DiT 1.7e-6 of 2.1 (past 113 positions), lyrics embedding 0, latent embedding 7e-7, decoder 1.3e-6, style text 1.2e-7 against MuQMuLan(texts=...).

How these were made

From DiffRhythm 2 at revision 9aa15742e4. The graphs were exported from the original PyTorch modules with torch.onnx (opset 17 / 18) and checked against them; language-model decoders were built with onnxruntime-genai's model builder (inputs_embeds in, KV cache in and out). Weight-only quantization: int8 is a per-output-channel int8 weight behind DequantizeLinear (or 8-bit MatMulNBits in the LM), int4 is ONNX Runtime's MatMulNBits (block 32). Each card above says which precision every graph uses. Every precision was benchmarked against the original before upload (the pass criteria are under How to run), and the sha256 column above is what the files must hash to.

License

Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.


Original model card

Di♪♪Rhythm 2: Efficient And High Fidelity Song Generation Via Block Flow Matching

License

Yuepeng Jiang, Huakang Chen, Ziqian Ning, Jixun Yao, Zerui Han, Di Wu, Meng Meng, Jian Luan, Zhonghua Fu, Lei Xie†

DiffRhythm 2 (Chinese: 谛韵, Dì Yùn) is the next-generation open-sourced music generation framework that advances the original DiffRhythm with a semi-autoregressive diffusion architecture. It is capable of generating full-length songs with precise lyric alignment and coherent musical structures. The name inherits the essence of DiffRhythm — “Diff” reflects its diffusion-based generative backbone, while “Rhythm” emphasizes its dedication to musicality and temporal flow. The Chinese name 谛韵 (Dì Yùn) continues this dual symbolism: “谛” (attentive listening) represents perceptual awareness, and “韵” (melodic charm) captures the expressive beauty of music.

📢 News and Updates

📋 TODOs

  • Support Colab.
  • Gradio support.
  • Song extension.
  • Instrumental music generation.
  • Release code and weights.
  • Release paper to Arxiv.

🔨 Inference

Following the steps below to clone the repository and install the environment.

# clone and enter the repositry
git clone https://github.com/ASLP-lab/DiffRhythm2.git
cd DiffRhythm2

# install the environment
## espeak-ng
# For Debian-like distribution (e.g. Ubuntu, Mint, etc.)
sudo apt-get install espeak-ng
# For RedHat-like distribution (e.g. CentOS, Fedora, etc.) 
sudo yum install espeak-ng
# For MacOS
brew install espeak-ng
# For Windows
# Please visit https://github.com/espeak-ng/espeak-ng/releases to download .msi installer

## install requirements
pip install -r requirements.txt

On Linux you can now simply use the inference script:

# For inference using a reference WAV file
bash inference.sh

Weights will be automatically downloaded from Hugging Face upon the first run.

Example files of lyrics and reference audio can be found in example.

📜 License & Disclaimer

DiffRhythm 2 (code and weights) is released under the Apache License 2.0. This open-source license allows you to freely use, modify, and distribute the model, as long as you include the appropriate copyright notice and disclaimer.

We do not make any profit from this model. Our goal is to provide a high-quality base model for music generation, fostering innovation in AI music and contributing to the advancement of human creativity. We hope that DiffRhythm 2 will serve as a foundation for further research and development in the field of AI-generated music.

DiffRhythm 2 enables the creation of original music across diverse genres, supporting applications in artistic creation, education, and entertainment. While designed for positive use cases, potential risks include unintentional copyright infringement through stylistic similarities, inappropriate blending of cultural musical elements, and misuse for generating harmful content. To ensure responsible deployment, users must implement verification mechanisms to confirm musical originality, disclose AI involvement in generated works, and obtain permissions when adapting protected styles.

Citation

@article{diffrhythm2,
  title={DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching},
  author={Jiang, Yuepeng and Chen, Huakang and Ning, Ziqian and Yao, Jixun and Han, Zerui and Wu, Di and Meng, Meng and Luan, Jian and Fu, Zhonghua and Xie, Lei},
  journal={arXiv preprint arXiv:2510.22950},
  year={2025}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for moonbox-dev/DiffRhythm2-ONNX

Quantized
(1)
this model

Papers for moonbox-dev/DiffRhythm2-ONNX