OmniSinger

OmniSinger is a fine-tuned OmniVoice checkpoint that generates singing from text with explicit note and duration control. It is used exactly like a regular OmniVoice checkpoint. The only difference is the input text format, described below.

Give it a short reference clip for the voice, a lyric line annotated with pitch and duration per word, and it sings the line in that voice.

Installation

pip install omnivoice soundfile torch

Quick start

import re

import soundfile as sf
import torch
from omnivoice import OmniVoice


def compute_duration_from_text(text: str) -> float:
    """Sum every `pitch:duration` pair in the text (centiseconds) -> seconds."""
    matches = re.findall(r"-?\d+:(\d+)", text)
    if not matches:
        raise ValueError("No {pitch:duration} tags found in text.")
    return sum(int(d) for d in matches) / 100.0


model = OmniVoice.from_pretrained(
    "mindvu/OmniSinger",  # this repo
    device_map="cuda:0",
    dtype=torch.float16,
)
# Apple Silicon: device_map="mps"
# Intel Arc GPU: device_map="xpu"

text = (
    "[singing] <silence>{0:28} hold{68:144} me{67:33} close{65:73} and{63:50} "
    "hold{59:30} me{69:36} fast{68:114} <silence>{0:45} the{65:24} magic{63:92} "
    "spell{56:45} you{66:40} cast{63:110} <silence>{0:45}"
)

duration = compute_duration_from_text(text)
print(f"Target duration: {duration:.2f}s")

audio = model.generate(
    text=text,
    ref_audio="reference.mp3",   # short clip of the voice you want
    # ref_text="Transcript of the reference clip.",  # optional, see below
    duration=duration,           # must match the total of the tags
    postprocess_output=False,
)  # list of np.ndarray, shape (T,), 24 kHz

sf.write("out.wav", audio[0], 24000)

If you omit ref_text, the transcript of the reference audio is obtained automatically with Whisper ASR.

Input text format

The text is a lyric line where every word carries a note tag and rests are written as <silence> tags.

[singing] word{pitch:duration} word{pitch:duration} <silence>{0:duration} ...

Element	Meaning
[singing]	Required prefix that switches the model to singing mode.
word{P:D}	A sung word. P is the pitch as a MIDI note number (e.g. 60 = C4, 69 = A4). D is the note length in centiseconds (1/100 s).
<silence>{0:D}	A rest of D centiseconds. Pitch is always 0.
word{P1:D1,P2:D2}	Melisma: one word sung over several notes, as comma-separated pitch:duration pairs.

Examples:

hold{68:144}              -> "hold" on note 68 for 1.44 s
<silence>{0:28}           -> 0.28 s rest
baby{60:104,65:104}       -> "baby" sung as two notes, 1.04 s each

Setting duration

duration (seconds) should equal the sum of all durations in the text, including silences and every note of a melisma. The helper compute_duration_from_text in the quick start does this for you:

total_seconds = sum(all D values) / 100

More examples

English with melisma

text = "[singing] <silence>{0:76} baby{60:69,65:139} i'm{59:20,55:21} <silence>{0:132} dancing{70:97,64:37} in{65:26} the{67:26} dark{60:48} <silence>{0:114} with{54:28} you{69:70} between{63:94} my{62:58} arms{56:42} <silence>{0:33}"

Vietnamese

text = "[singing] <silence>{0:28} bèo{68:144} dạt{67:33} mây{65:73} trôi{63:50} chốn{59:30} xa{69:36} xôi{68:114} <silence>{0:45} anh{65:24} ơi{63:92} em{56:45} vẫn{66:40} đợi{66:110} <silence>{0:45}"

Mixed Vietnamese / English

text = "[singing] <silence>{0:28} ôm{68:144} em{67:33} gần{65:73} và{63:50} cầm{59:30} tay{69:36} em{68:114} <silence>{0:45} the{65:24} magic{63:92} spell{56:45} you{66:40} cast{63:110} <silence>{0:45}"

Tips

• Reference audio: use a clean clip of a single speaker or singer, a few seconds to ~15 s. The output timbre follows it.
• Sync of notes and lyrics: every word needs a tag. Words without {pitch:duration} will not be timed correctly.
• Short phrases work best: split long songs into lines or phrases and generate them one at a time, then concatenate the audio.
• Rests matter: explicit <silence> tags at phrase boundaries give the most natural breathing and phrasing.
• Plain speech: OmniSinger can still be prompted like a normal OmniVoice model by leaving out [singing] and the note tags (and not passing a tag-derived duration), though it is optimized for singing.

Output

model.generate returns a list of NumPy arrays (one per input), mono, 24 kHz:

sf.write("out.wav", audio[0], 24000)

License

Please see the license section of the base OmniVoice model and of the data this checkpoint was fine-tuned on.

Disclaimer

Users are strictly prohibited from using this model for unauthorized voice cloning, voice impersonation, fraud, scams, or any other illegal or unethical activities. All users shall ensure full compliance with applicable local laws, regulations, and ethical standards. The developers assume no liability for any misuse of this model and advocate for responsible AI development and use, encouraging the community to uphold safety and ethical principles in AI research and applications.