quantum_tts

quantum_tts is a LoRA fine-tune of Orpheus-TTS (unsloth/orpheus-3b-0.1-ft), adapted to a single Kiswahili voice using SNAC-tokenized speech data. At inference time the LoRA adapter is combined with the base model via PEFT, which preserves the base model's original capabilities alongside the fine-tuned voice — including the ability to perform voice characteristic transfer to English text, despite no English training data being used for this speaker.

Model Details

• Base model: unsloth/orpheus-3b-0.1-ft (Llama-3B backbone)
• Fine-tuning method: LoRA adapter via Unsloth + PEFT
• Audio codec: SNAC (hubertsiuzdak/snac_24khz)
• Language(s): Kiswahili (training), with emergent English voice-transfer capability at inference
• License: CC0 1.0 (inherited from training data)
• Training data: rlabz/swa_lug_tts, tokenized into Orpheus's interleaved text/SNAC-code format

Intended Use

quantum_tts is intended for single-voice Kiswahili text-to-speech synthesis, and experimentally for cross-lingual voice transfer to English text using the same speaker's voice characteristics. Because the adapter is combined with the base model at inference (rather than replacing it), the model retains the general TTS capabilities of the Orpheus base — this is what enables it to speak English text in the fine-tuned voice even though English audio was never part of training.

This model is not intended as a general-purpose multi-speaker or multi-language TTS system.

How to Get Started

Load the model

from unsloth import FastLanguageModel
from peft import PeftModel

# Base model
base_model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/orpheus-3b-0.1-ft",
    max_seq_length=4096,   # choose any for long context
    dtype=None,            # None = auto detection
    # load_in_4bit=True,   # set True to reduce memory usage
)

# Attach the quantum_tts LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    "rlabz/quantum_tts",  # remove this step to run the base model only
)

Run inference

import gc
import torch
from IPython.display import display, Audio
from snac import SNAC

snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").to("cuda")

gc.collect()
torch.cuda.empty_cache()

FastLanguageModel.for_inference(model)
snac_model.to("cpu")  # free GPU for LLM generation

prompts_ = [(f"{chosen_voice}: " + p) if chosen_voice else p for p in prompts]

# Tokenize and wrap with Orpheus's special tokens
all_input_ids = [tokenizer(p, return_tensors="pt").input_ids for p in prompts_]

start_token = torch.tensor([[128259]], dtype=torch.int64)          # start of human
end_tokens  = torch.tensor([[128009, 128260]], dtype=torch.int64)  # EOT, end of human

all_modified_input_ids = [
    torch.cat([start_token, ids, end_tokens], dim=1) for ids in all_input_ids
]

# Pad batch
max_length = max(t.shape[1] for t in all_modified_input_ids)
all_padded_tensors, all_attention_masks = [], []
for modified in all_modified_input_ids:
    padding = max_length - modified.shape[1]
    padded = torch.cat([torch.full((1, padding), 128263, dtype=torch.int64), modified], dim=1)
    mask = torch.cat(
        [torch.zeros((1, padding), dtype=torch.int64),
         torch.ones((1, modified.shape[1]), dtype=torch.int64)], dim=1
    )
    all_padded_tensors.append(padded)
    all_attention_masks.append(mask)

input_ids = torch.cat(all_padded_tensors, dim=0).to("cuda")
attention_mask = torch.cat(all_attention_masks, dim=0).to("cuda")

# Generate
with torch.inference_mode():
    generated_ids = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        max_new_tokens=1200,
        do_sample=True,
        temperature=0.6,
        top_p=0.95,
        repetition_penalty=1.1,
        num_return_sequences=1,
        eos_token_id=128258,
        use_cache=True,
    )

# Extract SNAC audio tokens, then decode with SNAC to get waveforms
# (see full decoding + redistribute_codes logic in the repo's inference script)

The snippet above is abbreviated for readability — the full inference script (token extraction, redistribute_codes, and SNAC decoding into playable audio) is included in this project notebook.

Training Details

Training Data

Fine-tuned on the SNAC-tokenized version of rlabz/swa_lug_tts, filtered to a single Kiswahili speaker and formatted into Orpheus's interleaved input_ids / labels / attention_mask layout (text tokens + SNAC audio codes, per the Orpheus fine-tuning data format).

Training Procedure

Fine-tuned with Hugging Face Trainer on top of an Unsloth-loaded Orpheus base model, using LoRA adapters (via PEFT) rather than full fine-tuning.

Training hyperparameters:

Hyperparameter	Value
Per-device train batch size	1
Gradient accumulation steps	8
Effective batch size	8
Learning rate	3.4e-5
LR scheduler	cosine
Warmup steps	50
Max steps	600
Optimizer	adamw_8bit
Weight decay	0.001
Eval strategy	steps (every 100)
Save strategy	steps (every 100)
Load best model at end	Yes
Metric direction	greater_is_better=False (lower eval loss is better)
Seed	3407

from transformers import TrainingArguments, Trainer, DataCollatorForSeq2Seq

trainer = Trainer(
    model=model,
    data_collator=data_collator,
    train_dataset=dataset,
    eval_dataset=val_dataset,
    args=TrainingArguments(
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=3.5e-6,
        warmup_steps=50,
        max_steps=600,
        logging_steps=100,
        eval_strategy="steps",
        eval_steps=100,
        save_steps=100,
        load_best_model_at_end=True,
        greater_is_better=False,
        optim="adamw_8bit",
        weight_decay=0.001,
        lr_scheduler_type="cosine",
        seed=3407,
        output_dir="outputs",
        report_to="none",
    ),
)

Compute Infrastructure

Trained via Unsloth-accelerated fine-tuning on the Orpheus-3B backbone.

Notable Capability: Cross-Lingual Voice Transfer

Because inference combines the LoRA adapter with the unmodified base model (rather than serving a fully-merged fine-tuned checkpoint), quantum_tts retains the base Orpheus model's broader multilingual TTS ability. In practice this means the fine-tuned voice can be prompted with English text and will speak it in the same voice characteristics learned from the Kiswahili fine-tuning data — even though the training set contained no English audio for this speaker. This is an emergent property of the PEFT-based inference setup rather than something explicitly trained for, so quality and consistency on English text should be evaluated case by case.

Acknowledgements

• Orpheus-TTS by Canopy Labs
• SNAC neural audio codec
• Unsloth for efficient fine-tuning
• Training data derived from rlabz/swa_lug_tts

---