Hugging Face's logo

formospeech
/
cutetts-hakka-community-1

Text-to-Speech
Safetensors
Hakka Chinese
cutetts
zero-shot
voice-cloning
taiwanese-hakka
hakka
Eval Results (legacy)
Model card Files
xet
 Community

You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is released under the Creative Commons Attribution-NonCommercial 4.0 International License, also known as CC BY-NC 4.0.

Access is limited to non-commercial use. Users must provide appropriate attribution when sharing or adapting the model where required by the license. Commercial use, including use in paid products, paid services, or commercial deployment, is not permitted under this license.

Log in or Sign Up to review the conditions and access this model content.

CuteTTS Hakka Community 1

formospeech/cutetts-hakka-community-1 is a Taiwanese Hakka finetune of OPPOer/CuteTTS, a ~230M-parameter autoregressive TTS model on continuous latents.

A 4-step distilled version that is about 1.8–2x faster is available as formospeech/cutetts-hakka-community-1-distill.

Demo: try both models in the CuteTTS Taiwanese Hakka Space.

Summary

  • Base model: OPPOer/CuteTTS
  • Model type: CuteTTS (variant: base)
  • Primary use case: Taiwanese Hakka zero-shot voice cloning
  • Released checkpoint: step 27000 of a 30000-step finetune
  • Inference: 10 uniform flow-matching steps with LM-level classifier-free guidance (cfg_strength=2.0)
  • Supported generation mode: voice cloning from a reference recording. The reference transcript is not needed.

Usage

Install the Hakka-enabled CuteTTS fork:

This fork is required. The checkpoint was finetuned with the target dialect named in CuteTTS's voice-clone instruction, and upstream CuteTTS has no way to pass a dialect. The fork adds a dialect argument that takes the same labels as FormoSpeech/OmniVoice-hakka.

Access and Authentication

This model is hosted as a gated Hugging Face repository. Before using it:

  1. Visit the model page and request access.
  2. Log in with the same Hugging Face account that has been granted access.
  3. Authenticate your local environment with a Hugging Face access token.

A read token is sufficient for inference.

pip install -U huggingface_hub
hf auth login

Alternatively, you can provide the token through the HF_TOKEN environment variable:

export HF_TOKEN=hf_xxx

Do not hard-code your Hugging Face token in scripts, notebooks, or public repositories.

If you see an error such as Cannot access gated repo, make sure that:

  • your Hugging Face account has been granted access to this model;
  • hf auth whoami shows the expected account;
  • HF_HUB_DISABLE_IMPLICIT_TOKEN is not set.

Installation

Step 1 (optional): Install PyTorch

The fork works with PyTorch >= 2.5.1 and transformers 4.51 through 5.x. Step 2 alone installs the latest releases, whose default CUDA build needs a recent NVIDIA driver. With an older driver, install a matching PyTorch build first, e.g.

pip install torch==2.5.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121

See the PyTorch official site for other platforms.

Step 2: Install CuteTTS-hakka

pip install git+https://github.com/FormoSpeech/CuteTTS-hakka.git

Python API 

import soundfile as sf
from cutetts import CuteTTS

model = CuteTTS.from_pretrained("formospeech/cutetts-hakka-community-1", device="cuda")

result = model.generate(
    "客語語音合成測試。",
    mode="voice_clone",
    reference_audio="ref.wav",
    dialect="客語四縣腔",
)

sf.write("out.wav", result.waveform.squeeze(0).float().numpy(), result.sample_rate)

To synthesize many sentences, generate_batch decodes them together and is much faster (see Speed):

results = model.generate_batch(
    ["第一句。", "第二句。"],
    mode="voice_clone",
    reference_audio="ref.wav",  # or one path per sentence
    dialect="客語四縣腔",        # or one label per sentence
    batch_size=16,
)

generate_stream takes the same arguments as generate and yields audio chunks while decoding; generate_batch_stream does the same for generate_batch, yielding (index, chunk) pairs.

If the same voices are used again and again, encode each reference once with create_voice_prompt and pass the result as reference_audio. The audio is the same as passing the path, but the reference is not re-encoded on every request, which cuts time to first audio under batching several-fold (see Speed):

from cutetts import VoicePrompt

voice = model.create_voice_prompt("ref.wav")
voice.save("voice.safetensors")  # later: voice = VoicePrompt.load("voice.safetensors")
results = model.generate_batch(["第一句。", "第二句。"], mode="voice_clone", reference_audio=voice, dialect="客語四縣腔")

Command line 

python -m cutetts.cli --model-dir formospeech/cutetts-hakka-community-1 --mode voice_clone \
    --reference-audio ref.wav --dialect 客語四縣腔 \
    --text "客語語音合成測試。" --output out.wav

Input text

Write the text in Hakka Hanzi with normal punctuation (，。？！). The training transcripts carry their punctuation, so punctuated input reads better than unpunctuated input; see Evaluation. Write Latin code-switch words in lowercase, as the training text does (ai, covid).

Supported Hakka Dialect Labels

  • 客語四縣腔
  • 客語海陸腔
  • 客語大埔腔
  • 客語饒平腔
  • 客語詔安腔
  • 客語南四縣腔

Only Sixian and Hailu are evaluated below. The other dialects have far less training data (Nan-Sixian 68 h, Dapu 20 h, Zhaoan 13 h, Raoping 8 h).

Training Data

953.5 hours, 434,105 utterances, from:

Original datasets:

  • formospeech/hakkaradio_news_clean (train split; the test split is held out for evaluation)
  • formospeech/hat_tts_hailu_clean
  • formospeech/hat_tts_sixian_clean
  • formospeech/hakka_elearning_example_clean

Denoised -R datasets:

  • formospeech/hat_asr_sixian_reading_clean_r
  • formospeech/hat_asr_hailu_reading_clean_r
  • formospeech/hat_asr_nansixian_reading_clean_r
  • formospeech/hat_asr_sixian_broadcast_clean_r

By dialect: Sixian 544 h, Hailu 302 h, Nan-Sixian 68 h, Dapu 20 h, Zhaoan 13 h, Raoping 8 h.

Text Processing

Training uses cleaned hanzi_cln / pinyin_cln text:

  1. Samples containing <UNK> or <spn> are filtered out.
  2. <SIL> / <sil> markers are removed from the cleaned text.
  3. Punctuation is restored from the raw hanzi text.
  4. Latin code-switch words are lowercased.
  5. 50% of samples have 20% of their words replaced by toned Hakka pinyin (IndexTTS-style mixed input).

Training Config

  • Full finetune of the language model, the latent encoder, the diffusion head and the stop predictor. The audio VAE, speaker encoder and tokenizer are unchanged from CuteTTS.
  • Each training row is cut at a random word boundary (from MMS_FA forced alignment): the start of the utterance is the voice prompt, and the rest is the target. The prompt is at most 50% of the utterance.
  • Flow-matching loss with t = sigmoid(N(0, 1)), each patch repeated 4 times with independent noise, plus the stop-predictor loss; CFG condition dropout 0.1.
  • AdamW (betas 0.9/0.95), learning rate 2.5e-5, cosine decay to 0, 500 warmup steps, 30,000 steps.
  • 4 GPUs, each step about 4,000 tokens per GPU, packed without padding.
  • fp32 master weights, with the language model run under bf16 autocast.
  • Released checkpoint: step 27000.

Evaluation

Evaluation uses the same per-dialect test lists as formospeech/omnivoice-hakka-community-1. The lists are byte-identical, built from the test split of formospeech/hakkaradio_news_clean (commit e9e8762), configs Hakka_Sixian and Hakka_Hailu. It also uses the same metrics:

  • CER: formospeech/whisper-large-v2-taiwanese-hakka-v1, served via a vLLM OpenAI-compatible /v1/audio/transcriptions endpoint
  • SIM-o: WavLM-based speaker verification model from k2-fsa/TTS_eval_models
  • UTMOS: UTMOS predictor from k2-fsa/TTS_eval_models

CER is computed on NFKC-normalized, punctuation-stripped, whitespace-stripped text, comparing the ASR hypothesis with the ground-truth text field.

Generation uses the default settings (seed 42, cfg_strength=2.0, 10 uniform steps) with the item's dialect label.

The test lists' text field is hanzi_cln without punctuation, while the training text has its punctuation restored. The Input column gives what the model reads:

  • no punct: text, the same input as the OmniVoice and F5-TTS rows.
  • punct: text_restored, which is text with the source punctuation restored and Latin lowercased, as in training. CER is still scored against text.

Results (per dialect)

Hakka Sixian (客語四縣腔)

Model Input CER avg CER weighted SIM-o UTMOS
Ground truth — 5.24% 4.44% 0.771 3.12
CuteTTS Hakka (this release) no punct 2.08% 1.99% 0.826 3.16
CuteTTS Hakka (this release) punct 2.01% 1.93% 0.828 3.16
CuteTTS Hakka distill (4 steps) no punct 1.90% 1.98% 0.812 3.15
OmniVoice Hakka no punct 2.03% 1.85% 0.795 3.38
F5-TTS g2p no punct 11.08% 11.89% 0.807 2.96
F5-TTS gt_pinyin no punct 10.08% 11.31% 0.808 2.95

Hakka Hailu (客語海陸腔)

Model Input CER avg CER weighted SIM-o UTMOS
Ground truth — 3.03% 2.83% 0.786 3.67
CuteTTS Hakka (this release) no punct 2.31% 2.26% 0.833 3.69
CuteTTS Hakka (this release) punct 2.13% 2.05% 0.835 3.70
CuteTTS Hakka distill (4 steps) no punct 2.36% 2.28% 0.831 3.66
OmniVoice Hakka no punct 2.44% 2.35% 0.807 3.77
F5-TTS g2p no punct 6.42% 6.26% 0.791 3.36
F5-TTS gt_pinyin no punct 5.91% 5.76% 0.792 3.35

Bold marks the best value per column among the no punct rows. The OmniVoice, F5-TTS and ground-truth rows are copied from the OmniVoice Hakka Community 1 model card, measured on the same test lists with the same metrics.

Observations:

  • Speaker similarity is clearly higher than OmniVoice on both dialects (+0.03). CER is comparable, and UTMOS is lower, more so on Sixian.
  • Restoring the input punctuation lowers CER on both dialects. On a held-out validation check, the same sentences read with and without their punctuation gave 2.35% vs 2.69% CER (Sixian) and 1.69% vs 1.87% (Hailu).
  • Sentences with Latin acronyms (about 3% of the test set) have about 4x the CER of the rest, for every input condition. Part of that is the ASR model.

Speed

All speed numbers below were measured on one NVIDIA RTX A5000 (24 GB) with default settings (CuteTTS Hakka: 10 uniform steps with CFG; distill: 4 steps; OmniVoice: 32 steps, fp16; F5-TTS: 32 NFE including g2p). CuteTTS ran with PyTorch 2.13 and transformers 5.15, with the defaults of the fork at commit 0cde9dd (CUDA-graphed VAE decoder and local encoder).

Latency for a single request (batch 1). CuteTTS streams audio while it decodes (generate_stream). OmniVoice and F5-TTS return audio only once the whole utterance is done, so their time to first audio is the full generation time. "Precomputed voice" passes a VoicePrompt built beforehand instead of the reference path (see Usage):

Model Time to first audio with precomputed voice Full utterance
CuteTTS Hakka (this release) 142 ms 49 ms 2.43 s
CuteTTS Hakka distill 132 ms 38 ms 1.35 s
OmniVoice Hakka 2.16 s — 2.16 s
F5-TTS (g2p) 5.36 s — 5.36 s

Means over the same 64 test sentences as the throughput table below (about 7.3 s of audio each), after warm-up.

Throughput as audio seconds generated per wall-clock second (x real time; the batch-1 column is 1/RTF):

Model batch 1 2 4 8 16 32
CuteTTS Hakka (this release) 3.06x 4.14x 7.49x 11.20x 16.67x 20.27x
CuteTTS Hakka (this release), precomputed voice 3.06x 4.48x 7.66x 12.03x 18.77x 23.54x
CuteTTS Hakka distill 5.92x 8.14x 13.72x 20.06x 31.20x 39.67x
CuteTTS Hakka distill, precomputed voice 5.75x 8.62x 14.01x 24.31x 37.98x 55.13x
OmniVoice Hakka 3.43x 5.68x 5.57x 5.00x 4.67x 3.95x
F5-TTS (g2p) 1.45x — — — — —

Measured on the same 64 test sentences, kept in their list order, so each batch mixes short and long sentences, after an untimed warm-up batch. Each batch is decoded together: CuteTTS with generate_batch, OmniVoice with its batched generate. F5-TTS's pipeline does not batch requests with different reference audio. CuteTTS rows are the mean of two passes, run alternately with and without precomputed voices; passes differ by up to about 10% at small batch sizes, so differences of that size there are noise. "Precomputed voice" rows use a VoicePrompt made beforehand (untimed); the other rows, like OmniVoice and F5-TTS, encode the reference inside every request. Precomputing matters most at large batches, where per-request reference encoding is a larger share of the work. At batch 32, peak GPU memory is 2.6 GiB for CuteTTS and 12.8 GiB for OmniVoice.

Time to first audio when a whole batch is streamed with generate_batch_stream, averaged over every request in the batch. With a reference path, every request's reference is encoded before decoding starts, so the wait grows with the batch; with precomputed voices it stays low:

Model Reference batch 1 2 4 8 16 32
CuteTTS Hakka (this release) path 135 ms 207 ms 356 ms 562 ms 1183 ms 2080 ms
CuteTTS Hakka (this release) precomputed 52 ms 54 ms 55 ms 60 ms 79 ms 134 ms
CuteTTS Hakka distill path 132 ms 214 ms 349 ms 587 ms 1168 ms 2132 ms
CuteTTS Hakka distill precomputed 37 ms 39 ms 42 ms 50 ms 65 ms 97 ms

Reproducing this evaluation

The test lists are published under eval/:

ref_audio paths are relative (e.g. ref_audio/Hakka_Sixian/<id>.wav) and the audio is not bundled. Rebuild it with the script in FormoSpeech/OmniVoice-hakka:

python tools/build_omnivoice_hakka_community1_eval_list.py --out-root <your-output-dir>

Notes

  • Only voice cloning was finetuned. The text-only tts mode of the base model was not trained on Hakka, and dialect is rejected outside mode="voice_clone".
  • The reference recording sets the voice, the recording conditions and the speaking style. The evaluation used 3–15 s references by the same speaker, from a different recording.
  • Evaluation numbers come from the project-side Hakka benchmark setup, not from a standardized public benchmark leaderboard.

License

The weights are released under CC BY-NC 4.0, the same license as the OmniVoice Hakka release. The base model, OPPOer/CuteTTS, is licensed under Apache-2.0; its license is included as LICENSE-CuteTTS.

Downloads last month
4
Inference Providers NEW
Text-to-Speech
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for formospeech/cutetts-hakka-community-1

Base model

OPPOer/CuteTTS
Finetuned
(2)
this model
Finetunes
1 model

Space using formospeech/cutetts-hakka-community-1 1

Evaluation results