shiqianyan/TangYuGe-GPT-SoVITS-v2ProPlus

TangYuGe GPT-SoVITS v2ProPlus

Fine-tuned GPT-SoVITS v2ProPlus weights for the voice of Tang Yuge (唐语歌), a character from the visual novel Sparkling Memories (夏日花火). Chinese (Mandarin) only, 32 kHz mono output.

This repository is self-contained: fine-tuned GPT and SoVITS weights, 19 scene profiles with their reference clips and prompt texts, inference / HTTP-bridge scripts, and (optionally) every auxiliary and base model needed to run or continue training. Current release: GPT G-F2+DPO epoch 10 + SoVITS epoch 4 (weights/tangyuge_v2pp_dpo-e10.ckpt, weights/tangyuge_v2pp_e4_s564.pth).

Task	Text-to-speech, few-shot voice cloning (character voice)
Language	Mandarin Chinese only
Output	32 kHz, mono, WAV
Weights	GPT 155 MB + SoVITS 173 MB (fp16)
Verified with	GPT-SoVITS commit ea2d2a8, Python 3.10, PyTorch 2.4.0 + CUDA 12.4, RTX 4060 Laptop (8 GB)
Real-time factor	about 0.4–0.5 on the verified GPU (fp16, after warm-up)
License	Apache-2.0 (see Licensing)

Model Details

Model Description

• Maintainer: shiqianyan
• Model type: two-stage autoregressive TTS. A GPT (text → semantic tokens) and a SoVITS (semantic tokens + reference → waveform), both fully fine-tuned (not LoRA) from the official v2ProPlus base models.
• Language: Mandarin Chinese
• Fine-tuned from: lj1995/GPT-SoVITS (s1v3.ckpt, v2Pro/s2Gv2ProPlus.pth, v2Pro/s2Dv2ProPlus.pth)
• Sources: training code is the official GPT-SoVITS (commit ea2d2a8); only the learning-rate schedule launcher and a switch for the official DPO option were added (see Training Procedure).

Release history

main always holds the current release. Earlier GPT checkpoints are kept as tags.

Tag / ref	GPT checkpoint	What it is
main (tag v2pp-dpo-e10-20261023)	weights/tangyuge_v2pp_dpo-e10.ckpt	Current. Real + synthetic teacher data, cosine-annealed LR, official DPO
v2pp-d2-20261022	weights/tangyuge_v2pp_d2-e6.ckpt	Distillation round: 561 real + 561 synthetic clips, 1:1 (that tag still has the e14 file in weights/; use the d2 one)
v2pp-e14-20261020	weights/tangyuge_v2pp-e14.ckpt	First v2ProPlus release: trained on 561 real clips only

The SoVITS weight (tangyuge_v2pp_e4_s564.pth) is the same in all three.

hf download shiqianyan/TangYuGe-GPT-SoVITS-v2ProPlus --revision v2pp-d2-20261022 --local-dir TangYuGe-d2

The earlier v2Pro (non-Plus) fine-tune lives in shiqianyan/TangYuGe-GPT-SoVITS-v2Pro. The v2ProPlus line has a noticeably lower noise floor and background noise (see manifests/training_report.md) and is the recommended one.

Uses

Direct use

Generating Chinese dialogue in this character's voice for fan works, game mods, character chat bots and personal study or research. Pick a scene profile that matches the tone of the line (see Scene profiles); the reference clip and its prompt text control emotion and rhythm to a large degree.

Out-of-scope use

Non-Chinese text; mixed Chinese–English text (untested); anything that needs the original actor's exact performance; impersonating a real person; misleading or unlawful use. Please label generated audio as AI-synthesised.

Bias, Risks, and Limitations

• Short utterances can end early (e.g. only the first half of a four-syllable line is spoken). This is a known property of GPT-SoVITS autoregressive decoding; the conservative sampling defaults below reduce it but cannot rule it out.
• Electronic / "muffled" artefacts. Some outputs, especially with the serious reference (serious_script41_L0682, the default of main_hybrid_mid), have light electronic distortion or a "mouth full" quality. These artefacts are present in the shipped SoVITS epoch 4 and were not fixed by any later experiment. Automatic metrics (DNSMOS, ASR error rate) do not detect them, so no improvement is claimed.
• Non-speech sounds (laughter, breaths, sighs) have no dedicated tokens in GPT-SoVITS (the phoneme vocabulary is full). They only emerge when the text contains interjections such as 嘻嘻 / 哈哈 / 唉, and quality is below natural laughter. In our test, about three quarters of the expected event types were realised (event recall 0.77, see Evaluation).
• Reference clip and prompt text must be paired. A wrong prompt text noticeably degrades pronunciation and rhythm.
• Voice and timbre are limited by about 43 minutes of source recordings; strong emotion (shouting, crying) is weak.
• On some Intel hybrid-architecture CPUs (reproduced on i7-13650HX) PyTorch's oneDNN multi-threaded Conv1d returns wrong results. GPU inference is unaffected; for CPU, set torch.backends.mkldnn.enabled = False or run single-threaded.
• Part of the GPT training data is synthetic speech from a commercial TTS service imitating the same character. It is not redistributed here, and it can propagate that service's own mispronunciations.

Recommendations

Keep the default sampling parameters, use the scene profile closest to the text, listen to the output before publishing, and do not use it to imitate real people.

How to Get Started

1. Prepare GPT-SoVITS

git clone https://github.com/RVC-Boss/GPT-SoVITS.git
cd GPT-SoVITS && git checkout ea2d2a8      # install its Python dependencies as documented upstream

2. Download this repository

Put it next to GPT-SoVITS/:

cd .. && hf download shiqianyan/TangYuGe-GPT-SoVITS-v2ProPlus --local-dir TangYuGe-GPT-SoVITS-v2ProPlus

If you already run GPT-SoVITS v2ProPlus and have its auxiliary models (BERT, HuBERT, SV, G2PW, fast_langdetect), you only need the weights, profiles, references and scripts; skip the large overlay:

hf download shiqianyan/TangYuGe-GPT-SoVITS-v2ProPlus --exclude "GPT-SoVITS-overlay/*" --local-dir TangYuGe-GPT-SoVITS-v2ProPlus

3. Install the auxiliary models (unless you already have them)

python TangYuGe-GPT-SoVITS-v2ProPlus/scripts/install_pretrained.py --gptsovits-root GPT-SoVITS

The script verifies hashes and copies GPT-SoVITS-overlay/ into the GPT-SoVITS tree (existing identical files are skipped). The overlay contains two groups:

Group	Files	Needed for
Auxiliary models	chinese-roberta-wwm-ext-large (BERT), chinese-hubert-base, sv/ (speaker verification), text/G2PWModel, fast_langdetect	Inference
Official v2ProPlus base models	v2Pro/s2Gv2ProPlus.pth, v2Pro/s2Dv2ProPlus.pth, s1v3.ckpt	Continued training only

The fine-tuned weights are complete inference models. GPT-SoVITS drops the posterior encoder (enc_q) and the discriminator when saving, so continuing training from them needs the base models to restore those parts.

4. Synthesize

Use the GPT-SoVITS Python environment:

# default profile (serious / long lines)
python TangYuGe-GPT-SoVITS-v2ProPlus/scripts/run_tangyuge_infer.py --text "这里填写要合成的中文文本。" --output output.wav

# another scene: switch the profile
python TangYuGe-GPT-SoVITS-v2ProPlus/scripts/run_tangyuge_infer.py \
  --voice-profile TangYuGe-GPT-SoVITS-v2ProPlus/profiles/morning.yaml \
  --text "早上好呀，今天也要元气满满哦。" --output morning.wav

Add --gptsovits-root <path> if the two directories are not siblings, and --seed N for reproducibility.

5. HTTP service (optional)

python TangYuGe-GPT-SoVITS-v2ProPlus/scripts/start_tangyuge_bridge.py

This serves /tts on 127.0.0.1:9881 (the GPT-SoVITS api_v2 backend runs on 9880). The bridge picks a scene from the text, or you can name one:

import requests

r = requests.post(
    "http://127.0.0.1:9881/tts",
    json={"text": "早上好呀，今天也要元气满满哦。", "profile": "morning"},   # "profile" is optional
    timeout=120,
)
open("output.wav", "wb").write(r.content)

Request fields: text (required), profile / scene / reference (choose a scene), top_k, top_p, temperature, and the usual api_v2 fields. Helper endpoints: GET /health, GET /profiles, GET /references.

If you call the plain GPT-SoVITS api_v2 backend yourself, the weights are fixed by the server, but every request must still carry the scene's reference clip and its prompt text (from manifests/references.yaml):

{
  "text": "早上好呀，今天也要元气满满哦。",
  "text_lang": "zh",
  "ref_audio_path": "<repo>/references/morning_script43_L0993.wav",
  "prompt_text": "曦铭？这么早就来学校了呀。",
  "prompt_lang": "zh",
  "top_k": 5, "top_p": 0.8, "temperature": 0.7,
  "text_split_method": "cut5", "batch_size": 1,
  "media_type": "wav", "streaming_mode": false
}

6. GPT-SoVITS WebUI

After installing the auxiliary models, select GPT weights/tangyuge_v2pp_dpo-e10.ckpt, SoVITS weights/tangyuge_v2pp_e4_s564.pth, any clip from references/ together with its prompt_text from manifests/references.yaml, and the sampling parameters below.

Defaults

Parameter	Value
top_k / top_p / temperature	5 / 0.8 / 0.7 (conservative; fewer swallowed syllables than the upstream defaults)
Text splitting	cut5
Precision	fp16 (fp32 showed no measurable difference)
Repetition penalty	upstream default (1.35); 1.0 and 1.15 gave no improvement

Profile format

Each profile binds weights, config, reference clip and prompt text with relative paths, so the repository can be moved anywhere:

# abridged
name: tangyuge_v2pp_final_main
version: v2ProPlus
tts_config: ../configs/tts_infer_pack.yaml
weights:
  gpt: ../weights/tangyuge_v2pp_dpo-e10.ckpt
  sovits: ../weights/tangyuge_v2pp_e4_s564.pth
prompt:
  reference_id: serious_script41_L0682
  ref_audio_path: ../references/serious_script41_L0682.wav
  prompt_text: 气也撒了，如果你还不收敛，给在场辛苦的工作人员增添麻烦，丢人现眼的，也只有你自己了。
  prompt_lang: zh
defaults: {text_lang: zh, text_split_method: cut5, media_type: wav, batch_size: 1, top_k: 5, top_p: 0.8, temperature: 0.7}

Scene profiles

GPT-SoVITS takes its tone and rhythm largely from the reference clip, so the repository ships 19 scene references (each is an original in-game line plus its transcript), packaged as profiles/*.yaml:

Profile	Suited for
main_hybrid_mid (default)	serious explanations, long lines, steady emotion
morning	greetings, short chit-chat
neutral_polite	plain statements, polite replies
soft_daily	gentle, low-pressure daily talk
question_clear / question_firm	caring questions / challenging questions
lively_daily / casual_invite	light short lines / inviting a friend
narration_calm	narration
apology	apologies, polite retreat
bright_fireworks / high_brightness_exclaim / exclaim	bright anticipation / happy short lines / surprised exclamations
intimate_reassure / romantic_soft	intimate reassurance / shy romantic lines
command_gentle	reminders, dissuading
food_daily / cafeteria_ordering	food talk / ordering
low_energy_discussion	restrained explanations, calm discussion

When you change a reference clip, change its prompt text too.

Samples

File	Text
samples/sample_morning.wav	早上好呀，今天也要元气满满哦。
samples/sample_long_statement.wav	这个计划我已经反复确认过了，如果大家都愿意配合，烟火大会一定能顺利举办，到时候我们再一起庆祝吧。

Both were generated with seed 9001 by scripts/run_tangyuge_infer.py from this release.

Repository layout

├── README.md
├── LICENSE                          # Apache-2.0
├── THIRD_PARTY.md                   # sources of the bundled third-party models
├── weights/                         # fine-tuned weights (fp16)
│   ├── tangyuge_v2pp_dpo-e10.ckpt   #   GPT  (text -> semantic tokens)
│   └── tangyuge_v2pp_e4_s564.pth    #   SoVITS (semantic tokens -> waveform)
├── GPT-SoVITS-overlay/              # auxiliary models + base models, same layout as GPT-SoVITS
├── configs/tts_infer_pack.yaml      # GPT-SoVITS inference config
├── profiles/                        # 19 scene profiles
├── references/                      # 19 scene reference clips
├── samples/                         # example outputs
├── scripts/                         # install / inference / HTTP bridge scripts, GPT LR-schedule launcher
├── assets/                          # figures used in this card
└── manifests/
    ├── references.yaml              # reference clip <-> prompt text bindings
    ├── training_report.md           # full training and evaluation report (Chinese)
    └── checksums.sha256             # file hashes

Training Details

Training data

All voice data come from the character's in-game lines (702 clips, about 48.8 min, extracted from the game); the dataset itself is not released. Transcripts were checked with ASR: no obviously wrong labels (mean pinyin error 2.0%, none above 30%).

Stage	Data
SoVITS	561 real clips (1–12 s, at least 6 Chinese characters), about 43.1 min
GPT	1559 clips: 685 real (the 561 above + 124 short / interjection lines that the length and character-count rules had dropped; used for GPT only), 845 synthetic (about 57.5 min; speech from a commercial TTS service imitating the same voice, filtered as below), and 29 duplicated real clips that contain 我们

Synthetic-data filtering (all machine gates): duration 1–12 s, clipping ≤ 1%, pinyin edit distance to the text ≤ 1 (0 for lines of up to 4 syllables) with coverage ≥ 0.9 on a local Paraformer transcript, DNSMOS background score ≥ 3.5, and for laughter lines the event tag must appear in a speech-to-text transcript with event tags. Clips containing 我们 were removed from the synthetic part (their articulation did not match the real voice), and 5 clips the maintainer found unnatural were removed by ear.

Training Procedure

SoVITS (unchanged from the first release): started from s2Gv2ProPlus.pth / s2Dv2ProPlus.pth, batch size 4, learning rate 1e-4, fp16, 8 epochs saved, epoch 4 (step 564) selected.

GPT (this release): started from the official s1v3.ckpt, official training script and optimizer (ScaledAdam, optimizer step every 4 batches), batch size 4, 16-bit mixed precision, 10 epochs, one checkpoint per epoch.

• Learning rate. The upstream code pins the GPT learning rate to a constant 0.002, so each saved epoch is a snapshot taken at a high learning rate and pinyin error rate swings from epoch to epoch (1.5–4.7% in our runs). We replaced only the scheduler, with a launcher that leaves the GPT-SoVITS source untouched (scripts/s1_train_annealed.py, total updates set through the S1_TOTAL_UPDATES environment variable): linear warm-up (1e-5 → 2e-3 over 50 optimizer updates) then cosine decay to 1e-4. With the schedule the epoch-to-epoch swing dropped to 1.25–1.63%.
• DPO. The official "DPO training (experimental)" option (train.if_dpo: true in the s1 config) adds a preference loss (β = 0.2, reference-free) in which, for every batch, a "rejected" semantic sequence is built by repeating a random segment or deleting one, so the model learns to move away from repeated and swallowed syllables.
• Observation. In the DPO run the logged step count was twice that of the non-DPO runs (7769 vs 3889) and the learning rate hit its floor at about epoch 6 instead of epoch 10, because the total update count (970) had been set for the non-DPO run. Epochs 7–10 therefore ran at the minimum learning rate of 1e-4. We did not test a re-scaled schedule.

Variants compared (all from s1v3.ckpt): F / F2 (constant LR, 14 epochs; F2 drops synthetic 我们 clips), G-F / G-F2 (same data, annealed LR, 10 epochs), and G-F2+DPO (G-F2 plus official DPO, 10 epochs) which is released.

Higher training top-3 accuracy means more memorisation; the un-annealed runs keep climbing to 0.79 while the annealed ones end at about 0.60. The DPO run's loss includes a preference term and is not plotted.

We also re-trained SoVITS (real + synthetic data, 6 epochs, from the official base models) but did not adopt it: see Evaluation. Its logged losses are single-batch values every 100 steps and too noisy to read; only the KL term trends down.

Compute

One RTX 4060 Laptop (8 GB). The DPO GPT run (10 epochs) took about 24 minutes and used about 6.4 GB of VRAM at batch size 4; SoVITS epochs take about 2–5.5 minutes each depending on data size.

Evaluation

Protocol

Candidates are compared with a baseline on the same sentence, same reference clip and same random seed, and every figure is a paired difference with a 95% interval from a bootstrap over sentences.

• Test sentences: 100 sentences that are not in the training data (checked against the script); 12 extra lines (6 laughter, 6 interjection); and, for the final check, 59 further unseen sentences (checked against the training script and all earlier test sentences, with extra laughter, sigh and emotion lines).
• References: two real-voice scene references (serious_script41_L0682, question_clear_script34_L0420); 1–3 seeds per sentence depending on the test set.
• Metrics
  • Pinyin error rate (PER): toneless pinyin edit distance between the target text and a local Paraformer-large transcript; laughter lines excluded. Severe-error rate: share of utterances with PER > 10%.
  • Speaker similarity: cosine similarity of ERes2NetV2 embeddings against the real recordings.
  • DNSMOS (P.835): SIG (speech distortion), BAK (background noise); noise floor: 10th-percentile RMS of non-silent frames (dB).
  • Event recall: for sentences where a same-text reference rendition contains non-speech events (laughter, breath, sigh), the share of those event types that a tag-emitting speech-to-text service finds in our output (13–16 such sentences, one reference).
  • Prosody closeness: correlation of log-F0 and energy contours plus speaking-rate ratio against that reference rendition (weak at separating systems; versions differ by about 0.01).
• Composite score and gates (used to select checkpoints): speaker similarity 35%, emotion / non-speech events 25%, articulation 25%, cleanliness 15%. Each paired difference is mapped linearly to 0–100 (50 = same as baseline). Three gates are applied first: speaker similarity not lower by more than 0.01, PER not higher by more than 0.3 pp, severe errors not higher by more than 1.5 pp.

Results

Final comparison, 59 unseen sentences × 2 references × 3 seeds (354 utterances per system):

Metric	e14 (first release)	d2 (previous release)	This release
Pinyin error rate (laughter lines excluded)	4.42%	2.41%	2.53%
Severe-error rate (PER > 10%)	13.0%	10.3%	11.0%
Event recall (laugh / breath / sigh)	0.67	0.58	0.77
Speaker similarity	0.884	0.886	0.885
DNSMOS SIG / BAK	3.13 / 3.66	3.10 / 3.61	3.12 / 3.58
Noise floor (dB)	−57.2	−56.7	−56.6

Composite score with all test data pooled: 57.4 vs d2 (95% interval 48.2–67.3; 119 sentences, 474 pairs) and 66.8 vs e14 (51.4–73.6; 159 sentences, 554 pairs); all three gates pass.

How to read this. Articulation, timbre and cleanliness are on par with d2; the clear difference is that laughter / breath / sigh cues are realised about 0.2 more often. Intervals overlap for every metric, so this is "not worse and better at non-speech events", not a statistically significant across-the-board gain. Absolute PER depends heavily on the sentence set (short interjection lines are hard) and cannot be compared with earlier reports on other sets.

Per-epoch behaviour (100 sentences × 2 references, one seed, versus e14):

• G-F2+DPO: all 10 epochs pass the gates, articulation better than e14 at every epoch, event recall stays between 0.62 and 0.84. Epochs 2–10 form a plateau (composite 59–68). Epochs 6 and 10 were re-tested on unseen sentences and new seeds, and epoch 10 scored higher on the pooled data, so it is the released one. Epoch 2 scored highest on the first test set but fell back on unseen sentences.
• G-F2 (no DPO): event recall drops to 0.47–0.53 from epoch 6 on, and on unseen sentences it failed the severe-error gate. G-F reached 0.53–0.72 event recall and was not re-tested on unseen sentences.
• F / F2 (constant LR): large epoch-to-epoch swings.

SoVITS re-training did not beat the shipped epoch 4: pooled over two seeds (424 pairs) the best candidates scored 54.1 (95% interval 44.7–62.7) and 51.9 (44.6–61.6) against a baseline of 50, and paired with the new GPT the results went both ways (52.7 and 44.5). SoVITS was therefore left unchanged.

Limits of this evaluation

Two real references only; event recall on a small sentence set; the prosody metric barely separates systems; composite weights are a judgement call; automatic metrics cannot see the electronic artefacts described above. The released version was additionally accepted by listening to 30 random draws (10 stratified-random unseen sentences × 3 seeds) by the maintainer.

Technical Specifications

• GPT: autoregressive transformer decoder over semantic tokens, 24 layers, hidden size 512, 16 heads, feed-forward 2048, semantic vocabulary 1025 (1024 + EOS), phoneme vocabulary 732, BERT features from chinese-roberta-wwm-ext-large, training clips up to 54 s.
• SoVITS (v2ProPlus): VITS-style generator that decodes semantic tokens conditioned on a reference clip and a speaker-verification embedding (ERes2NetV2 sv/ model); 32 kHz output.
• Software: GPT-SoVITS ea2d2a8, Python 3.10, PyTorch 2.4.0 (CUDA 12.4). Hardware: NVIDIA GPU recommended; see the CPU note above.

Licensing and Rights

This repository is released under Apache-2.0; THIRD_PARTY.md lists the bundled third-party models and their notices (for example the fastText language-ID model is CC BY-SA 3.0 and keeps its own terms). The character, the game and the original recordings belong to their respective rights holders; the weights here are fine-tuned parameters only. You are responsible for complying with the rights holders' terms when you publish or monetise generated audio.

Citation

@misc{tangyuge_gptsovits_v2proplus,
  author       = {shiqianyan},
  title        = {TangYuGe GPT-SoVITS v2ProPlus},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/shiqianyan/TangYuGe-GPT-SoVITS-v2ProPlus}}
}
@software{gpt_sovits,
  author = {RVC-Boss and contributors},
  title  = {GPT-SoVITS},
  url    = {https://github.com/RVC-Boss/GPT-SoVITS}
}

Thanks to GPT-SoVITS, Chinese-BERT-wwm, chinese_speech_pretrain, 3D-Speaker, g2pW and fastText.

Model Card Contact

Open a Discussion on this repository.

---

中文摘要

本仓库是基于 GPT-SoVITS v2ProPlus 微调的唐语歌（《夏日花火》角色）中文语音合成模型，仓库自带推理所需辅助模型和继续训练用的官方底模，下载后即可使用。当前版本：GPT 为 weights/tangyuge_v2pp_dpo-e10.ckpt（真人 + TTS 合成教师集，学习率余弦退火，官方 DPO，第 10 轮），SoVITS 沿用第 4 轮。

• 用法：按上文 How to Get Started 安装；每次合成必须使用场景 reference 及其配对的 prompt_text（见 manifests/references.yaml），换 reference 时同时换文本；默认采样 top_k=5 / top_p=0.8 / temperature=0.7。已有 GPT-SoVITS 环境的话，用 --exclude "GPT-SoVITS-overlay/*" 只下载权重与脚本即可。
• 相对上一版（d2）：发音、音色、干净度持平；笑声 / 吸气 / 叹气的事件召回 0.58 → 0.77；综合分 57.4（95% 区间 48.2–67.3）。各区间互相重叠，不要理解为“全面显著更好”。
• 已知问题：短句偶发提前结束；严肃类 reference 下仍可能有轻微电音 / “含东西”的口腔感（机器指标测不出，本版没有声称改善）；笑声没有专用 token，只能靠 嘻嘻 / 哈哈 / 唉 等文本诱发；CPU 推理需关闭 oneDNN。
• 完整中文训练与评测报告：manifests/training_report.md。旧版权重按标签取回：v2pp-d2-20261022、v2pp-e14-20261020。