MOSS voice-profile LoRAs — 500 voices, rank 4
500 low-rank adapters for laion/moss-tts-local-transformer-4.55b-voice-acting-v2, one per synthetic voice profile. Each adapter makes the frozen base model speak as one specific, invented character voice — a "grave documentary narrator", an "anxious teenage hacker" — and ships with the reference clip and the profile card that define that character. Every one of them is rank 4, alpha 8, dropout 0.05 over the same 23 target modules, weighs about 34 MB, and beats the frozen base model on that voice's own held-out clips. None of them is a clone of a real person: every voice profile is synthetic, invented from a design spec and then generated.
If you have never seen this project before, read §1 to §3 and then run quickstart.py.
- 500 voices · 17.36 GB · ~34.4 MB per adapter · 2,506 files
- 8,589,312 trainable parameters per adapter — 96.1 % of them in the semantic backbone, only ~3.9 % in the acoustic path (§4)
- Median gain over base: 0.1571 nats of held-out loss (min 0.0751, max 0.3150). All 500 beat the base; none was shipped that did not.
- Base model
laion/moss-tts-local-transformer-4.55b-voice-acting-v2· codecOpenMOSS-Team/MOSS-Audio-Tokenizer-v2· reference audiovoices/<voice>/reference.wav, in this repo (why not the dataset that sounds like it)
Read this first
Not buried at the bottom, because these change how you should use the release. Full detail in §12.
- No human has listened to any of this in a controlled study. Every quality number here is held-out language-modelling loss on MOSS audio codes, plus ECAPA speaker cosine. Those are proxies for "sounds like the voice", not measurements of it.
- You need the reference clip as well as the weights. The base model is reference-conditioned. An adapter with no reference gives you a random speaker wearing the adapter's colouring — there is no useful "no reference" mode. The right clip ships at
voices/<voice>/reference.wav. - 8 of the 500 adapters are below the measured saturation point and 6 of those are drawn from voices the generator never cloned reliably. They are flagged and named (§8), not hidden.
- Do not read a voice's gender out of
caption_general. The corpus renders the gender and background-noise axes of that prose field with inverted polarity, and about 35 % of training instructions carried the buggy prose verbatim. Usecard_genderfromprofile.json/ the manifest (§12).
Relationship to TTS-AGI/moss-voice-profile-loras
There is an earlier, smaller LoRA repo, and it is not superseded silently, so here is what it actually is.
TTS-AGI/moss-voice-profile-loras (public, 168 files) is the 10-voice pilot and rank-ablation study that this release was designed from. It holds ten voices, each with a shipped adapter and six ablation arms (ranks/{r4,r8,r16}/{stage1,stage2}) plus ablation/ result tables. That sweep is where rank 4 was chosen; it is why every adapter here is rank 4 and not rank 16.
All ten of its voices — anime_088, emolia_c0542, emolia_c1682, emolia_c1699, emolia_c2570, k10_age3_bg1, k325_age3_bg1, k395_age3_bg1, k91_age5_bg0, mediathek_0184 — are present in this release, and the pilot repo's own README says so, describing this build as the one whose voices "will replace these same ten".
TTS-AGI/moss-voice-profile-loras |
this repo | |
|---|---|---|
| purpose | rank/stage ablation, pilot | production release |
| voices | 10 | 500 |
| ranks | 4, 8, 16 (60 ablation arms) | 4 only |
| ships reference clip | no | yes |
| ships profile card | no | yes |
| manifest | ablation/*.csv |
manifest.parquet, 56 columns |
Keep the pilot repo if you want the rank ablation or the two-stage arms. For actually generating a voice, use this one.
1. What these are
A voice profile here is a stable, invented speaker identity — not a real person and not a recording of one. Each profile started as a written design spec (name, tagline, gender, age, accent, register, timbre sliders, casting notes) and a single reference clip. That profile was then used to generate a large corpus of takes across emotion, delivery, character and edge-case prompts — roughly 38,000 candidate takes per voice. The adapter shipped here is fitted on the subset of those takes that most convincingly sound like the profile, as scored by an ECAPA speaker embedder against the reference.
The result is a 34 MB file that, attached to the 4.55 B base model, holds that one identity across the whole expressive range the corpus covers: emotions, deliveries, vocal bursts, English and German. Nothing in this release is trained on a real speaker's voice, and there is no real identity to recover from these weights.
These are stage-1 identity adapters (stage = "stage1" on all 500 rows). They are complete and usable as they are.
2. What is in the repo
README.md this file
MANIFEST_COLUMNS.md the column dictionary, also inlined at §10
manifest.parquet one row per voice, 500 rows x 56 columns
manifest.json the same rows as JSON
RELEASE.json build summary + the downstream consumer contract
quickstart.py runnable end-to-end example
voices/<voice>/
adapter_model.safetensors the LoRA weights, peft format (~34.4 MB)
adapter_config.json the peft config
reference.wav the voice's reference clip -- REQUIRED at generation time
profile.json the voice's profile card (name, tagline, casting, tags)
voice.json this voice's manifest row, standalone
500 voice directories × 5 files + 6 top-level files = 2,506 files, 17,359,449,359 bytes.
Choosing a voice. Use the profile-card columns in the manifest — voice_name, tagline, card_gender, card_age, card_accent, card_register, card_tags. These come from the design spec each voice was generated from. Do not use the corpus's caption_general prose for this (§12).
import pandas as pd
m = pd.read_parquet("manifest.parquet")
m[m["card_tags"].str.contains("narrator", na=False) & (m["card_gender"] == "Female")] \
[["voice", "voice_name", "tagline", "gain"]].sort_values("gain", ascending=False).head()
pandas gotcha:
rankis also a DataFrame method. Writem["rank"], neverm.rank. The same applies to any column whose name collides with a method.
If you are consuming this release from code, read manifest.parquet and treat voice as the key. Every row already points at that voice's final adapter — the choice between a retrain and its incumbent has already been made, so there is nothing to resolve:
m = pd.read_parquet("manifest.parquet").set_index("voice")
rel = m.loc["emolia_c0019", "adapter_dir"] # "voices/emolia_c0019" -- relative, use this
was_new = bool(m.loc["emolia_c0019", "retrained"]) # did this release retrain it?
why = m.loc["emolia_c0019", "retrain_status"] # shipped_retrain / rejected_worse / ...
The *_abs path columns and release_root record absolute paths on the machine that built the release and are provenance only; on the Hub, use adapter_dir. Never load from adapter_src — it points into build trees that are not part of this release.
3. How to use one
quickstart.py in this repo is the code below, runnable:
python quickstart.py --release . --voice emolia_c0019 --out hello.wav
Budget roughly 20 GB of VRAM for the base model plus the codec.
Three things actually trip people up:
(a) You need the reference clip as well as the adapter. Covered above; quickstart.py defaults to the shipped one, so in practice you do not have to find it.
(b) PeftModel.active_adapter is a plain instance attribute, not a property. It is assigned once in PeftModel.__init__ and is not kept in sync by everything that changes adapters — on the class itself it is simply None. active_adapters (plural) IS a property and does query the model. Read the plural one, or read peft_config directly, and never assume the singular attribute is current after juggling adapters.
(c) peft 0.20.0's offline path mangles subfolder. With HF_HUB_OFFLINE=1, passing subfolder= to PeftModel.from_pretrained puts the subfolder into the filename and passes it again as a kwarg, so it looks for <voice>/<voice>/adapter_model.safetensors and raises LocalEntryNotFoundError. Resolve to a plain local directory first, as below, and it works either way.
import numpy as np, soundfile as sf, torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoModel, AutoProcessor
BASE = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2"
CODEC = "OpenMOSS-Team/MOSS-Audio-Tokenizer-v2"
REPO = "laion/moss-voice-profile-loras-500"
VOICE = "emolia_c0019" # any directory name under voices/
# Pull just the one voice (~34 MB + a reference clip), not all 17 GB.
root = snapshot_download(REPO, allow_patterns=[f"voices/{VOICE}/*", "manifest.parquet"])
adir = f"{root}/voices/{VOICE}"
proc = AutoProcessor.from_pretrained(BASE, trust_remote_code=True, codec_path=CODEC)
proc.audio_tokenizer = proc.audio_tokenizer.to("cuda").eval()
# "some weights of MossTTSLocalModel were not initialised" warnings about audio_lm_heads /
# text_lm_head are benign -- those tensors are weight-tied to the embeddings, not missing.
model = AutoModel.from_pretrained(BASE, trust_remote_code=True,
dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()
model = PeftModel.from_pretrained(model, adir).eval() # a plain local directory
name = model.active_adapters[0] # PLURAL: the property (see (b))
print("rank:", model.peft_config[name].r, "alpha:", model.peft_config[name].lora_alpha)
conv = [[proc.build_user_message(
text="I have read the file, and that is exactly what worries me.",
instruction="A warm, unhurried voice, speaking just above a murmur.",
language="English", # "English" or "German"
reference=[f"{adir}/reference.wav"], # ships with the adapter -- do not omit
tokens=12)]] # a length HINT in words, not a limit
batch = proc(conv, mode="generation")
out = model.generate(input_ids=batch["input_ids"].cuda(),
attention_mask=batch["attention_mask"].cuda(),
max_new_frames=400, # the real ceiling: 400 frames ~ 32 s at 12.5 fps
do_sample=True,
text_temperature=0.7, text_top_k=50, text_top_p=1.0,
audio_temperature=1.0, audio_top_p=0.95, audio_top_k=30,
audio_repetition_penalty=1.1)
msg = proc.decode(out)[0]
if not msg.audio_codes_list: # a NORMAL, silent failure mode of this model -- always check
raise SystemExit("empty decode; retry with another seed")
w = msg.audio_codes_list[0].cpu().float().numpy()
sf.write("out.wav", np.ascontiguousarray(w.mean(0) if w.ndim > 1 else w), 48000)
To merge an adapter permanently: model = model.merge_and_unload(). These adapters were evaluated at scale 1.0 only; no other merge scale was measured, and nothing here was measured with two voice adapters attached at once.
4. What the adapter actually touches
Read straight out of the shipped adapter_model.safetensors, not out of the config. 536 tensors, 268 adapted modules, 8,589,312 trainable parameters, identical in shape across all 500 voices.
| config field | value |
|---|---|
r |
4 |
lora_alpha |
8 (always 2 × rank) |
lora_dropout |
0.05 |
bias |
none |
init_lora_weights |
True |
use_rslora / use_dora |
False / False |
modules_to_save |
None |
target_modules |
23 patterns |
base_model_name_or_path |
laion/moss-tts-local-transformer-4.55b-voice-acting-v2 |
Where those 8.59 M parameters go:
| block | modules adapted | params | share |
|---|---|---|---|
transformer.layers.0–35 — the semantic backbone |
36 layers × q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj = 252 |
8,257,536 | 96.14 % |
audio_lm_heads.0–11 — one head per MOSS codebook, each (4, 2560) → (1024, 4) |
12 | 172,032 | 2.00 % |
local_transformer.h.0 — the talker head, c_attn, c_proj, fc_in, fc_out |
4 | 159,744 | 1.86 % |
| total | 268 | 8,589,312 | 100 % |
This is the single most useful fact for deciding how to use these adapters. About 96 % of the capacity sits in the semantic/prosodic transformer and only about 3.9 % in the acoustic path — and of the local transformer's stack, only layer 0 is adapted at all. These adapters overwhelmingly change how the model decides what the voice does — phrasing, timing, prosodic shape, register — and only marginally retune the codebook decoding on the way out. If you expected a timbre patch, this is not one; the timbre still comes substantially from the reference clip, which is why §3(a) matters as much as it does.
All 500 adapters share this exact structure: rank = 4, lora_alpha = 8 and n_target_modules = 23 on every manifest row, and all 500 adapter_sha256 values are distinct.
5. How they were trained
Per voice:
- Take the voice's ~38,000 generated candidate takes and drop the empty ones, anything under 0.5 s, and every clip belonging to a held-out group (40 groups per voice, stratified by block and language, fixed seed). Held-out audio never enters training.
- Select the training pool by ECAPA speaker similarity against the voice's reference (§6, §7).
- Encode the pool to MOSS audio codes — 12 codebooks × 1024, 12.5 frames/s exactly (hop 3,840 at 48 kHz).
- Train rank 4 / alpha 8 / dropout 0.05 on the 23 target modules for 6 epochs, checkpointing and evaluating on the held-out clips after every epoch.
- Ship the epoch with the lowest held-out loss. If no epoch beat the frozen base model, ship nothing. That never happened: all 500 beat their base.
Captions are resampled every epoch from the stored measurements, so a clip is not welded to one phrasing of its description.
Why best-epoch selection rather than a fixed epoch count. Across the 499 runs of the original build the winning epoch was ep2 ×90, ep3 ×297, ep4 ×99, ep5 ×13 — there is no constant that is right for all voices. Every one of those runs overfits: the best epoch was never the last (499/499). But the rise after the minimum is strictly monotone in only 472/499 (94.6 %), so "it gets worse after the best epoch" is reliable while "it gets worse at every subsequent epoch" is not — an early-stop on the first uptick would misfire on about 5 % of voices. Picking the minimum over all six is what makes the 100 % beat-base rate real rather than lucky.
Across the shipped 500 adapters (which for 119 voices means the retrain's epoch, not the original run's) the chosen epoch is ep2 ×54, ep3 ×331, ep4 ×101, ep5 ×13, plus stage2 ×1 for emolia_c1699.
Cost. Summing gpu_h over the shipped adapters gives 378.7 GPU-hours (repair build 329.3, 3,000-clip retrain 47.3, floor-capped retrain 0.8, reference study 1.2). That column records only the stage that produced the shipped weights; counting the discarded challengers too, the build reports roughly 355 GPU-h for the original 499 plus 51 GPU-h for the retrain pass, ≈406 GPU-h in total. Encoding time is not included in either figure.
Provenance of the 500. 499 come from the identity-repair pipeline (src_tree = repair, lora3k or lora3k_fc); the 500th, emolia_c1699, is the original single-voice reference study and used a different, two-stage recipe (§8).
6. The two thresholds, and what they do and do not mean
Two numbers govern which clips a voice was trained on. They are both cosine similarities between ECAPA speaker embeddings, on a scale internal to this project, and they are easy to confuse.
| threshold | name | what it does |
|---|---|---|
| 0.40 | identity floor | Below this, a take is treated as a failed clone — the generator produced somebody else. |
| 0.60 | MIN_SPK_TRAIN |
The training-pool threshold of the original recipe: "train on the takes above 0.60". A purity-seeking cut, not a correctness cut. |
The original recipe took every take at spk_sim ≥ 0.60, capped at the 6,000 highest. If fewer than 800 cleared, it relaxed to that voice's own top 800 and flagged the voice relaxed; below 200 it refused to train at all. The retrain recipe replaced this with the voice's top 3,000 takes with a hard 0.40 floor (§7).
These are conventions of this project, not established perceptual facts. The project's own report is explicit: "The 0.40 floor is this project's ECAPA scale, not an established perceptual fact: an independent WavLM-tbr embedder puts 75.4 % of the selected failures above its own same-speaker threshold (0.472)." Two embedders disagree about three quarters of the rejected population.
Three different speaker embedders appear around this project and none of their scales are interchangeable. Worth spelling out, because a threshold copied across them is silently wrong:
| embedder | dim | used for | a threshold on that scale |
|---|---|---|---|
ECAPA (speechbrain/spkrec-ecapa-voxceleb) |
— | every spk_sim in this release |
0.40 floor, 0.60 training cut |
Orange/Speaker-wavLM-tbr |
128 | the independent cross-check quoted above | its own same-speaker point, 0.472 |
Orange/Speaker-wavLM-id |
250 | a different, ear-calibrated task elsewhere | 0.80 |
The 0.80 in particular belongs to -id and must not be applied to -tbr or to ECAPA. A cosine is only meaningful in the embedding it was computed in.
min_spk_effective in the manifest is the number that actually matters per voice: the lowest similarity that really entered that voice's pool, whatever rule produced it. Across the release: 378 voices realise ≥ 0.60, 119 land in [0.40, 0.60), **2 fall below 0.40** (emolia_c1340 at 0.388, emolia_c2142 at 0.387 — both kept pre-floor incumbents, see §8), and 1 is null (emolia_c1699, whose pool was not similarity-filtered).
One more caveat from the same report, about what hard identity filtering costs: "Identity is bought, not free: the reference run paid −11 % target strength and −14 % blend for its identity gain." Selecting for clips that sound like the profile is not the same as selecting for clips that are expressive or well articulated.
7. Why 3,000 clips
Over the 499 adapters of the original build, gain over base correlates with pool size at r = +0.532 and with pool purity (mean speaker similarity) at only r = +0.234. And it saturates. Measured on the pre-retrain state of all 500 voices:
| pool clips | n voices | median gain |
|---|---|---|
| 0–999 | 60 | 0.1063 |
| 1,000–1,999 | 35 | 0.1288 |
| 2,000–2,999 | 28 | 0.1541 |
| 3,000–4,999 | 54 | 0.1630 |
| 5,000–6,000 | 322 | 0.1626 |
| >6,000 | 1 | 0.2253 |
3,000–4,999 and 5,000–6,000 are indistinguishable. Going past ~3,000 clips buys nothing; staying under it costs a lot. The 123 voices trained on fewer than 3,000 clips had a median gain of 0.1221 against 0.1627 for the rest — about 25 % less improvement.
So those 123 voices, and only those, were retrained on the voice's top 3,000 takes by speaker similarity with a hard 0.40 floor — not on a flat 0.60 cut, because 124 of the 500 voices cannot reach even 800 clips at 0.60, which is exactly why they were under-trained in the first place. The other 377 were left alone: they are at or above saturation, and retraining them would have burned GPU-hours to move a number that has stopped moving.
Never ship a worse adapter. Every challenger was written to a separate tree and evaluated on the same held-out clips as the incumbent — the validation container was copied verbatim, not rebuilt, which is the only way two losses are comparable. The lower loss wins.
| outcome | voices |
|---|---|
| 3,000-clip retrain shipped | 115 |
| originals-only floor-capped variant shipped | 4 |
| all challengers rejected (incumbent shipped) | 4 |
| retrain did not finish (incumbent shipped) | 0 |
| not a target (already saturated, untouched) | 377 |
119 voices shipped a new adapter, 4 kept their incumbent, and 0 shipped worse — that last figure guaranteed by construction, since a losing challenger is discarded. Median gain across those 123: 0.1221 → 0.1427, a median per-voice improvement of +0.0219 (median +0.0227 among the winners). Voices below saturation: 123 → 8.
Pool composition beats pool size
Of the 123 retrain targets, 117 could fill 3,000 clips from their own original takes above the floor; 6 could not and had their pools topped up with audio regenerated by the identity-repair pipeline.
| retrain pool | improved on incumbent |
|---|---|
| all-original (117 voices) | 115 / 117 |
| contains regenerated audio (6 voices) | 0 / 6 |
Every pool containing regenerated audio regressed, without exception. All-original pools overwhelmingly improved, but not universally: emolia_c2305 and k492_age2_bg1 had clean 3,000-clip retrains that still lost to their incumbents, and shipped the incumbent. No shipped adapter in this release contains a single regenerated clip — pool_n_repaired = 0 and used_repaired = false on all 500 rows.
Gain by pool size across the shipped release now looks like this:
| pool clips | n voices | median gain |
|---|---|---|
| 0–999 | 2 | 0.0820 |
| 1,000–1,999 | 4 | 0.1050 |
| 2,000–2,999 | 2 | 0.1554 |
| 3,000–4,999 | 169 | 0.1492 |
| 5,000–6,000 | 322 | 0.1626 |
| >6,000 | 1 | 0.2253 |
8. Which voices are weak, and why
Honesty matters more here than a tidy release. 8 of the 500 adapters are still below the 3,000-clip saturation point and should be expected to clone their profile less reliably than the rest. They are flagged below_saturation = true in the manifest and named here:
| voice | pool clips | gain | status | sub-floor |
|---|---|---|---|---|
emolia_c1340 |
800 | 0.0751 | rejected_worse |
yes |
emolia_c2142 |
800 | 0.0888 | rejected_worse |
yes |
emolia_c1070 |
1,096 | 0.0931 | shipped_floorcap |
yes |
emolia_c0241 |
1,532 | 0.1049 | shipped_floorcap |
yes |
emolia_c0697 |
1,861 | 0.1052 | shipped_floorcap |
yes |
emolia_c0243 |
1,909 | 0.1220 | shipped_floorcap |
yes |
k492_age2_bg1 |
2,881 | 0.1543 | rejected_worse |
no |
emolia_c2305 |
2,933 | 0.1566 | rejected_worse |
no |
The six sub-floor voices
The most informative result in the release. Six voices could not supply 3,000 original takes above the 0.40 identity floor — their generator failed to sound like the profile most of the time. Their incumbent adapters had been trained on 800 clips selected by dropping the threshold until 800 existed, which for emolia_c1340 and emolia_c2142 meant reaching down to similarity 0.388 and 0.387, i.e. below the identity floor, including clips that are arguably a different speaker. Two repairs were tried, and they came out opposite ways.
Attempt 1 — more clips, from regenerated audio. Failed, six times out of six. Topping the pool up with identity-repair regenerations reached a full 3,000 clips at a far higher effective similarity threshold for every one of them. Every one came out worse than the 800-clip incumbent, losing roughly a third to a half of its gain. The regenerated audio is measurably degraded on content and delivery — for emolia_c1070, word error rate +96 %, naturalness −23 %, burst blend −38 % — and the validation clips are original audio. An adapter fitted on regenerated audio predicts regenerated audio.
Attempt 2 — fewer clips, but all original and all above the floor. Won, up to a point.
| voice | incumbent: clips / gain | 3k regen: clips / gain | floor-capped: clips / gain | shipped | final gain |
|---|---|---|---|---|---|
emolia_c0241 |
800 / 0.0915 | 3,000 (2,975 regen) / 0.0448 | 1,532 / 0.1049 | floorcap | 0.1049 |
emolia_c0243 |
800 / 0.1040 | 3,000 (2,929 regen) / 0.0672 | 1,909 / 0.1220 | floorcap | 0.1220 |
emolia_c0697 |
800 / 0.0894 | 3,000 (2,957 regen) / 0.0540 | 1,861 / 0.1052 | floorcap | 0.1052 |
emolia_c1070 |
800 / 0.0849 | 3,000 (2,865 regen) / 0.0512 | 1,096 / 0.0931 | floorcap | 0.0931 |
emolia_c1340 |
800 / 0.0751 | 3,000 (2,944 regen) / 0.0322 | 546 / 0.0681 | incumbent | 0.0751 |
emolia_c2142 |
800 / 0.0888 | 3,000 (2,861 regen) / 0.0559 | 632 / 0.0834 | incumbent | 0.0888 |
What this shows. The saturation curve in §7 is not about clip count as such — it is about the count of clips drawn from the distribution you are evaluated on. Three thousand regenerated clips lose to 800 originals. But cleanliness stops paying once the pool gets small enough: emolia_c1340 (546 clean clips) and emolia_c2142 (632) both lost to their 800-clip incumbents even though those incumbents contain sub-floor audio, while emolia_c1070 won on 1,096. The crossover therefore sits between 632 and 1,096 clips: above it a smaller clean pool wins, below it there is simply not enough data and the larger contaminated pool is the better bet. That is considerably more useful than "3,000 is the magic number", and it is the opposite of what a naive reading of the saturation table predicts.
The two voices whose pools reach below 0.40 are also the two with the lowest gains in the entire release — at least consistent with treating the floor as a training constraint and not only a reporting one.
All six remain the weakest voices here and stay flagged sub_floor_voice = true. Improved or not, each is fitted on a pool far under saturation, drawn from a voice the generator never cloned reliably. Expect less consistent identity from all six, and prefer other voices where you have the choice.
emolia_c1699 is not like the others
emolia_c1699 is the original single-voice reference study the whole project was built from. Its adapter comes from a different recipe — a two-stage curriculum over 18,823 unfiltered clips (plus a 1,580-clip stage-2 sharpening pass), selected by a rank sweep (4 / 8 / 16) and a stage sweep rather than the single-stage similarity-filtered recipe used for the other 499. It is rank 4 like the rest and its held-out loss is measured on the same kind of split, but it is not a comparable data point about the recipe — it has the highest gain in the release (0.2253) partly because it is the only voice trained this way. recipe = reference_study_2stage marks it, and pool_mean_spk / min_spk_effective are null for it because its pool was never similarity-filtered.
9. profile.json, reference.wav, and which gender field to trust
reference.wav is the voice's conditioning clip — the same clip the profile was defined by and the corpus was generated from, at the enhancement variant the profile card rates best by DNSMOS (reference_variant in the manifest says which of orig / sidon / cbx; across the release: 173 / 156 / 171). The base model is reference-conditioned, so this file is not optional. These 500 files are the only published copy of the profiles' reference clips — see the note below.
profile.json is the voice's design spec — the document the voice was generated from, not a measurement of the audio that came out:
{
"cid": "emolia_c0019",
"name": "Grave Documentary Narrator",
"tagline": "A weightful and measured voice that conveys historical gravity and calm authority.",
"gender": "Male",
"age": "Late 40s to 50s",
"language": "English",
"accent": "Standard American",
"register": "Mid-range baritone with a calm, downward inflection",
"timbre_profile": {"metallic": 1, "throat_guttural": 2, "falsetto": 0,
"chest_voice": 4, "roughness": 2, "brightness": 3},
"distinctive_features": "Precise enunciation combined with a somber, grounded resonance…",
"emotional_range": "Primarily serious and informative with a steady, cautionary undertone.",
"casting": {"classic_fantasy": {"role": "Royal Archivist", "direction": "…"},
"sci_fi": {…}, "mystery_horror": {…}, "contemporary": {…}},
"tags": ["english", "standard american", "male", "middle-aged", "narrator", …]
}
The casting blocks are genre-specific role + direction pairs and make good ready-made instruction= strings.
card_genderis the field to trust. It is the profile card'sgender, i.e. the intended design, and it is not derived from the corpus's buggycaption_generalprose (§12). Anyone wanting a voice's gender should readcard_gender(orprofile.json'sgender) and nothing else. Note it states intent, not a measurement of the produced audio. Across the release: Male 242, Female 236, Androgynous 21, null 1.
One voice, k532_age3_bg1, has a stub profile card with no descriptive fields, so voice_name, tagline and the card_* columns are null there. Its adapter and reference clip are unaffected. card_language is additionally null for 115 voices whose cards did not record one.
9b. The speaker name — name, and a third gender field
Every voice now also carries a given name: one unique, ordinary first name per profile, added
so that a model can be conditioned on "speak as Katrin" instead of on a reference clip. It is in
manifest.parquet / manifest.json (name, name_gender_class, name_confidence), in every
voices/<voice>/voice.json, and standalone in the new top-level names.csv / names.json.
profile.json is deliberately left byte-untouched.
"name": "Katrin", "name_gender_class": "female", "name_confidence": 0.86
Names are decorative labels for conditioning. They are not the character names — those stay
where they were, in voice_name (manifest, voice.json) and in name (profile.json, unchanged).
name_gender_class is measured, and it is not the same thing as card_gender.
| field | what it describes | who produced it |
|---|---|---|
card_gender |
the reference clip | a VLM writing the character card during voice selection |
name_gender_class |
the 40,256 takes this profile actually generated | VoiceNet vn_GEND_reg, thresholded on the corpus's own bimodal distribution |
A voice is male when at least 85 % of its takes score vn_GEND_reg >= 3.2103 (the 50/50
boundary of a 2-component Gaussian mixture fitted to 1,003,974 clips: modes at 1.587 and 4.867),
female when at most 15 % do, and uncertain in between — 209 male, 170 female, 121 uncertain.
High vn_GEND_reg is masculine (it correlates +0.87 with chest resonance and −0.55 with head
resonance across the 500 per-voice means); the inverted prose in caption_general in the older
corpus release is a captioning bug and was never the source here.
The two labels agree on 89.2 % of the 379 voices this classification calls confidently — which
is also a fourth independent confirmation of the polarity, since an inverted ladder would give
~11 % — and they disagree on 41. names.csv flags every one (agrees_with_card = NO).
The 41 track speaker-identity failure. Disagreement rate falls monotonically with how well a
voice reproduces its own reference: 15.8 % in the weakest quartile by frac_ge06 (share of
takes with spk_sim >= 0.60) down to 3.2 % in the strongest. frac_ge06 differs significantly
between agreeing and disagreeing voices (0.248 vs 0.181, Welch p = 0.011), as does
min_spk_effective (p = 0.032). By family: char 21.1 %, emolia 10.4 %, mediathek 7.0 %,
anime and refvoice 0 %.
It does not explain every case, though: the bare weak/strong split is not significant on its own
(chi-square p = 0.10), and seven of the ten largest disagreements are char voices specced
Androgynous that measure 5.1–5.3 — vague spec rather than failed generation. §12's caveats
apply; nobody has listened to any of them.
The uncertain bucket is mostly measured, not designed, ambiguity. Of the 121, only 12 are
specced Androgynous; 69 are specced Female and 40 Male but do not render consistently on one
side. Within-voice std(vn_GEND_reg) averages 1.288 there against 0.759 for decided
voices — they genuinely swing take to take. Filter card_gender == "Androgynous" if you want only
the designed-androgynous voices.
Reference audio is here, not in moss-voice-profile-references
Measured 2026-08-23 from that repo's own metadata.parquet, because earlier versions of this card
described it as the reference set for these voices and that is wrong:
| field | value |
|---|---|
| distinct voices | 1 — k325_age3_bg1 |
distinct profile |
4 (PEX3, PRGN4, RGN3, RW) |
distinct variant |
4 (raw, raw_sidon, vc, vc_sidon) |
| rows | 106,424 |
| overlap with the 500 profiles here | 1 of 500 |
It is a single-voice reference / voice-conversion ablation study. Useful for what it is, but it does not contain the reference clips of these 500 voices and never did.
The 500 reference clips are the voices/<voice>/reference.wav files in this repo, one per
profile, with reference_file / reference_variant / reference_best_version in
manifest.parquet. There is no standalone published dataset of them, so this release is the
reference-audio artefact.
10. manifest.parquet — the 56 columns
One row per voice, 500 rows, 56 columns. manifest.json carries identical rows, and each voice's own row is duplicated standalone at voices/<voice>/voice.json.
Identity and recipe
| column | type | meaning |
|---|---|---|
voice |
string | Voice identifier. The primary key, and the directory name under voices/. |
src_tree |
string | Which build produced the shipped weights: repair (the 500-voice identity-repair run, 380 rows), lora3k (the 3,000-clip retrain, 115), lora3k_fc (the originals-only floor-capped retrain, 4), pvlorarelease (the single-voice reference study, emolia_c1699 only, 1). |
recipe |
string | repair_single_stage = one stage, 6 epochs, pool by the 0.60-or-relaxed rule. retrain_top3000_floor040 = one stage, 6 epochs, pool = top 3,000 takes by speaker similarity with a hard 0.40 floor. retrain_floorcap_originals_only = the same but originals only, every original take above the floor and no more. reference_study_2stage = the two-stage curriculum used only for emolia_c1699. |
stage |
string | stage1 for every row. These are first-stage identity adapters. |
rank |
int | LoRA rank. 4 for every voice. (Access as m["rank"] — it shadows a DataFrame method.) |
lora_alpha |
int | LoRA alpha. Always 2 × rank, i.e. 8. |
n_target_modules |
int | Number of names in target_modules. 23 for every voice. |
base_model |
string | The base checkpoint the adapter attaches to. Identical for all 500. |
Training pool
| column | type | meaning |
|---|---|---|
pool_rows |
int | Number of training clips the shipped adapter was fitted on. |
pool_rows_stage2_sharpen |
float | emolia_c1699 only: size of the second-stage sharpening pass on top of pool_rows. Null for every other voice. |
pool_mean_spk |
float | Mean ECAPA cosine to the voice's reference over the training pool. Higher = a purer pool. Null for emolia_c1699. |
min_spk_effective |
float | The lowest speaker similarity that actually entered the pool — the threshold the pool realises, whatever rule produced it. This is the number to read, not the nominal threshold. |
pool_max_spk |
float | Highest speaker similarity in the pool. Only recorded for retrained voices. |
pool_n_orig |
int | How many pool clips are ORIGINAL generator output. |
pool_n_repaired |
int | How many pool clips are REGENERATED by the identity-repair pipeline. 0 on every shipped row. |
used_repaired |
bool | True when the pool had to be topped up with regenerated audio. False on every shipped row. |
Training and evaluation
| column | type | meaning |
|---|---|---|
epoch_chosen |
string | The epoch whose checkpoint is shipped, chosen as the minimum held-out loss over the epochs trained. stage2 for emolia_c1699. |
epochs_run |
int | How many epochs were trained before the best was picked. 6 for 499 voices; 1 for emolia_c1699. |
val_loss |
float | Held-out validation loss of the shipped adapter. Lower is better. Comparable across voices only together with base_val_loss, since each voice has its own held-out set. |
base_val_loss |
float | Held-out loss of the frozen base model on the same rows, adapter disabled. |
gain |
float | base_val_loss − val_loss. The headline number. Always positive here — an adapter that did not beat the base was never shipped. |
val_rows |
float | Number of held-out clips the loss was measured on, where recorded — populated for 1 row only; null elsewhere. |
val_gids |
float | Number of held-out groups, where recorded. Null on all rows in this build. |
gpu_h |
float | GPU-hours of the training stage that produced the shipped weights. Encoding not included, discarded challengers not included. |
Retrain bookkeeping
| column | type | meaning |
|---|---|---|
retrained |
bool | True if the shipped weights come from this release's retrain pass (119 rows). |
retrain_attempted |
bool | True if this voice was one of the 123 retrain targets, whether or not the retrain won. |
retrain_improved |
bool | True if the retrained adapter beat the incumbent on held-out loss and was shipped. |
retrain_status |
string | not_a_target (377, already at/above saturation), shipped_retrain (115), shipped_floorcap (4), rejected_worse (4, incumbent shipped), not_finished (0). |
candidate_shipped |
string | Which candidate won: incumbent, retrain_3k or floorcap. Null for voices never targeted. |
candidates_tried |
int | How many adapters were trained and compared (2 = incumbent + 3,000-clip retrain; 3 = both plus the originals-only floor-capped variant, the six sub-floor voices). Null for non-targets. |
old_val_loss |
float | The old_* columns always describe the adapter that was not shipped, measured on the same held-out rows. For shipped_retrain that is the replaced incumbent; for rejected_worse it is the retrain that lost. Null where only one adapter ever existed. |
old_gain |
float | Gain over base of the adapter that was not shipped. |
old_pool_rows |
int | Pool size of the adapter that was not shipped. |
delta_val_loss |
float | loser_val_loss − winner_val_loss — how much the shipped adapter wins by. Always > 0. |
Quality flags
| column | type | meaning |
|---|---|---|
below_saturation |
bool | True if pool_rows < 3000: the adapter is on the steep part of the pool-size/quality curve and is expected to be weaker. 8 rows. |
sub_floor_voice |
bool | True for the six voices whose ORIGINAL takes could not supply 3,000 clips above the 0.40 identity floor. 6 rows. |
Reference clip
| column | type | meaning |
|---|---|---|
reference_file |
string | Path, relative to the repo root, of this voice's reference clip. The base model is reference-conditioned: you need this file as well as the weights. |
reference_variant |
string | Which enhancement variant is shipped (orig, sidon or cbx), taken from the profile card's best_version. |
reference_best_version |
string | The variant the profile card names as best, by DNSMOS. Normally equal to reference_variant. |
Profile card
| column | type | meaning |
|---|---|---|
voice_name |
string | Human-readable name, e.g. "Grave Documentary Narrator". |
tagline |
string | One-line description of the voice. |
card_gender |
string | Intended gender, from the profile card — the field to use. Not derived from the buggy caption_general prose, and not a measurement of the produced audio. |
card_age |
string | Intended age range. |
card_language |
string | Primary language of the card. The corpus itself is English and German. Null for 115 voices. |
card_accent |
string | Intended accent. |
card_register |
string | Intended vocal register. |
card_source |
string | Upstream voice family: emolia (219), mediathek (124), char (115), refvoice (27) or anime (15). |
card_tags |
string | Comma-joined descriptive tags. Useful for filtering. |
Paths and integrity
| column | type | meaning |
|---|---|---|
adapter_dir |
string | This voice's adapter directory relative to the repo root (voices/<voice>). On the Hub, use this one. |
adapter_dir_abs |
string | Absolute path on the build machine. Provenance only. |
adapter_weights_abs |
string | Absolute path to adapter_model.safetensors on the build machine. Provenance only. |
adapter_config_abs |
string | Absolute path to adapter_config.json on the build machine. Provenance only. |
release_root |
string | Absolute path of the release root as built. Provenance only. |
adapter_src |
string | The build tree the weights were copied FROM. Provenance only — never load from this path. |
adapter_sha256 |
string | SHA-256 of the shipped adapter_model.safetensors. All 500 distinct. |
adapter_bytes |
int | Size of the shipped adapter_model.safetensors in bytes. Sums to 17,215,216,000. |
Useful slices:
import pandas as pd
m = pd.read_parquet("manifest.parquet")
m.nlargest(10, "gain")[["voice", "voice_name", "pool_rows", "gain"]] # the strongest
m[m["below_saturation"] | m["sub_floor_voice"]] # everything flagged weak
m[m["retrain_attempted"]].groupby("retrain_status")["gain"].describe() # what the retrain did
11. Related repositories
| repo | what it is |
|---|---|
laion/moss-tts-local-transformer-4.55b-voice-acting-v2 |
The base model. Every adapter here attaches to it: a 4.55 B reference-conditioned TTS model — roughly a 4 B semantic transformer plus a ~550 M talker head. ~9.1 GB in bf16. |
OpenMOSS-Team/MOSS-Audio-Tokenizer-v2 |
The audio codec. 12 codebooks × 1024 entries, 12.5 fps exactly (hop 3,840 at 48 kHz). Pass it as codec_path= to the processor; nothing decodes without it. |
TTS-AGI/moss-voice-profile-references |
Despite the name, one voice, not 500 — a reference/voice-conversion ablation on k325_age3_bg1 alone (106,424 rows, 4 pipeline variants). It is not the reference set for this release; see the note in §9. |
TTS-AGI/moss-voice-profile-loras |
The 10-voice pilot and rank-ablation study this release was designed from — see the section above. |
speechbrain/spkrec-ecapa-voxceleb |
The ECAPA speaker embedder behind every spk_sim number in this release. |
A further consolidated reference-voice dataset exists internally (TTS-AGI/moss-reference-voices-consolidated) but is private and not linkable — it is named only so the provenance chain is complete.
12. Limitations in full
- They are not validated perceptually. Every quality number in this release is a held-out language-modelling loss on MOSS codes, plus ECAPA speaker cosine. No listening test was run. "Higher gain" means "the model predicts this voice's audio tokens better", which is a proxy for, not a measurement of, how much it sounds like the voice.
gainis not strictly comparable across voices. Each voice has its own held-out set, so absolute losses live on different scales. The valid comparison is adapter vs. base within a voice — which is exactly whatgainis.- 8 voices are below the saturation point and 6 are sub-floor. They ship flagged and named (§8), not silently.
- They are not speaker clones of real people. Every profile is synthetic. There is no real identity to recover from these weights. Please do not present output as a real person's voice.
- They are not a substitute for the reference clip. Generation without a matching reference gives a random speaker with the adapter's colouring.
- The 0.40 and 0.60 thresholds do not transfer to another speaker embedder (§6).
- Identity is bought with expressiveness. Hard filtering on speaker similarity selects for clips that sound like the profile, not for clips that are expressive or well articulated; the project's reference run measured −11 % target emotion strength and −14 % blend for its identity gain.
- They are not stackable. Nothing here was measured with two voice adapters attached at once, or at a merge scale other than 1.0.
- Languages are English and German only. The corpus contains nothing else.
- They inherit the base model's failure modes, including empty decodes — always check
audio_codes_listbefore using the output. - Gender and background-noise prompting is unreliable. See below.
The caption_general polarity bug
Two of the descriptive axes in the corpus's rendered caption_general prose — GEND (gender) and BKGN (background noise) — are rendered with inverted polarity. It is a bug in the prose only: the numeric vn_GEND_* / vn_BKGN_* columns are correct and authoritative.
Why it affects these adapters, and how much. The training instruction for each clip is regenerated every epoch from the numeric VoiceNet buckets, and that path is unaffected. But with probability 0.35 the generator instead uses the clip's authored caption_general verbatim — the buggy prose. So roughly a third of the instructions each adapter saw during training carried inverted gender and background-noise wording. This applies to all 500 adapters equally, retrained and untouched alike. It is a property of the corpus these adapters were built from, and it cannot be corrected by retraining alone.
Practical consequence: do not rely on gender or background-noise wording in a prompt to steer these adapters on those two axes — the association they learned there is unreliable. All other axes are unaffected. And do not read a voice's gender out of caption_general; use card_gender (§9). This README deliberately never describes a voice using caption_general.
How the polarity was established, and why the obvious test cannot work
caption_general is generated deterministically from the numeric buckets, so it agrees with them by construction whichever way the ladder points. Comparing the prose against the bucket therefore proves nothing — a keyword check of exactly that kind over 1,112 gender-mentioning captions across 47 voices returned 52 % agreement, i.e. chance. The polarity has to be pinned against acoustic correlates instead:
| evidence | value |
|---|---|
bucket 6, captioned "strongly feminine": chest resonance vn_R_CHST_reg |
3.556 (corpus maximum) |
bucket 6: head resonance vn_R_HEAD_reg |
1.314 (corpus minimum) |
| bucket 6: brightness | 1.274 (darkest timbre) |
| bucket 0, captioned "strongly masculine" | the mirror image of the above |
corr(vn_GEND_reg, vn_R_CHST_reg) |
+0.578 |
corr(vn_GEND_reg, vn_BRGT_reg) |
−0.445 |
corr(vn_GEND_reg, vn_R_MASK_reg) |
−0.252 |
corr(vn_GEND_reg, vn_R_HEAD_reg) |
−0.178 |
Chest-versus-head resonance is the physiologically correct masculine/feminine marker, and all four correlates agree: the bucket the prose calls "feminine" is acoustically the masculine end. For BKGN the same approach gives corr(vn_BKGN_reg, vn_RCQL_reg) = +0.786 and corr(vn_BKGN_reg, qual_background_quality) = +0.402, the latter from an independent model head. Caveat: this is an acoustic-correlate argument. No listening test was run.
Provenance note: the corpus's published MOSS codes are unusable
The adapters here were trained from MOSS codes re-encoded directly from the source mp3s. They do not use the *.moss.npy arrays published in the vprof_base / vprof_repaired WebDataset copies of the corpus, and neither should you.
Those published codes encode half-speed audio. The corpus tokeniser decoded mp3s through a path that did frame.to_ndarray().reshape(-1), which is correct only for mono; PyAV returns (channels, nb_samples) for planar layouts and (1, nb_samples × channels) for packed ones, and a blind flatten turns both into a signal of length nb_samples × channels. The voice-profile audio is duplicated mono written as a 2-channel mp3 (max |L−R| = 0.000000), so the flatten produced [s0,s0,s1,s1,…] — a clean half-speed signal of exactly twice the length. Measured and independently confirmed twice: true duration 15.92 s against a stored dur_s of 31.84 s (ratio exactly 2.000), and moss_frames / true_duration = 25.00 fps against the nominal 12.5. vprof_base (1,907 shards) and vprof_repaired (25,993 shards) are affected; the mono datasets are not. The mp3s themselves are fine — only the derived codes and durations are wrong, and re-encoding from the audio, which is what this release did, recovers everything.
The methodological lesson is worth more than the bug. dur_s and moss_frames were both doubled, so the obvious integrity check — moss_frames == floor(dur_s × 12.5) — passed on every affected row, and a first scope test built on that check reported "0 affected shards". An internal-consistency check cannot detect a shared-mode error: when every derived quantity is wrong in the same way, they still agree with each other. Catching it required decoding the audio and comparing against an independent measurement.
13. Reproducing any of this
Per-voice voice.json records the exact pool size, effective threshold, epoch and both loss numbers for that adapter; manifest.parquet is the same data for all 500. The build code lives in vprof/lora3k/code/ (l3codes.py pool selection + encoding, l3train.py training and epoch choice, l3pack.py selection and packaging) and the trainer it shells out to is vprof/idloop/code/idlora.py, unchanged.
The two figures in this README that are not recomputable from the shipped manifest are the total GPU-hours including discarded challengers (≈406 GPU-h; the manifest's gpu_h column sums to 378.7 over shipped stages only) and the per-epoch validation curves behind the "472/499 strictly monotone" statement, which live in the build logs rather than the release. Everything else in §4 through §8 was recomputed from manifest.parquet and the shipped adapter_model.safetensors files when this README was written.
Citation
@misc{laion_moss_voice_profile_loras_500,
title = {MOSS Voice-Profile LoRAs: 500 synthetic character voices for
moss-tts-local-transformer-4.55b-voice-acting-v2},
author = {{LAION}},
year = {2026},
howpublished = {\url{https://huggingface.co/laion/moss-voice-profile-loras-500}}
}
Licence
This release — the 500 LoRA adapters, the reference clips, the profile cards, the manifests and this documentation — is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0).
You are free to share and adapt this material for any purpose, including commercially, provided you give appropriate credit, link to the licence, and indicate if changes were made.
Two things the licence does not cover, and which you must check separately:
- The base model
laion/moss-tts-local-transformer-4.55b-voice-acting-v2and the codecOpenMOSS-Team/MOSS-Audio-Tokenizer-v2carry their own licences. These adapters are useless without both. Read theirs before deploying. - Synthetic voices are still voices. Every profile here is invented and no adapter is trained on a real speaker, but generated speech can still be used to deceive. Do not present output from these adapters as a recording of a real person, and disclose synthetic speech where a listener could reasonably mistake it for human.
- Downloads last month
- -