laion/moss-va-sft3-voice-loras

500 voice adapters for MOSS voice-acting v2 SFT-3

One rank-16 LoRA per synthetic voice profile. Each is trained on every clip of that one voice — about 2,150 rows after dropping the worst decile by measured quality — across all emotions and all VoiceNet dimensions.

They are voice-specific and emotion-agnostic: an adapter learns identity, not affect. For affect use the 40 emotion adapters; the two are trained against the same base and stack cleanly.

The 500 voice profiles are LAION-generated and ship publicly in laion/laion-voice-profiles-sft.

Stacking identity and affect

The intended use is both at once, each with its own weight:

model = PeftModel.from_pretrained(model, "<dpo adapter>",   adapter_name="dpo")
model.load_adapter("<voice adapter>", adapter_name="voice")
model.load_adapter("<emotion adapter>", adapter_name="emo")
# `add_weighted_adapter` cannot be used: it requires equal rank, and DPO is rank 64 while these
# are rank 16. Activate all three and scale each -- the full snippet is in the Inference section.
model.base_model.set_adapter(["dpo", "voice", "emo"])

The emotion weight of 1.5 comes from a measured sweep — see the emotion-adapter card. The voice weight has not been swept; 1.0 is the trained value and the sensible starting point.

Honest limits

• Identity was never gated during voice-profile construction, only ranked, so profiles vary in how distinctive they are.
• No speaker-similarity metric was applied to these adapters. Whether an adapter reproduces its voice has to be judged by ear; we publish them so that can be done.
• The base model's known weakness under strong emotion — occasional drift into a different voice, odd timbre at high intensity — is not fixed by these adapters. Practical use benefits from generating several candidates and ranking them.

The 500 adapters

One per voice profile. Every adapter: rank 16, alpha 32, 5 epochs, cosine 1e-4 to 5e-6, 34.4 M trainable parameters. Training rows per voice: median 2,163, min 1,971, max 2,251. Steps per adapter: median 2,705. 500 of 500 finished with zero non-finite batches.

Each directory holds adapter_config.json, adapter_model.safetensors and a bucket.json recording the exact training set size, step count, wall time and prompt-format hash for that adapter.

Inference

import torch, torchaudio
from transformers import AutoProcessor, AutoModel

BASE = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3"

proc = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
model = AutoModel.from_pretrained(BASE, trust_remote_code=True,
                                  dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()

prompt = open("prompt.txt").read()          # the <user_inst> block, see "How to prompt" below
um = {"role": "user", "content": prompt, "audio_codes_list": []}
b = proc([[um]], mode="generation")

with torch.no_grad():
    out = model.generate(input_ids=b["input_ids"].cuda(),
                         attention_mask=b["attention_mask"].cuda(),
                         max_new_frames=340, do_sample=True,
                         audio_temperature=1.0, audio_top_p=0.95, audio_top_k=50,
                         audio_repetition_penalty=1.0)

# codes -> waveform. Use the processor's own decoder: calling the audio tokenizer directly, or
# reshaping its output, yields a two-channel result that flattens into audio at HALF SPEED and
# still sounds like speech. This project lost a whole corpus to that once.
wav = proc.decode_audio_codes([out_codes], return_stereo=False)[0].reshape(-1).float().cpu()
torchaudio.save("out.flac", wav[None], int(proc.model_config.sampling_rate), format="flac")

Loading adapters

from peft import PeftModel

# one adapter
model = PeftModel.from_pretrained(model, "laion/moss-va-sft3-dpo-lora")

# several, each with its own weight -- the usual case: identity from a voice adapter,
# affect from an emotion adapter, general quality from the DPO adapter.
#
# NOTE: `add_weighted_adapter(..., combination_type="linear")` does NOT work here. It raises
# `ValueError: All adapters must have the same r value`, because the DPO adapter is rank 64 and
# the voice / emotion adapters are rank 16. Activate them together instead and scale each one.
model = PeftModel.from_pretrained(model, "<dpo adapter path>", adapter_name="dpo")
model.load_adapter("<voice adapter path>", adapter_name="voice")
model.load_adapter("<emotion adapter path>", adapter_name="emo")

names = ["dpo", "voice", "emo"]
weights = {"dpo": 1.0, "voice": 1.0, "emo": 1.5}     # 1.5 for emotion is the measured optimum
model.base_model.set_adapter(names)                  # the TUNER takes a list; PeftModel does not
model.active_adapter = names[0]                      # must stay a str or generate() indexes a list

for mod in model.modules():
    sc = getattr(mod, "scaling", None)
    if isinstance(sc, dict):
        if not hasattr(mod, "_base_scaling"):
            mod._base_scaling = dict(sc)
        for k in sc:
            if k in weights:
                sc[k] = mod._base_scaling[k] * weights[k]

Scaling an adapter without re-merging

A LoRA layer computes h + scaling · B(A(x)), so multiplying the stored scaling is the merge weight — exact and reversible:

def set_lora_scale(model, w):
    for mod in model.modules():
        sc = getattr(mod, "scaling", None)
        if isinstance(sc, dict):
            if not hasattr(mod, "_base_scaling"):
                mod._base_scaling = dict(sc)
            for k in sc:
                sc[k] = mod._base_scaling[k] * w

How to prompt this model

Every request is one <user_inst> block. The fields are fixed — none may be added or removed:

<user_inst>
- Reference(s):
{None | Speaker: <name> | <|audio|>}
- Instruction:
{GENERAL: ... and/or SCRIPT: ...}
- Tokens:
{target length in audio frames}
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
{English | German}
- Text:
{the same script as under SCRIPT:, character for character}
</user_inst>

Field	What goes in it
Reference(s)	<|audio|> when a reference recording of the target voice is attached, Speaker: <name> when only a voice name is known, otherwise None.
Instruction	A GENERAL: line, a SCRIPT: block, or both.
Tokens	Target length in audio frames. The tokenizer runs at 12.5 frames per second, so 12.8 s = 160 frames. This is the length budget and the numbers in the script must add up to it.
Quality, Sound Event, Ambient Sound	Always None. Kept so the field layout matches the base model.
Language	English or German.
Text	The rendered script, byte-identical to the SCRIPT: block.

GENERAL: — who is speaking

Prose describing the voice and the clip: age and gender, energy and pace, tension, timbre, clarity, pitch range, breath, affect, which emotions are audible, style, recording quality.

GENERAL: A young adult masculine voice; delivery is normally alert, brisk, neutral tension;
timbre is neutral-toned, fairly smooth; average clarity, wide pitch range, light breath;
affect is mildly positive, slightly dominant; reads as bitterness, contempt; 9.5s, EN.

The phrase reads as … is where the emotion names live.

SCRIPT: — what to say, when, and how

Four kinds of tag, told apart by their brackets:

Tag	Means	Rule
[3.9 seconds duration]	the next sentence must take this long	square brackets, stands before the text, one per speech segment
[0.8 seconds pause]	silence of this length	square brackets; every gap of 0.2 s or more, including before the first word and after the last
(contented sigh, 0.2 seconds)	a non-speech vocalisation of this length	round brackets with a duration — label first, then the seconds
(clearly amused, warm and open, unguarded)	how to perform the next sentence	round brackets without a duration, stands before the duration tag

The disambiguation rule in one line: square bracket = a number of seconds; round bracket with a number = a vocal burst; round bracket without a number = a delivery direction. That is the only thing separating a burst from a direction, which is why directions never carry a number.

A complete example:

<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: A young adult feminine voice, warm and conversational; reads as amusement; 6.0s, EN.
SCRIPT:
[0.4 seconds pause] (intensely amused: letting it out / not hiding it, warm and open,
unguarded; bright, relaxed) [2.4 seconds duration] You are not going to believe this.
[0.3 seconds pause] (breathy giggle, 0.4 seconds) [0.2 seconds pause] (still intensely amused)
[2.3 seconds duration] He actually wore it to the wedding.
- Tokens:
75
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
[0.4 seconds pause] (intensely amused: letting it out / not hiding it, warm and open,
unguarded; bright, relaxed) [2.4 seconds duration] You are not going to believe this.
[0.3 seconds pause] (breathy giggle, 0.4 seconds) [0.2 seconds pause] (still intensely amused)
[2.3 seconds duration] He actually wore it to the wedding.
</user_inst>

0.4 + 2.4 + 0.3 + 0.4 + 0.2 + 2.3 = 6.0 s = 75 frames. If the numbers do not add up to the token budget the model has to choose which to honour, and length control is the thing it honours best.

Segmentation rules the training data followed

• split at sentence ends (. ! ? …) and at every vocal burst;
• any segment still longer than 12 s is split again at its largest internal gap;
• a duration is measured from the first word onset to the last word offset of that segment, so two sentences of 12 s and 8 s produce [12.0 seconds duration] and [8.0 seconds duration], never a single [20.0 seconds duration];
• gaps below 0.20 s are folded into the neighbouring speech instead of being printed, so the printed numbers still add up;
• a burst that overlaps speech prints only the part that does not overlap.

Vocal-burst labels that actually occur in training

low mumble, ahem, contented sigh, surprised gasp, chuckle, breathy giggle, childlike giggle, wistful sigh, exhausted groan, sharp inhale, resonant hum, scream, yawn, deep breath, soft hum, exasperated sigh, cackle, shriek, coughing, mournful wail, growl, purr.

Realistic lengths: median 0.28 s, 10th percentile 0.14 s, 90th percentile 0.48 s, longest observed 2.46 s. A sigh requested at 3 s is outside anything in the data.

Intensity bands

Delivery directions carry an intensity adverb drawn from the percentile band of the requested emotion. The same cutoffs are used by the training data, the reward and the evaluation:

band	percentile	adverbs
faint	0.40 – 0.70	barely, faintly, only slightly, just a little
moderate	0.70 – 0.90	clearly, plainly, noticeably, unmistakably
intense	0.90 – 0.98	strongly, intensely, very, deeply
extreme	0.98 – 1.00	overwhelmingly, extremely, utterly, completely

The family

🧩 Base model (required by every adapter here)	laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3
🎚️ Preference-tuning adapter	laion/moss-va-sft3-dpo-lora
🎭 40 emotion adapters	laion/moss-va-sft3-emotion-loras
🗣️ 500 voice adapters	laion/moss-va-sft3-voice-loras
🔊 Listening page — nine models on the same 80 prompts, with ASR transcripts	laion/moss-va-sft3-samples
🔬 Emotion adapters vs baseline, matched and neutral prompts	laion/moss-va-emotion-loras
🧪 Four-factor study — what actually drives the emotion score	laion/moss-va-four-factor-study
📄 Technical report	laion/moss-va-technical-report
📖 Voice-acting manual (v2 model)	projects.laion.ai/moss-voiceacting-manual
⬅️ Predecessors	voice-acting-v2 · -sft · -sft-dpo

Independent evaluation

These adapters ship with no speaker-similarity measurement; this is one. The section below is a first measurement, run against the checkpoint they were trained for. It is not a listening test and does not replace one.

Setup. Ten fixed English utterances of two to three sentences, rendered into the SFT3 timed-script format, identical in every condition and in the same order. Base is laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3 with laion/moss-va-sft3-dpo-lora-p2 at weight 1.0. One adapter is then added at one weight; nothing else changes, and the sampling seed is fixed. The prompt carries one reference recording of the voice emolia_c1699 from TTS-AGI/moss-voice-profile-references.

Metrics. Word error rate is measured with nvidia/parakeet-tdt-0.6b-v3 against the intended line. Extra words counts words transcribed after the last word that still matches the line — the direct measure of a take running on past its script. Genuineness is laion/voiceclap-commercial-genuineness (0–6) and blend is laion/voiceclap-commercial-vocalburst-blend (0–10). Speaker similarity is ECAPA cosine against that same reference recording.

Baseline, no adapter beyond DPO-p2: word error 0.043, 0.3 extra words, genuineness 1.17, blend 3.35, speaker similarity 0.513.

What this does not tell you. Ten utterances at one seed: the trend across weights is meaningful, a single cell is not. All lines are English, declarative and calm — nothing here describes behaviour on shouting, on German, or on lines carrying vocal bursts. Only one speaker was used. The two scorer models are themselves models, and their absolute levels are not calibrated against human judgement here; differences between conditions are the signal.

Method, all ten prompts, every hyper-parameter and the raw per-take data including transcripts: https://github.com/LAION-AI/Humaneness-Voice-Demo-Server/blob/main/docs/EXPERIMENTS.md

Results

One voice, emolia_c1699, at six weights. The prompt already carries a reference recording of that speaker, so the question is what the adapter adds on top of it.

condition	word error	extra words	takes w/ extra	genuineness 0–6	blend 0–10	speaker sim
voice @0.25	0.020	0.1	10%	1.21	3.29	0.581
voice @0.5	0.052	0.2	20%	1.33	2.69	0.566
voice @0.75	0.028	0.3	20%	1.55	4.48	0.589
voice @1.0	0.038	0.3	20%	1.57	4.45	0.600
voice @1.25	0.088	1.0	30%	1.42	3.06	0.609
voice @1.5	0.072	0.8	50%	1.64	3.08	0.645

Speaker similarity without any voice adapter, from the reference recording alone, is 0.513. The adapter adds 0.068 of that at weight 0.25 and only 0.019 more by 1.0, while word error rises from 0.020 to 0.038 and then to 0.088 at 1.25. Most of the identity is available at a quarter of the trained weight, and the demo downstream now runs it at 0.25.

If you use these adapters without a reference clip in the prompt the trade-off will be different — that case was not measured.