laion/moss-va-sft3-emotion-loras

40 emotion adapters for MOSS voice-acting v2 SFT-3

One rank-16 LoRA per emotion, each trained on the top 1 % of that emotion across the whole corpus — filtered so that measured naturalness (genuineness) and vocal-burst blend are both above the corpus median.

That filter is the point, not a detail. Measured on a 360-clip listening sample, the median genuineness percentile falls from 0.391 in the top 10 % of an emotion to 0.360 in the top 1 %: the most intense recordings in this corpus are on average the least genuine, so selecting on intensity alone teaches a model to overact.

These adapters are emotion-specific and voice-agnostic — an "Anger" adapter has seen many speakers. For voice identity use the 500 voice adapters instead; the two stack.

What they do, measured

17 adapters compared against the best general checkpoint on identical prompts, identical sampling, identical seed, all prompts from the same three speakers. Each adapter got ten prompts naming its emotion and ten naming no emotion at all.

	mean	adapters improving
emotion when asked for	+0.047	11 of 17
emotion when not asked for	+0.033	12 of 17
word error rate	0.190 vs 0.121 baseline	—

So they do push their emotion — but they push almost as hard when nobody asked. The selectivity ratio is only about 1.4 : 1, and they cost roughly half again as much transcription error. Do not merge one in permanently and expect neutral speech to survive.

Use the merge weight

A scale sweep over 31 adapters, 8 prompts each, at the extreme band with vocal bursts:

merge weight	emotion	genuineness	burst blend	WER median	WER mean	duration error
0 (base)	0.408	0.817	0.925	0.000	0.167	0.100 s
0.25	0.407	0.844	0.955	0.000	0.179	0.100 s
0.5	0.430	0.833	0.923	0.000	0.146	0.100 s
1.0	0.441	0.836	0.954	0.000	0.130	0.100 s
1.5	0.471	0.846	0.961	0.000	0.096	0.100 s
2.0	0.492	0.880	0.969	0.030	0.184	0.100 s

Emotion rises monotonically and, contrary to expectation, genuineness and burst blend rise with it. Timing is untouched at every weight. What breaks is intelligibility, and only between 1.5 and 2.0 — and it breaks as a tail of a few completely derailed clips, not as general decay: the median word error rate is still 0.000 at 1.5 and the mean is at its lowest there.

Recommended operating point: 1.5. Take 2.0 only if you are already generating best-of-N and can discard the failures. For soft or neutral requests, weight 0 — the natural mode is to derive the weight from the requested intensity band, which the prompt already states.

Layout

<Emotion>/adapter_config.json
<Emotion>/adapter_model.safetensors
<Emotion>/bucket.json          # training rows, steps, wall time, prompt-format hash

The 40 adapters

Every adapter: rank 16, alpha 32, 5 epochs, cosine 1e-4 to 5e-6, 34.4 M trainable parameters (0.83 % of the model). All finished with zero non-finite batches.

emotion	training rows	steps	minutes
Affection	5,235	6,545	87
Amusement	13,272	16,590	215
Anger	6,142	7,680	101
Astonishment_Surprise	4,899	6,125	78
Awe	3,247	4,060	50
Bitterness	3,364	4,205	52
Concentration	5,400	6,750	88
Confusion	10,028	12,535	170
Contemplation	6,867	8,585	119
Contempt	3,105	3,885	48
Contentment	3,445	4,310	58
Disappointment	7,827	9,785	115
Disgust	3,271	4,090	44
Distress	9,111	11,390	124
Doubt	9,004	11,255	136
Elation	6,046	7,560	88
Embarrassment	13,795	17,245	248
Emotional_Numbness	4,633	5,795	77
Fatigue_Exhaustion	9,321	11,655	151
Fear	7,472	9,340	122
Helplessness	9,672	12,090	162
Hope_Enthusiasm_Optimism	8,700	10,875	162
Impatience_and_Irritability	8,673	10,845	143
Infatuation	6,420	8,025	109
Interest	11,224	14,030	173
Intoxication_Altered_States_of_Consciousness	15,725	19,660	224
Jealousy_and_Envy	4,503	5,630	63
Longing	7,981	9,980	107
Malevolence_Malice	6,628	8,285	101
Pain	5,784	7,230	83
Pleasure_Ecstasy	2,976	3,720	42
Pride	6,784	8,480	100
Relief	6,570	8,215	107
Sadness	7,970	9,965	111
Sexual_Lust	9,606	12,010	172
Shame	7,475	9,345	118
Sourness	2,663	3,330	42
Teasing	3,302	4,130	52
Thankfulness_Gratitude	6,450	8,065	108
Triumph	8,685	10,860	154

Inference

import torch, torchaudio
from transformers import AutoProcessor, AutoModel

BASE = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3"

proc = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
model = AutoModel.from_pretrained(BASE, trust_remote_code=True,
                                  dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()

prompt = open("prompt.txt").read()          # the <user_inst> block, see "How to prompt" below
um = {"role": "user", "content": prompt, "audio_codes_list": []}
b = proc([[um]], mode="generation")

with torch.no_grad():
    out = model.generate(input_ids=b["input_ids"].cuda(),
                         attention_mask=b["attention_mask"].cuda(),
                         max_new_frames=340, do_sample=True,
                         audio_temperature=1.0, audio_top_p=0.95, audio_top_k=50,
                         audio_repetition_penalty=1.0)

# codes -> waveform. Use the processor's own decoder: calling the audio tokenizer directly, or
# reshaping its output, yields a two-channel result that flattens into audio at HALF SPEED and
# still sounds like speech. This project lost a whole corpus to that once.
wav = proc.decode_audio_codes([out_codes], return_stereo=False)[0].reshape(-1).float().cpu()
torchaudio.save("out.flac", wav[None], int(proc.model_config.sampling_rate), format="flac")

Loading adapters

from peft import PeftModel

# one adapter
model = PeftModel.from_pretrained(model, "laion/moss-va-sft3-dpo-lora")

# several, each with its own weight -- the usual case: identity from a voice adapter,
# affect from an emotion adapter, general quality from the DPO adapter.
#
# NOTE: `add_weighted_adapter(..., combination_type="linear")` does NOT work here. It raises
# `ValueError: All adapters must have the same r value`, because the DPO adapter is rank 64 and
# the voice / emotion adapters are rank 16. Activate them together instead and scale each one.
model = PeftModel.from_pretrained(model, "<dpo adapter path>", adapter_name="dpo")
model.load_adapter("<voice adapter path>", adapter_name="voice")
model.load_adapter("<emotion adapter path>", adapter_name="emo")

names = ["dpo", "voice", "emo"]
weights = {"dpo": 1.0, "voice": 1.0, "emo": 1.5}     # 1.5 for emotion is the measured optimum
model.base_model.set_adapter(names)                  # the TUNER takes a list; PeftModel does not
model.active_adapter = names[0]                      # must stay a str or generate() indexes a list

for mod in model.modules():
    sc = getattr(mod, "scaling", None)
    if isinstance(sc, dict):
        if not hasattr(mod, "_base_scaling"):
            mod._base_scaling = dict(sc)
        for k in sc:
            if k in weights:
                sc[k] = mod._base_scaling[k] * weights[k]

Scaling an adapter without re-merging

A LoRA layer computes h + scaling · B(A(x)), so multiplying the stored scaling is the merge weight — exact and reversible:

def set_lora_scale(model, w):
    for mod in model.modules():
        sc = getattr(mod, "scaling", None)
        if isinstance(sc, dict):
            if not hasattr(mod, "_base_scaling"):
                mod._base_scaling = dict(sc)
            for k in sc:
                sc[k] = mod._base_scaling[k] * w

How to prompt this model

Every request is one <user_inst> block. The fields are fixed — none may be added or removed:

<user_inst>
- Reference(s):
{None | Speaker: <name> | <|audio|>}
- Instruction:
{GENERAL: ... and/or SCRIPT: ...}
- Tokens:
{target length in audio frames}
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
{English | German}
- Text:
{the same script as under SCRIPT:, character for character}
</user_inst>

Field	What goes in it
Reference(s)	<|audio|> when a reference recording of the target voice is attached, Speaker: <name> when only a voice name is known, otherwise None.
Instruction	A GENERAL: line, a SCRIPT: block, or both.
Tokens	Target length in audio frames. The tokenizer runs at 12.5 frames per second, so 12.8 s = 160 frames. This is the length budget and the numbers in the script must add up to it.
Quality, Sound Event, Ambient Sound	Always None. Kept so the field layout matches the base model.
Language	English or German.
Text	The rendered script, byte-identical to the SCRIPT: block.

GENERAL: — who is speaking

Prose describing the voice and the clip: age and gender, energy and pace, tension, timbre, clarity, pitch range, breath, affect, which emotions are audible, style, recording quality.

GENERAL: A young adult masculine voice; delivery is normally alert, brisk, neutral tension;
timbre is neutral-toned, fairly smooth; average clarity, wide pitch range, light breath;
affect is mildly positive, slightly dominant; reads as bitterness, contempt; 9.5s, EN.

The phrase reads as … is where the emotion names live.

SCRIPT: — what to say, when, and how

Four kinds of tag, told apart by their brackets:

Tag	Means	Rule
[3.9 seconds duration]	the next sentence must take this long	square brackets, stands before the text, one per speech segment
[0.8 seconds pause]	silence of this length	square brackets; every gap of 0.2 s or more, including before the first word and after the last
(contented sigh, 0.2 seconds)	a non-speech vocalisation of this length	round brackets with a duration — label first, then the seconds
(clearly amused, warm and open, unguarded)	how to perform the next sentence	round brackets without a duration, stands before the duration tag

The disambiguation rule in one line: square bracket = a number of seconds; round bracket with a number = a vocal burst; round bracket without a number = a delivery direction. That is the only thing separating a burst from a direction, which is why directions never carry a number.

A complete example:

<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: A young adult feminine voice, warm and conversational; reads as amusement; 6.0s, EN.
SCRIPT:
[0.4 seconds pause] (intensely amused: letting it out / not hiding it, warm and open,
unguarded; bright, relaxed) [2.4 seconds duration] You are not going to believe this.
[0.3 seconds pause] (breathy giggle, 0.4 seconds) [0.2 seconds pause] (still intensely amused)
[2.3 seconds duration] He actually wore it to the wedding.
- Tokens:
75
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
[0.4 seconds pause] (intensely amused: letting it out / not hiding it, warm and open,
unguarded; bright, relaxed) [2.4 seconds duration] You are not going to believe this.
[0.3 seconds pause] (breathy giggle, 0.4 seconds) [0.2 seconds pause] (still intensely amused)
[2.3 seconds duration] He actually wore it to the wedding.
</user_inst>

0.4 + 2.4 + 0.3 + 0.4 + 0.2 + 2.3 = 6.0 s = 75 frames. If the numbers do not add up to the token budget the model has to choose which to honour, and length control is the thing it honours best.

Segmentation rules the training data followed

• split at sentence ends (. ! ? …) and at every vocal burst;
• any segment still longer than 12 s is split again at its largest internal gap;
• a duration is measured from the first word onset to the last word offset of that segment, so two sentences of 12 s and 8 s produce [12.0 seconds duration] and [8.0 seconds duration], never a single [20.0 seconds duration];
• gaps below 0.20 s are folded into the neighbouring speech instead of being printed, so the printed numbers still add up;
• a burst that overlaps speech prints only the part that does not overlap.

Vocal-burst labels that actually occur in training

low mumble, ahem, contented sigh, surprised gasp, chuckle, breathy giggle, childlike giggle, wistful sigh, exhausted groan, sharp inhale, resonant hum, scream, yawn, deep breath, soft hum, exasperated sigh, cackle, shriek, coughing, mournful wail, growl, purr.

Realistic lengths: median 0.28 s, 10th percentile 0.14 s, 90th percentile 0.48 s, longest observed 2.46 s. A sigh requested at 3 s is outside anything in the data.

Intensity bands

Delivery directions carry an intensity adverb drawn from the percentile band of the requested emotion. The same cutoffs are used by the training data, the reward and the evaluation:

band	percentile	adverbs
faint	0.40 – 0.70	barely, faintly, only slightly, just a little
moderate	0.70 – 0.90	clearly, plainly, noticeably, unmistakably
intense	0.90 – 0.98	strongly, intensely, very, deeply
extreme	0.98 – 1.00	overwhelmingly, extremely, utterly, completely

The family

🧩 Base model (required by every adapter here)	laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft3
🎚️ Preference-tuning adapter	laion/moss-va-sft3-dpo-lora
🎭 40 emotion adapters	laion/moss-va-sft3-emotion-loras
🗣️ 500 voice adapters	laion/moss-va-sft3-voice-loras
🔊 Listening page — nine models on the same 80 prompts, with ASR transcripts	laion/moss-va-sft3-samples
🔬 Emotion adapters vs baseline, matched and neutral prompts	laion/moss-va-emotion-loras
🧪 Four-factor study — what actually drives the emotion score	laion/moss-va-four-factor-study
📄 Technical report	laion/moss-va-technical-report
📖 Voice-acting manual (v2 model)	projects.laion.ai/moss-voiceacting-manual
⬅️ Predecessors	voice-acting-v2 · -sft · -sft-dpo