MOSS Mediathek emotion LoRAs

LoRA adapters for laion/moss-tts-local-transformer-4.55b-voice-acting-v2, trained on an emotion-selected subset of German public-broadcast (Mediathek) speech — real recorded human speech, not synthetic.

Nine checkpoints: ranks 16 / 32 / 64 × epochs 1 / 2 / 3.

Read this before using them

On the headline composite score, none of the nine checkpoints is distinguishable from the base model. Measured on a 400-clip held-out set, 40 generations per arm, scored with the VoiceNet/EmoNet/genuineness/blend stack:

arm	composite	genuineness	blend	target emotion
base (prompt only)	0.5015	3.179	4.732	1.468
r16_e1	0.5076	2.597	5.823	2.279
r16_e2	0.5086	2.736	5.612	2.399
r16_e3	0.4879	2.643	5.354	2.465
r32_e1	0.5114	2.648	5.815	2.382
r32_e2	0.4978	2.703	5.451	2.293
r32_e3	0.4979	2.604	5.618	2.251
r64_e1	0.4951	2.609	5.553	2.165
r64_e2 ← default	0.5275	2.909	5.701	2.509
r64_e3	0.5050	2.618	5.737	2.486

No arm reaches ~2 SE against base. diff_vs_base ranges −0.014 to +0.026, |z| ≤ 0.83.

But the composite is hiding a real and consistent trade-off, and that is the actual result:

Target emotion rises sharply and without exception: 1.468 → 2.17–2.51, i.e. +48 % to +71 % on every single checkpoint. If you want the requested emotion to actually come through, these adapters do that.
Blend improves everywhere: 4.732 → 5.35–5.82.
Genuineness falls everywhere: 3.179 → 2.60–2.91. The emotion reads as more performed and less felt.

The composite weights these against each other and they cancel. So the honest summary is not "the training did nothing" — it is "the training buys emotional intensity and burst blending, and pays for it in perceived sincerity." Whether that is a good trade depends entirely on your use case: for expressive character work it usually is; for intimate or documentary-style delivery it usually is not.

Which one to use

r64_e2 is the default: the best composite of the nine, the highest genuineness of any LoRA arm (so the smallest sincerity penalty), and essentially the top target-emotion score. If you want maximum emotional push and can accept lower genuineness, r16_e3 or r64_e3 score marginally higher on target emotion.

Use the usual dose band — 0.35–0.75. Above ~1.0 genuineness and intelligibility collapse.

Usage

import torch
from transformers import AutoModel, AutoProcessor
from peft import PeftModel

REPO = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2"
proc = AutoProcessor.from_pretrained(
    REPO, trust_remote_code=True,
    codec_path="OpenMOSS-Team/MOSS-Audio-Tokenizer-v2")
model = AutoModel.from_pretrained(          # AutoModel, NOT AutoModelForCausalLM
    REPO, trust_remote_code=True,
    dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()

model = PeftModel.from_pretrained(model, "laion/moss-mediathek-emotion-lora/r64_e2").eval()

conv = [[proc.build_user_message(
    text="Ich habe wirklich alles versucht.",           # spoken words ONLY
    instruction='GENERAL: A tired middle-aged man, close mic.\n'
                'SCRIPT:\n(quietly, on the edge of giving up) "Ich habe wirklich alles versucht."',
    language="German",
    tokens=6)]]
b = proc(conv, mode="generation")
out = model.generate(input_ids=b["input_ids"].cuda(),
                     attention_mask=b["attention_mask"].cuda(),
                     max_new_frames=400, do_sample=True,
                     audio_temperature=1.0, audio_top_p=0.95, audio_top_k=30)
wav = proc.decode(out)[0].audio_codes_list[0].cpu().float().numpy()   # 48 kHz

Notes that will otherwise cost you an afternoon:

Load the base with AutoModel, not AutoModelForCausalLM.
Missing audio_lm_heads.* / text_lm_head.weight warnings on load are benign.
instruction holds the whole GENERAL: …\nSCRIPT:… caption; text holds only the spoken words. Empty fields render as the literal string "None".

Training


base	`laion/moss-tts-local-transformer-4.55b-voice-acting-v2`
data	emotion-selected German Mediathek speech (real recordings)
held-out set	400 clips
LoRA alpha	128
dropout	0.05
ranks	16 / 32 / 64
epochs	1 / 2 / 3

A methodological note

Checkpoints here are ranked by generating audio and scoring it, not by validation loss. On a sibling run, validation loss rose from 4.42 to 5.44 across epochs 1→8 — which reads as severe overfitting — while listeners scored the two checkpoints identically. Validation loss has now disagreed with listening three times on this model family. Do not select checkpoints with it.

Model tree for laion/moss-mediathek-emotion-lora

Base model

laion/moss-tts-local-transformer-4.55b-voice-acting-v2

Adapter

(7)

this model

laion
/

moss-mediathek-emotion-lora