techprotrade/luuna-emotional-voice

Luuna Emotional Voice Engine

One identity, 27 emotions, continuous prosody, streaming, interruptible.

Luuna is a persistent AI companion. This engine decides how she speaks (semantic meaning + context + emotional state + previous state + urgency + punctuation), plans it as structured control data, and executes it on a swappable local TTS stack. It runs fully offline on one RTX 3090 24 GB; cloud APIs are optional adapters only.

Conversation Context
        ↓
LUUNA Emotion Director        (rules + optional LLM via Model Router)
        ↓
Prosody / Emotion Plan        {emotion, intensity, VAD, pace, pitch, energy, ...}
        ↓
Text Segmentation             [pause] markers, soft→strong intra-utterance arcs
        ↓
Expressive TTS Engine         adapters: chatterbox / voxcpm / xtts (swap freely)
        ↓
Audio Post Processor          pace/pitch/energy/breathiness + loudness
        ↓
LUUNA Voice Output            streaming chunks, crossfade-stitched, interruptible

Emotion state machine

• per-session temporal memory: previous state, target state, VAD, timestamps
• smooth transitions (distance-weighted, eased); same-family moves settle fast
• idle decay toward neutral (~90 s)
• emotion persistence: neutral text after a sad utterance stays in the sad family (state_blend), it does not snap to neutral
• intra-sentence performance: "Tom... ma ei tea. [pause_500] See ei tundu..." renders segment 1 soft/low-energy, 500 ms pause, segment 2 with full weight
• explicit instructions ALWAYS override inference

LUUNA VOICE V1 — verified state (2026-09-14)

GPU smoke on a10g-small (24 GB class): 8 Estonian emotions, all green. Samples: samples/gpu-smoke/ · raw numbers: samples/gpu-smoke/benchmark.json

Metric	Value
Model	ResembleAI/chatterbox (multilingual), chatterbox-tts==0.1.7
Model load	34.6 s
VRAM peak	3.41 GB (fits RTX 3090 with huge headroom)
Latency	2.6–5.4 s per multi-sentence utterance (RTF 0.80–0.91)
Prosody control	f0 median varies by emotion: happy 228 Hz, sad 112 Hz, neutral 140 Hz
Estonian	synthesized via grapheme fallback (language_id=en); et not in the model's language list

8 sample files (same voice identity across all): neutral, happy, sad, angry, whisper, excited, concerned, sarcastic — the concerned sample includes the [pause_500] intra-sentence marker from Luuna's real usage.

CPU test suite: 29/29 green (pytest exit 0, commit 12f4f44).

Canonical voice anchor (the "this is Luuna" identity)

Luuna's identity comes from ONE neutral reference clip — emotional references bleed their mood into the performance, so the anchor must be calm speech:

1. Place your chosen clip at C:\Luuna\automation\voice-io\references\luuna_anchor.wav (10–20 s, clean mono ≥ 24 kHz, neutral delivery)
2. Either set it in config/luuna.yaml under engine.reference_voice, or copy it to models\reference\luuna-neutral.wav (auto-detected), or POST /v1/luuna/voice/reference (multipart upload)
3. All 27 emotions render through that one anchor — the model modifies performance, not the speaker

Install (Windows 11, RTX 3090)

git clone <this-repo> ; cd luuna-emotional-voice
powershell -ExecutionPolicy Bypass -File scripts\install.ps1
# put Luuna's NEUTRAL reference clip at models\reference\luuna-neutral.wav

Run

# API server on 127.0.0.1:7890
powershell -ExecutionPolicy Bypass -File scripts\run_server.ps1

# Demo UI on 127.0.0.1:7891
powershell -ExecutionPolicy Bypass -File scripts\run_demo.ps1

# tests (CPU, no model downloads)
powershell -ExecutionPolicy Bypass -File scripts\run_tests.ps1

API

POST /v1/luuna/speak                        {text, session_id, language, emotion, intensity, stream}
POST /v1/luuna/emotion/analyze              -> performance plan
GET  /v1/luuna/emotion/state/{session_id}   -> emotional memory snapshot
POST /v1/luuna/emotion/state/{session_id}   -> manual state override
POST /v1/luuna/voice/reference              -> upload reference voice (multipart)
POST /v1/luuna/speak/{session_id}/cancel    -> interrupt
WS   /v1/luuna/speak/ws                     -> streamed base64 PCM chunks
GET  /health

Example:

POST /v1/luuna/speak
{ "text": "Tere Tom...", "session_id": "tom-main", "language": "et",
  "emotion": "auto", "stream": true }

Streamed PCM starts with a header line LUUNAPCM1 24000 int16\n, then raw 16-bit mono PCM frames per chunk.

Estonian path

No open TTS ships Estonian (verified 2026-09 against official language lists — Chatterbox multilingual: et raises "Unsupported language_id"; grapheme fallback renders õ/ä/ö/ü but rhythm is approximated from English). So:

1. EKKK (Estonian Emotional Speech Corpus, 1 female speaker, anger/joy/sadness/neutral, CC-BY) → Luuna's canonical voice + emotion labels
2. Common Voice et (51.7 h, CC-0) → general Estonian fluency
3. Fine-tune with training/finetune.py (VoxCPM2 official LoRA trainer)
4. Emotion conditioning via inline [emotion] tags — never from the reference audio (prevents reference-emotion bleeding)

See docs/ for the full model comparison, dataset spec and training runbook.

License notes

• Chatterbox: MIT. VoxCPM2: Apache-2.0. XTTS-v2: CPML (non-commercial, fallback/prototype only). Common Voice: CC-0. FLEURS: CC-BY. EKKK: CC-BY.