laion/moss-voice-acting-playground

🎭 MOSS Voice-Acting Playground — streaming server + web UI

A tiny FastAPI server + single-file web UI for the expressive voice-acting model laion/moss-tts-local-transformer-4.55b-voice-acting-v2 (4.55B local-transformer, 48 kHz). It does true frame-by-frame streaming — audio is decoded and sent as it is generated, so playback starts at the first chunk — with reference-voice cloning, a sliding-window click-free decoder, and time-to-first-audio metrics.

Files:

• server.py — the FastAPI server (loads the model + codec on one GPU, streams audio).
• index.html — the web UI (served at /), no build step, plain HTML/JS + Web Audio.
• requirements.txt — dependencies.

Run it

pip install -r requirements.txt
# serve on GPU 0, port 8770:
CUDA_VISIBLE_DEVICES=0 python -m uvicorn server:app --host 0.0.0.0 --port 8770

On first start it downloads the model (8 GB) + the codec OpenMOSS-Team/MOSS-Audio-Tokenizer-v2 (4 GB) and does a warmup pass. Needs ~24 GB VRAM (bf16 model + codec). Then open http://<server-ip>:8770/.

⚠️ Dummy host/port — the examples below use http://YOUR_SERVER_IP:8770 (and, for our current demo, a Cloudflare quick-tunnel URL). These are placeholders / ephemeral — replace them with your own host:port (or a stable tunnel / reverse proxy) when you deploy. The values checked into this repo (--port 8770, dev="cuda:0", the REF_DIR path, the model id) are just what we happen to run right now; treat them as dummies and adjust for your machine.

Prompt format (how the model was trained)

Two fields:

• instruction = the performance direction. Write GENERAL: (overall voice / age / mood / delivery) then SCRIPT: — the same lines again, annotated in position with (delivery cues), vocal bursts like (a soft laugh) / (gasp), and [pause] markers. This is where you direct the emotional arc (it may change mid-scene). Describe a high-quality studio recording and a genuine, spontaneous delivery for the most natural voice.
• text = just the plain spoken words (no cues, no brackets).

The words appear in both fields (plain in text, cue-annotated in the SCRIPT section) — that is intentional.

HTTP API

Base URL below = http://YOUR_SERVER_IP:8770 (dummy — replace with yours).

GET / → the web UI (HTML).

GET /api/models

{ "models": ["v2-best"], "default": "v2-best" }

POST /api/stream → streaming audio

Request body (JSON), all fields optional except you want text + instruction:

{
  "text": "Oh my god, hey! Is that really you?",
  "instruction": "GENERAL: a warm, overjoyed young man ... SCRIPT: (delighted) Oh my god, hey! ...",
  "language": "English",
  "reference": "none",
  "seed": 0,
  "audio_temperature": 1.0, "audio_top_p": 0.95, "audio_top_k": 25, "audio_repetition_penalty": 1.1,
  "text_temperature": 1.0, "text_top_p": 1.0, "text_top_k": 50,
  "max_new_tokens": 3200,
  "chunk_frames": 18,
  "model": "v2-best"
}

reference is "none", a built-in voice id, or a ref_id returned by /api/upload_ref (voice cloning). chunk_frames = how often audio is flushed (smaller = lower first-audio latency; quality is chunk-independent).

Response = application/octet-stream, a sequence of length-prefixed frames:

each frame:  [1 byte tag][4 bytes big-endian uint32 length][length bytes payload]
  tag = 0  -> payload is UTF-8 JSON metadata:
                {"type":"start","sr":48000,"gpu_first_ms":123}
                {"type":"end","gpu_total_ms":4210,"n_frames":260,"duration_sec":16.4}
  tag = 1  -> payload is raw PCM: int16, little-endian, mono, at sample rate `sr` (48000 Hz)

Concatenate all tag=1 payloads to get the full clip. Minimal Python client:

import requests, struct, numpy as np
BASE = "http://YOUR_SERVER_IP:8770"          # <-- DUMMY, replace
r = requests.post(f"{BASE}/api/stream", json={
    "text": "You dare enter my domain?",
    "instruction": "GENERAL: a mighty orc warrior, deep guttural growl. SCRIPT: (bellowing, furious) You dare enter my domain?",
    "language": "English", "chunk_frames": 18, "max_new_tokens": 1200,
}, stream=True)
buf, pcm, sr = b"", [], 48000
for part in r.iter_content(8192):
    buf += part
    while len(buf) >= 5:
        tag, ln = buf[0], struct.unpack(">I", buf[1:5])[0]
        if len(buf) < 5 + ln: break
        payload, buf = buf[5:5+ln], buf[5+ln:]
        if tag == 1: pcm.append(np.frombuffer(payload, "<i2"))
import soundfile as sf
sf.write("out.wav", np.concatenate(pcm).astype(np.float32)/32768, sr)

POST /api/upload_ref?ext=wav → clone a reference voice

Body = raw audio bytes (wav/flac/mp3). Returns {"ref_id": "...", "duration": 3.2, "frames": 40}. Pass the returned ref_id as "reference" in /api/stream to steer the timbre. An empty instruction with a reference gives the highest speaker similarity.

Notes

• Single GPU, requests are serialized (one generation at a time via a lock). For concurrent / high-throughput serving use SGLang-Omni (see the model card).
• The streaming decoder uses a sliding window with left-context + overlap-add crossfade, so long scenes stream click-free without an O(n²) full re-decode.
• Model / codec / prompt format are documented on the model card: laion/moss-tts-local-transformer-4.55b-voice-acting-v2.