Emotion Avatar Server — emotion-conditioned static talking head

Real-time-capable emotion-conditioned talking-head renderer built on IMTalker: a 32-D motion latent generator (flow matching) + a frozen neural renderer, with per-frame emotion + intensity conditioning. Ships with the v35 generator weights (eye-line-pinned training set), the renderer, wav2vec2, a reference avatar image, offline video-generation scripts and the live websocket server.

Live server: lip-sync and streaming fixes

The live path streams 2 s chunks (50 frames @ 25 fps) and is paced to the client audio clock. The version in this package fixes four bugs that made the mouth visibly lag or desync from the audio:

Wrong audio window (main desync). The server encoded a 5 s wav2vec left context but then kept the first 50 generated frames, i.e. the audio from ~3 s earlier. It now keeps the last target_frames of the encoded features, so motion is paired with the chunk being spoken.
Cold restarts every other chunk. FM.sample built the previous-context tensors (prev_gaze/prev_pose/prev_cam) with length 0 on the first chunk while prev_a was padded to 10, so a + pose + cam + gaze raised 60 vs 50 and the server silently regenerated the chunk cold. Previous context is now always exactly num_prev_frames, and the real prev_sample/prev_a_wav context is used.
Silence produced an open mouth. All-zero audio is out of distribution for wav2vec (it normalises per call), so pauses and the drain lead-in showed a wide-open mouth with no sound. Silent chunks now render the reference latent (still head, closed mouth) with a short crossfade (--idle_still, --idle_rms_threshold, --idle_crossfade_frames).
No realtime headroom. The fp32 renderer cost ~~21 ms/frame, i.e. exactly the realtime budget once PersonaPlex decoding is added, so the frame queue drained and the avatar fell seconds behind. The renderer now runs in bf16 (~~11.6 ms/frame); measured production is ~0.8 s per 2 s chunk (2.5x realtime).

Verification on a 14 s live session (350 frames, 1x realtime mic input):

check	result
chunked motion vs offline single-pass	per-frame cosine 0.9996, best shift 0 frames
lip aperture vs audio envelope	lag -2 frames (-80 ms), corr 0.46
packet arrival cadence	mean 40.0 ms, p95 41.1 ms, no gap > 120 ms
late / starved frames	0 / 0
carry failures	0
silent chunks	aperture 0.005 (closed) vs 0.076 on speech

code/emo_code/tools/ contains the scripts used for these measurements (parity_single.py compares chunked vs single-pass latents, parity_chunk.py renders a wav through the live chunk loop, diag_capture_rt.py captures a realtime websocket session).

Repository layout

imtalker/generator.ckpt          # v35 flow-matching generator (FMT, 38.8M params)
imtalker/renderer.ckpt           # frozen IMTalker renderer (512x512)
weights/wav2vec2-base-960h/      # audio encoder (used by the generator)
weights/ref.jpeg                 # reference avatar image
weights/renderer_narrow.pt       # distilled renderer (optional, for the student path)
weights/motion_student_38M.pt    # distilled causal student (optional, live path)
weights/motion_student_emo.pt    # emotion-conditioned student (optional)
weights/blink_motion.pt          # blink latent helper
weights/silence_helium_mean.pt   # silence prior for the student
emotion_avatar_code.zip          # all code: generator/, renderer/, static UI, live server, scripts
scripts/setup_env.sh             # create the venv + download the weights
scripts/generate_video.sh        # OFFLINE video generation with an emotion script
scripts/run_server.sh            # live websocket server (needs PersonaPlex weights, see below)
requirements.txt

PersonaPlex (the 7B speech LLM + Mimi codec) is not part of this repo. The offline video generation below needs no PersonaPlex at all; the live conversational server does, and expects those weights via env vars.

Install

bash scripts/setup_env.sh          # venv + pip + weights + unzips the code to code/emo_code

Requirements: Linux/Windows, Python 3.10+, a CUDA GPU with ≥8 GB for offline rendering (≥20 GB for the live server).

Generate a video with emotions (offline)

One command. Inputs: reference image, speech wav, and an emotion script.

bash scripts/generate_video.sh \
  face.jpg \
  speech.wav \
  "happy@0:0.50,neutral@4.0:0.30,surprised@8.0:0.55,angry@12.0:0.50,sad@16.0:0.45" \
  out_video

Emotion script format

<emotion>@<start_seconds>:<intensity>[,<emotion>@..., ...]

emotion ∈ angry, disgust, fear, happy, neutral, sad, surprised (index contempt also exists)
start_seconds — where in the audio that emotion begins (seconds)
intensity — 0..1, the strength of the expression (0.3–0.55 is the natural range; 0.8+ saturates and can distort)
Between two entries the renderer applies a 15-frame smoothstep ramp and an intensity ease, so turns are smooth. --ramp_frames controls the ramp length.

Any number of segments works; a single entry renders one constant emotion for the whole clip.

Useful flags (`render_conversation.py`)

flag	meaning
`--nfe 10`	ODE steps; higher = better mouth detail (10 is the trained setting, 7 is faster and slightly softer)
`--e_cfg_scale 2.5`	emotion classifier-free guidance; 2–3 is natural, >4 exaggerates
`--head_pin 0.6`	removes the slow head-drift component at inference (0 = off, 1 = strong)
`--latent_smooth 1.0`	gaussian smoothing of the 32-D latent over time (anti-jitter)
`--ramp_frames 15`	emotion transition length (frames @25 fps)
`--wav2vec_sec 2.0`	generator window (must stay 2.0 for these weights)

Example: 7 emotions on the same audio

for e in angry disgust fear happy neutral sad surprised; do
  bash scripts/generate_video.sh face.jpg speech.wav "${e}@0:0.5" "out_${e}"
done

How the emotion conditioning works

The generator is a flow-matching transformer over a 32-D latent sequence (25 fps) conditioned on wav2vec2 audio features. Emotion is a separate conditioning branch, injected in two places:

Emotion embedding — an 8-way one-hot (plus intensity) is projected and added to the transformer tokens: emo = emo_projection(one_hot) * emo_scale.
AdaLN / FiLM modulation — the same one-hot + intensity drives per-frame, per-channel gamma/beta that modulate the residual stream (EmotionFiLM). Intensity is what makes an expression strong or subtle; the class selects which expression.

Per-frame conditioning means the schedule above is followed frame by frame: at each frame the model sees the interpolated one-hot (during ramps it is a soft mixture of the two emotions) and the interpolated intensity. That is why the transitions look continuous instead of snapping.

At inference, e_cfg_scale applies classifier-free guidance between the conditioned and the null (no-emotion) prediction — the same idea as CFG in diffusion models — which sharpens the expression.

Practical notes from training/eval (v35)

Neutral closes the lips almost completely in this checkpoint; for speech rendering use a mild happy (0.22–0.30) as the "speaking neutral". The live server does this automatically (--neutral_as_happy 1).
Intensities: 0.30–0.55 natural, 0.6–0.75 expressive, ≥0.8 starts to distort.
angry and happy open the mouth the most; neutral the least.
Head pose comes from the reference image and is pinned at inference (head_pin); residual drift is a trained behaviour, not a conditioning input.

Live websocket server (conversational, needs PersonaPlex)

scripts/run_server.sh starts the real-time server: a browser sends microphone audio, PersonaPlex generates the spoken reply, and the reply text is classified into emotions on the fly (--emo_mode autotext) which condition the avatar. The browser UI is served at http://<host>:8998/.

Set these first (PersonaPlex weights are not included):

export MOSHI_WEIGHT=/path/personaplex/model.safetensors
export MIMI_WEIGHT=/path/mimi.safetensors
export TOKENIZER=/path/tokenizer_spm_32k_3.model
export MOSHI_SRC=/path/to/moshi_fork          # moshi python sources
export VOICE_DIR=/path/to/voices              # optional voice prompts
bash scripts/run_server.sh

Live path highlights:

2 s chunks (50 frames), audio + video in one binary packet → no A/V drift
previous motion/audio context carried between chunks
3 s of left context for the audio encoder (matches training features)
emotion classifier on CPU in a background thread; ramp 400 ms lookahead
--head_pin 0.6, --buffer_ms 1200, --drain_chunks 1 (2 s lead-in so the video pipeline is warm before speech starts)

Model checkpoint for the live server: imtalker/generator.ckpt (v35).

Version notes

v35 (this release): generator fine-tuned on an eye-line-pinned dataset (heads pinned to the reference framing → very stable head, exact interocular distance), with mouth-aperture weighting + teeth interpenetration supervision.
Training recipe: emo_ramp transitions, chunk-consistency, silence weighting, target_shift_steps 1 (40 ms lookahead).

License / credits

IMTalker (author code + renderer), wav2vec2 (Facebook), PersonaPlex (NVIDIA) — the latter only at runtime for the live server.

Downloads last month: -; Downloads are not tracked for this model. How to track