tsinghua-ee/ELLSA2

ELLSA 2 — Standalone Generator Checkpoints

ELLSA 2 studies full-duplex speech interaction with gestures and facial expressions. This model package provides the Standalone generator used as the gesture-expression module in the paper, plus its training initializer trained on ASR and RVQ tokenizers. The provided generator consumes aligned target/partner audio and a style motion clip and produces SMPL-H body/hand poses and FLAME facial parameters.

Available checkpoints

Path	Role
standalone_generator/	ELLSA's causal streaming dialog gesture-expression generator
base_asr_model/	Dual-stream streaming ASR-only base model at 50,000 updates
tokenizers/upper.pth	Causal six-codebook upper-body RVQ-VAE
tokenizers/hands.pth	Causal six-codebook hand RVQ-VAE
tokenizers/flame.pth	Causal face RVQ-VAE, four facial-expression + two mouth codebooks

The model operates in 0.8-second blocks: 10 acoustic embeddings per audio stream and six motion latent steps per block. Eighteen codebook indices per step are decoded into 30 fps motion. The style reference contains 612 SMPL-H frames. Default inference emits each new 24-frame block immediately, with zero right context. The 300-frame gesture and 240-frame expression caps retain at most 74 gesture and 59 expression/mouth latent steps per RVQ branch. Use --rvq-decode-mode whole for complete-sequence reference decoding.

Setup

Place this package next to the downloaded code and data packages:

ELLSA2/
├── code/
├── data/
└── checkpoints/           # This package

For another download location:

export ELLSA2_CHECKPOINT_ROOT=/path/to/checkpoints
export ELLSA2_DATA_ROOT=/path/to/data

Install the environment using the code guide.

Run the Standalone generator

From the code directory:

cd ../code
python infer.py --model-path ../checkpoints/standalone_generator --test-id fro_00038 --output-dir outputs/example

To use your own compatible trained model, replace --model-path with its checkpoint directory, including its configuration, weights, and tokenizer files.

The output NPZ contains body/hand poses, 108D facial motioncode, and discrete codes. See the inference guide for custom aligned audio and style inputs.

Train a standalone gesture-expression generator

Prepare source data and extract facial labels using the data guide. Then run the two-stage recipe:

cd ../code
conda activate ellsa2
NPROC_PER_NODE=8 bash scripts/train.sh --stage expression
NPROC_PER_NODE=8 bash scripts/train.sh --stage gesture_expression

Stage 1 initializes from base_asr_model/; stage 2 initializes from code/runs/expression/. To fine-tune the supplied joint generator, select --stage gesture_expression, pass --model_name_or_path ../checkpoints/standalone_generator, and choose a new --output_dir. Optimizer states are not packaged.

The supplied generator used an intermediate gesture-only stage before joint training. This two-stage expression/joint recipe trains a compatible model; it does not reproduce the supplied weights exactly.

External assets: sources and download guide

Inference to motion NPZ and the released training recipe need only the generator and RVQ weights above. The following assets support mesh rendering, objective geometry metrics, or optional raw facial-label extraction:

Asset	Used by	Source and terms	Included in upload
SMPL-H FEMALE	SMPL-H evaluation renderer	MANO / SMPL+H, model license; AMASS extended shape-space models	No
SMPL-X neutral 2020, 100 expression coefficients	Body/hand/face demo, inference rendering, motion skeleton metrics	SMPL-X, model license	No
FLAME 2020	Facial mesh metrics	FLAME, academic model license	No
KaoLRM mono configuration and weights	Optional raw facial-label extraction	CyberAgent AI Lab's KaoLRM, mono release, CC BY-NC 4.0	Yes

The SMPL-H, full SMPL-X, and FLAME 2020 licenses restrict redistribution, thus you need to download them from the official sites. After obtaining the geometry models, the local layout is:

external_assets/
├── smplh/SMPLH_FEMALE.pkl       # Download and prepare locally
├── smplx/SMPLX_NEUTRAL_2020.npz # Download locally
├── flame/
    └── flame_2020.pt         # Convert locally
└── kaolrm-mono/               # Included in this package
    ├── config.json
    ├── model.safetensors
    └── LICENSE_WEIGHT.txt

Convert the downloaded FLAME 2020 generic_model.pkl from the code directory:

python -m pip install chumpy==0.70 --no-build-isolation
python scripts/convert_flame.py --input /path/to/generic_model.pkl \
  --output ../checkpoints/external_assets/flame/flame_2020.pt