Ex-Omni

Ex-Omni is a omni-modal framework for generating response text, speech audio, and 3D facial animation from text or speech input.

---

Ex-Omni is an public release for omni-modal response generation. Given text or speech input, the system can produce response text, speech units / decoded audio, and 52-dimensional facial blendshape coefficients, with optional rendering into a talking-face video.

📖 Table of Contents

• Repository Structure
• Quick Start
  • Installation
  • Checkpoints and Assets
  • Launch the Demo
• Acknowledgements
• Citation

🗂️ Repository Structure

.
├── asset/                              # Download the mesh templates here, e.g. EmoTalk.npz
├── cosyvoice/                          # Runtime audio decoder modules
├── deploy.py                           # Main Gradio entrypoint
├── deploy_base.py                      # Shared inference pipeline and UI logic
├── ex_omni/
│   ├── constants.py                    # Runtime constants
│   ├── flow_inference.py               # Audio decoder wrapper
│   ├── render_utils.py                 # Blendshape rendering utilities
│   └── model/
│       ├── language_model/             # Omni model wrapper
│       ├── speech_encoder/             # Whisper speech encoder
│       ├── speech_projector/           # Speech projector
│       ├── speech_generator/           # Speech generator
│       └── blendshape_generator/       # Blendshape generator
├── requirements.txt                    # Python dependencies
└── LICENSE.txt                         # License file

⚙️ Quick Start

Installation

# 1. Create and activate environment
conda create -n Ex-Omni python=3.10 -y
conda activate Ex-Omni

# 2. Install PyTorch (example: CUDA 12.6)
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu1216

# 3. Install project dependencies
pip install -r requirements.txt

# 4. Install pytorch3d separately according to your CUDA / PyTorch version
# See official pytorch3d installation instructions

Checkpoints and Assets

Prepare the following assets before running inference:

Component	Expected Path	Source
Ex-Omni checkpoint	ckpt/Ex-Omni/
Flow decoder checkpoint	ckpt/glm-4-voice-decoder/flow.pt
HiFT decoder checkpoint	ckpt/glm-4-voice-decoder/hift.pt
EmoTalk mesh template	asset/EmoTalk.npz
Claire mesh template	asset/claire.npz

The Claire template is not redistributed in this repository. Please obtain the original Claire asset yourself and convert it into the .npz format.

Launch the Demo

python deploy.py \
  --model-path ckpt/Ex-Omni \
  --flow_ckpt_path ckpt/glm-4-voice-decoder/flow.pt \
  --hift_ckpt_path ckpt/glm-4-voice-decoder/hift.pt \
  --template_type emotalk \
  --port 8080

Then open:

http://localhost:8080

🙏 Acknowledgements

We would like to thank the authors of OpenOmni, LLaMA-Omni2, EmoTalk and UniTalker. Parts of the implementation and overall system design were developed with reference to their open-source release.

📄 Citation

If you use this project, please cite our paper:

@misc{zhang2026exomnienabling3dfacial,
      title={Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models}, 
      author={Haoyu Zhang and Zhipeng Li and Yiwen Guo and Tianshu Yu},
      year={2026},
      eprint={2602.07106},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2602.07106}, 
}