[ICCV 2025] FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait

Official PyTorch implementation of FLOAT — Flow Matching for Audio-driven Talking Portrait Video Generation — plus the pre-converted ONNX / TensorRT graphs used by the FLOAT Fast serving stack.

FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait
Taekyung Ki, Dongchan Min, Gyeongsu Chae

Project page: deepbrainai-research.github.io/float · Serving stack: github.com/mohamed-em2m/float-fast

Abstract: With the rapid advancement of diffusion-based generative models, portrait image animation has achieved remarkable results. However, it still faces challenges in temporally consistent video generation and fast sampling due to its iterative sampling nature. This paper presents FLOAT, an audio-driven talking portrait video generation method based on flow matching generative model. Instead of a pixel-based latent space, we take advantage of a learned orthogonal motion latent space, enabling efficient generation and editing of temporally consistent motion. To achieve this, we introduce a transformer-based vector field predictor with an effective frame-wise conditioning mechanism. Additionally, our method supports speech-driven emotion enhancement, enabling a natural incorporation of expressive motions. Extensive experiments demonstrate that our method outperforms state-of-the-art audio-driven talking portrait methods in terms of visual quality, motion fidelity, and efficiency.

TL;DR: FLOAT is a flow-matching based audio-driven talking portrait video generation method, which can enhance the speech-driven emotional motion.

Repository layout

mohamedemam/float/
├── float.pth                                   # the main model checkpoint (827 MB)
├── wav2vec2-base-960h/                         # audio encoder  (facebook/wav2vec2-base-960h)
├── wav2vec-english-speech-emotion-recognition/ # emotion encoder (r-f/wav2vec-…-recognition)
├── assets/                                     # figures used by this card
└── onnx/                                       # pre-converted ONNX graphs (see below)
    ├── fmt_step_v1.onnx            628 MB   # FMT vector-field step  (the ODE inner loop)
    ├── emotion_v1.onnx            1.27 GB   # speech-emotion classifier
    ├── wav2vec_v1.onnx             380 MB   # audio encoder + projection head
    ├── decoder_v1.onnx              94 MB   # frame decoder (StyleGAN-style synthesis)
    ├── refenc_v1.onnx               91 MB   # reference-image encoder
    ├── direction_v1.onnx            42 KB   # motion direction basis
    ├── decoder_v1.onnx.no_trt              # sidecar: the runtime must not use TensorRT here
    ├── backends.generated.yaml              # measured per-component backend plan
    └── .build_manifest.json                 # build provenance for the artifacts

ONNX / TensorRT accelerated inference

The onnx/ folder holds the same six components, exported to ONNX and ready to run on TensorRT / CUDA / CPU. They are produced by float-fast's scripts/convert_to_onnx.sh and are drop-in replacements for the torch modules — the ODE loop, the CFG combine and the chunk autoregression stay in Python, only the hot inner steps move into the graph.

Every artifact is gated before it is published: the converter compares the exported graph against the deployed torch model and refuses to pass a graph that does not reproduce it. Two rules make that gate meaningful, and both are easy to get wrong:

the reference is the deployed model, never the module that was traced;
both sides run the same computation (fp32 on both sides — TF32 alone moves a 22-conv stack by tens of uint8 levels, which would drown the graph's own error).

Measured for the graphs in this folder:

artifact	provider	fidelity vs the torch model
`fmt_step_v1.onnx`	TensorRT FP16	cosine 0.999996 (per ODE step)
`decoder_v1.onnx`	CUDA (fp32)	max 1 / mean 0.002 uint8 level on a 512×512 frame
`emotion_v1.onnx`, `wav2vec_v1.onnx`, `refenc_v1.onnx`, `direction_v1.onnx`	TensorRT FP16	cosine ≥ 0.99999

backends.generated.yaml is the measured plan for the machine the conversion ran on (an RTX A6000): emotion, audio and fmt on onnx/trt, refenc compiled, direction and decoder eager. Re-run the selector on your own box to re-measure — the choices are per-GPU:

git clone https://github.com/mohamed-em2m/float-fast && cd float-fast
uv run --frozen --no-sync python -m fast_float.onnx.select_backends \
    --onnx-dir ./onnx --out ./onnx/backends.generated.yaml --ckpt ./checkpoints/float.pth

Two things are deliberately not in this folder:

the TensorRT engine cache — engines are tied to the GPU and the TensorRT/CUDA versions, so the target box builds its own on first use (and caches them under onnx/trt_cache/);
a runtime config you can trust blindly — see the selector above.

Using the graphs with FLOAT Fast

# 1. fetch the checkpoints and the ONNX graphs
huggingface-cli download mohamedemam/float --include "onnx/*" --local-dir ./hf
huggingface-cli download mohamedemam/float --include "float.pth" --local-dir ./checkpoints
huggingface-cli download mohamedemam/float --include "wav2vec2-base-960h/*" --local-dir ./checkpoints
huggingface-cli download mohamedemam/float --include "wav2vec-english-speech-emotion-recognition/*" --local-dir ./checkpoints
cp -r hf/onnx ./onnx

# 2. build the environment (uv venv, deps, ONNX stack, TensorRT 10.9)
./scripts/setup.sh

# 3. re-measure the backends for this box, then serve
uv run --frozen --no-sync python -m fast_float.onnx.select_backends \
    --onnx-dir ./onnx --out ./onnx/backends.generated.yaml --ckpt ./checkpoints/float.pth
./scripts/run_all.sh        # backend :8000 + Studio frontend :3000

FLOAT Fast then serves the model over HTTP:

curl -X POST http://localhost:8000/generate \
  -H 'Content-Type: application/json' \
  -d '{"audio_base64": "'"$(base64 -w0 audio.wav)"'",
       "audio_format": "wav",
       "image_base64": "'"$(base64 -w0 portrait.jpg)"'"}'

It also streams the frames live over a WebSocket (/ws/stream/{request_id}) and over WebRTC (/offer), so a browser can play the portrait while the MP4 is still being written.

Generation Results

Result 1	Result 2

Result 3	Result 4

Our method runs faster than current diffusion-based methods with fewer sampling steps and lower memory cost. For more details, please refer to the paper.

Updates

[2026.10.21] 🧩 The onnx/ folder now hosts the pre-converted ONNX / TensorRT graphs for the FLOAT Fast serving stack (FMT step, emotion, audio, frame decoder, reference encoder, motion direction), each gated against the torch model before publication.
[2025.06.26] 🎉🎉 FLOAT is accepted to ICCV 2025 🎉🎉
[2025.02.17] The inference code and checkpoints are released under a Non-commercial License
[2024.12.03] Selected as a HuggingFace Daily Papers on December 3, 2024
[2024.12.02] The paper is publicly available on ArXiv

Getting started

Requirements

# The original FLOAT environment
conda create -n FLOAT python=3.8.5
conda activate FLOAT
sh environments.sh

# or manual installation
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

Tested on Linux with A100 and V100 GPUs.
The FLOAT Fast stack needs a newer stack (Python 3.10, PyTorch 2.7.1+cu126, ONNX Runtime 1.23, optionally TensorRT 10.9).

Preparing the checkpoints

Download the checkpoints — either from this repository:
```
huggingface-cli download mohamedemam/float --local-dir ./checkpoints
```
or from the original Google Drive (the FLOAT Fast scripts do this automatically).

The checkpoints should be organised as follows:

./checkpoints
|-- checkpoints_here
|-- float.pth                            # main model
|-- wav2vec2-base-960h/                  # audio encoder
|   |-- config.json
|   |-- feature_extractor_config.json
|   |-- model.safetensors
|   |-- preprocessor_config.json
|   |-- pytorch_model.bin
|   |-- special_tokens_map.json
|   |-- tokenizer_config.json
|   '-- vocab.json
'-- wav2vec-english-speech-emotion-recognition/   # emotion encoder
    |-- config.json
    |-- preprocessor_config.json
    |-- pytorch_model.bin
    '-- training_args.bin

The W2V based models come from wav2vec2-base-960h and wav2vec-english-speech-emotion-recognition.

Generating a talking portrait from a single image and audio

For the best quality, please mind the pre-processing:

FLOAT is trained on frontal head pose distributions — a non-frontal image gives suboptimal results.

The performance of talking-portrait methods depends on their training-time pre-processing (e.g. field of view). The inference code includes automatic face cropping, which may introduce black padding regions; it can be disabled in generate.py at the cost of quality.

Heavy background music hurts the result — extract the vocals first with ClearVoice.

1. Generating video 1 (emotion from audio)

You can generate a video with an emotion derived from the audio without specifying --emo. Adjust the intensity with --e_cfg_scale (default 1); for more emotion-intensive video try 5–10.

CUDA_VISIBLE_DEVICES=0 python generate.py \
    --ref_path path/to/reference/image \
    --aud_path path/to/audio \
    --seed 15 \
    --a_cfg_scale 2 \
    --e_cfg_scale 1 \
    --ckpt_path ./checkpoints/float.pth \
    --no_crop                    # [optional] skip cropping

2. Generating video 2 (redirecting the emotion)

Pass --emo to pick one of the seven emotions: ['angry', 'disgust', 'fear', 'happy', 'neutral', 'sad', 'surprise'].

CUDA_VISIBLE_DEVICES=0 python generate.py \
    --ref_path path/to/reference/image \
    --aud_path path/to/audio \
    --emo 'happy' \
    --seed 15 \
    --a_cfg_scale 2 \
    --e_cfg_scale 1 \
    --ckpt_path ./checkpoints/float.pth \
    --no_crop

3. Running the example

CUDA_VISIBLE_DEVICES=0 python generate.py \
    --ref_path assets/sam_altman.webp \
    --aud_path assets/aud-sample-vs-1.wav \
    --seed 15 \
    --a_cfg_scale 2 \
    --e_cfg_scale 1 \
    --ckpt_path ./checkpoints/float.pth

Before Crop	After Crop	Result

Intended use and limitations

Research use only. This work is licensed under CC BY-NC-ND 4.0: no commercial use, and no derivative works.
The model is trained on frontal head poses and is driven by a single reference image and a speech track; it does not handle large head rotations, multi-person scenes or non-speech audio.
The emotion is either predicted from the audio or set explicitly to one of the seven basic emotions.
The onnx/ graphs are community-produced (by the FLOAT Fast project) and are provided as-is; they are numerically validated against the torch model, but the upstream authors do not support them.

❗ License ❗

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. You may not use this work for commercial purposes and may use it only for research purposes. For any commercial inquiries or collaboration opportunities, please contact daniel@deepbrain.io.

Development

This repository is a research demonstration implementation and is provided as a one-time code drop. For any research-related inquiries, please contact the first author Taekyung Ki. This work was done during the first author's South Korean Alternative Military Service at DeepBrain AI. This repository includes only the inference code; the training code will not be released.

Citation

@inproceedings{ki2025float,
  title={Float: Generative motion latent flow matching for audio-driven talking portrait},
  author={Ki, Taekyung and Min, Dongchan and Chae, Gyeongsu},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={14699--14710},
  year={2025}
}

Related works

Acknowledgements

The source images and audio are collected from the internet and other baselines, such as LIA, SadTalker, EMO, VASA-1, Hallo, LivePortrait, Loopy, and others. We appreciate their valuable contributions to this field. We employ the Wav2Vec2.0-based speech emotion recognizer by Rob Field. We appreciate this good work.

Downloads last month: -

Papers for mohamedemam/float

FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait

Paper • 2412.01064 • Published Dec 2, 2024 • 48

Learning to Generate Conditional Tri-plane for 3D-aware Expression Controllable Portrait Animation

Paper • 2404.00636 • Published Mar 31, 2024