DiffuEraser ONNX
ONNX builds of DiffuEraser, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each
precision is one folder holding the graphs and a small Python pipeline.py that does the rest (image preprocessing,
the frame loop, mask handling); what the graphs take and return is under How to run.
Files
The last column is the benchmark against the original PyTorch model (details under How to run): how closely each output frame matches the original's on the same clips and masks.
| Precision | File | Size | sha256 | Benchmark |
|---|---|---|---|---|
| fp16 | fp16/brushnet.onnx |
0.8 MB | b097784e53a9 |
inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s) |
| fp16 | fp16/brushnet.onnx.data |
1.77 GB | 98ec3773cc7f |
inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s) |
| fp16 | fp16/pipeline.py |
13.0 kB | bae356dc2287 |
inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s) |
| fp16 | fp16/text_embeds.npy |
0.2 MB | 5aef4d79fd6b |
inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s) |
| fp16 | fp16/unet.onnx |
1.8 MB | 8f36ecc2a258 |
inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s) |
| fp16 | fp16/unet.onnx.data |
2.63 GB | 5179ab239ca4 |
inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s) |
| fp16 | fp16/vae_decoder.onnx |
99.1 MB | 4c4e4582fc9c |
inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s) |
| fp16 | fp16/vae_encoder.onnx |
68.4 MB | 58d65b351143 |
inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s) |
| int8 | int8/brushnet.onnx |
0.8 MB | 199ed052addd |
MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA) |
| int8 | int8/brushnet.onnx.data |
1.55 GB | deef2c36bc2d |
MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA) |
| int8 | int8/pipeline.py |
13.0 kB | bae356dc2287 |
MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA) |
| int8 | int8/text_embeds.npy |
0.2 MB | 5aef4d79fd6b |
MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA) |
| int8 | int8/unet.onnx |
1.9 MB | f71b146c146e |
MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA) |
| int8 | int8/unet.onnx.data |
1.97 GB | c9cbbd1007f8 |
MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA) |
| int8 | int8/vae_decoder.onnx |
99.1 MB | 4c4e4582fc9c |
MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA) |
| int8 | int8/vae_encoder.onnx |
68.4 MB | 58d65b351143 |
MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA) |
How to run
DiffuEraser (Alibaba Tongyi Lab: diffusion video inpainting and object removal from per-frame masks) as four ONNX graphs
plus pipeline.py, a numpy runtime that reproduces the repository's DiffuEraser.forward around them, with
run_diffueraser.py's settings: the PCM 2-step LoRA, guidance 0, mask dilation 8.
DiffuEraser refines a prior: the video as a first-pass inpainter left it. The repository makes the prior with ProPainter and passes it in, and so does this runtime. ProPainter's ONNX builds are at moonbox-dev/ProPainter-ONNX.
Licences.
- DiffuEraser: the weights are Apache-2.0.
- ProPainter: the prior model it expects is non-commercial (S-Lab License 1.0).
- Stable Diffusion 1.5: two parts of the pipeline come from it.
text_embeds.npy: SD1.5's CLIP text encoder output for the empty prompt (the only prompt DiffuEraser uses), under the CreativeML OpenRAIL-M license.- The VAE: sd-vae-ft-mse, MIT.
- PCM LoRA: the 2-step LoRA merged into the UNet comes from wangfuyun/PCM_Weights. Its card states no licence; the PCM code repository is Apache-2.0.
The graphs.
vae_encoder.onnx:image[N, 3, H, W] in [-1, 1] ->moments[N, 8, H/8, W/8] (mean and log-variance;pipeline.pysamples the latent).vae_decoder.onnx:latents[N, 4, H/8, W/8], already divided by 0.18215 ->image[N, 3, H, W].brushnet.onnx:sample[F, 4, h, w],timestep[1],text[F, 77, 768],cond[F, 5, h, w] -> 28 residuals (down_0..down_11,mid,up_0..up_14).condis the masked frames' latents plus the mask.unet.onnx: the motion UNet with the PCM LoRA merged. It takes the samesample,timestepandtext[1, 77, 768] plus the residuals, and returnsnoise[22, 4, h, w]. It always runs on clips of 22 frames, the length the model works at.
Frame sizes can be any multiple of 8.
pipeline.py reproduces the original's steps:
- Frames: resized so the long side is at most
max_size(the script uses 960), both sides multiples of 8. - Masks: eroded once, then dilated 8 times.
- Priors: the prior frames are VAE-encoded and noised.
- Pre-inference: videos over 44 frames first get a pass on 22 frames spread across them.
- Denoising: two TCD steps over windows of 22 frames that overlap by 5 (shifted windows on the second step), with the original's overlap blending.
- Output: decoded and composited into the input frames through a blurred mask.
It needs numpy, scipy, onnxruntime-gpu and Pillow. OpenCV (optional) makes the mask blur match the original exactly.
import sys
sys.path.insert(0, "fp16")
import pipeline
de = pipeline.DiffuEraser("fp16", "cuda")
# frames, masks, priors: lists of uint8 RGB arrays (masks white = remove; priors = the ProPainter output)
out = de(frames, masks, priors, max_size=960, seed=42)
Or from the command line: python fp16/pipeline.py frames/ masks/ priors/ out/ --cuda.
Running it.
- Inputs: at least 22 frames.
- Memory: at 640 x 352 the whole pipeline peaked at about 12.7 GB of system memory on Windows, where GPU allocations count toward it. GPU memory use is in the same range.
- Execution provider: the attention runs as ONNX Runtime's
com.microsoft::MultiHeadAttention, which uses memory-efficient kernels on CUDA. Plain attention would hold 22 x 8 x (h x w)² scores per layer, 14 GB at 848 x 480. CUDA is the tested path.
Precisions.
fp16/: every graph fp16. BrushNet and the UNet were exported from the fp16 models, as the original runs them. The VAE graphs do fp16 math with fp32 inputs / outputs.int8/: the BrushNet and UNet MatMuls 8-bit MatMulNBits (block 32), covering 27 % and 53 % of their weights. Their convolutions stay fp16, and the VAE is as infp16/.- No int4: the inpainted region fell to 27 dB on one clip, against the 30 dB pass mark, to save 0.5 GB.
- No bf16: fp16 passes.
Benchmark.
- Clips: the repository's example clips 1 and 2 (70 and 80 frames, 854 x 480, 15 fps) with their masks, and the prior the repository's ProPainter wrote for them.
- Original:
DiffuEraser.forwardin fp16 on CUDA, seed 42, atmax_img_size640 (640 x 352). The script's 960 needs more memory than the test machine allows. - Same draws: every random number the original drew (initial noise, each VAE latent sample) was recorded and replayed into the ONNX pipeline, so both sides see the same draws.
- Score: PSNR of every output frame against the original's, over the dilated mask, i.e. the inpainted pixels. Mean over frames, and the worst frame.
- Pass mark: at least 30 dB on every frame.
- Timing: seconds per clip on an RTX 3080 Laptop GPU (16 GB), DiffuEraser only. The prior took about 20 s more.
| inpainted region, clip 1: mean / worst | clip 2: mean / worst | seconds per clip | |
|---|---|---|---|
| original (PyTorch fp16, CUDA) | - | - | 44 / 47 |
fp16/ (CUDA) |
57.0 / 52.4 dB | 55.0 / 54.1 dB | 67 / 73 |
int8/ (CUDA) |
53.9 / 48.6 dB | 49.0 / 48.0 dB | 67 / 73 |
Both builds remove the same objects as the original, and side by side their frames look the same. The ONNX runtime is slower than the original on this GPU.
Before the benchmark, each graph was checked against its PyTorch module, at a frame size other than the one it was exported at (848 x 480; largest absolute difference):
- VAE encoder: 7e-3 of 28 (float32).
- VAE decoder: 6e-5 of 0.57 (float32).
- BrushNet: 0.016 of 11.7 (fp16 against fp16).
- UNet: 0.009 of 5.1 (fp16 against fp16).
Running pipeline.py's loop with the original PyTorch modules matches the original's frames to 58 / 56 dB.
How these were made
From DiffuEraser at revision ad510dca07. The graphs were exported from the original PyTorch modules with
torch.onnx (opset 17 / 18) and checked against them. Weight-only quantization: int8 is a per-output-channel int8
weight behind DequantizeLinear, int4 is ONNX Runtime's MatMulNBits (block 32). Each card above says which precision
every graph uses. Every precision was benchmarked against the original before upload (the pass mark is under How to run), and the
sha256 column above is what the files must hash to.
License
Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.
Original model card
DiffuEraser: A Diffusion Model for Video Inpainting
DiffuEraser is a diffusion model for video inpainting, which outperforms state-of-the-art model Propainter in both content completeness and temporal consistency while maintaining acceptable efficiency.
Model Sources
- Project Page: https://lixiaowen-xw.github.io/DiffuEraser-page
- Repository: https://github.com/lixiaowen-xw/DiffuEraser.git
- Paper: https://arxiv.org/abs/2501.10018
Citation
BibTeX:
@misc{li2025diffueraserdiffusionmodelvideo,
title={DiffuEraser: A Diffusion Model for Video Inpainting},
author={Xiaowen Li and Haolan Xue and Peiran Ren and Liefeng Bo},
year={2025},
eprint={2501.10018},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2501.10018},
}
Model Card Authors
Xiaowen Li / xwlxw@qq.com / lxw262398@alibaba-inc.com
Model tree for moonbox-dev/diffuEraser-ONNX
Base model
lixiaowen/diffuEraser