DiffuEraser ONNX

ONNX builds of DiffuEraser, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each precision is one folder holding the graphs and a small Python pipeline.py that does the rest (image preprocessing, the frame loop, mask handling); what the graphs take and return is under How to run.

Files

The last column is the benchmark against the original PyTorch model (details under How to run): how closely each output frame matches the original's on the same clips and masks.

Precision File Size sha256 Benchmark
fp16 fp16/brushnet.onnx 0.8 MB b097784e53a9 inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s)
fp16 fp16/brushnet.onnx.data 1.77 GB 98ec3773cc7f inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s)
fp16 fp16/pipeline.py 13.0 kB bae356dc2287 inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s)
fp16 fp16/text_embeds.npy 0.2 MB 5aef4d79fd6b inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s)
fp16 fp16/unet.onnx 1.8 MB 8f36ecc2a258 inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s)
fp16 fp16/unet.onnx.data 2.63 GB 5179ab239ca4 inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s)
fp16 fp16/vae_decoder.onnx 99.1 MB 4c4e4582fc9c inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s)
fp16 fp16/vae_encoder.onnx 68.4 MB 58d65b351143 inpainted-region PSNR vs the original: mean 57.0 / 55.0 dB, worst frame 52.4 dB; 67 / 73 s per clip (CUDA; original 44 / 47 s)
int8 int8/brushnet.onnx 0.8 MB 199ed052addd MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA)
int8 int8/brushnet.onnx.data 1.55 GB deef2c36bc2d MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA)
int8 int8/pipeline.py 13.0 kB bae356dc2287 MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA)
int8 int8/text_embeds.npy 0.2 MB 5aef4d79fd6b MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA)
int8 int8/unet.onnx 1.9 MB f71b146c146e MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA)
int8 int8/unet.onnx.data 1.97 GB c9cbbd1007f8 MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA)
int8 int8/vae_decoder.onnx 99.1 MB 4c4e4582fc9c MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA)
int8 int8/vae_encoder.onnx 68.4 MB 58d65b351143 MatMuls 8-bit, convolutions fp16: inpainted-region PSNR mean 53.9 / 49.0 dB, worst frame 48.0 dB; 67 / 73 s per clip (CUDA)

How to run

DiffuEraser (Alibaba Tongyi Lab: diffusion video inpainting and object removal from per-frame masks) as four ONNX graphs plus pipeline.py, a numpy runtime that reproduces the repository's DiffuEraser.forward around them, with run_diffueraser.py's settings: the PCM 2-step LoRA, guidance 0, mask dilation 8.

DiffuEraser refines a prior: the video as a first-pass inpainter left it. The repository makes the prior with ProPainter and passes it in, and so does this runtime. ProPainter's ONNX builds are at moonbox-dev/ProPainter-ONNX.

Licences.

  • DiffuEraser: the weights are Apache-2.0.
  • ProPainter: the prior model it expects is non-commercial (S-Lab License 1.0).
  • Stable Diffusion 1.5: two parts of the pipeline come from it.
    • text_embeds.npy: SD1.5's CLIP text encoder output for the empty prompt (the only prompt DiffuEraser uses), under the CreativeML OpenRAIL-M license.
    • The VAE: sd-vae-ft-mse, MIT.
  • PCM LoRA: the 2-step LoRA merged into the UNet comes from wangfuyun/PCM_Weights. Its card states no licence; the PCM code repository is Apache-2.0.

The graphs.

  • vae_encoder.onnx: image [N, 3, H, W] in [-1, 1] -> moments [N, 8, H/8, W/8] (mean and log-variance; pipeline.py samples the latent).
  • vae_decoder.onnx: latents [N, 4, H/8, W/8], already divided by 0.18215 -> image [N, 3, H, W].
  • brushnet.onnx: sample [F, 4, h, w], timestep [1], text [F, 77, 768], cond [F, 5, h, w] -> 28 residuals (down_0..down_11, mid, up_0..up_14). cond is the masked frames' latents plus the mask.
  • unet.onnx: the motion UNet with the PCM LoRA merged. It takes the same sample, timestep and text [1, 77, 768] plus the residuals, and returns noise [22, 4, h, w]. It always runs on clips of 22 frames, the length the model works at.

Frame sizes can be any multiple of 8.

pipeline.py reproduces the original's steps:

  • Frames: resized so the long side is at most max_size (the script uses 960), both sides multiples of 8.
  • Masks: eroded once, then dilated 8 times.
  • Priors: the prior frames are VAE-encoded and noised.
  • Pre-inference: videos over 44 frames first get a pass on 22 frames spread across them.
  • Denoising: two TCD steps over windows of 22 frames that overlap by 5 (shifted windows on the second step), with the original's overlap blending.
  • Output: decoded and composited into the input frames through a blurred mask.

It needs numpy, scipy, onnxruntime-gpu and Pillow. OpenCV (optional) makes the mask blur match the original exactly.

import sys
sys.path.insert(0, "fp16")
import pipeline
de = pipeline.DiffuEraser("fp16", "cuda")
# frames, masks, priors: lists of uint8 RGB arrays (masks white = remove; priors = the ProPainter output)
out = de(frames, masks, priors, max_size=960, seed=42)

Or from the command line: python fp16/pipeline.py frames/ masks/ priors/ out/ --cuda.

Running it.

  • Inputs: at least 22 frames.
  • Memory: at 640 x 352 the whole pipeline peaked at about 12.7 GB of system memory on Windows, where GPU allocations count toward it. GPU memory use is in the same range.
  • Execution provider: the attention runs as ONNX Runtime's com.microsoft::MultiHeadAttention, which uses memory-efficient kernels on CUDA. Plain attention would hold 22 x 8 x (h x w)² scores per layer, 14 GB at 848 x 480. CUDA is the tested path.

Precisions.

  • fp16/: every graph fp16. BrushNet and the UNet were exported from the fp16 models, as the original runs them. The VAE graphs do fp16 math with fp32 inputs / outputs.
  • int8/: the BrushNet and UNet MatMuls 8-bit MatMulNBits (block 32), covering 27 % and 53 % of their weights. Their convolutions stay fp16, and the VAE is as in fp16/.
  • No int4: the inpainted region fell to 27 dB on one clip, against the 30 dB pass mark, to save 0.5 GB.
  • No bf16: fp16 passes.

Benchmark.

  • Clips: the repository's example clips 1 and 2 (70 and 80 frames, 854 x 480, 15 fps) with their masks, and the prior the repository's ProPainter wrote for them.
  • Original: DiffuEraser.forward in fp16 on CUDA, seed 42, at max_img_size 640 (640 x 352). The script's 960 needs more memory than the test machine allows.
  • Same draws: every random number the original drew (initial noise, each VAE latent sample) was recorded and replayed into the ONNX pipeline, so both sides see the same draws.
  • Score: PSNR of every output frame against the original's, over the dilated mask, i.e. the inpainted pixels. Mean over frames, and the worst frame.
  • Pass mark: at least 30 dB on every frame.
  • Timing: seconds per clip on an RTX 3080 Laptop GPU (16 GB), DiffuEraser only. The prior took about 20 s more.
inpainted region, clip 1: mean / worst clip 2: mean / worst seconds per clip
original (PyTorch fp16, CUDA) - - 44 / 47
fp16/ (CUDA) 57.0 / 52.4 dB 55.0 / 54.1 dB 67 / 73
int8/ (CUDA) 53.9 / 48.6 dB 49.0 / 48.0 dB 67 / 73

Both builds remove the same objects as the original, and side by side their frames look the same. The ONNX runtime is slower than the original on this GPU.

Before the benchmark, each graph was checked against its PyTorch module, at a frame size other than the one it was exported at (848 x 480; largest absolute difference):

  • VAE encoder: 7e-3 of 28 (float32).
  • VAE decoder: 6e-5 of 0.57 (float32).
  • BrushNet: 0.016 of 11.7 (fp16 against fp16).
  • UNet: 0.009 of 5.1 (fp16 against fp16).

Running pipeline.py's loop with the original PyTorch modules matches the original's frames to 58 / 56 dB.

How these were made

From DiffuEraser at revision ad510dca07. The graphs were exported from the original PyTorch modules with torch.onnx (opset 17 / 18) and checked against them. Weight-only quantization: int8 is a per-output-channel int8 weight behind DequantizeLinear, int4 is ONNX Runtime's MatMulNBits (block 32). Each card above says which precision every graph uses. Every precision was benchmarked against the original before upload (the pass mark is under How to run), and the sha256 column above is what the files must hash to.

License

Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.


Original model card

DiffuEraser: A Diffusion Model for Video Inpainting

DiffuEraser is a diffusion model for video inpainting, which outperforms state-of-the-art model Propainter in both content completeness and temporal consistency while maintaining acceptable efficiency.

Model Sources

Citation

BibTeX:

@misc{li2025diffueraserdiffusionmodelvideo,
  title={DiffuEraser: A Diffusion Model for Video Inpainting}, 
  author={Xiaowen Li and Haolan Xue and Peiran Ren and Liefeng Bo},
  year={2025},
  eprint={2501.10018},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2501.10018}, 
}

Model Card Authors

Xiaowen Li / xwlxw@qq.com / lxw262398@alibaba-inc.com

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for moonbox-dev/diffuEraser-ONNX

Quantized
(1)
this model

Paper for moonbox-dev/diffuEraser-ONNX