- EfficientViT-SAM ONNX
- Original model card
- Efficient Vision Foundation Models for High-Resolution Generation and Perception
- News
- Content
- [ICLR 2025] Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models [paper] [readme] [poster]
- [CVPR 2024 eLVM Workshop] EfficientViT-SAM: Accelerated Segment Anything Model Without Accuracy Loss [paper] [online demo] [readme]
- [ICCV 2023] EfficientViT-Classification [paper] [readme]
- [ICCV 2023] EfficientViT-Segmentation [paper] [readme]
- EfficientViT-GazeSAM [readme]
- Getting Started
- Third-Party Implementation/Integration
- Contact
- Reference
- News
EfficientViT-SAM ONNX
ONNX builds of EfficientViT-SAM, made by Moonbox for on-device use with ONNX Runtime (CPU, CUDA, DirectML). Each
precision is one folder holding the graphs and a small Python pipeline.py that does the rest (image preprocessing,
prompt coordinates, mask upsampling); what the graphs take and return is under How to run.
Files
The last column is the benchmark against the original PyTorch model (details under How to run): IoU of every predicted mask with the original's on the same pictures and prompts.
| Precision | File | Size | sha256 | Benchmark |
|---|---|---|---|---|
| fp32 | fp32/decoder.onnx |
0.3 MB | 72ff3fdc36d9 |
mask IoU 0.9999 mean / 0.9991 worst |
| fp32 | fp32/decoder.onnx.data |
16.2 MB | df2017f2c3fc |
mask IoU 0.9999 mean / 0.9991 worst |
| fp32 | fp32/encoder.onnx |
0.4 MB | 35d5b72c6f74 |
mask IoU 0.9999 mean / 0.9991 worst |
| fp32 | fp32/encoder.onnx.data |
229.0 MB | f5552293e854 |
mask IoU 0.9999 mean / 0.9991 worst |
| fp32 | fp32/pipeline.py |
4.2 kB | 3cee6dcd2a80 |
mask IoU 0.9999 mean / 0.9991 worst |
How to run
EfficientViT-SAM L2 (MIT Han Lab: SAM's prompt encoder and mask decoder on an EfficientViT-L2 image encoder at 512 px,
61 M parameters) as two ONNX graphs plus pipeline.py, a numpy runtime that reproduces the repository's
EfficientViTSamPredictor around them.
encoder.onnx: EfficientViT-L2 + the SAM neck:image[1, 3, 512, 512] (normalised, padded) ->image_embeddings[1, 256, 64, 64]. Run it once per image.decoder.onnx: SAM's ONNX decoder wrapper (SamOnnxModel, all 4 masks; prompts and masks in the 1024 frame):image_embeddings,point_coords[1, N, 2] andpoint_labels[1, N] (1 / 0 points, 2 / 3 box corners, -1 padding),mask_input[1, 1, 256, 256],has_mask_input[1],orig_im_size[2] ->masks[1, 4, H, W] (logits at the image size),iou_predictions[1, 4],low_res_masks[1, 4, 256, 256].
pipeline.py resizes the longest side to 512 (Pillow bilinear, as the repository's SamResize), applies SAM's pixel
mean / std and pads bottom-right to 512 x 512. It scales the prompt coordinates to the 1024 frame, and returns masks 1-3
for multimask point prompts and mask 0 for a box. It needs numpy, onnxruntime (or onnxruntime-gpu) and Pillow:
import sys, numpy as np
from PIL import Image
sys.path.insert(0, "fp32")
import pipeline
sam = pipeline.EfficientViTSAM("fp32", "cuda")
emb, size = sam.embed(np.array(Image.open("photo.jpg").convert("RGB")))
masks, scores = sam.point(emb, size, [[410, 300]]) # 3 masks [3, H, W] (bool)
box_mask, score = sam.box(emb, size, [120, 80, 700, 560])
Precisions. Only fp32/: the model doesn't survive 16 bits. 30 conv + BatchNorm pairs (every MBConv point conv of
stages 3-4, eight in the neck, the stem) have BatchNorm scales above 20, up to 602 (running variance 5.5e-7). Their
outputs are differences of terms about 1000x larger than the result. Measured here: fp16 (decoder fp32) worst mask IoU
0.12 (mean 0.89); weight-only int8 mean 0.33. Neither is published.
Benchmark. Eight photos (the first eight of the lab's DIS5K-250 set). Each gets a positive point (the ground-truth foreground pixel nearest its centroid; 3 masks) and the ground truth's bounding box (1 mask), so 32 masks per build. The original is the repository's EfficientViTSamPredictor in float32. The build gets the image it preprocesses and the same prompt coordinates (the conversion measured alone). Pass: IoU with the original's mask at least 0.95 on every mask.
| worst mask IoU | mean mask IoU | |
|---|---|---|
fp32/ |
0.9991 | 0.9999 |
Graphs checked against the original modules (largest absolute difference, float32): encoder 7.0e-5 of 0.58, decoder 2.3e-5 of 14.8 (masks).
How these were made
From EfficientViT-SAM at revision de7d7733cc. The graphs were exported from the original PyTorch modules with
torch.onnx (opset 17 / 18) and checked against them. Weight-only quantization: int8 is a per-output-channel int8
weight behind DequantizeLinear, int4 is ONNX Runtime's MatMulNBits (block 32). Each card above says which precision
every graph uses. Every precision was benchmarked against the original before upload (pass: IoU at least 0.95 on every
mask, otherwise marked below threshold), and the sha256 column above is what the files must hash to.
License
apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.
Original model card
Efficient Vision Foundation Models for High-Resolution Generation and Perception
News
- (🔥 New) [2025/09/05] We will no longer maintain this codebase. All future updates and announcements will be made on DC-Gen.
- (🔥 New) [2025/01/24] We released DC-AE-SANA-1.1: doc.
- (🔥 New) [2025/01/23] DC-AE and SANA are accepted by ICLR 2025.
- (🔥 New) [2025/01/14] We released DC-AE+USiT models: model, training. Using the default training settings and sampling strategy, DC-AE+USiT-2B achieves 1.72 FID on ImageNet 512x512, surpassing the SOTA diffusion model EDM2-XXL and SOTA auto-regressive image generative models (MAGVIT-v2 and MAR-L).
- (🔥 New) [2024/12/24] diffusers supports DC-AE models. All DC-AE models in diffusers safetensors are released. Usage.
- [2024/10/21] DC-AE and EfficientViT block are used in our latest text-to-image diffusion model SANA! Check the project page for more details.
- [2024/10/15] We released Deep Compression Autoencoder (DC-AE): link!
- [2024/07/10] EfficientViT is used as the backbone in Grounding DINO 1.5 Edge for efficient open-set object detection.
- [2024/07/10] EfficientViT-SAM is used in MedficientSAM, the 1st place model in CVPR 2024 Segment Anything In Medical Images On Laptop Challenge.
- [2024/04/06] EfficientViT-SAM is accepted by eLVM@CVPR'24.
- [2024/03/19] Online demo of EfficientViT-SAM is available: https://evitsam.hanlab.ai/.
- [2024/02/07] We released EfficientViT-SAM, the first accelerated SAM model that matches/outperforms SAM-ViT-H's zero-shot performance, delivering the SOTA performance-efficiency trade-off.
- [2023/11/20] EfficientViT is available in the NVIDIA Jetson Generative AI Lab.
- [2023/09/12] EfficientViT is highlighted by MIT home page and MIT News.
- [2023/07/18] EfficientViT is accepted by ICCV 2023.
Content
[ICLR 2025] Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models [paper] [readme] [poster]
Deep Compression Autoencoder (DC-AE) is a new family of high-spatial compression autoencoders with a spatial compression ratio of up to 128 while maintaining reconstruction quality. It accelerates all latent diffusion models regardless of the diffusion model architecture.
Demo
Figure 1: We address the reconstruction accuracy drop of high spatial-compression autoencoders.
Figure 2: DC-AE speeds up latent diffusion models.
Figure 3: DC-AE enables efficient text-to-image generation on the laptop: SANA.
- Usage of Deep Compression Autoencoder
- Usage of DC-AE-Diffusion
- Evaluate Deep Compression Autoencoder
- Demo DC-AE-Diffusion Models
- Evaluate DC-AE-Diffusion Models
- Train DC-AE-Diffusion Models
- Reference
[CVPR 2024 eLVM Workshop] EfficientViT-SAM: Accelerated Segment Anything Model Without Accuracy Loss [paper] [online demo] [readme]
EfficientViT-SAM is a new family of accelerated segment anything models by replacing SAM's heavy image encoder with EfficientViT. It delivers a 48.9x measured TensorRT speedup on A100 GPU over SAM-ViT-H without sacrificing accuracy.
- Pretrained EfficientViT-SAM Models
- Usage of EfficientViT-SAM
- Evaluate EfficientViT-SAM
- Visualize EfficientViT-SAM
- Deploy EfficientViT-SAM
- Train EfficientViT-SAM
- Reference
[ICCV 2023] EfficientViT-Classification [paper] [readme]
Efficient image classification models with EfficientViT backbones.
- Pretrained EfficientViT Classification Models
- Usage of EfficientViT Classification Models
- Evaluate EfficientViT Classification Models
- Export EfficientViT Classification Models
- Train EfficientViT Classification Models
- Reference
[ICCV 2023] EfficientViT-Segmentation [paper] [readme]
Efficient semantic segmantation models with EfficientViT backbones.
- Pretrained EfficientViT Segmentation Models
- Usage of EfficientViT Segmentation Models
- Evaluate EfficientViT Segmentation Models
- Visualize EfficientViT Segmentation Models
- Export EfficientViT Segmentation Models
- Reference
EfficientViT-GazeSAM [readme]
Gaze-prompted image segmentation models capable of running in real time with TensorRT on an NVIDIA RTX 4070.
Getting Started
conda create -n efficientvit python=3.10
conda activate efficientvit
pip install -U -r requirements.txt
Third-Party Implementation/Integration
Contact
Reference
If EfficientViT or EfficientViT-SAM or DC-AE is useful or relevant to your research, please kindly recognize our contributions by citing our paper:
@inproceedings{cai2023efficientvit,
title={Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction},
author={Cai, Han and Li, Junyan and Hu, Muyan and Gan, Chuang and Han, Song},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={17302--17313},
year={2023}
}
@article{zhang2024efficientvit,
title={EfficientViT-SAM: Accelerated Segment Anything Model Without Performance Loss},
author={Zhang, Zhuoyang and Cai, Han and Han, Song},
journal={arXiv preprint arXiv:2402.05008},
year={2024}
}
@article{chen2024deep,
title={Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models},
author={Chen, Junyu and Cai, Han and Chen, Junsong and Xie, Enze and Yang, Shang and Tang, Haotian and Li, Muyang and Lu, Yao and Han, Song},
journal={arXiv preprint arXiv:2410.10733},
year={2024}
}



