wW-Ww/sam3-ONNX

SAM 3 for ONNX Runtime on DirectML: clicks, boxes and words from one image reading

An ONNX export of Meta's SAM 3 for still photographs, made for Photool's object masks. It runs on ONNX Runtime with the DirectML execution provider (any DirectX 12 graphics card on Windows), and on CUDA and CPU.

Both of SAM 3's ways of choosing are here, and they share one reading of the image:

file	runs	inputs → outputs
vision_fp16.onnx	once per image (0.4 s on an RTX 3070)	pixel_values (1, 3, 1008, 1008) → features_288/144/72 for words, image_embeddings.0/1/2 for clicks
prompt_encoder_mask_decoder.onnx	per click or box (~10 ms)	image_embeddings.*, input_points, input_labels, input_boxes → iou_scores, pred_masks (3 candidates, 288²), object_score_logits
text_fp16.onnx	once per phrase (~5 ms)	input_ids, attention_mask (1, 32) → text_features
detector.onnx	per phrase (0.2 s)	features_*, text_features, text_mask → pred_logits, pred_boxes, presence_logits, pred_masks (200 queries, 288²), semantic_seg

Input: the image stretched to 1008 × 1008, RGB scaled to 0..1, then (x - 0.5) / 0.5. Words use CLIP's tokenizer (the openai/clip-vit-base-patch32 vocabulary), 32 tokens padded with the end token. An instance counts as found when sigmoid(pred_logits) * sigmoid(presence_logits) > 0.5.

How it was made

• Weights: Meta's SAM 3 checkpoint, in the Transformers Sam3VideoModel format.
• vision_fp16.onnx: export_shared.py here. It runs the Perception Encoder once, then both the detector's neck and the tracker's neck, with the tracker's conv_s0/conv_s1 projections and no_memory_embedding applied, as Sam3TrackerModel.get_image_embeddings does.
• text_fp16.onnx, detector.onnx: the graph split of CardAnvilModels/sam3-ONNX, exported again in fp32 for DirectML.
• prompt_encoder_mask_decoder.onnx: from onnx-community/sam3-tracker-ONNX, unchanged.
• For DirectML (fix_for_dml.py):
  • Reshape with allowzero=1 is rewritten without it, since none of the shapes holds a 0.
  • In the detector, a Slice that takes one item out of a Concat is replaced by that item.
• Vision and text are converted to fp16 with fp32 inputs and outputs (to_fp16.py), because the fp32 vision graph fills an 8 GB card. The detector stays fp32.

Checked against the fp32 PyTorch model on photographs:

• the detector's masks agree to a mean IoU of 0.995;
• the tracker embeddings are within 2.2 to 2.8 % (relative L2), as close as onnx-community's own fp16 encoder or closer.

License

SAM 3 is Meta's, released under the SAM License, a copy of which comes with these files as the license asks. By using or distributing these files you agree to its terms.