parodiniccolo/sam3-dual-webgpu

SAM 3 image encoder for text and click prompts (ONNX, WebGPU)

One image encoder that feeds both SAM 3 decoders, so a browser reads each image once and serves text prompts and clicks:

• the text-prompt decoder of danilobukvic/sam3-text-onnx (revision fe8f8e1), and
• the point/box decoder of onnx-community/sam3-tracker-ONNX (revision 429305c, onnx/prompt_encoder_mask_decoder.onnx).

Files

• sam3_vision_dual_q4f16.onnx (+ .data, 311 MB): current. About 1.7 s per image on WebGPU (Apple M3 Max, Chrome).
• sam3_vision_dual_int4.onnx (+ .data, 321 MB): the first version, about 6.5 s per image. Kept for pinned revisions that use it.

How it was made

onnx-community/sam3-tracker-ONNX's vision_encoder_q4f16.onnx (SAM 3 ViT trunk, 4-bit weights, fp16 activations, plus the tracker neck), with the text detector's FPN neck from danilobukvic/sam3-text-onnx's vision_encoder_int4.onnx converted to fp16 and grafted onto the shared trunk output. The trunk weights are the same in both sources; the tracker's q4f16 export runs about 4.5x faster on WebGPU than the text export's int4 one. Input fixed to 1008×1008.

Contract

• Input: pixel_values float32 [1,3,1008,1008], RGB stretched to 1008×1008, px / 127.5 − 1.
• Outputs (float32): fpn_hidden_state_0..3 and fpn_position_encoding_0..3 (for the text decoder), and image_embeddings.0 [1,32,288,288], image_embeddings.1 [1,64,144,144], image_embeddings.2 [1,256,72,72] (for the tracker decoder).

Validation (onnxruntime CPU, one indoor still)

• Text path vs the text export's own encoder: best-mask IoU 0.994 ("round table") and 0.984 ("chair"), scores within 0.01.
• Click path vs the fp32 tracker: the larger masks agree at IoU 0.97–0.98, as with the stock q4f16 tracker.

Licence

The weights are Meta's SAM 3 and remain under the SAM License. This is not an official Meta release.