nodevorg/siglip2-base-patch16-256-coreml

SigLIP2 base-patch16-256 — CoreML (fp16 + int8-embedding text variant)

CoreML .mlpackage conversion of google/siglip2-base-patch16-256 for on-device (macOS/Apple Silicon) text/image embedding — built for Alarian's semantic frame search.

Provenance: converted directly from the Hugging Face transformers reference implementation and coremltools, by nodevorg. Not affiliated with Google. The conversion script is public: tools/siglip2-coreml/convert.py (in the nodevorg/studio repo, kept uncommitted pending review — ask if you want a copy). This project intentionally never looked at or referenced any third-party GPLv3 conversion script; it was written from the HF transformers SigLIP2 modeling code and coremltools docs only.

License: apache-2.0, inherited from the base model. Same terms apply here — free for commercial and closed-source use.

What's in this repo

file	what it is
ImageEncoder.mlpackage.zip	image tower → 768-dim embedding
TextEncoder.int8emb.mlpackage.zip	recommended — text tower → 768-dim embedding; token-embedding table int8, everything else fp16 (36% smaller, parity ≥0.9996)
TextEncoder.mlpackage.zip	text tower → 768-dim embedding, full fp16 (use only if you need max precision and don't care about download size)
tokenizer.json	Gemma BPE tokenizer (256k vocab), verbatim from the source repo
tokenizer_config.json	tokenizer config (padding_side=right, do_lower_case=true), verbatim
special_tokens_map.json	special tokens, verbatim

For app downloads, use TextEncoder.int8emb.mlpackage.zip. Same inputs/ outputs as TextEncoder.mlpackage (see below) — it's a drop-in replacement, 323 MiB instead of 506 MiB, with only the 256k-row token-embedding table quantized to int8 (linear_symmetric, per-channel/per-token-row scale); attention and MLP weights stay fp16. Measured parity cost: worst-case cosine similarity 0.9996 vs the fp16 model's 0.999999 — both pass the >0.99 gate by a wide margin, and text↔image rankings are unaffected (see the parity section below).

A note on architecture

google/siglip2-base-patch16-256's own config.json self-describes as model_type: siglip (not siglip2) — at this fixed-256-resolution, non-NaFlex tier, the SigLIP2-trained weights are architecturally identical to SigLIP v1 (transformers.SiglipModel); only the weights and the tokenizer (Gemma BPE, 256k vocab, vs. SigLIP v1's 32k SentencePiece) differ. AutoModel.from_pretrained resolves this checkpoint to SiglipModel, and that's what was converted.

Inputs / outputs

ImageEncoder.mlpackage

• Input: pixel_values — a CoreML ImageType, RGB, 256×256. The exact SigLIP preprocessing (SiglipImageProcessor defaults for this checkpoint: rescale_factor=1/255, image_mean=[.5,.5,.5], image_std=[.5,.5,.5]) is baked into the model as scale=1/127.5, bias=[-1,-1,-1] — i.e. the model itself maps raw RGB [0,255] to [-1, 1]. Just resize/squash your image to 256×256 and hand it the raw RGB pixels; no separate normalization step needed on the caller side. (Aspect-ratio squash, not center-crop — the model saw squashed images in training.)
• Output: image_embedding — 768-dim float16, already L2-normalized.

TextEncoder.mlpackage

• Input: input_ids — 1×64 int32 token ids. No attention_mask input: SigLIP's text tower trains with unmasked bidirectional attention over the full fixed-length 64-token sequence and pools the hidden state at the last sequence position (SiglipTextModel.forward, not an EOS-aware pool) — passing a mask would deviate from training. Tokenize with the included tokenizer.json/tokenizer_config.json using padding="max_length", max_length=64, and lowercase the input text (do_lower_case=True in the tokenizer config; the tokenizer will also do this itself if you feed it through its normalizer, but be aware if you hand-roll token ids).
• Output: text_embedding — 768-dim float16, already L2-normalized.

TextEncoder.int8emb.mlpackage has the exact same input/output names, shapes, and dtypes — only the internal token-embedding weight is quantized, which is invisible from the outside.

Both encoders normalize their pooled output inside the graph, so cosine similarity between an image and text embedding is just a dot product.

Conversion details

• minimum_deployment_target: macOS 15
• compute_precision: ct.precision.FLOAT16
• convert_to: mlprogram
• Traced via torch.jit.trace on thin wrapper modules around SiglipModel.vision_model / .text_model (not the full contrastive model — each .mlpackage is just the one branch).
• Pinned toolchain (see the sha256 table below for exact artifact hashes): torch==2.7.0, transformers==5.14.1, coremltools==9.0, numpy==2.3.5. Gotcha: numpy>=2.4.0 breaks coremltools==9.0's PyTorch frontend with TypeError: only 0-dimensional arrays can be converted to Python scalars on any model using torch.nn.MultiheadAttention (SigLIP's vision pooling head hits this) — confirmed via apple/coremltools#2633; pin numpy<2.4.0 until a coremltools release ships the fix from #2632.

int8-embedding quantization (TextEncoder.int8emb.mlpackage)

Built by post-training-quantizing the token-embedding weight of the already- converted fp16 TextEncoder.mlpackage, via coremltools.optimize.coreml: OpLinearQuantizerConfig(mode="linear_symmetric", dtype="int8", granularity="per_channel") applied to only the text_model.embeddings.token_embedding weight (256000×768) by op name — OptimizationConfig(global_config=None, op_name_configs={...}) so every other weight (attention, MLP) is left untouched at fp16. This is a gather weight: each forward pass reads exactly one row per token, so quantization error doesn't compound the way it would through a matmul chain, which is why it tolerates int8 so well.

A whole-model int8 pass (global_config instead of a single op_name_config) was also built and measured for comparison: smaller (≈270 MB unzipped vs ≈352 MB for embedding-only) but with visibly worse worst-case parity (0.9964 vs 0.9996 cosine). Embedding-only was shipped since it's the clearly-better size/parity trade-off — the extra ~30% file size buys back a large chunk of the precision the whole-model pass gives up, for a component (search ranking) where score margins matter.

Parity (PyTorch fp32 reference vs. converted CoreML, cosine similarity)

4 test images (solid red / green / blue squares + a synthetic sunset gradient) × 4 test strings, L2-normalized before comparison. Threshold: > 0.99 per item.

item	cosine(pytorch, coreml)
image: solid_red	0.9999988
image: solid_green	0.9999992
image: solid_blue	0.9999992
image: gradient_sky	0.9999987
text: "a red square"	0.9999995
text: "a photo of a sunset over the ocean"	0.9999997
text: "a solid blue color"	0.9999995
text: "a green field"	0.9999996

Worst per-item cosine similarity (fp16 text encoder): 0.9999987 — well above the 0.99 threshold.

Text↔image ranking also matches exactly between PyTorch and CoreML across all 4×4 pairs (e.g. "a red square" ranks solid_red highest in both backends; "a photo of a sunset over the ocean" ranks gradient_sky highest in both).

int8-embedding text encoder parity

Same fixture (4 images × 4 strings), same gate (>0.99 per item, unchanged rankings), reference is still PyTorch fp32:

text	cosine(pytorch, int8emb coreml)
"a red square"	0.99978
"a photo of a sunset over the ocean"	0.99976
"a solid blue color"	0.99962
"a green field"	0.99975

Worst per-item cosine similarity (int8-embedding text encoder): 0.99962 — passes the 0.99 gate with a wide margin. Text↔image rankings are identical to both the PyTorch reference and the fp16 CoreML model across all 4 queries (each color string still top-matches its own solid-color image; the sunset string still top-matches the gradient).

sha256

file	sha256
ImageEncoder.mlpackage.zip	406938456c8f8e91f01632cb74131fdfb993064d8d0bc8f6fbbd142709f1ba66
TextEncoder.int8emb.mlpackage.zip	11471872101a6ca88dd51ed9de668b728ae9d09c0f853aeac8a6cc96f1c109d6
TextEncoder.mlpackage.zip	4f31f38a1729a0044f0ffc95a940dc1a6a2fdcd5fcf78a03cbb662bada6cbafb
tokenizer.json	cb9140fae3ac5122c972d37adf83e1248471a38147ad76f8215c8872c6fd8322
tokenizer_config.json	14afe629fe4959b9e0d51e1852b8d9f7ad074f90a1a7125a4fcdd17f06e78fc8
special_tokens_map.json	baec30ea10906f16adb8c18af7a34023002c1746542612b8b41c9f09e1351351

File sizes

file	size
ImageEncoder.mlpackage.zip	163 MiB
TextEncoder.int8emb.mlpackage.zip	323 MiB (36% smaller than the fp16 text encoder)
TextEncoder.mlpackage.zip	506 MiB
tokenizer.json	33 MiB

The text encoder is large mostly because of the Gemma tokenizer's 256k-entry vocabulary embedding table (256000 × 768 × 2 bytes ≈ 375 MiB alone, fp16; ≈188 MiB at int8, which is most of where the int8-embedding variant's size savings come from).