houssemzaier/iris-scene-encoders

Iris scene encoders

The image towers of two CLIP-style models, converted to run on a phone: SigLIP 2 base (Google) and TinyCLIP 39M (Microsoft), each as a Core ML model for iOS and a LiteRT model for Android. The Iris app runs one of them on a camera frame now and then, and compares its embedding with text vectors computed ahead of time from short descriptions ("a person in a kitchen", "a person wearing a jacket or a coat"...). The closest descriptions give the conversation some context: where the user is, what they wear. The text towers stay on the computer that computes those vectors.

Files

File	Platform	What it is	Size	sha256
siglip2-base-patch16-224-coreml-int8-posemb-fp16-compiled.aar	iOS	SigLIP 2 base image encoder, compiled Core ML, int8 weights	83.5 MB	6ff9b3125634c464af6150347279f6b3ff075b751140cf96bd579b145a5d148d
siglip2-base-patch16-224-fp16.tflite	Android	SigLIP 2 base image encoder, LiteRT, fp16 weights	185.5 MB	34486bb151857b9c68582a8629a93983791eee6a1fad06d3487a55870e38c851
tinyclip-vit-39m-16-coreml-int8-compiled.aar	iOS	TinyCLIP ViT-39M/16 image encoder, compiled Core ML, int8 weights	35.0 MB	bf6bfbc18b90ad9bf95c5c60b8f4fe7bb8715782e64eae1d8e579e03ba16a022
tinyclip-vit-39m-16-int8.tflite	Android	TinyCLIP ViT-39M/16 image encoder, LiteRT, int8 weights	40.1 MB	1716c72db1b2bc2f6d3fa7e5fe59c492f659319cd98917b1ff7b379d83c2f014

Inputs and outputs

• Core ML (.aar): an Apple Archive (lzfse) holding one entry, SceneEncoder.mlmodelc, an ML Program compiled for iOS 17 with fp16 compute. Input image: 224 x 224 RGB, pixel values scaled by 1/255, the model's own normalization inside. Output embedding: Float32 [1, 768] for SigLIP 2, [1, 512] for TinyCLIP, L2-normalized. Unpack it with aa extract (or the AppleArchive framework) and run it through Vision (VNCoreMLRequest).
• LiteRT (.tflite): input [1, 224, 224, 3] float32 RGB, with MediaPipe image metadata (mean 0, std 255: the model's own normalization is inside the graph). Output: the embedding, 768 or 512 floats, L2-normalized. It runs in MediaPipe Tasks Vision's Image Embedder (1.1.1, CPU).
• Framing: SigLIP 2 takes the whole frame squeezed to 224 x 224, TinyCLIP the frame's centered square. Shrink in filtered steps or with an area filter: a single bilinear resize from 640 pixels costs accuracy.

How they were made

From google/siglip2-base-patch16-224 at 75de2d55ec2d0b4efc50b3e9ad70dba96a7b2fa2 and wkcn/TinyCLIP-ViT-39M-16-Text-19M-YFCC15M at 07a4b0bc751cb64fecd2b661c048c1dd98d69444, image tower only: SigLIP 2 keeps its attention-pooling head (92.9M parameters), TinyCLIP its visual projection (38.6M).

• Core ML: coremltools 9.0 with torch 2.7.0, ML Program, int8 per-channel weights. For SigLIP 2 the position embedding (1, 196, 768) stays in fp16: a single int8 scale for it cost two held-out pictures. Compiled with xcrun coremlcompiler 3600.25.1.
• LiteRT: litert-torch 0.10.0, AI Edge Quantizer 0.10.0 and the MediaPipe 0.10.21 metadata writer. SigLIP 2 keeps fp16 weights (int8 lost too much on this model), TinyCLIP has int8 dynamic-range weights.

These files are modified versions of the original models: converted and quantized as described above.

Checks

Cosine between each file's embedding and the original PyTorch model in fp32, on 111 pictures:

Model	Core ML	LiteRT, same pixels
SigLIP 2 base	median 0.9994, min 0.9980	min 0.999998
TinyCLIP 39M	median 0.9997, min 0.9992	min 0.997

On 80 held-out pictures of people (first-ranked activity among six), the converted files rank the right one as often as the originals: 65 to 67 for SigLIP 2 base, 66 for TinyCLIP. On an iPhone 16 Plus, a reading takes a median 14 ms (SigLIP 2) and 8 ms (TinyCLIP).

Use and limits

• Made for scene context: the place, the clothing and its colour, glasses or a hat. Not for identifying anyone, and not for inferring sensitive traits.
• The text vectors are not in this repository: compute them with the original text towers, at the same commits, for your own descriptions.

Licenses

• SigLIP 2 files: Apache License 2.0, from Google's SigLIP 2 (LICENSES/Apache-2.0.txt).
• TinyCLIP files: MIT License, Copyright (c) Microsoft Corporation (LICENSES/MIT-TinyCLIP.txt).