sonealabs/dejashot-image-model

nomic-embed-vision v1.5, Core ML int8 (DejaShot)

This is a Core ML conversion of the image tower of nomic-ai/nomic-embed-vision-v1.5, with its weights quantized to 8 bits.

DejaShot, a Mac app, downloads it to find photos by describing them. It runs on the Mac and sends nothing anywhere.

The model's image vectors share a vector space with nomic-embed-text v1.5, so a text query can be compared directly with an image.

File

File	Size	SHA-256
nomic-embed-vision-v1.5.int8.mlpackage.zip	93,110,125 bytes	48b8546c28cf05a08855f185f8628a33df8604695b52c68c689261d94663cfb8

The zip holds one .mlpackage, a Core ML ML program. It requires macOS 14 or later.

Model

• Source: nomic-ai/nomic-embed-vision-v1.5 at revision e3a725bce72db07ca4adb1d83da08903f3ee02f8. The SHA-256 of each weight file was checked before conversion.
• Input: pixel_values, a float tensor of shape 1 × 3 × 224 × 224, prepared by the base model's own image processor (see its preprocessor_config.json).
• Output: embedding, a 768-number image vector normalized to length 1. It is the normalized CLS token, as in the base model.
• Precision:
  • The model was converted with fp16 compute.
  • Its weights were then quantized to int8 with coremltools' linear_quantize_weights (linear symmetric, per channel).
• Tooling: coremltools 9.0, torch 2.7.0, transformers 4.51.3.

For search, pair it with nomic-embed-text v1.5 and its search_query: prefix. Score a match by the cosine similarity between the text vector and the image vector.

Changes from the original

• Only the image tower was traced with PyTorch and converted to a Core ML ML program, with a fixed input shape.
• A tensor_split used by the rotary embedding was rewritten into operations coremltools can convert. The arithmetic is unchanged.
• The weights were quantized from fp16 to int8, per channel.
• No retraining or fine-tuning.

License and credit

The original model is nomic-embed-vision-v1.5 by Nomic AI, released under the Apache License 2.0. This conversion is released under the same license. See http://www.apache.org/licenses/LICENSE-2.0.

If you use it, please cite Nomic's paper:

@article{nussbaum2024nomicembedvision,
  title={Nomic Embed Vision: Expanding the Latent Space},
  author={Nussbaum, Zach and Duderstadt, Brandon and Mulyar, Andriy},
  journal={arXiv preprint arXiv:2406.18587},
  year={2024}
}

Converted by SoneaLabs for DejaShot.