grabbou/clip-vit-base-patch32-CoreAI

CLIP ViT-B/32 encoders for Core AI

openai/clip-vit-base-patch32 exported to Apple Core AI as one .aimodel with separate text and image encoders, so text and images embed on their own into the same space.

Apple's models/clip recipe traces one function that needs an image and a fixed batch of three prompts on every call, so this export uses export.py, based on it (coreai-torch 0.4.3, torch 2.13):

uv run --script export.py --output-dir exports --dtype float16

Folder	Contents	Size	Platforms
clip-vit-base-patch32/	clip-vit-base-patch32.aimodel with encode_text(input_ids, attention_mask) -> text_embeds ([batch, 77] int32) and encode_image(pixel_values) -> image_embeds ([batch, 3, 224, 224] float16, CLIP mean and std), batch 1–64, 512-dim L2-normalized embeddings. tokenizer/: the CLIP tokenizer	290 MB	iOS, macOS

Use

With @react-native-ai/core-ai, download clip-vit-base-patch32/ into app storage, then embed text with the AI SDK and images with the native API:

import { coreAI, CoreAI } from '@react-native-ai/core-ai'
import { cosineSimilarity, embedMany } from 'ai'

const model = coreAI.embeddingModel({
  id: 'clip-vit-base-patch32',
  source: { type: 'file', uri: pathToFolder },
})
const { embeddings } = await embedMany({ model, values: ['a photo of a cat', 'a photo of a car'] })
const { embeddings: [image] } = await model.embedImages([imageUri])

cosineSimilarity(embeddings[0], image) // higher for the matching caption

License

CLIP is MIT licensed. This repository only converts it to Core AI format.