gustavecortal/apophenia-clip-vit-base-patch32

CLIP ViT-B/32 for apophenia

The CLIP ViT-B/32 files that apophenia runs in the browser with ONNX Runtime Web, on WebGPU or WebAssembly. They are Xenova's ONNX export of OpenAI's CLIP at 4-bit weights with 16-bit activations (q4f16), with two changes. The text model stores its token table in 8 bits, which makes it 47.3 MB instead of 72.5. A copy of the vision model reads 160-pixel pictures instead of 224, which halves its work per picture.

File	Source
onnx/vision_model_q4f16_160.onnx	Xenova's vision_model_q4f16.onnx for 160 × 160 input: the 7 × 7 grid of position embeddings resized to 5 × 5 (bilinear), the reshapes set to 26 tokens
onnx/vision_model_q4f16.onnx	Xenova's file, unchanged, which apophenia used before the 160-pixel copy
onnx/text_model_q4f16_t8.onnx	Xenova's text_model_q4f16.onnx, token table (49,408 × 512) in int8 with an fp16 scale per row
tokenizer.json, tokenizer_config.json	Xenova's files, unchanged

The token lookup becomes a Gather of int8 rows, a Cast to fp16 and a multiplication by the row's scale, all standard ONNX operators. For three test phrases, its text embeddings have a cosine of 0.9998 or more with the original's, in Node and in the browser on both backends. Over 3,600 transformed photographs and 16 phrases, the direction scores apophenia computes follow the original's with a correlation of 0.995 at worst, and follow fp32's as closely as the original does.

The 160-pixel model was not fine-tuned. Its direction scores follow the 224-pixel model's with a correlation of 0.79 at the median, yet from the same 1,200 candidates per photograph, a blind reviewer preferred its picks in 4 of 6 cases, with one tie. In the browser on WebAssembly it takes 45 ms per picture against 71 (Edge, Intel i5-14400F).

CLIP was released by OpenAI under the MIT licence (github.com/openai/CLIP).