satorean/jina-clip-v2-split-onnx

Jina CLIP v2 split ONNX

Independent text and vision ONNX encoders derived from jinaai/jina-clip-v2.

File	Precision	Size	Input
jina-clip-v2-text.onnx	FP16	1.0 GB	input_ids: int64 [batch, sequence]
jina-clip-v2-vision.onnx	FP16	585 MB	pixel_values: float32 [batch, 3, 512, 512]
jina-clip-v2-text-int8.onnx	Dynamic INT8, optimized attention graph	540 MB	input_ids: int64 [batch, sequence]
jina-clip-v2-vision-int8.onnx	Dynamic INT8	301 MB	pixel_values: float32 [batch, 3, 512, 512]

Every model is self-contained, supports dynamic batch sizes, and returns normalized float32 embeddings named embeddings with shape [batch, 1024]. The text models also support dynamic sequence lengths.

Which precision should I use?

• CPU inference: use the dynamic INT8 models. They are substantially smaller and faster than FP16 on tested CPU runtimes. The INT8 text graph uses integer linear layers with FP32 attention activations and an optimized MatMul-based attention structure.
• GPU inference: use the base FP16 models. FP16 is designed for GPU acceleration, while the dynamic quantization operators in the INT8 artifacts primarily target CPU execution.

FP16 is not recommended for CPU inference. CPU providers may promote unsupported FP16 operations to FP32 or insert casts, increasing both latency and memory use.

Preprocessing and inference

Use the tokenizer and image processor from the base model:

import onnxruntime as ort
from transformers import AutoImageProcessor, AutoTokenizer

base_model = "jinaai/jina-clip-v2"
tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
image_processor = AutoImageProcessor.from_pretrained(base_model, trust_remote_code=True)

text_session = ort.InferenceSession("jina-clip-v2-text-int8.onnx")
tokens = tokenizer(["A photo of a cat"], padding=True, truncation=True, return_tensors="np")
text_embeddings = text_session.run(["embeddings"], {"input_ids": tokens.input_ids})[0]

# `images` is a list of PIL images.
vision_session = ort.InferenceSession("jina-clip-v2-vision-int8.onnx")
pixels = image_processor(images=images, return_tensors="np").pixel_values
image_embeddings = vision_session.run(["embeddings"], {"pixel_values": pixels})[0]

Matryoshka dimensions

The trained dimensions are 32, 64, 128, 256, 512, 768, and 1024. To reproduce the base model's truncate_dim=N behavior, truncate the prefix and normalize it again:

import numpy as np

embedding = embedding[:, :dimension]
embedding /= np.linalg.norm(embedding, axis=1, keepdims=True)

Validation

The dynamic INT8 files were compared directly with the original, unsplit jinaai/jina-clip-v2 model through its public encode_text and encode_image methods.

Dimension	Text mean cosine	Text minimum cosine	Vision cosine
32	0.9949	0.9933	0.9984
64	0.9951	0.9936	0.9985
128	0.9942	0.9930	0.9975
256	0.9929	0.9915	0.9964
512	0.9918	0.9907	0.9953
768	0.9913	0.9905	0.9946
1024	0.9911	0.9904	0.9943

Text validation used a multilingual batch. Vision validation used deterministic images and the base model's original image processor. INT8 outputs were finite and unit-normalized, and batching was verified at batch sizes 1 and 2.

Optimized INT8 text performance

The current jina-clip-v2-text-int8.onnx replaces the original attention Einsums with algebraically equivalent Transpose and MatMul operations. It remains bit-identical to the previous INT8 graph while giving ONNX Runtime a more efficient CPU execution structure.

The following measurements used ONNX Runtime 1.27 CPU on an 18-core Intel Core i9-7980XE. Inputs contained exactly 1024 real tokens with no padding. Times are warmed median inference times; peak RSS includes the model session.

Batch	Previous INT8 time	Current INT8 time	Speedup	Previous peak RSS	Current peak RSS
1	1.425 s	0.686 s	51.9%	1.26 GiB	1.29 GiB
2	2.033 s	1.517 s	25.4%	1.77 GiB	1.83 GiB
4	4.250 s	3.250 s	23.5%	2.78 GiB	2.93 GiB
8	9.379 s	6.966 s	25.7%	5.48 GiB	5.07 GiB
16	19.686 s	14.455 s	26.6%	10.09 GiB	9.32 GiB

The optimized model is consistently faster. Peak memory is slightly higher at small batches on this Linux runtime, but becomes lower at batches 8 and 16. On a 16 GB Apple Silicon machine, the optimized graph also completed [16, 1024] where the previous graph was killed under memory pressure. Memory behavior depends on the CPU provider and allocator.

Correctness checks covered random inputs at sequence lengths 1, 12, 77, 511, 1024, and 2048, multiple batch sizes, a padded multilingual batch, normalization, and all trained Matryoshka dimensions. Outputs were bit-identical to the previous INT8 graph. The text input remains dynamic and supports up to 8192 tokens, although ordinary full attention still has quadratic memory use.

License and attribution

This is a quantized and structurally split derivative of jinaai/jina-clip-v2. It retains the base model's CC BY-NC 4.0 license and non-commercial restriction. Refer to the base model card for intended use, limitations, training details, and complete attribution.