pigzhou/fg-clip2-base-immich

fg-clip2-base — Immich Smart Search ONNX package

ONNX conversion of qihoo360/fg-clip2-base (bilingual zh/en, fine-grained CLIP, 384M params) in the exact cache layout and input/output convention expected by Immich's Smart Search (immich_ml OpenClip loaders). Intended for upstream support in immich-app/immich (whitelist + CLIP_MODEL_INFO entry); until merged, use it by placing the files in the ML cache folder and patching the two whitelists (see below).

Layout (drop into <MACHINE_LEARNING_CACHE_FOLDER>/clip/fg-clip2-base/)

config.json                       # {"text_cfg": {"context_length": 64}}
textual/model.onnx                # input "text" int32 (1,64) -> output "embedding" (1,768)
textual/tokenizer.json
textual/tokenizer_config.json     # pad_token <pad> = 0
visual/model.onnx                 # input "image" float32 (1,3,512,512) -> "embedding" (1,768)
visual/preprocess_cfg.json        # {"size":512,"mean/std":[0.5]*3,"interpolation":"bicubic"}

Conversion notes

• Torch reference: attn_implementation="eager", fp32, opset 17, dynamo=False.
• Text: short-walk path; no attention mask (official training/eval attends to pad tokens; pooling is the fixed last position, index 63 at context 64). The per-row projection head loop is applied batched. Tokenization via tokenizers with enable_padding(length=64) — verified byte-identical to the HF tokenizer, and (input_ids != 0) == attention_mask holds (unused, kept for reference).
• Vision: Immich-style preprocessing (short-side 512 bicubic + center crop + (x-0.5)/0.5), in-graph patchify in (gh, gw, ph, pw, C) order (SigLIP2 fast processor order), 16×16→32×32 antialias bilinear position-interpolation precomputed and burned as a constant, pooling head residual + mlp(layernorm(residual)).
• Outputs are L2-normalized; pgvector cosine distance is scale-invariant.

Validation

• All outputs cos = 1.0 vs the torch reference (13 texts, 9 images), cross-process.
• Immich production loaders (OpenClipTextualEncoder / OpenClipVisualEncoder) load the package and reproduce the reference (cos = 1.0); retrieval sanity 13/13 top-1 on a bilingual query set.
• Subsampled Immich-style benchmark (recall@1/5/10 avg, text→image; XM3600 1000-img gallery seed 1234 + full XTD-10; identical harness/preprocessing for all models): fg-clip2-base 86.67 overall (XM3600 en/zh 86.20/90.07, XTD-10 89.13/81.30) vs nllb-clip-base-siglip__v1 81.67 and (English-only) ViT-B-16-SigLIP-384 80.20 en / ~0.7% zh. Latency batch=1, 8-core CPU f32: text ~35 ms, image ~0.75 s.

Usage with Immich (pending upstream)

Requires two whitelist entries (PR prepared for immich-app/immich): machine-learning/immich_ml/models/constants.py _OPENCLIP_MODELS += fg-clip2-base; server/src/constants.ts CLIP_MODEL_INFO += 'fg-clip2-base': { dimSize: 768 }. Then set the model name in Administration → Settings → Smart Search and re-run the Smart Search job.

License: Apache-2.0 (inherits qihoo360/FG-CLIP).