rootlocalghost/aura-gallery-clip-onnx

🌸 Aura Gallery — CLIP ViT-B/32 ONNX DirectML Model Suite

This repository provides the optimized ONNX multimodal vision and text embedding models used by Aura Gallery, powered by ONNX Runtime (ort) and Microsoft DirectML for native hardware-accelerated execution on Windows DirectX 12 and Intel Arc A770 GPU (with zero Python runtime dependencies).

---

📦 Model Information

• Base Architecture: OpenAI CLIP (ViT-B/32)
• Format: ONNX format with external weights (.onnx + .onnx.data)
• Embedding Dimension: 512 dimensions (L2-normalized)
• Primary Execution Provider: Microsoft DirectML (DmlExecutionProvider), fallback to CPU
• Hardware Target: Intel Arc A770 GPU, Windows DirectX 12

Files Included:

• vision.onnx & vision.onnx.data: Vision transformer encoder (takes normalized 3x224x224 RGB image tensor, outputs 512-d normalized embedding).
• text.onnx & text.onnx.data: Text transformer encoder (takes 77-token sequence, outputs 512-d normalized embedding).
• tokenizer.json & tokenizer_config.json: Fast Hugging Face BPE tokenizer with CLIP vocab.
• preprocessor_config.json: Standard CLIP normalization parameters (Mean: [0.48145466, 0.4578275, 0.40821073], Std: [0.26862954, 0.26130258, 0.27577711], Size: 224x224).

---

⚡ DirectML & Rust Usage

In Rust with ort v2:

use ort::session::Session;
use ort::ep::DirectML;

// Load vision encoder with DirectML GPU acceleration
let vision_session = Session::builder()?
    .with_execution_providers([DirectML::default().build()])?
    .commit_from_file("vision.onnx")?;

// Load text encoder
let text_session = Session::builder()?
    .with_execution_providers([DirectML::default().build()])?
    .commit_from_file("text.onnx")?;

---

💡 Intended Use

1. Natural Language Image Discovery: Match natural queries like "smiling at the beach at sunset" or "cozy hoodie indoors" to photo embeddings via cosine similarity.
2. Zero-Shot Tagging: Classify images across curated scene, outfit, and mood categories.
3. Visual Similarity ("Find Similar"): Search nearest neighbors across image embeddings in SQLite with sub-millisecond retrieval.