CLIP ViT-L/14 in OpenVINO™ format

Original model details

The CLIP model was developed by OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero-shot manner. It was not developed for general model deployment - to deploy models like CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they’re being deployed within.

Model type

The model uses a ViT-L/14 Transformer architecture as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.

OpenVINO™ optimization

To increase the efficiency of the model during inference, we utilized the OpenVINO™ toolkit for optimization. The table below showcases the inference time improvements achieved with OpenVINO™ compared to the original PyTorch implementation:

Metric	PyTorch Inference Time (sec)	OpenVINO™ Inference Time (sec)	Similarity
mean	0.52	0.46	1
std	0.11	0.09	0
min	0.39	0.36	1
max	0.70	0.62	1

OpenVINO offers a 1.12x speedup in inference time compared to PyTorch. It was measured on same image in 100 iterations on Intel(R) Xeon(R) CPU @ 2.20GHz (CPU family: 6, Model: 79).

The results indicate that the OpenVINO™ optimization provides a consistent improvement in inference time while maintaining the same level of accuracy (as indicated by the similarity score).

Usage

You can utilize this optimized model for faster inferences in environments where time is a critical factor. Ensure you have the necessary libraries and dependencies installed to leverage the usage of OpenVINO™.

pip install transformers huggingface_hub openvino-dev

Then use it for inference:

import os

import numpy as np
from PIL import Image
from huggingface_hub import snapshot_download
from openvino.runtime import Core
from scipy.special import softmax
from transformers import CLIPProcessor

# Download the OV model
ov_path = snapshot_download(repo_id="scaleflex/clip-vit-large-patch14-openvino")
# Load preprocessor for model input
processor = CLIPProcessor.from_pretrained("scaleflex/clip-vit-large-patch14-openvino")
ov_model_xml = os.path.join(ov_path, "clip-vit-large-patch14.xml")

image = Image.open("face.png")  # download this example image: http://sample.li/face.png
input_labels = [
    "businessman",
    "dog playing in the garden",
    "beautiful woman",
    "big city",
    "lake in the mountain",
]
text_descriptions = [f"This is a photo of a {label}" for label in input_labels]
inputs = processor(
    text=text_descriptions, images=[image], return_tensors="pt", padding=True
)

# Create OpenVINO core object instance
core = Core()

ov_model = core.read_model(model=ov_model_xml)
# Compile model for loading on device
compiled_model = core.compile_model(ov_model)
# Obtain output tensor for getting predictions
logits_per_image_out = compiled_model.output(0)
# Run inference on preprocessed data and get image-text similarity score
ov_logits_per_image = compiled_model(dict(inputs))[logits_per_image_out]
# Perform softmax on score
probs = softmax(ov_logits_per_image, axis=1)
max_index = np.argmax(probs)

# Use the index to get the corresponding label
label_with_max_prob = input_labels[max_index]
print(
    f"The label with the highest probability is: '{label_with_max_prob}' with a probability of {probs[0][max_index] * 100:.2f}%"
)
# The label with the highest probability is: 'beautiful woman' with a probability of 97.87%