mounkie/surgical_instruments-lora-adapter

PEFT LoRA Adapter

This repository contains a LoRA adapter for OpenCLIP vision inference.

• Base model: open-clip:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
• Adapter target: visual.transformer
• Task: image_embedding

Usage

import open_clip
import torch
from peft import PeftModel

MODEL_NAME = 'hf-hub:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K'
ADAPTER_REPO = 'mounkie/surgical_instruments-lora-adapter'

model, _, preprocess = open_clip.create_model_and_transforms(MODEL_NAME)
model.visual.transformer = PeftModel.from_pretrained(
    model.visual.transformer,
    ADAPTER_REPO,
)
model.eval()

# image = preprocess(pil_image).unsqueeze(0)
# with torch.no_grad():
#     image_features = model.encode_image(image)
#     image_features = image_features / image_features.norm(dim=-1, keepdim=True)