Model card has 0 info how to use amazing! I want to use ViT-bigG-14, laion2b_s39b_b160k to generate captions for a given folder of images
#13
by
MonsterMMORPG
- opened
I want to use ViT-bigG-14, laion2b_s39b_b160k to generate captions for a given folder of images
And save them with same file name
Any example code please?
Thank you so much
import torch
from PIL import Image
import open_clip
model, _, preprocess = open_clip.create_model_and_transforms('ViT-bigG-14', pretrained='.checkpoints/CLIP-ViT-bigG-14-laion2B-39B-b160k.bin')
tokenizer = open_clip.get_tokenizer('ViT-bigG-14')
image = preprocess(Image.open("animals.jpg")).unsqueeze(0)
text = tokenizer(["a dog","a cat"])
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)
import torch
from PIL import Image
import open_clip
model, _, preprocess = open_clip.create_model_and_transforms('ViT-bigG-14', pretrained='.checkpoints/CLIP-ViT-bigG-14-laion2B-39B-b160k.bin')
tokenizer = open_clip.get_tokenizer('ViT-bigG-14')
image = preprocess(Image.open("animals.jpg")).unsqueeze(0)
text = tokenizer(["a dog","a cat"])
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs)
thanks but this is not what i was looking
this is token based chances
actually i found how to use and even have a auto installer right now
Ah! I missed the captions part. Happy inferencing.