ruclip-vit-base-patch16-384

RuCLIP (Russian Contrastive Language–Image Pretraining) is a multimodal model for obtaining images and text similarities and rearranging captions and pictures. RuCLIP builds on a large body of work on zero-shot transfer, computer vision, natural language processing and multimodal learning.

Model was trained by Sber AI and SberDevices teams.

• Task: text ranking; image ranking; zero-shot image classification;
• Type: encoder
• Num Parameters: 150M
• Training Data Volume: 240 million text-image pairs
• Language: Russian
• Context Length: 77
• Transformer Layers: 12
• Transformer Width: 512
• Transformer Heads: 8
• Image Size: 384
• Vision Layers: 12
• Vision Width: 768
• Vision Patch Size: 16

Usage Github

pip install ruclip

clip, processor = ruclip.load("ruclip-vit-base-patch16-384", device="cuda")

Performance

We have evaluated the performance on the following datasets:

Dataset	Metric Name	Metric Result
Food101	acc	0.689
CIFAR10	acc	0.845
CIFAR100	acc	0.569
Birdsnap	acc	0.195
SUN397	acc	0.521
Stanford Cars	acc	0.626
DTD	acc	0.421
MNIST	acc	0.478
STL10	acc	0.964
PCam	acc	0.501
CLEVR	acc	0.132
Rendered SST2	acc	0.525
ImageNet	acc	0.482
FGVC Aircraft	mean-per-class	0.046
Oxford Pets	mean-per-class	0.635
Caltech101	mean-per-class	0.835
Flowers102	mean-per-class	0.452
HatefulMemes	roc-auc	0.543

Authors

• Alex Shonenkov: Github, Kaggle GM
• Daniil Chesakov: Github
• Denis Dimitrov: Github
• Igor Pavlov: Github