File size: 2,731 Bytes
1a8bfa7 a4c8314 1a8bfa7 a4c8314 2340d70 a4c8314 6872d9c a4c8314 6872d9c a4c8314 6872d9c a4c8314 6872d9c a4c8314 6872d9c a4c8314 efa005f a4c8314 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 |
---
license: apache-2.0
pipeline_tag: feature-extraction
tags:
- clip
- vision
datasets:
- Ziyang/yfcc15m
- conceptual_captions
---
<h1 align="center">UForm</h1>
<h3 align="center">
Multi-Modal Inference Library<br/>
For Semantic Search Applications<br/>
</h3>
---
UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
This is model card of the __English only model__ with:
* 12 layers BERT (6 layers for unimodal encoding and rest layers for multimodal encoding)
* ViT-L/14 (image resolution is 224x224)
* Multiple embedding sizes: 64, 256, 512, 768
If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual).
## Evaluation
The following metrics were obtained with multimodal re-ranking (text-to-image retrieval):
| Dataset |Recall@1 | Recall@5 | Recall@10 |
| :------ | ------: | --------: | --------: |
| Zero-Shot Flickr | 0.693 | 0.875 | 0.923 |
| Zero-Shot MS-COCO | 0.382 | 0.617 | 0.728 |
ImageNet-Top1: 0.518 \
ImageNet-Top5: 0.756
## Installation
```bash
pip install uform[onnx-gpu]
```
## Usage
To load the model:
```python
import uform
model, processor = uform.get_model_onnx('unum-cloud/uform-vl-english-large', device='gpu', dtype='fp16')
```
To encode data:
```python
from PIL import Image
text = 'a small red panda in a zoo'
image = Image.open('red_panda.jpg')
image_data = processor.preprocess_image(image)
text_data = processor.preprocess_text(text)
image_features, image_embedding = model.encode_image(image_data, return_features=True)
text_features, text_embedding = model.encode_text(text_data, return_features=True)
score, joint_embedding = model.encode_multimodal(
image_features=image_features,
text_features=text_features,
attention_mask=text_data['attention_mask'],
return_scores=True
)
```
There are two options to calculate semantic compatibility between an image and a text: cosine similarity and [Matching Score](#matching-score).
### Cosine Similarity
__Pros__:
- Computationally cheap.
- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
- Suitable for retrieval in large collections.
__Cons__:
- Takes into account only coarse-grained features.
### Matching Score
Unlike cosine similarity, unimodal embedding are not enough.
Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.
__Pros__:
- Joint embedding captures fine-grained features.
- Suitable for re-ranking – sorting retrieval result.
__Cons__:
- Resource-intensive.
- Not suitable for retrieval in large collections. |