File size: 3,239 Bytes
039f477 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 |
---
license: apache-2.0
pipeline_tag: feature-extraction
tags:
- clip
- vision
datasets:
- sbu_captions
- visual_genome
- ChristophSchuhmann/MS_COCO_2017_URL_TEXT
---
<h1 align="center">UForm</h1>
<h3 align="center">
Multi-Modal Inference Library<br/>
For Semantic Search Applications<br/>
</h3>
---
UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
This is model card of the __English only model__ with:
* 4 layers BERT (2 layers for unimodal encoding and rest layers for multimodal encoding)
* ViT-B/16 (image resolution is 224x224)
If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual).
## Evaluation
The following metrics were obtained with multimodal re-ranking:
| Dataset | Recall@1 | Recall@5 | Recall@10 |
| :-------- | ------: | --------: | --------: |
| Zero-Shot Flickr | 0.727 | 0.915 | 0.949 |
| MS-COCO (train split was in training data) | 0.510 | 0.761 | 0.838 |
## Installation
```bash
pip install uform[torch]
```
## Usage
To load the model:
```python
import uform
model, processor = uform.get_model('unum-cloud/uform-vl-english')
```
To encode data:
```python
from PIL import Image
text = 'a small red panda in a zoo'
image = Image.open('red_panda.jpg')
image_data = processor.preprocess_image(image)
text_data = processor.preprocess_text(text)
image_features, image_embedding = model.encode_image(image_data, return_features=True)
text_features, text_embedding = model.encode_text(text_data, return_features=True)
```
To get features:
```python
image_features, image_embedding = model.encode_image(image_data, return_features=True)
text_features, text_embedding = model.encode_text(text_data, return_features=True)
```
These features can later be used to produce joint multimodal encodings faster, as the first layers of the transformer can be skipped:
```python
joint_embedding = model.encode_multimodal(
image_features=image_features,
text_features=text_features,
attention_mask=text_data['attention_mask']
)
```
There are two options to calculate semantic compatibility between an image and a text: [Cosine Similarity](#cosine-similarity) and [Matching Score](#matching-score).
### Cosine Similarity
```python
import torch.nn.functional as F
similarity = F.cosine_similarity(image_embedding, text_embedding)
```
The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match.
__Pros__:
- Computationally cheap.
- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
- Suitable for retrieval in large collections.
__Cons__:
- Takes into account only coarse-grained features.
### Matching Score
Unlike cosine similarity, unimodal embedding are not enough.
Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.
```python
score = model.get_matching_scores(joint_embedding)
```
__Pros__:
- Joint embedding captures fine-grained features.
- Suitable for re-ranking – sorting retrieval result.
__Cons__:
- Resource-intensive.
- Not suitable for retrieval in large collections.
|