Create README.md

README.md

@@ -0,0 +1,135 @@
+---
+license: apache-2.0
+pipeline_tag: feature-extraction
+tags:
+- clip
+- vision
+datasets:
+- Ziyang/yfcc15m
+- conceptual_captions
+---
+<h1 align="center">UForm</h1>
+<h3 align="center">
+Multi-Modal Inference Library<br/>
+For Semantic Search Applications<br/>
+</h3>
+
+---
+
+UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
+
+This is model card of the __English only model__ with:
+
+* 4 layers BERT (2 layers for unimodal encoding and rest layers for multimodal encoding)
+* ViT-S/16 (image resolution is 224x224)
+
+
+If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual).
+
+## Evaluation
+
+The following metrics were obtained with multimodal re-ranking (text-to-image retrieval):
+
+| Dataset   |Recall@1 |  Recall@5 | Recall@10 |
+| :------   | ------: | --------: | --------: |
+| Zero-Shot Flickr    | 0.565 | 0.790 | 0.860 |
+| Zero-Shot MS-COCO   | 0.281 | 0.525 | 0.645 |
+
+ImageNet-Top1: 0.361 \
+ImageNet-Top5: 0.608
+
+## Installation
+
+```bash
+pip install uform[onnx-gpu]
+```
+
+## Usage
+
+To load the model:
+
+```python
+import uform
+
+model = uform.get_model_onnx('unum-cloud/uform-vl-english-small', device='gpu', dtype='fp16')
+```
+
+To encode data:
+
+```python
+from PIL import Image
+
+text = 'a small red panda in a zoo'
+image = Image.open('red_panda.jpg')
+
+image_data = model.preprocess_image(image)
+text_data = model.preprocess_text(text)
+
+image_embedding = model.encode_image(image_data)
+text_embedding = model.encode_text(text_data)
+score, joint_embedding = model.encode_multimodal(
+    image_features=image_features,
+    text_features=text_features,
+    attention_mask=text_data['attention_mask'],
+    return_scores=True
+)
+```
+
+To get features:
+
+```python
+image_features, image_embedding = model.encode_image(image_data, return_features=True)
+text_features, text_embedding = model.encode_text(text_data, return_features=True)
+```
+
+These features can later be used to produce joint multimodal encodings faster, as the first layers of the transformer can be skipped:
+
+```python
+joint_embedding = model.encode_multimodal(
+    image_features=image_features,
+    text_features=text_features,
+    attention_mask=text_data['attention_mask']
+)
+```
+
+There are two options to calculate semantic compatibility between an image and a text: [Cosine Similarity](#cosine-similarity) and [Matching Score](#matching-score).
+
+### Cosine Similarity
+
+```python
+import torch.nn.functional as F
+
+similarity = F.cosine_similarity(image_embedding, text_embedding)
+```
+
+The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match.
+
+__Pros__:
+
+- Computationally cheap.
+- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
+- Suitable for retrieval in large collections.
+
+__Cons__:
+
+- Takes into account only coarse-grained features.
+
+
+### Matching Score 
+
+Unlike cosine similarity, unimodal embedding are not enough.
+Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.
+
+```python
+score = model.get_matching_scores(joint_embedding)
+```
+
+__Pros__:
+
+- Joint embedding captures fine-grained features.
+- Suitable for re-ranking – sorting retrieval result.
+
+__Cons__:
+
+- Resource-intensive.
+- Not suitable for retrieval in large collections.