Update README.md

README.md

@@ -10,38 +10,38 @@ datasets:
 ---
 <h1 align="center">UForm</h1>
 <h3 align="center">
-Multi-Modal Inference Library<br/>
-For Semantic Search Applications<br/>
+Multi-Modal Pocket-Sized AI<br/>
+For Content Understaning and Generation<br/>
+In Python, JavaScript, and Swift<br/>
 </h3>
 
 ---
 
-UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
+The `uform3-image-text-english-small` UForm model is a tiny vision and English language encoder, mapping them into a shared vector space.
+This model is made of:
 
-This is model card of the __English only model__ with:
+* Text encoder: 4-layer BERT.
+* Visual encoder: ViT-S/16 for images of 224x224 resolution.
 
-* 4 layers BERT (2 layers for unimodal encoding and rest layers for multimodal encoding)
-* ViT-S/16 (image resolution is 224x224)
-
-
-If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual).
+Unlike most CLIP-like multomodal models, this model shares 2 layers between the text and visual encoder to allow for more data- and parameter-efficient training.
+Also unlike most models, UForm provides checkpoints compatible with PyTorch, ONNX, and CoreML, covering the absolute majority of AI-capable devices, with pre-quantized weights and inference code.
+If you need a larger, more accurate, or multilingual model, check our [HuggingFace Hub](https://huggingface.co/unum-cloud/).
+For more details on running the model, check out the [UForm GitHub repository](https://github.com/unum-cloud/uform/).
 
 ## Evaluation
 
-The following metrics were obtained with multimodal re-ranking (text-to-image retrieval):
+For zero-shot ImageNet classification the model achieves Top-1 accuracy of 36.1% and Top-5 of 60.8%.
+On text-to-image retrieval it reaches 86% Recall@10 for Flickr:
 
 | Dataset   |Recall@1 |  Recall@5 | Recall@10 |
 | :------   | ------: | --------: | --------: |
 | Zero-Shot Flickr    | 0.565 | 0.790 | 0.860 |
 | Zero-Shot MS-COCO   | 0.281 | 0.525 | 0.645 |
 
-ImageNet-Top1: 0.361 \
-ImageNet-Top5: 0.608
-
 ## Installation
 
 ```bash
-pip install uform[torch]
+pip install "uform[torch,onnx]"
 ```
 
 ## Usage
@@ -49,82 +49,33 @@ pip install uform[torch]
 To load the model:
 
 ```python
-import uform
-
-model, processor = uform.get_model('unum-cloud/uform-vl-english-small')
-```
-
-To encode data:
+from uform import get_model, Modality
 
-```python
+import requests
+from io import BytesIO
 from PIL import Image
 
-text = 'a small red panda in a zoo'
-image = Image.open('red_panda.jpg')
-
-image_data = processor.preprocess_image(image)
-text_data = processor.preprocess_text(text)
+model_name = 'unum-cloud/uform3-image-text-english-small'
+modalities = [Modality.TEXT_ENCODER, Modality.IMAGE_ENCODER]
+processors, models = get_model(model_name, modalities=modalities)
 
-image_features, image_embedding = model.encode_image(image_data, return_features=True)
-text_features, text_embedding = model.encode_text(text_data, return_features=True)
-joint_embedding = model.encode_multimodal(image=image_data, text=text_data)
+model_text = models[Modality.TEXT_ENCODER]
+model_image = models[Modality.IMAGE_ENCODER]
+processor_text = processors[Modality.TEXT_ENCODER]
+processor_image = processors[Modality.IMAGE_ENCODER]
 ```
 
-To get features:
+To encode the content:
 
 ```python
-image_features, image_embedding = model.encode_image(image_data, return_features=True)
-text_features, text_embedding = model.encode_text(text_data, return_features=True)
+text = 'a cityscape bathed in the warm glow of the sun, with varied architecture and a towering, snow-capped mountain rising majestically in the background'
+image_url = 'https://media-cdn.tripadvisor.com/media/photo-s/1b/28/6b/53/lovely-armenia.jpg'
+image_url = Image.open(BytesIO(requests.get(image_url).content))
+
+image_data = processor_image(image)
+text_data = processor_text(text)
+image_features, image_embedding = model_image.encode(image_data, return_features=True)
+text_features, text_embedding = model_text.encode(text_data, return_features=True)
 ```
 
-These features can later be used to produce joint multimodal encodings faster, as the first layers of the transformer can be skipped:
-
-```python
-joint_embedding = model.encode_multimodal(
-    image_features=image_features,
-    text_features=text_features,
-    attention_mask=text_data['attention_mask']
-)
-```
-
-There are two options to calculate semantic compatibility between an image and a text: [Cosine Similarity](#cosine-similarity) and [Matching Score](#matching-score).
-
-### Cosine Similarity
-
-```python
-import torch.nn.functional as F
-
-similarity = F.cosine_similarity(image_embedding, text_embedding)
-```
-
-The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match.
-
-__Pros__:
-
-- Computationally cheap.
-- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
-- Suitable for retrieval in large collections.
-
-__Cons__:
-
-- Takes into account only coarse-grained features.
-
-
-### Matching Score 
-
-Unlike cosine similarity, unimodal embedding are not enough.
-Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.
-
-```python
-score = model.get_matching_scores(joint_embedding)
-```
-
-__Pros__:
-
-- Joint embedding captures fine-grained features.
-- Suitable for re-ranking – sorting retrieval result.
-
-__Cons__:
 
-- Resource-intensive.
-- Not suitable for retrieval in large collections.