Update README.md

README.md

@@ -5,41 +5,45 @@ tags:
 - clip
 - vision
 datasets:
-- sbu_captions
-- visual_genome
-- ChristophSchuhmann/MS_COCO_2017_URL_TEXT
+- Ziyang/yfcc15m
+- conceptual_captions
 ---
 <h1 align="center">UForm</h1>
 <h3 align="center">
-Multi-Modal Inference Library<br/>
-For Semantic Search Applications<br/>
+Pocket-Sized Multimodal AI<br/>
+For Content Understanding and Generation<br/>
+In Python, JavaScript, and Swift<br/>
 </h3>
 
 ---
 
-UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
+The `uform3-image-text-english-base` UForm model is a tiny vision and English language encoder, mapping them into a shared vector space.
+This model produces up to __256-dimensional embeddings__ and is made of:
 
-This is model card of the __English only model__ with:
+* Text encoder: 4-layer BERT for up to 64 input tokens.
+* Visual encoder: ViT-B/16 for images of 224 x 224 resolution.
 
-* 4 layers BERT (2 layers for unimodal encoding and rest layers for multimodal encoding)
-* ViT-B/16 (image resolution is 224x224)
-
-
-If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual).
+Unlike most CLIP-like multomodal models, this model shares 2 layers between the text and visual encoder to allow for more data- and parameter-efficient training.
+Also unlike most models, UForm provides checkpoints compatible with PyTorch, ONNX, and CoreML, covering the absolute majority of AI-capable devices, with pre-quantized weights and inference code.
+If you need a larger, more accurate, or multilingual model, check our [HuggingFace Hub](https://huggingface.co/unum-cloud/).
+For more details on running the model, check out the [UForm GitHub repository](https://github.com/unum-cloud/uform/).
 
 ## Evaluation
 
-The following metrics were obtained with multimodal re-ranking:
+For zero-shot ImageNet classification the model achieves Top-1 accuracy of 36.1% and Top-5 of 60.8%.
+On text-to-image retrieval it reaches 86% Recall@10 for Flickr:
 
 | Dataset |  Recall@1 |  Recall@5 | Recall@10 |
 | :-------- | ------: | --------: | --------: |
 | Zero-Shot Flickr | 0.727 | 0.915 | 0.949 |
-| MS-COCO (train split was in training data) | 0.510 | 0.761 | 0.838 |
+| MS-COCO ¹ | 0.510 | 0.761 | 0.838 |
+
+> ¹ It's important to note, that the MS-COCO train split was present in the training data.
 
 ## Installation
 
 ```bash
-pip install uform[torch]
+pip install "uform[torch,onnx]"
 ```
 
 ## Usage
@@ -47,82 +51,31 @@ pip install uform[torch]
 To load the model:
 
 ```python
-import uform
-
-model, processor = uform.get_model('unum-cloud/uform-vl-english')
-```
+from uform import get_model, Modality
 
-To encode data:
-
-```python
+import requests
+from io import BytesIO
 from PIL import Image
 
-text = 'a small red panda in a zoo'
-image = Image.open('red_panda.jpg')
-
-image_data = processor.preprocess_image(image)
-text_data = processor.preprocess_text(text)
-
-image_features, image_embedding = model.encode_image(image_data, return_features=True)
-text_features, text_embedding = model.encode_text(text_data, return_features=True)
-```
-
-To get features:
-
-```python
-image_features, image_embedding = model.encode_image(image_data, return_features=True)
-text_features, text_embedding = model.encode_text(text_data, return_features=True)
-```
-
-These features can later be used to produce joint multimodal encodings faster, as the first layers of the transformer can be skipped:
-
-```python
-joint_embedding = model.encode_multimodal(
-    image_features=image_features,
-    text_features=text_features,
-    attention_mask=text_data['attention_mask']
-)
-```
-
-There are two options to calculate semantic compatibility between an image and a text: [Cosine Similarity](#cosine-similarity) and [Matching Score](#matching-score).
-
-### Cosine Similarity
-
-```python
-import torch.nn.functional as F
+model_name = 'unum-cloud/uform3-image-text-english-base'
+modalities = [Modality.TEXT_ENCODER, Modality.IMAGE_ENCODER]
+processors, models = get_model(model_name, modalities=modalities)
 
-similarity = F.cosine_similarity(image_embedding, text_embedding)
+model_text = models[Modality.TEXT_ENCODER]
+model_image = models[Modality.IMAGE_ENCODER]
+processor_text = processors[Modality.TEXT_ENCODER]
+processor_image = processors[Modality.IMAGE_ENCODER]
 ```
 
-The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match.
-
-__Pros__:
-
-- Computationally cheap.
-- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
-- Suitable for retrieval in large collections.
-
-__Cons__:
-
-- Takes into account only coarse-grained features.
-
-
-### Matching Score 
-
-Unlike cosine similarity, unimodal embedding are not enough.
-Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.
+To encode the content:
 
 ```python
-score = model.get_matching_scores(joint_embedding)
+text = 'a cityscape bathed in the warm glow of the sun, with varied architecture and a towering, snow-capped mountain rising majestically in the background'
+image_url = 'https://media-cdn.tripadvisor.com/media/photo-s/1b/28/6b/53/lovely-armenia.jpg'
+image_url = Image.open(BytesIO(requests.get(image_url).content))
+
+image_data = processor_image(image)
+text_data = processor_text(text)
+image_features, image_embedding = model_image.encode(image_data, return_features=True)
+text_features, text_embedding = model_text.encode(text_data, return_features=True)
 ```
-
-__Pros__:
-
-- Joint embedding captures fine-grained features.
-- Suitable for re-ranking – sorting retrieval result.
-
-__Cons__:
-
-- Resource-intensive.
-- Not suitable for retrieval in large collections.
-