Upload folder using huggingface_hub

README.md

@@ -1,16 +1,130 @@
 ---
 license: apache-2.0
-language:
-  - en
-library_name: UForm
 pipeline_tag: feature-extraction
 tags:
-  - clip
-  - vision
-  - transformers.js
-  - transformers
+- clip
+- vision
 datasets:
-  - sbu_captions
-  - visual_genome
-  - ChristophSchuhmann/MS_COCO_2017_URL_TEXT
----
+- Ziyang/yfcc15m
+- conceptual_captions
+---
+<h1 align="center">UForm</h1>
+<h3 align="center">
+Multi-Modal Inference Library<br/>
+For Semantic Search Applications<br/>
+</h3>
+
+---
+
+UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
+
+This is model card of the __English only model__ with:
+
+* 4 layers BERT (2 layers for unimodal encoding and rest layers for multimodal encoding)
+* ViT-S/16 (image resolution is 224x224)
+
+
+If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual).
+
+## Evaluation
+
+The following metrics were obtained with multimodal re-ranking (text-to-image retrieval):
+
+| Dataset   |Recall@1 |  Recall@5 | Recall@10 |
+| :------   | ------: | --------: | --------: |
+| Zero-Shot Flickr    | 0.565 | 0.790 | 0.860 |
+| Zero-Shot MS-COCO   | 0.281 | 0.525 | 0.645 |
+
+ImageNet-Top1: 0.361 \
+ImageNet-Top5: 0.608
+
+## Installation
+
+```bash
+pip install uform[torch]
+```
+
+## Usage
+
+To load the model:
+
+```python
+import uform
+
+model, processor = uform.get_model('unum-cloud/uform-vl-english-small')
+```
+
+To encode data:
+
+```python
+from PIL import Image
+
+text = 'a small red panda in a zoo'
+image = Image.open('red_panda.jpg')
+
+image_data = processor.preprocess_image(image)
+text_data = processor.preprocess_text(text)
+
+image_features, image_embedding = model.encode_image(image_data, return_features=True)
+text_features, text_embedding = model.encode_text(text_data, return_features=True)
+joint_embedding = model.encode_multimodal(image=image_data, text=text_data)
+```
+
+To get features:
+
+```python
+image_features, image_embedding = model.encode_image(image_data, return_features=True)
+text_features, text_embedding = model.encode_text(text_data, return_features=True)
+```
+
+These features can later be used to produce joint multimodal encodings faster, as the first layers of the transformer can be skipped:
+
+```python
+joint_embedding = model.encode_multimodal(
+    image_features=image_features,
+    text_features=text_features,
+    attention_mask=text_data['attention_mask']
+)
+```
+
+There are two options to calculate semantic compatibility between an image and a text: [Cosine Similarity](#cosine-similarity) and [Matching Score](#matching-score).
+
+### Cosine Similarity
+
+```python
+import torch.nn.functional as F
+
+similarity = F.cosine_similarity(image_embedding, text_embedding)
+```
+
+The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match.
+
+__Pros__:
+
+- Computationally cheap.
+- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
+- Suitable for retrieval in large collections.
+
+__Cons__:
+
+- Takes into account only coarse-grained features.
+
+
+### Matching Score 
+
+Unlike cosine similarity, unimodal embedding are not enough.
+Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.
+
+```python
+score = model.get_matching_scores(joint_embedding)
+```
+
+__Pros__:
+
+- Joint embedding captures fine-grained features.
+- Suitable for re-ranking – sorting retrieval result.
+
+__Cons__:
+
+- Resource-intensive.
+- Not suitable for retrieval in large collections.

config.json

@@ -19,14 +19,22 @@
         "dropout_prob": 0.1
     },
     "image_encoder": {
+        "normalization_means": [
+            0.48145466,
+            0.4578275,
+            0.40821073
+        ],
+        "normalization_deviations": [
+            0.26862954,
+            0.26130258,
+            0.27577711
+        ],
         "dim": 384,
         "patch_size": 16,
         "image_size": 224,
         "num_layers": 12,
         "num_heads": 6,
         "embedding_dim": 256,
-        "normalization_means": [0.48145466, 0.4578275, 0.40821073],
-        "normalization_deviations": [0.26862954, 0.26130258, 0.27577711],
         "pooling": "cls"
     }
 }

image_encoder.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f06b44c342f406cb5d3bfa2c009b55bbac455ba97bb07846b6161287aee4c1a3
-size 111195
+oid sha256:f18da9209de45eca55e0e0f23f3f3f7c4ad1a03d96533a7c5108df0a125c1663
+size 111190

image_encoder.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:80df8d05daadf940deb3a21f1f7fa4b03dd5e7ee4a2ec0be9cca87a6751ebc31
+oid sha256:750f259efc991efc9f32aae3cb8960508e8ae68122b7fa83f26b6ba97b78d1cc
 size 87106624

image_encoder.mlpackage/Manifest.json

@@ -1,18 +1,18 @@
 {
     "fileFormatVersion": "1.0.0",
     "itemInfoEntries": {
-        "20458914-6BBA-4AF3-ADB8-60CBA7CE7713": {
-            "author": "com.apple.CoreML",
-            "description": "CoreML Model Specification",
-            "name": "model.mlmodel",
-            "path": "com.apple.CoreML/model.mlmodel"
-        },
-        "CD25C0E2-C2BB-4C45-8EF7-2D23E3E550A6": {
+        "703D1BA1-CC3F-468B-B079-5914AE99ECF4": {
             "author": "com.apple.CoreML",
             "description": "CoreML Model Weights",
             "name": "weights",
             "path": "com.apple.CoreML/weights"
+        },
+        "C8AB2870-74DF-4EDF-9F73-4903BE34700E": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
         }
     },
-    "rootModelIdentifier": "20458914-6BBA-4AF3-ADB8-60CBA7CE7713"
+    "rootModelIdentifier": "C8AB2870-74DF-4EDF-9F73-4903BE34700E"
 }

image_encoder.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0eb66a62c8edb29ee6791882d7153501327d8450798a75685a835c051eadd54b
-size 22589727
+oid sha256:f0d6bc2354318ad9d5a53cad2aa219e307bcdcf3010aa9a8cdb3f385c574015a
+size 22589738

image_encoder.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b194aad10463422dcce1605116b505301cd5cc24382b59869ac59090dcd8de42
-size 43604334
+oid sha256:ad037e636f42ca84f3e2c2a3c1180649f46507ff5c771f6d7def14481f84dbb6
+size 43620486

text_encoder.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d022e8345991dbf0352357afd63df088628115e22066e864506f9b71de5c480f
+oid sha256:091bd543cf3c3ec3eddd4b0d50fca6309b47e15163c69c8e4a380c708ed9a320
 size 151459264

text_encoder.mlpackage/Manifest.json

@@ -1,18 +1,18 @@
 {
     "fileFormatVersion": "1.0.0",
     "itemInfoEntries": {
-        "74D647A6-216E-48BE-A0A6-BD4BF4EEB3CA": {
-            "author": "com.apple.CoreML",
-            "description": "CoreML Model Weights",
-            "name": "weights",
-            "path": "com.apple.CoreML/weights"
-        },
-        "BE4077DB-78CC-4549-B7F9-0F6AA0AEFF56": {
+        "702D723F-AA56-4B0B-A29C-6F7D643170BA": {
             "author": "com.apple.CoreML",
             "description": "CoreML Model Specification",
             "name": "model.mlmodel",
             "path": "com.apple.CoreML/model.mlmodel"
+        },
+        "D5AF71DD-F123-410D-9CED-F0017B203BF1": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
         }
     },
-    "rootModelIdentifier": "BE4077DB-78CC-4549-B7F9-0F6AA0AEFF56"
+    "rootModelIdentifier": "702D723F-AA56-4B0B-A29C-6F7D643170BA"
 }

text_encoder.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c74e2c8fdbc145780e2473b35553ad0fef523ac7a7ac17d4920532179fdbcf70
+oid sha256:c0b8c0dca234622eb2ee57acf22c893394a79126b7dc4be2ed7cb7dcb1095aca
 size 37994692

text_encoder.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e1d667c4601292fa2197face6fa472726ce69165e9674dc7f3f7ca66b35d82e5
-size 114152802
+oid sha256:d8636c1447e28b39b9d1d51902497acd65007aad26b3cf84912fd917046bccb7
+size 114159394