Upload folder using huggingface_hub

.gitattributes

@@ -25,7 +25,6 @@
 *.safetensors filter=lfs diff=lfs merge=lfs -text
 saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
 *.tflite filter=lfs diff=lfs merge=lfs -text
 *.tgz filter=lfs diff=lfs merge=lfs -text
 *.wasm filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,128 @@
+---
+license: apache-2.0
+pipeline_tag: feature-extraction
+tags:
+- clip
+- vision
+datasets:
+- sbu_captions
+- visual_genome
+- ChristophSchuhmann/MS_COCO_2017_URL_TEXT
+---
+<h1 align="center">UForm</h1>
+<h3 align="center">
+Multi-Modal Inference Library<br/>
+For Semantic Search Applications<br/>
+</h3>
+
+---
+
+UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
+
+This is model card of the __English only model__ with:
+
+* 4 layers BERT (2 layers for unimodal encoding and rest layers for multimodal encoding)
+* ViT-B/16 (image resolution is 224x224)
+
+
+If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual).
+
+## Evaluation
+
+The following metrics were obtained with multimodal re-ranking:
+
+| Dataset |  Recall@1 |  Recall@5 | Recall@10 |
+| :-------- | ------: | --------: | --------: |
+| Zero-Shot Flickr | 0.727 | 0.915 | 0.949 |
+| MS-COCO (train split was in training data) | 0.510 | 0.761 | 0.838 |
+
+## Installation
+
+```bash
+pip install uform[torch]
+```
+
+## Usage
+
+To load the model:
+
+```python
+import uform
+
+model, processor = uform.get_model('unum-cloud/uform-vl-english')
+```
+
+To encode data:
+
+```python
+from PIL import Image
+
+text = 'a small red panda in a zoo'
+image = Image.open('red_panda.jpg')
+
+image_data = processor.preprocess_image(image)
+text_data = processor.preprocess_text(text)
+
+image_features, image_embedding = model.encode_image(image_data, return_features=True)
+text_features, text_embedding = model.encode_text(text_data, return_features=True)
+```
+
+To get features:
+
+```python
+image_features, image_embedding = model.encode_image(image_data, return_features=True)
+text_features, text_embedding = model.encode_text(text_data, return_features=True)
+```
+
+These features can later be used to produce joint multimodal encodings faster, as the first layers of the transformer can be skipped:
+
+```python
+joint_embedding = model.encode_multimodal(
+    image_features=image_features,
+    text_features=text_features,
+    attention_mask=text_data['attention_mask']
+)
+```
+
+There are two options to calculate semantic compatibility between an image and a text: [Cosine Similarity](#cosine-similarity) and [Matching Score](#matching-score).
+
+### Cosine Similarity
+
+```python
+import torch.nn.functional as F
+
+similarity = F.cosine_similarity(image_embedding, text_embedding)
+```
+
+The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match.
+
+__Pros__:
+
+- Computationally cheap.
+- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
+- Suitable for retrieval in large collections.
+
+__Cons__:
+
+- Takes into account only coarse-grained features.
+
+
+### Matching Score 
+
+Unlike cosine similarity, unimodal embedding are not enough.
+Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.
+
+```python
+score = model.get_matching_scores(joint_embedding)
+```
+
+__Pros__:
+
+- Joint embedding captures fine-grained features.
+- Suitable for re-ranking – sorting retrieval result.
+
+__Cons__:
+
+- Resource-intensive.
+- Not suitable for retrieval in large collections.
+

config.json

@@ -0,0 +1,40 @@
+{
+    "text_encoder": {
+        "tokenizer_class": "bert",
+        "model_type": "bert",
+        "dim": 768,
+        "context_dim": 768,
+        "vocab_size": 30522,
+        "padding_idx": 0,
+        "num_layers": 4,
+        "num_heads": 12,
+        "embedding_dim": 256,
+        "multimodal_layers_ids": [
+            2,
+            3
+        ],
+        "head_one_neuron": false,
+        "pooling": "cls",
+        "max_position_embeddings": 77,
+        "dropout_prob": 0.1
+    },
+    "image_encoder": {
+        "normalization_means": [
+            0.48145466,
+            0.4578275,
+            0.40821073
+        ],
+        "normalization_deviations": [
+            0.26862954,
+            0.26130258,
+            0.27577711
+        ],
+        "dim": 768,
+        "patch_size": 16,
+        "image_size": 224,
+        "num_layers": 12,
+        "num_heads": 12,
+        "embedding_dim": 256,
+        "pooling": "cls"
+    }
+}

image_encoder.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:101d2a485d7f1fb9b13bb2e9b21017d41e0484c09d7252fa48cd45663bf6a7a0
+size 111239

image_encoder.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d795737604c1677ff6b2e5fa8bceface13912a9baecea99b6ef3e24efa050464
+size 344067136

image_encoder.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "ADEDA51D-BFFF-46BC-AF60-BF0FB2F3BB19": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        },
+        "F1FDEC64-5002-491E-8C51-99A28404DD54": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        }
+    },
+    "rootModelIdentifier": "ADEDA51D-BFFF-46BC-AF60-BF0FB2F3BB19"
+}

image_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54ea2e32abb19d5a8415143f3d2d87ce3e96ccc0a031653af198db6fd13ecef8
+size 87267681

image_encoder.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea4548002193d407577533b6a98596a9903f9576c4c8de574b06050aaa5c92d7
+size 172100742

image_encoder.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:800f75efc508c0eb3445ec270152194e692860656eef32b8ebab888ff5946351
+size 172047072

text_encoder.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1428abd50aa8fe334c81f5bc88c8fb62b9903ba05ad94e61d0f6d73f3cf5c42
+size 21788

text_encoder.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8dd08b0a4867b25785d44b8dce1c37174894c788a732dd18c2ab3f8b47f048d3
+size 151499200

text_encoder.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "546A7C3A-6CCD-42C4-A20B-507DBA62CEA6": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        },
+        "D48DD071-5007-4390-BF8E-9D05D8483749": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        }
+    },
+    "rootModelIdentifier": "546A7C3A-6CCD-42C4-A20B-507DBA62CEA6"
+}

text_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac4e7765b4951b4ebe9001f35d7deb1dadf77cde697c363ba90d2c44f71c052b
+size 38004782

text_encoder.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50bc23ed1f2642decba5e2bc7eca5bc9d83cdc7e13f44fd00298395d90849cb3
+size 113589216

text_encoder.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7001e423ec999a2bfbd33683f48f8163942d3e9a7259944db170129e0857ffdf
+size 113566956