Update README.md

README.md

@@ -10,7 +10,7 @@ license: apache-2.0
 
 # sentence-transformers/clip-ViT-B-32-multilingual-v1
 
-This is a multi-lingual version of the OpenAI CLIP-ViT-B32 model. You can map text (in 50+ languages) and images to a common dense vector space such that images and the matching texts are close.
+This is a multi-lingual version of the OpenAI CLIP-ViT-B32 model. You can map text (in 50+ languages) and images to a common dense vector space such that images and the matching texts are close. This model can be used for **image search** (users search through a large collection of images) and for **multi-lingual zero-shot image classification** (image labels are defined as text).
 
 
 ## Usage (Sentence-Transformers)
@@ -24,21 +24,69 @@ pip install -U sentence-transformers
 Then you can use the model like this:
 
 ```python
-from sentence_transformers import SentenceTransformer
-sentences = ["This is an example sentence", "Each sentence is converted"]
+from sentence_transformers import SentenceTransformer, util
+from PIL import Image, ImageFile
+import requests
+import torch
 
-model = SentenceTransformer('sentence-transformers/clip-ViT-B-32-multilingual-v1')
-embeddings = model.encode(sentences)
-print(embeddings)
-```
+# We use the original clip-ViT-B-32 for encoding images
+img_model = SentenceTransformer('clip-ViT-B-32')
+
+# Our text embedding model is aligned to the img_model and maps 50+
+# languages to the same vector space
+text_model = SentenceTransformer('sentence-transformers/clip-ViT-B-32-multilingual-v1')
+
+
+# Now we load and encode the images
+def load_image(url_or_path):
+    if url_or_path.startswith("http://") or url_or_path.startswith("https://"):
+        return Image.open(requests.get(url_or_path, stream=True).raw)
+    else:
+        return Image.open(url_or_path)
+
+# We load 3 images. You can either pass URLs or
+# a path on your disc
+img_paths = [
+    # Dog image
+    "https://unsplash.com/photos/QtxgNsmJQSs/download?ixid=MnwxMjA3fDB8MXxhbGx8fHx8fHx8fHwxNjM1ODQ0MjY3&w=640",
+
+    # Cat image
+    "https://unsplash.com/photos/9UUoGaaHtNE/download?ixid=MnwxMjA3fDB8MXxzZWFyY2h8Mnx8Y2F0fHwwfHx8fDE2MzU4NDI1ODQ&w=640",
 
+    # Beach image
+    "https://unsplash.com/photos/Siuwr3uCir0/download?ixid=MnwxMjA3fDB8MXxzZWFyY2h8NHx8YmVhY2h8fDB8fHx8MTYzNTg0MjYzMg&w=640"
+]
 
+images = [load_image(img) for img in img_paths]
 
-## Evaluation Results
+# Map images to the vector space
+img_embeddings = img_model.encode(images)
+
+# Now we encode our text:
+texts = [
+    "A dog in the snow",
+    "Eine Katze",  # German: A cat
+    "Una playa con palmeras."  # Spanish: a beach with palm trees
+]
+
+text_embeddings = text_model.encode(texts)
+
+# Compute cosine similarities:
+cos_sim = util.cos_sim(text_embeddings, img_embeddings)
+
+for text, scores in zip(texts, cos_sim):
+    max_img_idx = torch.argmax(scores)
+    print("Text:", text)
+    print("Score:", scores[max_img_idx] )
+    print("Path:", img_paths[max_img_idx], "\n")
+
+```
 
+## Multilingual Image Search - Demo
+For a demo of multilingual image search, have a look at: [Image_Search-multilingual.ipynb](https://github.com/UKPLab/sentence-transformers/tree/master/examples/applications/image-search/Image_Search-multilingual.ipynb) ( [Colab version](https://colab.research.google.com/drive/1N6woBKL4dzYsHboDNqtv-8gjZglKOZcn?usp=sharing) )
 
+For more details on image search and zero-shot image classification, have a look at the documentation on [SBERT.net](https://www.sbert.net/examples/applications/image-search/README.html).
 
-For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=sentence-transformers/clip-ViT-B-32-multilingual-v1)