patomp commited on
Commit
79c61fa
1 Parent(s): e49b33f

Create readme usage and benchmark

Browse files
Files changed (1) hide show
  1. README.md +59 -0
README.md ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## Requirements
2
+
3
+ ```bash
4
+ pip install pythainlp
5
+ pip install git+https://github.com/openai/CLIP.git
6
+ ```
7
+
8
+ ## Usage
9
+
10
+ Encode a text by
11
+ ```python
12
+ from transformers import AutoModel
13
+
14
+ text = 'หมากำลังวิ่งในสนามหญ้า'
15
+ model = AutoModel.from_pretrained("patomp/thai-light-multimodal-clip-and-distill", trust_remote_code=True)
16
+
17
+ embeddings = model(text)
18
+ print("Text features shape:", embeddings.shape)
19
+
20
+ ```
21
+
22
+ Encode an image by
23
+ ```python
24
+ import torch
25
+ import clip
26
+ import requests
27
+ from PIL import Image
28
+
29
+ device = "cuda" if torch.cuda.is_available() else "cpu"
30
+ model, preprocess = clip.load("ViT-B/32", device=device)
31
+
32
+ url = "http://images.cocodataset.org/val2017/000000039769.jpg"
33
+ image = Image.open(requests.get(url, stream=True).raw)
34
+ image = preprocess(image).unsqueeze(0).to(device)
35
+
36
+ with torch.no_grad():
37
+ image_features = model.encode_image(image)
38
+
39
+ print("Image features shape:", image_features.shape)
40
+ ```
41
+
42
+ ## Benchmark
43
+
44
+ On the test set of [Thai MS COCO 2014 dataset](https://huggingface.co/datasets/patomp/thai-mscoco-2014-captions)
45
+
46
+ | Model \ Metrics | text-find-image recall@1 | text-find-image recall@10 | image-find-text recall@1 | image-find-text recall@10 | # text samples per second* |
47
+ | :--- | --- | --- | --- | --- | --- |
48
+ | **Multilingual Encoder** | | | | | |
49
+ | [clip-ViT-B-32-multilingual-v1](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1) | 0.075 | 0.242 | 0.096 | 0.286 | - |
50
+ | [XLM-Roberta-Large-Vit-B-32](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-32) | **0.226** | **0.565** | **0.265** | **0.596** | 20 |
51
+ | **Thai Encoder (WangchanBERTa-based)** | | | | | 48 |
52
+ | [Thai-Cross-CLIP](https://github.com/vikimark/Thai-Cross-CLIP) | 0.167 | 0.475 | 0.197 | 0.523 | |
53
+ | **Thai Encoder (Thai2Fit-based)** | | | | | |
54
+ | [thai-light-multimodal-clip-and-distill](https://huggingface.co/patomp/thai-light-multimodal-clip-and-distill) | 0.082 | **0.328** | 0.118 |**0.401**| 450 |
55
+ | [thai-light-multimodal-distill](https://huggingface.co/patomp/thai-light-multimodal-distill) | **0.084** | 0.319 | 0.122 |**0.401**| 450 |
56
+
57
+ ## Reference
58
+
59
+ Some part of this content referenced from https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-32