vinid commited on
Commit
1a47e59
1 Parent(s): a44a03a

new stuff in readme.md

Browse files
Files changed (1) hide show
  1. readme.md +39 -3
readme.md CHANGED
@@ -36,9 +36,14 @@ However, this kind of text, without more information, is not useful to learn a g
36
 
37
  Example: ....
38
 
39
- + MSCOCO-IT.
 
 
40
 
41
- + Conceptual Captions.
 
 
 
42
 
43
 
44
  ## Better Augmentations
@@ -66,6 +71,10 @@ To better understand how well our clip-italian model works we run an experimenta
66
 
67
  ### mCLIP
68
 
 
 
 
 
69
  ### Experiments Replication
70
  We provide two colab notebooks to replicate both experiments.
71
 
@@ -77,15 +86,25 @@ We selected two different tasks:
77
 
78
  ### Image Retrieval
79
 
 
 
 
 
80
  | MRR | CLIP-Italian | mCLIP |
81
  | --------------- | ------------ |-------|
82
  | MRR@1 | **0.3797** | 0.2874|
83
  | MRR@5 | **0.5039** | 0.3957|
84
  | MRR@10 | **0.5204** | 0.4129|
85
 
 
 
 
86
  [Colab: Image Retrieval Evaluation](https://colab.research.google.com/drive/1bLVwVKpAndpEDHqjzxVPr_9nGrSbuOQd?usp=sharing)
87
 
88
- ### Zero-shot classification
 
 
 
89
 
90
  | Accuracy | CLIP-Italian | mCLIP |
91
  | --------------- | ------------ |-------|
@@ -96,11 +115,28 @@ We selected two different tasks:
96
 
97
  [Colab: ImageNet Zero Shot Evaluation](https://colab.research.google.com/drive/1zfWeVWY79XXH63Ci-pk8xxx3Vu_RRgW-?usp=sharing)
98
 
 
 
 
 
 
99
  ## Qualitative Evaluation
100
 
 
 
 
 
101
  # Broader Outlook
102
 
 
 
 
 
 
 
 
103
 
 
104
 
105
  # Other Notes
106
  This readme has been designed using resources from Flaticon.com
 
36
 
37
  Example: ....
38
 
39
+ + MSCOCO-IT. This image-caption dataset comes from the work by Antonio et al., 2019. The captions comes from the original
40
+ MSCOCO dataset and have been translated with Microsoft Translator. The 2017 version of the MSCOCO training set contains more than
41
+ 100K images, for each image more than one caption is available.
42
 
43
+ + Conceptual Captions. This image-caption dataset comes from the work by Sharma et al., 2018. There are more than 3mln image-caption pairs in
44
+ this dataset and these have been collected from the web. We downloaded the images with the URLs provided by the dataset, but we
45
+ could not retrieve them all. Eventually, we had to translate the captions to Italian. We have been able to collect
46
+ a dataset with 700K translated captions.
47
 
48
 
49
  ## Better Augmentations
 
71
 
72
  ### mCLIP
73
 
74
+ The multilingual CLIP (henceforth, mCLIP), is a model introduced by [Nils Reimers](https://www.sbert.net/docs/pretrained_models.html) in his
75
+ [sentence-transformer](https://www.sbert.net/index.html) library. mCLIP is based on a multilingual encoder
76
+ that was created through multilingual knowledge distillation (see Reimers et al., 2020).
77
+
78
  ### Experiments Replication
79
  We provide two colab notebooks to replicate both experiments.
80
 
 
86
 
87
  ### Image Retrieval
88
 
89
+ This experiment is run against the MSCOCO-IT validation set (that we haven't used in training). Given in input
90
+ a caption, we search for the most similar image in the MSCOCO-IT validation set. As evaluation metrics
91
+ we use the MRR.
92
+
93
  | MRR | CLIP-Italian | mCLIP |
94
  | --------------- | ------------ |-------|
95
  | MRR@1 | **0.3797** | 0.2874|
96
  | MRR@5 | **0.5039** | 0.3957|
97
  | MRR@10 | **0.5204** | 0.4129|
98
 
99
+ Is it true that we used MSCOCO-IT in training, and this might give us an advantage. However the original CLIP model was trained
100
+ on 400million images (and some of them probably were from MSCOCO).
101
+
102
  [Colab: Image Retrieval Evaluation](https://colab.research.google.com/drive/1bLVwVKpAndpEDHqjzxVPr_9nGrSbuOQd?usp=sharing)
103
 
104
+ ### Zero-shot image classification
105
+
106
+ This experiment replicates the original one run by OpenAI on zero-shot image classification.
107
+
108
 
109
  | Accuracy | CLIP-Italian | mCLIP |
110
  | --------------- | ------------ |-------|
 
115
 
116
  [Colab: ImageNet Zero Shot Evaluation](https://colab.research.google.com/drive/1zfWeVWY79XXH63Ci-pk8xxx3Vu_RRgW-?usp=sharing)
117
 
118
+ Our results confirm that CLIP-Italian is very competitive and beats mCLIP on the two different task
119
+ we have been testing. Note, however, that our results are lower than those shown in the original OpenAI
120
+ paper (see, Radford et al., 2021), considering that our results are in line with those obtained by mCLIP we think that
121
+ the translated image labels might have had an impact on the final scores.
122
+
123
  ## Qualitative Evaluation
124
 
125
+ ### Colors
126
+
127
+ ### Numbers
128
+
129
  # Broader Outlook
130
 
131
+ # References
132
+
133
+ Antonio, S., Croce, D., & Basili, R. (2019). Large scale datasets for Image and Video Captioning in Italian. IJCoL. Italian Journal of Computational Linguistics, 5(5-2), 49-60.
134
+
135
+ Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018, July). Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2556-2565).
136
+
137
+ Reimers, N., & Gurevych, I. (2020, November). Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 4512-4525).
138
 
139
+ Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML.
140
 
141
  # Other Notes
142
  This readme has been designed using resources from Flaticon.com