Paper: Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations (ICLR-2024)

Reviewer comment (Experiments):
BabelNet reliance. This work relies entirely on BabelNet and assumes that the mapping between WordNet and other resources is high quality. However, BabelNet is automated and has a known percentage of error, potentially affecting the label mapping [1];

Author response:
Yes, BabelNet is created automatically and thus not not error-free. But quoting [1]  “For example, in BabelNet 5.0 more than 90% of the mapping between Wikipedia pages and WordNet synsets has been manually validated by experts, resulting in an overall mapping precision above 99.5%.” & “Wiktionary entries are now integrated automatically using a BERT-based neural model, finetuned to associate a word sense definition to its correct synset, attaining an F1 score of 92% on a manually annotated test set.” . 

This suggests an error rate that is, to us, in an acceptable range (and substantially lower than any machine translation of concepts in isolation would yield). There is a clear trade-off between the range of “translation” errors and the number of supported languages: manual annotation is (virtually) error free, but does not scale to many languages; MT scales to hundreds of languages, but is very error-prone for concept translation; by “trusting” BabelNet, we can greatly increase the number of languages, if we accept the small error rate present in the resource – getting this way close to a “sweet spot” of this trade-off between the error rate and number of supported languages. In sum, Babel-ImageNet offers a fairly reliable evaluation for a large number of languages, without expensive human annotation, and helps in particular with low(er)-resource languages.

The correlation with retrieval datasets (Figure 3) and the near-perfect correlation w.r.t. the four existing translated ImageNet variants (Arabic, Chinese, Japanese, and Italian) in Figure 7 (Appendix) further empirically validate our BabelNet-derived translations.

Reviewer comment (Experiments):
Using WordNet synsets for translations may introduce limitations, as not all concepts or words have direct equivalents in WordNet or BabelNet, potentially impacting the completeness of translations for some languages.

Author response:
Can you clarify this point? ImageNet itself is constructed from WordNet synsets so all classes map to a WordNet synset and BabelNet is constructed also from WordNet so all WordNet synsets also appear in BabelNet. The availability of translations in any particular language, however, is affected by other factors, e.g., is there a WordNet equivalent in a language, how big is the Wikipedia/Wikidata of that language, etc.

Reviewer comment (Evaluation):
Considering that the paper belongs to the "datasets and benchmarks" area, the methodology employed (mapping from ImageNet to WordNet and then to BabelNet) is expectedly straightforward. However, I think there's also some weakness in the data cleaning and validation since the obtained multilingual data is used to evaluate models, but those same benchmarks cannot be used to assess the quality of the data itself;

Author response:
We have commented on the reliability of BabelNet in the first point above. Also, we have validated (Section 5) the meaningfulness of Image-BabelNet via the (small) set of languages that also exist in retrieval benchmarks, showing substantial correlation as well as for the four languages for which translations of ImageNet exist (Appendix, Figure 7). See also our reply in relation to BabelPic below.

Reviewer comment (Experiments):
The paper's process of removing words with identical English counterparts in the class label translation and cleaning may not be fully justified, as there can be legitimate shared words between the English and language-specific vocabulary;

Author response:
This is an accurate observation – there are such instances, for example, ‘Zebra’ or ‘Axolotl’. However, as we write in §3, what the multilingual models will use in those cases are their more accurate/stronger English representation, diluting the insight into how good the representational quality is for the other language. By keeping such translations, as a consequence, benchmarks for other languages would “reward” the models for their English “competency”. To illustrate (with an extreme example), imagine a language X with 200 classes, 100 of which are identical to the English label. If a model achieves 50% accuracy on language X, is it actually useful for X or is it just correct on the subset of the X’s concept vocabulary that overlaps with the “English” one? By filtering out all English-matching words, we avoid this problem.

Reviewer comment (Experiments):
There's a missing reference to another text-image dataset produced by *manual* annotation over BabelNet synsets [1]. In general, I'm curious about the possible usage of this dataset as a proxy to evaluate at least the Babel-ImageNet methodology part that performs prompt translation. I'd like to kindly ask the authors what they think about it or if they have alternative ideas toward strengthening data validation, which I think is the strongest weakness in this current version of the manuscript;

Author response:
Thank you for the reference!

Could you clarify what you mean by “the Babel-ImageNet methodology part that performs prompt translation”  because we do not understand how that relates to BabelPic? 

Regarding BabelPic methodology, they want to automatically verify if an image and their non-concrete concepts (e.g. “sadness”) match and they test if visual QA models can perform this verification using gold-label human-annotated image-concept data. For Babel-ImageNet, the verification of the match between the WordNet synset and image was already performed by humans in the construction of ImageNet itself.

Reviewer comment (Writing):
Not a direct weakness, so I'm listing it here: multiple usages of the verb "demonstrate" (e.g., "we demonstrate that OpenCLIP performance strongly correlates with the distribution of languages in LAION5B"). Personally, given the empirical nature of this work, I wouldn't suggest using such a theoretical-related term.

Author response:
Thank you for the suggestion. We were not aware that “demonstrate” has such a strong theoretical meaning. We were using “demonstrate” simply as a synonym for “show” (and not a synonym for “prove”). We are happy to change this and simply replace instances of “demonstrate” with “show”.

Reviewer comment (Novelty):
My concern is about the simplicity of the method, which merely translates English ImageNet labels using BabelNet.

Author response:
Thank you for your review!
> My concern is about the simplicity of the method, which merely translates English ImageNet labels using BabelNet. While the resulting benchmark proves useful, the method's contribution appears to be limited

See our general response to all reviewers. We do not disagree that our creation method is straightforward and simple, but we see do not perceive this as a weakness.

Reviewer comment (Novelty):
While the resulting benchmark proves useful, the method's contribution appears to be limited.

Author response:
Thank you for your review!
> My concern is about the simplicity of the method, which merely translates English ImageNet labels using BabelNet. While the resulting benchmark proves useful, the method's contribution appears to be limited

See our general response to all reviewers. We do not disagree that our creation method is straightforward and simple, but we see do not perceive this as a weakness.

Reviewer comment (Experiments):
Have you considered using GPT-4/ChatGPT to prompt the model to translate English ImageNet labels? Perhaps combining GPT-4/ChatGPT with WordNet could yield better results.

Author response:
We have not but it is an interesting idea to explore: LLMs are surprisingly good at translations and if we provide them additional context (e.g. via WordNet synonyms or definitions), we could likely reduce the problem of polysemy that designated NMT models struggle with. However, the main problem here is the same as in traditional MT models: performance degrades massively for low-resource languages, which makes human verification necessary again, which is exactly the problem we want to avoid by exploiting BabelNet which is massively multilingual and readily available.