Paper: mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs (ICLR-2024)

Reviewer comment (Experiments):
One weakness is to name a translated dataset as a multilingual dataset, which doesn't cover the real benefits of multilingual at all (i.e. multiple cultures, examples from different locations from local people speaking those languages). This is misleading as the future work will use the translated dataset which will reinforce the emphasis of English centric research, under the translated "multilingual" directions.

Author response:
We fully agree with the reviewer, but want to point out that this is, unfortunately, a general problem in multilingual vision-language models. For example, CCLM, Ernie-UniX2, or UC2 all also train with machine-translated data. PaLI models are trained with both machine-translated data and (proprietary) multilingual web-crawled data. There is multilingual caption data available in Laion5B but these are noisy web-crawled captions. For high-quality captions and other tasks like VQA, there is only English data available.

However, our evaluation setup does take this problem into account and includes test datasets that require both multilingual and multicultural knowledge: XM3600 in particular selected and annotated images from 36 countries with speakers from those countries, and MaRVL is built from the ground up (and also annotated by native people) to include images and concepts from 5 different countries. 

Nonetheless, you make a valid point and we will make sure to add a short discussion on true multilingual data vs. “multilingual” data obtained through translation and disadvantages/pitfalls of the latter.

Reviewer comment (Novelty):
The idea of applying Q-Former to LLMs is not novel and has been explored by the previous BLIP papers. I have not found BLIP baselines reported (at least on EN) in the paper (except for the right sub-table of Table 1). More comparisons with BLIP baselines, especially on the impact of multilingual data to the original English tasks would be nice to show the impact of this paper.

Author response:
The comparison with BLIP-2 models is difficult because their underlying English LLMs are better on English tasks than multilingual models. Monolingual language-specific LLMs (especially if the language is English) generally outperform multilingual counterparts for that particular language. This phenomenon (i.e., the downside of sharing representation space across many languages) is known as the curse of multilinguality [1,2].  

We did consider including the zero-shot GQA results for BLIP-2 and InstructBLIP Flan-T5-XL results in Table 2 (44.0 and 48.4% accuracy, our mBLIP mT0-XL has 42.6) but decided against it for the sake of readability of Table 2, which is already quite full with information.

Reviewer comment (Experiments):
The comparison of mBLIP with PaLI is quite unclear. Is mBLIP a zero-shot model or finetuned model? If it's the latter then it doesn't make any sense to claim that mBLIP outperforms (zero-shot) PaLI.

Author response:
This is a good question with no straightforward answer:

PaLI-X is pre-trained with (amongst other data) millions of captions translated to the 36 XM3600 languages. For fine-tuning, they are then trained on ~5M examples of MSCOCO again translated to the 36 languages. 

mBLIP is trained with our task mix for 2 epochs, which includes 400k MSCOCO captions automatically translated to 95 languages. 

So on one hand, mBLIP has seen data for some of the downstream fine-tuning tasks in its re-alignment training (but orders of magnitude less than the fine-tuned PaLI) but on the other hand, we did not optimize our setup for the 36 XM3600 languages (and PaLI does). 

To make the comparison more complicated, our mBLIP models have 5 to 8B parameters of which we train only 128M. Pali-X has 55B parameters, all of which are updated during fine-tuning training.

In sum, this means that mBLIP has some advantage regarding the (type of the) training data, whereas Pali-X has an advantage in terms of training languages (matching exactly those from XM3600), model size, and size of the (pre)training data. We thus believe that it is overall fair to compare mBLIP against Pali-X (zero-shot).

Reviewer comment (Novelty):
The mixture of task highlighted in the paper as a contribution to the success of the model, although not considered in BLIP2, was actually explored in PALI. This fact further deteriorates the novelty of this work.

Author response:
It would appear that we forgot to cite Pali when motivating our task mix in §3.2; we will fix this. We did however cite InstructBLIP, another Vision-LLM trained with a task mix, so we did not intend to claim this as a particular novelty of our approach.

However, unlike Pali, who use a rigid (and English-only) input format for instructions, we use diverse templates and additionally incorporate translated user instruction data (from Llava) so that our mBLIP models can handle a wider range of user inputs as illustrated in Figure 3.

Reviewer comment (Evaluation):
It's not a surprise that task mix helps most in VQA and VNLI tasks because such an objective in included in the pre-training stage. The mixture of task actually hurts the captioning performance.

Author response:
When comparing the results in Table 3 (row 2 for captions only, last row for the task mix), we do not see that the task mix hurts captioning performance: while English results for XM3600 and xFlickrCo are slightly lower, multilingual results are higher.

Also, VNLI is not included in the task mix, which is why our model has poor zero-shot results there (in zero-shot, i.e., without fine-tuning for VNLI, it never predicts the neutral class).

Reviewer comment (Novelty):
The model can be trained on 4 cheap RTX-3090 GPUs, which is great. But this claim in the conclusion has nothing to do with the contribution of the work, i.e., any similar models will enjoy the speed-up with 8-bit quantization and LoRA PEFT.

Author response:
First, while LoRA and 8-bit quantization do contribute to the training efficiency, the core driver of our efficient training is the (comparatively small) size of our training data and our task mix – precisely because we re-align a pre-trained “English” Q-former to a new, multilingual LLM, we are able to achieve multilingual performance competitive to that of much larger models that have been trained from scratch with orders of magnitude less training data.

Second, the fact that our approach works with other models should be seen as a positive: if next week, a new SOTA multilingual LLM is released, we have a proven computationally effective method of inducing a corresponding Vision-LLM.

Reviewer comment (Experiments):
The paper does not state the data portion used to pre-train the model. In the model pre-training, 3 tasks are considered, but the captioning dataset looks much bigger than VQAs. Meanwhile, the small size of VQA could contribute less to comprehensive pre-training, but only provide some demonstration to downstream QA tasks (that's probably why QA and VNLI tasks perform well).

Author response:
We do report this. In §3.2, we refer to Appendix C.1 for details of our training data where table 5 lists the exact number of images and samples used from each dataset.

As can be seen there, we limit the caption datasets to prevent this very problem you describe where the larger caption dataset would (proportionally) overshadow the other tasks.

Reviewer comment (Experiments):
The paper claims that based on a few prior works, the re-alignment does not need larger dataset. However, some of the listed prior works (on arXiv) have very weak quantitative studies, which may not support authors' claim. Thus, such a claim has to be validated by ablation studies like re-alignment with different size of pre-training data, e.g., 3M, 12M, 120M, 400M, etc.

Author response:
We cite two works to support this claim. We can agree that Zhu et al., 2023 lacks proper quantitative evaluation; Zhang et al., 2023, however, features extensive experiments that clearly indicate that one can get comparable results with re-alignment on far less data. 

While the suggested experiment would be interesting – we do not dispute that with more data, you can (very likely) get better results, but we also suspect that the gains would diminish logarithmically with more data. The proposed data-scaling experiment/ablation is also too expensive for us to run: training with 100M examples would take at least 40 days on our compute.

In addition, there is another problem: for the greater data scale (>100M images), only image-caption data is available, which would greatly skew the proportions in our task mix.