Paper: Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages (ICLR-2024)

Reviewer comment (Evaluation):
The paper claims to “generalize to other languages in a zero-shot manner.” This is not true, and very misleading. In fact, the chat model had to be fine-tuned on target language data, otherwise it would generate text in English. Calling the behavior “quasi-zero-shot” is also (i) different from the literature, and especially (ii) misleading: the model cannot generate text in the target language without supervision. Such claims, while valid for the text-to-image model, should be removed.

Author response:
**Q1: Claim about 'quasi-zero-shot' Term**       
In MLLM literature, to the best of our knowledge, there has been no report on the phenomenon where a model exclusively pre-trained on English multimodal data, can provide accurate responses in English for instructions in other languages. In this context, the foundational MLLM capability has been successfully generalized from English to the target language, requiring merely a switch in language to fully realize this capability. On the one hand, this demonstrates a remarkable strength: the model's ability to comprehend and respond to Chinese multimodal instructions without being exposed to any Chinese multimodal data. On the other hand, such capability is imperfect as the model cannot respond in Chinese, whereas only very little instructing tuning data is needed to switch the language. Therefore, we use the term “quasi-zero-shot” to describe this subtle capability. 

We appreciate your valuable feedback and have refined the claim to a more precise expression that "generalize to other languages in a (quasi-)zero-shot manner", based on the quasi-zero-shot performance on image-to-text generation and zero-shot performance on text-to-image generation.

Reviewer comment (Experiments):
The paper claims to propose “an effective training paradigm for training large multimodal models in low-resource languages.” However, none of the languages tested were low-resource. There is a significant difference in the representations of the LM model between high- and low-resource languages, as well as in the quality of translations that an MT system can provide for those languages. None of these things are true for Chinese. Any claims about performance and applicability to low-resource languages need to be removed as they are never verified.

Author response:
**Q2: Language Choice**
- The claim about low-resource language. In this paper, we refer to the low-resource language under a multimodal scenario. Please refer to the general response for a more detailed discussion. We have revised our paper to avoid misunderstanding with the term ‘low-resource’ language.
- Language Diversity. Thanks for your advice! We have expanded our evaluation on a larger range of languages based on the IGLUE benchmark, which covers more diverse languages and obtains good performance, such as Russian in Cyrillic script with a 47.68 score on xGQA. We leave a wider range of language models in future work.

Reviewer comment (Experiments):
The paper also claims to be multilingual. Yet, most of the experiments are done in a bilingual setup (English+Chinese) or in a few, high-resource European languages. These languages share the same script, similar topology. Claims about multilinguality should be supported by a better selection of languages that increase diversity in scripts, topology, and resource availability.

Author response:
**Q2: Language Choice**
- The claim about low-resource language. In this paper, we refer to the low-resource language under a multimodal scenario. Please refer to the general response for a more detailed discussion. We have revised our paper to avoid misunderstanding with the term ‘low-resource’ language.
- Language Diversity. Thanks for your advice! We have expanded our evaluation on a larger range of languages based on the IGLUE benchmark, which covers more diverse languages and obtains good performance, such as Russian in Cyrillic script with a 47.68 score on xGQA. We leave a wider range of language models in future work.

Reviewer comment (Experiments):
The model is evaluated on unusual benchmarks for multilingual multimodal modeling. These datasets are obtained through machine translation of the corresponding English dataset. The authors claim to fix minor errors (only for Chinese, and not even for all the other 6 European languages), yet do not provide any indication of the quality of the resulting test data.

Author response:
**Q3: Experimental Setup**
  - **Quality of test data**  
  For the Chinese test data, we manually check the data quality. This operation has been adopted in previously well-recognized benchmarks such as IGLUE to ensure the test data quality.      
  For other languages, note that directly using machine translation to translate the test benchmark without manual checking is also widely adopted in the literature [1][2][3]. Nonetheless, we agree that evaluation on manually validated benchmarks can better support the findings. During the response period, we additionally evaluate the performance of these languages on manually validated benchmark IGLUE.

Reviewer comment (Evaluation):
The evaluation scores are based on GPT-4, which might not understand languages beyond English well. This is a hypothesis that might or might not hold true, but the fact that such concern exists is enough to undermine the evaluation setup used by the authors.

Author response:
**Q3: Experimental Setup**
  - **Using GPT-4 as evaluator**
    - **Feasibility**: GPT-4 has been widely accepted as an evaluator for AI chatbots, as also discussed in AlpacaEval [4], MTBench [5], and LLaVA Bench [6]. For example, the AlpacaEval benchmark is an authoritative benchmark for evaluating LLMs in following open-ended instructions. It uses GPT-4 as an automatic evaluator and shows a 0.94 Pearson correlation with human judgment. Our focus is to evaluate the performance of models in open-ended multimodal chats. Compared with traditional string-match-based metrics, GPT-4 evaluation is found to be more robust against the high diversity of natural language. 
    - **Multilingual Capability**: As for the multilingual capability, GPT-4 presents its strong multilingual capability on MMLU across languages in its report, which shows only a slight drop in the language we chose compared with English. We note that previous works [7][8] in the literature also use GPT-4 (ChatGPT) to evaluate multilingual tasks. For example, GEMBA [7] is a machine translation metric based on GPT-4 evaluation. We believe these results from the literature show the soundness of using GPT-4 to evaluate mainstream European languages.

Reviewer comment (Experiments):
There exist multilingual multimodal benchmarks. For instance, IGLUE tests for 4 multimodal tasks in 20 languages, with data *manually* collected by native speakers. MaRVL and XM3600 even include images sourced from the countries where a language is spoken. These benchmarks are adopted in the literature and completely missing (even as part of the related work) in this paper.

Author response:
**Q3: Experimental Setup**
  - **Performance on IGLUE benchmark**  
  Thanks for your valuable advice. We perform an additional evaluation on the IGLUE. Please refer to the general response for the results.  

**Q4: Missing related work**  
Thanks for your suggestions. We will incorporate the missing related work and benchmark in the final version.

Reviewer comment (Novelty):
There is a lack of discussion of relevant related work. Models like {m,x}UNITER, CCLM, and TD-MML are missing. The latter in particular shows how machine translation data can be very helpful during pretraining if filtering out poor translations, which provides a contrasting point to the one made by the authors in this paper. A discussion of these findings and how they differ is important to guide the community.

Author response:
**Q4: Missing related work**  
Thanks for your suggestions. We will incorporate the missing related work and benchmark in the final version.

*****

**Q5: Inconsistency with the previous conclusion**  
In our work, we use a powerful multimodal LLM (10B) as the backbone language model, which already possesses strong cross-lingual generalization capability. In this case, only a small number of data in the target language is needed during SFT periods to switch on the target language in form. In this context, introducing more translation data can only bring a little marginal improvement.  
In contrast, TD-MML [1] employs translation data in the context of a traditional language understanding model with relatively limited capacity and generalization capability. For such models, introducing more translation data can lead to more substantial improvements.  
Thanks for the suggestion. We will incorporate this discussion into our final version.

Reviewer comment (Experiments):
Most of the tables present absolute comparisons, showing how great the proposed method is. However that might be less interesting from a reader's perspective without proper baselines and ablation tests as discussed in the above section (due to different dataset mixtures and tuning tricks used). It's highly recommended to take a few tables, and perform multiple in-depth analysis and comparisons with controlled study, to make sure there are enough take aways and confidence in the current MPM design choices.

Author response:
**Q3: In-depth analysis and comparisons with controlled study**           
We highly agree that analysis in controlled comparison is valuable for readers. For the main focus, we conducted a comprehensive ablation study with controlled comparison to analyze the effect of data from different sources (Table 5 and discussion in Sec. 5.3), which yields the following findings:
- We ablate the languages of the pretraining dataset. The results show that relying solely on a native Chinese dataset is insufficient for achieving good performance for both image-to-text and text-to-image tasks.
- We ablate the languages of the instruction tuning dataset. The results show that English data also plays a crucial role in improving the chat capability of the model in Chinese during the instruction tuning stage.
- We ablate the filter process to the native Chinese dataset. The results show that the filtering process applied to the native dataset is essential for the model performance in Chinese.
- We ablate the translated dataset in the pretraining stage. The results show that incorporating the native Chinese dataset and the translated Chinese dataset during the pre-training stage yields a marginal improvement in the model performance.

We believe the analysis above offers a helpful know-how handbook for building strong non-English MLLMs. We will explore more data and training strategies as kindly suggested in our future work.

Reviewer comment (Writing):
There are some inconsistencies in the argumentation used throughout the paper. For example, in Section 4.1, the authors argue that translating a large-scale pretraining dataset will consume substantial computing resources, which is a fair argument. However, in Section 4.2, the authors describe that they train a version of their model called VisCPM-Chat+ using 136 million translated examples from LAION-COCO. The authors could improve their argumentation if they clarify whether it is or is not challenging to create such translated examples. Bear in mind that other researchers have already translated pretraining datasets, e.g. Thapliyal+ EMNLP 2022.

Author response:
**Q1: Why translate the dataset?**       
It is certainly true that the translation of the 136M examples did consume substantial computational resources. In our experiments, we employed the bilingual LLM, CPM-Bee 10B, for translation. The translation process consumed 4,533 GPU hours on NVIDIA A100 80G for 136 million examples. We would like to clarify that the goal of this operation is to empirically validate the hypothesis of MPM in a controlled setting, verifying whether translated datasets are necessary for training MLLMs. Our experimental results show that marginal improvement can be achieved even with high-quality large-scale datasets translated by strong bilingual LLMs. Considering the cost vs improvement, the conclusion is that translating pretraining data is unfavored for training non-English MLLMs. We hope the empirical results contribute to a better understanding of the role of data translation in training non-English MLLMs.

Reviewer comment (Experiments):
The motivation for this paper is to create multimodal models for low-resourced languages via the proposed pivoting method but the main languages used in the experiments are Chinese and English, neither of which could ever be described as low-resource languages. The remaining languages used for the experiments in Section 5.1.3 are also hardly low-resource: German, French, Spanish, Italian, and Portuguese. Using these languages and claiming that the method applies to low-resource settings affects the credibility of the conclusions that one can draw for actual low-resource languages. See, for example, Joshi+ ACL 2020 for a discussion on low-resource languages.

Author response:
**Q2: The meaning of low-resource languages**          
Thanks for your suggestion. Please refer to the general response for a more detailed discussion. In general, we focus on the low-resource nature of languages in the multimodal scenario. We have revised our paper to avoid confusion with the terminology "low-resource languages" in pure language resources.

Reviewer comment (Evaluation):
In Section 5.1, why is it methodologically sound to use GPT-4 to evaluate model output?

Author response:
**Q3: Methodological soundness to use GPT-4 for evaluation**        
GPT-4 has been widely accepted as an evaluator for AI chatbots, as also discussed in AlpacaEval [1], MTBench [2], and LLaVA Bench [3]. For example, the AlpacaEval benchmark is an authoritative benchmark for evaluating LLMs in following open-ended instructions. It uses GPT-4 as an automatic evaluator and shows a 0.94 Pearson correlation with human judgment. Our focus is to evaluate the performance of models in open-ended multimodal chats. Compared with traditional string-match-based metrics, GPT-4 evaluation is found to be more robust against the high diversity of natural language. Therefore, we follow previous works and use GPT-4 to evaluate MLLM conversation performance.

Reviewer comment (Evaluation):
In Figure 2, why is a CLIPscore threshold of 0.18 used to define high-quality? How was this determined?

Author response:
**Q4: CLIP score threshold**         
We referred to Taiyidiffusion's criteria for data volume and filtering standards (including CLIP score) in training Chinese text-to-image models. We conducted a thorough manual check for the correlation between image-text alignment and the CLIP score. Here are some specific findings about image-text pairs with low CLIP scores:
- Broken Images: Some image URLs are broken or redirected to the generic default image for downloading. 
- Unrelated captions: Some captions are entirely irrelevant to the associated image content.
- Incomplete sentences: Some captions are truncated or missing punctuation

Reviewer comment (Experiments):
Did you also use M2M-100 to translate the 136M examples in the COCO-LAION dataset?

Author response:
**Q6: Choice of Machine Translation System**          
We use the CPM-Bee 10B for all En-Zh translations due to its better performance. For other languages, we use M2M100 12B for translation in German, French, Spanish, Italian, and Portuguese.

Reviewer comment (Reproducibility):
Which version of M2M-100 did you use for translation?

Author response:
**Q6: Choice of Machine Translation System**          
We use the CPM-Bee 10B for all En-Zh translations due to its better performance. For other languages, we use M2M100 12B for translation in German, French, Spanish, Italian, and Portuguese.

Reviewer comment (Evaluation):
In Section 5.2.2, you describe an FID of 9.5 or 9.9 "comparable" to Stable Diffusion FID is 8.6. Is this a reasonable claim?

Author response:
**Q7: Conclusion about FID performance**         
We draw a conclusion on the performance comparison between our model (9.5/9.9 FID) and Stable Diffusion (8.6 FID) referring to the literature. For instance, Unidiffuser[4], with an FID of 9.7, concluded in their paper that their model's performance on single text-to-image generation tasks is "on par" with custom diffusion models such as Stable Diffusion.

Reviewer comment (Experiments):
How does your model perform on a larger set of languages in the multilingual multimodal benchmark IGLUE by Bugliarello+ ICML 2022?

Author response:
**Q8: Performance on IGLUE**          
Thanks for your advice. Please refer to the general response for the performance of MPM on IGLUE.