Paper: Beyond Language: Empowering Unsupervised Machine Translation with Cross-modal Alignment (ICLR-2024)

Reviewer comment (Experiments):
The experimental setup is arguably slightly outdated in that the paper trains a very small transformer from scratch for the task. While it is fair to do this for fair comparison to other work, the contributions of the paper could be extended and more applicable to a contemporary training setup if they used a pre-trained initialization (in lieu of stage (1) of the UMMT pipeline). This hurts the impact of the paper in its current form as it is less clear how this work could be interpreted/extended by a current reader. At a minimum, the paper should address why a pretrained model would not be appropriate here if this is the case.

Author response:
In stage 1, the choice to refrain from utilizing a pre-trained model is made to ensure a fair comparison with prior research. This decision does not imply that our approach is incapable of leveraging pre-trained models. Notably, MASS does provide pre-trained models for developers, although we opted not to use them. It is important to highlight that employing pre-trained models in stage 1 is a feasible option.

Reviewer comment (Experiments):
The work has minimal diligence for the handling of data for the task. 3 datasets are combined, chopped, shuffled and split into different groups and splits for each stage but this is not justified or introspected upon. It is not clear why the work takes the current course of action. Furthermore, it is not clear if different splits (i.e. more data in (1) or (2) would be more beneficial given the purported low-resource benefit of the method).

Author response:
In our work, to ensure a fair comparison, the datasets and the amount of data used at each stage are strictly consistent with the baseline. For Stage 1, a monolingual corpus from WMT with a size of 10.145 million was used. In Stage 2, a monolingual image-text pair dataset from Multi30k and Mscoco, with a size of 75,000, was utilized. In this step, data from both datasets were evenly distributed between the source and target languages to ensure a uniform data distribution, consistent with other UMMT baselines.

It is crucial to emphasize that the datasets and data volumes used at each stage are fixed, and there is no arbitrary allocation. The even distribution of Multi30K and MsCOCO in Stage 2 aligns consistently with other UMMT baselines. The WMT data utilized in Stage 1 is complete, and the text data from Stage 2 is employed in Stage 3, maintaining full consistency with the baseline throughout.

Reviewer comment (Experiments):
I am concerned some of the analyses and ablations are straw men. For the retrieval task, I can see little reason why MASS should be expected to produce text-to-image alignment as this was not the intention of this model. 5.4 appears to only re-enforce prior work and I am not sure why this is a core contribution. The analysis in 5.5 appears to make conclusions from a sample of 2 outputs which is not sufficient for a qualitative or quantitative conclusion.

Author response:
For the retrieval task, we did not specifically demand that the MASS model possess retrieval capabilities. Our focus was on conducting a comparison between the performance before and after cross-modal training to illustrate how cross-modal training enables the model to learn cross-modal alignment. Section 5.4 primarily analyzes the impact of different noise levels on the experimental results. It's important to note that we did not emphasize this as our core contribution. In Section 5.5, we selected two representative examples for qualitative illustration. Due to space constraints, we were unable to present more sentences. We will address this by adding an appendix in the final version to provide additional examples.

Reviewer comment (Experiments):
Comparisons to other systems are not current. For text-only comparisons there is no reference to OPUS, NLLB, M2M100, MBART50-(one,many)-to-(many,one) or any recent comparator for text-only MT. This makes your results harder to contextualize. These models can be run with the compute available to the authors.

Author response:
Firstly, comparing our model to NLLB, M2M100 is unfair as they were trained using a substantial amount of parallel corpora. Secondly, multilingual translation and unsupervised machine translation are two distinct machine translation tasks, making a direct comparison impractical. Lastly, it is crucial to emphasize that, in Stage 1, any encoder-decoder-based pre-trained model could be adopted, including LLMs like FlanT5. However, due to hardware limitations (we only have 20GB 3090s), we couldn't conduct experiments on a larger scale. We will explore further with more extensive computational resources in the future.

Reviewer comment (Writing):
The framing of the Initialization stage is confusing and needs more contextualization — is this optional after pre-training. what prior work uses or skips this? how was better pre-training changed the need for initialization? Furthermore, you describe Initialization somewhat circularly, in that you say that it is important because it happens and do not clearly state what the purpose of this stage is and why it is needed.

Author response:
In Section 2, we provide a detailed overview of the roles and purposes of the three stages. The Language Modeling (LM) stage aims to enable the model to learn monolingual representations. The Initialization stage is designed to equip the model with a rudimentary translation capability, offering a good initial value for subsequent iterative back-translation. The Iterative Back-Translation stage guides the model to learn the alignment between the source and target languages.

Certainly, the Initialization stage is not strictly necessary, similar to how iteration can start with a randomly chosen initial value. However, opting for a random initial value often leads to lower-quality iterative results and less training stability. Initialization, on the other hand, provides a solid starting point for iteration, resulting in improved iterative outcomes and enhanced training stability. This underscores the importance of the Initialization stage.

Reviewer comment (Writing):
Many spelling and grammar errors i.e. "langauge", "acheive", "We" in the middle of a sentence.

Author response:
We deeply apologize for the occurrence of such errors in the article and appreciate your pointing them out. We will promptly address and correct them in the next version.

Reviewer comment (Theory):
Why must decoders be language specific?

Author response:
The main reason for this is to reduce the accumulation of errors in Stage 3: iterative back-translation. If a unified decoder is used and translation of wrong language occurs during the back-translation process, it will accumulate in the iteration, thereby impacting the overall model training. This could result in poor training outcomes, and in some cases, the model fails to converge.

Reviewer comment (Experiments):
Did you consider using a [CLS] type token for the encoding to avoid needing to pool the encodings?

Author response:
Using [cls] is certainly feasible, and it's a very good suggestion. It's highly likely to yield better results. Thank you very much for bringing this up. We will further explore and experiment with this suggestion to observe its effects.

Reviewer comment (Experiments):
Did you consider that your improvements, compared to models trained not on image-text data, may be because you are better fitting to the distribution of sequence lengths of this kind of data which other MT models are unaware of?

Author response:
The issues you pointed out are typically present in traditional Multimodal Machine Translation (MMT). In traditional supervised MMT tasks, there is ample textual information, and the model's utilization of image information is limited, with images playing a relatively minor role. However, in Unsupervised Multimodal Translation (UMMT), the absence of textual information makes it challenging. If images are replaced with noise or chosen randomly, contrastive learning becomes impossible, and learning cross-modal alignment becomes unfeasible. This highlights the crucial role played by images in this context, rather than better fitting to the distribution of sequence lengths.

Reviewer comment (Writing):
There are some typos and grammatical errors (I missed some but here is an example. Kindly check and make the necessary corrections): a. "Therefore, we propose a novel unsupervised multi-modal method to achieve better initialization. method semantically aligns source-target languages into a shared latent space through contrastive learning, using images as pivots"

Author response:
**5. Concerns on  typos and grammatical errors**

Thank you for pointing out errors and inaccuracies in our expression and writing. We apologize for any confusion this may have caused. We will diligently address these issues and rectify the errors in the upcoming version.

Reviewer comment (Theory):
This method heavily relies on the assumption that the two source and language spaces are approximately isomorphic. What if the two spaces are not isomorphic?

Author response:
**1. Concerns on isomorphic spaces**

Firstly, we did not assume any necessity for isomorphism between the source languages, nor did we employ any normalization methods that would enforce convergence towards isomorphism among distant languages. However, the suggestions you provided from this perspective are highly valuable. Therefore, we will conduct further analysis based on your input.

Constrained by the dataset, both previous works and our work have conducted experiments in English, German, and French. For English and French, there is a substantial amount of shared vocabulary, indicating a higher degree of similarity. In the case of English and German, the differences between them are relatively greater, but they still belong to the same language family. Therefore, we chose to conduct experiments with Czech, a language that does not belong to the same language family as English. English belongs to the Indo-European language family, while Czech belongs to the West Slavic language group. The substantial differences between them fulfill the criteria for low similarity. You can refer to Table 1 below for Czech experimental results.

**Table 1 Results on Multi30K EN-CS Flickr2016 set**
|      |           |        |       |        |           |        |       |        |      |         |
|:----:|:---------:|:------:|:-----:|:------:|:---------:|:------:|:-----:|:------:|:----:|:-------:|
|      | Test_2016 |        |       |        | Test_2018 |        |       |        | Avg  |         |
|      | en-cs     |        | cs-en |        | en-cs     |        | cs-en |        |      |         |
|      | BLEU      | METEOR | BLEU  | METEOR | BLEU      | METEOR | BLEU  | METEOR | BLEU | METEOR  |
| MASS | 20.1      | 23.9   | 27.1  | 29.3   | 16.1      | 21.2   | 22.3  | 27.1   | 21.4 | 25.4    |
| Ours | **24.2**      | **26.4**   | **30.8**  | **32.2**   | **20.0**        | **24.1**   | **26.6**  | **30.4**    |**25.4** | **28.3**    |

Reviewer comment (Theory):
What happens to your model when the two spaces are not isomorphic?

Author response:
**1. Concerns on isomorphic spaces**

Firstly, we did not assume any necessity for isomorphism between the source languages, nor did we employ any normalization methods that would enforce convergence towards isomorphism among distant languages. However, the suggestions you provided from this perspective are highly valuable. Therefore, we will conduct further analysis based on your input.

Constrained by the dataset, both previous works and our work have conducted experiments in English, German, and French. For English and French, there is a substantial amount of shared vocabulary, indicating a higher degree of similarity. In the case of English and German, the differences between them are relatively greater, but they still belong to the same language family. Therefore, we chose to conduct experiments with Czech, a language that does not belong to the same language family as English. English belongs to the Indo-European language family, while Czech belongs to the West Slavic language group. The substantial differences between them fulfill the criteria for low similarity. You can refer to Table 1 below for Czech experimental results.

**Table 1 Results on Multi30K EN-CS Flickr2016 set**
|      |           |        |       |        |           |        |       |        |      |         |
|:----:|:---------:|:------:|:-----:|:------:|:---------:|:------:|:-----:|:------:|:----:|:-------:|
|      | Test_2016 |        |       |        | Test_2018 |        |       |        | Avg  |         |
|      | en-cs     |        | cs-en |        | en-cs     |        | cs-en |        |      |         |
|      | BLEU      | METEOR | BLEU  | METEOR | BLEU      | METEOR | BLEU  | METEOR | BLEU | METEOR  |
| MASS | 20.1      | 23.9   | 27.1  | 29.3   | 16.1      | 21.2   | 22.3  | 27.1   | 21.4 | 25.4    |
| Ours | **24.2**      | **26.4**   | **30.8**  | **32.2**   | **20.0**        | **24.1**   | **26.6**  | **30.4**    |**25.4** | **28.3**    |

Reviewer comment (Experiments):
A more comprehensive experiment on more languages is needed. The shared space of EN-FR, and EN-DE, is highly approximately isomorphic even under procrustes. Include other languages and report them to make your model more generalizable.

Author response:
**1. Concerns on isomorphic spaces**

Firstly, we did not assume any necessity for isomorphism between the source languages, nor did we employ any normalization methods that would enforce convergence towards isomorphism among distant languages. However, the suggestions you provided from this perspective are highly valuable. Therefore, we will conduct further analysis based on your input.

Constrained by the dataset, both previous works and our work have conducted experiments in English, German, and French. For English and French, there is a substantial amount of shared vocabulary, indicating a higher degree of similarity. In the case of English and German, the differences between them are relatively greater, but they still belong to the same language family. Therefore, we chose to conduct experiments with Czech, a language that does not belong to the same language family as English. English belongs to the Indo-European language family, while Czech belongs to the West Slavic language group. The substantial differences between them fulfill the criteria for low similarity. You can refer to Table 1 below for Czech experimental results.

**Table 1 Results on Multi30K EN-CS Flickr2016 set**
|      |           |        |       |        |           |        |       |        |      |         |
|:----:|:---------:|:------:|:-----:|:------:|:---------:|:------:|:-----:|:------:|:----:|:-------:|
|      | Test_2016 |        |       |        | Test_2018 |        |       |        | Avg  |         |
|      | en-cs     |        | cs-en |        | en-cs     |        | cs-en |        |      |         |
|      | BLEU      | METEOR | BLEU  | METEOR | BLEU      | METEOR | BLEU  | METEOR | BLEU | METEOR  |
| MASS | 20.1      | 23.9   | 27.1  | 29.3   | 16.1      | 21.2   | 22.3  | 27.1   | 21.4 | 25.4    |
| Ours | **24.2**      | **26.4**   | **30.8**  | **32.2**   | **20.0**        | **24.1**   | **26.6**  | **30.4**    |**25.4** | **28.3**    |

Reviewer comment (Evaluation):
By reporting metrics such as singular value gap and effective condition number (see https://aclanthology.org/2020.emnlp-main.186.pdf and https://openreview.net/forum?id=Nh7CtbyoqV5) could show how the shared representation become after your proposed method is applied. A before and after table should be good to report.

Author response:
**2. Concerns on singular value gap and effective condition number**

Thank you very much for bringing up these metrics. We can use these metrics to quantitatively represent how the shared representation become after your proposed method is applied. Therefore, we have reported the status of **Effective Rank, Effective Condition Number,  Singular Value Gap** in the table 2, 3, 4 below.

**Table 2  Effective Rank**
|                    |     |     |      |
|:------------------:|:---:|:---:|:----:|
|                    | EN  | DE  | FR   |
| Before Contrastive | 207 | 225 | 207  |
| After Contrastive  | 224 | 232 | 222  |

**Table 3 Effective Condition Number**
|                    |      |      |       |
|:------------------:|:----:|:----:|:-----:|
|                    | EN   | DE   | FR    |
| Before Contrastive | 24.3 | 19.4 | 24.3  |
| After Contrastive  | 16.7 | 17.4 | 17.1  |


**Table 4 Singular Value Gap**
|                    |       |        |
|:------------------:|:-----:|:------:|
|                    | EN-DE | EN-FR  |
| Before Contrastive | 63.3  | 4.7   |
| After Contrastive  | 2.7   | 0.1    |


From the experimental results, it can be observed that before contrastive learning, the singular value gap between German and English is larger compared to English and French. This indicates a greater dissimilarity between English and German (63.3) than between English and French (4.7). Through contrastive learning, the singular value gaps in both English-German and English-French pairs significantly reduced, and the effective condition number also decreased relatively. This further emphasizes the effectiveness of our method.

Reviewer comment (Experiments):
This work primarily evaluates English-German and English-French translations, which are typically high-resource translation tasks. It would be valuable to see an evaluation of real low-resource languages to better gauge the method's effectiveness in such scenarios.

Author response:
**1. Concerns on real low-resource languages**

Constrained by the dataset, both previous works and our work have conducted experiments in English, German, and French. However, it is necessary and valuable to validate our approach on truly low-resource languages. Therefore, we conducted additional experiments using a Czech language dataset from the Multi30K Test2016 and Test2018 datasets. 
Our method demonstrates a substantial improvement compared to the text-only baseline, with an average BLEU score increase of 4.0 and an average METEOR score increase of 2.9 across all the language directions. This indicates that our method also performs well on real low-resource languages.

**Table 1 Results on Multi30K EN-CS Flickr2016 set**
|      |           |        |       |        |           |        |       |        |      |         |
|:----:|:---------:|:------:|:-----:|:------:|:---------:|:------:|:-----:|:------:|:----:|:-------:|
|      | Test_2016 |        |       |        | Test_2018 |        |       |        |      |         |
|      | en-cs     |        | cs-en |        | en-cs     |        | cs-en |        |      |         |
|      | BLEU      | METEOR | BLEU  | METEOR | BLEU      | METEOR | BLEU  | METEOR | BLEU | METEOR  |
| MASS | 20.1      | 23.9   | 27.1  | 29.3   | 16.1      | 21.2   | 22.3  | 27.1   | 21.4 | 25.4    |
| Ours | **24.2**      | **26.4**   | **30.8**  | **32.2**   | **20.0**        | **24.1**   | **26.6**  | **30.4**    |**25.4** | **28.3**    |

To further supplement our findings, we conducted experiments on English-to-German and English-to-French translation under low-resource settings, specifically using only 75,000 monolingual data samples. 

**Table 2 Results on low-resource setting (75k data only).**
|      |       |        |       |        |       |        |       |        |      |         |
|:----:|:-----:|:------:|:-----:|:------:|:-----:|:------:|:-----:|:------:|:----:|:-------:|
|      | en-de |        | de-en |        | en-fr |        | fr-en |        | Avg  |         |
|      | BLEU  | METEOR | BLEU  | METEOR | BLEU  | METEOR | BLEU  | METEOR | BLEU | METEOR  |
| MASS | 18.3  | 35.8   | 19.7  | 23.4   | 24    | 41.5   | 23.9  | 27     | 21.5 | 31.9    |
| Ours | **27.5**  | **47.6**   | **32.3**  | **32.2**   | **36.7**  | **54.6**   | **36.9**  | **34.3**   | **33.6** | **42.2**    |

The experimental results indicate that in simulated low-resource scenarios, our method continues to yield significant improvements. In contrast, text-only method without large-scale monolingual pretraining experiences a severe performance drop, and the model convergence during training is notably slower.

Reviewer comment (Experiments):
The paper should provide insights into the effectiveness of the alignment method when applied to languages with low similarity. This would offer a more comprehensive understanding of its performance across various language pairs.

Author response:
**2. Concerns on languages with low similarity**

For English and French, there is a substantial amount of shared vocabulary, indicating a higher degree of similarity. In the case of English and German, the differences between them are relatively greater, but they still belong to the same language family. Therefore, we chose to conduct experiments with Czech, a language that does not belong to the same language family as English. English belongs to the Indo-European language family, while Czech belongs to the West Slavic language group. The substantial differences between them fulfill the criteria for low similarity. 
You can refer to Table 1 above for Czech experimental results.

Reviewer comment (Novelty):
In comparison to methods that rely on bilingual dictionaries to enhance alignment, such as denoising synthetic code-switched data, the paper should discuss whether the introduction of images offers clear advantages or if these two approaches are complementary.

Author response:
**3. Concerns on comparison to methods that rely on bilingual dictionaries**

Actually, we have already compared our approach with the method relying on bilingual dictionaries in our paper. In Section 5.2 (Translation Quality), we compared the pre-back-translation results. Here, "MUSE"[1] represents the method initialized with a bilingual dictionary. Regarding the post-back-translation results, in the Baselines, the UMT[2] method employs the initialization method based on a bilingual dictionary. The results obtained by our method are significantly superior to UMT method.

Additionally, obtaining a high-quality bilingual dictionary requires a significant amount of additional data expenses, such as through manual annotation or extraction from bilingual parallel data. This is not aligned with the objectives of Unsupervised Neural Machine Translation (UNMT). Relatively speaking, our method achieves high-quality translation with lower data collection costs as it only requires monolingual text-image pairs.

Reviewer comment (Novelty):
The authors should consider discussing whether there have been more recent developments in text-only unsupervised translation methods, as this would help place their approach in the context of the latest advancements in the field.

Author response:
**4. Concerns on more recent text-only unsupervised translation methods**

Recently, many state-of-the-art text-only translation models are based on larger-scale models for multilingual translation. These models are not entirely unsupervised; their training often involves extensive parallel corpora. In these models, zero-shot tasks are common for languages without parallel data. Our method can be operated at the representation level, narrowing the gap between representations of different languages. This helps alleviate off-target issues (outputting in the wrong language) in multilingual translation models, thereby improving overall performance. Consequently, our approach is equally applicable to large language models. In future explorations, we plan to delve into related research. Thank you for your valuable suggestions!

Reviewer comment (Experiments):
While the authors demonstrate superiority over existing systems on high-resource language pairs (en<->de and en<->fr), they should explore the effectiveness of real low-resource languages.

Author response:
**1. Concerns on real low-resource languages**

Constrained by the dataset, both previous works and our work have conducted experiments in English, German, and French. However, it is necessary and valuable to validate our approach on truly low-resource languages. Therefore, we conducted additional experiments using a Czech language dataset from the Multi30K Test2016 and Test2018 datasets. 
Our method demonstrates a substantial improvement compared to the text-only baseline, with an average BLEU score increase of 4.0 and an average METEOR score increase of 2.9 across all the language directions. This indicates that our method also performs well on real low-resource languages.

**Table 1 Results on Multi30K EN-CS Flickr2016 set**
|      |           |        |       |        |           |        |       |        |      |         |
|:----:|:---------:|:------:|:-----:|:------:|:---------:|:------:|:-----:|:------:|:----:|:-------:|
|      | Test_2016 |        |       |        | Test_2018 |        |       |        |      |         |
|      | en-cs     |        | cs-en |        | en-cs     |        | cs-en |        |      |         |
|      | BLEU      | METEOR | BLEU  | METEOR | BLEU      | METEOR | BLEU  | METEOR | BLEU | METEOR  |
| MASS | 20.1      | 23.9   | 27.1  | 29.3   | 16.1      | 21.2   | 22.3  | 27.1   | 21.4 | 25.4    |
| Ours | **24.2**      | **26.4**   | **30.8**  | **32.2**   | **20.0**        | **24.1**   | **26.6**  | **30.4**    |**25.4** | **28.3**    |

To further supplement our findings, we conducted experiments on English-to-German and English-to-French translation under low-resource settings, specifically using only 75,000 monolingual data samples. 

**Table 2 Results on low-resource setting (75k data only).**
|      |       |        |       |        |       |        |       |        |      |         |
|:----:|:-----:|:------:|:-----:|:------:|:-----:|:------:|:-----:|:------:|:----:|:-------:|
|      | en-de |        | de-en |        | en-fr |        | fr-en |        | Avg  |         |
|      | BLEU  | METEOR | BLEU  | METEOR | BLEU  | METEOR | BLEU  | METEOR | BLEU | METEOR  |
| MASS | 18.3  | 35.8   | 19.7  | 23.4   | 24    | 41.5   | 23.9  | 27     | 21.5 | 31.9    |
| Ours | **27.5**  | **47.6**   | **32.3**  | **32.2**   | **36.7**  | **54.6**   | **36.9**  | **34.3**   | **33.6** | **42.2**    |

The experimental results indicate that in simulated low-resource scenarios, our method continues to yield significant improvements. In contrast, text-only method without large-scale monolingual pretraining experiences a severe performance drop, and the model convergence during training is notably slower.

Reviewer comment (Experiments):
The text-image pairs need annotation, which means that the unsupervised machine translation needs cross-modality annotation, which is more expensive than text-only annotation.

Author response:
**1. Concerns on the cost of cross-modality annotation**

In Unsupervised Multi-modal Machine Translation (UMMT), the correspondence between source language and target language is missing, and visual information serves as additional data to compensate for this absence. Although there is costly for data collection, our cross-modal model has achieved a significant improvement in performance compared to the text-only baseline. Additionally, we only utilize monolingual image-text annotations,  it is easier to recruit monolingual annotators to describe an image than to find multilingual translators to translate sentences.which results in relatively lower data collection costs.

Reviewer comment (Theory):
In Section 3.3, why text tokens and image tokens in different positions (i ≠ j) is regarded as negative examples, and why in the same position can be regarded as positive examples? In text-only translation tasks, the same position of the src sentence and tgt sentence always do not refer to the same thing. In other words, the mapping of image tokens and language tokens is simply one-to-one position mapping.

Author response:
**2. Concerns on the image token and token-level contrastive learning**

We utilize the **Vision Transformer (ViT)** as our image encoder. In Vision Transformer,  the input image is divided into fixed-size non-overlapping patches. Each patch is treated as a token. After that, each patch is linearly embedded to flatten it into a 1D vector. These vectors serve as the initial input tokens for the transformer model. Finally, it encodes the image to a sequence $v = (v_0, v_1, ..., v_m)$, where $v_0$ is the special $[class]$ token and the others are the representation of image patches. For example, a single image is divided into 9 patches through a 3x3 grid. Each patch serves as a token, and when combined with a special [class] token, it forms a sequence with a length of 10 as input to the model. Consequently, a vector of dimensions (1, 10, 512) is obtained after processing through the model.

Therefore, each image token represents the vector representation of a patch of the image. We utilize a selective attention module to filter out image region representations related to the semantics of each text token from multiple patch representations, and unify the sequence length. Consequently, the attention sequences filtered for each token correspond one-to-one. Instantly, $w_1$ and $v^t_1$ is positive examples, $w_1$ and $v^t_1$ is negative examples. The token-level contrastive learning is trained between sequence w and $v^t$ but not $w$ and $v$.  

A core module here is the **Selective Attention** module, which filters image information based on each text token. As a result, beyond positional correspondence between sequences, there exists a strong semantic relevance. Due to limitations in the length of the paper, we don't present the detailed image feature extraction process in the original text. You can refer to Figure 1 of <On Vision Features in Multimodal Machine Translation>[1] and <An image is worth 16x16 words: Transformers for image recognition at scale>[2] for a more in-depth understanding. It will greatly aid your comprehension.

Reviewer comment (Evaluation):
Do you only visualize 6 examples in Figure 2? Where are the 6 examples from? Is it sufficient to support that your approach is truly successful?

Author response:
**3. Concerns on the visualization**

We randomly pick 6 examples from the Test2016 set of Multi30k on EN-DE direction. and due to space constraints, additional examples couldn't be included in a single diagram. For a more comprehensive analysis, please refer to the text-to-image retrieval section (Sec 5.2), where you'll find that the overall matching performance between images and text has considerably improved across the entire test set. Therefore, Figure 2 only serves as a visual example specifically for the text-to-image retrieval part, aiming to enhance reader understanding.

Reviewer comment (Experiments):
Why IWSLT data is regarded as out-of-domain data compared with the data you use for initialization which includes MsCOCO and Multi30K? Can you prove the domain mismatch in the two kind of datasets?

Author response:
**4. Concerns on IWSLT dataset**

The Multi30k and MsCOCO dataset is primarily designed for image captioning tasks. It consists of images paired with descriptions, making it suitable for tasks related to visual understanding and language generation. The IWSLT dataset, on the other hand, is created for speech translation tasks. It includes spoken language data in various languages and is commonly used for evaluating the performance of machine translation systems for spoken content. We provide some examples below to help you understand. 

**Multi30k:**

[English]: A man in a blue shirt is standing on a ladder cleaning a window.

[German]: Ein Mann in einem blauen Hemd steht auf einer Leiter und putzt ein Fenster.

[English]: A group of people are sitting on a wall and playing musical instruments.

[German]: Eine Gruppe von Menschen sitzt auf einer Mauer und spielt Musikinstrumente.

**IWSLT:**

[English]: Can you help me find my keys? I think I left them on the kitchen counter.

[German]: Kannst du mir helfen, meine Schlüssel zu finden? Ich glaube, ich habe sie auf der Küchentheke liegen lassen.

[English]: The conference will be held in the main auditorium, starting at 9:00 AM.

[German]: Die Konferenz findet im Hauptauditorium statt und beginnt um 9:00 Uhr.

These examples illustrate typical sentences from each dataset, showcasing the diversity of language and content in Multi30k (image captioning) and IWSLT (text translation).


Thank you again for your valuable time and effort in the review. If our answer solves your problems precisely, we would appreciate it if you could reassess our work. If you have any further questions, please don't hesitate to reach out to me for discussion.