Paper: A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
There were certain flaws in the method, and prompts affected the results.

Author response:
**Regarding your concern about prompt & stability :**
>There were certain flaws in the method, and prompts affected the results.

>The stability of the proposed method was not verified.

Thank you for your insightful feedback!  We wholeheartedly acknowledge the idea that different prompts can influence translation performance, and that enriching these prompts could further improve results. Our primary focus, however, is to demonstrate that a properly formulated prompt enables moderate-sized large language models (LLMs) to achieve performance comparable to state-of-the-art (SOTA) encoder-decoder translation models or GPT3.5/4. The prompts we have employed are widely recognized and validated in the field of machine translation, as evidenced by previous research ([1], [2], [3]). Nonetheless, we agree that exploring alternative prompts could potentially yield even better translation outcomes! Hence, we re-train ALMA-7B-LoRA model with **two new prompts** as suggested in [4]:

Recap that:
- Our prompt is: Translate this from <src> to <tgt>:\n<src>: <input>\n <tgt>:
- New Prompt 1: <src>: <input>\n<tgt>:
- New Prompt 2: <input> Translate this from <src> to <tgt>:

|                | Avg. en>xx |       | Avg. xx>en |       |
|----------------|------------|-------|------------|-------|
| Prompt (for ALMA-7B-LoRA) | BLEU  | Comet22 | BLEU  | Comet22 |
| Our prompt               | **29.78** | **86.80**   | **34.34** | **84.24**   |
| New prompt 1             | 27.63 | 85.44   | 33.434| 83.60   |
| New prompt 2             | 27.44 | 85.31   | 33.314| 83.85   |

We conducted evaluations of the two new prompts using the WMT’22 test set and report the average scores for both translation directions. The performance of both new prompts is comparable, though they slightly underperform compared to our originally proposed prompt.

Reference: 

[1] Hendy A, Abdelrehim M, Sharaf A, Raunak V, Gabr M, Matsushita H, Kim YJ, Afify M, Awadalla HH. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210. 2023 Feb 18.

[2] Chen Y, Liu Y, Meng F, Chen Y, Xu J, Zhou J. Improving Translation Faithfulness of Large Language Models via Augmenting Instructions. arXiv preprint arXiv:2308.12674. 2023 Aug 24.

[3] Zeng J, Meng F, Yin Y, Zhou J. Tim: Teaching large language models to translate with comparison. arXiv preprint arXiv:2307.04408. 2023 Jul 10.

[4] Zhang B, Haddow B, Birch A. Prompting large language model for machine translation: A case study. arXiv preprint arXiv:2301.07069. 2023 Jan 17.

Reviewer comment (Experiments):
The stability of the proposed method was not verified.

Author response:
**Regarding your concern about prompt & stability :**
>There were certain flaws in the method, and prompts affected the results.

>The stability of the proposed method was not verified.

Thank you for your insightful feedback!  We wholeheartedly acknowledge the idea that different prompts can influence translation performance, and that enriching these prompts could further improve results. Our primary focus, however, is to demonstrate that a properly formulated prompt enables moderate-sized large language models (LLMs) to achieve performance comparable to state-of-the-art (SOTA) encoder-decoder translation models or GPT3.5/4. The prompts we have employed are widely recognized and validated in the field of machine translation, as evidenced by previous research ([1], [2], [3]). Nonetheless, we agree that exploring alternative prompts could potentially yield even better translation outcomes! Hence, we re-train ALMA-7B-LoRA model with **two new prompts** as suggested in [4]:

Recap that:
- Our prompt is: Translate this from <src> to <tgt>:\n<src>: <input>\n <tgt>:
- New Prompt 1: <src>: <input>\n<tgt>:
- New Prompt 2: <input> Translate this from <src> to <tgt>:

|                | Avg. en>xx |       | Avg. xx>en |       |
|----------------|------------|-------|------------|-------|
| Prompt (for ALMA-7B-LoRA) | BLEU  | Comet22 | BLEU  | Comet22 |
| Our prompt               | **29.78** | **86.80**   | **34.34** | **84.24**   |
| New prompt 1             | 27.63 | 85.44   | 33.434| 83.60   |
| New prompt 2             | 27.44 | 85.31   | 33.314| 83.85   |

We conducted evaluations of the two new prompts using the WMT’22 test set and report the average scores for both translation directions. The performance of both new prompts is comparable, though they slightly underperform compared to our originally proposed prompt.

Reference: 

[1] Hendy A, Abdelrehim M, Sharaf A, Raunak V, Gabr M, Matsushita H, Kim YJ, Afify M, Awadalla HH. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210. 2023 Feb 18.

[2] Chen Y, Liu Y, Meng F, Chen Y, Xu J, Zhou J. Improving Translation Faithfulness of Large Language Models via Augmenting Instructions. arXiv preprint arXiv:2308.12674. 2023 Aug 24.

[3] Zeng J, Meng F, Yin Y, Zhou J. Tim: Teaching large language models to translate with comparison. arXiv preprint arXiv:2307.04408. 2023 Jul 10.

[4] Zhang B, Haddow B, Birch A. Prompting large language model for machine translation: A case study. arXiv preprint arXiv:2301.07069. 2023 Jan 17.

Reviewer comment (Experiments):
Typically, if an LLM was fine-tuned with specific bilingual corpora to enhance translation capabilities between those two languages, it might impair other NLP capabilities of the LLM, such as document summarization and logical question answering. Did this issue not arise in this study?

Author response:
**Regarding your question 1:**
>Typically, if an LLM was fine-tuned with specific bilingual corpora to enhance translation capabilities between those two languages, it might impair other NLP capabilities of the LLM, such as document summarization and logical question answering. Did this issue not arise in this study?

Thank you for posing such an insightful question, which indeed addresses a key aspect of our research. You are likely aware that the first stage of our process, monolingual fine-tuning, is designed to enhance the large language model's (LLM) proficiency in general non-English languages. The second stage, involving parallel data fine-tuning, is specifically tailored for machine translation.

In our study, we have confirmed that the first stage of fine-tuning does not significantly impact the model's performance in English across various tasks and highly improves for non-English languages, as shown in Table 12. For the second stage, we have explored the use of Low-Rank Adaptation (LoRA) during fine-tuning with bilingual corpora. This approach is intended to avoid any potential negative impact on other NLP capabilities after fine-tuning. Importantly, LoRA is utilized as a **lightweight, plug-and-play module**. This means that, based on the model fine-tuned in the first stage, we can extend its application to develop multilingual summarization or question-answering capabilities by training a corresponding LoRA module for each task. This approach ensures that our fine-tuning process enhances specific functionalities while preserving the model's overall proficiency in diverse NLP tasks.

Reviewer comment (Evaluation):
If possible, please consider using the 'Instruct Score' from EMNLP 2023 (Instructscore: Towards Explainable Text Generation Evaluation with Automatic Feedback) as a metric. I believe it's a better benchmark for evaluating LLM-MT.

Author response:
**Regarding your Concern about evaluation metric:**
Thank you for your constructive feedback. We acknowledge that finding reliable evaluation metrics remains a challenging aspect of machine translation research. During our experiments, we reported results using COMET-22 and BLEU, which were the predominant metrics at that time. These metrics are not only widely accepted but also facilitate easy comparison within the research community. To address your concerns, we have also evaluated our ALMA models using three additional automatic metrics:

Thank you for recommending InstructScore, which is an intriguing automatic evaluation metric. We observed that the prompt for this metric (found at https://github.com/xu1998hz/SEScore3/blob/586f7f22ced93b136cbde818e86fbddcdd8580ce/InstructScore.py#L67C19-L67C19) is specifically tailored for translations from Chinese to English. Therefore, we limited our evaluation of the Instructscore to this language pair. The results, as detailed in the table below, demonstrate that our method surpasses GPT-3.5-D in performance, with higher scores indicating better results.

|             | zh->en (Instructscore) |
|-------------|------------------------|
| GPT-3.5-D   | -5.2944                |
| ALMA-13B-LoRA | **-4.5856**              |

To provide a more thorough assessment using recently released evaluation metrics, we conducted evaluations using wmt23-cometkiwi-da-xxl, and Unbabel/XCOMET-XXL. Both of these are 10 billion parameter models that support all languages. The results, as presented in the table below, indicate that our method continues to outperform GPT-3.5-D when evaluated with these latest metrics.

|              | Avg. xx->en |       |            |
|-|---|--|---|
|              | Comet22     | wmt23-cometkiwi-da-xxl | XCOMET-XXL |
| GPT-3.5-D    | 83.90       | 80.06 | 84.59      |
| ALMA-13B-LoRA| **84.59**       | **81.50** | **86.74**      |


|              | Avg. en->xx |       |            |
|------|---|-------|------|
|              | Comet22     | wmt23-cometkiwi-da-xxl | XCOMET-XXL |
| GPT-3.5-D    | 84.59       | 75.99 | 88.10      |
| ALMA-13B-LoRA| **87.00**       | **82.66** | **92.76**      |

Reviewer comment (Experiments):
Given the powerful In-Context Learning capabilities of large language models, it would be worth exploring whether adding relevant knowledge to the prompt could further enhance translation capabilities.

Author response:
**Regarding your concern about prompt & stability :**
>There were certain flaws in the method, and prompts affected the results.

>The stability of the proposed method was not verified.

Thank you for your insightful feedback!  We wholeheartedly acknowledge the idea that different prompts can influence translation performance, and that enriching these prompts could further improve results. Our primary focus, however, is to demonstrate that a properly formulated prompt enables moderate-sized large language models (LLMs) to achieve performance comparable to state-of-the-art (SOTA) encoder-decoder translation models or GPT3.5/4. The prompts we have employed are widely recognized and validated in the field of machine translation, as evidenced by previous research ([1], [2], [3]). Nonetheless, we agree that exploring alternative prompts could potentially yield even better translation outcomes! Hence, we re-train ALMA-7B-LoRA model with **two new prompts** as suggested in [4]:

Recap that:
- Our prompt is: Translate this from <src> to <tgt>:\n<src>: <input>\n <tgt>:
- New Prompt 1: <src>: <input>\n<tgt>:
- New Prompt 2: <input> Translate this from <src> to <tgt>:

|                | Avg. en>xx |       | Avg. xx>en |       |
|----------------|------------|-------|------------|-------|
| Prompt (for ALMA-7B-LoRA) | BLEU  | Comet22 | BLEU  | Comet22 |
| Our prompt               | **29.78** | **86.80**   | **34.34** | **84.24**   |
| New prompt 1             | 27.63 | 85.44   | 33.434| 83.60   |
| New prompt 2             | 27.44 | 85.31   | 33.314| 83.85   |

We conducted evaluations of the two new prompts using the WMT’22 test set and report the average scores for both translation directions. The performance of both new prompts is comparable, though they slightly underperform compared to our originally proposed prompt.

Reference: 

[1] Hendy A, Abdelrehim M, Sharaf A, Raunak V, Gabr M, Matsushita H, Kim YJ, Afify M, Awadalla HH. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210. 2023 Feb 18.

[2] Chen Y, Liu Y, Meng F, Chen Y, Xu J, Zhou J. Improving Translation Faithfulness of Large Language Models via Augmenting Instructions. arXiv preprint arXiv:2308.12674. 2023 Aug 24.

[3] Zeng J, Meng F, Yin Y, Zhou J. Tim: Teaching large language models to translate with comparison. arXiv preprint arXiv:2307.04408. 2023 Jul 10.

[4] Zhang B, Haddow B, Birch A. Prompting large language model for machine translation: A case study. arXiv preprint arXiv:2301.07069. 2023 Jan 17.

Reviewer comment (Experiments):
The section "Small Training Data Is Enough" contained many uncertain descriptions, which should be rigorous in a paper and supported by convincing data. Moreover, the best-performing commercial LLM, GPT-4, remained proprietary, leaving us in the dark about the amount of data used, the number of parameters trained, and potential data leakage issues during translation metric testing.

Author response:
**Regarding your question 5:**
>The section "Small Training Data Is Enough" contained many uncertain descriptions, which should be rigorous in a paper and supported by convincing data. Moreover, the best-performing commercial LLM, GPT-4, remained proprietary, leaving us in the dark about the amount of data used, the number of parameters trained, and potential data leakage issues during translation metric testing.

Thank you for your thorough examination of our claim in the "Small Training Data Is Enough" section. We appreciate the opportunity to engage in a more detailed discussion on this topic. In light of your feedback, we acknowledge that certain descriptions in this section, such as "well-trained LLM may not necessitate substantial parallel data," might be perceived as overly broad and lacking rigor. We intend to revise this to specifically reference "LLaMA-2" instead of a general "well-trained LLM."

Additionally, we fully agree with your observation regarding the unknown aspects of GPT-4. This lack of transparency is precisely why our study is significant. Our goal is to develop and implement multilingual LLMs that are not only entirely transparent but also freely available for real-world applications. This approach is intended to provide a clear and open understanding of the LLM's functioning and training, contrasting with the more opaque nature of models like GPT-4.

Reviewer comment (Writing):
The sentence in the "Large Parallel Data Wash Out the Knowledge" section: "As expected, it tends up with a similar performance in both BLEU and COMET evaluations (triangle in Figure 4)," was hard to understand.

Author response:
**Regarding your question 6:**
>The sentence in the "Large Parallel Data Wash Out the Knowledge" section: "As expected, it tends up with a similar performance in both BLEU and COMET evaluations (triangle in Figure 4)," was hard to understand.

Thank you for your question, which invites a deeper exploration of our hypothesis. We propose that excessive parallel data (for instance, 20 million sentences) might dilute the pre-existing knowledge in pre-trained LLMs like LLaMA-2 and MPT. A pertinent question arises: To what extent does this "knowledge washout" occur?

To investigate this, we conducted an experiment where we trained a model from scratch with 20 million sentences. This simulates an extreme scenario where all pre-trained knowledge is effectively erased, leaving the model with knowledge derived solely from the 20 million parallel sentences. The COMET score for this model, trained from scratch, was 77.5. This score is strikingly close to those of LLaMA-2 or MPT, which hover around 79. This similarity suggests that, with an increase in parallel data and extended training, all LLMs sharing a similar architecture (even those trained from scratch) might eventually converge towards similar translation models, effectively erasing their previous specialized knowledge.

Therefore, we conclude that using a large volume of parallel data may not be the most appropriate approach for LLMs, as it can lead to the loss of their uniquely trained features and knowledge base.

Reviewer comment (Novelty):
The statement of "paradigm shift" is somehow overestimated. The authors claim the finding/proposal is a "paradigm shift" as highlighted in the title, which might be inadequate. The recipe generally follows the pretraining-finetuning paradigm which has already been well-established since BERT and mBart/T5. In addition, similar solutions have already been used in prior studies, such as BigTranslate.

Author response:
We genuinely appreciate the valuable feedback provided by the reviewer and have addressed them in a point-by-point manner below. We are more than willing to engage in further discussions with the reviewers should any follow-up questions arise.

**Regarding your concern about statement:**
>The statement of "paradigm shift" is somehow overestimated.

We are grateful for your feedback and would like to provide further clarification. “The new paradigm” specifically addresses machine translation in a new method, rather than proposing a general methodology. While the general concept of "pre-training + fine-tuning" is indeed widely used in NLP tasks, it represents a broad strategy rather than a specific methodology. We acknowledge that the concepts of 'modeling' (using a decoder-only model) and 'training' (through causal language modeling) are not novel in themselves. However, applying these techniques to machine translation is a significant innovation (Note that we discuss three ‘training’ method for translation: casual language modeling, prefix language modeling, and mixture of denoisers in Appendix A, but the simplest CLM performs the best).  Traditionally, machine translation has primarily relied on processing vast quantities (millions) of parallel sentences using an encoder-decoder architecture. The application of decoder-only models to this field is basically unexplored and has rarely outperformed the conventional encoder-decoder approach.

The remarkable performance of models like ChatGPT has sparked interest in decoder-only large language models (LLMs) for various applications. There must be potential for machine translation. However, most recent study on machine translation for LLMs fails to achieve comparative performance to conventional encoder-decoder models. This area remains largely untapped, posing critical questions:

- How should we train decoder-only models for translation tasks which were traditionally training on encoder-decoder architecture?
- Given that LLMs are already extensively trained, is there still a need for large volumes of parallel data for additional training? Could this potentially impair the model's effectiveness?
- Considering that LLMs are predominantly trained in English, how can we enhance their proficiency in other languages to improve translation quality?

Our paper seeks to address these questions, proposing a novel and efficient training methodology for machine translation. We are the first to utilize 7B/13B LLMs for machine translation, and achieve performance comparable to state-of-the-art encoder-decoder models. This breakthrough lays the groundwork for future research in integrating LLMs with machine translation, marking an exciting new direction in the field.

We recognize that previous studies, such as BigTranslate, have implemented similar fine-tuning approaches. However, there are significant differences between their methods and ours, particularly in acknowledging the negative impacts of excessive parallel data and the importance of data quality. Their approach resulted in performances that did not meet our expectations. Our goal is to provide an accurate and effective training "recipe" for decoder-only LLMs in machine translation, ensuring high performance. In our revised manuscript, we commit to clearly delineating our method's relationship to the broader "pre-training and fine-tuning" approach, emphasizing the unique aspects of our strategy and avoiding any unintentional overstatements.

Reviewer comment (Evaluation):
The few-shot prompting results are highly undervalued. As shown in Table 11, the quality gap between the HW 5-shot and the proposal is only 0.2 COMET on XX->En, although the finetuning used a large amount of monolingual corpus and more parallel data. Few-shot performance should be able to be further enhanced via beam search and optimized prompt construction, and it should be used as the fair baseline rather than 0-shot prompting. It is also misleading to state in Appendix I that "ICL substantially underperforms our stage-2 finetuning".

Author response:
**Regarding your concern about Few-shot learning:**
>The few-shot prompting results are highly undervalued.

Thank you for your valuable feedback and for taking the time to review the appendix. We would like to clarify an important aspect: the model evaluated for few-shot learning is not the standard LLaMA-2-13B, but rather LLaMA-2-13B after fine-tuning with 12 billion monolingual data tokens. This means that both our supervised fine-tuning and few-shot learning methods are built upon a model that has extensively utilized monolingual corpus. For contexts where monolingual data fine-tuning is not applied, we kindly direct you to Table 6 (Appendix E). In this table, it is evident that a 5-shot fine-tuning approach applied solely on LLaMA-2-13B falls behind the ALMA-13B-LoRA model by 12.18 COMET points on en$\rightarrow$xx direction.

However, we agree with the reviewer's suggestion that we need to exercise caution with our wording, particularly since few-shot learning performance can be significantly influenced by the quality of the prompts used. Nonetheless, supervised fine-tuning still offers certain advantages under these conditions:
- Performance Edge: Supervised fine-tuning, albeit marginally, continues to outperform few-shot learning, especially in English-to-other language (en$\rightarrow$xx) translations. This observation is consistent with findings from several previous studies ([1], [2]).
- Stability of Results: The performance of few-shot learning tends to vary considerably, heavily dependent on the quality of examples (shots) included in the prompt, as demonstrated in Table 11.
- Efficiency in Resource Use: Few-shot learning typically requires a longer prefix, consuming more memory. If we consider a hypothetical scenario where each input token incurs a cost, supervised fine-tuning would be more economical in terms of resource utilization.


Reference:

[1] Liu H, Tam D, Muqeeth M, Mohta J, Huang T, Bansal M, Raffel CA. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems. 2022 Dec 6;35:1950-65.

[2] Mosbach M, Pimentel T, Ravfogel S, Klakow D, Elazar Y. Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation. arXiv preprint arXiv:2305.16938. 2023 May 26.]

Reviewer comment (Experiments):
The proposed recipe might not apply to other LLMs and languages. Finetuning performance is highly dependent on pretraining conditions and the downstream task. When the downstream tasks highly correlate with the pretraining, the demand for a large volume of the supervised corpus is reduced as highlighted in this paper. However, as the MPT-7B performance indicates in Figure 4, when the correlation is low, adding more supervised data is almost always helpful. In other words, the findings in this paper might not be generalizable. There is also a question of whether the finetuning results for MPT-7B are available and whether the recipe applies to low-resource languages like Gu.

Author response:
**Regarding your concern about generalization:**
>The proposed recipe might not apply to other LLMs and languages.

> However, as the MPT-7B performance indicates in Figure 4, when the correlation is low, adding more supervised data is almost always helpful.

Thank you for your insightful comment. We concur with your observation that the effectiveness of downstream tasks is closely linked to the pre-training stage. This understanding actually underscores the importance of non-English monolingual fine-tuning for these models. LLMs mainly trained in English are likely to exhibit performance trends similar to the MPT model, especially in languages like Russian, which they have not been exposed to during training. In such cases, any improvements in these models are solely reliant on the Russian knowledge extracted from parallel data. However, learning from parallel data can wash out the knowledge learned during the pre-training phase — to the extent that a model initialized randomly could achieve comparable performance to MPT after being trained with 20 million parallel data sentences. The primary aim of monolingual fine-tuning is to equip the model with a general understanding of a new language. Following this, parallel data fine-tuning is employed to guide the model in accurately utilizing this knowledge for effective translation. Our suggested approach is designed to be **universally applicable across all LLMs and languages**. It involves fine-tuning the model on monolingual data of the target language, regardless of its resource availability, and subsequently fine-tuning on the corresponding parallel data. This method ensures that the model not only learns the new language but also becomes proficient in applying this knowledge for translation tasks.

>In other words, the findings in this paper might not be generalization. Do you also have the finetuning results for MPT-7B? Does the recipe also apply to low-resource languages like Gu?

To demonstrate the generalization of our methods to other large language models (LLMs), we extended our experiments to Mistral-7B, another well-known but recently released model. We fine-tuned Mistral-7B using 18 billion tokens of monolingual data, which included a mix of five new languages as detailed in our paper. We then evaluated its performance on the same multilingual test set used for our previous experiments. The results, as presented below, show that ALMA models, whether based on LLaMA-2-7B or Mistral-7B, are capable of achieving comparable levels of performance. This indicates the potential of our fine-tuning approach to be effectively generalized across different LLM architectures.

|                | Avg. en>xx |       | Avg. xx>en |       |
|----------------|------------|-------|------------|-------|
|                | BLEU       | Comet22 | BLEU       | Comet22 |
| ALMA-LLaMA-7B-LoRA | 29.78 | 86.37 | 34.31 | 84.12 |
| ALMA-Mistral-7B-LoRA | 30.78 | 86.48 | 34.07 | 84.09 |

To demonstrate the adaptability of our methods to low-resource languages, specifically Gujarati (gu), we initially fine-tuned LLaMA-2-7B using 0.1 billion Gujarati tokens and subsequently fine-tuned it on the Flores-200 development data. The model was then evaluated using the Flores-200 devtest dataset. The results of this experiment are detailed below. We compared the performance of our fine-tuned model against the zero-shot performance of LLaMA-2-7B and the outputs from BigTranslate. These comparisons revealed that while the other models produced some nonsensical outputs, our fine-tuned model significantly outperformed them, showcasing the effectiveness of our method. It's important to note that this was a preliminary experiment, constrained by time limitations, and does not represent the peak potential performance for Gujarati. We anticipate that the performance could be significantly enhanced with access to more monolingual data, higher-quality data, and more refined training practices. This experiment serves as evidence that our method can be successfully extended to low-resource languages like Gujarati.

|                | en>gu       |       | gu>en       |       |
|----------------|-------------|-------|-------------|-------|
|                | BLEU        | COMET22 | BLEU        | COMET22 |
| LLAMA-2-7B, zero-shot | 0.48      | 38.88 | 0.18      | 39.33  |
| BigTranslate   | 0.23      | 43.80 | 1.87      | 51.09  |
| ALMA-7B-GU     | **11.31**     | **79.42** | **15.05**     | **66.60**  |

Reviewer comment (Experiments):
How's your fine-tuned models compared with the dedicated encoder-decoder based translation models in similar size? Please discuss in both high and low resource settings.

Author response:
**Regarding your question about dedicated encoder-decoder model**:  
>How's your fine-tuned models compared with the dedicated encoder-decoder based translation models in similar size?  

Thank you for your insightful question! Most online translation models are generally smaller, often adhering to the transformer-big architecture [1], which results in around 300 million parameters. We'd like to clarify that in our paper, the NLLB-200 model serves as a benchmark for one of the strongest encoder-decoder based translation models available that is comparable in size to our ALMA models. The NLLB-54B's base model has 3 billion parameters, with additional parameters derived from a Mixture of Experts (MoE) approach. Our research demonstrates that for high-resource languages such as Chinese, German, Russian, Czech, and the low-resource language Icelandic, our ALMA models either match or outperform the dedicated encoder-decoder model NLLB-54B. However, we acknowledge that for very low-resource languages, where even monolingual data is limited, further research is required to optimize positive cross-lingual transfer. We plan to explore these avenues in future studies.  

Reference:  
[1] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. Advances in neural information processing systems. 2017;30.

Reviewer comment (Experiments):
If fine-tuning with large amount of parallel data is not optimal, how about using the during pre-training phase? If targeting at the highest translation performance, what's your take? Please include the dedicated translation model in the discussion.

Author response:
**Regarding your question about using parallel data for pre-training:**  
>If fine-tuning with large amount of parallel data is not optimal, how about using them during the pre-training phase?  

Thank you for your intriguing question. In fact, at an early stage in our research, we experimented with treating parallel data as monolingual data during the pre-training phase. Specifically, we used 20 million English-Russian parallel sentences, amounting to a total of 1.8 billion tokens. Our initial approach involved fine-tuning the LLaMA-2-7B model on these 1.8 billion tokens, mimicking the pre-training process, followed by further fine-tuning on English to Russian translation using the Flores development and test datasets (comprising approximately 2,000 sentences in total). To ensure a fair comparison, we also conducted an experiment where we fine-tuned LLaMA-2-7B on 1.8 billion tokens from the Oscar dataset, consisting of both English and Russian data, and this was also followed by fine-tuning on the same English-Russian parallel data. The outcomes of these comparative tests are presented below:  

|                              | BLEU  | Comet22 |  
|------------------------------|-------|---------|  
| Parallel data as monolingual data | 25.26 |  85.29  |  
| Oscar as monolingual data    | **26.43** |  **86.36**  |  

The results indicate that treating parallel data as monolingual data is beneficial, but it does not yield as favorable outcomes as using authentic monolingual data, such as that from the Oscar dataset. We hypothesize that the advantage of using Oscar's monolingual data lies in its document-level context, which enables ALMA models to learn more natural expressions in Russian. In contrast, treating parallel data as monolingual data limits learning to sentence-level information, which may not provide the same depth of linguistic understanding and fluency.

Reviewer comment (Experiments):
What would be the results of evaluating on some out-of-domain test sets?

Author response:
**Regarding your question about out-of-domain testing:**  
>What would be the results of evaluating on some out-of-domain test sets?  

Thank you for providing an insightful question! We actually have conducted such an experiment at Appendix E and we encourage the reviewer to take a look with any chances. Here, we only compare the ALMA model fine-tuning only Flores to the one fine-tuned by Flores+WMT data. Since the test data is WMT’22 (even though it has 4 difference domains), we think WMT training data we use is “in-domain” data, and Flores is out-of-domain data. Our results show that training with out-of-domain data may lead the lexical match drop (BLEU score down), while COMET scores are similar or even better.