Paper: SCALE: Synergized Collaboration of Asymmetric Language Translation Engines (ICLR-2024)

Reviewer comment (Experiments):
I like the idea; however, the paper focuses on translation into English and does not demonstrate the efficacy of translating from English to these low-resource languages. I suspect that SCALE has limitations in translating from English, given that refinement is core to SCALE. Although GPT-4 excels in English, it falls short in low-resource languages, suggesting that SCALE may not be effective for En→X.

Author response:
We appreciate your insightful comment and agree that evaluating the efficacy of En→X translation is crucial. To address this, we have expanded our experiments to include translations from English to six low-resource languages, using the same experimental setup as that of SCALE-refine. The results, gauged by COMET-22 and COMETKiwi, are presented below, and all corresponding outputs have been appended to the supplementary zip files for your reference. These additional experiments lead us to several observations:

1. SCALE exemplifies its proficiency in translating from English to a diverse range of languages, spanning various scripts and linguistic families.
2. While SCALE shows improvement over few-shot GPT-4 and NLLB, the magnitude of this improvement is less pronounced compared to the X→En setting. This indicates that the present limitation for LLM translators mainly lies in the generation of the target language, rather than the comprehension of the source language, as also suggested in [1].
3. Due to the limited transparency of the GPT-4 model, it is challenging to attribute these inconsistencies to specific factors like tokenization, data coverage, or the proportion of languages in the training data. Regardless, we are striving to identify a potential indicator, specifically the number of native speakers, to illustrate the extent of a language's resources. As illustrated in the accompanying table, the performance of few-shot GPT-4 diminishes with the number of native speakers, while our framework, SCALE, consistently and effectively mitigates this language bias, outperforming few-shot GPT-4 and the supervised NLLB model in 6/6 directions.

Recognizing the significance of these findings, we would include this discussion in the final version of our paper. 

|                   |   Javanese    |     Tamil     |    Urdu     |    Amharic    |     Khmer     |    Nepali     |
| --------------- | :-----------: | :-----------: | :---------: | :-----------: | :-----------: | :-----------: |
| # Native Speakers |     98 M      |    84.12 M    |   71.29 M   |     25 M      |     18.4M     |     16 M      |
|  few-shot GPT-4   |   83.9/75.8   |   83.5/80.8   |  80.0/80.4  |   77.1/73.4   |   74.3/74.7   |   80.1/86.4   |
|     NLLB-3.3B     |   86.4/76.4   |   86.5/82.9   |  80.7/80.4  |   84.4/80.7   |   76.3/77.8   |   77.0/82.0   |
|       SCALE       | **86.6/77.5** | **87.8/84.7** | **82/81.7** | **84.7/81.7** | **79.6/80.3** | **82.6/87.8** |

Reviewer comment (Experiments):
SCALE seems confined to low-resource languages, as LLM already performs excellently in translating between high-resource languages, such as German and French. The paper should make it clearer for readers how SCALE fares in the context of translating high-resource languages.

Author response:
We acknowledge your concern about the perceived focus of SCALE on low-resource languages and apologize if our manuscript did not communicate this aspect clearly. Your suggestion to clarify the motivation of our experimental design is well taken. 

While it is indeed true that current LLMs demonstrate exceptional performance in high-resource settings, our curiosity was piqued regarding the performance of SCALE in high-resource languages. We carried out experiments in three languages: German, Czech, and Chinese, under the same settings as SCALE-refine. We utilized COMET-22 and COMETKiwi to measure the translations. As shown below, the results reveal that SCALE offers consistent improvements even for high-resource languages, albeit less significantly than for low-resource languages, and it exhibits robust performance even when paired with a less performant STM (especially in en-zh direction). This is remarkable, given the already strong performances of models such as GPT-4 and NLLB. All the generated output has been updated to the supplementary zip files for your reference.

We assure you that these insights will be integrated into the revised version of our manuscript to provide a more comprehensive understanding.

|                |     en-de     |     de-en     |     en-cs     |     cs-en     |     en-zh     |     zh-en     |
| ------------ | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |
| few-shot GPT-4 |   88.5/84.4   | **90.0**/84.6 |   92.0/86.8   |   88.4/85.0   | 88.8/**84.7** |   87.3/84.3   |
|   NLLB-3.3B    |   86.9/82.9   |   89.1/84.2   |   90.1/84.8   |   88.4/85.5   |   78.0/70.9   |   86.1/83.7   |
|     SCALE      | **89.0/85.1** | 89.9/**84.7** | **92.4/87.1** | **89.2/86.0** | **89.1/84.7** | **87.8/84.9** |

Reviewer comment (Experiments):
In presenting "pivoting" (Sec. 4.2) and "updating" (Sec. 4.3), the paper restricts its experiments to LAO→En and Xhosa→En, respectively, which doesn't sufficiently substantiate this aspect. Conducting more comprehensive experiments would provide stronger support for the concept.

Author response:
We appreciate your observation regarding the limited range of experiments for the "pivoting" and "updating" sections. However, it is important to clarify that SCALE in the pivoting section was not limited to Lao→En translations. We expanded the translation from Lao to 8 other languages, encapsulating both low- and high-resource settings. Since the language selection in both the pivoting and updating sections was not specifically curated, we believe that this showcases the versatility of our method.

In accordance with your suggestion for a more exhaustive set of experiments, we have conducted additional experiments in both pivoting and updating settings. These new experiments include a broader range of languages, offering robust support for our concept. For the pivoting setting, we extended our experiments from Xhosa to eight other languages, covering both low- and high-resource languages, employing both few-shot GPT-4 and SCALE-pivot. We utilized COMET-22 to measure the performance. The results of these tests are presented below and we observe a similar pattern as exhibited in "pivoting" (Sec. 4.2). All the generated output has been updated to the supplementary zip files for your reference.

|                | Armenian | Assamese | Amharic  |   Lao    |  German  |  Czech   | Bulgarian |  Greek   |
| -------------- | :------: | :------: | :------: | :------: | :------: | :------: | :-------: | :------: |
| # Resource     |   Low    |   Low    |   Low    |   Low    |   High   |   High   |   High    |   High   |
| few-shot GPT-4 |   68.3   |   65.6   |   69.3   |   58.5   |   74.4   |   80.8   |   79.1    |   78.6   |
| SCALE-pivot    | **71.7** | **67.4** | **71.4** | **59.6** | **77.9** | **83.7** | **82.7**  | **81.4** |

For updating setting, we have broadened our scope to include three additional languages: Lao, Armenian, and Amharic. To assess the translations, we utilized the COMET-22 and COMETKiwi evaluation metrics. The results, which exhibit a pattern similar to that described in the "updating" section (Sec. 4.3), are presented below. For your reference, we have also updated the supplementary zip files with all generated outputs.

|                |           |     Xhosa     |      Lao      |   Armenian    |    Amharic    |
| :------------- | :-------: | :-----------: | :-----------: | :-----------: | :-----------: |
| STM            |    M2M    |   68.0/59.0   |   67.8/57.5   |   75.9/58.9   |   72.3/54.8   |
|                | NLLB-600M |   78.1/70.4   |   84.6/70.3   |   86.3/73.4   |   84.7/69.2   |
|                | NLLB-1.3B |   80.7/74.0   |   86.9/73.8   |   87.7/75.6   |   86.2/71.9   |
|                | NLLB-3.3B |   80.7/74.0   |   86.9/73.8   |   88.3/77.0   |   86.9/73.6   |
| few-shot GPT-4 |           |   78.8/70.8   |   80.0/63.7   |   86.2/73.1   |   83.2/67.3   |
| SCALE w/       |    M2M    |   79.1/71.6   |   82.5/67.3   |   86.7/74.1   |   84.6/69.7   |
|                | NLLB-600M |   81.3/74.6   |   86.3/72.9   |   88.1/76.3   |   87.3/74.2   |
|                | NLLB-1.3B |   81.9/75.3   |   86.6/73.5   |   88.5/77.0   |   87.8/75.1   |
|                | NLLB-3.3B | **82.1/75.7** | **87.2/74.4** | **88.8/77.8** | **88.0/75.7** |

Reviewer comment (Evaluation):
The paper's assertion that SCALE "mitigates both language bias and parallel data bias" seems peculiar to me. It feels intuitive rather than conclusively factual. Defining "language bias" and "parallel data bias," followed by a quantitative analysis, might be necessary for clarity.

Author response:
We appreciate your feedback regarding the need for explicit definitions and comprehensive analysis of the terms "language bias" and "parallel data bias". We regret any confusion that may have been caused by their absence in our initial submission.

When referring to "language bias", we are addressing the overrepresentation of English in the pre-training corpus of LLMs. This often leads to superior performance in English while underperforming in other languages, particularly those with limited resources. We have conducted experiments involving refinement and pivoting, which have shown that SCALE provides more equitable performance across these underrepresented languages. We interpret this as an effective mitigation of the inherent language bias present in models such as GPT-4.

On the other hand, "parallel data bias" pertains to the issues faced by supervised models when dealing with substandard or skewed samples from coarsely-mined parallel data [2]. Regarding SCALE, as the final translation is not generated by a STM, we posit that SCALE has the potential to alleviate this bias to a considerable extent. We present empirical evidence in our robustness analysis, demonstrating that SCALE maintains its effectiveness even when less satisfactory translations are provided by a STM. 

We hope that these clarifications address your concerns, and we thank you for your constructive feedback.

Reviewer comment (Experiments):
The choice of a less advanced model like GPT-2 for evaluating GPT-4's fluency strikes me as odd. While employing GPT-2 might yield reasonable comparative results for less sophisticated generation models, but I do not think it is proper here.

Author response:
We understand your concerns about our choice of using GPT-2 in our experiments. The use of GPT-2 in this context is solely as an evaluator to measure the fluency of translations, a role it has successfully fulfilled in recent studies such as [2]. Despite being a model from four years ago, GPT-2 still serves as an effective evaluator for assigning perplexity scores to English sentences.

To address your concern and ensure the reliability of our results, we also conducted the same experiments using Llama-2-7B and Llama-2-13B. The results of these tests are shown below. While Llama2 gives different absolute values of perplexity of STM, GPT-4, and SCALE, the relative order remains identical to that obtained with GPT-2, thereby reinforcing our confidence in the original findings.

This is the original results produced by GPT-2-XL:

|       | Assamese  | Armenian  |  Amharic  |   Xhosa   |  Uyghur   |   Khmer   |  Nepali   |  Sindhi   |
| ----- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |
| NLLB  |   45.00   |   34.02   |   45.29   |   42.11   |   37.51   |   34.49   |   39.90   |   40.80   |
| GPT-4 | **34.61** |   36.11   |   38.83   |   34.70   |   32.47   |   32.73   |   34.75   | **31.82** |
| SCALE |   36.35   | **31.01** | **33.98** | **33.26** | **32.34** | **30.79** | **34.09** |   32.97   |

This is the results produced by Llama-2-7B:

|       | Assamese  | Armenian  |  Amharic  |   Xhosa   | Uyghur |   Khmer   |  Nepali   |  Sindhi   |
| ----- | :-------: | :-------: | :-------: | :-------: | :----: | :-------: | :-------: | :-------: |
| NLLB  |   27.08   |   21.39   |   26.88   |   25.72   | 24.53  |   22.70   |   23.74   |   25.32   |
| GPT-4 |   21.77   |   21.18   |   23.64   |   21.26   | 20.50  |   20.63   |   21.33   | **19.90** |
| SCALE | **21.68** | **18.94** | **21.20** | **20.82** | **20.29**  | **19.63** | **20.24** |   20.37   |

This is the results produced by Llama-2-13B:

|       | Assamese  | Armenian  |  Amharic  |   Xhosa   |  Uyghur   |   Khmer   |  Nepali   |  Sindhi   |
| ----- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |
| NLLB  |   27.41   |   21.37   |   26.73   |   26.01   |   24.74   |   22.90   |   23.68   |   25.42   |
| GPT-4 | **22.01** |   21.53   |   23.75   |   21.01   |   20.58   |   20.68   |   21.23   | **20.07** |
| SCALE |   22.11   | **18.84** | **21.21** | **20.63** | **20.49** | **19.80** | **20.12** |   20.41   |

[1] Raunak, Vikas, et al. "Leveraging GPT-4 for Automatic Translation Post-Editing." *arXiv preprint arXiv:2305.14878* (2023).

[2] Hendy, Amr, et al. "How good are gpt models at machine translation? a comprehensive evaluation." *arXiv preprint arXiv:2302.09210* (2023).

Reviewer comment (Novelty):
The idea is relatively straightforward. The SCALE framework and experiments are more engineering-oriented, lacking scientific insight.

Author response:
We acknowledge your viewpoint, but we stand firmly to dispute your claim that our work is predominantly engineering-oriented and lacks scientific insight. We wish to assert with utmost certainty that our work is not confined to the realm of engineering but makes significant strides in contributing to the scientific literature in these crucial ways:

1. Originality: We have pioneered an approach that seamlessly integrates two distinct translation paradigms within a single framework. This is a first in the field and enriches the existing body of knowledge by demonstrating how these two asymmetric translation models can work in synergy through the mechanism of triplet in-context learning.
2. Rigorous Experimentation: Our research encompasses stringent experiments that authenticate our approach. These tests extend beyond mere engineering prowess to involve scientific investigation, presenting empirical proof of how triplet in-context learning can bolster the performance of translation models in a variety of scenarios, including refinement, pivoting, and updating.
3. In-depth Analysis: We have conducted a thorough scrutiny of the translation traits of STM, LLM, and SCALE, delineating the subtle dissimilarities among these three systems. Furthermore, we have undertaken a sweeping analysis of SCALE to validate each critical design in our framework, enhancing our comprehension of SCALE in terms of its robustness and latency cost.
4. Far-reaching Implications: The outcomes of our research have profound implications for the design of future translation systems. They equip practitioners with strategic insights on when to employ large language models, specialized models, or a blend of the two. Additionally, our study tackles a pivotal question in machine translation literature in the era of Large Language Models: Should we continue to train specialized translation models, or should we wholeheartedly adopt LLM-based translation? Our paper breaks new ground by proposing a feasible solution that amalgamates these two systems within a single framework.

We are confident that this articulates the scientific merit of our work. We will make it a point to highlight these aspects more prominently in our revised manuscript to forestall any potential misconceptions.

Reviewer comment (Experiments):
The baselines used for comparison are weak. There are many previous works, such as back translation, pretraining, and other improvements for low-resource languages, which may require fewer resources and perform better.

Author response:
We would like to clarify that we have selected three types of baselines for our study: supervised models (NLLB, M2M100), few-shot LLMs (XGLM, GPT-3.5, and GPT-4), and one commercial translation system (Microsoft Translator). To our knowledge, these models are among the strongest and most representative of their respective categories. In particular, NLLB and MS Translator have already employed techniques such as back-translation and pretraining, and NLLB remains the state-of-the-art model on the Flores datasets.

Moreover, our SCALE framework is not restricted to certain translation models. We believe that SCALE would also benefit from other strong baseline models by employing it as STM. If you could provide any additional models you believe we overlooked, we would greatly appreciate it. This would allow us to further validate and enhance the applicability of our SCALE framework.

Reviewer comment (Experiments):
Why not conduct translation experiments on languages with slightly higher resources, such as German, French, or Chinese? In these languages, the improvements of SCALE may be limited.

Author response:
In our experiment design, we primarily evaluate SCALE in the challenging low-resource setting. This decision was informed by current research findings that indicate Large Language Models already perform comparably with supervised models on high-resource languages, yet struggle with low-resource languages \[1][2]. Therefore, we believe it is more meaningful and challenging to focus on improving LLMs in this lower-resource setting.

To resolve some of your concerns, we still would like to test SCALE performance in the high-resource setting. We carried out experiments in three languages on the Flores datasets: German, Czech, and Chinese, under the same settings as SCALE-refine. We utilized COMET-22 and COMETKiwi to measure the translations. As shown below, the results reveal that SCALE offers consistent improvements even for these high-resource languages, albeit less significantly than for low-resource languages as you predicted, and it exhibits robust performance even when paired with a less performant STM (especially in en-zh direction). This is remarkable, given the already strong performances of models such as GPT-4 and NLLB. All the generated output has been updated to the supplementary zip files for your reference.

We assure you that these insights will be integrated into the revised version of our manuscript to provide a more comprehensive understanding.

|                | en-de         | de-en         | en-cs         | cs-en         | en-zh         | zh-en         |
| -------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| few-shot GPT-4 | 88.5/84.4     | **90.0**/84.6 | 92.0/86.8     | 88.4/85.0     | 88.8/**84.7** | 87.3/84.3     |
| NLLB-3.3B      | 86.9/82.9     | 89.1/84.2     | 90.1/84.8     | 88.4/85.5     | 78.0/70.9     | 86.1/83.7     |
| SCALE          | **89.0/85.1** | 89.9/**84.7** | **92.4/87.1** | **89.2/86.0** | **89.1/84.7** | **87.8/84.9** |

We hope that our responses have addressed your concerns adequately. We would greatly appreciate it if you could consider revising your score in light of these clarifications. Please feel free to ask any further questions or voice any other concerns you might have.

[1] Garcia, Xavier, et al. "The unreasonable effectiveness of few-shot learning for machine translation." *International Conference on Machine Learning*. PMLR, 2023.

[2] Vilar, David, et al. "Prompting palm for translation: Assessing strategies and performance." *arXiv preprint arXiv:2211.09102* (2022).