Paper: Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts (ICLR-2024)

Reviewer comment (Novelty):
A major weakness in this work is that the author essentially reintroduces the concept of back-translation, a well-established technique in machine translation. Yet, this paper does not make any reference to the original, popular back-translation work by Sennrich in 2016, which raises questions about the author's familiarity with prior research and the potential reinvention of existing concepts.

Author response:
1. **Our paper does not, in absolutely anyway, claim to invent back-translation!**  We cited numerous back-translation papers (Edunov et al., 2018; Lample et al., 2018; Conneau & Lample, 2019; Liu et al., 2020; Nguyen et al., 2022b). (Sennrich et al., 2016) was the first, and has been improved and refined in various works we cited above. Each of them is better than (Sennrich et al.) or complement it in other settings. **We thank the reviewer for bringing up this well-known work, and we will cite (Sennrich et al., 2016) in the paper!** But we would like to emphasize that we initially skipped its citation in the same way many papers nowadays have also stop citing (Vaswani, et al, 2017) Transformer paper due to its utmost popularity, but instead cite other improved versions, and not because we are not aware of it. We sincerely plead that missing an amendable citation is not punishable with rejection.

Reviewer comment (Novelty):
The primary distinction is that it is now presented in the form of a few-shot in-context learning, rather than for training purposes. One of the proposed comparisons involves fine-tuning using synthetic data in the opposite direction, which basically is the original back-translation concept. The absence of this reference is significant because it represents the core idea of this paper.

Author response:
**We follow your advice to discuss extensively about back-translation** in the camera-ready updated paper once accepted. Particularly, we will rewrite and/or add the following:

**In the introduction:**

Our LDP prompting technique is a simple application of back-translation (Sennrich et al., 2016) for the translation tasks under the extreme constraint where we can only use a general LLM without substantial amounts of parallel or unlabeled data for the low-resource languages, and especially where we cannot train or fine-tune the LLM if it is closed-source (OpenAI's GPTs) or too large (~175B params).

**In the related work:**

Our LDP prompting technique applies the concept of back-translation (Sennrich et al., 2016) in a novel situation of low-resource unsupervised translation, where a general pre-trained language model is used on low-resource languages without any further training. There are major distinctions between our method and standard back-translation (Sennrich et al., 2016). The first one is that standard back-translation requires two backward and forward supervised translation models to generate a huge amounts synthetic data from unlabeled target-language data. This renders the method inapplicable in our setup because only one generic language-model is available and no parallel or huge unlabeled data is available. The second difference is that our LDP method does not require any training to work well and use less than 8 unlabeled data samples. The third one is that our LDP method only use one-way back-translation (X->En) for both En->X and X->En translation directions, in order to avoid poor generative performance in low-resource languages.

Reviewer comment (Experiments):
The synthetic back-translation data was generated by providing in-context examples across a diverse set of languages (Figure 1a). However, this crucial idea is not thoroughly explored, considering the potential variability in the types of languages, examples, and diversity that could be explored. Conducting an ablation study involving different examples and languages would strengthen the paper's claims.

Author response:
4. **Ablation study**: **Table 4a** examines the different types of prompts, language tags and with/without synthetic backtranslation. **Table 4b** explored different language choices, from the number of languages to use, which groups of languages (distant or close), and repeating the same languages. We also conduct other ablation studies in **figure 3, figure 4, table 5, figure 5, and figure 6**.

Reviewer comment (Experiments):
While the author comments that the LDP method (Figure 1c) is superior to the standard few-shot approach (Figure 1b), there is a lack of experimental results to substantiate this claim.

Author response:
5. **Figure 1b vs 1c:** **Table 4a** compares Fig. 1b (without BT) vs Fig. 1c (with BT) and shows that Fig. 1c outperforms Fig. 1b greatly for En-->X direction, but observes no difference for X-->En direction. This is consistent with the explanations in figures 1a, 1b, and 1c. Please note that Fig. 1b is **not** standard supervised few-shot prompting, but our LDP method **without** synthetic intra-lingual backtranslation data.

Reviewer comment (Experiments):
Is there any specific reason on choosing BLOOM over BLOOMZ (instruct-tuned version of BLOOM)? I think it will be a more fair comparison vs InstructGPT.

Author response:
6. **Why BLOOM?** Because LDP is a fully unsupervised prompt technique, as we explain in page 6, we do not want the examined models to be contaminated with supervised data and especially test sets. BLOOMz was trained on test sets! InstructGPT experiments are to show that our method works on instruction-tuned models. We also tested on GPT-3 base and observe similar results, but its performance was much worse than BLOOM in absolute terms for these low-resource languages across different settings. We explain GPTs' weaknesses in figure 6.

Reviewer comment (Experiments):
One of the strengths of the original back-translation approach lies in its ability to generate synthetic data at scale, and the size of the generated data can influence performance. However, I am uncertain about the data size used for your fine-tuning comparison.

Author response:
7. **Finetuning data size:** sizes are very small! **<500-1000** samples for extremely low-resource languages like Tsonga, and **~50K** samples for more popular ones like Hindi. We understand BT needs millions of samples in data size, but it was difficult to obtain such data for low-resource languages. We will put these details in paper.

Reviewer comment (Reproducibility):
This paper has reproducibility issues. The results in the paper cannot be replicated, as the approaches are evaluated on 200 randomly sampled sentences from the test set, while there is no explanation or source provided that details which sentences are included or how to reproduce this selection (e.g. which random seed). Random data selection is also used in one of the baselines, namely supervised prompting (A.2) without providing details.

Author response:
**Results are reproducible!** We stated that in the paper **"For each of the 68 language pairs, we sample randomly and evaluate 200 sentences from each test set with the same seed to limit the cost of API calls"** . The seed is **0**! We also used seed **0** to collect the prompts for supervised prompting (A.2). In short, **every test set in the paper is the same across all experiments and baselines**.

Reviewer comment (Novelty):
Is LDP truly an unsupervised prompting method, or are some aspects more like obtaining data for a kind of weak supervision?

Author response:
LDP itself is **truly a fully and completely unsupervised** method, both from prompting or data collection perspectives. The only thing that makes the entire process not *fully unsupervised*  is which underlying model is used. InstructGPT is not an unsupervised model and we do not know which data it was trained on. Still, we demonstrate that our method can improve InstructGPT zero-shot performances for the target language where we do not have any supervised data at hand!

Reviewer comment (Experiments):
What are your criteria for distinguishing high-resource from low-resource languages?

Author response:
Which language is high or low resource is subjective. In our paper, we assume language with < 0.1% pretraining data or <1GB data size from the CC100 corpora as very low-resource.

Reviewer comment (Writing):
In what way does improving prompting performance for certain low-resource scenarios ’democratize’ LLMs (title)?

Author response:
**Democratizing**: our method is a simple strategy that can boost LLM's performance for low-resource languages that we would not have been able to achieve with naive zero-shot prompting, and without supervised data from those low-resource languages. For example, practitioners can build a simple summarization app for Swahili-speaking community, without ever needing to collect any supervised Swahili data, which would be hard to collect.

In this way, under-represented communities can benefits from existing LLMs without needing to invest in human annotations, training computations.

Reviewer comment (Experiments):
How are the diverse language sets selected? Do the authors observe any correlation between linguistically similar language selection versus a random set of languages?

Author response:
1. As demonstrated in **table 4b**, our method work best if all languages in prompts are different from each other and **must be very distant from the target language** you want to solve. Other than that, **as we have tried different combinations of language choices**, there is no significant difference in terms of performances.

Reviewer comment (Experiments):
Did the authors study the relation between the number of languages to be selected and the number of examples in the prompt?

Author response:
2. Number of languages or number of examples are limit by the context length, which is around 6-8. **table 4b** also shows that there is no difference if we put 3 (Fr,Es,Pt) or 7 (Ar,Fr,Es,Pt,Vi,Zh,Id) languages in the prompt. Meanwhile, repeating the same languages (Zh,Zh,Zh,Zh) may hurt performance.

Reviewer comment (Experiments):
It would be interesting to see the performance difference between selecting prompts from diverse languages versus creating synthetic prompts for just the pair of languages of interest.

Author response:
4. *Diverse vs Synthetic intra-lingual Prompts*: **table 4a** explains the requested results, indicated as LDP *without* vs *with* intralingual back-translation prompts. In short, there is no difference between them for X-->En direction, but ones with synthetic back-translation significantly outperforms the other and match supervised prompting for En-->X direction.