Paper: When Is Multilinguality a Curse? Language Modeling for 252 High- and Low-Resource Languages (ICLR-2024)

Reviewer comment (Writing):
I found some parts of the papers are a bit repetitive. For example, the first and the second paragraphs in the introduction section have extensive similar claims. The authors should improve the flow of writing to reduce such redundancy.

Author response:
*"For example, the first and the second paragraphs in the introduction section have extensive similar claims."* Thanks for pointing this out! We have removed redundancy in those paragraphs in this revision.

Reviewer comment (Experiments):
In the paragraph "Multilingual tokenizers", the author mentioned that the target language and the added multilingual datasets are tokenized separately. And the size for multilingual vocab is set to 32K tokens, which is the same as the monolingual tokenizer. Is this a limitation since 32K tokens might not be enough to support ten languages (especially when the scripts of the ten languages are different)?

Author response:
*"Is this a limitation since 32K tokens might not be enough to support ten languages?"* We agree that 32K tokens may not be enough to cover ten languages with high word coverage, so performance on the added ten languages may be non-optimal. For this reason, we only evaluate performance in the target language, which retains its 32K-vocab monolingual tokenizer. Increasing the multilingual vocabulary size would significantly increase computation costs due to added embedding parameters for each model.

Reviewer comment (Evaluation):
In the very interesting part of "Syntactic similarity of added languages drives results", the authors mentioned that abstract linguistic similarity might be more beneficial than surface-level similarity. I was wondering how conceptual similarity [1] would hold in this case. Could the authors give any thought to this?

Author response:
*"I was wondering how conceptual similarity would hold in this case."* The same analyses (correlations and variance partitioning; Section 6.1) could evaluate the effects of conceptual similarity of added languages on target language performance. To facilitate such evaluations, we plan to release the evaluation log-likelihood scores of all our models. Conceptual similarity is particularly interesting as an abstract similarity measure that is largely separate from structural (syntactic) similarity.

Reviewer comment (Evaluation):
The authors mentioned that the degradation is even slightly larger when the added languages are "similar" to the target languages. This sounds a bit counter-intuitive. Could the authors give some possible explanations?

Author response:
*"Degradation is even slightly larger when the added languages are 'similar' to the target languages. This sounds a bit counter-intuitive. Could the authors give some possible explanations?"* We agree that this result is surprising. One possible explanation is that more similar languages affect the target language more due to surface level and representational similarities between languages. In low-resource scenarios, these influences from other languages appear to improve predictions; however, in high-resource scenarios, when models are learning more fine-grained language-specific nuances, these influences might hurt performance, making similar languages hurt performance more.

Reviewer comment (Evaluation):
I found it a strong (unsupported/contradicting) claim when the authors argue that "the multilingual pre-training may not be beneficial to any languages involved". Because the obtained results already show that low-resource languages clearly benefit from multilinguality and that the high-resource performance degradation reduces as the model size increases. Providing supportive arguments regarding this claim might be useful.

Author response:
*"Strong (unsupported/contradicting) claim when the authors argue that 'the multilingual pre-training may not be beneficial to any languages involved.'"* Thank you for pointing out this ambiguity in the original text. By this, we intend to mean that multilingual pre-training is *non-optimal* for any languages involved. Multilingual pre-training degrades high-resource language performance relative to a monolingual model (Figure 5). An excess of multilingual data (e.g. massively multilingual pre-training with many dissimilar languages) degrades low-resource language performance relative to a targeted multilingual model with a moderate amount of data from similar languages (Figure 3). Even if model sizes are dramatically increased, models optimized for many languages simultaneously would likely require more parameters than would be computationally feasible (Section 7). We will clarify these claims in the final version of the manuscript (and the abstract wording has been updated in the PDF).

Reviewer comment (Experiments):
While the authors carried out a dramatic number of experiments, they do not provide detailed reasons why such a huge amount of computing resources would be needed. I assume selecting a set of languages that represent specific scenarios would be enough to get supportive results. And that could help in securing some resources that could be used to evaluate relatively larger language models which could have probably given a different picture of the results.

Author response:
*"I assume selecting a set of languages that represent specific scenarios would be enough to get supportive results."* Due to the diversity of human language (e.g. Joshi et al., 2020, "The state and fate of linguistic diversity and inclusion in the NLP world"), results from a small set of languages are not always reflective of results overall. One of the main aims of this study is to create a sample of languages that better represent the diversity of the world’s languages, because we don't know how insights from diverse languages might change our inferences. Our study includes languages from 5 continents, from 29 language families, and with 30 writing systems, along with a wide variety of linguistic typological features (see WALS database; https://wals.info/).  

*"And that could help in securing some resources that could be used to evaluate relatively larger language models which could have probably given a different picture of the results."* Unfortunately, contemporary LLMs require extremely large amounts of compute. Because pre-training costs are linear with respect to parameter count (Deshpande et al., 2023, "Honey, I shrunk the language: Language model behavior at reduced scale"), a 1B-parameter model would take ~22x the compute of one of our largest models (45M parameters). Thus, to run 1B-parameter models, we would need to reduce our experimental conditions by about 22x (e.g. reduce languages from 252 to 11). Of course, larger models could be tested with larger compute budgets, but we note that directions of effect are consistent across the three model sizes we evaluated; for additional comments on model size, see response to Reviewer HmV5.

Reviewer comment (Writing):
In the first line of page 6, it was mentioned that a total of 8454 multilingual language models were pre-trained while in the abstract it was mentioned that over 10000 models were trained. Is that a typo?

Author response:
*"It was mentioned that a total of 8454 multilingual language models were pre-trained while in the abstract it was mentioned that over 10000 models were trained."* We pre-train 8454 multilingual models and 1989 monolingual baseline models.

Reviewer comment (Novelty):
Unsurprising results.

Author response:
*"Their conclusions were already common knowledge."* We agree with the reviewer that there is previous evidence for the curse of multilinguality in language models (and evidence of benefits for low-resource languages; Section 2), but it is limited in generality. Previous findings are based on relatively few scenarios. For example, XLM-R considers only five subsets of pre-training languages for downstream tasks in 16 languages. Other studies (e.g. mBERT, BLOOM, and XGLM) simply pre-select a set of languages and fix a sampling parameter to upsample low-resource languages. Yet, despite the lack of controlled studies evaluating the effects of multilinguality on language modeling performance, language models are increasingly pre-trained on multilingual data (e.g. GPT-4 and PaLM 2).

Thus, our work makes several novel contributions:
1. It contributes a controlled method, a new measure (estimated monolingual tokens), and experimental manipulations that quantify effects of multilingual pre-training in ways that are comparable across languages and informative for pre-training in practice.
2. It quantifies these effects in over 250 languages. We plan to release the evaluation log-likelihood scores of all our models.
3. It confirms and quantifies several effects that are commonly assumed (i.e. we estimate comparable dataset sizes for low-resource performance improvements and high-resource performance degradations, for similar and dissimilar languages).
4. It uncovers several results that may be unexpected: syntactic similarity plays a greater role than vocabulary overlap in low-resource performance improvements, low-resource performance improvements decrease given excessive multilingual data, and more similar languages degrade high-resource performance more than dissimilar languages (see next point).

Reviewer comment (Evaluation):
Why do you think that selecting most related vs least related had relatively little impact?

Author response:
*"Why do you think that selecting most related vs least related had relatively little impact?"* We agree that the effects of language similarity are small in many, although not all (e.g. Figure 3, top right), cases. Some possible explanations are:
* Effects of multilinguality may be driven by linguistic similarities not captured well by our similarity metric (i.e. not syntactic, geographic, or lexical similarities). For example, see conceptual similarity (suggested by Reviewer Dd5N).
* The benefits of multilinguality may be based on fairly abstract linguistic properties that are shared to a large degree across most or many languages, regardless of relatedness (e.g. hierarchical structure in general). The drawbacks of multilinguality may be based on nuances that are highly language-specific, regardless of linguistic relatedness or similarity.
* We do not have closely related languages for some languages (particularly some low-resource languages), which might make the similar languages condition closer to the dissimilar languages condition for those languages.

Reviewer comment (Experiments):
The model sizes are all on the small end. One can see from e.g. Figs. 3 and 5 that, as model size increases, multilingual pretraining hurts performance less and less (and actually becomes beneficial, in a few cases). The obvious next step would then be to grow model size further. I fully acknowledge, however, that this kind of experiment would have prohibitive computational costs, and I believe that the paper is already valuable as-is. I further appreciate the authors explicitly calling out and addressing this limitation in sec. 7.1.

Author response:
*"The model sizes are all on the small end."* Our experiments take approximately 17700 A6000 GPU hours total, which is still less than 1/1500x the FLOPs required to train the original 175B-parameter GPT-3 model (computational costs added to Appendix A.3). Other research groups may have the computational resources to extend our experiments to larger models, but we believe that the sizes of our models are a reasonable compromise between informativity of results and environmental/computational costs. Our results still cover three model sizes, 250 languages, and four experimental manipulations. Directions of effect are consistent across the three model sizes we evaluated.

The added benefits of running similar experiments for larger models may also be limited. As dataset sizes increase to hundreds of billions of tokens, it is likely that larger models will still hit multilingual capacity limitations (e.g. 70B-parameter models are required just for English; Hoffmann et al., 2022, "Training compute-optimal large language models"). Furthermore, smaller models can perform comparably to larger models in low-resource scenarios (Section 7.1), making them useful for efficient low-resource language technologies and low-compute settings such as laptops and mobile phones. Nevertheless, results for larger models could be verified with larger compute budgets.

Reviewer comment (Experiments):
Do you expect these results to hold also for related tasks such as machine translation, speech recognition, etc.? Based on the current SOTA models for these related tasks, would you agree that the story is likely to be similar there too?

Author response:
*"Do you expect these results to hold also for related tasks such as machine translation, speech recognition, etc.?"* Although we cannot guarantee the same results for other tasks, we believe that results may be similar for tasks with similar setups. For example, tasks such as translation still use Transformer text generation models, although often with encoder-decoder architectures. Like in language modeling, massive multilinguality in machine translation appears to benefit low-resource languages (NLLB team, 2022, "No language left behind: Scaling human-centered machine translation"; Bapna et al., 2022, "Building machine translation systems for the next thousand languages"), but the benefits and drawbacks of massive multilinguality are rarely quantified using controlled studies. Results for less similar tasks and architectures (e.g. speech models) would need to be verified empirically. In any case, multilingual language modeling over text (e.g. for text generation; GPT-4, PaLM 2, BLOOM, XGLM) is increasingly popular as a task, with direct implications for downstream applications such as chatbots and virtual assistants.