Original: U NI M AX : F AIRER AND MORE E FFECTIVE L ANGUAGE S AMPLING FOR L ARGE -S CALE M ULTILINGUAL P RETRAINING
Revised: U NI M AX : F AIRER AND MORE E FFECTIVE L ANGUAGE S AMPLING FOR L ARGE -S CALE M ULTILINGUAL P RETRAINING

Original: Anonymous authors
Revised: Tay Sharan Narang

Original: Paper under double-blind review
Revised: Orhan Firat

Original: A BSTRACT
Revised: A BSTRACT

Original: Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance between different languages.
Revised: Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance between different languages.

Original: However previous work has not systematically evaluated the efﬁcacy of different pretraining language distributions across model scales.
Revised: However previous work has not systematically evaluated the efﬁcacy of different pretraining language distributions across model scales.

Original: In this paper, we propose a new sampling method, U NI M AX , that delivers more uniform coverage of head languages while mitigating overﬁtting on tail languages by explicitly capping the number of repeats over each language’s corpus.
Revised: In this paper, we propose a new sampling method, U NI M AX , that delivers more uniform coverage of head languages while mitigating overﬁtting on tail languages by explicitly capping the number of repeats over each language’s corpus.

Original: We perform an extensive series of ablations testing a range of sampling strategies on a suite of multilingual benchmarks, while varying model scale.
Revised: We perform an extensive series of ablations testing a range of sampling strategies on a suite of multilingual benchmarks, while varying model scale.

Original: We ﬁnd that U NI M AX outperforms standard temperature-based sampling, and the beneﬁts persist as scale increases.
Revised: We ﬁnd that U NI M AX outperforms standard temperature-based sampling, and the beneﬁts persist as scale increases.

Original: As part of our contribution, we release an improved and refreshed variant of the mC multilingual corpus consisting of 29 trillion characters across 107 languages. In addition we release full code to reproduce our experiments.
Revised: As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with U NI M AX sampling.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: State-of-the-art multilingual models (Xue et al., 2021; 2022; Goyal et al., 2021, inter alia ) utilize large-scale self-supervised learning, which involves jointly training on many languages.
Revised: State-of-the-art multilingual models (Xue et al., 2021; 2022; Goyal et al., 2021, inter alia ) utilize large-scale self-supervised learning, which involves jointly training on many languages.

Original: Because data availability varies greatly across languages, multilingual pretraining can be characterized as multitask learning (or multi-objective optimization) with severe data imbalance.
Revised: Because data availability varies greatly across languages, multilingual pretraining can be characterized as multitask learning (or multi-objective optimization) with severe data imbalance.

Original: Typically English is the highest-resource language (or task) with orders of magnitude larger size than lower-resource languages.
Revised: Typically English is the highest-resource language (or task) with orders of magnitude larger size than lower-resource languages.

Original: For example, in the mC4 corpus (Xue et al., 2021), English has roughly 9 .
Revised: For example, in the mC4 corpus (Xue et al., 2021), English has roughly 9 .

Original: 7 trillion characters, which is over 92 , 000 times larger than the lowest resource language, Yoruba.
Revised: 7 trillion characters, which is over 92 , 000 times larger than the lowest resource language, Yoruba.

Original: As a result, a key problem in designing such models is the “language balancing” problem: in what proportions should we balance the pretraining languages?
Revised: As a result, a key problem in designing such models is the “language balancing” problem: in what proportions should we balance the pretraining languages?

Original: Deriving the optimal balance is a difﬁcult open research problem due to the high cost of pretraining.
Revised: Deriving the optimal balance is a difﬁcult open research problem due to the high cost of pretraining.

Original: The standard approach to this problem has been to upsample languages with smaller datasets, using a temperature hyperparameter τ (Devlin et al., 2019).
Revised: The standard approach to this problem has been to upsample languages with smaller datasets, using a temperature hyperparameter τ (Devlin et al., 2019).

Original: However, one shortcoming of this approach is that choosing τ based on the desired distribution among higher-resources languages may result in examples from the lowest-resource languages being repeated excessively.
Revised: However, one shortcoming of this approach is that choosing τ based on the desired distribution among higher-resources languages may result in examples from the lowest-resource languages being repeated excessively.

Original: Figure 1a shows the number of epochs covered for each language in the mC4 corpus.
Revised: Figure 1a shows the number of epochs covered for each language in the mC4 corpus.

Original: When using τ = 3 .
Revised: When using τ = 3 .

Original: 33 and a trillion token budget (the values used in popular models such as mT5 and ByT5), the lowest-resource languages are repeated over 100 times.
Revised: 33 and a trillion token budget (the values used in popular models such as mT5 and ByT5), the lowest-resource languages are repeated over 100 times.

Original: This excessive repetition can have several unwanted consequences: (i) it leads to overﬁtting, which degrades performance on downstream tasks (Raffel et al., 2020; Lee et al., 2022; Hernandez et al., 2022), (ii) it increases the risk of memorizing private or sensitive content (Carlini et al., 2021; Lee et al., 2022), and (iii) it wastes training cycles that could have been devoted to unique examples.
Revised: This excessive repetition can have several unwanted consequences: (i) it leads to overﬁtting, which degrades performance on downstream tasks (Raffel et al., 2020; Lee et al., 2022; Hernandez et al., 2022), (ii) it increases the risk of memorizing private or sensitive content (Carlini et al., 2021; Lee et al., 2022), and (iii) it wastes training cycles that could have been devoted to unique examples.

Original: As models continue to grow in scale (Chowdhery et al., 2022; Brown et al., 2020; Smith et al., 2022), these issues with temperature sampling grow more pressing, as larger models beneﬁt from longer training (Hoffmann et al., 2022), overﬁt more easily, and have a greater capacity to memorize.
Revised: As models continue to grow in scale (Chowdhery et al., 2022; Brown et al., 2020; Smith et al., 2022), these issues with temperature sampling grow more pressing, as larger models beneﬁt from longer training (Hoffmann et al., 2022), overﬁt more easily, and have a greater capacity to memorize.

Original: This paper proposes a new paradigm for sampling across languages and datasets that ameliorates the above mentioned problems.

Original: We propose U NI M AX (uniform + max), a conceptually simple but highly effective two-pronged sampling approach that results in fairer and more effective language

Original: =3.33 (1 / [Figure] budget)
Revised: =3.33 (1 / 8 budget)

Original: UniMax (1 / 8 budget) = [Figure] [Figure] (1x 
Revised: UniMax (1 / 8 budget) =3.33 (1x budget)

Original: [Figure]

Original: UniMax [Figure] 1x
Revised: UniMax (1x budget)

Original: [Figure]

Original: Language rank
Revised: Language rank

Original: N u m b e r o f
Revised: N u m b e r o f

Original: e po
Revised: e po

Original: c h s
Revised: c h s

Revised: [Figure]

Original: = =3.
Revised: = =3.

Original: UniMax (1× budget)
Revised: UniMax (1× budget)

Original: UniMax (1 / 8 budget)
Revised: UniMax (1 / 8 budget)

Original: Language rank
Revised: Language rank

Revised: P r

Original: [Figure]
Revised: [Figure]

Original:  ng
Revised: a i n i ng

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Revised: m p li ng

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  distributions for pretraining multilingual language models that work well across model scales.
Revised: whereas U NI M AX provides more uniform distribu- tions without excessive upsampling. This paper proposes a new paradigm for sampling across languages and datasets that ameliorates the above mentioned problems. We propose U NI M AX (uniform + max), a conceptually simple but highly effective two-pronged sampling approach that results in fairer and more effective language distributions for pretraining multilingual language models that work well across model scales.

Original: One of the main assumptions we make is that practical large-scale training jobs operate with a ﬁxed amount of compute, which is often translated into a ﬁxed training token budget (Raffel et al., 2020).
Revised: One of the main assumptions we make is that practical large-scale training jobs operate with a ﬁxed amount of compute, which is often translated into a ﬁxed training token budget (Raffel et al., 2020).

Original: U NI M AX starts by pre-allocating training tokens to underrepresented datasets based on the number of allowed max repeats ( N ).
Revised: U NI M AX starts by pre-allocating training tokens to underrepresented datasets based on the number of allowed max repeats ( N ).

Original: For the remaining budget, we prioritize “linguistic utility” (Blasi et al., 2022) by allocating uniformly across all languages with sufﬁcient data to avoid exceeding the prescribed number of per-language epochs.
Revised: For the remaining budget, we prioritize “linguistic utility” (Blasi et al., 2022) by allocating uniformly across all languages with sufﬁcient data to avoid exceeding the prescribed number of per-language epochs.

Original: Unlike previous approaches, this means U NI M AX is relatively resistant to distribution biases that arise due to artifacts of the corpus generation process (i.e., web crawlers).
Revised: Unlike previous approaches, this means U NI M AX is relatively resistant to distribution biases that arise due to artifacts of the corpus generation process (i.e., web crawlers).

Original: To take a concrete example, the mC4 corpus contains 70 × more English than
Revised: To take a concrete example, the mC4 corpus contains 70 × more English than

Original: Chinese text.
Revised: Chinese text.

Original: While mT5’s temperature sampling ( τ = 3 .
Revised: While mT5’s temperature sampling ( τ = 3 .

Original: 33 ) results in training on 3 .
Revised: 33 ) results in training on 3 .

Original: 4 × more
Revised: 4 × more

Original: English than Chinese, U NI M AX will assign equal training tokens to the two languages, provided that this doesn’t result in repeating the 39 billion available Chinese tokens more than N times.
Revised: English than Chinese, U NI M AX will assign equal training tokens to the two languages, provided that this doesn’t result in repeating the 39 billion available Chinese tokens more than N times.

Original: Another key beneﬁt of U NI M AX is that it is robust to model scaling.
Revised: Another key beneﬁt of U NI M AX is that it is robust to model scaling.

Original: In considering language sampling strategies at scale, it is important to carefully control how many times a dataset can be repeated during training to avoid overﬁtting and memorization.
Revised: In considering language sampling strategies at scale, it is important to carefully control how many times a dataset can be repeated during training to avoid overﬁtting and memorization.

Original: Our proposed method explicitly controls the extent of data repeats of any language, providing a direct countermeasure to overﬁtting on low-resource languages, without imposing any reprioritization on higher-resource languages.
Revised: Our proposed method explicitly controls the extent of data repeats of any language, providing a direct countermeasure to overﬁtting on low-resource languages, without imposing any reprioritization on higher-resource languages.

Original: Our key contributions are to: (1) Propose U NI M AX , a simple but effective language sampling strategy that provides more uniform coverage of high-resource languages while mitigating overﬁtting on low-resource languages.
Revised: Our key contributions are to: (1) Propose U NI M AX , a simple but effective language sampling strategy that provides more uniform coverage of high-resource languages while mitigating overﬁtting on low-resource languages.

Original: (2) Perform an extensive series of ablations testing a range of sampling strategies on a suite of multilingual benchmarks, while varying model scale.
Revised: (2) Perform an extensive series of ablations testing a range of sampling strategies on a suite of multilingual benchmarks, while varying model scale.

Original: (3) Release an improved and refreshed variant of the mC4 multilingual corpus consisting of 29 trillion characters across 107 languages.
Revised: (3) Release an improved and refreshed variant of the mC4 multilingual corpus consisting of 29 trillion characters across 107 languages.

Original: (4) Release pretrained model checkpoints using U NI M AX sampling.
Revised: (4) Release pretrained model checkpoints using U NI M AX sampling.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: While (massively-)multilingual models enjoy the beneﬁts of positive transfer across languages, the sheer number of languages reduces the effective capacity of the model per task.
Revised: While (massively-)multilingual models enjoy the beneﬁts of positive transfer across languages, the sheer number of languages reduces the effective capacity of the model per task.

Original: This competition among languages for limited model capacity is the well-known problem of “capacity bottleneck” (Arivazhagan et al., 2019), also known as the “curse of multilinguality” (Conneau et al., 2020).
Revised: This competition among languages for limited model capacity is the well-known problem of “capacity bottleneck” (Arivazhagan et al., 2019), also known as the “curse of multilinguality” (Conneau et al., 2020).

Original: In exploring the interaction between language balancing and model scale, previous work has ablated sampling temperature with models at or below 1 billion parameters (Conneau et al., 2020; Xue et al., 2021), but we believe our study is the ﬁrst to systematically explore balancing strategies at scales above 1 billion parameters.
Revised: In exploring the interaction between language balancing and model scale, previous work has ablated sampling temperature with models at or below 1 billion parameters (Conneau et al., 2020; Xue et al., 2021), but we believe our study is the ﬁrst to systematically explore balancing strategies at scales above 1 billion parameters.

Original: Additionally, our work targets general-purpose encoder-decoder models, whereas most previous work used encoder-only models or targeted machine translation exclusively.
Revised: Additionally, our work targets general-purpose encoder-decoder models, whereas most previous work used encoder-only models or targeted machine translation exclusively.

Revised: Michel et al.

Revised: (2021) explore adaptive mixing strategies in a similar setting, and also ﬁnd that ﬁxed uniform mixing is a strong baseline for multilingual pretraining.

Original: Within the context of machine translation 1 , Jean et al.
Revised: Within the context of machine translation 1 , Jean et al.

Original: (2019) proposed using a bi-level optimization algorithm maximizing the validation utility to adjust the sampling weights per language.
Revised: (2019) proposed using a bi-level optimization algorithm maximizing the validation utility to adjust the sampling weights per language.

Original: Arivazhagan et al.
Revised: Arivazhagan et al.

Original: (2019) was the ﬁrst testing various temperatures ( τ = 1 , 5 , 100 ) to train a multilingual translation model for 212 language pairs.
Revised: (2019) was the ﬁrst testing various temperatures ( τ = 1 , 5 , 100 ) to train a multilingual translation model for 212 language pairs.

Original: The study suggested over-sampling low-resource while sub-sampling high-resource languages to maximize a uniform utility.
Revised: The study suggested over-sampling low-resource while sub-sampling high-resource languages to maximize a uniform utility.

Original: The over-training/over-ﬁtting due to over-sampling was alleviated by increasing the number of tasks in that study.
Revised: The over-training/over-ﬁtting due to over-sampling was alleviated by increasing the number of tasks in that study.

Original: Wang et al.
Revised: Wang et al.

Original: (2020b) proposed another bi-level approach, using differentiable data selection (Wang et al., 2020a).
Revised: (2020b) proposed another bi-level approach, using differentiable data selection (Wang et al., 2020a).

Original: The study includes a uniform language distribution baseline ( τ = ∞ ), but this differs from UniMax in having no limit on repetition of examples in low-resource languages.
Revised: The study includes a uniform language distribution baseline ( τ = ∞ ), but this differs from UniMax in having no limit on repetition of examples in low-resource languages.

Original: Recently Wang et al.
Revised: Recently Wang et al.

Original: (2021) proposed using the gradient alignment information across languages to adjust the language weights,
Revised: (2021) proposed using the gradient alignment information across languages to adjust the language weights,

Original: but it is unclear if this can be practically scaled to 100 or more languages.
Revised: but it is unclear if this can be practically scaled to 100 or more languages.

Original: Work on cross-lingual transfer learning has repeatedly found that English may not be the “ideal” source language for transfer.
Revised: Work on cross-lingual transfer learning has repeatedly found that English may not be the “ideal” source language for transfer.

Original: Malkin et al.
Revised: Malkin et al.

Original: (2022) conduct an analysis of zero-shot transfer, and calculate a “donation score” for each of 22 languages, measuring whether its inclusion in pretraining helps or hurts performance in other languages.
Revised: (2022) conduct an analysis of zero-shot transfer, and calculate a “donation score” for each of 22 languages, measuring whether its inclusion in pretraining helps or hurts performance in other languages.

Original: They ﬁnd English is among the “least donating”— one of only four languages whose inclusion results in an overall degradation in performance for other languages.
Revised: They ﬁnd English is among the “least donating”— one of only four languages whose inclusion results in an overall degradation in performance for other languages.

Original: In a similar exploration, Turc et al.
Revised: In a similar exploration, Turc et al.

Original: ﬁnd that German and Russian are more effective source languages for zero-shot transfer than English.
Revised: ﬁnd that German and Russian are more effective source languages for zero-shot transfer than English.

Original: These ﬁndings support the idea of testing sampling strategies where English doesn’t outweigh other languages.
Revised: These ﬁndings support the idea of testing sampling strategies where English doesn’t outweigh other languages.

Original: We focus here on the problem of balancing languages with signiﬁcant web data.
Revised: We focus here on the problem of balancing languages with signiﬁcant web data.

Original: In particular, we limit our scope to the approximately hundred languages that occur with some frequency in the
Revised: In particular, we limit our scope to the approximately hundred languages that occur with some frequency in the

Original: CommonCrawl corpus, as detected by CLD3 language detection.
Revised: CommonCrawl corpus, as detected by CLD3 language detection.

Original: Other methods are likely needed to scale NLP methods to thousands of languages, e.g., see Wang et al.
Revised: Other methods are likely needed to scale NLP methods to thousands of languages, e.g., see Wang et al.

Original: (2022) and Bapna et al.
Revised: (2022) and Bapna et al.

Original:  S AMPLING M ETHODS
Revised:  S AMPLING M ETHODS

Original: Temperature-based sampling In order to deﬁne a sampling distribution over languages, we use an empirical distribution for each language
Revised: Temperature-based sampling In order to deﬁne a sampling distribution over languages, we use an empirical distribution for each language

Original: [Equation]
Revised: [Equation]

Original: where n l represents the “size” of language l , as discussed in Section § 3.1.
Revised: where n l represents the “size” of language l , as discussed in Section § 3.1.

Original: A temperature-based sampling strategy uses a distribution q deﬁned by exponentiating p l by the inverse of the temperature and renormalizing.
Revised: A temperature-based sampling strategy uses a distribution q deﬁned by exponentiating p l by the inverse of the temperature and renormalizing.

Original: [Equation]
Revised: [Equation]

Original: Commonly used temperature values in the literature are summarized in Table 1.
Revised: Commonly used temperature values in the literature are summarized in Table 1.

Original: Higher temperature makes the distribution “ﬂatter”, approaching the uniform distribution as τ → ∞ .
Revised: Higher temperature makes the distribution “ﬂatter”, approaching the uniform distribution as τ → ∞ .

Original: In the other extreme with τ = 0 , the entire probability mass is contained in the language with the highest probability.
Revised: In the other extreme with τ = 0 , the entire probability mass is contained in the language with the highest probability.

Original: Note, there is no guarantee that there exists a value of τ that achieves the desired balance on high- and mid-resource languages while still avoiding overﬁtting on the tail languages.
Revised: Note, there is no guarantee that there exists a value of τ that achieves the desired balance on high- and mid-resource languages while still avoiding overﬁtting on the tail languages.

Original: U NI M AX sampling For U NI M AX sampling, we start with a predeﬁned character budget C .
Revised: U NI M AX sampling For U NI M AX sampling, we start with a predeﬁned character budget C .

Original: In practice, this is typically deﬁned by the training compute allocated for the training job.
Revised: In practice, this is typically deﬁned by the training compute allocated for the training job.

Original: The goal  of U NI M AX sampling is to allocate the character budget to languages as uniformly as possible, without using more than N epochs of any language.
Revised: The goal [Table] of U NI M AX sampling is to allocate the character budget to languages as uniformly as possible, without using more than N epochs of any language.

Original: The ﬁrst step is to sort the languages based on the character count in the training corpus.
Revised: The ﬁrst step is to sort the languages based on the character count in the training corpus.

Original: We iterate over languages starting from the one with the lowest character count.
Revised: We iterate over languages starting from the one with the lowest character count.

Original: At each iteration, we check if the remaining character budget can be split evenly among the remaining languages without using more than N epochs of any language.
Revised: At each iteration, we check if the remaining character budget can be split evenly among the remaining languages without using more than N epochs of any language.

Original: If so,[Table] we allocate the budget uniformly.
Revised: If so, we allocate the budget uniformly.

Original: If not, language l is allocated N epochs worth of characters and the remaining budget is reduced.
Revised: If not, language l is allocated N epochs worth of characters and the remaining budget is reduced.

Original: Algorithm 1 formalizes this procedure.
Revised: Algorithm 1 formalizes this procedure.

Original: Algorithm1: U NI M AX
Revised: Algorithm

Original: Inputs : Character count cl of each languagel in all the languages L of the training corpus
Revised: Inputs : Character count c of each language in all the languages L of the training corpus

Original: Total character budget C
Revised: Total character budget C

Original: The number of epochs per language N
Revised: The number of epochs per language N

Original: Output: Sampling distribution p l of each language //
Revised: Output: Sampling distribution p l of each language //

Original: Sort the languages by increasing number of character counts
Revised: Sort the languages by increasing number of character counts

Original: L ← sortByCount ( L )
Revised: L ← sortByCount ( L )

Original: B ← C // Initialize the remaining budget to the total character budget i ←for l in L do b l ←
Revised: B ← C // Initialize the remaining budget to the total character budget i ←for l in L do b l ←

Original: B len ( L ) − i //
Revised: B len ( L ) − i //

Original: Compute the remaining budget per-language
Revised: Compute the remaining budget per-language

Original: if b l > c l × N then //
Revised: if b l > c l × N then //

Original: If per-language budget exceeds N epochs of l , use N epochs
Revised: If per-language budget exceeds N epochs of l , use N epochs

Original: U l ← c l × N else
Revised: U l ← c l × N else

Original: U l ← b l // Otherwise use uniform per-language budget end if
Revised: U l ← b l // Otherwise use uniform per-language budget end if

Original: B ← B − U l // Update the remaining budget i ← i + 1 end for p ← normalize ( U ) return p
Revised: B ← B − U l // Update the remaining budget i ← i + 1 end for p ← normalize ( U ) return p

Original: 3. V OCABULARY GENERATION PROCESS
Revised: 3. V OCABULARY GENERATION PROCESS

Original: Following Xue et al.
Revised: Following Xue et al.

Original: (2021), we use SentencePiece tokenization (Kudo & Richardson, 2018), which consumes a corpus to produce a subword-level tokenizer.
Revised: (2021), we use SentencePiece tokenization (Kudo & Richardson, 2018), which consumes a corpus to produce a subword-level tokenizer.

Original: This corpus is sampled from the available training data, typically using the same distribution used during model training.
Revised: This corpus is sampled from the available training data, typically using the same distribution used during model training.

Original: In the multilingual setting, however, this amounts to an instance of the language balancing problem.
Revised: In the multilingual setting, however, this amounts to an instance of the language balancing problem.

Original: While many works explore varying the temperature of the training distribution, few perform the analogous change on the vocabulary-learning distribution.
Revised: While many works explore varying the temperature of the training distribution, few perform the analogous change on the vocabulary-learning distribution.

Original: This could have drastic consequences in certain situations.
Revised: This could have drastic consequences in certain situations.

Original: For example, if the vocabulary learning distribution heavily favors Chinese, but the training distribution heavily favors English, models using this vocabulary may experience poor performance due to insufﬁcient vocabulary coverage (Chung et al., 2020).
Revised: For example, if the vocabulary learning distribution heavily favors Chinese, but the training distribution heavily favors English, models using this vocabulary may experience poor performance due to insufﬁcient vocabulary coverage (Chung et al., 2020).

Original: Most sampling strategies require some notion of “size” ( n l ) of each language l .
Revised: Most sampling strategies require some notion of “size” ( n l ) of each language l .

Original: This is typically computed by counting words or tokens using a pre-existing multilingual tokenizer.
Revised: This is typically computed by counting words or tokens using a pre-existing multilingual tokenizer.

Original: But beyond the unwanted complexity of requiring one “pre-tokenizer” to train a second tokenizer, such strategies do not generalize well to the massively multilingual setting, as tokenization across 100 + languages is non-trivial, particularly in languages written without spaces (e.g. Chinese, Japanese, Thai).
Revised: But beyond the unwanted complexity of requiring one “pre-tokenizer” to train a second tokenizer, such strategies do not generalize well to the massively multilingual setting, as tokenization across 100 + languages is non-trivial, particularly in languages written without spaces (e.g. Chinese, Japanese, Thai).

Original: We propose to resolve the above issues by training separate vocabularies for each sampling strategy, using character counts as our measure of sub-corpus size.
Revised: We propose to resolve the above issues by training separate vocabularies for each sampling strategy, using character counts as our measure of sub-corpus size.

Original: As previous work has tended to measure training budget in tokens , when comparing against such approaches, we assume a token containscharacters on average, following Xue et al.
Revised: As previous work has tended to measure training budget in tokens , when comparing against such approaches, we assume a token containscharacters on average, following Xue et al.

Original: We use this same conversion rate to determine the total character budget C in Algorithm 1, given a real training budget in terms of tokens.
Revised: We use this same conversion rate to determine the total character budget C in Algorithm 1, given a real training budget in terms of tokens.

Original:  E XPERIMENTS 4. P RETRAINING C ORPUS
Revised:  E XPERIMENTS 4. P RETRAINING C ORPUS

Original: To ensure meaningful results when comparing language sampling strategies, it is critical that the examples in our pretraining corpus are correctly labeled for language.
Revised: To ensure meaningful results when comparing language sampling strategies, it is critical that the examples in our pretraining corpus are correctly labeled for language.

Original: However language detection is far from a solved problem (Caswell et al., 2020), and audits have found low accuracy across a range of public multilingual datasets (Kreutzer et al., 2022).
Revised: However language detection is far from a solved problem (Caswell et al., 2020), and audits have found low accuracy across a range of public multilingual datasets (Kreutzer et al., 2022).

Original: As one severe example, only 40% of the documents in the Marathi language bucket of the popular mC4 dataset (3.0.1) were found to be well-formed Marathi, with 50% coming from other languages, and 10% being non-language.
Revised: As one severe example, only 40% of the documents in the Marathi language bucket of the popular mC4 dataset (3.0.1) were found to be well-formed Marathi, with 50% coming from other languages, and 10% being non-language.

Original: To mitigate this problem, we construct a new multilingual corpus by ﬁltering mC4 (Xue et al., 2021) to remove documents whose language ID conﬁdence is below 0 .
Revised: To mitigate this problem, we construct a new multilingual corpus by ﬁltering mC4 (Xue et al., 2021) to remove documents whose language ID conﬁdence is below 0 .

Original: Compared to the original mC corpus (which used a threshold of 0 . 7 ), this removes 6.1% of documents; however only 5.1% of characters are removed, as the ﬁltered low-conﬁdence documents also tend to be shorter.
Revised: Compared to the original mC corpus (which used a threshold of 0 . 7 ), this removes 6.1% of documents; however only 5.1% of characters are removed, as the ﬁltered low-conﬁdence documents also tend to be shorter.

Original: The most ﬁltered languages are Welsh ( 86 %), Sindhi ( 85 %) and Luxembourgish ( 84 %).
Revised: The most ﬁltered languages are Welsh ( 86 %), Sindhi ( 85 %) and Luxembourgish ( 84 %).

Original: Of the 20 languages where > 50 % of documents are ﬁltered, nearly all ( 19 of 20 ) are low-resource (mC4 rank > 40 ), with the only exception being Indonesian (rank 31).

Original: Manual inspection of several hundred examples across a subset of languages indicated a clear reduction in mislabeled documents.
Revised: Inspection of several hundred examples across a subset of languages indicated a clear reduction in mislabeled documents.

Original: 4. V OCABULARY
Revised: 4. V OCABULARY

Original: As discussed in Section § 3.1, we train a dedicated SentencePiece vocabulary for each language sampling method considered.
Revised: As discussed in Section § 3.1, we train a dedicated SentencePiece vocabulary for each language sampling method considered.

Original: We analyze these vocabularies in terms of token lengths and coverage of various writing scripts in Appendix A. Overall, the expected trends are observed that (i) higher temperature, (ii) smaller U NI M AX training budget, and (iii) higher U NI M AX max-epoch threshold ( N ) all lead to vocabularies that allocate capacity more uniformly across languages.
Revised: We analyze these vocabularies in terms of token lengths and coverage of various writing scripts in Appendix A. Overall, the expected trends are observed that (i) higher temperature, (ii) smaller U NI M AX training budget, and (iii) higher U NI M AX max-epoch threshold ( N ) all lead to vocabularies that allocate capacity more uniformly across languages.

Original: This more uniform allocation results in fewer tokens from higher-resource language scripts (e.g. Latin), more tokens from lower-resource language scripts (e.g. Devanagari), and a shift towards shorter tokens.
Revised: This more uniform allocation results in fewer tokens from higher-resource language scripts (e.g. Latin), more tokens from lower-resource language scripts (e.g. Devanagari), and a shift towards shorter tokens.

Original: 4. E VALUATION T ASKS
Revised: 4. E VALUATION T ASKS

Original: In selecting evaluation tasks, we aim to satisfy several key properties.
Revised: In selecting evaluation tasks, we aim to satisfy several key properties.

Original: First, tasks should be linguistically diverse , covering a range of languages from distinct families and regions, including both high- and low-resource languages.
Revised: First, tasks should be linguistically diverse , covering a range of languages from distinct families and regions, including both high- and low-resource languages.

Original: Second, tasks should be free of language bias .
Revised: Second, tasks should be free of language bias .

Original: For example, the task training data and evaluation metrics should be well-balanced across languages.
Revised: For example, the task training data and evaluation metrics should be well-balanced across languages.

Original: We also avoid benchmarks where English plays a special role, including datasets constructed in English and translated post-hoc to other languages, as well as zero-shot transfer tasks where English is the sole source language.
Revised: We also avoid benchmarks where English plays a special role, including datasets constructed in English and translated post-hoc to other languages, as well as zero-shot transfer tasks where English is the sole source language.

Original: Finally, to the degree possible, benchmarks should be realistic , such that performing better on the benchmark gives us conﬁdence that a model will do better on actual tasks facing language technology users.
Revised: Finally, to the degree possible, benchmarks should be realistic , such that performing better on the benchmark gives us conﬁdence that a model will do better on actual tasks facing language technology users.

Original: This is in contrast to “intermediate structure” tasks such as part-of-speech tagging.
Revised: This is in contrast to “intermediate structure” tasks such as part-of-speech tagging.

Original: TyDi QA (Clark et al., 2020) is a multilingual question-answering benchmark covering a range of typologically diverse languages.
Revised: TyDi QA (Clark et al., 2020) is a multilingual question-answering benchmark covering a range of typologically diverse languages.

Original: Questions are written from scratch by native speakers in each language, ensuring culturally relevant content and the absence of “translationese”.
Revised: Questions are written from scratch by native speakers in each language, ensuring culturally relevant content and the absence of “translationese”.

Original: We use the “GoldP” task, which covers 9 languages and asks models to identify the span answering a question within a speciﬁc passage.
Revised: We use the “GoldP” task, which covers 9 languages.

Original: To evaluate candidate models, we use the “in-language multitask” setting (Hu et al., 2020)—ﬁne-tuning on a mixture of all available languages, and evaluating in each language separately.
Revised: To evaluate candidate models, we use the “in-language multitask” setting (Hu et al., 2020)—ﬁne-tuning on a mixture of all available languages, and evaluating in each language separately.

Original: To maximize per-language performance, we select per-language checkpoints based on the validation performance, and report validation metrics, as no test set is provided.
Revised: To maximize per-language performance, we select per-language checkpoints based on the validation performance, and report validation metrics, as no test set is provided.

Original: The WMT21 shared task on large-scale multilingual machine translation (Wenzek et al., 2021) tests the ability of single model to translate across many languages.
Revised: The WMT21 shared task on large-scale multilingual machine translation (Wenzek et al., 2021) tests the ability of single model to translate across many languages.

Original: We focus on the “small track” tasks, each testing translation between 6 languages, in all 30 combinations.
Revised: We focus on the “small track” tasks, each testing translation between 6 languages, in all 30 combinations.

Original: As with TyDi QA, we ﬁnetune a single multilingual model on the mixture of all tasks and select per-language-pair checkpoints based on the test set performance.
Revised: As with TyDi QA, we ﬁne- tune a single multilingual model on the mixture of all tasks and select per-language-pair checkpoints based on the test set performance.

Original: 2 To reduce heavy English bias, we restrict training data to 1 ,examples per language pair by randomly subsampling from the training data.
Revised: 2 To reduce heavy English bias, we restrict training data to 1 ,examples per language pair by randomly subsampling from the training data.

Original: Limiting the size of the ﬁne-tuning dataset also increases the importance of transferring knowledge from pretraining, thereby making more apparent the differences between various pretraining sampling strategies.
Revised: Limiting the size of the ﬁne-tuning dataset also increases the importance of transferring knowledge from pretraining, thereby making more apparent the differences between various pretraining sampling strategies.

Original: We also evaluate on several widely adopted multilingual benchmarks: XNLI (Conneau et al., 2018),
Revised: We also evaluate on several widely adopted multilingual benchmarks: XNLI (Conneau et al., 2018),

Original: XQuAD (Artetxe et al., 2020), MLQA (Lewis et al., 2020) and PAWS-X
Revised: XQuAD (Artetxe et al., 2020), MLQA (Lewis et al., 2020) and PAWS-X

Original: (Yang et al., 2019).
Revised: (Yang et al., 2019).

Original: While these do not satisfy all of our desiderata above (e.g., some are translated from English and some skew towards high-resource languages), their popularity still makes them valuable points of reference.
Revised: While these do not satisfy all of our desiderata above (e.g., some are translated from English and some skew towards high-resource languages), their popularity still makes them valuable points of reference.

Original: 4. T RAINING S ETUP
Revised: 4. T RAINING S ETUP

Original: We closely follow mT5 (Xue et al., 2021) for model architecture and training procedure.
Revised: We closely follow mT5 (Xue et al., 2021) for model architecture and training procedure.

Original: Speciﬁcally, we use an encoder-decoder Transformer architecture.
Revised: Speciﬁcally, we use an encoder-decoder Transformer architecture.

Original: We use the span corruption pretraining objective from T5 (Raffel et al., 2020) on a multilingual corpus consisting of 101 languages plus
Revised: We use the span corruption pretraining objective from T5 (Raffel et al., 2020) on a multilingual corpus consisting of 101 languages plus

Original: Latin-script variants (e.g. ru-Latn ).
Revised: Latin-script variants (e.g. ru-Latn ).

Original: We use batch size of 1024 sequences where each sequence is deﬁned by selecting a chunk of 568 tokens from the training corpus.
Revised: We use batch size of 1024 sequences where each sequence is deﬁned by selecting a chunk of 568 tokens from the training corpus.

Original: This is then split into 512 input and 114 target tokens.
Revised: This is then split into 512 input and 114 target tokens.

Original: For the results in Section § 5, the number of training steps is 250 , 000 unless otherwise stated.
Revised: For the results in Section § 5, the number of training steps is 250 , 000 unless otherwise stated.

Original: Additional training details can be found inthe Appendix C.
Revised: Additional training details can be found in Appendix C.

Original:  R ESULTS 5. P RETRAINING LOSS
Revised:  R ESULTS 5. P RETRAINING LOSS

Original: One challenge in evaluating multilingual models is the relative paucity of benchmarks that cover a broad range languages and are free from language bias (see Section § 4.3).
Revised: One challenge in evaluating multilingual models is the relative paucity of benchmarks that cover a broad range languages and are free from language bias (see Section § 4.3).

Original: Numerous studies have established strong correlation between pretraining and ﬁne-tuning performance (Devlin et al., 2019; Narang et al., 2021; Tay et al., 2022, inter alia ).
Revised: Numerous studies have established strong correlation between pretraining and ﬁne-tuning performance (Devlin et al., 2019; Narang et al., 2021; Tay et al., 2022, inter alia ).

Original: Thus, in addition to targeted downstream evaluations, it can be valuable to monitor and analyze performance on held-out pretraining data, which by deﬁnition covers all pretraining languages.
Revised: Thus, in addition to targeted downstream evaluations, it can be valuable to monitor and analyze performance on held-out pretraining data, which by deﬁnition covers all pretraining languages.

Original: Figure 2 shows pretraining loss curves of models trained with three sampling strategies, for both English ( en ) and Yoruba ( yo ), the highest and lowest-resource languages in our study.
Revised: Figure 2 shows pretraining loss curves of models trained with three sampling strategies, for both English ( en ) and Yoruba ( yo ), the highest and lowest-resource languages in our study.

Original: We plot three model sizes, as model scale controls the tradeoffs between positive cross-lingual transfer and overﬁtting.
Revised: We plot three model sizes, as model scale controls the tradeoffs between positive cross-lingual transfer and overﬁtting.

Original: In Figure 2a, we observe that the loss values for en are much lower than for yo , and model scale alone does little to close this gap.
Revised: In Figure 2a, we observe that the loss values for en are much lower than for yo , and model scale alone does little to close this gap.

Original: This suggests that downstream performance on low-resource languages will suffer if they are not upsampled.
Revised: This suggests that downstream performance on low-resource languages will suffer if they are not upsampled.

Original: On the other hand, too much upsampling is problematic, as shown in Figure 2b.
Revised: On the other hand, too much upsampling is problematic, as shown in Figure 2b.

Original: Crucially, overﬁtting only emerges with scale: Large shows no obvious overﬁtting, XL shows weak overﬁtting, and it becomes conspicuous at XXL size.
Revised: Crucially, overﬁtting only emerges with scale: Large shows no obvious overﬁtting, XL shows weak overﬁtting, and it becomes conspicuous at XXL size.

Original: Moreover, while the effects of repetition may appear to be limited to Yoruba, results from Hernandez et al.
Revised: Moreover, while the effects of repetition may appear to be limited to Yoruba, results from Hernandez et al.

Original: (2022) suggest that overﬁtting in one language is likely to hurt performance across the board; the authors ﬁnd that even repeating 0 .
Revised: (2022) suggest that overﬁtting in one language is likely to hurt performance across the board; the authors ﬁnd that even repeating 0 .

Original: 1% of data 100 times can be as harmful as halving model size.
Revised: 1% of data 100 times can be as harmful as halving model size.

Original: Figure 3 shows the pretraining loss curve for a longer train, corresponding to 1 / 2 character budget.
Revised: Figure 3 shows the pretraining loss curve for a longer train, corresponding to 1 / 2 character budget.

Original: In this case, overﬁtting emerges even for the Large model after around 300,000 steps.
Revised: In this case, overﬁtting emerges even for the Large model after around 300,000 steps.

Original: Given that previous work (Conneau et al., 2020; Xue et al., 2021) studied sampling distributions at smaller model sizes or for shorter training steps, the importance of overﬁtting may have been overlooked.
Revised: Given that previous work (Conneau et al., 2020; Xue et al., 2021) studied sampling distributions at smaller model sizes or for shorter training steps, the importance of overﬁtting may have been overlooked.

Original: Notably, across Figures 2c and 3c, U NI M AX closes the loss gap between high- and low-resource languages without showing any sign of overﬁtting.
Revised: Notably, across Figures 2c and 3c, U NI M AX closes the loss gap between high- and low-resource languages without showing any sign of overﬁtting.

Original: 5. D OWNSTREAM EVALUATIONS
Revised: 5. D OWNSTREAM EVALUATIONS

Original: Figure 4 shows the average TyDi QA performance for the three sampling strategies discussed in
Revised: Figure 4 shows the average TyDi QA performance for the three sampling strategies discussed in

Original: Section § 3.
Revised: Section § 3.

Original: Our ﬁrst observation is that U NI M AX outperforms the other two consistently across model scales.
Revised: Our ﬁrst observation is that U NI M AX outperforms the other two consistently across model scales.

Original: We can get additional insights by correlating per-language performance with thepretraining portion of each language, as shown in Figure 5. Generally, languages seen more during to unstable performance.
Revised: We can get additional insights by correlating per-language performance with the to unstable performance.

Original: The WMT21 Serbian training data covers Cyrillic and Latin scripts, and we observed high variance in eval metrics across checkpoints, depending on whether the model happened to use the same script as the references.
Revised: The WMT21 Serbian training data covers Cyrillic and Latin scripts, and we observed high variance in eval metrics across checkpoints, depending on whether the model happened to use the same script as the references.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: pretraining portion of each language, as shown in Figure 5.

Original:  pretraining tend to have higher performance, but this beneﬁt seems to have its limit.
Revised: Generally, languages seen more during pretraining tend to have higher performance, but this beneﬁt seems to have its limit.

Original: For example, τ = 1 sampling allocates 47.7% of training to en
Revised: For example, τ = 1 sampling allocates 47.7% of training to en

Original: but this only translates to 1% better performance than U NI M AX , which allots only 1% to en .
Revised: but this only translates to 1% better performance than U NI M AX , which allots only 1% to en .

Original: We believe this surprisingly small delta is attributable to the beneﬁt of positive transfer from non-English languages onto English.
Revised: We believe this surprisingly small delta is attributable to the beneﬁt of positive transfer from non-English languages onto English.

Original: Additionally, we note that
Revised: Additionally, we note that

Original: U NI M AX outperforms τ = 3 .
Revised: U NI M AX outperforms τ = 3 .

Original: 33 on Swahili, despite seeing fewer Swahili examples during training.
Revised: 33 on Swahili, despite seeing fewer Swahili examples during training.

Original: See Table 7 for full per-language metrics across sampling strategies and model scales.
Revised: See Table 7 for full per-language metrics across sampling strategies and model scales.

Original: Figure 6 (left) shows that U NI M AX also outperforms temperature sampling on WMT21, across all model sizes considered.
Revised: Figure 6 (left) shows that U NI M AX also outperforms temperature sampling on WMT21, across all model sizes considered.

Original: Figure 6 (right) shows that the beneﬁt is spread out across the majority of 
Revised: Figure 6 (right) shows that the beneﬁt is spread out across the majority of language pairs, as opposed to only beneﬁting a small subset. We also note that the beneﬁt is more

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: language pairs, as opposed to only beneﬁting a small subset.

Original: We also note that the beneﬁt is more pronounced for languages pairs where the target language is non-English, see Fig. 8 for details.
Revised: pronounced for languages pairs where the target language is non-English, see Fig. 8 for details.

Original: Appendix D provides evaluations on additional benchmarks, which show similar trends.
Revised: Appendix D provides evaluations on additional benchmarks, which show similar trends.

Original: 5. F URTHER ABLATIONS
Revised: 5. F URTHER ABLATIONS

Original: Our core experiments described in the previous sections were done across three model scales, but limited the character budget to 1 / 8 , due to compute resource constraints.
Revised: Our core experiments described in the previous sections were done across three model scales, but limited the character budget to 1 / 8 , due to compute resource constraints.

Original: In this section, we perform two additional ablations, but limiting to Large size.
Revised: In this section, we perform two additional ablations, but limiting to Large size.

Original: First, in Table 2 (left) we show the effect of increasing the budget by 4 × .
Revised: First, in Table 2 (left) we show the effect of increasing the budget by 4 × .

Original: Note, longer training changes the “shape” of the U NI M AX distribution (cf.
Revised: Note, longer training changes the “shape” of the U NI M AX distribution (cf.

Original: Figure 1b), as the increased budget means more languages will hit their max- N epoch cap.
Revised: Figure 1b), as the increased budget means more languages will hit their max- N epoch cap.

Original: Overall, we ﬁnd that U NI M AX still outperforms temperature sampling, with the longer train boosting performance across the board.
Revised: Overall, we ﬁnd that U NI M AX still outperforms temperature sampling, with the longer train boosting performance across the board.

Original: Second, we ablate the U NI M AX max-epoch parameter N .
Revised: Second, we ablate the U NI M AX max-epoch parameter N .

Original: So far, we have used N = 1 , i.e. no example is repeated.
Revised: So far, we have used N = 1 , i.e. no example is repeated.

Original: Table 2 (right) shows TyDi QA results for N ∈ { 1 , 5 , 10 } .
Revised: Table 2 (right) shows TyDi QA results for N ∈ { 1 , 5 , 10 } .

Original: We observe the best performance when disallowing repeats entirely, but the effect is small.
Revised: We observe the best performance when disallowing repeats entirely, but the effect is small.

Original: We also note that the optimal setting of N likely depends on the character budget, as for large enough budgets, U NI M AX -1 will see vanishingly little of the lowest-resource languages.
Revised: We also note that the optimal setting of N likely depends on the character budget, as for large enough budgets, U NI M AX -1 will see vanishingly little of the lowest-resource languages

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  F ULL -B UDGET M ODEL
Revised:  UM T5 M ODELS

Original: We put our above ﬁndings into practice by training a “full-budget” model over a trillion tokens.
Revised: We put our above ﬁndings into practice by training a suite of “umT5” models over a trillion tokens.

Original: For this ﬁnal model, we also update the training corpus and add analysis comparing to mT5, which is the most direct point of comparison.
Revised: For these ﬁnal models, we also update the training corpus and add analysis comparing to mT5, which is the most direct point of comparison.

Original: As large language models are increasingly used for knowledgeintensive tasks (Petroni et al., 2019; Roberts et al., 2020; Petroni et al., 2021), it is important that training corpora are up-to-date.
Revised: As large language models are increasingly used for knowledge-intensive tasks (Petroni et al., 2019; Roberts et al., 2020; Petroni et al., 2021), it is important that training corpora are up-to-date.

Original: Given that the mC4 corpus isnow over two years old, we construct an improved variant of the corpus covering crawled documents through August 2022.
Revised: Given that the mC4 corpus is over two years old, we update the corpus (update version 3.1.0) to cover crawled documents through August 2022.

Original: Beyond adding fresh documents, we make threekey changes to mC4.
Revised: Beyond adding fresh documents, we make three changes to mC4.

Original: First, as in Section § 4.1, we raise the language detection conﬁdence threshold from 0 .
Revised: First, as in Section § 4.1, we raise the language detection conﬁdence threshold from 0 .

Original: 7 to 0 .
Revised: 7 to 0 .

Original: 95 , which we found to increase accuracy and reduce documents with little or no natural language.
Revised: 95 , which we found to increase accuracy and reduce documents with little or no natural language.

Original: Second, we adjust the mC4 bad word ﬁlters to be soft ﬁlters, allowing a random 0 .
Revised: Second, we adjust the mC4 bad word ﬁlters to be soft ﬁlters, allowing a random 0 .

Original: 1% of documents with bad words to pass through.
Revised: 1% of documents with bad words to pass through.

Original: This ensures that models trained on our corpus will have at least a minimal exposure to any term, which is likely to help ondownstream tasks like toxicity detection.
Revised: This ensures that models trained on our corpus will have at least a minimal exposure to any term, which is likely to help on tasks like toxicity detection.

Original: Finally, we remove from mC4’s bad words lists any term that results in ﬁltering > 10% of documents from the language in question.
Revised: Finally, we remove from mC4’s bad words lists any term that results in ﬁltering > 10% of documents from the language in question.

Original: After applying these changes, our resulting training corpus consists of 28 .
Revised: Applying these changes, our resulting training corpus consists of 28 .

Original: 8 trillion characters from 9 . 0 billion documents—a 35% increase in documents over mC4.
Revised: 8 trillion characters from 9 . 0 billion documents—a 35% increase in documents over mC4.

Original: This increase is due primarily to added documents, and persists despite the more aggressive ﬁltering during language detection.
Revised: This increase is due primarily to added documents, and persists despite the more aggressive ﬁltering during language detection.

Original: See Table 8 for full corpus statistics.
Revised: See Table 8 for full corpus statistics.

Original: We closely follow the training setup and evaluation tasks from the mT5 paper for a fair comparison.
Revised: We closely follow the training setup and evaluation tasks from the mT5 paper for a fair comparison.

Original: Table 3 shows that our full budget U NI M AX model outperforms mT5 on  all tasks considered.
Revised: Table 3 shows that umT5 outperforms mT5 on most tasks, across all sizes, and particularly at the largest size.

Original:  See Appendix E for an additional ablation isolating the effect of the data refresh.
Revised: 3 See Appendix E for an additional ablation isolating the effect of the data refresh.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduced U NI M AX , a language sampling strategy that comes close to being uniform across languages—as close as possible without introducing harmful repetition.
Revised: In this paper, we introduced U NI M AX , a language sampling strategy that comes close to being uniform across languages—as close as possible without introducing harmful repetition.

Original: We showedthat this method performs well across several benchmarks andseveral model scales, up to 13 billion parameters.
Revised: We showed this method performs well across several benchmarks and model scales, up to 13 billion parameters.

Original: Given thedirect guarantees it provides against repetition, we expectthat U NI M AX should alsobe well-suited to even larger model sizes, where issues of overﬁtting and memorization are known to grow more severe.
Revised: Given the guarantees it provides against repetition, we expect U NI M AX is also well-suited to even larger model sizes, where issues of overﬁtting and memorization are known to grow more severe.

Original: Our study was limited to a single pretraining paradigm.
Revised: Our study focused on a single pretraining paradigm.

Original: In future work, it would be interesting to test whether U NI M AXstill confers gains in other settings, such as with: (i) encoder-only or decoderonly models, (ii) models trained with parallel data, and (iii) models with dedicated parameters per language (Pfeiffer et al., 2022).
Revised: Future work is needed to test if U NI M AX confers gains in other settings, such as: (i) encoder-only or decoder-only models, (ii) models trained with parallel data, and (iii) models with dedicated parameters per language (Pfeiffer et al., 2022).

Original: Beyond evaluation metrics, there is an a priori reason to prefer more uniform language distributions: they can be seen as more equitable, in that they come closer to treating each language as an equally priority.
Revised: Beyond evaluation metrics, there is an a priori reason to prefer more uniform language distributions: they can be seen as more equitable, in that they come closer to treating each language as an equally priority.

Original: However there is an important remaining question about whether “language” is the right unit over which to normalize.
Revised: However there is an important remaining question about whether “language” is the right unit over which to normalize.

Original: One alternative that we think deserves further exploration is inspired by the notion of “demographic utility” discussed by Blasi et al.
Revised: One alternative that we think deserves further exploration is inspired by the notion of “demographic utility” discussed by Blasi et al.

Original: (2022), which treats each speaker as equal in balancing across languages.
Revised: (2022), which treats each speaker as equal in balancing across languages.

Original: See Appendix B for further analysis.
Revised: See Appendix B for further analysis.

Original: We are eager to see if future research can formulate successful sampling strategies that take into account both linguistic and demographic utility.
Revised: We are eager to see if future research can formulate successful sampling strategies that take into account both linguistic and demographic utility.

Original: Ideally the language distributions output by these strategies can lead to stronger-performing and more equitable pretrained models that better serve a wider range of users and use cases.
Revised: Ideally the language distributions output by these strategies can lead to stronger-performing and more equitable pretrained models that better serve a wider range of users and use cases.

Original: AV OCABULARY A NALYSIS
Revised: A NALYSIS

Original: We analyze the makeup of the sub-word vocabularies resulting from different language sampling strategies in Table 4.
Revised: We analyze the makeup of the sub-word vocabularies resulting from different language sampling strategies in Table 4.

Original: To see how capacity is allocated across languages, we measure the proportion of tokens using scripts associated with speciﬁc languages or language groups.
Revised: To see how capacity is allocated across languages, we measure the proportion of tokens using scripts associated with speciﬁc languages or language groups.

Original: We observe that increasing τ from 1 to 3 .
Revised: We observe that increasing τ from 1 to 3 .

Original: 33 reduces the amount of Latin-script and Cyrillic tokens, presumably due to heavy reduction of English and Russian, the two most prevalent languages in mC4.
Revised: 33 reduces the amount of Latin-script and Cyrillic tokens, presumably due to heavy reduction of English and Russian, the two most prevalent languages in mC4.

Original: However, as expected, raising temperature increases allocation to lower-ranking languages (Arabic, Chinese,
Revised: However, as expected, raising temperature increases allocation to lower-ranking languages (Arabic, Chinese,

Original: Greek, Hindi, Japanese, Korean, Thai).
Revised: Greek, Hindi, Japanese, Korean, Thai).

Original: Overall, the U NI M AX vocabularies are fairly similar to τ = 3 .
Revised: Overall, the U NI M AX vocabularies are fairly similar to τ = 3 .

Original: 33 , with the expected trend that increasing pretraining budget ( 1 × → 4 × ) shifts more  allocation to high-resource languages, while increasing the max-epoch threshold (UM1 → UM10) shifts more allocation to low-resource languages.
Revised: 33 , with the expected trend that increasing pretraining budget ( 1 × → 4 × ) shifts more [Table] allocation to high-resource languages, while increasing the max-epoch threshold (UM1 → UM10) shifts more allocation to low-resource languages.

Original: Looking at token lengths, we observe three factors contribute to shorter tokens: higher temperature, smaller U NI M AX budget, and higher U NI M AX max-epoch threshold.
Revised: Looking at token lengths, we observe three factors contribute to shorter tokens: higher temperature, smaller U NI M AX budget, and higher U NI M AX max-epoch threshold.

Original: This is expected, as if a[Table] few high-resource languages dominate the vocabulary training corpus, the sub-word optimizer will assign more capacity to rare words from these languages, as opposed to covering more frequent (and shorter) words from a wider range of languages.Comparing to mT
Revised: This is expected, as if a few high-resource languages dominate the vocabulary training corpus, the sub-word optimizer will assign more capacity to rare words from these languages, as opposed to covering more frequent (and shorter) words from a wider range of languages.

Original: [Table] we note that our vocabularies have longer tokens overall [Table] and include more tokens containing the SentencePiece meta-symbol indicating whitespace ( ).
Revised: Comparing to mT5, we note that our vocabularies have longer tokens overall, and include more tokens containing the SentencePiece meta-symbol indicating whitespace ( ).

Original: We suspect these differences stem from mT5 using a whitespace-splitting “pre-tokenizer” to derive preliminary word counts which are fed to the sub-word training algorithm.
Revised: We suspect these differences stem from mT5 using a whitespace-splitting “pre-tokenizer” to derive preliminary word counts which are fed to the sub-word training algorithm.

Original: By comparison, our vocabulary is trained on raw text, with no need for a pre-tokenizer.
Revised: By comparison, our vocabulary is trained on raw text, with no need for a pre-tokenizer.

Original: We also note that mT5 contains many more tokens including punctuation.
Revised: We also note that mT5 contains many more tokens including punctuation.

Original: This is most likely due to the presence of low-quality non-language documents that are ﬁltered by our higher language detection threshold, as described in section § 4. [Table]
Revised: This is most likely due to the presence of low-quality non-language documents that are ﬁltered by our higher language detection threshold, as described in section § 4.1.

Original: B R EPRESENTATION R ATIOS
Revised: B R EPRESENTATION R ATIOS

Original: To shed light on how moving towards a more population-based sampling strategy would reshape language distributions, we deﬁne the “representation ratio” R ( l, t ) of a language l within a training distribution t to be the ratio between the language’s rate of use within t and its rate of native speakers ( s ) among the world population ( w ), as shown in equation (3).
Revised: To shed light on how moving towards a more population-based sampling strategy would reshape language distributions, we deﬁne the “representation ratio” R ( l, t ) of a language l within a training distribution t to be the ratio between the language’s rate of use within t and its rate of native speakers ( s ) among the world population ( w ), as shown in equation (3).

Original: We say a language is “overrepresented” ( R > 1 ) or “underrepresented” ( R < 1 ) to indicate that its prevalence in training is more or less than expected based on its global L1 speaker population.
Revised: We say a language is “overrepresented” ( R > 1 ) or “underrepresented” ( R < 1 ) to indicate that its prevalence in training is more or less than expected based on its global L1 speaker population.

Original: [Equation]
Revised: [Equation]

Original: Figure 7 plots representation ratios for the mT5 language distribution, with native speaker counts taken from Wikipedia.
Revised: Figure 7 plots representation ratios for the mT5 language distribution, with native speaker counts taken from Wikipedia.

Original: 3 We observe a wide range of representation, ranging from 7 × underrepresented to over 900 × overrepresented.
Revised: 4 We observe a wide range of representation, ranging from 7 × underrep- [Figure] resented to over 900 × overrepresented.

Original: We also note a clear pattern whereby languages of Europe tend to have higher representation (Mann–Whitney U =566 , p< 1e − 5 , two-tailed), echoing a general bias observed by Blasi et al.
Revised: We also note a clear pattern whereby languages of Europe tend to have higher representation (Mann–Whitney U =566 , p< 1e − 5 , two-tailed), echoing a general bias observed by Blasi et al.

Original: for NLP resources.
Revised: for NLP resources.

Original: [Figure] s [Figure] Overall, we ﬁnd that languages with large speaker populations, and particularly non-European languages are heavily underrepresented in commonly used pretraining distributions.
Revised: Overall, we ﬁnd that languages with large speaker populations, and particularly non-European languages are heavily underrepresented in commonly used pretraining distributions.

Original: For example, the top 5 underrepresented languages in mT5 training are all languages of Asia and Africa withmillion or more native speakers: Chinese, Punjabi, Yoruba, Bengali and Hindi.
Revised: For example, the top 5 underrepresented languages in mT5 training are all languages of Asia and Africa withmillion or more native speakers: Chinese, Punjabi, Yoruba, Bengali and Hindi.

Original: C A DDITIONAL TRAINING DETAILS
Revised: C A DDITIONAL TRAINING DETAILS

Original: The model architectures used in this study are the same as mT5 models, except that relative position embeddings are not shared across layers.
Revised: The model architectures used in this study are the same as mT5 models, except that relative position embeddings are not shared across layers.

Original: In all of our models, the vocabulary size is 256, subwords [Figure] and byte-level fallback is enabled, so unknown tokens are broken down into UTF-8 bytes.
Revised: In all of our models, the vocabulary size is 256, subwords, and byte-level fallback is enabled, so unknown tokens are broken down into UTF-8 bytes.

Original: We use the T5X library (Roberts et al., 2022) to train the models using Google Cloud TPUs.
Revised: We use the T5X library (Roberts et al., 2022) to train the models using Google Cloud TPUs.

Original: For pretraining, we use Adafactor optimizer (Shazeer & Stern, 2018) with a constant learning rate of 0 .
Revised: For pretraining, we use Adafactor optimizer (Shazeer & Stern, 2018) with a constant learning rate of 0 .

Original: 01 in the ﬁrst 10 , 000 steps and inverse square root decay afterwards.
Revised: 01 in the ﬁrst 10 , 000 steps and inverse square root decay afterwards.

Original: For ﬁnetuning, we use
Revised: For ﬁnetuning, we use

Original: Adafactor with a constant learning rate of 5e − 5 .
Revised: Adafactor with a constant learning rate of 5e − 5 .

Original: Unlike mT5, we do not use loss normalization factor.
Revised: Unlike mT5, we do not use loss normalization factor.

Original: Instead we use the number of real target tokens as the effective loss normalization.
Revised: Instead we use the number of real target tokens as the effective loss normalization.

Original: Finally, we do not factorize the second moment of the Adafactor states and we also use momentum, neither of which are used in T5 and mT5 studies.
Revised: Finally, we do not factorize the second moment of the Adafactor states and we also use momentum, neither of which are used in T5 and mT5 studies.

Original: D A DDITIONAL BENCHMARKS
Revised: D A DDITIONAL BENCHMARKS

Original: As a further comparison, we ﬁne-tune and evaluate the nine pretrained models from Sections 4 and 5 on three additional benchmarks: XNLI zero-shot, XNLI translate-train, and XQuAD.
Revised: As a further comparison, we ﬁne-tune and evaluate the nine pretrained models from Sections 4 and 5 on three additional benchmarks: XNLI zero-shot, XNLI translate-train, and XQuAD.

Original: We observe
Revised: We observe

Original: U NI M AX performs the best overall, although τ = 3 .
Revised: U NI M AX performs the best overall, although τ = 3 .

Original: 33 is a close second.
Revised: 33 is a close second.

Original: E A BLATION ON M C4 REFRESH
Revised: E A BLATION ON M C4 REFRESH

Original: To isolate the effect of refreshing the mC4 data, we train two additional models to 100 , 000 steps using U NI M AX sampling, under the same “full-budget” setting as in Section § 6.
Revised: To isolate the effect of refreshing the mC4 data, we train two additional models to 100 , 000 steps using U NI M AX sampling, under the same “full-budget” setting as in Section § 6.

Original: These models differ only in that one trains on the original mC4 data, while the other uses our refreshed mC4 corpus.
Revised: These models differ only in that one trains on the original mC4 data, while the other uses our refreshed mC4 corpus.

Original: Table 6 shows results ﬁne-tuning these two models on the XNLI task.
Revised: Table 6 shows results ﬁne-tuning these two models on the XNLI task.

Original: We observe that the data refresh alone only gives a small boost ( +0 . 1 ), supporting the view that the gains of U NI M AX over mT5 ( +1 . 0 ) in Table 3 are primarily due to improved language sampling.
Revised: We observe that the data refresh alone only gives a small boost ( +0 . 1 ), supporting the view that the gains of U NI M AX over mT5 ( +1 . 0 ) in Table 3 are primarily due to improved language sampling.

Original: While the inclusion of more recently crawled documents does not help on this particular benchmark (which is also several years old), we expect that the refreshed data will be useful to practitioners, and should help on tasks requiring up-to-date knowledge.
Revised: While the inclusion of more recently crawled documents does not help on this particular benchmark (which is also several years old), we expect that the refreshed data will be useful to practitioners, and should help on tasks requiring up-to-date knowledge.

Original: F A DDITIONAL TABLES

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: F A DDITIONAL TABLES

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: 1.48 ﬁ 1.

Original: [Table] [Table] [Table] [Table]
Revised: 1.48 no 1.

Original: [Table] [Table] [Table] [Table]
Revised: 1.48 bg 1.

Original: [Table] [Table] [Table] [Table]
Revised: 1.48 th 1.

Original: [Figure] [Table] 1.48 ﬁ 1. 1.48 no 1.
Revised: 1.48 sk 1.

Original: 1.48bg 1. 1.48 th 1. 1.48 sk 1. 1.48 hi 1.
Revised: 1.48 hi 1.

Original: 1.48 ko 1.
Revised: 1.48 ko 1.

Original: 1.48 lt 1.
Revised: 1.48 lt 1.

Original: 1.48 iw 1.
Revised: 1.48 iw 1.

Original: 1.48 ca 1.
Revised: 1.48 ca 1.

Original: [Table]

Original: [Table]

Original: 48 id 1.
Revised: 1.48 id 1.

Original: 1.48 sl 1.
Revised: 1.48 sl 1.

Original: 1.48 et 1.
Revised: 1.48 et 1.

Original: 1.48 lv 0.
Revised: 1.48 lv 0.

Original: 1.48 bn 0.
Revised: 1.48 bn 0.

Original: 1.48 sq 0.
Revised: 1.48 sq 0.

Original: 1.48 az 0.
Revised: 1.48 az 0.

Original: 1.48 sr 0.
Revised: 1.48 sr 0.

Original: 1.48 ta 0.
Revised: 1.48 ta 0.

Original: 1.48 ms 0.
Revised: 1.48 ms 0.

Original: 1.48 is 0.
Revised: 1.48 is 0.

Original: 1.48 kk 0.
Revised: 1.48 kk 0.

Original: 1.48 mr 0.
Revised: 1.48 mr 0.

Original: 1.48 ne 0.
Revised: 1.48 ne 0.

Original: 1.48 ur 0. 1.48 ka 0.
Revised: 1.48 ur 0. 1.48 ka 0.

Original: [Table]
Revised: [Table]

Revised: si

Revised: [Table]

Revised: [Table]

Revised: 52 ru

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: pa
Revised: pa

Original: [Table]
Revised: [Table]

Revised: ga

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: ps
Revised: ps

Revised: [Table]

Revised: [Table]

Revised: 25 ku

Original: [Table]
Revised: [Table]

Original: ku
Revised: ha

Original: [Table]
Revised: [Table]

Revised: 19 ceb

Revised: [Table]

Revised: fy

Revised: [Table]

Original: mg
Revised: mg

Original: [Table]
Revised: [Table]

Original: am
Revised: am

Original: [Table]
Revised: [Table]

Original: ht
Revised: sd

Original: [Table]
Revised: [Table]

Original: yi
Revised: 15 gd [Table] [Table] 14 ht

Original: [Table]
Revised: [Table]

Original: lo
Revised: [Table] [Table] 14 yi 0. [Table]

Original: [Table]
Revised: [Table]

Original: hi-
Revised: 13 lo

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: zu
Revised: [Table] [Table] [Table] hi-Latn 0.

Original: [Table]
Revised: [Table]

Original: jv
Revised: [Table] [Table] 12 zu

Original: [Table] [Table]
Revised: [Table] [Table]

Original: .12 hmn
Revised: 12 jv

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: yo
Revised: 0.12 hmn 0.

Original: [Table]
Revised: [Table]