Original: Adapting BigScience Multilingual Model to Unseen Languages
Revised: Adapting BigScience Multilingual Model to Unseen Languages

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: We benchmark different strategies of addingnew languages (German and Korean) into the
Revised: We benchmark different strategies of adding new languages (German and Korean) into the

Original: BigScience’s pretrained multilingual languagemodel with 1.3 billion parameters that currentlysupports 13 languages.
Revised: BigScience’s pretrained multilingual language model with 1.3 billion parameters that currently supports 13 languages.

Original: We investigate the factors that affect the language adaptability ofthe model and the trade-offs between computational costs and expected performance.
Revised: We investigate the factors that affect the language adaptability of the model and the trade-offs between computational costs and expected performance.

Original:  Introduction
Revised:  Introduction

Original: Pretrained multilingual language models (LMs)have enabled cross-lingual transfer (Artetxe et al. ,2020; Conneau et al., 2020b; K et al., 2021).
Revised: Pretrained multilingual language models (LMs) have enabled cross-lingual transfer (Artetxe et al., 2020; Conneau et al., 2020b; K et al., 2021).

Original: Whileseveral works suggest that such knowledge transfergoes beyond just vocabulary sharing across languages (Artetxe et al., 2020; K et al., 2021), othershave shown that the models’ performance is sensitive to the quality of the tokenization it relieson (Pfeiffer et al., 2021b).
Revised: While several works suggest that such knowledge transfer goes beyond just vocabulary sharing across languages (Artetxe et al., 2020; K et al., 2021), others have shown that the models’ performance is sensitive to the quality of the tokenization it relies on (Pfeiffer et al., 2021b).

Original: Moreover, the qualityof knowledge transfer can degrade for unseen languages, especially when the scripts are unknown tothe model (Muller et al., 2021).
Revised: Moreover, the quality of knowledge transfer can degrade for unseen languages, especially when the scripts are unknown to the model (Muller et al., 2021).

Original: Data availability, but also the curse of multilinguality (Conneau et al., 2020a) makes training asingle model covering all the languages challenging: there is a trade off between the number oflanguages in pretraining data, model capacity andthe downstream performance for each individuallanguage.
Revised: Data availability, but also the curse of multilinguality (Conneau et al., 2020a) makes training a single model covering all the languages challenging: there is a trade off between the number of languages in pretraining data, model capacity and the downstream performance for each individual language.

Original: Finally, it is hard to anticipate any potential usage of pretrained LM in advance, henceit is important to study a posteoriori adaptationto new tasks/languages.
Revised: Finally, it is hard to anticipate any potential usage of pretrained LM in advance, hence it is important to study a posteoriori adaptation to new tasks/languages.

Original: Recently proposed methods include continual pretraining of the model(restricted to the embedding layer training only insome cases)
Revised: Recently proposed methods include continual pretraining of the model (restricted to the embedding layer training only in some cases)

Original: (Artetxe et al., 2020; Chau et al., 2020;
Revised: (Artetxe et al., 2020; Chau et al., 2020;

Original: Muller et al., 2021; Zhang et al., 2020;Wang et al.
Revised: Muller et al., 2021; Zhang et al., 2020;

Original: ,2020), or training of language-specific adapters(Pfeiffer et al., 2020, 2021a,b; Philip et al., 2020;Üstün et al., 2021; Berard, 2021) for the target language.
Revised: Wang et al., 2020), or training of language-specific adapters (Pfeiffer et al., 2020, 2021a,b; Philip et al., 2020; Üstün et al., 2021; Berard, 2021) for the target language.

Original: The core motivation behind these methodsis to benefit from knowledge transfer encoded in the pretrained LM for the new language processing at a small computational cost (compared to full model retraining) .
Revised: The core motivation behind these methods is to benefit from knowledge transfer encoded in the pretrained LM for the new language processing at a small computational cost (compared to full model retraining) .

Original: In this work, we aim at better understanding the trade-offs between the amount of compute and the final downstream task performance.
Revised: In this work, we aim at better understanding the trade-offs between the amount of compute and the final downstream task performance.

Original: Specifically, we study the impact of the following three factors: original pretraining steps, adaptation strategies, and adapters’ capacity on the Natural Language Inference (NLI) task.
Revised: Specifically, we study the impact of the following three factors: original pretraining steps, adaptation strategies, and adapters’ capacity on the Natural Language Inference (NLI) task.

Original: As part of the initiative of BigScience, we experiment with its multilingual LM, which currently only supports languages.
Revised: As part of the initiative of BigScience, we experiment with its multilingual LM, which currently only supportslanguages.

Original: We investigate how researchers could adapt BigScience’s full-open-access multilingual models to their languages of interest.
Revised: We investigate how researchers could adapt BigScience’s full-open-access multilingual models to their languages of interest.

Original: Moreover,
Revised: Moreover,

Original: BigScience open source its intermediate pretraining checkpoints, therefore we also study how does the adaptability to new language changes with the amount of pretraining.
Revised: BigScience open source its intermediate pretraining checkpoints, therefore we also study how does the adaptability to new language changes with the amount of pretraining.

Original: Our main findings are: (1) the most promising strategy is training of the embedding layers and adapters simultaneously; (2) position encoding is an important component to adapt; (3) pretraining steps benefit zero-shot performance; (4) adaptation to low-resource languages can be done with low parameter budget.
Revised: Our main findings are: (1) the most promising strategy is training of the embedding layers and adapters simultaneously; (2) position encoding is an important component to adapt; (3) pretraining steps benefit zero-shot performance; (4) adaptation to low-resource languages can be done with low parameter budget.

Original:  Background and Related Work
Revised:  Background and Related Work

Original: Most of the works investigating extension of pretrained models to new languages consider models such as  (Devlin et al., 2019) and XLM-R (Conneau et al., 2020a) that were pretrained to 100+ languages with Masked Language Modeling (MLM) objective.
Revised: Most of the works investigating extension of pretrained models to new languages consider models such as BERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020a) that were pretrained to 100+ languages with Masked Language Modeling (MLM) objective.

Original: Artetxe et al.
Revised: Artetxe et al.

Original: Pfeiffer et al.
Revised: Pfeiffer et al.

Original: (2021b) demonstrate that it is possible to add new language to these models by training new embedding layer.
Revised: (2021b) demonstrate that it is possible to add new language to these models by training new embedding layer.

Original: Muller et al.
Revised: Muller et al.

Original: (2021) continues training the pretrained mBERT on the new languagedata, and finds that transliteration of languages using non-latin script boosts performance on these
Revised: (2021) continues training the pretrained mBERT on the new language

Original: Decoder Block Decoder Block
Revised: Decoder Block Decoder Block

Original: Embeddings
Revised: Embeddings

Original: Embeddings
Revised: Embeddings

Original: Invertible Adapter
Revised: Invertible Adapter

Original: Language Adapter Language Adapter
Revised: Language Adapter Language Adapter

Original: Invertible Adapter ...
Revised: Invertible Adapter ...

Original: TDecoder Block
Revised: TDecoder Block

Original: Decoder Block
Revised: Decoder Block

Original: Embeddings
Revised: Embeddings

Original: Embeddings ...
Revised: Embeddings ...

Original: T
Revised: T

Original: Decoder Block Decoder Block
Revised: Decoder Block Decoder Block

Original: Embeddings
Revised: Embeddings

Original: Embeddings Invertible Adapter
Revised: Embeddings Invertible Adapter

Original: Invertible Adapter
Revised: Invertible Adapter

Original: Language Adapter
Revised: Language Adapter

Original: Language Adapter ...
Revised: Language Adapter ...

Original: TDecoder Block
Revised: TDecoder Block

Original: Decoder Block
Revised: Decoder Block

Original: Embeddings
Revised: Embeddings

Original: Embeddings ... frozen frozen
Revised: Embeddings ... frozen frozen

Original: [Figure]
Revised: [Figure]

Original: T  languages.
Revised: T data, and finds that transliteration of languages using non-latin script boosts performance on these languages.

Original: Berard (2021) adds new languages into pretrained multilingual NMT model by training embedding layer and adapter layers.
Revised: Berard (2021) adds new languages into pretrained multilingual NMT model by training embedding layer and adapter layers.

Original: It shows thathigher adaptation cost is required for new targetlanguages (as opposed to new source languages).
Revised: It shows that higher adaptation cost is required for new target languages (as opposed to new source languages).

Original: Closest to our work (Ebrahimi and Kann, 2021)studies different approaches (i.e., continued pretraining, vocabulary expansion and adapter layers) allowing to extend XLM-R model to 1600languages.
Revised: Closest to our work (Ebrahimi and Kann, 2021) studies different approaches (i.e., continued pretraining, vocabulary expansion and adapter layers) allowing to extend XLM-R model tolanguages.

Original: They conclude that continued pretraining is the most promising direction.
Revised: They conclude that continued pretraining is the most promising direction.

Original: However, thecost of such pretraining will grow with the size ofthe pretrained model and can be quite prohibitivefor many researchers working with low-resourcelanguages.
Revised: However, the cost of such pretraining will grow with the size of the pretrained model and can be quite prohibitive for many researchers working with low-resource languages.

Original: Our work aims to compare lightweight strategiesof adaptation to new language (embedding layertraining and adapter layers) across the followingprevious unstudied dimensions: we consider the(1) adaptation parameter budget and (2) intermediate pretraining checkpoints by analyzing how theyimpact autoregressive model’s adaptability to newlanguages.
Revised: Our work aims to compare lightweight strategies of adaptation to new language (embedding layer training and adapter layers) across the following previously unstudied dimensions: we consider the (1) adaptation parameter budget and (2) intermediate pretraining checkpoints by analyzing how they impact autoregressive model’s adaptability to new languages.

Original:  Experimental settings 
Revised:  Experimental settings 3. BigScience Pretrained Multilingual Language Model

Original: 3. BigScience Pretrained Multilingual Language Model

Original: BigScience open-sources one of its pretrained multilingual autoregressive LMs 1 (Anonymous, 2022)that relies on transformer architecture with 24 decoder layers,2048 hidden units per layer, 16 at- tention heads and 1.3B parameters.
Revised: BigScience open-sources one of its pretrained multilingual autoregressive LMs 1 (Hesslow et al., 2022) that relies on transformer architecture with 24 decoder layers, 16 attention heads, embedding dimension of 2048, feed forward layer with dimensionality of 8192, and 1.3B parameters.

Original: This model is trained with Next Work Prediction objective ( aka GPT-style).
Revised: This model is trained with Next Word Prediction objective ( aka GPT-style).

Original: It is pretrained on OSCAR deduplicated subcorpora (Ortiz Suárez et al., 2019) for the following thirteen languages for a total of training 400B tokens: Indonesian, Basque, Vietnamese, Chinese, Urdu, Spanish, Catalan, Portuguese, French, English, Hindi, Arabic, and Bengali.
Revised: It is pretrained on OSCAR deduplicated subcorpora (Ortiz Suárez et al., 2019) for the following thirteen languages for a total of training 400B tokens: Indonesian, Basque, Vietnamese, Chinese, Urdu, Spanish, Catalan, Portuguese, French, English, Hindi, Arabic, and Bengali.

Original: See Appendix A for further model pretraining details.
Revised: See Appendix A for further model pretraining details.

Original: In our experiments, we use the model checkpoint saved after 12,000, 100,500, and 118,500 pretraining steps (final checkpoint).
Revised: In our experiments, we use the model checkpoint saved after 12,000, 100,500, and 118,500 pretraining steps (final checkpoint).

Original: 3. New Languages and Data
Revised: 3. New Languages and Data

Original: The need for adaptation of pretrained multilingual
Revised: The need for adaptation of pretrained multilingual

Original: LM to new languages typically raises for lowresource languages for three reasons: first, the lack of the available data for such languages; second, the lack of the computational resourceswith which people working on low-resource languages may experience; lastly, the lack of the evaluation resources, which makes it hard to assess the quality of the adapted model properly.
Revised: LM to new languages typically raises for lowresource languages for three reasons: first, the lack of the available data for such languages; second, the lack of the computational resources which people working on low-resource languages may experience; lastly, the lack of the evaluation resources, which makes it hard to assess the quality of the adapted model properly.

Original: In this study we simulate low-resource settings by restricting the amount of training samples for high-resource languages.
Revised: In this study we simulate low-resource settings by restricting the amount of training samples for high-resource languages.

Original: This allows us to benefit from good-quality evaluation datasets available for the high resource languages.
Revised: This allows us to benefit from good-quality evaluation datasets available for the high resource languages.

Original: We expect that our observations could generalize to real low-resource scenarios.
Revised: We expect that our observations could generalize to real low-resource scenarios.

Original: We’ve chosen two languages for the adaptation of BigScience model: (1) German (easy language) that belongs in thesame Germanic language family as
Revised: We’ve chosen two languages for the adaptation of BigScience model: (1) German (easy language) that belongs in the

Original: English and shares the Latin script with several pretraining languages; (2) Korean (difficult language)

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  that belongs to Koreanic language family and uses Hangul script system, that none of the languagesof pretrained LM cover.
Revised: same Germanic language family as English and shares the Latin script with several pretraining languages; (2) Korean (difficult language) that belongs to Koreanic language family and uses Hangul script system, that none of the languages of pretrained LM cover.

Original: We train the pretrained model’s byte-level BPEtokenizer on German and Korean deduplicated OSCAR subcorpora (Ortiz Suárez et al., 2019) anduse 100,000 samples from OSCAR subcorpora ofthe respective languages for adaptation.
Revised: We train the pretrained model’s byte-level BPE tokenizer with the vocabulary size of 130,000 on German and Korean deduplicated OSCAR subcorpora (Ortiz Suárez et al., 2019), which has 21.5B and 1.1B words respectively. Then, we use 100, samples from OSCAR subcorpora of the respective languages for adaptation.

Original: 3. Language Adaptation Strategies
Revised: 3. Language Adaptation Strategies

Original: We experiment with the following three languageadaptation strategies that involve finetuning of theembedding layer and adapters layers (see Figure 1).
Revised: We experiment with the following three language adaptation strategies that involve finetuning of the embedding layer and adapters layers (see Figure 1).

Original: See Appendix B for further training details.
Revised: See Appendix B for further training details.

Original: Embedding-only.
Revised: Embedding-only.

Original: The embedding layer consistsof token embeddings and positional embeddings.
Revised: The embedding layer consists of token embeddings and positional embeddings.

Original: We follow Artetxe et al.
Revised: We follow Artetxe et al.

Original: (2020) by learning new token embeddings and positional embeddings whilefreezing the rest of the transformer parameters.
Revised: (2020) by learning new token embeddings and positional embeddings while freezing the rest of the transformer parameters.

Original: Embedding-then-Adapters.
Revised: Embedding-then-Adapters.

Original: Following the
Revised: Following the

Original: Embedding-only approach, we first train theembedding layer on the target language pretrainingdata for 25,000 steps.
Revised: Embedding-only approach, we first train the embedding layer on the target language pretraining data for 25,000 steps.

Original: Then, we freeze theentire transformer and add MAD-X invertibleand language adapters (Pfeiffer et al., 2020).
Revised: Then, we freeze the entire transformer and add MAD-X invertible and language adapters (Pfeiffer et al., 2020).

Original: We subsequently finetune the adapters for an additional 25,000 steps.
Revised: We subsequently finetune the adapters for an additional 25,000 steps.

Original: Embedding-and-Adapters.
Revised: Embedding-and-Adapters.

Original: Instead of training the embedding layer and the adapters separately, we train both of them at the same time for 50, steps.
Revised: Instead of training the embedding layer and the adapters separately, we train both of them at the same time for 50, steps.

Original: We also vary the adapter capacity, which is defined by the reduction factor (also known as compression rate (Rücklé et al., 2021)) in the adapter’s bottleneck layer.
Revised: We also vary the adapter capacity, which is defined by the reduction factor (also known as compression rate (Rücklé et al., 2021)) in the adapter’s bottleneck layer.

Original: A smaller reduction value would lead to a larger amount of adapter parameters and, accordingly, a larger adaptation capacity.
Revised: A smaller reduction value would lead to a larger amount of adapter parameters and, accordingly, a larger adaptation capacity.

Original: We consider the reduction factors of 16 (default), 48, and 384.
Revised: We consider the reduction factors of 16 (default), 48, and 384.

Original: 3. Evaluation
Revised: 3. Evaluation

Original: We evaluate our models on the natural language inference (NLI) task, using XNLI dataset (Conneau et al., 2018) for German and KLUE-NLI dataset (Park et al., 2021) for Korean. [-
-]
Revised: We evaluate our models on the natural language inference (NLI) task, using XNLI dataset (Conneau et al., 2018) for German and KLUE-NLI dataset (Park et al., 2021) for Korean. [-
-]2 Table 2 reports the statistics of the NLI datasets.

Original: We adopt the three following  settings with increasing levels of task supervision.
Revised: We adopt the three following evaluation settings with increasing levels of task supervision.

Original: Prompt-based Zero-shot
Revised: Prompt-based Zero-shot

Original: Following Lin et al.(2021), we use the German and Korean cloze-styleprompt templates (see Table 1).
Revised: Following Lin et al. (2021), we use the German and Korean cloze-style prompt templates (see Table 1).

Original: The zero-shot prediction is the candidate label verbalizer, which replaces the [MASK] token, that maximizes the LMbased likelihood of the prompt sentence.
Revised: The zero-shot prediction is the candidate label verbalizer, which replaces the [MASK] token, that maximizes the LM based likelihood of the prompt sentence.

Original: Cross-lingual
Revised: Cross-lingual

Original: In this evaluation setting, we follow Pfeiffer et al.
Revised: In this evaluation setting, we follow Pfeiffer et al.

Original: and train task adapters on
Revised: and train task adapters on

Original: English task dataset.
Revised: English task dataset.

Original: These adapters are then usedon the target language (German or Korean) evaluation set.
Revised: These adapters are then used on the target language (German or Korean) evaluation set.

Original: We adopt adapter size (reduction factor= 16) and run training for 2 epochs with the batchsize of 32, learning rate of 5 e − 5, and maximum sequence length of 128 input tokens on a single [-V
-]
Revised: We adopt adapter size (reduction factor = 16) and run training for 2 epochs with the batch size of 32, learning rate of 5 e − 5, and maximum sequence length of 128 input tokens on a single [-V
-]V

Original: GPU, which takes around 5 hours to complete.
Revised: GPU, which takes around 5 hours to complete and produces around 0.65 kg of CO 2 . [Table]

Original: Supervised Finetuning
Revised: Supervised Finetuning

Original: Our finetuning strategiesare identical to the cross-lingual evaluation settingabove, except that the classification head and thetask adapters are finetuned on the target languagedataset.
Revised: Our finetuning strategies are identical to the cross-lingual evaluation setting above, except that the classification head and the task adapters are finetuned on the target language dataset.

Original: Note that this implies that we train differenttask adapter for each model.
Revised: Note that this implies that we train different task adapter for each model.

Original:  Results and Discussion
Revised:  Results and Discussion

Original: Table 2 reports the results on different adaptationstrategies.
Revised: Table 3 reports the results on different adaptation strategies.

Original: Baselines.
Revised: Baselines.

Original: XLM-R and mBERT are pretrainedmultilingual LMs that were trained on 100+ languages and include both German and Korean languages.
Revised: XLM-R and mBERT are pretrained multilingual LMs that were trained on 100+ languages and include both German and Korean languages.

Original: Therefore the adaptation of these modelscan be seen as an upper bound performance onecould achieve on classification tasks.
Revised: Therefore the adaptation of these models can be seen as an upper bound performance one could achieve on classification tasks.

Original: XGLM (Linet al., 2021) is a multilingual autoregressive LMthat was pretrained on 30 languages, including both
Revised: XGLM (Lin et al., 2021) is a multilingual autoregressive LM that was pretrained on 30 languages, including both

Original: German and Korean.
Revised: German and Korean.

Original: It serves as the (upper-bound)baseline for the prompt-based evaluation settingdue to its zero-shot learning capability throughprompts on NLI (Lin et al., 2021).
Revised: It serves as the (upper-bound) baseline for the prompt-based evaluation setting due to its zero-shot learning capability through prompts on NLI (Lin et al., 2021).

Original: Language Adaptation Strategies.
Revised: Language Adaptation Strategies.

Original: First, original BigScience multilingual model (4) performspoorly on German.
Revised: First, original BigScience multilingual model (4) performs poorly on German.

Original: 3  Model (6) that trains embedding layers and adapters separately seems to perform worse than models training them simultaneously (models (7) and (12) ).Training adapter layers(7) boosts performance compared to the model updating embedding layers only (5) , especially underthe cross-lingual setting for both German and Ko- rean languages
Revised: 3 Training adapter layers (7) , (6) boosts performance compared to the model updating embedding layers only (5) , especially under the cross-lingual setting for both German and Korean languages. Model (6) that trains embedding layers and adapters separately seems to perform worse than models training them simultaneously (models (7) and (13) ).

Original: We also observe that it is easier to adapt to German, where adapted models (5) – (10) almost close the gap with the upper bound baselines ( (1) – (3) ), but this is not the case for Korean, where this gap is still important.
Revised: We also observe that it is easier to adapt to German, where adapted models (5) – (10) almost close the gap with the upper bound baselines ( (1) – (3) ), but this is not the case for Korean, where this gap is still important.

Original: This is in line with Muller et al.’s (2021) findings suggesting that knowledge transfer is more efficient between related languages.
Revised: This is in line with Muller et al.’s (2021) findings suggesting that knowledge transfer is more efficient between related languages.

Original: Original Pretraining Steps.
Revised: Original Pretraining Steps.

Original: The models (7) , (10) and (11) compare the effect of the pretraining steps on the new language adaptability.
Revised: The models (7) , (10) and (11) compare the effect of the pretraining steps on the new language adaptability.

Original: We observe that, after language adaptation, the early pretraining checkpoint (row (11) in Table 2) only performs as good as a non-adapted model (4) when evaluated with prompts.
Revised: We observe that, after language adaptation, the early pretraining checkpoint (row (11) in Table 3) only performs as good as a non-adapted model (4) when evaluated with prompts.

Original: However, after further finetuning on the downstream task, its performance catches up with later checkpoints.
Revised: However, after further finetuning on the downstream task, its performance catches up with later checkpoints.

Original: Thispreliminary result suggests that researchers who are interested in adapting a multilingual LM to their language of interest may not need a large number of pretraining steps.
Revised: This result suggests that the model is capable to recover a lot of task knowledge during finetuning. Whether the knowledge encoded in the pretrained model was useful needs further investigation.

Original: Adapters’ Capacity
Revised: Adapters’ Capacity.

Original: The models with different adapters’ capacity (models (7) , (8) and (9) ) achieve comparable downstream performance when adapting to German and Korean languages.
Revised: The models with different adapters’ capacity (models (7) , (8) and (9) ) achieve comparable downstream performance when adapting to German and Korean languages.

Original: The small impact of parameter budget on the final performance could be due to the limited amount of training data.
Revised: The small impact of parameter budget on the final performance could be due to the limited amount of training data.

Original: This implies that in the real low-resource settings there is no need for a very large parameter budget.
Revised: This implies that in the real low-resource settings there is no need for a very large parameter budget.

Original: Positional Embeddings.
Revised: Positional Embeddings.

Original: Our results (model (7) vs model (12) in Table 2) highlight the importance of adapting the positional embedding layer of the 
Revised: Our results (models (7) vs (13) , models (10) vs (12) ) highlight the importance of adapting the positional embedding layer of the LM.

Original: LM, which gives 6.5% accuracy boost for crosslingual NLI.
Revised: For German, the adapted positional embeddings gives 6.5% accuracy boost for crosslingual NLI; for Korean, we obtain 4.9% accuracy improvement for cross-lingual NLI and 12.7% improvement for the supervised setting.

Original: This is in line with the findings by Ravishankar and Søgaard (2021).
Revised: This is in line with the findings by Ravishankar and Søgaard (2021).

Original:  Conclusion and Future Work 
Revised:  Limitations and Future Work Languages

Original: In this work we’ve compared different adaptation strategies to add a new language into pretrained model and identified important components for adaptation.
Revised: In our language adaptation setup, we use pseudo-low resource languages by limiting the number of samples for German and Korean, which are classified as high-resource languages by Joshi et al. In our future work, we plan to explore language adaptation to truly low-resource languages such as Sinhala and Burmese. Evaluation Tasks

Original: Limitations.
Revised: We only evaluate our models on the natural language inference task, which is a high-level NLP classification task. We plan to extend our evaluation task suite to include generative tasks such as multilingual abstractive summarization (Hasan et al., 2021) and sequence labeling tasks such as named entity recognition (Rahimi et al., 2019) We acknowledge that the performances might not generalize to truly low-resource languages because they have exceptionally limited labeled resources. Training Examples

Original: Our work only studies adaptation of multilingual LMs totwo unseen languages on a single task.
Revised: We adapt the model to a new language by training on 100,000 samples of the new language. In reality, many languages may have far more monolingual data available.

Original: In the future, we want to include more languages for adaptation  and evaluate the models on more NLU and NLG tasks.
Revised: Our future experiments would include language adaptation with varying number of training samples and evaluate the effect of larger training samples on downstream performance and the use of adapters. Origianl Pretraining Steps As we observe that the model still performs well on downstream tasks after finetuning despite only 12,000 of original pretraining steps, we plan to evaluate how important it is to start from the pre-trained model by comparing the performance with a randomly initialized pretrained model. Tokenizers

Revised: We retrain the BigScience tokenizer with its original vocabulary size of 130,000 on the entire subcorpora of the new language.

Original: We also plan to explore the impact of the  size of language adaptation training samples on model’s downstream performance.
Revised: In our future work, we plan to explore the effect of the vocabulary size of the tokenizers and the number of samples used to train the tokenizers.  Conclusion In this work we’ve compared different adaptation strategies to add a new language into pretrained BigScience model and identified important components for adaptation.

Revised: Acknowledgements

Revised: The authors would like to thank anonymous reviewers for their helpful feedback.

Original: A BigScience Pretraining
Revised: A BigScience Pretraining

Original: The BigScience multilingual autoregressive language model (LM) uses the GPT architecture andthe causal language modeling objective.
Revised: The BigScience multilingual autoregressive language model (LM) uses the GPT architecture and the causal language modeling objective.

Original: Its tokenizer is the byte-level BPE with a vocab size of130,001.
Revised: Its tokenizer is the byte-level BPE with a vocab size of 130,001.

Original: For pretraining, it uses the Adam optimizer with a batch size 512, a learning rate of 2ewith cosine decay over 16,927,083 samples and warmup over 216,320 samples, weight decay of 0.1, and gradient clipping of 1.0.
Revised: For pretraining, it uses the Adam optimizer with a batch size 512, a learning rate of 2ewith cosine decay over 16,927,083 samples and warmup over 216,320 samples, weight decay of 0.1, and gradient clipping of 1.0.

Original: The deduplicate subcorpora from OSCAR (Ortiz Suárez et al., 2019)  for pretraining the model are sampled using the following probabilities: Indonesian (0.0554), Basque (0.0184), Vietnamese (0.0684), Chinese (0.1339),
Revised: Table 4 reports the statistics of the deduplicate subcorpora from OSCAR (Ortiz Suárez et al., 2019) used for pretraining the BigScience model.

Original: Urdu (0.0267), Spanish (0.1118), Catalan (0.0395), Portuguese (0.0867), French (0.1110), English (0.2107), Hindi (0.0398), Arabic (0.0638) and Bengali (0.0339).
Revised: [Table]

Original: B Language Adaptation Training Details
Revised: B Language Adaptation Training Details

Original: For all the language adaptation strategies, we train by using the AdamW optimizer with default parameters, batch size of 8, learning rate of 1 e − with linear decay, and maximum sequence length of 1,024.
Revised: For all the language adaptation strategies, we train by using the AdamW optimizer with default parameters, batch size of 8, learning rate of 1 e −with linear decay, and maximum sequence length of 1,024.

Original: It takes around 30 hours to complete the training on a single V100 GPU machine.
Revised: It takes around 30 hours to complete the training on a single V100 GPU machine.

Original: C Relationship between NLI Task 
Revised: C Relationship between NLI Task Performance and Adapter Capacity

Original: Performance and Adapter Capacity

Original: We convert the reduction factor of language adapters to total adapter capacity and plot the
Revised: We convert the reduction factor of language adapters to total adapter capacity and plot the

Original: NLI performance against it in Figure 2.
Revised: NLI performance against it in Figure 2.

Original: The figure shows comparable performance across all NLI evaluation settings for both German and Korean languages even with small adapter capacity.
Revised: The figure shows comparable performance across all NLI evaluation settings for both German and Korean languages even with small adapter capacity.

Original: [Figure]
Revised: [Figure]