Original: A N E XPLORATION OF V OCABULARY S IZE AND T RANSFER E FFECTS IN M ULTILINGUAL L ANGUAGE M ODELS FOR A FRICAN L ANGUAGES
Revised: A N E XPLORATION OF V OCABULARY S IZE AND T RANSFER E FFECTS IN M ULTILINGUAL L ANGUAGE M ODELS FOR A FRICAN L ANGUAGES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multilingual pretrained language models have been shown to work well on many languages, even those they were not originally pretrained on.
Revised: Multilingual pretrained language models have been shown to work well on many languages, even those they were not originally pretrained on.

Original: Despite their empirical success in downstream tasks, there is still a gap in understanding of “what makes them tick”.
Revised: Despite their empirical success in downstream tasks, there is still a gap in understanding of “what makes them tick”.

Original: In this paper, we try to understand the effects of sharing a vocabulary space on the cross-lingual abilities of a multilingual model.
Revised: In this paper, we try to understand the effects of sharing a vocabulary space on the cross-lingual abilities of a multilingual model.

Original: We train multiple monolingual and multilingual models and compare their effectiveness on downstream tasks.
Revised: We train multiple monolingual and multilingual models and compare their effectiveness on downstream tasks.

Original: In monolingual models, a single language occupies the entire vocabulary space, limiting possible cross-lingual transfer.
Revised: In monolingual models, a single language occupies the entire vocabulary space, limiting possible cross-lingual transfer.

Original: Whereas in a multilingual setting, the model benefits from cross-lingual transfer with a trade-off of having to split the vocabulary space between multiple languages.
Revised: Whereas in a multilingual setting, the model benefits from cross-lingual transfer with the tradeoff of having to split the vocabulary space between multiple languages.

Original: We present a comprehensive study of the effects of a shared vocabulary space, cross-script pretraining, and high-resource transfer on the cross-lingual abilities of multilingual models in zero- and few-shot settings.
Revised: We present a comprehensive study of the effects of a shared vocabulary space, cross-script pretraining, and high-resource transfer on the cross-lingual abilities of multilingual models in zero- and few-shot settings.

Original: From our study, we observe that scaling the number of languages is beneficial for cross-lingual transfer in low-resource multilingual models up until a point, after which transfer effects saturate.
Revised: From our study, we observe that scaling the number of languages is beneficial for cross-lingual transfer in lowresource multilingual models up until a point, after which transfer effects saturate.

Original: We find that there is not much benefit from pretraining low-resource multilingual models with a high-resource language, and that cross-lingual transfer is possible even when the languages belong to different scripts.
Revised: We find that there is not much benefit from pretraining low-resource multilingual models with a high-resource language, and that cross-lingual transfer is possible even when the languages are written with different scripts.

Original: This empirical study is conducted in the context of three linguistically different low-resource African languages—Amharic, Hausa, and Swahili—and evaluation was performed on two different tasks, text classification and named entity recognition.
Revised: This empirical study was conducted in the context of three linguistically different low-resource African languages—Amharic, Hausa, and Swahili—and evaluation was performed on two different tasks, text classification and named entity recognition.

Original: During the course of our experiments, we also performed an audit of the quality of two common low-resource language corpora (Common Crawl and BBC News data).
Revised: During the course of our experiments, we also performed an audit of the quality of two common low-resource language corpora, Common Crawl and BBC News.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Circa 2017, transformers (Vaswani et al., 2017) emerged as the de facto architecture for training language models, and have since become the backbone of many state-of-the-art neural models.
Revised: Circa 2017, transformers (Vaswani et al., 2017) emerged as the de facto architecture for building language models, and have since become the backbone of many state-of-the-art neural models.

Original: Unsupervised learning of text representations using transformers has resulted in significant improvements on multiple tasks in NLP.
Revised: Unsupervised learning of text representations using transformers has resulted in significant improvements on multiple tasks in NLP.

Original: Models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have shown excellent results when fine-tuned and evaluated on different downstream tasks.
Revised: Models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have shown impressive results when fine-tuned and evaluated on different downstream tasks.

Original: The multilingual variants of these models such as mBERT, XLM-R (Conneau et al., 2020),
Revised: The multilingual variants of these models such as mBERT, XLM-R (Conneau et al., 2020),

Original: RemBERT (Chung et al., 2021), and mT5 (Xue et al., 2021), all trained on over 100 languages, havealso pushed the state of the art on cross-lingual understanding tasks by pretraining a single model on several languages in an unsupervised manner with no aligned data.
Revised: RemBERT (Chung et al., 2021), and mT5 (Xue et al., 2021), all trained on over 100 languages, have pushed the state of the art on cross-lingual understanding tasks by pretraining a single model on several languages in an unsupervised manner with no aligned data.

Original: Despite being trained with no explicit supervision, no alignment between the languages, and no explicit cross-lingual objectives, mBERT and XLM-R produce representations that are able to generalize well on a number of languages, even in zero-shot settings.
Revised: Despite being trained with no explicit supervision, no alignment between the languages, and no explicit cross-lingual objectives, mBERT and XLM-R produce representations that are able to generalize well on a number of languages, even in zero-shot settings.

Original: The aforementioned models have been trained on large language combinations consisting of highand low-resource languages, usually amounting to hundreds of gigabytes of data.
Revised: The aforementioned models have been trained on large language combinations consisting of highand low-resource languages, usually amounting to hundreds of gigabytes of data.

Original: However, in
Revised: However, with

Original: CamemBERT (Martin et al., 2020) and in Micheli et al.
Revised: CamemBERT (Martin et al., 2020) and in Micheli et al.

Original: , researchers showed that monolingual language models can be trained using relatively little data.
Revised: , researchers showed that monolingual language models can be trained using relatively little data.

Original: More recently, Ogueji et al.
Revised: More recently, Ogueji et al.

Original: released AfriBERTa, a multilingual language model that was pretrained on 11 low-resource African languages.
Revised: released AfriBERTa, a multilingual language model that was pretrained on 11 low-resource African languages.

Original: It achieved competitive results on multiple downstream tasks despite being pretrained on a much smaller corpus compared to XLM-R. These empirical results show that even in the “small data regime”, pretraining withlinguistically similar low-resource languages can yield competitive results.
Revised: AfriBERTa has achieved competitive results on multiple downstream tasks despite being pretrained on a much smaller corpus compared to XLM-R. These empirical results show that even in the “small data regime”, pretraining with low-resource languages can yield competitive results.

Original: In this work, we try to develop an understanding of the success of low-resource multilingual models by exploring certain components of the model.
Revised: In this work, our goal is to develop an understanding of the successes of low-resource multilingual models by exploring certain components of the model.

Original: We explore these key components using AfriBERTa, in the context of three linguistically different African languages:
Revised: We explore these key components using AfriBERTa, in the context of three linguistically different African languages:

Original: Amharic, Hausa, Swahili.
Revised: Amharic, Hausa, Swahili.

Original: These languages are spoken by millions of users across different regions in Africa with lots of digital exposure.
Revised: These languages are spoken by millions of users across different regions in Africa with lots of digital exposure.

Original: However, these languages have very limited support for natural language technologies (Nekoto et al., 2020).
Revised: However, these languages have very limited support for natural language technologies (Nekoto et al., 2020).

Original: We investigate the trade-off between the benefits of cross-lingual transfer and the sharing of vocabulary space in low-resource settings.
Revised: We investigate the tradeoff between the benefits of cross-lingual transfer and the sharing of vocabulary space in low-resource settings.

Original: By cross-lingual transfer, we mean the ability of a model to better perform tasks in one language by leveraging pretraining in a different language, an effect that is well known and dates to the earliest usages of multilingual models (Wu & Dredze, 2019).
Revised: By cross-lingual transfer, we mean the ability of a model to better perform tasks in one language by leveraging pretraining in a different language, an effect that is well known and dates to the earliest usages of multilingual models (Wu & Dredze, 2019).

Original: Specifically, we use a varying number of monolingual and multilingual configurations to determine the effect of shared vocabulary space on the transfer abilities of multilingual models.
Revised: Specifically, we use different monolingual and multilingual configurations to determine the effect of shared vocabulary spaces on the transfer abilities of multilingual models.

Original: We train monolingual models and multilingual models from scratch on three different languages,  fine-tune and evaluate each model on downstream token and sentence classification tasks.
Revised: We train monolingual models and multilingual models from scratch on three different languages, then fine-tune and evaluate each model on downstream token and sentence classification tasks.

Original: We also ran a token analysis to further ascertain the extent of token overlap across all three languages in the vocabulary.
Revised: We also performed a token analysis to further ascertain the extent of token overlap across all three languages.

Original: Finally, to understand the role of training data quality on theimproved results shown by AfriBERTa, we train three sets of models of different parameter sizes on both corpora: BBC data used in AfriBERTa and  Common Crawl (CC-100) data used in XLM-R (Conneau & Lample, 2019).
Revised: Finally, to understand the role of pretraining corpus data quality on the results demonstrated by AfriBERTa, we train three sets of models of different parameter sizes on the BBC data used in AfriBERTa and the Common Crawl (CC-100) data used in XLM-R (Conneau & Lample, 2019).

Original: Training configurations and hyperparameters were kept constant while varying the dataset.
Revised: Training configurations and hyperparameters were kept constant while varying the dataset, thus isolating the impact of the pretraining corpus.

Original: We find thatboth models  yield similar results when evaluated on the same tasks.
Revised: We find that models trained on both datasets yield similar results when evaluated on the same tasks.

Original: Our findings can be summarized as follows:
Revised: Our findings can be summarized as follows:

Original: • Low-resource multilingual models do not appear to benefit from pretraining together with a high-resource language (English, in this case).
Revised: • Low-resource multilingual models do not appear to benefit from pretraining together with a high-resource language (English, in this case).

Original: F 1 score on downstream token and text classification tasks remain approximately the same for models jointly pretrained on multiple lowresource languages and English.
Revised: F 1 score on downstream token and text classification tasks remain approximately the same for models jointly pretrained on multiple lowresource languages and English.

Original: • Cross-lingual transfer is possible among languages, even when they belong to entirely different scripts.
Revised: • Cross-lingual transfer is possible among languages in monolingual models, even when they belong to entirely different scripts.

Original: This shows that monolingual models are sometimes able to capture multilingual representations.
Revised: This shows that monolingual models are sometimes able to capture multilingual representations.

Original: • Monolingual models trained with a multilingual vocabulary generally perform better than those trained with a monolingual vocabulary.
Revised: • Monolingual models trained with a multilingual vocabulary generally perform better than those trained with a monolingual vocabulary when transferring to languages with different scripts.

Original: Overall, monolingual models only outperform multilingual models when evaluated on a task in the same language they were pretrained on.
Revised: Overall, monolingual models only outperform multilingual models when evaluated on a task in the same language they were pretrained on.

Revised: However, even in such cases, the difference is often marginal.

Original: • More languages provide better cross-lingual transfer up until a point, beyond which crosslingual effects seems to degrade.
Revised: • More languages provide better cross-lingual transfer up until a point, beyond which crosslingual effects seem to degrade.

Original: This is evident when comparing results across the spectrum of XLM-R (100 languages) to AfriBERTa (11 languages) to our 3L models, and to monolingual models.
Revised: This is evident when comparing results across the spectrum of XLM-R (100 languages) to AfriBERTa (11 languages) to our 3L models, and to monolingual models.

Original: This is referred to as the curse of multilinguality by Conneau et al.
Revised: This is referred to as the curse of multilinguality by Conneau et al.

Original: (2020) • There does not appear to be much difference in the quality of Common Crawl and BBC data.
Revised: (2020) • There does not appear to be much difference in the quality of Common Crawl and BBC data.

Original: Models trained separately on both corpora yield similar results when evaluated on the same downstream datasets.
Revised: Models trained separately on both corpora yield similar results when evaluated on the same downstream datasets.

Original: • In a learned vocabulary, token overlap between the languages can be very high, especially among typologically similar languages.
Revised: • In a learned vocabulary, token overlap between the languages can be very high, especially among typologically similar languages.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Unsupervised multilingual models have been shown to generalize in a zero-shot cross-lingual setting.
Revised: Unsupervised multilingual models have been shown to generalize in a zero-shot cross-lingual setting.

Original: One hypothesis about this generalization is the deep abstractions resulting from shared vocabulary and joint training across multiple languages (Pires et al., 2019; Wu & Dredze, 2019).
Revised: One hypothesis about this generalization is the deep abstractions resulting from shared vocabu- [Table] lary and joint training across multiple languages (Pires et al., 2019; Wu & Dredze, 2019).

Original: However,
Revised: However,

Original: Artetxe et al.
Revised: Artetxe et al.

Original: (2020) contradicted this and empirically showed that neither shared vocabulary nor joint multilingual training is necessary for transfer.
Revised: (2020) contradicted this and empirically showed that neither shared vocabulary nor joint multilingual training is necessary for transfer.

Original: They showed that deep monolingual models learn some abstractions that generalize across languages.
Revised: They showed that deep monolingual models learn some abstractions that generalize across languages.

Original: Another hypothesis proposed as a way to explain the success of multilingual language models has to do with some level of language similarity.
Revised: Another hypothesis proposed as a way to explain the success of multilingual language models has to do with some level of language similarity.

Original: This could be lexical similarity (shared words or word-parts) or structural similarities (word-ordering[Table] or word-frequency), or both, but K et al.
Revised: This could be lexical similarity (shared words or word-parts), structural similarities (word-ordering or word-frequency), or both, but K et al.

Original: (2020) showed that m-BERT is cross-lingual even when there is no word overlap.
Revised: (2020) showed that mBERT is cross-lingual even when there is no word overlap.

Original: In another study done by K et al. , they found that the size and depth of a multilingual model play a more important role to its cross-lingual success compared to the lexical overlap between the languages.
Revised: In another study, the same researchers found that the size and depth of a multilingual model play a more important role to its cross-lingual success compared to the lexical overlap between the languages.

Original: In their work, Ammar et al.
Revised: Related, Ammar et al.

Original: (2016) showed that multilingual embeddings improve the transfer capabilities of multilingual models by augmenting them with lexical information.
Revised: (2016) showed that multilingual embeddings improved the transfer capabilities of multilingual models by augmenting them with lexical information.

Original: They demonstrated this by training a single multilingual model for dependency parsing and use it to parse sentences in multiple languages.
Revised: They demonstrated this by training a single multilingual model for dependency parsing and used it to parse sentences in multiple languages.

Original: However, much of this study has been done w.r.t models that were trained on datasets that contain a larger percentage of high-resource languages, with very few African languages despite the 2000+ languages spoken on the  continent (Eberhard et al., 2019).
Revised: However, most of these studies have been conducted with models that were pretrained on corpora with high-resource languages, with very little representation of the 2000+ languages spoken on the African continent (Eberhard et al., 2019).

Original:  mBERT was trained on 104 languages and only 3  were African while XLM-R only contained about 8 African languages out of 100 languages.
Revised: For example, mBERT was trained on 104 languages, only 3 of which were African, while XLM-R only contained about 8 African languages out of 100 languages.

Original: Hence, our work aims to bridge this knowledge gap by exploring the factors that contribute to multilinguality with a focus on low-resource African languages.
Revised: Hence, our work aims to bridge this knowledge gap by exploring the factors that contribute to multilinguality with a focus on low-resource African languages.

Original:  M ETHODOLOGY 3. D ATA
Revised:  M ETHODOLOGY 3. D ATA

Original: Pretraining corpora.
Revised: Pretraining corpora.

Original: We select a subset of the languages on which both AfriBERTa and XLMR were originally pretrained (Amharic, Hausa, and Swahili).
Revised: We select a subset of the languages on which both AfriBERTa and XLMR were originally pretrained (Amharic, Hausa, and Swahili).

Original: These languages belong to different written scripts: Hausa and Swahili belong to the Latin Script 1 while Amharic belongs to the Ge‘ez
Revised: These languages belong to different written scripts: Hausa and Swahili belong to the Latin Script 1 while Amharic belongs to the Ge‘ez

Original: Script.
Revised: Script.

Original: This allows us to investigate cross-script transfer effects.
Revised: This allows us to investigate cross-script transfer effects.

Original: We have two different pretraining corpora—one from the British Broadcasting Corporation (BBC)
Revised: We have two different pretraining corpora—one from the British Broadcasting Corporation (BBC)

Original: News 3 and the other from the Common Crawl.
Revised: News 3 and the other from the Common Crawl.

Original: 4 As the Common Crawl includes BBC as one of its sources and the time periods of the snapshots we use intersect, we acknowledge that there might be some overlap between both datasets.
Revised: 4 As the Common Crawl includes BBC as one of its sources and the time periods of the snapshots we use intersect, we acknowledge that there might be some overlap between both datasets.

Original: We randomly sample sentences from the Common Crawl corpus to match the size of data in  BBC corpus.
Revised: We randomly sample sentences from the Common Crawl corpus to match the size of data in the BBC corpus.

Revised: We use this sample of the Common Crawl corpus only in our investigation of possible quality differences between both sources.

Original: We also sample enough English sentences from the Common Crawl to match the size of the Swahili corpus, which has the largest corpora of the three low-resource languages we consider.
Revised: We also sample enough English sentences from the Common Crawl to match the size of the Swahili corpus, which has the largest corpora of the three low-resource languages we consider.

Original: We use this in our investigations of the impact of highresource transfer.
Revised: We use this in our investigations of the impact of high-resource transfer.

Original: Table 1 shows the sizes of the datasets with a breakdown of the number of tokens and sentences in each language.
Revised: Table 1 shows the sizes of the datasets with a breakdown of the number of tokens and sentences in each language.

Revised: Named Entity Recognition.

Revised: We perform NER on the publicly available MasakhaNER dataset (Adelani et al., 2021), containing 10 African languages:

Revised: Amharic, Hausa, Igbo, Kinyarwanda, Luganda, Luo, Nigerian Pidgin, Swahili, Wolof, and Yor`ub´a.

Revised: This work and subsequent work (Ogueji et al., 2021) demonstrated that XLM-R and AfriBERTa exhibit impressive cross-lingual capabilities by generalizing to previously unseen languages during pretraining.

Original: Text Classification.
Revised: Text Classification.

Original: We perform text classification on news title topic classification datasets for
Revised: We perform text classification on news title topic classification datasets for

Original: Hausa and Yor`ub´a (Hedderich et al., 2020).
Revised: Hausa and Yor`ub´a (Hedderich et al., 2020).

Original: The authors established strong baselines using multilingual PLMs and multilingual PLMs + English adaptive fine-tuning.
Revised: The authors established strong baselines using multilingual pretrained language models with and without English adaptive fine-tuning.

Original: They found that both mBERT and XLM-R outperform simpler neural network and RNN baselines in few-shot and zero-shot settings.
Revised: They found that both mBERT and XLM-R outperform simpler neural network and recurrent neural network (RNN) baselines in few-shot and zero-shot settings.

Original: In Ogueji et al.
Revised: In Ogueji et al.

Original: , AfriBERTa outperformed all previously established baselines.
Revised: , AfriBERTa outperformed all previously established baselines.

Original: Named Entity Recognition.

Original: We perform NER on the publicly available MasakhaNER dataset (Adelani et al., 2021), containing 10 African languages:

Original: Amharic, Hausa, Igbo, Kinyarwanda, Luganda, Luo, Nigerian Pidgin, Swahili, Wolof, and Yor`ub´a.

Original: This work and subsequent work (Ogueji et al., 2021) demonstrated that XLM-R and AfriBERTa exhibit impressive cross-lingual capabilities by generalizing to previously unseen languages during pretraining.

Original: 3. M ODELS
Revised: 3. M ODELS

Original: We train multiple transformer models in our experiments.
Revised: We train multiple transformer models in our experiments.

Original: All models share the same architecture as
Revised: All models share the same architecture as

Original: AfriBERTa, which was trained with the same training objective in Conneau et al.
Revised: AfriBERTa, which was trained with the same training objective as Conneau et al.

Original: 3L multilingual models : multilingual transformer models jointly pretrained on Amharic, Hausa, and Swahili.
Revised: 3L multilingual models : multilingual transformer models jointly pretrained on Amharic, Hausa, and Swahili.

Original: These models serve to explore pretraining on a set of three typographically different languages and to compare downstream results to similar models trained on a much larger set of languages.
Revised: These models serve to explore pretraining on a set of three typographically different languages and to compare downstream results to similar models trained on a much larger set of languages.

Original: This enables us to investigate the impact of a shared vocabulary on the transfer abilities of a multilingual model in low-resource settings.
Revised: This enables us to investigate the impact of a shared vocabulary on the transfer abilities of a multilingual model in low-resource settings.

Original: We process all the languages using the same learned subword vocabulary of 20000 tokens generated using the unigram language model (Kudo, 2018) trained with SentencePiece (Kudo & Richardson, 2018).
Revised: We process all the languages using the same learned subword vocabulary of 20000 tokens generated using the unigram language model (Kudo, 2018) trained with SentencePiece (Kudo & Richardson, 2018).

Original: 3L + En multilingual model : a multilingual transformer model jointly trained on three low-resource languages (Amharic, Hausa, and Swahili) and a high-resource language (English).
Revised: 3L + En multilingual model : a multilingual transformer model jointly trained on three low-resource languages (Amharic, Hausa, and Swahili) and a high-resource language (English).

Original: Since this model differs only by the inclusion of English, it allows us to isolate the effect of transfer from a highresource language in our setting.
Revised: Since this model differs only by the inclusion of English, it allows us to isolate the effect of transfer from a highresource language in our setting.

Original: Monolingual Models : three monolingual models, each on Amharic, Hausa, and Swahili.
Revised: Monolingual Models : three monolingual models, each on Amharic, Hausa, and Swahili.

Original: We pretrain on the aforementioned languages, fine-tune the models using task-specific supervised training data and evaluate that task in different languages.
Revised: We pretrain on the aforementioned languages, fine-tune the models using task-specific supervised training data and evaluate that task in different languages.

Original: This allows us to observe the benefits of allocating  entire vocabulary space to one language with no cross-lingual transfer.
Revised: This allows us to observe the benefits of allocating the entire vocabulary space to one language with no cross-lingual transfer.

Original: 3. E XPERIMENTAL S ETUP
Revised: 3. E XPERIMENTAL S ETUP

Original: Our experiments largely follow Ogueji et al.
Revised: Our experiments largely follow Ogueji et al.

Original: We pretrain models using a batch size of 32 and 8 gradient accumulation  steps.
Revised: We pretrain models using a batch size ofand gradient accumulation of 8 steps.

Original: We use a learning rate of 1e-4 with AdamW as  optimizer.
Revised: We use a learning rate of 1e-4 with AdamW as the optimizer.

Original: We pretrain monolingual models for 40000 steps with 4000 linear warm-up steps and multilingual models for 40000 steps with 4000 linear warm-up steps. 
Revised: We pretrain all models for 40000 steps with 4000 linear warm-up steps. Ogueji et al.

Original: We use a vocabulary size of 20000 for  multilingual  models, and for the monolingual models we use different vocabulary sizes for each language corresponding to the  vocabulary space they occupy in the multilingual tokenizers:for Amharic, 9000 for Hausa, and 8000 for Swahili.
Revised: (2021) showed that medium vocabulary sizes often outperform larger ones for small datasets. Following this, we use a vocabulary size of 20000 for all our multilingual models. For monolingual models, our initial explorations also showed that the 20000 vocabulary size produced the best results across model sizes. Additionally, using the same vocabulary size across all models allow us to explore the effects of shared vocabulary on cross-lingual transfer.

Original: For NER and text classification, we train models by adding a linear classification layer to the pretrained models and fine-tuning all parameters.
Revised: For NER and text classification, we train models by adding a linear classification layer to the pretrained models and fine-tuning all parameters.

Original: All F 1 scores reported are averaged over 5 runs with different random seeds.
Revised: All F 1 scores reported are averaged over 5 runs with different random seeds.

Original:  R ESULTS AND D ISCUSSION  4. T OKENIZATION E FFECTS IN M ONOLINGUAL M ODELS
Revised:  R ESULTS AND D ISCUSSION . 4. T OKENIZATION E FFECTS IN M ONOLINGUAL M ODELS

Original: We pretrain monolingual models in two different settings: (1) using monolingual tokenizers trained on the individual languages and (2) using the tokenizeras from our 3L multilingual models.
Revised: We pretrain monolingual models in two different settings: (1) using monolingual tokenizers trained on the individual languages and (2) using the tokenizer from our 3L multilingual models.

Original: In the monolingual tokenizers, the pretraining language occupies  entire vocabulary, limiting possible  transfer to other downstream task languages.
Revised: In the monolingual tokenizers, the pretraining language occupies the entire vocabulary, limiting possible [Table] [Table] transfer to other downstream task languages.

Original: However, the opposite is the case with multilingual tokenizers where multiple languages share  vocabulary space but this benefits possible cross-lingual transfer.
Revised: However, the opposite is the case with multilingual tokenizers, where multiple languages share the vocabulary space, but this benefits possible crosslingual transfer.

Original: The multilingual tokenizers were trained on a concatenation of Amharic, Hausa, and Swahili corpora.
Revised: The multilingual tokenizers were trained on a concatenation of Amharic, Hausa, and Swahili corpora.

Original: We comparethe resultswe obtained from both sets of models in Table 2.
Revised: We compare results from both sets of models in Tables 2 and 3.

Revised: For reference, we report comparable AfriBERTa results copied from Ogueji et al.

Original:  We observe an improvement by an average of 2 F 1 points on text classification using multilingual tokenizers. We find even more surprising results in named entity recognition, where models trained using multilingual tokenizers outperform by up to 7 F 1 points on the three languages seen by the tokenizer.
Revised: (2021), which used a multilingual tokenizer but is also pretrained with more languages. We observe that a multilingual vocabulary is generally more effective than a monolingual vocabulary when fine-tuning on a language in a script different from that used during pretraining. From Table 2, we can see that monolingual Hausa and Swahili have higher F 1 scores on Amharic NER when using a multilingual tokenizer.

Original: However,  results are mixed on unseen languages, where models with multilingual tokenizers reach up to 15 F 1 points improvement on Ibo but is 4 F 1 points worse on Wolof.
Revised: However, when languages are in the same script, our results are worse by 1.5 F 1 points. Results on unseen languages also degrade when we use multilingual tokenizers. Except for Kinyarwanda, we do not gain improvements in F 1 for NER on unseen languages by using the multilin-

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: gual tokenizer.

Revised: Instead, the monolingual tokenizer is up to 7 F 1 points better (such as on PCM) and 1.4 F 1 points better on average.

Revised: As Table 3 shows, F 1 scores improve by up to 4 points for monolingual Amharic on text classification when we use multilingual tokenizers.

Revised: For monolingual Hausa and Swahili, which are of the same script as the task languages, F 1 scores generally remain close regardless of the tokenizers used.

Original: Nevertheless, these results show that simply training with a multilingual tokenizer can improve  transfer in cases where we know the language of  downstream tasks.
Revised: Nevertheless, these results show that simply training with a multilingual tokenizer can improve cross-script transfer in cases where we know the language of the downstream tasks.

Original: We hypothesize that in these cases, the model learns generalizable language abstractions and embeddings for tokens of the language it is pretrained on.
Revised: We hypothesize that in these cases, the model learns generalizable language abstractions and embeddings for tokens of the language it is pretrained on.

Original: During fine-tuning, token embeddings are then learned for the task language.
Revised: During fine-tuning, token embeddings are then learned for the task language.

Original: Thus, the model benefits from monolingual transfer and having vocabulary items to adequately represent languages it is fine-tuned on.
Revised: Thus, the model benefits from monolingual transfer and having vocabulary items to adequately represent languages it is fine-tuned on.

Original: 4. M ONOLINGUAL VS . M ULTILINGUAL M ODELS
Revised: 4. M ONOLINGUAL VS . M ULTILINGUAL M ODELS

Original: We pretrain multilingual models in two different settings.
Revised: We pretrain multilingual models in two different settings.

Original: In the first condition, we pretrain on three languages:
Revised: In the first condition, we pretrain on three languages:

Original: Amharic, Hausa and Swahili.
Revised: Amharic, Hausa, and Swahili.

Original: In the second condition, we add a high-resource language,
Revised: In the second condition, we add a high-resource language,

Original: English, and pretrain similarly.
Revised: English, and pretrain similarly.

Original: Results are presented in Tables 4 and 5.
Revised: Results are presented in Tables 4 and 5.

Original: Table 5 presents the NER results in greater detail.
Revised: Table 4 presents the NER results in greater detail.

Original: From these results, we can see that F 1 increases as we scale the number of languages from our 3L models to AfriBERTa.
Revised: From these results, we can see that F 1 increases as we scale the number of languages from our 3L models to AfriBERTa.

Original: However, this does not scale to XLM-R and mBERT with more languages.
Revised: However, this does not scale to XLM-R and mBERT with more languages.

Original: When we compare results in Tables 2 and 5, multilingual models are generally better than all variants of our monolingual models.
Revised: When we compare results in Tables 2 and 4, multilingual models are in most cases better than all variants of our monolingual models.

Original: Monolingual models only remain competitive with multilingual models when the task language is the same as the pretraining language.
Revised: Generally, monolingual models only remain competitive with multilingual models when the task language is the same as the pretraining language.

Original: This demonstrates that multilingual models benefit from cross-lingual transferbetween the languages.
Revised: This demonstrates that multilingual models benefit from cross-lingual transfer

Original: For the multilingual model trained without English, this result confirms that transfer is occurring without any high-resource language.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: between the languages. For the multilingual models trained without English, this result confirms that transfer is occurring without any high-resource language.

Original: As Table 4 shows, high-resource transfer benefits text classification for both Yor ` ub ´ a and Hausa.
Revised: As Table 5 shows, high-resource transfer benefits text classification for both Yor ` ub ´ a and Hausa.

Original: However, it does not lead to any improvements on the NER task for all languages present during pretraining, as well as several languages unseen during pretraining.
Revised: However, it does not lead to any improvements on the NER task for all languages present during pretraining, as well as several languages unseen during pretraining.

Original: Across the other unseen languages, we see improvements by up to 3 F 1 points on Luo and Nigerian Pidgin.
Revised: Across the other unseen languages, we see improvements by up to 3 F 1 points on Luo and Nigerian Pidgin.

Original: However, the transfer effects observed on Nigerian Pidgin is likely explained by its similarity with English.
Revised: However, the transfer effects observed on Nigerian Pidgin is likely explained by its similarity with English.

Original: For monolingual models, we also find that transferring from languages with the same Subject-VerbObject order produced better F 1 .
Revised: For monolingual models, we also find that transferring from languages with the same subject–verb– object (SVO) order produced higher F 1 scores.

Original: When we compare the F 1 of all models to what we obtain with models trained and evaluated on the same language, we find that the drop-off when transferring from Amharic, an SOV language, to SVO languages was less than when transferring from SVO languages (Hausa and Swahili) to Amharic.
Revised: When we compare the F 1 of all models to what we obtain with models trained and evaluated on the same language, we find that the drop-off when transferring from Amharic, an SOV language, to SVO languages was less than when transferring from SVO languages (Hausa and Swahili) to Amharic.

Original: This trend was also noted by Pires et al.
Revised: This trend was also noted by Pires et al.

Original: (2019) for zero-shot POS accuracy.
Revised: (2019) for zero-shot POS accuracy.

Original: 4. E VALUATION OF P RETRAINING C ORPUS Q UALITY
Revised: 4. E VALUATION OF P RETRAINING C ORPUS Q UALITY

Original: Common Crawl and BBC News data are two of the most common sources of unsupervised data for low-resource languages.
Revised: Common Crawl and BBC News data are two of the most common sources of unsupervised data for low-resource languages.

Original: They have been used in training multiple language models, e.g., Common Crawl was used in pretraining XLM-R while BBC was used in pretraining AfriBERTa.
Revised: They have been used in training multiple language models, e.g., Common Crawl was used in pretraining XLM-R while BBC was used in pretraining AfriBERTa.

Original: To investigate whether the difference in data quality of both sources explains some of the performance differences between both models, we attempt to evaluate the quality of both datasets.
Revised: To investigate whether differences in data quality of the corpora explain some of the effectiveness differences between the models, we attempt to evaluate the quality of both datasets.

Original: We trained two fixed capacity multilingual models under the same experimental conditions on both corpora.
Revised: We trained two fixedcapacity multilingual models under the same experimental conditions on both corpora.

Original: In Table 1, we show a breakdown of the size of data, the number of sentences and tokens used in training.
Revised: In Table 1, we show a breakdown of the size of data and the number of sentences and tokens used in pretraining.

Original: We compare the downstream NER and Text classification results from both models in Table 6.
Revised: We compare the downstream NER and text classification results from both models in Table 6.

Original: Based on these results, we find that there does not appear to be much difference in the quality of both corpora, with only Amharic showing a difference of up to 2 F 1 points.[Table] [Table]
Revised: Based on these results, we find that there does not appear to be much difference in the quality of both corpora, with only Amharic showing a difference of up to 2 F 1 points.

Original: 4. T OKEN A NALYSIS
Revised: 4. T OKEN A NALYSIS

Original: We run a token analysis to determine the extent of token overlap between the languages in the learned vocabulary.
Revised: We perform a token analysis to determine the extent of token overlap between the languages in the learned vocabulary.

Original: Tables 7 and 8 show a breakdown of the token overlap between the languages and the number of unique subword tokens present in each language corpus, with respect to the tokenizers.

Original: Despite competing for space in a fixed vocabulary, we observed that some of the tokens are inherently shared between the languages.
Revised: Despite competing for space in a fixed vocabulary, we observe that some of the tokens are inherently shared between the languages.

Original: To get the extent of token overlap between two languages, we extract the set of unique tokens in both corpus using the learned3L multilingual tokenizer, and then find the intersecting tokens present in both languages.
Revised: To compare the extent of token overlap between two languages, we extract the set of unique tokens in both corpora using a learned multilingual tokenizer, and then find the intersecting tokens present in both languages.

Revised: We perform this comparison using our learned 3L multilingual tokenizer as well as the AfriBERTa 11L tokenizer (Ogueji et al., 2021).

Revised: Tables 7 and 8 show breakdowns of the token overlap between the languages and the number of unique subword tokens present in each language corpus, with respect to the tokenizers.

Original: We observe that although the overlap is higher in typologically similar languages (e.g., Hausa and Swahili, Amharic and Tigrinya), there exists some amount of token overlap between languages that belong to different scripts. 
Revised: We observe that although the overlap is higher in typologically similar languages (e.g., Hausa and Swahili, Amharic and Tigrinya), there exists some amount of token overlap between languages that belong to different scripts. By random sampling and manual spot checking, we notice that some [Table]

Revised: [Table]

Original: By random sampling and manual spot checking, we notice that some of the overlap between typologically different languages can be attributed to the presence of named entities (e.g., “BBC”, “Twitter”, etc.) in the corpus.
Revised: Gah represents Gahuza [Table] of the overlap between typologically different languages can be attributed to the presence of named entities (e.g., “BBC”, “Twitter”, etc.) in the corpus.

Original: Consequently, the tokenizers trained separately on Hausa and Swahili contain very few Amharic tokens, limiting possible transfer.
Revised: Consequently, the tokenizers trained separately on Hausa and Swahili contain very few Amharic tokens, limiting possible transfer.

Original: As Table 2 shows, transfer may be improved in such cases by pretraining with a multilingual tokenizer that has seen both scripts.
Revised: As Tableshows, transfer may be improved in such cases by pretraining with a multilingual tokenizer that has seen both scripts.

Original: We gain 7 F 1 points for the model pretrained on Swahili this way.
Revised: We gain 7 F 1 points for the model pretrained on Swahili this way.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: This paper provides an empirical study into the trade-offs between cross-lingual transfer effects and shared vocabulary space for multilingual language models trained on low-resource languages.
Revised: This paper provides an empirical study into the tradeoffs between cross-lingual transfer effects and shared vocabulary space for multilingual language models pretrained on low-resource languages.

Original: It also presents an evaluation of the data quality of two of the most common data sources for unsupervised training of low-resourced African languages.
Revised: It also presents an evaluation of the data quality of two of the most common data sources for pretraining language models for low-resource African languages.

Original: In our experiments, we trained multiple models in different multilingual configurations and evaluated on the same downstream tasks and datasets.
Revised: In our experiments, we trained multiple models in different multilingual configurations and evaluated on the same downstream tasks and datasets.

Original: We also compared our results with AfriBERTa and XLM-R, trained with larger number of languages occupying the same subword vocabulary space.
Revised: We also compared our results with AfriBERTa and XLM-R, pretrained with more languages occupying the same subword vocabulary space.

Original: This way, we are able to draw conclusions on model effectiveness as the number of languages the model is trained on increases.
Revised: This way, we are able to draw conclusions on model effectiveness as the number of languages the model is pretrained on increases.

Original: [Table]

Original: We observed that models achieved better F 1 on downstream tasks as we move from monolingual models to our multilingual models trained on three languages to multilingual models trained onlanguages (AfriBERTa).
Revised: We observed that models achieved better F 1 on downstream tasks as we move from monolingual models to our multilingual models pretrained on three languages to multilingual models pretrained on 11 languages (AfriBERTa).

Original: On the other hand, XLM-R, trained on  100 languages, is not convincingly better than AfriBERTa.
Revised: On the other hand, XLM-R, pretrained on over 100 languages, is not convincingly better than AfriBERTa.

Original: Wealso found that sometimes, monolingual models are able to learn crosslingual abstractions that generalize across languages. Furthermore, we showed that the effectiveness of monolingual models on  tasks may be improved by simply using a multilingual tokenizer.
Revised: We found that sometimes, monolingual models are able to learn cross-lingual abstractions that generalize across languages, and that the effectiveness of monolingual models on downstream tasks may be improved by simply using a multilingual tokenizer.

Revised: This points to how considerations of intended use of models for low-resource languages can impact pretraining decisions.

Original: Also, multilingual models may not benefit much from high-resource transfer when jointly pretrained on low-resource languages and English.
Revised: Finally, multilingual models may not benefit much from high-resource transfer when jointly pretrained on low-resource languages and English.

Original: These findings have practical implications for the development of NLP tools for African languages.
Revised: Together, our findings have practical implications for the development of NLP tools for African languages.

Original: We hope that this research would be useful in understanding the inner workings of multilingual models and would spur further research in low-resource languages.
Revised: We hope that this research would be useful in understanding the inner workings of multilingual models and would spur further research in low-resource languages.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This research was supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada and an AI for Social Good grant from the Waterloo AI Institute; computational resources were provided by Compute Ontario and Compute Canada..findings Attention is all you need 3f5ee243547dee91fbd053c1c4a845aa
Revised: This research was supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada and an AI for Social Good grant from the Waterloo AI Institute; computational resources were provided by Compute Ontario and Compute Canada.