M ono B yte: A Pool of Monolingual Byte-level Language Models (2022)

Abstract: The zero-shot cross-lingual ability of models pretrained on multilingual and even monolingual corpora has spurred many hypotheses to explain this intriguing empirical result. However, due to the costs of pretraining, most research uses public models whose pretraining methodology, such as the choice of tokenization, corpus size, and computational budget, might differ drastically. When researchers pretrain their own models, they often do so under a constrained budget, and the resulting models might underperform significantly compared to SOTA models. These experimental differences led to various inconsistent conclusions about the nature of the cross-lingual ability of these models. To help further research on the topic, we released 10 monolingual byte-level models rigorously pretrained under the same configuration with a large compute budget (equivalent to 420 days on a V100) and corpora that are 4 times larger than the original BERT’s. Because they are tokenizer-free, the problem of unseen token embeddings is eliminated, thus allowing researchers to try a wider range of cross-lingual experiments in languages with different scripts. Additionally, we release two models pretrained on non-natural language texts that can be used in sanity-check experiments. Experiments on QA and NLI tasks show that our monolingual models achieve competitive performance to the multilingual one, and hence can be served to strengthen our understanding of cross-lingual transferability in language models.

## introduction shortly after the publication of bert @xcite , researchers showed that multilingual models 1 https://huggingface.co/monobyte pretrained on the masked language modeling objective can achieve remarkable zero-shot cross-lingual performance on various nlp tasks (i.e., a multilingual model finetuned on a high-resource language and directly evaluated in other languages) @xcite . these empirical results triggered a wave of research that aimed to explain this behavior. @xcite raised the "anchor tokens" hypothesis, i.e., that tokens shared between two languages act as a reference point so models can learn similar concepts. @xcite and @xcite questioned these findings and pointed out that, even without a shared vocabulary, models achieve great cross-lingual transfer performance by leveraging shared parameters only. perhaps even more surprising is the performance of monolingual models in a cross-lingual setting. for instance, models pretrained and finetuned only on english can perform well in french tasks @xcite . @xcite also show that monolingual models finetuned on a foreign language (i.e., a language that differs from pretraining) achieve comparable results to models finetuned on their language. recent work investigates properties of the pretraining corpora that contribute to the model's performance on natural language tasks. models are pretrained on non-natural language corpora, such as code, music, proteins, and artificial languages and their performances are compared with natural language models (papadimitriou and jurafsky, 2020; chiang and yi @xcite @xcite . evidence shows that pretraining on a corpus that contains artificial recursive or hierarchical structure between tokens results in a similar performance when compared to models pretrained on a natural language. however, an outstanding problem is that these experiments are made either 1) using models pretrained by different research groups, whose pretraining configuration differs widely, or 2) using models pretrained under a compute budget or pretraining corpus that are orders of magnitude smaller than the ones used by sota models. for instance, chiang and yi lee (2020) pretrain their models on a dataset 200 times smaller than bert's. this is problematic because certain skills are learned only when corpus size and training budget are large enough @xcite . these differences in the pretraining methodology and undertrained models make it difficult to draw a conclusion about the nature of the crosslingual ability of such models. additionally, since most models use subword tokenization, it is difficult to experiment with languages that do not have a significant subword vocabulary overlap. for example, if a model is pretrained and finetuned on english, it cannot be tested in chinese because there is very little token overlap in their vocabularies, and hence, chinese token embeddings are not learned. this is a problem even for languages with the same script. for example, many spanish embeddings are not learned in an english-only pretraining, and hence it is difficult to tell whether a model's inability to learn cross-lingual representations is due only to tokenization issues @xcite . following @xcite , who advocate for the release of pretraining models, corpus, and script as a way to strengthen the field, we release byte-level models pretrained on large corpora from the same domain and using exactly the same training setup and compute budget. because they rely only on bytes to represent strings, they can be used to compare languages that use different scripts. each model takes approximately 210 hours of pretraining, resulting in more than three months of tpu computing. the models are available at https://huggingface.co/monobyte and the code used for finetuning can be found at https://github.com/lersouza/lang-agnostic . ## related work in the last years, many monolingual versions of bert @xcite @xcite @xcite @xcite @xcite and t5 @xcite @xcite have been released. the authors often claim that these versions outperform multilingual models in their languages. @xcite showed that the tokenizer plays a critical role in achieving those results. more recently, @xcite released the byt5 model, which would overcome this issue by leveraging a byte-level vocabulary. this model is pretrained on several languages and achieves great results when compared to mt5 @xcite . however, no monolingual version of byt5 has been released, which makes it difficult to conduct further investigation on the model's pretraining and cross-lingual performance that is not impacted by the tokenization in different languages. current research on the properties of pretraining often relies on monolingual models or models pretrained on artificial languages. researchers often rely on models released by other groups ( @xcite or pretrain models in a very controlled setup @xcite @xcite . in both cases, it is not clear if the results suffer from these different methodologies or undertrained models. @xcite , for instance, suggests that language contamination (i.e., a monolingual corpus composed of sentences in more languages) is the reason for monolingual zero-shot effectiveness. the same happens with models pretrained on nonnatural language corpora (chiang and yi @xcite @xcite , where the corpus size differs from the ones trained on natural language, which might affect the conclusions drawn from the experiments. we hope to bridge this gap by releasing the set of monolingual models presented in this paper. ## pretraining to train our monolingual models, we used mc4 @xcite , the same corpus used to train mt5 and byt5, which comprises 101 natural languages extracted from the common crawl web scrape. for each model, we selected only the documents written in the specific language, which was originally identified using cld3 by @xcite . the mc4 is a large corpus with different language distributions, with some being more represented than others. thus, we trimmed each pre-training corpus in approximately 65 billion utf-8 bytes for all languages. the only exception was bengali, which was trimmed on 32 billion bytes, corresponding to its total size. the pretraining was conducted on a tpu vm v3-8 using flax @xcite library and the pretraining script for t5-like span-masked language modeling available on huggingface transformers library. 2 we chose the smaller architecture of byt5, with 300 million parameters, and used similar hyper-parameters as reported on byt5 @xcite and mt5 @xcite papers. we did not experiment with larger models due to their computational cost. we set the sequence length to 1024 tokens (utf-8 bytes) and train for 1 million steps with a batch of 2 16 bytes, resulting in approximately 65 billion bytes. since we could only fit a batch of 8 examples per device, we used gradient accumulation of 128 steps to achieve a larger batch size. the learning rate starts at 10 - foot_1 and decays linearly throughout the pretraining. in comparison with the original byt5, our models are pretrained on a 16 times smaller corpus. however, our pretraining corpus is 4 times larger than bert's, which was trained on 3,300m words @xcite -i.e., approximately 16 billion characters, considering an average of 5 characters per word in english @xcite . in comparison with gpt-2, our corpus is 1.6 times larger @xcite . compared with other monolingual models research, @xcite pretrained on a corpus of approximately 77gb, according to the reported number of steps and tokens per batch. ri and tsuruoka (2022) pretrained on a corpus of 1.2gb of artificially generated text. a more extensive comparison to other work is presented in to sanity-check and validate the performance of our pretrained models, we compared their results with different baselines proposed in the literature. the first synthetic pretraining strategy, called nesting parentheses, was proposed by @xcite , which recursively generates random symbols respecting a hierarchical structure. this was proposed to evaluate how this recursion ## experiments our models are evaluated on two downstream tasks: natural language inference (nli) and question answering (qa). we use a similar setting as the in-language model, described by @xcite , where the model is finetuned and evaluated in the subset of the task that corresponds to its pretraining language. the following subsections provide details about the datasets used and the results. the monolingual models are compared to the byt5 small checkpoint. the latter is expected to perform better since it is pretrained on a much larger set of a more diverse corpus, as investigated by @xcite . for additional information on the finetuning procedure, please refer to appendix a. ## conclusion in this work, we introduced a pool of 10 tokenizerfree language models pretrained on large monolingual corpora. we demonstrated that our monolin-gual models can achieve competitive results to the multilingual byt5, although having a smaller and less diverse pretraining -single language, compared to all languages concatenated. the main goal of our work was to release a set of monolingual models that carefully follow the same pretraining methodology to support the research of monolingual language models, pretraining properties, and cross-linguality. we hope that by releasing our models, we bridge this gap of having a more controlled and reproducible setup for rigorous experiments.