A fri T e VA : Extending ?Small Data? Pretraining Approaches to Sequence-to-Sequence Models (2022)

Abstract: Pretrained language models represent the state of the art in NLP, but the successful construction of such models often requires large amounts of data and computational resources. Thus, the paucity of data for low-resource languages impedes the development of robust NLP capabilities for these languages. There has been some recent success in pretraining encoderonly models solely on a combination of lowresource African languages, exemplified by AfriBERTa. In this work, we extend the approach of “small data” pretraining to encoder– decoder models. We introduce AfriTeVa, a family of sequence-to-sequence models derived from T5 that are pretrained on 10 African languages from scratch. With a pretraining corpus of only around 1GB, we show that it is possible to achieve competitive downstream effectiveness for machine translation and text classification, compared to larger models trained on much more data. All the code and model checkpoints described in this work are publicly available at https://github.com/castorini/afriteva .

## introduction transfer learning has driven many recent advances in natural language processing, and leveraging pretrained models for downstream tasks has produced state-of-the-art results on many tasks. these results can be attributed to general-purpose knowledge that is gained when a model is pretrained on a data-rich task @xcite . this paradigm also extends to multilingual settings, where a model is pretrained on text in multiple languages and then fine-tuned for downstream tasks in those languages. some of these models, for example, mbert and xml-r @xcite , have been trained on large combination of languages comprised of high-resource and low-resource languages, amounting to many gigabytes of data. due to the effectiveness of transfer learning on downstream tasks, t5 @xcite introduced a unified framework where all nlp tasks can be framed as a text-to-text problem, enabling us to train a single model for multiple tasks. this framework is simple and effective by enabling knowledge transfer from high-resource to low-resource tasks @xcite . unlike bert-based models, which are encoder-only models, t5 and its multilingual variants such as mt5 @xcite and byt5 @xcite are encoder-decoder models that are more suited for natural language tasks involving generation. both mt5 and byt5 were trained on 100+ languages, of which only 13 were low-resource african languages, making up less than 6% of the total training data. despite the existence of 2000+ african languages @xcite , only a few of them are featured in pretraining, and thus it is unclear how effective these models generalize to those languages. the paucity of data for many african languages has been a stumbling block for developing robust nlp capabilities. however, some works have shown that it is possible to train language models with smaller amounts of data, albeit on encoderonly models. for example, @xcite obtained good results on the french question answering dataset (fquad) by pretraining on as little as 100mb of text. directly related to our present study, afriberta @xcite pretrained a roberta-based model from scratch on 10 african languages with only around 1gb of data, outperforming mbert and xlm-r on tasks in several languages. given this context, we pose the following research question: research question: can "small data" pretraining for low-resource african languages exemplified by afriberta be extended from encoder-only models to encoder-decoder models? to answer this research question, we pretrained encoder-decoder models in low-resource settings using relatively little data and evaluated our models against other models that have been pretrained on much more data. we introduce afriteva, a family of pretrained transformer-based sequenceto-sequence models derived from t5, pretrained on 10 low-resource african languages. afriteva gets its name from the fact that "v" is the roman numeral for "5", which reflects its membership in the t5 family. we pretrained from random initialization with only around 1gb of data (using the same corpus as afriberta) and evaluated our models on text classification and machine translation. to the best of our knowledge, this is the first encoderdecoder model pretrained solely on low-resource african languages. with respect to our research question, our results are suggestive but not conclusive. afriteva demonstrates better results than mt5, but falls short of other models pretrained with richer resources. however, existing experiments conflate several factors that we have not successfully untangled. nevertheless, our preliminary study sets the ground for future work. ## experimental setup following the t5 architecture @xcite , we consider 3 model sizes for afriteva: small (64m parameters), base (229m parameters), and large (745m parameters). each model is similar in configuration to their t5 counterparts. ## conclusions in this work, we present afriteva, a family of multilingual t5 models that were pretrained from scratch on 10 low-resource african languages with only around 1gb of data (with an additional variant model that includes english data in pretraining). answering our research question, we have verified that it is possible to pretrain encoder-decoder models on relatively small amounts of data, but there remain conflating factors we have yet to fully understand. although we do not reach the state of the art, our models achieve competitive results on text classification and machine translation benchmarks. we also highlight some of the limitations of evaluating sequence-to-sequence models for african languages. finally, we release code and pretrained models to drive further work in multilingual models for african languages.