F in GPT : Large Generative Models for a Small Language (2023)

Abstract: Large language models (LLMs) excel in many tasks in NLP and beyond, but most open models have very limited coverage of smaller languages and LLM work tends to focus on languages where nearly unlimited data is available for pretraining. In this work, we study the challenges of creating LLMs for Finnish, a language spoken by less than 0.1% of the world population. We compile an extensive dataset of Finnish combining web crawls, news, social media and eBooks. We pursue two approaches to pretrain models: 1) we train seven monolingual models from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the pretraining of the multilingual BLOOM model on a mix of its original training data and Finnish, resulting in a 176 billion parameter model we call BLUUMI. For model evaluation, we introduce FIN-bench, a version of BIG-bench with Finnish tasks. We also assess other model qualities such as toxicity and bias. Our models and tools are openly available at https://turkunlp.org/gpt3-finnish .

## introduction neural language models based on the transformer architecture @xcite have revolutionized natural language processing (nlp) in recent years, advancing the state of the art in tasks ranging from text classification to open-ended text generation. generative, decoder-only language models such as the generative pretrained transformer (gpt) @xcite series have been a particular focus of interest in part due to their multitask and few-shot capabilities @xcite . the ability of such models to implicitly learn to perform tasks that they have not been directly trained on has been considered to be closely tied to the scale of the model @xcite and, perhaps even more importantly, to the number of training tokens @xcite @xcite . most work on such models focuses on english, often entirely excluding other languages, and assumes that hundreds of billions of tokens of text are readily available for model training. in this study, we consider the challenges of introducing large generative models for finnish, a uralic language natively spoken by fewer than 6 million people. while the language is comparatively well represented in online resources relative to this number, less than 1% of texts available in e.g. wikipedia and common crawl are finnish @xcite . as the other members in the language family are either even smaller and lesser-resourced or quite distant, the resources for creating models for the language are quite limited. finnish has been represented to some degree in transformer-based models since the release of the original multilingual bert model @xcite , and a dedicated monolingual bert for the language was previously created by @xcite . also some generative models for finnish have been previously introduced by the "finnish-nlp" group foot_0 and @xcite , but as training llms is very expensive and finnish is constrained by the size of available data, models exceeding a billion parameters have been so far missing from the finnish nlp landscape. we compile a broad-coverage dataset of finnish and train monolingual models up to 13 billion parameters for 300 billion tokens (approx. 8 epochs). we also perform continued pretraining of the 176billion parameter bloom model @xcite to extend its coverage of finnish, introduce novel evaluation datasets, and assess multiple aspects of the resulting models. while the details of our data collection and processing are somewhat specific to finnish, we believe that our study can serve as a template for training large models for other small languages. ## models our models are based on the gpt architecture @xcite and we follow the pretraining approach developed for the bloom family of large multilingual language models @xcite . we train monolingual finnish models with up to 13 billion parameters from scratch, following gpt-3 @xcite in terms of the number of layers, dimensionality, and number of attention heads ( ## data we next present the sources of training data, preprocessing steps, data statistics and analysis. ## pretraining this work leverages the lumi supercomputer, 12 as of this writing the third-largest and seventh greenest in the world @xcite . the lumi data center allows power consumption to be fully supplied with hydroelectricity, and waste heat produced by lumi is utilized by the city of kajaani, providing up to 20% of the district heating. training was done on up to 192 nodes, each consisting of 4 amd instinct mi250x gpus, a single 64-core amd trento cpu and 512gb of memory. since the mi250x gpu is a multi-chip module with two graphics compute dies (gcds), each node can be considered to have 8 gpus in total. in this perspective, the training utilized up to 1536 gpus. the 64-core cpu is configured as 4 numa nodes linked to the gpus. because of a "low noise" mode used on the nodes, only 63 cores were available for training. we train our models on an adapted version of bloom's pretraining framework, megatron-deepspeed. 13 by combining features from megatron @xcite and deepspeed @xcite , the megatron-deepspeed framework can be used for training large language models with pipeline, tensor and data parallelization across gpus and compute nodes. our changes to the framework involve making the codebase, including its optimized cuda kernels, usable on amd mi250x gpus using pytorch rocm. to leverage the capabilities of mi250x, rocm enables the use of gpu matrix cores through its rocblas and miopen library implementations that, in turn, are leveraged by pytorch. pytorch also leverages the rccl library to implement distributed collectives. rccl also uses a hip port of the aws openfabrics interface (ofi) plugin foot_12 to enable communication directly through to the slingshot fabric provider for improved performance at scale. for the monolingual finnish models trained from scratch, we follow @xcite also in setting the batch size and maximum learning rate in addition to the model architecture parameters. for the continued pretraining of bloom to create the bluumi model, we retain the original bloom parameters @xcite . the pretraining parameter values are shown in figure 1 shows the loss curves for held-out validation data for the models trained from scratch, showing a stable pretraining process for all models and the expected pattern of larger models achieving lower loss. ## evaluation we next present a few-shot evaluation dataset for finnish and compare the capability of the models using this data. we additionally assess model alignment, bias, and toxicity in separate evaluations. ## discussion and conclusions in this study, we compiled an extensive dataset of finnish and created in total eight new large lan17 finnishnlp-models were left out of this evaluation as they appear to have been trained without an eos token. guage models: seven monolingual finnish models ranging from 185 million to 13 billion parameters and a multilingual 176-billion parameter model, bluumi. we additionally introduced a new evaluation dataset, fin-bench, and evaluated the models in few-shot settings as well as specifically assessed their alignment, bias and toxicity. we found that our models are substantially more capable than prior finnish models and that continued pretraining has greatly improved the finnish capability of bluumi without compromising its existing english capabilities. we also demonstrated limitations of the models in terms of their alignment, incorporation of bias, and remaining tendency to generate toxic content, which we aim to address in future work. we hope our models will serve as foundation models for finnish that can be used in research and leveraged through instruction finetuning and other alignment methods @xcite to create a range of capable tools for processing finnish text. in future work, we hope to continue our study of efficient and environmentally sustainable approaches for creating capable open foundation models for lesser-resourced languages.