nano T 5: Fast & Simple Pre-training and Fine-tuning of T5 Models with Limited Resources (2023)

Abstract: State-of-the-art language models like T5 have revolutionized the NLP landscape, but their computational demands hinder a large portion of the research community. To address this challenge, we present nanoT5, a specially-optimized PyTorch framework for efficient pre-training and fine-tuning of T5 models. Drawing on insights from optimizer differences and prioritizing efficiency, nanoT5 allows a T5-Base model to be pre-trained on a single GPU in just 16 hours, without any loss in performance. With the introduction of this open-source framework, we hope to widen the accessibility to language modelling research and cater to the community’s demand for more user-friendly T5 (Encoder-Decoder) implementations. We make our contributions, including configurations, codebase, pre-training insights, and pre-trained models, available to the public.

## introduction the transformative power of large pre-trained language models such as @xcite , t5 @xcite , and palm @xcite is undeniable. however, their massive computational requirements remain a barrier for many researchers. notably, models like t5 require extensive datasets and significant computational resources for their pre-training @xcite . furthermore, many open-source implementations lean heavily on tpu accelerators @xcite , which are not as available to the academic community as gpus. recognizing this gap, we introduce nanot5, a resource-efficient, open-source pytorch framework designed for the pre-training and fine-tuning of t5 models. inspired by pioneering efforts such as nanogpt @xcite and cramming @xcite , nanot5 uniquely concentrates on enhancing the training pipeline specifically for t5 encoder-decoder models. our framework includes optimized configurations and scripts, enabling researchers to pre-train a t5-base model with 248m parameters on a single gpu in just 16 hours. every facet, from data preprocessing and model architecture to the learning rate schedule, has been tuned for both efficiency and adaptability. with nanot5, users can seamlessly initiate model pre-training within minutes of accessing our github repository. this paper underscores two main innovations: first, we delve into the nuances between the adam and adafactor optimizer performances as detailed in @xcite , suggesting a version of adamw @xcite , augmented with matrix-wise learning rate scaling based on root mean square. this variant showcases better speed and robustness compared to the default adafactor @xcite . second, we demonstrate that t5 models trained with nanot5, housing around 250m parameters, can achieve performance akin to the publicly-available checkpoints while requiring 150x less pre-training data. our primary motivation stems from the growing demand for reproducible and tuned baselines @xcite , enabling fast and smallscale hypothesis validation in the evolving realm of large pre-trained transformers. with nanot5, we address a gap highlighted by community requests 123 , providing an approachable platform for working with t5 (encoder-decoder) architecture. to our understanding, nanot5 pioneers the effort to reproduce t5 v1.1 pre-training using pytorch, deviating from prior jax/flax implementations. we invite the community to explore our training configurations, codebase, and pre-trained models, all of which are available at https://github.com/piotrnawrot/nanot5 . ## related work the landscape of open-source repositories tailored for efficient pre-training of transformer language models is vast. notably, nanogpt @xcite sheds light on decoder-only models, while cramming @xcite homes in on the optimal pre-training of the encoder-only bert architecture @xcite . contrastingly, with nanot5, we sought to bridge the existing gap by providing a standalone research template tailored for the t5-style (encoder-decoder) models. to expedite the training process of nanot5 we incorporated various optimizations. these encompass mixed precision training @xcite , compiled runtimes @xcite , and more. additionally, we delved into the potential of efficient training methodologies such as recent optimizers @xcite , and fast attention mechanism @xcite , which are elaborated further in section 4.3. it's crucial to note that while we evaluated various efficient algorithms, we consciously opted against those, such as @xcite , that would modify the core model structure. instead, our intent with nanot5 was to cultivate a straightforward baseline for further research endeavors. the standout contribution of our work in terms of efficient training algorithms is the adamw variant, with the rms matrix scaling, which improves t5 pre-training convergence. ## methodology our validation strategy seeks to replicate the t5base pre-training outcomes detailed in @xcite and the fine-tuning results of tk-instruct on the super natural-instructions (sni) meta-dataset @xcite . ## conclusions in this study, we demonstrated the feasibility of pre-training a substantial model like t5 under resource constraints, specifically using a single a100 gpu within a 24-hour timeframe. through selection of optimization methods and configurations, we achieved results comparable to large-scale training settings. our intention in sharing the codebase, configurations, and training logs is to bridge the gap between research accessibility and computational resource limitations in the nlp domain. we invite and welcome community suggestions to further refine and enhance our approach. moving forward, we aim to enrich our codebase by incorporating additional training objectives, such as those suggested by @xcite , in hopes of further optimizing the training pipeline.