Zelda Rose: a tool for hassle-free training of transformer models (2023)

Abstract: Zelda Rose is a command line interface for pretraining transformer-based models. Its purpose is to enable an easy start for users interested in training these ubiquitous models, but unable or unwilling to engage with more comprehensive — but more complex — frameworks and the complex interactions between libraries for managing models, datasets and computations. Training a model requires no code on the user’s part and produce models directly compatible with the HuggingFace ecosystem, allowing quick and easy distribution and reuse. A particular care is given to lowering the cost of maintainability and future-proofing, by making the code as modular as possible and taking advantage of third-party libraries to limit ad-hoc code to the strict minimum.

## introduction since their advent in machine translation @xcite and as a mean to obtain contextual word representations @xcite , transformer models have become ubiquitous in natural language processing. the latter use in particular is almost impossible to avoid to develop state-ofthe-art nlp systems, the usual workflow being a self-supervised pretraining step using unannotated data, followed by a fine-tuning step for a downstream task, either as a module in a larger neural architecture or as an end-to-end predictor. using these models and fine-tuning them on downstream tasks has been made easier by libraries such as allennlp @xcite , fairseq @xcite , machamp (van der @xcite , @xcite . . . and most of all transformers @xcite . however, among these, only fairseq and transformers provide interfaces to pretrain them. while these libraries have considerably lowered the barrier of entry for using these models, produc-ing new ones remains an involved process. in particular training existing models on new data is not trivial, making it hard to develop models for new languages or specialty domains. this difficulty is largely due to the number of moving pieces and options required for training these models with limited resources in a reasonable time. zelda rose is meant to make pretraining transformers models as simple as possible, in particular for users who would benefit from being able to train or refine them on their own specialized domains, but who are not necessarily interested in controlling or customizing every possible aspect of the training process. this include among researchers in domains other than nlp, software engineers in the process of porting existing nlp tools to new languages, and, generally speaking, consumers of transformer models as opposed to researchers interested in improving the models themselves. it must therefore be a small tool, with a limited and clearly defined purpose, as easy to use and cheap to maintain as possible. to this end, pretraining a transformer model using zelda rose does not require writing any code (although it is easy to write code to extend its capability or customize it), but running a simple command. the configuration is done with entirely with configuration files and command line options. it can use any local datasets, models and configurations as well as refer to huggingface hub repository. it is also transparently compatible with the slurm scheduler to make it easier to run on computing clusters. ## related works to our knowledge, only two mainstream libraries providing interfaces for pretraining transformer models are fairseq @xcite and transformers @xcite (via its trainer module). both of these are complete frameworks, providing model implementations, data process-ing tools and training interfaces. while this allows for a complete customization of the architecture of the models and of the training process, it can makes it less straightforward to train a known model on new data or to get a precise sense of what is happening during training, since it requires an extensive knowledge of their often convoluted code bases. in other words, while they are essential tools to build on, they are not necessarily the most ergonomic for the need of all users. the tight coupling of their model training utilities with their library code also makes it hard to extend them, for instance to make them compatible with new hardware platforms or training techniques. to avoid these complications, some works (such as @xcite for deberta), chose to build projects entirely around the purpose of training a single model, writing ad-hoc code more or less from scratch. this obviates the need for learning all the details of a specific framework, but is much more involved in terms of engineering to design the model and training code, run them and ensure the reusability of the resulting artifacts. overall, this model would not be suitable for the use cases that we target with zelda rose. between these two extremes of being completely integrated in an end-to-end framework or to build from ad-hoc code, the choice we made for zelda rose is a loose coupling with frameworks. in practice, this means that users can train new models by providing a configuration and a dataset to a command line tool, which trains a model using only high-level interfaces of thirdparty libraries, each specialized in a specific aspect. for instance we use pytorch-lightning (falcon and the pytorch lightning team, 2019) to manage the training process, while the models implementations come from transformers, which allows to benefits from all their respective innovations while avoiding being restricted by their limitations. ## challenges the main challenge with such a library is its maintainability given a limited time budget. indeed, while the systematic reliance on third-party libraries rather than ad-hoc code as often as possible lets us benefit from their latest improvement with very little engineering code a priori, it also makes zelda rose dependent upon them for correctness and backward compatibility. this means that every new release of a dependency has to be checked with great care, as it could be introducing new bugs and it is often the case that slight but significant behavior changes go unnoticed or undocumented. undocumented behavior and implementation details in the dependencies were also a burden for the initial engineering effort and for subsequent extension of zelda rose (for instance when adding new tasks). indeed, for libraries of these sizes and providing so many functionality, the documentation does not always follow the speed of bleeding edge evolutions, and checking their source code directly is often necessary to ensure that we use their interfaces correctly. in other cases it was the lack of support for certain features required for the reproduction of existing work (such as embedding tying in the elec-tra and deberta models) that made reaching into private interfaces and monkey-patching necessary. while the dynamic nature of these libraries allow this, it makes part of our code much more brittle than we would hope for and these parts have to be checked for correctness with each new release. this regular checking of non-regressions is made harder by the nature of the tool. since train-ing neural networks is not in general deterministic and since bugs tend to manifest not as outright errors but as degradation in performances, ensuring that a modification did not introduce a bug can be challenging. indeed, it is often the case that such a degradation would not be observed on toy examples but only on the larger scale of real world examples. however, continuously running automated tests (in a continuous integration pipeline for instance) at these scales is not realistic, reducing the safety provided by tests. moreover, the complexity of the dependencies makes unit testing challenging, since it would often require mocking internals of these libraries. automated tests ran as part of the continuous integration pipeline of zelda rose are therefore limited to so-called smoke tests, which verify if the tool runs, is able to train model in a variety of configuration and produces viable output. nonregression tests, in contrast, are run manually, by training a few select models in realistic conditions. since these are much more costly, they are only ran before a new version of zelda rose is released. ## conclusion zelda rose makes possible it to pretrain transformer models with very little effort, even in complex environments. no custom code is needed to train a model on new data, which should allow more people to participate in resource development efforts. however, the modular design of this tool also makes it easy to extend and to integrate future developments in transformer models. building upon high level state-of-the-art frameworks rather than custom engineering allows to benefit from the latest innovations without too much maintenance work. it is not without downsides as it still requires some work to ensure that new releases of these dependencies do not introduce bugs and it makes the parts of zelda rose more complex to test individually. overall, however, this design choice is still a net gain. planned future development in zelda rose will focus on adding more tasks, which will be the occasion to make the design even more modular by allowing tasks to be external plugins. more efforts will also be made on the instrumentation and testing, to make the testing process more cumbersome and add as many checks as possible of the reproducibility of training results. finally, since the aim of this tool is to be useful beyond its developers, future improvements will also in a large part be guided by the requests of users, which we hope will be numerous and relevant!