SOTASTREAM : A Streaming Approach to Machine Translation Training (2023)

Abstract: Many machine translation toolkits make use of a data preparation step wherein raw data is transformed into a tensor format that can be used directly by the trainer. This preparation step is increasingly at odds with modern research and development practices because this process produces a static, unchangeable version of the training data, making common training-time needs difficult (e.g., subword sampling), time-consuming (preprocessing with large data can take days), expensive (e.g., disk space), and cumbersome (managing experiment combinatorics). We propose an alternative approach that separates the generation of data from the consumption of that data. In this approach, there is no separate pre-processing step; data generation produces an infinite stream of permutations of the raw training data, which the trainer tensorizes and batches as it is consumed. Additionally, this data stream can be manipulated by a set of user-definable operators that provide on-the-fly modifications, such as data normalization, augmentation or filtering. We release an open-source toolkit, SOTASTREAM, that implements this approach: https://github.com/marian-nmt/sotastream. We show that it cuts training time, adds flexibility, reduces experiment management complexity, and reduces disk space, all without affecting the accuracy of the trained models.

## introduction a cumbersome component of training machine translation systems is working with large amounts of data. modern high-resource parallel datasets are often on the order of hundreds of millions of parallel sentences, and backtranslation easily doubles that @xcite . because this data is too large to fit into main memory, toolkits such as fairseq @xcite and sockeye @xcite ) make use of a preprocessing step, which transforms the training data from its raw state into a static sequence of tensors. these tensors can then be read in via an index and memory-mapped shards, allowing for quick assembly into batches at training time. while this offline preprocessing prevents data loading from becoming a bottleneck in training, it creates a number of other problems: these problems exist for construction of any model, but are exacerbated in research settings, which often explore variations of the training data. we describe an alternative that factors generation of data from the consumption of that data by the training toolkit. this view presents the training data as an infinite stream of permutations of the raw training samples. this stream is then consumed by the training toolkit, which tensorizes it on the fly, consuming data into a buffer from which it can assemble batches. this framework eliminates all the problems above: variants of the data are independent of any model; arbitrary manipulations can be applied on the fly; preprocessing time is amortized over training, which can start as soon as the first batch can be constructed;and no extra disk space or management is required. we release an open-source implementation of the proposed data generation framework called so-tastream 1 . sotastream is written in python and uses infinibatch foot_1 to provide a stream of data over permutations of data sources. it additionally provides an easily-extendable set of mixers, augmentors, and filters that allow data to be probabilistically manipulated on the fly. a particular configuration of manipulators is provided by the user in the form of a dynamically-loadable pipeline, which defines a parameterizable recipe that can be used for training. sotastream uses multiprocessing to reach high throughput levels that prevent starvation of the training toolkit. and finally, it employs a standard unix api, writing data to stdout. after presenting this framework ( § 2), we conduct a quality comparison to demonstrate that it does not reduce model quality ( § 4). we then investigate stream bandwidth under various pipelines as well as necessary toolkit consumption needs ( § 5). we conclude by demonstrating a number of use cases ( § 6). ## training from data streams the core idea underlying sotastream is to cleanly separate data generation from consumption of that data during training. the data generator is responsible for producing training samples, and the trainer consumes them. this factorization allows us to separate properties of the data (such as their sources, mixing ratios, and augmentations) from properties of training and the model (such as tensor format, batch size, and so on). the current approach relies on standard unix i/o pipes as an interface between these two pieces. however, sotastream could also be used to generate data for offline uses, or modified to allow consumption through some other api, such as a library call that returns a generator. ## experimental setup our experimental goal is to demonstrate that the many advantages of sotastream do not come at a cost in accuracy ( § 4) or speed ( § 5). we do this by comparing to a number of other data loading methods. in order to isolate the effects of changing the data loader, we conduct all of our experiments within the marian toolkit. marian does not support offline data preprocessing; instead, we compare a number of different streaming settings that cover best-case scenarios for data loading. ## quality comparison ## speed next we ask whether sotastream has a negative effect on speed. we examine speed in three settings: generation speed ( § 5.1), marian's consumption speed ( § 5.2), and total runtime ( § 5.3). ## example use cases in this section we show example use cases how so-tastream can be used to simply and easily modify data on the fly. this provides all the advantages of training for robustness without the cumbersome task of generating (and managing) data that has been preprocessed in many different forms, which are combinatorial and impose high costs on the complexity of managing training runs. ## related work to our knowledge, sotastream is novel in presenting a framework for the generation of training data as a distinct component in the model training pipeline. it emphasizes a clean separation between data generation and training, multithreading for throughput, and the use of the standard unix pipeline interface. it is not the first to propose streaming data, however. although fairseq's documentation emphasizes a preprocessing step, foot_7 , fairseq can also read and process raw data on the fly if it can be loaded completely into memory. pytorch @xcite also provides "iterable-style" datapipes for iterating over data samples, foot_8 but as far as we know, they are not widely used for machine translation training. they could, however, provide an interface to sotastream for python-based training toolkits. there are many libraries focused on data augmentation. a number of these are focused just on text augmentations, including nlpaug @xcite , textattack @xcite , and textflint @xcite . another tool is augly @xcite , a multimodal tool for text, audio, images, and video that provides robust training against adversarial perturbations. many of these libraries could be useful within sotastream's general framework. ## conclusion the data-preprocessing approach that is common in machine translation model training makes it possible to work with increasingly large datasets, but this ability does not come without costs. it is timeconsuming to copy and process data, and can be expensive to store on disk. if data is compiled with model-specific parameters that tie the data to a particular model training, it prevents or at least complicates reusability. this problem is further exacerbated by research settings where one of the experimental parameters is manipulations of the training data, since each variant (and potentially their cross-products) must be written to disk and then managed. we have described an approach that separates data generation from data consumption, and shared sotastream, an implementation that makes use of the standard unix pipeline. the requirement is that preprocessing must now be computed on the fly. our experiments show that this does not slow down training, nor does it affect the accuracy of the models trained. the approach provides flexibility, saves processing time and disk space, and simplifies experiment management.