M6-10T: A S HARING -D ELINKING P ARADIGM FOR E FFICIENT M ULTI -T RILLION P ARAMETER P RETRAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent expeditious developments in deep learning algorithms, distributed training, and even hardware design for large models have enabled training extreme-scale models, say GPT-3 and Switch Transformer possessing hundreds of billions or even trillions of parameters.
However, under limited resources, extreme-scale model training that requires enormous amounts of computes and memory footprint suffers from frustratingly low efﬁciency in model convergence.
In this paper, we propose a simple training strategy called “Pseudo-to-Real” for high-memoryfootprint-required large models.
Pseudo-to-Real is compatible with large models with architecture of sequential layers.
We demonstrate a practice of pretraining unprecedented 10 -trillion-parameter model, an order of magnitude larger than the state-of-the-art, on solely 512 GPUs within 10 days. Besides demonstrating the application of Pseudo-to-Real, we also provide a technique, Granular CPU ofﬂoading, to manage CPU memory for training large model and maintain high GPU utilities.
Fast training of extreme-scale models on a decent amount of resources can bring much smaller carbon footprint and contribute to greener AI.
I NTRODUCTION
The expeditious growing of foundation models on broad data highly contributes to the development of the whole deep learning and artiﬁcial intelligence community.
Foundation models with self-supervised learning on big data have become an emerging paradigm of artiﬁcial intelligence systems [3], as they mostly possess high transferability to a wide range of downstream tasks and even multiple modalities.
The scale of foundation models across domains, including natural language processing, computer vision, and cross-modality representation learning, have been growing tremendously from millions to trillions of parameters [9; 24; 25; 27; 34; 38; 4; 30; 15; 12; 17; 49; 48; 44; 46; 39] thanks to the concurrent advancement in distributed training framework [21; 28; 32; 29; 2; 41; 31] and hardware design, and these studies have made a demonstration of the neural scaling law [13].
However, the training of these transformer-based models incurs high ﬁnancial costs and even environmental damage due to the massive carbon footprint and thus training extreme-scale models under a decent amount of resources but with high efﬁciency should be a fundamental goal for both the research and industrial communities to achieve, which promotes the progress of greener AI [23; 35].
Generally there are two tracks of research in large-scale pretraining, dense models and sparse expert models respectively.
An typical case of large-scale dense models is GPT-3 [4]. It is a 175-billion-parameter transformer model trained with 10000 GPUs for months, incurring striking ﬁnancial and environmental costs.
Researchers have been looking for methods to training largescale models with a decent amount of costs. Solutions include effective management of memory with gradient and optimizer state partitioning [28] or more efﬁcient model parallelism and pipeline parallelism [38; 21; 41].
A series of following studies apply those techniques to realize fast training of 10-billion-parameter transformers with hundreds of GPUs in 1 − 2 months.
Sparse expert models with large model capacity are capable of fast training owing to the combination of data parallelism and expert parallelism [36; 15; 12; 33], and it is even accessible to train a 1-trillion-parameter transformer with no more than 500 GPUs [44].
Be there as it may, a question emerges in our mind: is it possible to train an extreme-scale model with only a decent amount of resources, e.g., training a 10-trillion-parameter model with 500 GPUs?
Such training requires large memory for parameters, including weights, gradients, and even optimizer states.
Tackling the problem requires the utilization of external memory except for GPU memory, for instance, CPU memory or even NVMe storage [32; 29].
These methods resolve the problem of high memory footprint, but instead, their extra cost is low training efﬁciency caused by the frequent swap in-and-out between memories.
In this paper, we provide a solution to training large models that require high memory footprint, and we demonstrate a successful practice of pretraining unprecedented extreme-scale models with over 10 trillion parameters, an order of magnitude larger than the previous state-of-the-arts [12; 44]. The whole pretraining was conducted on solely 512 NVIDIA-V100 GPUs and lasts around 10 days.
A simple and effective training strategy called “Pseudo-to-Real” enables sharing and delinking parameters.
This training strategy is compatible with architectures built by stacking layers with an identical structure, including dense models like GPT [24; 25; 4], BERT [9], or sparse expert models like M6 [17; 44].
It is essentially a two-stage training, where in the ﬁrst stage, we apply cross-layer parameter sharing that requires much less memory footprint for efﬁcient convergence, and in the second, we delink the parameters for better performance. It ﬁrst trains a relatively small model but with a computation graph of a large one with the utilization of cross-layer parameter sharing, and we name it “Pseudo Giant”.
Then it builds a correspondingly large model and delinks the parameters of the shared layer for second-stage model initialization.
In this way, we achieve fast convergence in the ﬁrst stage as the training costs much less memory and speeds up with large batches.
Parameter sharing that addresses the communication overhead improves training speed as well.
The second-stage training is responsible for the ﬁnal convergence for better performance.
We unlock the secret of pretraining an unprecedented extreme-scale model with over 10 trillion parameters on limited resources of 512 GPUs. Compared with the previous M6-T on around
GPUs, we do not have a signiﬁcant increase in computation resources but level up the model scale by an order of magnitude. Besides the application of the “Pseudo-to-Real” training strategy, we provide a faster ofﬂoading mechanism for both management of CPU memory for parameter storage and utility of GPUs. We successfully train the M6-10T within 10 days to reach strong performance in log perplexity evaluation and outperform the baseline M6-T.
Contributions at a glance are below:
• We illustrate the training difﬁculty of extreme-scale models on limited resources and provide a simple but effective solution called “Pseudo-to-Real”. Upstream and downstream evaluation demonstrates the effectiveness of the strategy. • We further demonstrate a successful practice of pretraining a 10-trillion-parameter model on 512 GPUs and reach an outstanding performance within 10 days.
R ELATED W ORK
Large-Scale Pretraining In recent years, pretrained language models with growing magnitudes of parameters have been proposed, keeping to raise the validated upper limit of scaling law for model capacity w.r.t the number of parameters [13].
Earlier milestones of extreme-large models come from GPT-2 [25] and Megatron-LM [38], which demonstrates that scaling the transformer model up to billions of parameters can result in improvement on language modeling benchmarks [20; 22].
Turing-NLG [34], as a successor, implements a 17-billion-parameter transformer and achieves further lower perplexity.
Similar phenomena are also observed on classiﬁcation language tasks by T model [27].
The GPT-3 [4] pushes the boundary of model scale to 1750 billion parameters and demonstrates its striking effectiveness on downstream tasks in even zero-shot settings.
Furthermore, large-scale pretraining has recently demonstrated success in other ﬁelds, including pretraining on other languages or cross-lingual pretrianing [43; 48; 46; 7], cross-modal pretraining [26; 30; 17; 10] and code generation [5].
Along with the beneﬁt from increasing the model scale, the concern of unaffordable pretraining cost in time, computation resource and energy keeps emerging [23; 3], resulting in the strong demand for more efﬁcient and greener large-scale pretraining [35].
Methods to Train Larger Models Faster Our work is about designing algorithm to train extremescale models efﬁciently. Researchers have been demonstrating different types of methods to reach this objective.
To reduce the computational cost during training, some studies introduce sparsity to the model.
Mixture-of-Experts (MoE) [36] was proposed and was reintroduced in Mesh Tensorﬂow [37].
[Figure] It shows that MoE with sparsity can signiﬁcantly improve the model scale efﬁciently without increasing computation, and the models achieved state-of-the-art performance in language modeling and machine translation. GShard [15] extended it to a tremendously large scale of 600B parameters with the sophisticated collaborated design of model architecture and demonstrated its effectiveness across over 100 languages. Similarly, Switch Transformer reached 1 .
6 trillion parameters and showed its strong performance on NLU tasks. Those models with high sparsity are computationally efﬁcient, and therefore though they possess large model capacity they still can be trained with high efﬁciency. Most other studies still focus on training dense models to validate the scaling law, and thus the emerged problem is the distributed training of large models. The most inﬂuential distributed framework should be DeepSpeed that proposed ZERO [28] that partitions optimizer states and gradients to multiple
GPU devices, and ZERO-ofﬂoad [32] as well as ZERO-Inﬁnity [29] can even ofﬂoad parameters to CPU memory and NVMe storage. The memory footprint management makes training extremely large models on limited resources possible. However, such ofﬂoading mechanisms still have some defects that they may fail to fully utilize the fast hardware.
For example, when ofﬂoading all parameters to the CPU, the GPU memory can be idle.
In this work, we tackle this issue by proposing a granular ofﬂoading mechanism that can determine which parameters to be ofﬂoaded.
In addition to reducing the amount of computation in a single iteration, another route to speedup training is to reduce the needed iterations for model convergence.
Child et al.
[6] and Xiong et al.
[42] propose to put forward the layernorm operations in transformer blocks for more stabilized and faster convergence.
You et al.
[45] employs a layer-wise adaptive optimizer to enable super-large pretraining batches.
Zhang & He [47] progressively increases the layer dropping rate in a stochastic manner which signiﬁcantly speedups pretraining.
A PPROACH
This section describes our proposed two-stage training strategy, “Pseudo-to-Real”, and shows experimental results to validate its effectiveness for training high-memory-footprint-required large models.
3. M ODEL A RCHITECTURE
Choice in model architecture depends on several factors.
First, the architecture should contain a sequence of stacking layers, as the sequential structure enables cross-layer parameter sharing.
We prefer a simple encoder or decoder architecture, instead of an encoder-decoder framework where there are cross attentions that bring extra parameters and incur difﬁculties in activation checkpointing.
Second, a model of such architecture should be compatible with different types of downstream tasks including understanding and generation, and it is even better that it can be compatible with multiple modalities.
Third, as we mention that dense models and sparse expert models are two main tracks of large-scale pretraining, we prefer the model that can ﬂexibly become whether dense or sparse expert models.
Therefore, we select M6 [17] as an option, and we evaluate the effects of our method on M of different scales and types.
M6 is built with stacking transformer layers, which includes self attention and feed-forward neural nets (FFN).
For the transformation from dense models to sparse expert models, we should only replace FFN layers with the Mixture-of-Expert (MoE) layers.
MoE consists of multiple experts, which are usually FFNs distributed on different devices.
A gating network decides the dispatching and combining behaviors of each token and thus tokens can be processed in diverse devices.
Such mechanism is a combination of data parallelism and expert parallelism, and thus it is highly efﬁcient though with large model capacity.
For the training, to realize the learning of both understanding and generation, the model is trained with text denoising and language modeling on plain text data and with image-based text denoising and image captioning on multimodal data.
The model is compatible with different types of downstream tasks and can process information of multiple modalities.
3. P SEUDO - TO -R EAL
This section demonstrates the details of “Pseudo-to-Real” two-stage training strategy that enables fast training of high-memory-footprint-required transformer models.
The strategy consists of two stages.
The ﬁrst stage trains a model with many fewer parameters but with a large computation graph (“Pseudo Giant”), and the second stage trains a corresponding large model (“Real Giant”) initialized with the delinked weights of the shared layer.
Thus we name the strategy “Pseudo-to-Real”, and the general idea is demonstrated in Figure 1.
3.2. “P SEUDO ” S TAGE : L AYER - WISE P ARAMETER S HARING
The core of “Pseudo” stage is to train a Pseudo-Giant that shares parameters across layers.
Cross-layer parameter sharing has proved successful in maintaining satisfactory performance while keeping a much smaller amount of parameters.
The method was ﬁrst mentioned in the original Transformer [40], and Dehghani and Bai
[1] both illustrated that it is effective for vanilla transformer with encoder-decoder work.
Lan et al.
[14] introduced the method to pretraining and proposed a lite
BERT with different sharing techniques.
Owing to its effectiveness, we introduce it to training an extreme-scale model, and we hypothesize that the ﬁrst-stage training can gain beneﬁts from crossparameter sharing as it can address communication overhead and it consumes much less memory footprint.
Also, as Pseudo Giant with much fewer parameters is not bounded by memory, it can be trained with large batches for acceleration.
Suppose we build an M6 model with L layers that share parameters across all layers.
The Pseudo Giant though consists of a computation graph of a L -layer transformer, its number of weight parameters and optimizer states should be 1 /L of those of the original one.
As to the gradients, we can accumulate the gradients of each layer in the backward computation process, and therefore the amount of gradients becomes 2 /L of the original one.
Such saving in memory enables much faster training with larger batches.
Also, it is capable to use fewer resources even due to lower memory consumption.
This can also be applied to MoE models, as their architecture is stacking transformer layers.
However, different from dense models, MoE models partition their weights to all devices and redistribute token representations by sparse activation.
This limits the ﬂexible usage of GPU resources.
It is available to choose different numbers of GPU devices at different stages in training dense models. In contrast, the restoration of MoE model parameters requires the same number of GPU devices used at the last stage. To tackle this problem, we take advantage of the memory efﬁciency of the Pseudo stage, and distribute more experts to a single GPU and partition experts to more GPUs.
Suppose we train a model with 512 experts at each MoE layer. It is possible to train a Pseudo Giant with only 256 GPUs, where there are 2 experts on each GPU, and then train a Real Giant on 512 GPUs where there is only 1 expert on each GPU.
3.2. “R EAL ” S TAGE : D ELINKING THE S HARED P ARAMETERS
We name a large model without cross-layer parameter sharing “Real Giant”, in comparison with Pseudo Giant.
Both Pseudo Giant and Real Giant share a computation graph, but they have different numbers of parameters. In this work, we discover how to build a connection between the two models.
Given a Pseudo Giant fully trained until convergence, we apply the delinking of cross-layer shared parameters to accelerate Real Giant training. There is no need to train a large model from scratch.
The model can start its convergence from low perplexity.
Embedding initialization can be directly restored, but the layer weights should be treated specially.
In practice, there is only one layer of weights θ shared in Pseudo Giant, and there are L layers of weights { θ 1 , θ 2 , · · · , θ L } in Real Giant.
Thanks to their identical structure, each layer of Real Giant can be initialized with θ shared .
Without further training, this model is equivalent to a fully-trained Pseudo
Giant.
This extremely simple training strategy is highly beneﬁcial for the high-memory-footprint-required large models, especially extreme-scale models like the 10-trillion-parameter M6.
While the ﬁrst stage of training saves much time for faster convergence, we can use a decent amount of computational resources in this stage as lower efﬁciency in this stage becomes acceptable.
Therefore, in the practice of training an extreme-scale M6, we apply CPU ofﬂoading to utilize CPU memory.
Therefore, we can use a limited amount of resources, e.g., 512 GPUs, to train an unprecedented 10-trillion-parameter model efﬁciently, which is an order of magnitude larger than the state-of-the-arts.
3.2. T IMING FOR S WITCHING
A question naturally emerges: when should we switch from the Pseudo stage to the Real stage?
As mentioned above, the greatest advantage of Pseudo stage for training is the signiﬁcantly faster convergence speed.
Yet the performance of Pseudo Giant is bounded by its limitation in the number of parameters.
Training Pseudo Giant until its ﬁnal convergence apparently incurs much waste of time.
In practice, we present a simple strategy to determine the training step to switch from Pseudo to Real based on the convergence speed.
During the training of the Pseudo stage, we evaluate a training step in a ﬁxed interval by attempting to transfer it into the Real stage and training for a small while (e.g., 30 minutes).
After that, we will revert the model parameters to the evaluated step and continue the training of the Pseudo stage for the same training time as the Real stage.
If the decreasing speed of loss in the Real stage surpasses that of the Pseudo stage, we determine the evaluated training step as the best switching point for the next-stage training. 3. E XPERIMENTS
In this section, we provide a series of experiments to evaluate the effectiveness of the training strategy by observing the models’ upstream and downstream performance.
3.3. E XPERIMENTAL S ETUP
We conduct experiments for pretraining and ﬁnetuning to analyze model competence in upstream and downstream tasks.
Following the classical data setup for pretraining and ﬁnetuning, we pretrain the model on BookCorpus [50] and English Wikipedia [9], which are corpora with around 16GB of plain texts.
We validate the effectiveness of “Pseudo-to-Real” training strategy by implementing a medium-size
M6 model to conduct extensive analyses on both upstream and downstream performance.
To satisfy the requirements of high memory footprint where the two-staged training can make difference in training efﬁciency, we conduct experiments on a 1.4B-parameter model, the largest model trained on a single NVIDIA V100-32GB [32].
The corresponding Pseudo Giant is a model with the same computation graph but sharing parameters across layers.
Following Radford et al.
[24] and Lewis et al.
[16], we use a vocabulary of around 50 , 000 subwords.
Each sample consists of sentences from an identical passage, and we use a sequence length of 512 and correspondingly truncate or pad the sequence.
We build an M6 model with 36 layers of transformer, [Table] whose hidden size is 1024 and intermediate size is 16384 .
As our evaluation is conducted on plain text data, we adopt the two tasks, text denoising and language modeling, for pretraining. We name the one with cross-layer parameter sharing “Pseudo” and the one without it “Real” in Table 1.
The total number of parameters of Real Giant is around 1 .
4 billion and that of its corresponding Pseudo Giant is around 90 million.
We use “P2R” referring to “Pseudo-to-Real” to represent the model trained with both “Pseudo” and “Real” stages.
“Pseudo” and “Real” refer to the models that are pretrained from scratch, in contrast with “P2R”.
Furthermore, we build an M6 model of 358 M parameters as a baseline. It has a smaller intermediate size of 4096 and consists of 24 layers.
We name it “Base” in Table 1.
Following the common practice in pretraining [9; 18], we apply AdamW optimizer [19] for optimization.
To determine the most suitable learning rate of the two stages for fast convergence, we have made some preliminary tests and ﬁnally used the peak learning rate of 2 e − 4 for “Pseudo” stage and 8 e − 5 for “Real” stage, respectively.
We use a cosine decaying mechanism for learning rate scheduling and a warm-up ratio of 0 .
For both Pseudo and Real Giant, we train them with a total batch size of 6 , 144 .
In practice, we use a micro-batch size of 32 and a gradient accumulation step of 4 , and we train our models on 48 NVIDIA-V100 GPU devices.
We create two setups for better comparison. The one is “limited budget”, where models have been trained for the same duration of time. Correspondingly, “Pseudo” has been trained for around 25 , 000 steps, Real has been trained for around 4 , 700 steps, and “P2R” has been trained for 12 , 000 steps. The other is “training for longer”, which demonstrates a wall-time performance of M6-1B for better comparison.
We compare the model performance by the downstream evaluation on language modeling and text summarization tasks, which cover capabilities including language modeling and generation. To be more speciﬁc, the downstream datasets include: • WikiText103 : a classical language modeling evaluation benchmark dataset that consists of Wikipedia articles.
We evaluate the Perplexity (PPL) of the pretrained model, which is a per-token exponential cross-entropy loss that reﬂects probability distribution over texts.
• Gigaword : a dataset for summarization which consists of around 3.8M articles and summaries, and an effective benchmark to evaluate the model’s capability in text generation for abstractive summarization.
For better comparison, we also present the other pretrained models as baselines to demonstrate that the models can achieve strong performance in downstream tasks.
Furthermore, we also compare the model quality on the time basis to reﬂect the advantage of Pseudo-to-Real training strategy. To view their training efﬁciency, we focus on their training speed on the condition that we attempt to exhaust the GPU utility. Table 1 demonstrates the training speed of the models. We report their training speed on 48 NVIDIA-V100 GPU devices with their consumed samples per second.
Obviously training Real Giant model from scratch is highly time-consuming, and Pseudo Giant training has an advantage of around 5 times of convergence speed over Real Giant training. We also observe the loss convergence of both P2R and Real Giant trained from scratch. In Figurewe present the development of pretraining language modeling loss, which is the log perplexity, on [Table]
the time basis. The log perplexity of P2R decreases much faster than that of the Real Giant, with an advantage larger than 0 . [Figure] We add other strong baseline pretrained models for the two tasks for comparison.
M6-base model matches the performance of pretrianed models, including UniLM, Megatron-LM GPT, etc.
Experimental results are consistent with our hypothesis that Pseudo-to-Real training can speed up training effectively.
In the setup of limited budget, the M6 model trained with
Pseudo-to-Real can outperform the Pseudo Giant and Real Giant in both language modeling and text generation. We also add an M6-1B model trained for a longer time to show its walltime performance on downstream tasks. However, we did not reach the best performance in Gigaword.
As it is not related to the training strategy, we leave this issue to future work for further discussion about how to ﬁnetune large models on downstream tasks.
T OWARDS A 10-T RILLION -P ARAMETER M ODEL
Previously, training large-scale models brings tons of challenges to the collaboration algorithm design, distributed training, as well as hardware design, etc.
Training a GPT-3 of 175 B parameters with a combination of data parallelism on over 500 GB of data should cost around 300 GPU-years.
Later with the emergence of partitioning on optimizer states, gradients, and even weights, GPU memory can be fully utilized without performance degradation.
Now we can even use the CPU memory or even NVMe storage to store the parameters, but we have to bear the costs of efﬁciency.
Therefore, we attempt to tackle the difﬁculty of extreme-scale model training from the perspective of algorithm design and thus we apply the aforementioned Pseudo-to-Real training strategy to train an extreme-scale model.
4. M ODEL S ETUP
We design a 10-trillion-parameter M6 model with the combination of existing methods and proposed strategies to demonstrate a case of how to train an extreme-scale model efﬁciently.
In comparison with the previous studies of trillion-parameter models [12; 44], this one is almost 10 times larger.
To efﬁciently utilize the memory, we adopt Mixture-of-Experts and we replace every FFN layer with the memory-efﬁcient MoE layers.
Notably, we remove the auxiliary loss that consumes memory
F B A W
F B A W
F
B
A
W
F nB nA nW nF n
B n
A n
W n ...
[Figure]
and demonstrates little effects on model quality, and we follow Yang et al.
[44] to apply expert prototyping for improved model quality and training stability.
To be more speciﬁc, the hidden size d model is 1024 and the intermediate size d ff is 9984 .
The number of model layers is 48 .
For each
MoE layer, there are 10240 experts distributed on multiple devices.
We use 80 prototypes of experts based on our experience in preliminary experiments.
The training batch size per GPU device is 8 .
For the settings of hardware, we implement our models on a cluster of 8-GPU workers connected by
RDMA networks with a bandwidth of 100Gb.
The CPU memory of each worker is around 750GB.
4. G RANULAR CPU OFFLOADING
To utilize CPU memory for larger models with fewer resources, we apply the Granular CPU ofﬂoading which has a higher efﬁciency compared with the conventional CPU ofﬂoading.
Previously we note that conventional ofﬂoading mechanisms ofﬂoad all parameters, which may fail to effectively utilize
GPU.
To improve the efﬁciency of CPU ofﬂoading, we propose a new CPU ofﬂoading mechanism called Granular CPU ofﬂoading.
The training process is composed of phases including “Forward (Fn)”, “Backward(Bn)” and “Apply (An)”.
Ofﬂoading all model parameters to CPU in Fn and
Bn requires loading parameters from CPU to GPU memory twice.
Activation checkpointing that brings recomputation needs the parameters loaded in Bn.
An requires the gradients ofﬂoaded from
GPU memory to CPU memory.
Assume the model parameter size is W , the above processes bring parameter movement of size 4 ∗ W .
In ofﬂoading, PCIE is the bottleneck of the whole training process.
We observe that when ofﬂoading all parameters with recomputation, the GPU memory is idle.
We can ﬁll up the GPU memory by selective ofﬂoading, leaving part of the model in GPU memory.
In this way, the model can be accelerated by reducing across-device memory copy.
In our preliminary experiment, with the setting of training a 48-layer 78B-parameter M6 model on 8 NVIDIA-V100 GPU devices, the step-time costs 89 seconds when fully ofﬂoading the parameters into CPU memory.
In comparison, granularly ofﬂoading the ﬁrst 24 layers into CPU memory and leaving the remaining 24 layers on GPU reduces the step-time to only 45 seconds.
The signiﬁcant difference in training step-time indicates that the time-cost of parameter movement between CPU and GPU will dominate the training step-time when ofﬂoading is employed, thus the granularity of ofﬂoading and the utilization of GPU should be seriously considered in extreme-scale pretraining.
In addition, ofﬂoading the whole model can result in OOM error in the CPU when the model is extremely large.
With Granular CPU ofﬂoading, we successfully implement a 10-trillion-parameter M6 model on solely 512 NVIDIA-V100 GPUs.
Furthermore, at the Pseudo stage, we can train a Pseudo Giant with a computation graph of 10 trillion parameters only with 256 GPU devices without the utilization of
CPU memory for ofﬂoading.
Thus in our practice, we train a Pseudo Giant with only 256 GPUs, and then partition the experts and redistribute them to 512 GPUs.
This saves the usage of GPU resources, which is more resource-efﬁcient and also more environmentally friendly.
[Figure]
which limits has an advantage
4. A NALYSIS
Training Efﬁciency We pretrain the M6-10T with Pseudo-to-Real training strategy for around 10 days, and we additionally train an M6-10T from scratch without the strategy for around 3 days for comparison.
The Real stage training can be facilitated without Out-of-Memory errors with the help of Granular CPU ofﬂoading, but its step-time is only around 180 s.
In contrast, the step-time of
Pseudo stage is only 14 s without the cost of ofﬂoading, which greatly boosts the training efﬁciency of M6-10T P2R. The M6-10T with P2R has been trained for 15 k steps, but the one from scratch has been trained for solely 1 . 3 k steps.
We record the log perplexity of both models trained on the
M6-Corpus on the time basis in Figure 4(a).
Results show that within the same length of time M6-10T with P2R can outperform the one trained from scratch by a large margin.
Convergence Analysis We have trained M6-10T with Pseudo-to-Real strategy for around 10 days, and the model converges to a low level of log perplexity based on the upstream evaluation on the
M6-Corpus.
For better comparison, we also show the convergence performance of the 1T-parameter
M6-T model proposed in the previous work.
As shown in Figure 4(b), the observation is consistent with our intuition that the model with a larger capacity can converge faster on the sample basis, and it should achieve the best performance on language modeling.
What leaves open is whether it can positively lead to better downstream performance concerning different types of downstream tasks.
Finetuning extremely large models should be difﬁcult and there is still much room for us to discover the potential of extreme-scale models.
However, the contribution of this work is leveling up their training efﬁciency, which can be regarded as an initial step to investigate the secrets of super large models.
C ONCLUSION AND F UTURE W ORK
Pseudo-to-Real training strategy is a simple and effective way to train large-scale models that are highly memory consuming, and it is also essential to training extremely large models with limited resources with signiﬁcantly higher training efﬁciency.
We unlock pretraining unprecedented extremescale models with 10 trillion parameters with limited resources of 512 GPUs in 10 days.
Besides the application of Pseudo-to-Real training strategy, we further provide Granular CPU ofﬂoading to enhance GPU utility while breaking the GPU memory wall with a cost in efﬁciency.
The advances take a leap towards extreme-scale model training beyond implementation on limited resources.
With only a few GPU cards, training large models with tens or hundreds of parameters has become accessible to many researchers.
We believe that this can motivate low carbon dioxide production and encourages the progress of green AI.
E THICS S TATEMENT
This work is highly concerned with large-scale language models and multimodal pretrained models.
These models have been pretrained on broad data of plain texts and image-text pairs, which might contain harmful information, such as hate speech, terrorism, pornography, etc.
We have put much efforts to remove these kinds of data in our datasets by quality evaluation on texts and images.
However, this problem cannot be eliminated and ignored, and it is common in the pretraining community.
For those models that are not trained on commonly-used public datasets, we will carefully release the model checkpoints before careful evaluation and also limit the access to avoid misconduct
R EPRODUCIBILITY S TATEMENT
This work is generally reproducible.
Following the description in Section 3, researchers can easily implement the training strategy on the codebases for pretraining, including Huggingface Transformer 1 ,
Fairseq 2 .