GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Models
Abstract
High-concurrency asynchronous training upon parameter server (PS) architectureand high-performance synchronous training upon all-reduce (AR) architectureare the most commonly deployed distributed training modes for recommendationmodels.
Although synchronous AR training is designed to have higher trainingefficiency, asynchronous PS training would be a better choice for training speedwhen there are stragglers (slow workers) in the shared cluster, especially underlimited computing resources.
An ideal way to take full advantage of these twotraining modes is to switch between them upon the cluster status.
However, switching training modes often requires tuning hyper-parameters, which is extremelytime- and resource-consuming.
We find two obstacles to a tuning-free approach:the different distribution of the gradient values and the stale gradients from thestragglers.
This paper proposes Global Batch gradients Aggregation (GBA) over PS, which aggregates and applies gradients with the same global batch size as thesynchronous training.
A token-control process is implemented to assemble thegradients and decay the gradients with severe staleness.
We provide the convergence analysis to reveal that GBA has comparable convergence properties with thesynchronous training, and demonstrate the robustness of GBA the recommendation models against the gradient staleness.
Experiments on three industrial-scalerecommendation tasks show that GBA is an effective tuning-free approach forswitching.
Compared to the state-of-the-art derived asynchronous training, GBAachieves up to 0.2% improvement on the AUC metric, which is significant for therecommendation models.
Meanwhile, under the strained hardware resource, GBAspeeds up at least 2.4x compared to synchronous training.
Introduction
Nowadays, recommendation models with a large volume of parameters and high computationalcomplexity have become the mainstream in the deep learning communities [13].
Accelerating thetraining of these recommendation models is a trending issue and recently synchronous training uponhigh-performance computing (HPC) has dominated the training speed records [18, 16, 31].
Theresource requirements of the synchronous training upon AR are more rigorous than the asynchronoustraining upon PS [1].
In a shared training cluster with dynamic status [3], the synchronous trainingwould be retarded by a few straggling workers.
Thus, its training speed may be even much slowerthan the high-concurrency asynchronous training.
Should it be possible to switch the training mode according to the cluster status, we will haveaccess to making full use of the limited hardware resources.
Switching the training mode for aspecific model usually demands tuning of the hyper-parameters for guarantees of accuracy.
Re-tuning the hyper-parameters is common in the one-shot training workloads (e.g., the general CV or NLPworkloads)
However, it is not applicable for the continual learning or the lifelong training ofthe recommendation models [10], as tuning would be highly time- and resource-consuming.
Whenswitching the training mode of representative recommendation models, we confront three mainchallenges from our shared cluster: 1) Model accuracy may suffer from a sudden drop after switching,requiring the model to be retrained on a large amount of data to reach the comparable accuracybefore switching; 2) The distribution of gradient values is different between synchronous trainingand asynchronous training, making the models under two training modes difficult to reach the sameaccuracy by tuning the hyper-parameters;3)
The cluster status imposes staleness on the asynchronoustraining, and staleness negatively impacts the aggregation of gradients, especially for the denseparameters.
We conduct a systematic investigation of the training workloads of recommendation models to tacklethe above challenges.
It is found that when the global batch size (i.e., the actual batch size of gradientaggregation) is the same, the distribution of gradient values of asynchronous training tends to besimilar to that of synchronous training.
Besides, we notice that due to the high sparsity, the embeddingparameters in recommendation models are less frequently updated than the dense parameters, leadingto a stronger tolerance for staleness than the general CV or NLP deep learning models.
Based onthese insights, we propose Global Batch gradients Aggregation (GBA), which ensures the modelkeeps the same global batch size when switched between the synchronous and asynchronous training.
GBA is implemented by a token-control mechanism, which resorts to bounding the staleness andmaking gradient aggregation [12].
The mechanism suppresses the staleness following a stalenessdecay strategy over the token index.
The faster nodes would take more tokens without waiting,and thereby GBA trains as fast as the asynchronous mode.
Furthermore, the convergence analysisshows that GBA has comparable convergence properties with the synchronous mode, even underhigh staleness for recommendation models.
We conduct an extensive evaluation on three continuallearning of recommendation tasks.
The results reveal that GBA performs well on both accuracy andefficiency with the same hyper-parameters.
Particularly, GBA improves the AUC metric by 0.2%on average compared to the state-of-the-art training modes of asynchronous training.
Besides, GBApresents at least 2.4x speedup over the synchronous AR training in the cluster with strained hardwareresources.
To the best of our knowledge, this is the first work to approach switching between synchronous andasynchronous training without tuning the hyper-parameters.
GBA has been deployed in our sharedtraining cluster.
The tuning-free switching enables our users to dynamically change the trainingmodes between GBA and the synchronous HPC training for the continual learning tasks.
The overalltraining efficiency of these training workloads is thereby significantly improved, and the hardwareutilization within the cluster is also raised by a large margin.
Related Work
Distributed training mode .
PS [21] and AR [18] are two mainstream architectures for the trainingworkloads of recommendation models, accompanied by the asynchronous training and synchronoustraining, respectively.
Researchers are enthusiastic about the pipeline, communication, and computation optimization for the AR architecture of recommender systems [31].
Meanwhile, to improvethe training efficiency of the PS architecture, researchers propose a category of semi-synchronoustraining mode [12].
For example, Hop-BS [24] restricts the gradient updates under the boundedstaleness, and Hop-BW [24] ignores the gradients from the stragglers with the well-shuffled andredundancy data.
Recently, a category of decentralized training has been proposed in many studies toscale out the AR architecture.
Local all-reduce [25], local update [26], exponential graph [28], andmany topology-aware solutions have proven promising in the NLP and CV tasks.
However, owing tothe sparsity in recommendation models, the inconsistent parameters among workers and the droppedgradients of the scarce IDs would intolerably degrade the accuracy.
Besides, these training modeshardly consider the requirements to switch to another training mode according to the cluster status,though switching is beneficial to improve the training efficiency in the shared training clusters.
[Figure]
[Figure]
Staleness and noisy gradients .
The indeterminate or even inferior model accuracy of asynchronoustraining is mainly attributed to the staleness [7] and the noisy gradients [27] caused by the small batch.
Although prior research has pointed out that the converged giant model is less sensitive to staleness[8], staleness is still a negative factor in the accuracy of the continual recommendation training.
Manyefforts have been put into controlling staleness via Taylor expansion [32], weighted penalty [35], etc.
Recent works present a large-batch training paradigm with specially-designed optimizers to scale thegradients before updating [29], and point out that it can reach the best accuracy by merely adjustingbatch size [11, 30].
There are also attempts to change gradient aggregation strategies during theasynchronous training to achieve stable model accuracy [22].
GBA generalizes the staleness controlparadigm to the recommendation workloads by token-control mechanism, which finds the balancebetween bounding staleness and ignoring gradients.
GBA runs with the same global batch size as thesynchronous mode, ensuring the effective switching between GBA and synchronous training withouttuning hyper-parameters.
Preliminaries
[Figure]
3.1 Distributed Training of Recommendation Models
[Figure] Recommendation models usually comprise two modules: the sparse module contains the embeddinglayers with the embedding parameters , mapping the categorical IDs into numerical space; the densemodule contains the computational blocks with the dense parameters , such as attention and MLP[4, 33], to exploit the feature interactions.
The main difference between the two kinds of parametersis the occurrence ratio in each training batch.
Each training batch needs all the dense parameters,yet only a tiny amount of embedding parameters are required according to the feature IDs in thedata shard.
The latest development of recommendation models introduces high complexity and alarge volume of parameters, making distributed training essential to improve training efficiency.
Thesynchronous HPC training mode usually adopts the AR architecture, where the dense parameters arereplicated, and the embedding parameters are partitioned on each worker.
HPC should be deployed bymonopolizing a few high-performance workers and making full use of the associated resources, whichmay be retarded by the slow workers [19].
PS architecture is usually coupled with asynchronous highconcurrency training where the parameters are placed on PSs, and the workers are responsible for thecomputation.
On the one hand, the high concurrency mechanism activates the fragmentary resourcesin the training cluster by deploying hundreds of workers.
On the other hand, the asynchronoustraining brings in gradient staleness , which occurs when the gradient is calculated based on theparameters of an old version and applied to the parameters of a new version.
3.2 Observations and Insights within a Shared Training Cluster
We investigate the training workloads of recommendation models from a shared training cluster toobserve the obstacles and necessities of switching training modes.
[Figure] [Figure]
Observation 1: Cluster status determines the performance of training modes.
Figure 1 illustratesthe average CPU utilization within a real shared cluster, and the corresponding samples/queries persecond (QPS) of a YouTubeDNN [6] model by the synchronous and asynchronous training mode.
Theutilization and QPS are normalized by their maximal value, respectively.
When the cluster is relativelyvacant, models trained in the synchronous mode can fully occupy the hardware resources, satisfying
HPC conditions and presenting high efficiency.
When there are plenty of heterogeneous workloadsin the cluster, slow workers dominate the training speed, making the asynchronous training moderun much faster than the synchronous mode.
We also implement two approaches of local all-reduce 1 .
Since the status of each device in the cluster is constantly changing, the local all-reduce-based modewould not work well when confronting resource shortages.
Observation 2: Directly switching training mode brings sudden drop on accuracy.
We run
DeepFM [23] over Criteo-4GB [15] (few parameters, fast convergence) and YouTubeDNN on Privatedataset (trillions of parameters, slow convergence) in the shared cluster.
We tune the hyper-parametersfrom scratch for the best model accuracy of both asynchronous and synchronous mode, and denotethe two sets of hyper-parameters as set A and set S , respectively.
After training in one training mode,we evaluate the tendency of the training AUC after switching to the other training mode with set Aor set S .
Figure 2 illustrates that after switching from the synchronous mode to the asynchronousmode, the AUC encounters sudden drop and even decreases to 0.5.
The AUC drop also appears in theopposite-side switching, indicating that this condition is irrelevant to whether the model had beenconverged.
These observations imply that directly switching the training mode requires heavy effortin re-tuning the hyper-parameter.
Inherently, training modes would lead to different convergenceor minima owing to the difference in batch size, learning rate and many other factors, which havealready received in-depth theoretical research [2, 17].
We provide theoretical analysis to explain thesudden drop in Appendix D.
We then probe into the insights from asynchronously training recommendation models.
Insight 1: Distribution of the gradient values is related to the aggregated batch size.
We attemptto investigate the reason for observation 2 from the gradient aspect.
We implement asynchronousbulk synchronous parallel (BSP) on the YouTubeDNN recommendation task, which asynchronouslyaggregates K gradients from workers before applying the values to the parameters.
Here, we set
K to 100, the same as the number of workers.
Besides, we compare the synchronous trainingin 6.4K local batch size (64 workers).
Figure 3 plots the distribution of the L2-norm of gradientvalues from the synchronous training and BSP with various local batch sizes.
It is evident that thebatch size determines the mean and variance of the distribution.
The distribution of BSP resemblessynchronous training when the aggregation size is similar (i.e., BSP-4K).
The result suggests that thesame aggregation size could lead to a similar distribution of gradient values.
However, there is still agap in model accuracy after equalizing the global batch size between the asynchronous training andthe synchronous training, mainly induced by gradient staleness.
PS
Worker
Worker 𝑵 𝒂 − 𝟏 Worker 𝟐 Worker
PS …… 2 1 1 … ∇ ∇ ∇ … ∇
Gradient Buffer 𝑀 − 1 0 token gradient(partition) 1 2 … 2 1 1 … ∇ ∇ ∇ … ∇
Gradient Buffer 𝑀 −1 0 token 1 2 … 0 0 … 0 1 1 … 1 2 2 … 2 … 𝐝
𝐝
𝐝… 𝐝 𝑄−
Token List (Size 𝑄 )
Data List (Size 𝑄 ) worker sends token and gradient to PS data and token are sent to worker 𝑀 gradients in gradient buffer would be aggregated
Size 𝑀
Size 𝑀 Size 𝑀
Dense Parameters
S t a l e n e ss 2 ∇ 1 ∇ 1 ∇ … … 2 ∇ …
Embedding Parameters
Non-Update Update gradient(partition)
token gradient Insight 2: The gradient staleness imposes different impact on the embedding parameters andthe dense parameters. Due to the skewed distribution, most IDs would merely appear in a smallnumber of batches, as depicted in Fig. It means that in the recommendation models, only a tinyportion of IDs would be involved in every single batch, and the embedding parameters are lessfrequently updated than the dense parameters.
Therefore, the embedding parameters tend to be morerobust on the gradient staleness than the dense parameters (for example, considering a worker intraining, there could be five updates for the dense parameters, yet only two updates for the embeddingof a specific ID).
Global Batch based Gradients Aggregation
4.1 Training Recommendation Models with GBA
Switching the distributed training mode for recommendation models should get rid of tuning thehyper-parameters.
We introduce the concept of global batch size , which is defined as the actual batchsize when gradients are aggregated and applied, and propose GBA for the tuning-free switching.
Wedenote the local batch size , i.e., the actual batch size on each worker and the number of workers, as
B s and N s for the synchronous training, B a and N a for the asynchronous training.
Then, the globalbatch size in synchronous training, denoted by G s , can be calculated as B s × N s .
Following Insight1, GBA remains the global batch size unchanged when we switch the distributed training modelfrom synchronous training to asynchronous training.
For each step, all the dense parameters wouldbe updated, and only a small number of embedding parameters would be updated.
Then the denseparameters and embedding parameters obtain different gradient staleness during training.
Hence,we define the data staleness as the unified staleness in training recommendation models.
The datastaleness describes the gap between the global step when the worker begins to ingest a data batchand the global step when the calculated gradient is applied.
Obviously, the data staleness in thesynchronous training mode is constantly zero.
Based on data staleness, we implement GBA by atoken-control mechanism on the PS architecture to cope with the sparsity and the dynamic clusterstatus.
Figure 5 illustrates the architecture of the proposed token-control mechanism.
Over the canonical PS,we prepare a queue called data list to arrange the data (addresses) by batches.
Given a dataset D ,suppose we can split it into Q batches of size B a , denoted by D =
( d 0 , d 1 , . . .
, d Q − 1 ) .
Meanwhile,we establish another queue called token list to yield the token of each individual batch.
The tokenlist contains Q tokens, denoted by ( t 0 , t 1 , . .
, t Q − 1 ) , each attached to one batch in the data list toindicate the global step when this batch is sent to a worker.
The token value starts from zero, andeach token value repeats M times in the token list.
Here, M is the number of batches we use toaggregate gradients.
Under this setting, we can deduce that there will be K =
⌈ QM ⌉ gradient updates during the training.
Then we set t i =
⌊ iK ⌋ , ∀ i ∈ { 0 , 1 , . .
, Q − 1 } to ensure that the token list yieldsthe token value in ascending order.
Apart from the two queues, we also employ a gradient buffer toreceive the gradients calculated by the workers with the corresponding tokens of the gradients.
Tobe consistent with the tokens, the capacity of the gradient buffer is set to M , and therefore the PSswould aggregate M gradients before applying them to the variables.
Note that each PS maintains anindividual gradient buffer to deal with the gradients of the corresponding partitions of the variables.
During the training process, a worker would pull the parameters from PS, a batch from the datalist, and a token from the token list simultaneously before ingesting the data and computing thegradient locally.
When a worker completes calculating the gradient of a batch, the gradient and thecorresponding token are sent to the gradient buffer on PS.
Then, the worker immediately proceedsto work on the next batch.
In this way, the fast workers can keep working without waiting for theslow ones.
When the gradient buffer reaches the capacity of M pairs of gradients and tokens, allthe gradients are aggregated to apply once, and at the same time, the buffer will be cleared.
This iswhat we call finishing a global step, and thereby the global batch size in GBA can be calculated as
G a
=
B a × M .
According to the design, we aim to keep the global batch size consistent in switching,that is, G s =
G a .
Hence, we can set the size of the gradient buffer to be M
= B s × N s B a .
We would use
M workers in GBA, i.e., N a = M , to avoid the intrinsic gradient staleness led by the inconsistencybetween the number of workers and the number of batches to aggregate.
At the update of global step k , denote τ ( m, k ) the m -th token in the gradient buffer.
When weaggregate the gradients in the gradient buffer, we shall decay the gradients that suffer from severestaleness.
GBA could employ different staleness decay strategies to mitigate the negative impactfrom the staleness according to the token index, and in this work we define it as:
[Equation]
where ι is the threshold of tolerance.
If f ( τ ( m, k ) , k ) = 0 , we exclude the m -th gradient in the bufferdue to the severe staleness; otherwise, we aggregate the gradient.
As we can see, tokens help identifywhether the corresponding gradients are stale and how many stale steps are behind the current globalstep.
In this case, although the token is designed over the data staleness, the negative impact from thecanonical gradient staleness can also be mitigated.
4.2 Convergence Analysis
We have seen much research on the convergence analysis of the synchronous and asynchronoustraining.
Following the assumptions and convergence analysis in Dutta et al.
[7], the expectation oferror after k steps of gradient updates in the synchronous training can be deduced by:
[Equation]
where w k denotes the parameter in step k , η denotes learning rate, L is the Lipschitz constant and σdenotes the variance of gradients.
F ( w ) is the empirical risk function that is strongly convex withparameter c .
E [ F ( w k )] − F ∗ is the expected gap of the risk function from its optimal value, used asthe error after k steps.
As mentioned in Eqn.
(2), The first term in the right, i.e. ηLσ 2 2 c ( N s B s ) , would bethe error floor, and (1 − ηc ) is the decay rate.
The proposed GBA is derived upon the asynchronousgradient aggregation.
We assume that, for some γ ≤ 1 ,
[Equation]
Here, γ is a measure of gradients impact induced by the staleness; smaller value of γ indicates thatstaleness makes less accuracy deterioration of the model.
Besides, ζ indicates the average probabilitythat any parameter in the model would be both updated in step k and step τ ( m, k ) .
Intuitively, ζwould be far below 1 in the recommendation models due to the strong sparsity.
Then, the error of
GBA after k steps of aggregated updates would become (Appendix A presents the proof):
[Equation]
where γ ′ = 1 − γ + p 0 2 and p 0 is a lower bound on the conditional probability that the token equals tothe global step, i.e., τ ( m, k ) =
k .
Equation (4) proves the convergence of GBA.
Considering the errorfloors of Eqn.
(2) and Eqn.
(4), M × B a should be set close to N s × B s to make GBA tuning-free.
Itis exactly the global batch size we use in GBA, consistent with our main idea of keeping global batchsize unchanged.
Recall that with the embedding parameters, ζ < 1 makes γ lower than the trainingof general CV or NLP models.
Consequently, the error floor remains low in GBA.
Evaluation
5.1 Settings
[Table]
We conduct systematical evaluations to examine the performance of GBA and make a fine-grainedanalysis.
The evaluations involve three industrial-scale recommendation tasks: 1) On the Criteo-1TBdataset [14], we implement DeepFM, where the hyper-parameters on Criteo-4GB (AUC 0.8043)are utilized; 2)
On the Alimama dataset [9], we implement DIEN [34] and use the recommendedhyper-parameters in the original design; 3) On the Private dataset, we implement YouTubeDNN, andwe tune the best hyper-parameters.
The models are implemented in DeepRec [5] with the expandable
HashTables.
Detailed information on the dataset and the models are listed in Tab. 1.
We imitate thecontinual training without changing the hyper-parameters to the models, and ensure a similar clusterstatus for all evaluations.
Inheriting from a pre-trained checkpoint, we repeatedly train on the dataof every day and evaluate the data of the subsequent day.
The training cluster is equipped with a
Tesla-V100S GPU and Skylake CPU.
We focus on AUC as the accuracy metric and global/local QPS(QPS of all/single workers) as the efficiency metric.
We select several state-of-the-art PS-based training modes: Bounded staleness (Hop-BS) restrictsthe maximal differences of gradient version between the fastest and the slowest workers, controlledby b 1 ; Bulk synchronous parallel (BSP) aggregates a pre-set number b 2 of gradients when applyinggradients to the parameters, regardless of the gradient version; Backup worker (Hop-BW) ignoresthe pre-set number b 3 of gradients from the slowest workers during each gradient aggregation.
Weenumerate the specialized hyper-parameters of each training mode and record the statistics whenreaching its best AUC.
5.2 Performance of Training Modes
We first examine the performance of GBA.
Figure 6(a-c) records the AUC tendencies after switchingfrom synchronous to the other training modes over the three recommendation tasks.
We mainly focuson the AUC at the first day and the last day, as well as the average AUC scores throughout the datasets.
Although Hop-BW eliminates staleness, the ignorance of a large volume of data makes it performthe worst (also taken as evidence why we tend not to use local all-reduce in training recommendersystems).
The manipulation of global batch size contributes to the best performance on both sides ofswitching.
Compared to the best baselines (i.e., Hop-BW), GBA improves AUC by at least 0.2% onaverage over the three datasets, which has the potential to increase by 1% revenue in the real-worldbusiness.
Meanwhile, after switching from synchronous training, GBA obtains immediate good [Figure] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] accuracy (AUC at the first day), while there are explicit re-convergence on the other training modes,as depicted in Figure 6(g). Figure 6(d-f,h) illustrates the AUCs of these training modes after switching to synchronous training.
We can see that GBA tends to obtain at least equal accuracy to the continuous synchronoustraining without switching.
On the contrary, the models inherited from the other baselines requireconsuming more data to reach the desired accuracy of the synchronous training.
The tendency ofthe AUC gaps between the synchronous training and the compared training modes reflects that theparameters trained by GBA are the most compatible with the synchronous training.
It verifies thatswitching from GBA to synchronous training is also tuning-free.
We collect metrics of the training efficiency during the above experiments, and report their global
QPS in Tab. 2.
The results reveal that GBA performs similarly to the asynchronous training.
Although
Hop-BS works better than BSP and Hop-BW in accuracy, it struggles to deal with the slow workers.
It indicates that when facing a resource shortage in the shared cluster, GBA can provide similaraccuracy with synchronous training mode, while running as fast as the asynchronous training mode.
5.3 Fine-grained Analysis
We further probe into the performance of GBA.
Here, we take the recommendation task on the Privatedataset (the most complex model) as an example, switching from the synchronous mode to GBA.
We first analyze how the cluster status affects the performance of GBA.
The experiments are repeatedin the cluster during different periods of a day.
We collect AUC, QPS, average gradient staleness onthe dense parameters (for fair comparison among the baselines), and the number of excluded batches,as shown in Tab. 3.
From the results, we can infer that GBA properly finds the balance betweenthe staleness and the excluded data (as defined in Eqn.
(1)), i.e., excluding fewer data comparedto Hop-BW and suppressing the staleness to the same level of Hop-BS.
GBA also shows strongrobustness on the dynamic cluster status, obtaining stable performance on AUC.
[Table]
[Figure] [Figure]
We then examine the impact of the batch sizes in GBA.
Figure 7 depicts the average AUC score andthe global QPS when we modify the local batch size (the number of workers is thereby changed)and keep the global batch size unchanged.
Considering the hardware limitation on worker and thecommunication overhead on PS, we vary the number of workers from 100 to 800.
We can see asteady state of the AUC score (i e., absolute difference less than 10 − 4 ), while the training achievesa significant efficiency boost when using more workers.
It can thereby be inferred that GBA has agood capability of scaling out.
Besides, we fix the number of workers to 400 and change the localbatch size for each worker, which means that the global batch size would differ.
As shown in Fig.
8,the inconsistent global batch size with the synchronous training makes the training encounter lower AUC scores after switching.
Although the larger global batch size may have the potential to achievebetter accuracy (i.e., owing to the stable and accurate gradient), the experiment indicates the modelwould hardly reach its best accuracy without tuning.
These results verify that using the same globalbatch size in GBA as in the synchronous training is necessary to get rid of tuning when switching thetraining mode.
Conclusion
A tuning-free switching approach is demanded to take full advantage of the synchronous andasynchronous training, which can improve the training efficiency in the shared cluster.
We raiseinsights from the investigation over the production training workloads that the inconsistent batch sizeand the gradient staleness are two main reasons to fail the switching regarding the model accuracy.
Then GBA training mode is proposed for asynchronously training recommendation models viaaggregating gradients by the global batch size.
GBA enables switching between synchronous trainingand asynchronous training of the continual learning tasks with the accuracy and efficiency guarantees.
With GBA, users can freely switch the training modes according to the status of the shared trainingclusters, without tuning hyper-parameters.
GBA is implemented through a token-control mechanismto ensure that the faster worker should contribute more gradients to the aggregation while the gradientsfrom the straggling workers would be decayed.
Evaluations of three representative continual trainingtasks of recommender systems reveal that GBA achieves similar accuracy with the synchronoustraining, while resembling the efficiency of the canonical asynchronous training.
Currently, GBArequires the users to select the training mode according to their own judgment on the cluster status.
Inthe future, we will attempt to make GBA be adaptive to the cluster status. The guidelines of automaticswitching would be derived from more analyses upon the training trace logs.
It could be formulatedas an optimization problem under many control factors including but not limited to the overall QPS,training cost, and task scheduling with priority.
Paper Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?Yes.
(b) Have you read the ethics review guidelines and ensured that your paper conforms to them?Yes.
(c) Did you discuss any potential negative societal impacts of your work?Not applicable.(d)
Did you describe the limitations of your work?
Yes.
We mentioned in the conclusion that the users have to manually select the training mode basedon their own judgement of the status of the training cluster.
If you are including theoretical results...
Did you state the full set of assumptions of all theoretical results?
Yes.
Did you include complete proofs of all theoretical results?
Yes.
The theories in this paper can be deduced in a straightforward manner.
We provide the fullproofs in the appendix.
If you ran experiments...
Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
No, because the code is currently proprietary.
We are working on making pull request to integrate our
GBASyncReplicasOptimizer into some open-source training frameworks.
Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
Yes, as listed in Table 1.
Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
No.
The status of the training cluster and the asynchronous training manner introduce the randomnessto the accuracy.
The experiments were conducted on data of many days in the continual learning toshow that the results are convincing.
Did you include the amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
Yes, as listed in Table 1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
If your work uses
Not applicable.
We provide the URLs of three out of the four datasets (Criteo-4GB, Criteo-1TB,
Alimama, Private) used in this paper and the last visited time.
Did you include any new assets either in the supplemental material or as a URL?
No.
Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Not applicable.
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Not applicable.
The data used in this paper does not contain personally identifiable information oroffensive content.
If you used crowdsourcing or conducted research with human subjects...
Not applicable.