Large-batch Optimization for Dense Visual Predictions
Anonymous Author(s)
AffiliationAddress email
Abstract
Training a large-scale deep neural network in a large-scale dataset is challenging and time-consuming.
The recent breakthrough of large-batch optimizationis a promising way to tackle this challenge.
However, although the current advanced algorithms such as LARS and LAMB succeed in classification models,the complicated pipelines of dense visual predictions such as object detectionand segmentation still suffer from the heavy performance drop in the large-batchtraining regime.
To address this challenge, we propose a simple yet effectivealgorithm, named Adaptive Gradient Variance Modulator (AGVM), which cantrain dense visual predictors with very large batch size, enabling several benefitsmore appealing than prior arts.
Firstly, AGVM can align the gradient variancesbetween different modules in the dense visual predictors, such as backbone, featurepyramid network (FPN), detection, and segmentation heads.
We show that trainingwith a large batch size can fail with the gradient variances misaligned among them,which is a phenomenon primarily overlooked in previous work.
Secondly, AGVMis a plug-and-play module that generalizes well to many different architectures( e.g., CNNs and Transformers) and different tasks ( e.g., object detection, instancesegmentation, semantic segmentation, and panoptic segmentation).
It is also compatible with different optimizers ( e.g., SGD and AdamW).
Thirdly, a theoreticalanalysis of AGVM is provided.
Extensive experiments on the COCO and ADE20Kdatasets demonstrate the superiority of AGVM.
For example, it scales the batchsize from 1056 (the recent best) to 10k for object detection. AGVM can train Faster R-CNN+ResNet50 in 4 minutes without losing performance.
It enables training anobject detector with one billion parameters in just 3.5 hours, reducing the trainingtime by 20.9 × , whilst achieving 62.2 mAP on COCO.
The deliverables are releasedat https://anonymized-agvm.github.io/
Introduction
The recent successes in many tasks of dense visual predictions rely on the large-scale datasets [1, 2, 3],the increase of computational power ( e.g., GPUs), and the parallel training paradigm with largesample batches.
Sufficient computational resource enables large-batch training, greatly reducing thetraining time [4].
However, although simply scaling the batch size allows fewer iterations to updatethe parameters of deep neural networks, it often leads to dramatic drop of generalization performance[5, 6, 7].
To reduce the generalization gap in the large-batch training paradigm, LARS [8] scales the batchsize of a plain ResNet50 from 8k to 32k without losing accuracy, enabling to train an imageclassification model on ImageNet in a few minutes.
However, different from the plain networkarchitectures in ImageNet classification, many tasks of dense visual predictions, such as objectdetection [9, 10, 11, 12, 13] and segmentation [14, 15, 16, 17], are solved by more complicatedpipelines, which consist of multiple different modules, such as region proposal network (RPN) [9],
[Figure]
feature pyramid network (FPN) [18], detection head, and segmentation head.
Nevertheless, therecent advanced large-batch optimization methods such as LARS [8] and LAMB [6] are typicallynot sufficient to achieve good generalization performance in dense visual predictions.
The longtraining time of dense predictors greatly limits the researchers from making full use of the increasingcomputational power and large-scale datasets.
To address the above challenge, we present a novel large-batch training algorithm, named Adaptive
Gradient Variance Modulator (AGVM), which can train different complicated dense predictors withvery large batch size, significantly reducing their training time while maintaining the generalizationperformance.
The design of AGVM is motivated by a training phenomenon overlooked in priorarts.
We call it gradient variance misalignment, which would present when a visual dense predictionpipeline contains many different modules and is trained with a large mini-batch, where differentmodules ( e.g., backbone, RPN, FPN, and heads) can have different gradient variance magnitudes,impeding the generalization ability.
As shown in the first row of Fig.1, where Mask R-CNN [14] with ResNet50 [19] as the backbone istrained using different batch sizes, we compare the gradient variances of different network modules,including backbone, FPN, RPN, detection head, and mask head.
We see that when the batch sizeis small ( 32 in the first figure), the gradient variances of different network modules are similarthroughout the training process.
When the batch size increases from 256 to 1024 ( 2 nd ∼ 4 thfigures), the gradient variances misalign in different modules whose variance gap enlarges duringtraining.
Training fails when batch size equals 1024.
More importantly, the gradient variances havesignificantly smaller values in the RPN, FPN, detection head, and mask head compared to that in thebackbone, and their gradient variances change sharply in the late stage of training (two figures inthe middle).
We find that such misalignment undesirably burdens the large-batch training, leading tosevere performance drop and even training failure.
More observations on various visual tasks andnetworks can be found in Appendix A.2.
The above empirical analysis naturally inspires us to design a simple yet effective method AGVM fortraining dense visual predictors with multiple modules using very large batch size.
AGVM directly modulates the misaligned variance of gradient, making it consistent between different networkmodules throughout training.
As shown in the second row of Fig.1, AGVM significantly outperformsthe recent approaches of large-batch training in four different visual prediction tasks with variousbatch sizes from 32 to 2048.
For example, AGVM enables us to train an object detector with a hugebatch size 1536 (where prior arts may fail), reducing training time by more than 35 × compared tothe regular training setup.
This work makes three main contributions .
Firstly , we carefully design AGVM, which to ourknowledge, is the first large-batch optimization method for various dense prediction networks andtasks.
We evaluate AGVM in different architectures ( e.g., CNNs and Transformers), solvers ( e.g., SGDand AdamW), and tasks ( e.g., object detection, instance segmentation, semantic segmentation, andpanoptic segmentation).
Secondly , we provide a convergence analysis of AGVM, which converges toa stable point in a general non-convex optimization setting.
We also conduct an empirical analysis thatreveals an important insight: the inconsistency of effective batch size between different modules wouldaggravate the gradient variance misalignment when batch size is large, leading to performance dropand even training failure.
We believe this insight may facilitate future research for large-scale trainingof complicated vision systems.
Thirdly , extensive experiments are conducted to evaluate AGVM,which achieves many new state-of-the-art performances on large-batch training.
For example, AGVMscales the batch size of object detection from 1056 to 10k, while maintaining the generalizationperformance.
In particular, it enables training of the widely-used Faster R-CNN+ResNet50 within 4minutes without performance drop.
More importantly, AGVM can train a detector with one billionparameters within just 3.5 hours, which reduces the training time by 20.9 × , while achieving atop-ranking mAP 62.2 on the COCO dataset.
Codes and models will be released.
Preliminary and Notation
Let S =
{ ( x i , y i ) } ni =1 denote a dataset with n training samples, where x i and y i represent a datapoint and its label respectively.
We can estimate the value of a loss function L : R d → R usinga mini-batch of samples that are randomly sampled, and obtain l ( w t ) = 1 b (cid:80) j ∈ S t L ( w t , ( x j , y j )) ,where S t denotes the mini-batch at the t -th iteration with batch size | S t | =
b and w t represents theparameters of a deep neural network.
We can apply stochastic gradient descent (SGD), one of themost representative algorithms, to update the parameters w t .
The SGD update equation with learningrate η t is:
[Equation]
where ∇ l ( w t ) represents the gradient of the loss function with respect to w t .
Layerwise Scaling Ratio.
In large-batch training, You et al.
[8] observe that the ratio between thenorm of the layer weights and the norm of the gradients is unstable ( i.e., oscillate a lot), leading totraining failure.
You et al.
[8] present the LARS algorithm, which adopts a layerwise scaling ratio,∥ w ( i ) t ∥ / ∥∇ l ( w ( i ) t )+ λw ( i ) t ∥ , to modify the magnitude of the gradient of the i -th layer ∇ l ( w ( i ) t ) , wherew ( i ) t and λ indicate the parameters of the i -th layer and the weight decay coefficient, respectively.
Furthermore, LAMB [6] improves LARS by combining the AdamW optimizer with the layerwisescaling ratio.
It can be formulated as r t = m t / √ v t + ϵ , where m t = β 1 m t − 1 + (1 − β 1 ) ∇ l ( w t )and v t = β 2 v t − 1 + (1 − β 2 ) ∇ l ( w t ) 2 .
The layerwise scaling ratio of LAMB can be computed by∥ w ( i ) t ∥ /
∥ r ( i ) t + λw ( i ) t ∥ .
Sharpness-aware Minimization.
Large-batch training often converges to a sharp local minima,resulting in undesired generalization performance.
The sharpness-aware minimization (SAM) [20]algorithm explicitly penalizes the sharp minima and finds the parameters whose neighbors (in anl p -ball) have low training loss function values using the following objective function:
[Equation]
To solve the above equation, SAM applies one-step gradient ascent to determine ϵ =ρ ∇ l ( w t ) / ∥∇ l ( w t ) ∥ .
Its gradient is then approximated by ∇ l SAM ( w t )
≈ ∇ l ( w t ) | w t + ϵ .
However, SAM involves two sequential gradient computations at each iteration and thus doubles thecomputational cost.
Gradient Variance Estimation.
Qin et al.
[21] utilize the cosine similarity between two aggregatedgradients from the replicas in a distributed training system, to estimate the gradient variance between
SGD and GD efficiently.
Specifically, we can compute the gradient for each sample in the t -thmini-batch S t of batch size b , denoted by r 1 ,t , ..., r j,t , ..., r b,t .
We have ∇ l ( w t ) = 1 b (cid:80) bj =1 r j,t .
Wesplit the above gradients into two groups and average each group, obtaining G t, 1 = 2 b (cid:80) b 2 j =1 r 2 j − 1 ,tand G t, 2 = 2 b (cid:80) b 2 j =1 r 2 j,t , respectively.
Then the gradient variance can be measured by Φ t =1 − cos( G t, 1 , G t, 2 ) , where cos( · , · ) is the cosine similarity function.
Our Approach
Our goal is to perform large-batch training for dense visual predictors with many different networkmodules.
As illustrated in Fig.1, the inconsistency of gradient variances among different modulesneed to be modulated.
Gradient Variance across Modules.
We derive an updated (considering learning rate) gradientvariance to delve into the difference of network modules in complicated dense visual predictionpipelines.
The updated gradient variance of the i -th network module at the t -th iteration can beformulated as:
[Equation]
where n and b are the number of training samples and the mini-batch size, respectively.
η t isthe learning rate.
g ( i ) t indicates the gradient of the i -th network module.
G ( i ) t, 1 and G ( i ) t, 2 are twogroups of the gradient estimation as discussed above.
Since each entry in the vector g ( i ) t could beassumed i.i.d.
in a massive dataset following [21, 22], Φ ( i ) t is thus proportional to the above updatedgradient variance.
At each training iteration, we can approximate the updated gradient variance byΦ ( i ) t
= η 2 t (1 − cos( G ( i ) t, 1 , G ( i ) t, 2 )) .
Note that Φ ( i ) t for i -th module has been normalized by the numberof parameters, so Φ ( i ) t of different modules are comparable.
For consistency of presentation, westill call Φ ( i ) t gradient variance, which enables us to estimate the gradient variance of each networkmodule at each training iteration.
More discussions can be found in Appendix A.1.
Adaptive Gradient Variance Modulator (AGVM).
Let M be a set of modules in a complicateddense prediction pipeline, where M has h different modules.
At the t -th iteration, we have a set oflearning rates, { ˆ η ( i ) t | i ∈ { 1 , 2 , ..., h }} , corresponding to different modules.
We treat the Backbone( i = 1 ) as the anchor and modulate other modules making their gradient variances consistent with the
Backbone .
Specifically, we adjust the module learning rates ˆ η ( i ) t by using the ratio between Φ (1)
t andΦ ( i ) t .
The update rule for each network module can be written as:
[Equation]
ˆ
[Equation]
where η t is the global learning rate.
However, simply adjusting the learning rates on-the-fly wouldeasily yield training failure due to the transitory large variance ratio that impedes the optimization.
We propose a momentum update to address this problem.
Let α ∈ [0 , 1) be a momentum coefficient,we have:
[Equation]
5) which can reduce the influence of unstable variance.
Note that we update µ ( i ) t each τ iterations.
Discussion on Momentum and Weight Decay.
In practice, the weight decay is widely used as aregularizer and is tightly coupled with the learning rate and the momentum.
For instance, the gradientg ( i ) t will be replaced by the momentum, such as m ( i ) t
= β 1 m ( i ) t − 1 + (1 − β 1 )( g ( i ) t + λw ( i ) t ) [6, 23],where β 1 and λ indicate the momentum coefficient and the weight decay coefficient, respectively.
We observe that it’s also important to modulate the learning rate by Eq.(4) when weight decay ispresented.
In addition, since the above m t is a momentum-based moving average of ( g ( i ) t + λw ( i ) t ) ,we can directly apply ˆ η ( i ) t onto m ( i ) t .
[Table]
Extensions to Different Optimization Algorithms.
AGVM can be easily embedded into differentoptimization algorithms such as SGD and AdamW.
We demonstrate the details in Appendix A.6:
Alg.1 and Alg.2, respectively.
They can be easily implemented using the popular deep learningplatform e.g., PyTorch [24]
[Table]
Discussion on Convergence Rate.
With AGVM, the SGD and the AdamW optimizers still haveappealing convergence properties in the general non-convex settings.
Considering some mild assumptions in stochastic optimization and the case without heavy-ball momentum ( β 1 = 0 ), SGDand AdamW achieve O (1 / √ T ) and O (ln( T ) / √ T ) convergence rate respectively with appropriatechoice of the learning rate η t .
We present the analysis in Appendix A.4.
Comparisons with Existing Works.
The purpose of exploring large-batch training is to speed upmodel training with increasing computational power, as well as enable us to explore the larger dataset.
As shown in Table 1, the seminal works such as LARS [8], LAMB [6], and SAM [20] have madegreat contributions to large-batch training for plain vision pipelines e.g., image-level prediction,despite that they often require hyper-parameter tuning by experienced engineers.
For complicatedpipelines of dense visual predictions, they are typically not sufficient to achieve desired generalizationperformance.
MegDet [25] and PMD-LAMB [26] contribute the preliminary attempts by applyinglarge-batch training on object detection.
Different from these approaches, we revisit the designparadigm of the complicated dense visual perception pipelines and present a simple and effectivesolution, AGVM, which is insensitive to hyperparameter tuning and can be easily plugged into manyvisual perception pipelines.
For example, AGVM can perform stable training with an unprecedentedbatch size 10K, which could greatly reduce the training time.
Moreover, AGVM adds a negligiblecomputational overhead in training, unlike SAM which involves two sequential (non-parallelizable)gradient computations at each iteration, resulting in a significant increase of the training time.
Experiments
Dataset.
We conduct comprehensive experiments on the MS-COCO 2017 [2] and the ADE20K
[27]datasets.
Specifically, we perform various tasks of object detection, instance segmentation, andpanoptic segmentation on COCO, and conduct semantic segmentation on ADE20K.
Baselines.
Since the prior arts of large-batch optimization methods can be divided into two types,
SGD-based methods ( i.e., LARS [8], MegDet [25]) and AdamW-based methods ( i.e., LAMB [6],
PMD-LAMB [26]).
For fair comparison, we introduce two training configurations using SGD and
AdamW with AGVM, respectively.
The details of the hyper-parameter settings can be found in
Appendix A.5.
Pipelines and Models.
To evaluate the generalization ability of AGVM, we conduct extensiveexperiments on different pipelines, including RetinaNet [28], Faster R-CNN [9], Mask R-CNN [14],
Panoptic FPN [29], and Semantic FPN [29].
For the backbone networks, we use ResNet [19] and
Swin Transformer [30].
We strictly follow the official implementations of these pipelines and models.
Implementation Details.
We implement AGVM in Pytorch and reproduce PMD-LAMB with theofficial implementation of LAMB [6].
We also evaluate LARS [8] and SAM [20] by borrowing their
[Table]
[Table] [Table]
official implementations.
To make fair comparisons
[Table]
we follow the same learning rate scaling methodin all experiments.
For SGD optimizer
[Table]
we use linear learning rate scaling when batch size is less than128 (256 on semantic segmentation).
When the batch size is greater than [Table] we use the square rootof learning rate scaling to avoid divergence in the training process.
For PMD-LAMB and LAMB, wefollow the learning rate scaling scheme in [26].
We apply a learning rate warm-up scheme to avoiddivergence when the learning rate is large.
The implementation details can be found in Appendix A.5.
4.1 Comparisons to the State-of-The-Art Methods
Table 3 compares the results of object detection on COCO dataset with different backbones and batchsizes.
We compare the mAP and the number of iterations of LAMB, PMD-LAMB, and AGVM usingthe AdamW optimizer.
To our knowledge, AGVM reports the first result that successfully scales thebatch size to 1536 with negligible performance drop compared to small-batch training using LAMB.
We also see that AGVM contributes significant improvements along with the continuous increase ofthe batch size.
By scaling the batch size larger than 1024 for different backbones, AGVM can still [Table] achieve 36.6 and 42.8 mAP without heavy hyper-parameter tuning.
In conclusion, compared with
LAMB and PMD-LAMB, AGVM achieves more accurate results whilst reducing training iterationsand runtime.
AGVM can be embedded in CNN and Transformer models.
Generalize to various pipelines, architectures, and optimizers.
AGVM can be generalized todifferent tasks, pipelines, architectures, and optimizers.
Table 2 compares MegDet, SAM, LARS, and
AGVM in different dense visual prediction tasks, including object detection, instance segmentation,semantic segmentation, and panoptic segmentation on COCO and ADE20K.
We evaluate fourrepresentative pipelines ( e.g., Faster R-CNN, Mask R-CNN, Semantic FPN, and Panoptic FPN) withdifferent batch sizes from 32 to 1024.
We see that scaling the batch size only allows fewer iterationsto update weights in previous methods, whose performances drop a lot and even have training failurewhen the batch size is 1024 (denoted by “NaN”).
In contrast, AGVM yields surprising results inall tasks when increasing the batch size.
Table 6 reports the performance of AGVM trained withdifferent optimizers, SGD and AdamW. AGVM works well with both of them.
[Table]
[Table] [Table] [Table]
[Table]
Training COCO in 4 minutes.
With AGVM, we can push the frontier of fast training time on
COCO.
We employ Faster R-CNN with ResNet50-FPN as the detector and use the same experimentalsetting as [26].
Then we explore how fast AGVM can reach the 36.6 mAP@0.5:0.95 reported in[26] (which needs 12 minutes to train).
Different from the hardware setup in Fig.
1 (batch size 8 per
GPU)
[Table]
this experiment is conducted on 768 NVIDIA A100 GPUs.
As shown in Table 4, we reducethe original small-batch training time from 2.5 hours to only 4.2 minutes, which is the fastest recordto our knowledge.
Scaling the batch size to 10k.
We also try to push the frontier of large batch size in dense visualprediction tasks.
We choose RetinaNet with ResNet18 as the detector, which is trained for 24 epochs(2 × ) using the AdamW optimizer.
For batch size 4k and 10k, the learning rates are 0.001 and 0.0015,respectively.
The mAP results on COCO are shown in Table 5.
Without bells and whistles, the batchsize is successfully scaled to 10k with reasonable performance, while PMD-LAMB fails (“NaN”).
Scaling the detector to 1-Billion parameters.
We evaluate AGVM on an extremely-large detectorusing the UniNet [31].
We extend it to one billion parameters by following the design in [31].
The detailed settings are released in Appendix A.5.
Table 8 shows that AGVM still stabilizes andaccelerates the training process in such a large model regime.
Both AdamW and PMD-LAMB diverge
[Table]
in the early training stage.
AGVM can reduce the training time from 3 days (batch size 32) to 3.5hours using 480 NVIDIA A100 GPUs, achieving a 62.2 box mAP on COCO test-dev benchmark,whilst reducing the training wall-clock time by more than 20 times.
4.2 Ablation Study
Insensitive to hyper-parameter τ and α .
We study the effect of the interval parameter τ , whichmeans we update µ ( i ) t every τ iterations, as well as the coefficient of moving average α using Mask
R-CNN.
The experimental results in Table 9 indicate that AGVM is not sensitive to these twohyper-parameters.
In practice, we employ τ = 10 and α = 0 .
97 by default.
When the batch size issignificantly large ( e.g., larger than 1K), we reduce the interval to τ = 5 to update µ ( i ) t faster.
Anchor module selection.
In AGVM, we choose the backbone network as the anchor and modulateother modules to make their gradient variances consistent with the backbone.
To deeply investigatethis selection, we choose different modules as the anchors.
As shown in Table 7, we see that thebackbone is the optimal anchor because the backbone network plays the most important role in densevisual predictions.
Delving into the gradient variance misalignment.
We answer an important question: what causesthe gradient variance misalignment for dense visual predictors?
To tackle this question, we revisitthe data flow of dense prediction pipelines and find that the effective batch size is not consistentbetween different network modules.
For instance, due to the shared detection head ( i.e., classifiersand regressors) in all the levels of the FPN and different region proposals, the detection head has adifferent effective batch size compared to the backbone.
Similarly, the RPN (or detection head in
RetinaNet) shared by all FPN levels and pixel-wise loss computation lead to the increased effectivebatch size in RPN.
Similar to a previous work [22], we find that a larger effective batch size leads tolower gradient variance of modules ( e.g., RPN, detection head).
To explore these analysis, we conduct an progressive ablation study using the RetinaNet, as shown bythe different gradient variance curves in Fig.2.
We have three observations.
(1) Intuitively, the sharedhead leads to the unavoidable batch size misalignment between the backbone and the detection head.
For example, given an input mini-batch size B , the valid mini-batch size for the detection head is NB ,where N is the pyramidal feature number.
This motivates us to directly replace the shared detectionhead by independent detection heads.
As illustrated by the second figure in Fig.2, the gradientvariance misalignment between the detection head and the backbone has been significantly reduced.(2) Furthermore, compared to the plain network architecture, we argue that the effective batch size isalso related to the bottom-up and top-down pathways in FPN.
To evaluate this, we remove FPN and only adopt the final-level feature map to perform detection.
As shown in the third figure in Fig.2,this alleviates the variance difference between the backbone and the detection head.
In the fourthfigure, we remove the FPN and randomly ignore 75% pixels for loss computation in the predictionsgenerated by detection head.
This leads to a near-constant trend of variance throughout trainingtowards convergence between the backbone and the detection head.
We have done a similar studyusing Faster R-CNN, whose results and discussions can be found in Appendix A.3.
Related Work
We review related work on two aspects including dense visual predictions and large-batch optimization.
We defer the related work on dense visual predictions to Appendix A.7.
Large-batch Optimization.
For large scale deep model training, it is significant to adopt a largerbatch size for better hardware utilization and system scalability.
However, large-batch training isprone to converge to a sharp minima, resulting in undesired generalization ability [7].
The main reasonis that the number of iterations will decrease when we fix the number of epochs in large-batch settings.
Researchers [32, 33] try to carefully tune the hyper-parameters to narrow this generalization gap.
Indetail, by incorporating learning rate warm-up and linear scaling, Goyal et al.
[5] successfully train
ResNet50 with batch size 8192 without loss in generalization performance.
Recently, to avoid thesehand-tuned methods, the adaptive learning rate on large-batch training has gained enormous attentionfrom researchers.
For example, LARS and LAMB algorithms [8, 6] enable researchers to scale thebatch size for ResNet50/BERT to 32k/64k.
Both LARS and LAMB leverage the norm of weightsand gradients to adjust the learning rate of each layer.
These adaptive methods enable researchers totrain ImageNet in a few minutes [34, 35, 36].
Johnson et al.
[37] propose AdaScale SGD, a novellearning rate schedule rule for stabilizing the warm-up stage.
However, it highly depends on theparallelism degree of the system.
Liu et al.
[38] use adversarial learning to further scale the batchsize to 96k.
More recently, Sharpness-aware Minimization (SAM) [20] introduces a procedure tominimize the loss value and loss sharpness to close the generalization gap.
However, SAM suffersfrom training efficiency since the update rule of SAM involves two sequential gradient computationat each iteration.
There are few works [39, 40] towards improving the efficiency of SAM.
Recently,effort [41] has been made on how to choose an appropriate batch size and corresponding learningrate for large-batch training.
And Qin et al.
[21] propose Simigrad, which utilizes a lightweight andautomated adaptive batching method to enable fine-grained adaptive batch size.
However, rather thanclassification tasks, there are few works towards large-batch training for object detection.
Peng et al.[25] implement cross-GPU batch normalization to stabilize the training process and Wang et al.
[26]propose PMD-LAMB to reduce the negative effects of the lagging historical gradients.
They canscale the training of widely used Faster R-CNN+ResNet50 Detector with batch size 256/1056 withnegligible performance drop.
Conclusion
The complicated pipelines of dense visual predictions suffer from heavy performance drop in largebatch training.
In this paper, we propose and fully study AGVM, which enables module-wise learningrate scaling and successfully scales the batch size to larger than 10K with desired generalization performance.
We also provide a convergence analysis, showing that AGVM+SGD and AGVM+AdamWboth converge to a stable point in the general non-convex setting.
Furthermore, we have conductedextensive experiments to show that AGVM can generalize to different complicated pipelines andchallenging tasks, including object detection, instance segmentation, semantic segmentation, andpanoptic segmentation.
We report unprecedented better performance on large-batch training withvery large batch size.
For example, AGVM trains Faster R-CNN+ResNet50 using batch size of 1536in 4.2 minutes without loss of performance.
By increasing the object detector UniNet to one billionparameters, AGVM can achieve 62.2 mAP on COCO using a batch size of 960 in just 3.5 hours,reducing the training time by 20.9 × compared to the normal small-batch training.
Limitation and Potential Negative Societal Impact.
Module partitioning is important to estimatethe effective batch size quantitatively.
For some pipelines without explicit modularity such as theheatmap-based pose estimation, we need to do more empirical analysis.
We will investigate it in thefuture.
The potential negative social impact is to use the proposed algorithm to speed up the trainingof fraud models such as DeepFake.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Please seeAppendix A.4.
(b) Did you include complete proofs of all theoretical results?
[Yes] Please see AppendixA.4.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No] Codes andmodels will be released in the future.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please see Section 4 and Appendix A.5.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No] All our results are using fixed seed since the uncertainty
in dense visual predictions is very small.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please see Section 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Please see Section 4.
(b) Did you mention the license of the assets?
[Yes] The data used in our work is open
source and can be used for adademic research.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using
/curating contains personally identifiableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
you include the full text of instructions given to participants and screenshots, if
[Table] applicable
you describe any potential participant risks, links to Institutional Review [Table]
Board
the
participants the total amount spent participant compensation
N [Table]