High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary gradients to accelerate distributed deep learning in data parallelism. Our approach requires only three numerical levels {−1, 0, 1}, which can aggressively reduce the communication time. We mathematically prove the convergence of TernGrad under the assumption of a bound on gradients. Guided by the bound, we propose layer-wise ternarizing and gradient clipping to improve its convergence. Our experiments show that applying TernGrad on AlexNet doesn't incur any accuracy loss and can even improve accuracy. The accuracy loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a performance model is proposed to study the scalability of TernGrad. Experiments show significant speed gains for various deep neural networks. From the perspective of inference acceleration, sparse and quantized Deep Neural Networks (DNNs) have been widely studied, such as [17][18][19][20][21][22]. However, these methods generally arXiv:1705.07878v4 [cs.LG] 18 Sep 2017 aggravate the training effort. Researches such as sparse logistic regression and Lasso optimization problems [4][12][23] took advantage of the sparsity inherent in models and achieved remarkable speedup for distributed training. A more generic and important topic is how to accelerate the distributed training of dense models by utilizing sparsity and quantization techniques. For instance, Aji and Heafield [24] proposed to heuristically sparsify dense gradients by dropping off small values in order to reduce gradient communication. For the same purpose, quantizing gradients to low-precision values with smaller bit width has also been extensively studied [22][25][26][27]. Our work belongs to the category of gradient quantization, which is an orthogonal approach to sparsity methods. We propose TernGrad that quantizes gradients to ternary levels {−1, 0, 1} to reduce the overhead of gradient synchronization. Furthermore, we propose scaler sharing and parameter localization, which can replace parameter synchronization with a low-precision gradient pulling. Comparing with previous works, our major contributions include: (1) we mathematically prove the convergence of TernGrad in general by proposing a statistical bound on gradients; (2) we propose layer-wise ternarizing and gradient clipping to move this bound closer toward the bound of standard SGD. These simple techniques successfully improve the convergence; (3) we build a performance model to evaluate the speed of training methods with compressed gradients, like TernGrad. Our source code is available at https://github.com/wenwei202/terngrad.
TernGrad REF quantizes each element of a gradient vector to three numerical values {−1, 0, 1}. The gradient approximationg is shown to be unbiased and training converges under the assumption of a bound on gradients.
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
ArXiv
ArXiv
Computer Science Mathematics