A NALYSIS OF Q UANTIZED M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Weight-quantized networks have small storage and fast inference, but training can still be time-consuming.
This can be improved with distributed learning.
To reduce the high communication cost due to worker-server synchronization, recently gradient quantization has also been proposed to train networks with full-precision weights.
In this paper, we theoretically study how the combination of both weight and gradient quantization affects convergence.
We show that (i) weight-quantized networks converge to an error related to the weight quantization resolution and weight dimension; (ii) quantizing gradients slows convergence by a factor related to the gradient quantization resolution and dimension; and (iii) clipping the gradient before quantization renders this factor dimension-free, thus allowing the use of fewer bits for gradient quantization.
Empirical experiments conﬁrm the theoretical convergence results, and demonstrate that quantized networks can speed up training and have comparable performance as full-precision networks.
I NTRODUCTION
Deep neural networks are usually huge.
Such a high demand in time and space can signiﬁcantly limit the deployment on low-end devices.
To alleviate this problem, many approaches have been recently proposed to compress deep networks.
One direction is network quantization, which represents each network weight with a small number of bits.
Besides signiﬁcantly reducing the model size, it also accelerates network training and inference.
Many weight quantization methods aim at approximating the full-precision weights in each iteration (Courbariaux et al., 2015;
Lin et al., 2016; Rastegari et al., 2016; Li & Liu, 2016;
Lin et al., 2017; Guo et al., 2017).
Recently, loss-aware quantization minimizes the loss directly w.r.t.
the quantized weights (Hou et al., 2017; Hou & Kwok, 2018; Leng et al., 2018), and often achieves better performance than approximation-based methods.
Distributed learning can further speed up training of weight-quantized networks (Dean et al., 2012).
A key challenge is on reducing the expensive communication cost incurred during synchronization of the gradients and model parameters (Li et al., 2014a;b).
Recently, algorithms that sparsify (Aji &
Heaﬁeld, 2017; Wangni et al., 2017) or quantize the gradients (Seide et al., 2014; Wen et al., 2017;
Alistarh et al., 2017; Bernstein et al., 2018) have been proposed.
In this paper, we consider quantization of both the weights and gradients in a distributed environment.
Quantizing both weights and gradients has been explored in the DoReFa-Net
(Zhou et al., 2016), QNN (Hubara et al.), WAGE (Wu et al., 2018) and ZipML (Zhang et al., 2017).
We differ from them in two aspects.
First, existing methods mainly consider learning with a single machine, and gradient quantization is used to reduce the computations in backpropagation.
On the other hand, we consider a distributed environment, and use gradient quantization to reduce communication cost and accelerate distributed learning of weight-quantized networks.
Weight update and quantization are performed at the server, and only quantized weights are transmitted over the network, leading to a reduction in communication cost from server to workers.
Gradient quantization is performed by the workers.
As only quantized gradients are transmitted, worker-to-server communication is also reduced.
Moreover, instead of keeping both the full-precision and quantized weights as in a nondistributed environment, here the server only keeps the full-precision weights, while each worker only keeps the quantized weights.
Thus, workers can be run on low-end devices with small storage.
Second, while DoReFa-Net, QNN and WAGE show impressive empirical results on the quantized network, theoretical guarantees are not provided.
ZipML provides convergence analysis, but is limited to stochastic weight quantization, square loss with the linear model, and requires the stochastic gradients to be unbiased.
This can be restrictive as most state-of-the-art weight quantization methods (Rastegari et al., 2016; Lin et al., 2016; Li & Liu, 2016;
Guo et al., 2017; Hou et al., 2017; Hou & Kwok, 2018) are deterministic, and the resultant stochastic gradients are biased.
In this paper, we relax the restrictions on the loss function, and study in an online learning setting how biased stochastic gradients affect convergence.
The main ﬁndings are:
With either full-precision or quantized gradients, the average regret of loss-aware weight quantization does not converge to zero, but to an error related to the weight quantization resolution ∆ w and dimension d .
The smaller the ∆ w or d , the smaller is the error (Theorems 1 and 2).
With either full-precision or quantized gradients, the average regret converges with a O (1 / √ T ) rate to the error, where T is the number of iterations.
However, gradient quantization slows convergence (relative to using full-precision gradients) by a factor related to gradient quantization resolution ∆ g and d .
The smaller the ∆ g or d , the faster is the convergence (Theorems 1 and 2).
Empirically, gradient clipping has been found effective in use with quantized gradient (Wen et al., 2017).
We show theoretically that for gradients following the normal distribution, gradient clipping renders the speed degradation mentioned above dimension-free.
However, an additional error is incurred.
The convergence speedup and error are related to how aggressive clipping is performed.
More aggressive clipping results in faster convergence, but a larger error (Theorem 3).
Empirical results verify the convergence analysis and show that quantizing gradients can significantly reduce communication cost, speed up training of weight-quantized networks, and obtain accuracy comparable with the use of full-precision gradients.
Notations.
For a vector x , √ x is the element-wise square root, x 2 is the element-wise square,
Diag ( x ) returns a diagonal matrix with x on the diagonal, and x (cid:12) y is the element-wise multiplication of vectors x and y .
For a matrix Q , (cid:107) x (cid:107) 2 Q = x (cid:62) Qx .
For a matrix X , √ X is the element-wise square root, and diag ( X ) returns a vector extracted from the diagonal elements of X .
P RELIMINARIES 2. O NLINE L EARNING
Online learning continually adapts the model with a sequence of observations.
It has been commonly used in the analysis of deep learning optimizers (Duchi et al., 2011; Kingma & Ba, 2015;
Reddi et al., 2018).
At time t , the algorithm picks a model with parameter w t ∈ S , where S is a convex compact set.
The algorithm then incurs a loss f t ( w t ) .
After T rounds, performance is evaluated by the regret R ( T ) =
(cid:80) Tt =1 f t ( w t ) − f t ( w ∗ ) and average regret R ( T ) /T , where w ∗ =
arg min w ∈S (cid:80) Tt =1 f t ( w ) is the best model parameter in hindsight.
2. W EIGHT Q UANTIZATION
In BinaryConnect (Courbariaux et al., 2015), each weight is binarized using the sign function either deterministically or stochastically.
In ternary-connect (Lin et al., 2016), each weight is stochastically quantized to {− 1 , 0 , 1 } .
Stochastic weight quantization often suffers severe accuracy degradation, while deterministic weight quantization (as in the binary-weight-network (BWN) (Rastegari et al., 2016) and ternary weight network (TWN) (Li & Liu, 2016)) achieves much better performance.
In this paper, we will focus on loss-aware weight quantization, which further improves performance by considering the effect of weight quantization on the loss.
Examples include loss-aware binarization (LAB) (Hou et al., 2017) and loss-aware quantization (LAQ)
(Hou & Kwok, 2018).
Let the full-precision weights from all L layers in the deep network be w , and the corresponding quantized weight is denoted Q w ( w )
=
ˆ w where Q w ( · ) is the weight quantization function.
At the t th iteration, the second-order Taylor expansion of f t ( ˆ w ) , i.e., f t ( ˆ w t ) + ∇ f t ( ˆ w t ) (cid:62) ( ˆ w − ˆ w t ) + 12 ( ˆ w − ˆ w t ) (cid:62) H t ( ˆ w − ˆ w t ) is minimized w.r.t.
ˆ w , where H t is the Hessian at ˆ w t .
H t is expensive to compute.
In practice, this is approximated by Diag ( √ ˆ v t ) where ˆ v t is the moving average: ˆ v t =
β ˆ v t − 1 + (1 − β ) ˆ g 2 t
=
(cid:80) tj =1 (1 − β ) β t − j ˆ g 2 j with β (cid:39) 1 , and is readily available in popular deep network optimizers such as RMSProp and Adam.
Moreover, Diag ( √ ˆ v t ) is also an estimate of Diag ( (cid:112) diag ( H 2 t ))
(Dauphin et al., 2015).
Computationally, the quantized weight can be obtained by ﬁrst performing a preconditioned gradient descent w
[Equation]
followed by quantization via solving of the following optimization problem:
[Equation]
For binarization, S w = {− 1 , +1 } , and a simple closed-form solution is obtained in (Hou et al., 2017).
For m -bit linear quantization, S w = {− M k , . . .
, − M 1 , M 0 , M 1 , . . .
, M k } , where k
= 2 m − 1 − 1 , 0 =
M 0 < · · · < M k are uniformly spaced, with weight quantization resolution ∆ w = M r +1 − M r .
An efﬁcient approximate solution of (1) is obtained in (Hou & Kwok, 2018).
2. S TOCHASTIC G RADIENT Q UANTIZATION
In a distributed learning environment, the main bottleneck is often on worker-to-server communication.
This cost can be signiﬁcantly reduced by quantizing the gradients (computed by the workers) before pushing to the server (Seide et al., 2014; Wen et al., 2017; Alistarh et al., 2017).
For example, assuming that the full-precision gradient is 32-bit, the communication cost is reduced 32 /m times when gradients are quantized to m bits.
For the more advanced all-reduce communication model (Rabenseifner, 2004), the number of bits used for gradients increases by one at every aggregation step in the worst case 2 .
However, even considering this effect, our proposed method still signiﬁcantly reduces the network communication and speeds up training as will be shown in Figure 4 in the experiment section.
In TernGrad (Wen et al., 2017), the gradients are ternarized.
Here, we consider the more general m -bit linear quantization (Alistarh et al., 2017):
Q g ( g t ) =
s t · sign ( g t ) (cid:12) q t , where g t is the stochastic gradient, s t = (cid:107) g t (cid:107) ∞ , q t ∈ ( S g ) d , and
S g = {− B k , . . .
, − B 1 , B 0 , B 1 , . . .
, B k } , with k = 2 m − 1 − 1 ,
= B 0 < B 1 < · · · < B k are uniformly spaced, and the gradient quantization resolution ∆ g = B r +1 − B r .
The i th element q t,i in q t is equal to B r +1 with probability ( | g t,i | /s t − B r ) /
( B r +1 − B r ) , and B r otherwise.
Here, r is an index satisfying B r ≤ | g t,i | /s
t < B r +1 .
Note that Q g ( g t ) is an unbiased estimator of g t .
P ROPERTIES OF Q UANTIZED N ETWORK
In this section, we consider quantization of both weights and gradients in a distributed environment with N workers in data parallelism (Figure 1).
At the t th iteration, worker n ∈ { 1 , 2 , . . .
, N } computes the full-precision gradient
ˆ g ( n ) t w.r.t.
the quantized weight and quantizes ˆ g ( n ) t to ˜ g ( n ) t =
Q g (ˆ g ( n ) t ) .
The quantized gradients are then synchronized and averaged at the parameter server as: ˜ g t = 1 N (cid:80)
Nn =1 ˜ g ( n ) t .
The server updates the second moment ˜ v t based on ˜ g t , and also the fullprecision weight as w t +1 =
w t − η t Diag ( √ ˜ v t ) − 1 ˜ g t .
The weight is quantized using loss-aware weight quantization to produce ˆ w t
+1 =
Q w ( w t +1 ) , which is then sent back to all the workers.
[Figure]
3. A SSUMPTIONS
Analysis on quantized networks has only been performed on models with (i) full-precision gradients and weights quantized by stochastic weight quantization (Li et al., 2017; De Sa et al., 2018), or simple deterministic weight quantization using the sign (Li et al., 2017); (ii) full-precision weights and quantized gradients (Alistarh et al., 2017; Wen et al., 2017; Bernstein et al., 2018); (iii) quantized weights and quantized gradients (Zhang et al., 2017), but limited to stochastic weight quantization, square loss on linear model (i.e., f t ( w t ) =
( x (cid:62) t w t − y t ) 2 ) in Section 2.1), and unbiased gradient.
In this paper, we study the more advanced loss-aware weight quantization.
As it is deterministic and has biased gradients, the above analysis do not apply.
Moreover, we relax the assumptions on f t as: ( A1 ) f t is convex; ( A2 ) f t is twice differentiable with Lipschitz-continuous gradient; and ( A3 ) f t has bounded gradient, i.e., (cid:107)∇ f t ( w ) (cid:107) ≤ G and (cid:107)∇ f t ( w ) (cid:107) ∞
≤ G ∞ for all w ∈ S .
These assumptions have been commonly used in convex
online learning (Hazan, 2016; Duchi et al., 2011;
Kingma & Ba, 2015) and quantized networks (Alistarh et al., 2017; Li et al., 2017).
Obviously, the convexity assumption A1 does not hold for deep networks.
However, this facilitates analysis of deep learning models, and has been used in (Kingma & Ba, 2015; Reddi et al., 2018;
Li et al., 2017; De Sa et al., 2018).
Moreover, as will be seen, it helps to explain the empirical behavior in Section 4.
As in (Duchi et al., 2011; Kingma & Ba, 2015; Li et al., 2017), we assume that (cid:107) w m − w n (cid:107) ≤ D and (cid:107) w m − w n (cid:107) ∞ ≤ D ∞ for all w m , w n ∈ S .
Moreover, the learning rate η t decays as η/ √ t , where η is a constant (Hazan, 2016; Duchi et al., 2011; Kingma & Ba, 2015; Li et al., 2017).
For simplicity of notations, we denote the full-precision gradient ∇ f t ( w t ) w.r.t.
the full-precision weight by g t , and the full-precision gradient ∇ f t ( Q w ( w t )) w.r.t.
the quantized weight by ˆ g t .
As f t is twice differentiable (Assumption A2 ), using the mean value theorem, there exists p ∈ (0 , 1) such that g t − ˆ g t = ∇ f t ( w t ) − ∇ f t ( ˆ w t )
= ∇ 2 f t ( ˆ w t + p ( w t − ˆ w t ))
( w t − ˆ w t ) .
Let H (cid:48) t = ∇ 2 f t ( ˆ w t + p ( w t − ˆ w t )) be the Hessian at ˆ w t + p ( w t − ˆ w t ) .
Moreover, let α = max { α 1 , . .
. , α T } , where α t is the scaling parameter in (1) at the t th iteration.
3. W EIGHT Q UANTIZATION WITH F ULL -P RECISION G RADIENT
When only weights are quantized, the update for loss-aware weight quantization is
[Equation]
where ˆ v t is the moving average of the (squared) gradients ˆ g 2 t as discussed in Section 2.2.
Theorem 1.
For loss-aware weight quantization with full-precision gradients and η t = η/ √ t ,
[Equation]
For standard online gradient descent with the same learning rate scheme, R ( T ) /T converges to zero at a O (1 / √ T ) rate (Hazan, 2016).
From Theorem 1, the average regret converges at the same rate, but only to a nonzero error √ LD (cid:113) D 2 + dα 2 ∆ 2 wrelated to the weight quantization resolution ∆ w and dimension d .
3. W EIGHT Q UANTIZATION W ITH Q UANTIZED G RADIENT
When both weights and gradients are quantized, the update for loss-aware weight quantization is
[Equation]
where ˜ g t is the stochastically quantized gradient Q g ( ∇ f t ( Q w ( w t ))) .
The second moment ˜ v t is the moving average of the (squared) quantized gradients ˜
g 2 t .
The following Proposition shows that gradient quantization signiﬁcantly blows up the norm of the quantized gradient relative to its fullprecision counterparts.
Moreover, the difference increases with the gradient quantization resolution ∆ g and dimension d . Proposition 1.
[Equation]
Theorem 2.
For loss-aware weight quantization with quantized gradients and η t = η/ √ t ,
[Equation]
The regrets in (2) and (4) are of the same form and differ only in the gradient used.
Similarly, for the average regrets in (3) and (5), quantizing gradients slows convergence by a factor of (cid:113) 1+ √ 2 d − 1∆ g + 1 , which is a direct consequence of the blowup in Proposition 1.
These observations can be problematic as (i) deep networks typically have a large d ; and (ii) distributed learning prefers using a small number of bits for the gradients, and thus a large ∆ g .
3. W EIGHT Q UANTIZATION WITH Q UANTIZED C LIPPED G RADIENTS
To reduce convergence speed degradation caused by gradient quantization, gradient clipping has been proposed as an empirical solution (Wen et al., 2017).
The gradient ˆ g t is clipped to Clip (ˆ g t ) , where Clip (ˆ g t,i )
=
ˆ g t,i if | ˆ g t,i | ≤ cσ , and sign (ˆ g t,i ) · cσ otherwise.
Here, c is a constant clipping factor, and σ is the standard deviation of elements in ˆ g t .
The update then becomes
[Equation]
where ˇ g t ≡ Q g ( Clip (ˆ g t ))
≡ Q g ( Clip ( ∇ f t ( Q w ( w t )))) is the quantized clipped gradient.
The second moment ˇ v t is computed using the (squared) quantized clipped gradient ˇ g 2 t .
As shown in Figure 2(a) of (Wen et al., 2017), the distribution of gradients before quantization is close to the normal distribution.
Recall from Section 3.3 that the difference between E ( (cid:107) ˜
g t (cid:107) 2 ) of the quantized gradient ˜ g t and the full-precision gradient (cid:107) ˆ g t (cid:107) 2 is related to the dimension d .
The following Proposition shows that E ( (cid:107)
ˇ g t (cid:107) 2 ) /
E ( (cid:107) ˆ g t (cid:107) 2 ) becomes independent of d if ˆ g t follows the normal distribution and clipping is used.
Proposition 2.
Assume that ˆ g t follows N ( 0 , σ 2 I ) , we have E ( (cid:107)
ˇ g t (cid:107) 2 ) ≤ ((2 /π )
12 c ∆ g +1) E ( (cid:107) ˆ g t (cid:107) 2 ) .
However, the quantized clipped gradient may now be biased (i.e., E (ˇ g t ) =
Clip (ˆ g t ) (cid:54) =
ˆ g t ).
The following Proposition shows that the bias is related to the clipping factor c .
A larger c (i.e., less severe gradient clipping) leads to smaller bias.
Proposition 3.
Assume that ˆ g t follows N ( 0 , σ 2 I ) , we have E ( (cid:107) Clip (ˆ g t ) − ˆ g t (cid:107) 2 ) ≤ dσ 2 (2 /π )
12 F ( c ) , where F ( c ) = − ce − c 2 2 + (cid:112) π 2 (1 + c 2 )(1 − erf ( c √ 2 )) , and erf ( z ) = 2 √ π (cid:82) z[-
-]
e − t 2 dt is the error function.
Theorem 3.
Assume that ˆ g t follows N ( 0 , σ 2 I ) .
For loss-aware weight quantization with quantized clipped gradients and η t = η/ √ t ,
[Equation]
[Equation]
Note that terms involving ˜ g t in Theorem 2 are replaced by ˇ g t .
Moreover, the regret has an additional term D (cid:80)
Tt =1 E ( (cid:112) (cid:107) Clip
(ˆ g t ) − ˆ g t (cid:107) 2 ) over that in Theorem 2.
Comparing the average regrets in Theorems 1 and 3, gradient clipping before quantization slows convergence by a factor of (cid:113) (2 /π ) 12 c ∆ g + 1 , as compared to using full-precision gradients.
This is independent of d as the increase in E ( (cid:107) ˇ g t (cid:107) 2 ) is independent of d (Proposition 2).
Hence, a ∆ g larger than the one in
Theorem 2 can be used, and this reduces the communication cost in distributed learning.
A larger c (i.e., less severe gradient clipping) makes F ( c ) smaller (Figure 2).
Compared with (4) in Theorem 2, the extra error √ dDσ (2 /π ) 14 (cid:112) F ( c ) in (7) is thus smaller, but convergence is also slower.
Hence, there is a trade-off between the two. [Figure]
E XPERIMENTS 4. S YNTHETIC D ATA
In this section, we ﬁrst study the effect of dimension d on the convergence speed and ﬁnal error.
As popular deep networks usually have hand-crafted architectures, we consider here a linear model with square loss as in (Zhang et al., 2017).
Each entry of the model parameter is generated by uniform sampling from [ − 0 . 5 , 0 . 5] .
Samples x i ’s are generated such that each entry of x i is drawn uniformly from [ − 0 .
5 , 0 . 5] , and the corresponding output y i from N ( x (cid:62) i w ∗ , (0 . 2) 2 ) .
At the t th iteration, a mini-batch of B = 64 samples are drawn to form X t =
[ x 1 , . . .
, x B ] and y t =
[ y 1 , . . .
, y B ] (cid:62) .
The corresponding loss is f t ( w t ) =
(cid:107) X
(cid:62) t w t − y t (cid:107) 2 / 2 B .
The weights are quantized to 1 bit using LAB.
The gradients are either full-precision (denoted FP) or stochastically quantized to 2 bits (denoted
SQ2).
The optimizer is RMSProp, and the learning rate is η t =
η/ √ t , where η = 0 .
Figure 3 shows convergence of the average training loss (cid:80)
Tt =1 f t ( w t ) /T , which differs from the average regret only by a constant.
3 As can be seen, for both full-precision and quantized gradients, a larger d leads to a larger loss upon convergence.
Moreover, convergence is slower for larger d , particularly when the gradients are quantized.
These agree with the results in Theorems 1 and 2.
4. CIFAR-
We follow (Wen et al., 2017) and use the same train/test split, Cifarnet model, data preprocessing, augmentation and distributed Tensorﬂow setup.
Adam is used as the optimizer.
The learning rate is decayed by a factor of 0.1 every 200 epochs.
4.2. W EIGHT Q UANTIZATION R ESOLUTION ∆ w
Weights are quantized to 1 bit (LAB), 2 bits (LAQ2), or m bits (LAQ m ).
The gradients are fullprecision (FP) or stochastically quantized to m = { 2 , 3 , 4 } bits (SQ m ) without gradient clipping.
Two workers are used in this experiment.
Figure 5 shows convergence of the average training loss with different numbers of bits for the quantized weight.
With full-precision or quantized gradients, weight-quantized networks have larger training losses than full-precision networks upon convergence.
The more bits are used, the smaller is the ﬁnal loss.
This agrees with the results in Theorems 1 and 2.
Table 1 shows the test set accuracies.
Weight-quantized networks are less accurate than their full-precision counterparts, but the degradation is small when 3 or 4 bits are used. full-precision.
[Figure]
[Table]
[Table]
4.2. G RADIENT Q UANTIZATION R ESOLUTION ∆ g
Figure 6 shows convergence of the average training loss with different numbers of bits for the quantized gradients, again without gradient clipping.
Using fewer bits yields a larger ﬁnal error, and using 2- or 3-bit gradients yields larger training loss and worse accuracy than full-precision gradients (Figure 6 and Table 1).
The fewer bits for the gradients, the larger the gap.
The degradation is negligible when 4 bits are used.
Indeed, 4-bit gradient sometimes has even better accuracy than full-precision gradient, as its inherent randomness encourages escape from poor sharp minima (Wen et al., 2017).
Moreover, using a larger m results in faster convergence, which agrees with Theorem 2.
[Figure]
4.2. G RADIENT C LIPPING
In this section, we perform experiments on gradient clipping, with clipping factor c in { 1 , 2 , 3 } .
LAQ2 is used for weight quantization and SQ2 for gradient quantization.
Figure 7(a) shows histograms of the full-precision gradients before clipping.
As can be seen, the gradients at each layer before clipping roughly follow the normal distribution, which veriﬁes the assumption in Section 3.4.
Figure 7(b) shows the average (cid:107) ˜
g t (cid:107) 2 / (cid:107) ˆ g t (cid:107) 2 (for non-clipped gradients) and (cid:107) ˇ g t (cid:107) 2 / (cid:107) ˆ g t (cid:107)(for clipped gradients) over all iterations.
The dimensionalities ( d ) of the various Cifarnet layers are “conv1”: 1600, “conv2”: 1600, “fc3”: 884736, “fc4”: 73728, “softmax”: 1920.
Layers with large d have large (cid:107) ˜
g t (cid:107) 2 / (cid:107) ˆ g t (cid:107) 2 values, which agrees with Proposition 1.
With clipped gradients, (cid:107) ˇ g t (cid:107) 2 / (cid:107) ˆ g t (cid:107) 2 is much smaller and does not depend on d , agreeing with Proposition 3.
Figure 7(c) shows convergence of the average training loss.
Using a smaller c (more aggressive clipping) leads to faster training (at the early stage of training) but larger ﬁnal training loss, agreeing with Theorem 3.
[Figure] Figure 8 shows convergence of the average training loss with different numbers of bits for the quantized clipped gradient, with c = 3 .
By comparing 4 with Figure 6, gradient clipping achieves faster convergence, especially when the number of gradient bits is small.
For example, 2-bit clipped gradient has comparable speed (Figure 8) and accuracy (Table 1) as full-precision gradient. [Figure]
4.2. V ARYING THE N UMBER OF W ORKERS
We use 3-bit quantized weight (LAQ3), and gradients are full-precision or stochastically quantized to m = { 2 , 3 , 4 } bits (SQ m ).
The setup follows (Wen et al., 2017), and is in Appendix C. Tableshows the testing accuracies with varying number of workers N .
Observations are similar to those in Section 4.2.3.
2-bit quantized clipped gradient has comparable performance as full-precision gradient, while the non-clipped counterpart requires 3 to 4 bits for comparable performance.
4. I MAGE N ET
In this section, we train the AlexNet on ImageNet.
We follow (Wen et al., 2017) and use the same data preprocessing, augmentation, learning rate, and mini-batch size.
Quantization is not performed in the ﬁrst and last layers, as is common in the literature (Zhou et al., 2016; Zhu et al., 2017; Polino et al., 2018; Wen et al., 2017).
We use Adam as the optimizer.
We experiment with 4-bit loss-aware weight quantization (LAQ4), and the gradients are either full-precision or quantized to 3 bits (SQ3).
Table 3 shows the accuracies with different numbers of workers.
Weight-quantized networks have slightly worse accuracies than full-precision networks.
Quantized clipped gradient outperforms the non-clipped counterpart, and achieves comparable accuracy as full-precision gradient.
[Table]
Figure 4 shows the speedup in distributed training of a weight-quantized network with quantized/full-precision gradient compared to training with one worker using full-precision gra- dient.
We use the performance model in (Wen et al., 2017), which combines lightweight proﬁling on a single node with analytical communication modeling.
We also use the all-reduce communication model (Rabenseifner, 2004), in which each GPU communicates with its neighbor until all gradients are accumulated to a single GPU.
We do not include the server’s computation effort on weight quantization and the worker’s effort on gradient clipping, which are negligible compared to the forward and backward propagations in the worker.
As can be seen, when the bandwidth is small (Figure 4(a)), communication is the bottleneck, and using quantizing gradients is signiﬁcantly faster than the use of full-precision gradients.
With a larger bandwidth (Figure 4(b)), the difference in speedups is smaller.
Moreover, note that on the 1Gbps Ethernet with quantized gradients, its speedup is similar to those on the 10Gbps Ethernet with full-precision gradients.
C ONCLUSION
In this paper, we studied loss-aware weight-quantized networks with quantized gradient for efﬁcient communication in a distributed environment.
Convergence analysis is provided for weight-quantized networks with full-precision, quantized and quantized clipped gradients.
Empirical experiments conﬁrm the theoretical results, and demonstrate that quantized networks can speed up training and have comparable performance as full-precision networks.
A L EMMAS
Lemma 1.
E (˜ g t )
=
ˆ g t , and E ( (cid:107)
˜
g t − ˆ g t (cid:107) 2 ) ≤ ∆ g (cid:107) ˆ g t (cid:107) ∞ (cid:107) ˆ g t (cid:107) 1 .
A. P ROOF FOR L EMMA
Proof.
For the i th element ˜ g t,i of the quantized gradient ˜ g t , denote its two adjacent quantized values as B r,i and B r
+1 ,i with B r,i ≤ | g t,i | /s
t < B r
+
,i , its expectation satisﬁes:
[Equation]
Thus E (˜ g t ) =
ˆ g t , and the variance of the quantized gradients satisfy
[Equation]
ˆ g t (cid:107) ∞ (cid:107) ˆ g t (cid:107) 1 .
Lemma 2.
For v t = (cid:80) tj =1 (1 − β ) β t − j g 2 j with ∀ j ∈ { 1 , 2 · · · , t } , (cid:107) g j (cid:107) ∞ <
G ∞ , then its i th element v t,i satisﬁes v t,i ≥
[Equation]
i ,
[Equation]
≤
[Equation]
∞ , and (cid:107)√
[Equation]
=
[Equation]
(cid:88)
[Equation]
= (
[Equation]
(cid:107) 2 .
Proof.
v t,i =
t (cid:88)
j = (
[Equation] [Equation]
(1 −
[Equation]
) g
[Equation]
,i .
√ v t,i = (cid:118)(cid:117)(cid:117)(cid:116) t (cid:88)
j = (1 − β ) β t − j g 2 j,i ≤ (cid:118)(cid:117)(cid:117)(cid:116) t (cid:88)
j = (1 − β ) β t − j G 2 ∞ ≤ G ∞ .
(cid:107)√ v t (cid:107) 2 = (cid:118)(cid:117)(cid:117)(cid:116)
d (cid:88) i = v t,i = (cid:118)(cid:117)(cid:117)(cid:116)
d (cid:88) i
= t (cid:88)
j = (1 − β ) β t − j g 2 j,i = (cid:118)(cid:117)(cid:117)(cid:116) t (cid:88)
j = (1 − β ) β t − j (cid:107) g j (cid:107)
[Equation]
Lemma 3.
[Lemma 10.3 in (Kingma & Ba, 2015)] Let g 1: T,i =
[ g 1 ,i , g 2 ,i , . .
. , g T,i ] (cid:62)
∈ R T be the vector containing the i th element of the gradients for all iterations up to T , and g t be bounded as in
Assumption A3 ,
[Equation]
B P ROOFS B. P ROOF FOR T HEOREM
Proof.
When only weights are quantized, the update for loss-aware weight quantization is
[Equation]
Consider the i th parameter of w t .
the update for it is w t +1 ,i = w t,i − η t ˆ g t,i √
ˆ v t,i , which implies
[Equation]
After rearranging,
[Equation]
Since f t is convex, we have
[Equation] [Equation]
As f t is convex (Assumption
A1 ) and twice differentiable (Assumption
A2 ),
[Equation]
u
[Equation]
∇
[Equation]
v
[Equation]
≤
[Equation]
u −
[Equation]
2 for any u , v is equivalent to ∇ 2 f t (cid:22) L I where I is the identity matrix.
Thus from the domain bound assumption (cid:107) w m − w n (cid:107) 2 ≤ D , we have
[Equation]
Combining (9) and (10) sum over all the dimensions for i ∈ { 1 , 2 · · · , d } and over all the iterations for t ∈ { 1 , 2 , · · · , T } , we have
[Equation]
The ﬁrst inequality comes from Lemma 2.
The second inequality comes from the deﬁnition of H (cid:48) t .
The third inequality comes from the deﬁnition of H (cid:48) t and Cauchy’s inequality.
The fourth inequality comes from (11).
The last inequality comes from Lemma 2.
For m -bit ( m > 1 ) loss-aware weight quantization in (8), as ˆ w t = arg min w (cid:107) w − w t (cid:107)
Diag ( √ ˆ v t − 1 ) = (cid:80) di
=1 (cid:112) ˆ v t − 1 ,i ( w − w t,i ) 2 .
If w t,i is within the representable range, i.e., α t M k − 1 ≤ w t,i ≤ α t M k , as (cid:112) ˆ v t − 1 ,i > 0 , the optimal ˆ w t,i satisﬁes ˆ w t,i ∈ { α t M r,i ,
α t M r +1 ,i } , where r is the index that satisﬁes α t M r,i ≤ w t,i ≤
α t M r +1 ,i .
Since α = max { α 1 , · · · , α T } , we have
[Equation]
In the other case, if w t,i is exterior of the representable range, the optimal ˆ w t,i is just the nearest representable value of w t,i , thus
[Equation]
From (13) and (14), and sum over all the dimensions we have
[Equation]
From (11) and (15),
[Equation]
[Equation]
From (16) and Lemma 2, we have from (12)
[Equation]
Thus the average regret
[Equation]
B. P ROOF FOR P ROPOSITION
Proof.
From Lemma 1,
[Equation]
Denote { x 1 , x 2 , . . .
, x d } as the absolute values of the elements in ˆ g t sorted in ascending order.
From
Cauchy inequality, we have (cid:107) ˆ g t (cid:107) ∞ (cid:107) ˆ g (cid:107)= x d d (cid:88) i
= x i =
d (cid:88) i =
x d x i ≤ d − 1 (cid:88) i =
(1 + √ 2 d − x 2 i +
[Equation]
The equality holds iff x 1 = x 2 = · · · = √1+ √ 2 d − 1 x d .
Thus we have
[Equation]
B. P ROOF FOR T HEOREM
Proof.
When both weights and gradients are quantized, the update is
[Equation]
Similar to the proof for Theorem
[Equation]
and using that E (˜ g t )
=
ˆ g t
[Equation]
we can get
[Equation]
As (16) in the proof for Theorem 1 still hold, using Proposition 1 and Assumption A3 , we have
E ( R ( T )) ≤
[Equation]
[Equation]
B. P ROOF FOR P ROPOSITION
Proof.
From Lemma 1,
E ( Q g ( Clip (ˆ g t ))) =
Clip (ˆ g t ) , and
E ( (cid:107) Q g ( Clip (ˆ g t ))
(cid:107) 2 ) ≤ E (∆ g (cid:107) Clip
(ˆ g t ) (cid:107) ∞ (cid:107) Clip
(ˆ g t ) (cid:107)
+ (cid:107) Clip (ˆ g t ) (cid:107) 2 ) .
As [ˆ g t ] i ∼ N (0 , σ 2 ) , its pdf is f ( x ) = 1 √ 2 πσ 2 e − x 2 2 σ 2 , thus
E ( (cid:107) Clip (ˆ g t ) (cid:107) 1 ) ≤ E ( (cid:107) ˆ g t (cid:107) 1 ) =
[Equation]
(cid:88)
[Equation]
=
E (
[Equation]
[ˆ g
[Equation] [Equation]
| ) =
[Equation]
(cid:90)
[Equation]
∞
[Equation]
( x ) d
[Equation]
2 d
[Equation] [Equation]
x ) d x
[Equation]
d (cid:90)
[Equation]
∞
[Equation] [Equation]
(cid:12)(cid:12)(cid:12) + ∞
[Equation]
(2 /π )
12 dσ.
As E ( (cid:107) Clip (ˆ g t ) (cid:107) 2 ) ≤ E ( (cid:107) ˆ g t (cid:107) 2 ) , and that E ( (cid:107) ˆ g t (cid:107) 2 ) = dσ 2 , we have
E ( (cid:107) Q g ( Clip (ˆ g t ))
(cid:107) 2 ) ≤ E (∆ g (cid:107) Clip
(ˆ g t ) (cid:107) ∞ (cid:107) Clip
(ˆ g t ) (cid:107)
+ (cid:107) Clip
(ˆ
[Table]
t ) (cid:107)
[Equation]
B. P ROOF FOR P ROPOSITION
Proof.
[Equation]
B. P ROOF FOR T HEOREM
Proof.
When both weights and gradients are quantized, and gradient clipping is applied before gradient quantization, the update then becomes
[Equation]
Similar to the proof for Theorem 1, and using that E ( Q g ( Clip (ˆ g t ))) =
Clip
[Equation]
ˆ g t ) , we can get
E ( R ( T )) ≤
d (cid:88) i =
E ( (cid:112) ˇ
[Equation] [Equation]
i
[Equation]
( w
[Equation]
i
[Equation]
w ∗ i ) 2 ) + d (cid:88) i = T (cid:88) t
[Equation]
E
[Equation]
( (cid:112)
[Equation]
(cid:112)
[Equation]
(cid:33)
+
T (cid:88) t =
E ( (cid:104) w t − w ∗ , g t − ˇ g t (cid:105) )
+ d (cid:88) i =
T (cid:88) t =
η t 2 E   ˇ g 2 t,i (cid:113) (1 − β )ˇ g 2 t,i   ≤
D 2 ∞ 2 η d (cid:88) i =
E ( (cid:112) T ˇ v T,i )
+
T (cid:88) t =
E (cid:104) w t − w ∗ , g t − ˆ g t (cid:105) +
T (cid:88) t =
E (cid:104) w t − w ∗ , ˆ g t − Clip (ˆ g t ) (cid:105) + ηG ∞ √ 1 − β d
[Equation]
i =
[Equation]
[Table]
=
[Equation]
(16) in the proof for Theorem 1 still hold.
Similar to the proof for Theorem 2, using the domain bound assumption ( (cid:107) w m − w n (cid:107) 2 ≤ D and (cid:107) w m − w n (cid:107) ∞ ≤ D ∞ for any w m , w n ∈ S )
, Proposition 3 and that (cid:80) Tt =1 1 t ≤ 2 √ T , we have
[Equation]
C E XPERIMENTAL S ETUP
The detailed setup for the CIFAR-10 and ImageNet data sets are shown in Table 4.
[Table]
D C OMPARISON WITH S TOCHASTIC W EIGHT Q UANTIZATION D. L INEAR M ODELS
For stochastic weight quantization, the gradient is only unbiased for linear models i.e., f t ( w t ) =
( x (cid:62) t w t − y t ) 2 ) .
Speciﬁcally, denote the unbiased stochastic weight quantization function as Q w (i.e.
E ( Q w ( w ))
= w for any w ∈ R d ), then the expectation of the gradient w.r.t.
the quantized weights is also unbiased
[Equation]
where g t = 2 x t ( x (cid:62) t w t − y t ) is the full-precision gradient w.r.t.
the full-precision weight.
Moreover, if we further use an unbiased stochastic gradient quantization function (i.e., E ( Q g ( g )) =
g for any g ∈ R d ), then the quantized gradient w.r.t.
the quantized weight is also unbiased
[Equation]
In this case, convergence analysis of gradient quantization Wen et al.
; Alistarh et al.
(2017) and ZipML Zhang et al.
(2017) that based on unbiased approximation of SGD gradients, can be easily extended here.
Moreover, the error term will also disappear.
D. N ON - LINEAR MODELS
For non-linear models, however, using an unbiased stochastic weight quantization does not necessarily guarantee the gradient w.r.t.
the quantized weight or the quantized gradient w.r.t.
the quantized weight to be unbiased.
Take the linear stochastic weight quantization as an example, where the quantization function also satisﬁes E ( Q w ( w ))
= w , the convergence and error when using fullprecision, quantized, quantized clipped gradients are shown in Theorems 4, 5 and 6, respectively.
Denote Q w as the stochastic weight quantization function.
Q w is deﬁned in a similar way as the gradient quantization function in Section 2.3: Q w ( w t ) =
s t · sign ( w t ) (cid:12) q t , where w t is the weight before quantization, s t = (cid:107) w t (cid:107) ∞ , q t ∈ ( S w ) d , and S w = {− M k , . .
. , − M 1 , M 0 , M 1 , . .
{+, M k } , with k = 2 m − 1 − 1 , 
+}
= M 0 < M 1 < · · · < M k are uniformly spaced, and the weight quantization resolution ∆ w =
M r +1 − M r .
The i th element q t,i in q t is equal to M r +1 with probability ( | w t,i | /s t − M r ) /
( M r +1 − M r ) , and M r otherwise.
Here, r is an index satisfying
M r ≤ | w t,i | /s t < M r +1 .
When only weights are quantized, the update with full-precision gradient is
[Equation]
[Equation]
We still denote the full-precision gradient ∇ w f t ( Q w ( w t )) as ˆ g t .
{+As f t is twice differentiable (Assumption A2 ), using the mean value theorem, there exists p ∈ (0 , 1) such that (cid:107) g t − ˆ g t (cid:107) 
+}
= (cid:107)∇
[Equation]
=
[Equation]
p
[Equation]
w t − ˆ w
[Equation]
( w t − ˆ w t ) (cid:107) 2 .
Let H (cid:48) t = ∇ 2 w f t ( ˆ w t + p ( w t − ˆ w t )) be the Hessian at ˆ w t + p ( w t − ˆ w t ) .
Theorem 4.
For stochastic weight quantization (scheme (20)) with full-precision weights, and η t = η/ √ t ,
[Equation]
When both weights and gradients are quantized
[Equation]
the update is
[Equation]
Theorem 5.
For stochastic weight quantization with quantized gradients (scheme (21)) and η t = η/ √ t ,
[Equation]
Theorem 6.
Assume that ˆ g t follows N ( 0 , σ 2 I ) .
For stochastic weight quantization with quantized clipped gradients and η t = η/ √ t ,
[Equation]
As can be seen from Theorems 4-6, there is also an error term LD (cid:113)
[Equation]
related to the weight quantization resolution and dimension.
Moreover, this error term can be potentially larger than the one induced by the loss-aware weight quantization in Theorem 1, as D ∞ can be much larger than α and dD 2 ∞ be much larger than D 2 .
We provide a sketch of proof for Theorem 4 in the next section, the proofs for Theorems 5 and 6 can be extended from it in a similar way as the loss-aware weight quantization.
D.2. P ROOF FOR T HEOREM
Proof.
Consider the i th parameter of w t .
the update for it is w t +1 ,i = w t,i − η t ˆ g t,i which implies
[Equation]
After rearranging,
[Equation]
Since f t is convex, we have
[Equation]
As f t is convex (Assumption
A1 ) and twice differentiable (Assumption
A2 ),
(cid:107)∇ f
[Equation]
( u
[Equation]
∇ f
[Equation]
( v ) (cid:107) 2 ≤ L (cid:107) u − v (cid:107) 2 for any u , v is equivalent to ∇ 2 f t (cid:22) L I where I is the identity matrix.
Thus from the domain bound assumption (cid:107) w m − w n
[Equation]
2 ≤ D , we have
[Equation]
Combining (22) and (D.2.1), and sum over all the dimensions for i ∈ { 1 , 2 · · · , d } and over all the iterations for t ∈ { 1 , 2 , · · · , T } , we have
[Equation]
The second inequality comes from the deﬁnition of H (cid:48) t .
The third inequality comes from the deﬁnition of H (cid:48) t and Cauchy’s inequality.
The last inequality comes from (cid:80)
Tt =1 1 √ t ≤ 2 √ T .
For m -bit ( m > 1 ) stochastic linear weight quantization, as s t M k − 1 ≤ w t,i ≤ s t M k , and s t ≤ D ∞ we have
[Equation]
[Equation]
Thus the regret satisﬁes
[Equation]
Thus the average regret
[Equation]