P ROX Q UANT : Quantized Neural Networks via Proximal Operators
Anonymous Author(s)
AfﬁliationAddress email
Abstract
To make deep neural networks feasible in resource-constrained environments (suchas mobile devices), it is beneﬁcial to quantize models by using low-precisionweights.
One common technique for quantizing neural networks is the straightthrough gradient method, which enables back-propagation through the quantizationmapping.
Despite its empirical success, little is understood about why the straightthrough gradient method works.
Building upon a novel observation that the straight-through gradient method is infact identical to the well-known Nesterov’s dual-averaging algorithm on a quantization constrained optimization problem, we propose a more principled alternativeapproach, called P ROX Q UANT , that formulates quantized network training as aregularized learning problem instead and optimizes it via the prox-gradient method.
P ROX Q UANT does back-propagation on the underlying full-precision vector andapplies an efﬁcient prox-operator in between stochastic gradient steps to encouragequantizedness.
For quantizing ResNets and LSTMs, P ROX Q UANT outperformsstate-of-the-art results on binary quantization and is on par with state-of-the-art onmulti-bit quantization.
For binary quantization, our analysis shows both theoretically and experimentally that P ROX Q UANT is more stable than the straight-throughgradient method (i.e. BinaryConnect), challenging the indispensability of thestraight-through gradient method and providing a powerful alternative.
Introduction
In this paper we formulate the problem of model quantization as a regularized learning problem andpropose to solve it with a proximal gradient method.
Our contributions are summarized as follows.
• We present a uniﬁed framework for deﬁning regularization functionals that encourage
binary, ternary, and multi-bit quantized parameters, through penalizing the distance toquantized sets.
For binary quantization, the resulting regularizer is a W -shaped non-smooth
regularizer, which shrinks parameters towards either − 1 or 1 in the same way that the L
norm regularization shrinks parameters towards 0 .
We demonstrate that the prox-operators
for regularizers that come out of our framework often admit linear-time solutions (or linear
time approximation heuristics) which result in numerically exact quantized parameters.
• We propose training quantized networks using P ROX Q UANT (Algorithm 1) — a stochastic
proximal gradient method with a homotopy scheme.
Compared with the straight-through gra-
dient method, P ROX Q UANT has access to additional gradient information at non-quantized
points, and its homotopy scheme prevents potential overshoot early in the training.
Algo-
rithmically, P ROX Q UANT involves just adding a simple proximal step with respect to a
quantization-inducing regularizer after each stochastic gradient step, thus can be efﬁciently
implemented under any major deep learning frameworks without incurring signiﬁcant system overhead and be used as a modular component to add to the training pipeline of any
deep networks to result in a quantized network.
• We demonstrate the effectiveness and ﬂexibility of P ROX Q UANT through systematic experi-
ments on (1) image classiﬁcation with ResNets (Section 3.1); (2) language modeling withLSTMs (Section 3.2).
The P ROX Q UANT method outperforms the state-of-the-art resultson binary quantization and is comparable with the state-of-the-art on ternary and multi-bitquantization.• For binary nets, we show that BinaryConnect suffers from more optimization instability thanP ROX Q UANT through (1) a theoretical characterization of convergence for BinaryConnect(Section 4.1) and (2) a sign change experiment on CIFAR-10 (Section G).
Experimentally,P ROX Q UANT ﬁnds better binary nets that is also closer to the initialization in the signchange metric.
We present the main ingredients of our contribution in this extended abstract.
See the Appendices Bfor the prior work, C for the notation, A and D for the motivation and preliminary discussions aboutthe straight-through gradient method and prox operators.
Quantized net training via regularized learning
We propose the P ROX Q UANT algorithm, which adds a quantization-inducing regularizer onto theloss and optimizes via the (non-lazy) prox-gradient method with a ﬁnite λ .
The prototypical versionof P ROX Q UANT is described in Algorithm 1.
Algorithm 1 P
Require:
Regularizer R that induces desired quantizedness, initialization θ 0 , learning rates { η t } t ≥ 0 , regularization strengths { λ t } t ≥while not converged do
Perform the prox-gradient step
[Equation]
The inner SGD step in eq.
(2) can be replaced by any preferred stochastic optimization method such as Momentum SGD or Adam [Kingma and Ba, 2014].
end while
Compared to usual full-precision training, P ROX Q UANT only adds a prox step after each stochasticgradient step, hence can be implemented straightforwardly upon existing full-precision training.
Asthe prox step does not need to know how the gradient step is performed, our method adapts to otherstochastic optimizers as well such as Adam.
Further, each iteration is a prox-gradient step over theobjective L ( θ ) + λ t R ( θ ) with learning rates η t , and by choosing ( η t , λ t ) we obtain a joint controlover the speed of training and falling onto the quantized set.
Details of choosing the regularizer R , deriving the prox-operator prox λR , and choosing the regularization strength are deferred to Appendix E.
Experiments
We evaluate the performance of P ROX Q UANT on two tasks: image classiﬁcation with ResNets, andlanguage modeling with LSTMs.
On both tasks, we show that the default straight-through gradientmethod is not the only choice, and our P ROX Q UANT can achieve the same and often better results.
3.1 Image classiﬁcation on CIFAR-10
Problem setup We perform image classiﬁcation on the CIFAR-10 dataset, which contains 50000training images and 10000 test images of size 32x32.
We apply a commonly used data augmentationstrategy (pad by 4 pixels on each side, randomly crop to 32x32, do a horizontal ﬂip with probability0.5, and normalize).
Our models are ResNets
[He et al., 2016] of depth 20, 32, and 44 with weightsquantized to binary or ternary.
Method We use P ROX Q UANT with suitable regularizers in the binary case and the ternary case,which we respectively denote as PQ-B and PQ-T.
The training is initialized at pre-trained fullprecision nets (warm-start).
For the regularization strength we use the homotopy method λ t = λ · twith λ = 10 − 4 .
We initialize at pre-trained full-precision networks and use the Adam optimizerwith constant learning rate 0.01.
To accelerate training in the ﬁnal stage, we do a hard quantizationθ (cid:55)→ q ( θ ) at epoch 400 and keeps training till the 600-th epoch to stabilize the BatchNorm layers.
We compare with BinaryConnect (BC) for binary nets and Trained Ternary Quantization (TTQ)
[Zhuet al., 2016] for ternary nets.
For BinaryConnect, we haven’t found reported results with ResNetson CIFAR-10, and we train with the recommended Adam optimizer with learning rate decay [Courbariaux et al., 2015]
(initial learning rate 0.01, multiply by 0.1 at epoch 81 and 122, hard-quantize atepoch 400), which we ﬁnd leads to the best result for BinaryConnect.
Result The top-1 classiﬁcation errors are reported in Table 1.
For binary nets, our P ROX Q UANTBinary consistently yields better results than BinaryConnect.
For ternary nets, our results arecomparable with the reported results of TTQ, and the best performance of our method over 4 runs(from the same initialization) is slightly better than TTQ
[Table]
3.2 Language modeling with LSTMs
[Table]
See Appendix F for details.
Stability analysis of binary quantization4.1 Convergence characterization for BinaryConnect
We now show that BinaryConnect has a very stringent convergence condition.
Consider the BinaryConnect method with batch gradients:
[Equation]
Deﬁnition 4.1 (Fixed point and convergence) .
We say that s ∈ {± 1 } d is a ﬁxed point of the
BinaryConnect algorithm, if s
= s in eq.
(3) implies that s t =
s for all t = 1 , 2 , ... .
We say that
the
BinaryConnect algorithm converges if there exists t < ∞ such that s t is a ﬁxed point.
Theorem 4.1.
Assume that the learning rates satisfy (cid:80) ∞ t =
η t = ∞ , then s ∈ {± 1 } d is a ﬁxedpoint for BinaryConnect eq.
(3) if and only if sign ( ∇ L ( s )
[ i ])
= − s [ i ] for all i ∈
[ d ] such that∇ L ( θ )
[ i ] (cid:54) = 0 .
Such a point may not exist, in which case BinaryConnect does not converge for anyinitialization θ 0 ∈ R d .
In a sign change experiment on CIFAR-10 (see Appendix G), we are going to see that BinaryConnectindeed fails to converge to a ﬁxed sign pattern, corroborating Theorem 4.1.
A Drawback of the straight-through gradient method
Typically, training a quantized network involves (1) the design of a quantizer q that maps afull-precision parameter to a k -bit quantized parameter, and (2) the straight-through gradientmethod [Courbariaux et al., 2015] that enables back-propagation from the quantized parameterback onto the original full-precision parameter, which is critical to the success of quantized networktraining.
With quantizer q , an iterate of the straight-through gradient method (see Figure 1a) proceedsas θ t +
=
θ t − η t (cid:101) ∇ L ( θ ) | θ = q ( θ t ) , and q ( (cid:98) θ ) (for the converged (cid:98) θ ) is taken as the output model.
Fortraining binary networks, choosing q ( · ) = sign ( · ) gives the BinaryConnect method [Courbariauxet al., 2015].
Though appealingly simple and empirically effective, it is information-theoretically rather mysteriouswhy the straight-through gradient method works well, at least in the binary case: while the goal isto ﬁnd a parameter θ ∈ {± 1 } d with low loss, the algorithm only has access to stochastic gradientsat {± 1 } d .
As this is a discrete set, a priori , gradients in this set do not necessarily contain anyinformation about the function values.
Indeed, a simple one-dimensional example (Figure 1b) showsthat BinaryConnect fails to ﬁnd the minimizer of fairly simple convex Lipschitz functions in {± 1 } ,due to a lack of gradient information in between.
[Equation]
[Table]
=">AAACC3icdVDLSgMxFM34tr6qLt2EFkEQhqS2tt0JblxWsFro1JJJUxvMPEzuCGWYvRt/xY0LRdz6A+78G9OHoKIHQg7n3Mu99/ixkgYI+XBmZufmFxaXlnMrq2vrG/nNrXMTJZqLJo9UpFs+M0LJUDRBghKtWAsW+Epc+NfHI//iVmgjo/AMhrHoBOwqlH3JGVipmy94fqR6ZhjYL/VgIIBdpp4OcHyTdVPYp1nWzReJWyJ1cljFxD2oVOq0ZkmVViu0jKlLxiiiKRrd/LvXi3gSiBC4Ysa0KYmhkzINkiuR5bzEiJjxa3Yl2paGLBCmk45vyfCuVXq4H2n7QsBj9XtHygIzWtdWBgwG5rc3Ev/y2gn0a51UhnECIuSTQf1EYYjwKBjck1pwUENLGNfS7or5gGnGwcaXsyF8XYr/J+cllxKXnpaLR6VpHEtoBxXQHqKoio7QCWqgJuLoDj2gJ/Ts3DuPzovzOimdcaY92+gHnLdPjnGb9Q==</latexit>
[Figure]
B Prior work
Methodologies Han et al.
[2015] propose Deep Compression, which compresses a DNN viasparsiﬁcation, nearest-neighbor clustering, and Huffman coding.
This architecture is then made intoa specially designed hardware for efﬁcient inference [Han et al., 2016].
In a parallel line of work,
Courbariaux et al.
[2015] propose BinaryConnect that enables the training of binary neural networks, and Li and Liu [2016], Zhu et al.
[2016] extend this method into ternary quantization.
Training andinference on quantized nets can be made more efﬁcient by also quantizing the activation [Hubaraet al., 2017, Rastegari et al., 2016, Zhou et al., 2016], and such networks have achieved impressiveperformance on large-scale tasks such as ImageNet classiﬁcation [Rastegari et al., 2016, Zhu et al.,2016].
In the NLP land, quantized language models have been successfully trained using alternatingmulti-bit quantization [Xu et al., 2018].
Theories Li et al.
[2017] prove the convergence rate of stochastic rounding and BinaryConnecton convex problems and demonstrate the advantage of BinaryConnect over stochastic rounding onnon-convex problems.
Anderson and Berg [2017] demonstrate the effectiveness of binary networksthrough the observation that the angles between high-dimensional vectors are approximately preservedwhen binarized, and thus high-quality feature extraction with binary weights is possible.
Ding et
al.[2018] show a universal approximation theorem for quantized ReLU networks.
Principled methods Hou and Kwok [2018] propose a proximal Newton algorithm for model quantization, which makes use of the additional Hessian information but is hence slightly more expensivein each iteration.
Sun and Sun [2018] perform model quantization through a Wasserstein regularization term and minimize via the adversarial representation, similar as in Wasserstein GANs [Arjovskyet al., 2017].
Their method has the potential of generalizing to other generic requirements on theparameter, but might be hard to tune due to the instability of the inner maximization problem.
While preparing this manuscript, we discovered the independent work of Carreira-Perpinán [2017],
Carreira-Perpinán and Idelbayev [2017].
They formulate quantized network training as a constrainedoptimization problem and propose to solve them via augmented Lagrangian methods.
From anoptimization perspective, our views are largely complementary: they treat the quantization as aconstraint, whereas we encourage quantization through a regularizer.
Due to time constraints, wedid not do experimental comparison (they only reported results on VGG whereas we focus on
ResNets) – as they solve a full augmented Lagrangian minimization in between each compressionstep, successful training of their LC algorithm will at least require a careful tuning of this inneroptimization procedure.
C Notation
Throughout the paper, we let θ ∈ R
d denote the parameters of a neural network, L ( θ ) denote the lossfunction over the entire dataset (the empirical risk), and (cid:101) ∇ L denote the stochastic gradient of L (e.g.over a minibatch).
For the regularization method, we denote the set of quantized parameters by Q ,the regularizer by R ( θ ) and the regularization strength by λ .
The learning rates are denoted by η t fort ≥ 0 .
We let Proj S :
R d → R d denote the standard Euclidean projection onto a set S ⊂ R d andprox f denote the proximal operator w.r.t.
function f
(details in Appendix D.3).
The p -Wassersteindistance is denoted as W p .
We will restrict attention to Wasserstein distances on R and p ∈ { 1 , 2 } .
D Preliminaries
The optimization difﬁculty of training quantized models is that they involve a discrete parameter spaceand hence efﬁcient local-search methods are often prohibitive.
For example, the problem of traininga binary neural network is to minimize L ( θ ) for θ ∈ {± 1 } d .
Projected SGD on this set will notmove unless with an unreasonably large stepsize [Li et al., 2017], whereas greedy nearest-neighborsearch requires d forward passes which is intractable for neural networks where d is on the orderof millions.
Alternatively, quantized training can also be cast as minimizing L ( q ( θ )) for θ ∈ R dand an appropriate quantizer q that maps a real vector to a nearby quantized vector, but θ (cid:55)→ q ( θ ) isoften non-differentiable and piecewise constant (such as the binary case q ( · ) = sign ( · ) ), and thusback-propagation through q does not work.
D.1 The straight-through gradient method
The pioneering work of BinaryConnect [Courbariaux et al., 2015] proposes to solve this problem viathe straight-through gradient method , that is, propagate the gradient with respect to q ( θ ) unaltered to θ , i.e. to let ∂L∂θ :
= ∂L∂ q ( θ ) .
One iterate of the straight-through gradient method (with the SGDoptimizer) is
[Equation]
This enables the real vector θ to move in the entire Euclidean space, and taking q ( θ ) at the end oftraining gives a valid quantized model.
Such a customized back-propagation rule yields good empirical performance in training quantized nets and has thus become a standard practice [Courbariauxet al., 2015, Zhu et al., 2016, Xu et al., 2018].
However, as we have discussed, it is informationtheoretically unclear how the straight-through method works, and it does fail on very simple convex
Lipschitz functions (Figure 1b).
D.2 Straight-through gradient as lazy projection
Our ﬁrst observation is that the straight-through gradient method is equivalent to Nesterov’s dualaveraging method, or a lazy projected SGD [Xiao, 2010].
In the binary case, we wish to minimize
L ( θ ) over Q = {± 1 } d , and the lazy projected SGD proceeds as
[Equation]
Written compactly, this is θ t +
=
θ t − η t [Equation] ∇ L ( θ ) | θ = q ( θ t ) , which is exactly the straight-through gradientmethod: take the gradient at the quantized vector and perform the update on the original real vector.
D.3 Projection as a limiting proximal operator
We take a broader point of view that a projection is also a limiting proximal operator with a suitableregularizer, to allow more generality and to motivate our proposed algorithm.
Given any set Q , onecould identify a regularizer R : R d → R ≥ 0 such that the following hold:
[Equation]
In the case Q = {± 1 } d for example, one could take
[Equation]
The proximal operator (or prox operator)
[Parikh and Boyd, 2014] with respect to R and strengthλ > 0 is
[Equation]
In the limiting case λ = ∞ , the argmin has to satisfy R ( θ ) = 0 , i.e. θ ∈ Q , and the prox operator isto minimize (cid:107)
θ − θ 0 (cid:107) 22 over θ ∈ Q , which is the Euclidean projection onto Q .
Hence, projection isalso a prox operator with λ = ∞ , and the straight-through gradient estimate is equivalent to a lazyproximal gradient descent with and λ = ∞ .
While the prox operator with λ = ∞ correponds to “hard” projection onto the discrete set Q , whenλ < ∞ it becomes a “soft” projection that moves towards Q .
Compared with the hard projection,a ﬁnite λ is less aggressive and has the potential advantage of avoiding overshoot early in training.
Further, as the prox operator does not strictly enforce quantizedness, it is in principle able to querythe gradients at every point in the space, and therefore has access to more information than thestraight-through gradient method.
E Details on the P ROX Q UANT algorithmE.1 Regularization for model quantization
We deﬁne a ﬂexible class of quantization-inducing regularizers through “distance to the quantizedset”, derive efﬁcient algorithms of their corresponding prox operator, and propose a homotopy method for choosing the regularization strengths.
Our regularization perspective subsumes most existingalgorithms for model-quantization (e.g.,[Courbariaux et al., 2015, Han et al., 2015, Xu et al.
, 2018])as limits of certain regularizers with strength λ → ∞ .
Our proposed method can be viewed as aprincipled generalization of these methods to λ < ∞ .
Let Q ⊂ R d be a set of quantized parameter vectors.
An ideal regularizer for quantization would beto vanish on Q and reﬂect some type of distance to Q when θ / ∈ Q .
To achieve this, we propose L 1and L 2 regularizers of the form
[Equation]
This is a highly ﬂexible framework for designing regularizers, as one could specify any Q and choosebetween L 1 and L 2 .
Speciﬁcally, Q encodes certain desired quantization structure.
By appropriatelychoosing Q , we can specify which part of the parameter vector to quantize 1 , the number of bits toquantize to, whether we allow adaptively-chosen quantization levels and so on.
The choice of distance metrics will result in distinct properties in the regularized solutions.
Forexample, choosing the L 1 version leads to non-smooth regularizers that induce exact quantizednessin the same way that L 1 norm regularization induces sparsity [Tibshirani, 1996], whereas choosingthe squared L 2 version leads to smooth regularizers that induce quantizedness “softly”.
In the following, we present a few examples of regularizers under our framework eq.
(7) which inducebinary weights, ternary weights and multi-bit quantization.
We will also derive efﬁcient algorithms(or approximation heuristics) for solving the prox operators corresponding to these regularizers,which generalize the projection operators used in the straight-through gradient algorithms.
Binary neural nets
In a binary neural net, the entries of θ are in {± 1 } .
A natural choice would betaking Q = {− 1 , 1 } d .
The resulting L 1 regularizer is
[Equation]
This is exactly the binary regularizer R bin that we discussed earlier in eq.
Figure 2 plots the
W-shaped one-dimensional component of R bin from which we see its effect for inducing {± 1 }quantization in analog to L 1 regularization for inducing exact sparsity.
[Figure]
The prox operator with respect to R bin , despite being a non-convexoptimization problem, admits a simple analytical solution:
[Equation]
We note that the choice of the L 1 version is not unique: the squared
L 2 version works as well, whose prox operator is given by ( θ +λ sign ( θ )) /
(1 + λ ) .
Multi-bit quantization with adaptive levels.
Following [Xuet al., 2018], we consider k -bit quantized parameters with a structured adaptively-chosen set of quantization levels, which translatesinto
Q = (cid:40)
[Equation]
The squared L 2 regularizer for this structure is
[Equation]
which is also the alternating minimization objective in [Xu et al., 2018].
We now derive the prox operator for the regularizer eq.
For any θ , we have
[Equation]
This is a joint minimization problem in ( (cid:101) θ, B, α ) , and we adopt an alternating minimization scheduleto solve it:
(1) Minimize over (cid:101) θ given ( B, α ) , which has a closed-form solution
[Equation]
(2) Minimize over ( B, α ) given (cid:101) θ , which does not depend on θ 0 , and can be done via calling thealternating quantizer of [Xu et al., 2018]: Bα = q alt ( (cid:101) θ ) .
Together, the prox operator generalizes the alternating minimization procedure in [Xu et al., 2018], asλ governs a trade-off between quantization and closeness to θ .
To see that this is a strict generalization,note that for any λ the solution of eq.
(12) will be an interpolation between the input θ and its Euclideanprojection to Q .
As λ → + ∞ , the prox operator collapses to the projection.
Ternary quantization Ternary quantization is a variant of 2-bit quantization, in which weights areconstrained to be in {− α, 0 , β } for real values α, β > 0 .
For ternary quantization, we use an approximate version of the alternating prox operator eq.
(12):compute [Equation] by initializing at [Equation]
=
[Equation]
and repeating
[Equation]
where q is the ternary quantizer deﬁned as
[Equation]
This is a straightforward extension of the TWN quantizer [Li and Liu, 2016] that allows differentlevels for positives and negatives.
We ﬁnd that two rounds of alternating computation in eq.
(13)achieves a good performance, which we use in our experiments.
E.2 Homotopy method for regularization strength
Recall that the larger λ t is, the more aggressive θ t +1 will move towards the quantized set.
An idealchoice would be to (1) force the net to be exactly quantized upon convergence, and (2) not be tooaggressive such that the quantized net at convergence is sub-optimal.
We let λ t be a linearly increasing sequence, i.e. λ t := λ · t for some hyper-parameter λ > 0 whichwe term as the regularization rate .
With this choice, the stochastic gradient steps will start offclose to full-precision training and gradually move towards exact quantizedness, hence the name“homotopy method”.
The parameter λ can be tuned by minimizing the validation loss, and controlsthe aggressiveness of falling onto the quantization constraint.
There is nothing special about thelinear increasing scheme, but it is simple enough and works well as we shall see in the experiments.
F Experiments on LSTMs
Problem setup We perform language modeling with LSTMs Hochreiter and Schmidhuber [1997]on the Penn Treebank (PTB) dataset [Marcus et al., 1993], which contains 929K training tokens,73K validation tokens, and 82K test tokens.
Our model is a standard one-hidden-layer LSTM withembedding dimension 300 and hidden dimension 300.
We train quantized LSTMs with the encoder,transition matrix, and the decoder quantized to k -bits for k ∈ { 1 , 2 , 3 } .
The quantization is performedin a row-wise fashion, so that each row of the matrix has its own codebook { α 1 , . . .
, α k } .
Method We compare our multi-bit P ROX Q UANT to the state-of-the-art alternating minimizationalgorithm with straight-through gradients [Xu et al., 2018].
Training is initialized at a pre-trainedfull-precision LSTM.
We use the SGD optimizer with initial learning rate 20.0 and decay by a factorof 1.2 when the validation error does not improve over an epoch.
We train for 80 epochs withbatch size 20, BPTT 30, dropout with probability 0.5, and clip the gradient norms to 0 .
Theregularization rate λ is tuned by ﬁnding the best performance on the validation set.
In addition tomulti-bit quantization, we also report the results for binary LSTMs (weights in {± 1 } ), comparing
BinaryConnect and our P ROX Q UANT -Binary.
Result We report the perplexity-per-word (PPW, lower is better) in Table 2.
The performanceof P ROX Q UANT is comparable with the Straight-through gradient method.
On Binary LSTMs,
P ROX Q UANT -Binary beats BinaryConnect by a large margin.
These results demonstrate that P ROXQ UANT offers a powerful alternative for training recurrent networks. [Table]
G Sign change experiment
We experimentally compare the training dynamics of P ROX Q UANT -Binary and BinaryConnectthrough the sign change metric.
The sign change metric between any θ 1 and θ 2 is the proportion oftheir different signs, i.e. the (rescaled) Hamming distance:
SignChange ( θ 1 , θ 2 ) = (cid:107) sign ( θ 1 ) − sign ( θ 2 ) (cid:107)2 d ∈ [0 , 1] .
In R d , the space of all full-precision parameters, the sign change is a natural distance metric thatrepresents the closeness of the binarization of two parameters.
Recall in our CIFAR-10 experiments (Section 3.1), for both BinaryConnect and P ROX Q UANT , weinitialize at a good full-precision net θ 0 and stop at a converged binary network (cid:98) θ ∈ {± 1 } d .
Weare interested in SignChange ( θ 0 , θ t ) along the training path, as well as SignChange ( θ 0 , (cid:98) θ ) , i.e. thedistance of the ﬁnal output model to the initialization.
As P ROX Q UANT converges to higher-performance solutions than BinaryConnect, we expect that ifwe run both methods from a same warm start, the sign change of P ROX Q UANT should be higher thanthat of BinaryConnect, as in general one needs to travel farther to ﬁnd a better net.
However, we ﬁnd that this is not the case: P ROX Q UANT produces binary nets with both lower signchanges and higher performances, compared with BinaryConnect.
This ﬁnding is consistent inall layers, across different warm starts, and across differnent runs from each same warm start (see
Figure 3 and Table 3 in Appendix G.1).
This shows that for every warm start position, there is agood binary net nearby which can be found by P ROX Q UANT but not BinaryConnect, suggesting that
BinaryConnect, and in general the straight-through gradient method, suffers from higher optimizationinstability than P ROX Q UANT .
This result here is also consistent with Theorem 4.1: the signs in
BinaryConnect never stop changing until we manually freeze the signs at epoch 400.
G.1 Detailed sign change results on ResNet-
[Figure]
[Table] [Table]