Original: R IEMANNIAN M ANIFOLD E MBEDDINGS FOR S TRAIGHT -T HROUGH E STIMATOR
Revised: R IEMANNIAN M ANIFOLD E MBEDDINGS FOR S TRAIGHT -T HROUGH E STIMATOR

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Quantized Neural Networks (QNNs) aim at replacing full-precision weights W with quantized weights ˆ W , which make it possible to deploy large models to mobile and miniaturized devices easily.
Revised: Quantized Neural Networks (QNNs) aim at replacing full-precision weights W with quantized weights ˆ W , which make it possible to deploy large models to mobile and miniaturized devices easily.

Original: However, either inﬁnite or zero gradients caused by non-differentiable quantization signiﬁcantly affect the training of quantized models.
Revised: However, either inﬁnite or zero gradients caused by non-differentiable quantization signiﬁcantly affect the training of quantized models.

Original: In order to address this problem, most training-based quantization methods use Straight-Through Estimator (STE) to approximate gradients ∇ W w.r.t.
Revised: In order to address this problem, most training-based quantization methods use Straight-Through Estimator (STE) to approximate gradients ∇ W w.r.t.

Original: W with gradients ∇ ˆ W w.r.t.
Revised: W with gradients ∇ ˆ W w.r.t.

Original: ˆ W where the premise is that W must be clipped to [ − 1 , +1] .
Revised: ˆ W where the premise is that W must be clipped to [ − 1 , +1] .

Original: However, the simple application of STE brings with the gradient mismatch problem, which affects the stability of the training process.
Revised: However, the simple application of STE brings with the gradient mismatch problem, which affects the stability of the training process.

Original: In this paper, we propose to revise an approximated gradient for penetrating the quantization function with manifold learning.
Revised: In this paper, we propose to revise an approximated gradient for penetrating the quantization function with manifold learning.

Original: Speciﬁcally, by viewing the parameter space as a metric tensor in the Riemannian manifold, we introduce the Manifold Quantization (ManiQuant) via revised STE to alleviate the gradient mismatch problem.
Revised: Speciﬁcally, by viewing the parameter space as a metric tensor in the Riemannian manifold, we introduce the Manifold Quantization (ManiQuant) via revised STE to alleviate the gradient mismatch problem.

Original: The ablation studies and experimental results demonstrate that our proposed method has a better and more stable performance with various deep neural networks on CIFAR10/100 and ImageNet datasets.
Revised: The ablation studies and experimental results demonstrate that our proposed method has a better and more stable performance with various deep neural networks on CIFAR10/100 and ImageNet datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks can handle many complex tasks due to their large number of trainable parameters and strong nonlinear capabilities (Krizhevsky et al., 2012).
Revised: Neural networks can handle many complex tasks due to their large number of trainable parameters and strong nonlinear capabilities (Krizhevsky et al., 2012).

Original: However, the massive amount of models and calculations hinder the application of neural networks on mobile and miniaturized devices, which naturally comes with constraints on computing power and resources.
Revised: However, the massive amount of models and calculations hinder the application of neural networks on mobile and miniaturized devices, which naturally comes with constraints on computing power and resources.

Original: Neural network quantization is considered an efﬁcient solution in the inference that alleviates the number of parameters and optimizes the computation by reducing the bit width of weights or activations (Courbariaux et al., 2016; Li et al., 2016; Zhu et al., 2016).
Revised: Neural network quantization is considered an efﬁcient solution in the inference that alleviates the number of parameters and optimizes the computation by reducing the bit width of weights or activations (Courbariaux et al., 2016; Li et al., 2016; Zhu et al., 2016).

Original: Existing neural network quantization methods can be roughly divided into two categories: “STE” and “Non-STE” methods.
Revised: Existing neural network quantization methods can be roughly divided into two categories: “STE” and “Non-STE” methods.

Original: Most of the quantization methods adopted by the QNNs belong to the former, i.e. there is always a non-differentiable quantization function during training.
Revised: Most of the quantization methods adopted by the QNNs belong to the former, i.e. there is always a non-differentiable quantization function during training.

Original: The role of
Revised: The role of

Original: STE is to penetrate this non-differentiable quantization function and pass the gradients in backpropogation (Hinton, 2012), e.g. DeepShift (Elhoushi et al., 2019), INT8 (Zhu et al., 2020), AQE (Chen et al., 2020), etc.
Revised: STE is to penetrate this non-differentiable quantization function and pass the gradients in backpropogation (Hinton, 2012), e.g. DeepShift (Elhoushi et al., 2019), INT8 (Zhu et al., 2020), AQE (Chen et al., 2020), etc.

Original: “Non-STE” methods refer to maintain feasible quantization during training, which does not need to apply STE directly to all full-precision weights.
Revised: “Non-STE” methods refer to maintain feasible quantization during training, which does not need to apply STE directly to all full-precision weights.

Original: For example, Zhou et al.
Revised: For example, Zhou et al.

Original: (Zhou et al., 2017) divided the weights into two groups until all parameters are quantized, where the ﬁrst group is directly quantized and ﬁxed; the second group needs to be retrained to make up for the decrease of accuracy caused by quantization of the ﬁrst group.
Revised: (Zhou et al., 2017) divided the weights into two groups until all parameters are quantized, where the ﬁrst group is directly quantized and ﬁxed; the second group needs to be retrained to make up for the decrease of accuracy caused by quantization of the ﬁrst group.

Original: Louizos et al.
Revised: Louizos et al.

Original: (Louizos et al., 2019) introduced a differentiable quantizer that can transform the continuous distributions of weights and activations to categorical distributions with gradient-based optimization.
Revised: (Louizos et al., 2019) introduced a differentiable quantizer that can transform the continuous distributions of weights and activations to categorical distributions with gradient-based optimization.

Original: However, “Non-STE” methods face the setting and inﬂuence of heavy hyper-parameters in the training process.
Revised: However, “Non-STE” methods face the setting and inﬂuence of heavy hyper-parameters in the training process.

Original: Relatively, “STE” methods are wider choices for quantized models from simplicity and versatility.
Revised: Relatively, “STE” methods are wider choices for quantized models from simplicity and versatility.

Original: To approximate ∇ W w.r.t.
Revised: To approximate ∇ W w.r.t.

Original: W and complete stable quantization training, Courbariaux et al.
Revised: W and complete stable quantization training, Courbariaux et al.

Original: (Courbariaux et al., 2016) binarized the neural networks by the approximated gradients using STE:
Revised: (Courbariaux et al., 2016) binarized the neural networks by the approximated gradients using STE:

Original: [Equation]
Revised: [Equation]

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: otherwise .
Revised: otherwise .

Original: However, it will inevitably bring the gradient mismatch problem raised by just simply application of
Revised: However, it will inevitably bring the gradient mismatch problem raised by just simply application of

Original: STE.
Revised: STE.

Original: To overcome this challenge, Zhou et al.
Revised: To overcome this challenge, Zhou et al.

Original: (Zhou et al., 2016) proposed to transform W to ˜ W :
Revised: (Zhou et al., 2016) proposed to transform W to ˜ W :

Original: [Equation]
Revised: [Equation]

Original: and then quantize it using a quantization function Q ( · ) .
Revised: and then quantize it using a quantization function Q ( · ) .

Original: During backpropogation, the gradients ∇ W w.r.t.
Revised: During backpropogation, the gradients ∇ W w.r.t.

Original: W can be further computed using the chain rule:
Revised: W can be further computed using the chain rule:

Original: [Equation]
Revised: [Equation]

Original: Based on this work, Chen et al.
Revised: Based on this work, Chen et al.

Original: (Chen et al., 2019) proposed to learn ∇ W by fully-connected neural networks or LSTM, which replaces this gradients via a meta quantizer M φ parameterized by φ across layers:
Revised: (Chen et al., 2019) proposed to learn ∇ W by fully-connected neural networks or LSTM, which replaces this gradients via a meta quantizer M φ parameterized by φ across layers:

Original: [Equation]
Revised: [Equation]

Original: However, such methods not only add many additional parameters, but also increase the difﬁculty of the training of quantized models.
Revised: However, such methods not only add many additional parameters, but also increase the difﬁculty of the training of quantized models.

Original: In this paper, we introduce the Manifold Quantization (ManiQuant) to train a quantized model via embedding Riemannian manifolds for STE.
Revised: In this paper, we introduce the Manifold Quantization (ManiQuant) to train a quantized model via embedding Riemannian manifolds for STE.

Original: In Figure 1, we treat the parameter space in a quantized model as a Riemannian manifold, which will alleviate the gradient mismatch problem caused by non-differentiable quantization and assist quantized models in achieving a more stable convergence and better performance.
Revised: In Figure 1, we treat the parameter space in a quantized model as a Riemannian manifold, which will alleviate the gradient mismatch problem caused by non-differentiable quantization and assist quantized models in achieving a more stable convergence and better performance.

Original: The main contributions of this work are three-fold: First , weattribute the gradient mismatch problem to Cram´er-Rao Lower Bound of the variance of STE, and propose to use Fisher Information Matrix embedding to alleviate it.
Revised: The main contributions of this work are three-fold: First , we propose to use Fisher Information Matrix embedding to alleviate the gradient mismatch problem.

Original: Second , we deﬁne a novel Hyperbolic divergence by a convex function with geometric structure.
Revised: Second , we deﬁne a novel Hyperbolic divergence by a convex function with geometric structure.

Original: With the constraint of Hyperbolic divergence, we introduce a weak curvature manifold that forms the background of ManiQuant.
Revised: With the constraint of Hyperbolic divergence, we introduce a weak curvature manifold that forms the background of ManiQuant.

Original: Third , based on the Second , we propose to use weak curvature metric embeddings for STE as the weak curvature gradient to approximate ∇ W without extra parameters and training complexity.
Revised: Third , based on the Second , we propose to use weak curvature metric embeddings for STE as the weak curvature gradient to approximate ∇ W without extra parameters and training complexity.

Original:  R ELATED W ORK 2. G ENERAL G RADIENT
Revised:  R ELATED W ORK 2. G ENERAL G RADIENT

Original: For a neural network with l layers, the full-precision weight matrix of each layer is just marked as W i .
Revised: For a neural network with l layers, the full-precision weight matrix of each layer is just marked as W i .

Original: By deﬁning the operation vec( · ) that vectorizes matrices by stacking their columns together, the total parameter vector of the neural network can be denoted as θ = (cid:104) vec ( W 1 ) (cid:62) , vec ( W 2 ) (cid:62) , . .
Revised: By deﬁning the operation vec( · ) that vectorizes matrices by stacking their columns together, the total parameter vector of the neural network can be denoted as θ = (cid:104) vec ( W 1 ) (cid:62) , vec ( W 2 ) (cid:62) , . .

Original: , vec ( W l ) (cid:62) (cid:105) (cid:62) .
Revised: , vec ( W l ) (cid:62) (cid:105) (cid:62) .

Original: The parameter vector is given as a column vector.
Revised: The parameter vector is given as a column vector.

Original: Let θ ∈ R n be a parameter space on which a loss function L associated with weight is well-deﬁned.
Revised: Let θ ∈ R n be a parameter space on which a loss function L associated with weight is well-deﬁned.

Original: It is relatively easy to express the general gradient in training:
Revised: It is relatively easy to express the general gradient in training:

Original: [Equation]
Revised: [Equation]

Original: which is the steepest descent method in Euclidean space with an orthonormal coordinate.
Revised: which is the steepest descent method in Euclidean space with an orthonormal coordinate.

Original: Note that the negative gradient represents the direction of the steepest descent.
Revised: Note that the negative gradient represents the direction of the steepest descent.

Original: When the Euclidean space is considered, the Euclidean divergence between two sufﬁciently closed points θ and θ (cid:48) is actually deﬁned by default:
Revised: When the Euclidean space is considered, the Euclidean divergence between two sufﬁciently closed points θ and θ (cid:48) is actually deﬁned by default:

Original: [Equation]
Revised: [Equation]

Original: which is a half of the square of the Euclidean distance identiﬁed by the Euclidean metric δ ij , so that
Revised: which is a half of the square of the Euclidean distance identiﬁed by the Euclidean metric δ ij , so that

Original: [Equation]
Revised: [Equation]

Original: 2. N ATURAL G RADIENT
Revised: 2. N ATURAL G RADIENT

Original: The general gradient only considers the parameter update along the gradient direction that does not involve the metric tensor for the parameter space of the problem.
Revised: The general gradient only considers the parameter update along the gradient direction that does not involve the metric tensor for the parameter space of the problem.

Original: Now, we attach a Riemannian metric G ij ( θ ) for the parameter space to form a Riemannian manifold ( M , G ij ) (Amari, 1998; 2016).
Revised: Now, we attach a Riemannian metric G ij ( θ ) for the parameter space to form a Riemannian manifold ( M , G ij ) (Amari, 1998; 2016).

Original: In that case, the steepest descent direction also depends on the quadratic form introduced by a small incremental vector d θ that connects θ and θ + d θ , whose form is given by
Revised: In that case, the steepest descent direction also depends on the quadratic form introduced by a small incremental vector d θ that connects θ and θ + d θ , whose form is given by

Original: [Equation]
Revised: [Equation]

Original: where dθ i is the component of d θ .
Revised: where dθ i is the component of d θ .

Original: Under the constraint ds 2 , the steepest descent direction is toward the optimization goal of L ( θ + d θ ) .
Revised: Under the constraint ds 2 , the steepest descent direction is toward the optimization goal of L ( θ + d θ ) .

Original: Intuitively, it measures the Kullback-Leibler (KL) divergence between two distributions p ( x | θ ) and p ( x | θ + d θ ) of network model, which is equivalent to the interval of two adjacent points on a Riemannian manifold.
Revised: Intuitively, it measures the Kullback-Leibler (KL) divergence between two distributions p ( x | θ ) and p ( x | θ + d θ ) of network model, which is equivalent to the interval of two adjacent points on a Riemannian manifold.

Original: The KL divergence under the Riemannian metric is well approximated through the Fisher Information Matrix (FIM) with the second-order
Revised: The KL divergence under the Riemannian metric is well approximated through the Fisher Information Matrix (FIM) with the second-order

Original: Taylor expansion of the KL divergence (see Appendix B.3)
Revised: Taylor expansion of the KL divergence (see Appendix B.3)

Original: (Ba et al., 2016):
Revised: (Ba et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: We see that FIM is equal to the negative expected Hessian of log likelihood.
Revised: We see that FIM is equal to the negative expected Hessian of log likelihood.

Original: Furthermore, Amari deduced that the Riemannian metric is given by the FIM (Amari, 1998).
Revised: Furthermore, Amari deduced that the Riemannian metric is given by the FIM (Amari, 1998).

Original: By the Lagrangian form, we have the natural gradient (see Appendix C.1)
Revised: By the Lagrangian form, we have the natural gradient (see Appendix C.1)

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: which is the steepest descent method in a Riemannian space and F ( θ ) is the FIM with a parameter vector (the natural gradient also known as the Riemannian gradient).
Revised: which is the steepest descent method in a Riemannian space and F ( θ ) is the FIM with a parameter vector (the natural gradient also known as the Riemannian gradient).

Original: Empirically, the immediate application of FIM is as drop-in replacement of Hessian in second order optimization algorithm.
Revised: Empirically, the immediate application of FIM is as drop-in replacement of Hessian in second order optimization algorithm.

Original: Note that ˜ ∇ θ L will return to ∇ θ L when G ij ( θ ) is equal to the Euclidean metric δ ij , i.e. the identity matrix I .
Revised: Note that ˜ ∇ θ L will return to ∇ θ L when G ij ( θ ) is equal to the Euclidean metric δ ij , i.e. the identity matrix I .

Original:  M ANIFOLD Q UANTIZATION
Revised:  M ANIFOLD Q UANTIZATION

Original: Considering a quantized model of each layer, we notate the quantized weight matrix using ˆ W i for differentiating full-precision weight matrix W i .
Revised: Considering a quantized model of each layer, we notate the quantized weight matrix using ˆ W i for differentiating full-precision weight matrix W i .

Original: Note that we deﬁne the quantized parameter vector as ˆ θ , which is similar to the full-precision parameter vector θ .
Revised: Note that we deﬁne the quantized parameter vector as ˆ θ , which is similar to the full-precision parameter vector θ .

Original: During training a QNN, the quantization function Q ( · ) is a one-to-one mapping from full-precision values to quantized values, which can be expressed as ˆ θ = Q ( θ ) .
Revised: During training a QNN, the quantization function Q ( · ) is a one-to-one mapping from full-precision values to quantized values, which can be expressed as ˆ θ = Q ( θ ) .

Original: By involving the practice to train a neural network with low bit-width, the process of the quantization needs to be further designed that plays a vital role in the ﬁnal performance, as for the other parts, e.g. via mapping from an input pattern
Revised: By involving the practice to train a neural network with low bit-width, the process of the quantization needs to be further designed that plays a vital role in the ﬁnal performance, as for the other parts, e.g. via mapping from an input pattern

Original: ˆ a
Revised: ˆ a

Original: i − 1 to output ˆ a i can still be imitated in the same way as full-precision neural networks:
Revised: i − 1 to output ˆ a i can still be imitated in the same way as full-precision neural networks:

Original: [Equation]
Revised: [Equation]

Original: where f i is non-linear function acted on element-wise.
Revised: where f i is non-linear function acted on element-wise.

Original: To distinguish the full-precision model, we mark the notation “ˆ” to represent the operation through a quantization function in a quantized model, whether for weight matrix ˆ W i or activation vector ˆ a i .3. T HE R OOT OF G RADIENT M ISMATCH Theorem 1 The derivative of the loss function w.r.t. the binary neuron 1 can be estimated by
Revised: To distinguish the full-precision model, we mark the notation “ˆ” to represent the operation through a quantization function in a quantized model, whether for weight matrix ˆ W i or activation vector ˆ a i .

Original: Straight-Through Estimator

Original: [Equation]

Original: where the noise source (cid:15) obeys the Uniform distribution from − 1 to 1 : (cid:15) ∼ U ( − 1 , 1) .

Original: Simultaneously, this estimator in [ − 1 , 1] is unbiased.

Original: Proof .

Original: The proofs can be found in Appendix A.

Original: [Equation]

Original: Deﬁnition 1 (Cram´er-Rao Lower Bound)

Original: If ∂L/∂a is an unbiased estimator of ∂L/∂ ˆ a , where the probability density function p ( x, ∂L/∂ ˆ a ) satisﬁes some regularity conditions 3 , then there exists the variance of an unbiased estimator ∂L/∂a such that

Original: [Equation]

Original: where I

Original: [Equation]

Original: Deﬁnition 1 implies that the variance of any unbiased estimator is bounded by the reciprocal of the

Original: Fisher information I ( · ) .

Original: Combining with Theorem 1, we see that the root of gradient mismatch comes from Cram ´ er-Rao Lower Bound of the variance of STE.

Original: In other words, increasing the Fisher information can alleviate the gradient mismatch problem to a certain extent.

Original: Next, we will achieve this goal by embedding Riemannian manifolds into the STE.

Original: 3. F ISHER I NFORMATION M ATRIX E MBEDDING FOR STE
Revised: 3. F ISHER I NFORMATION M ATRIX E MBEDDING FOR STE

Original: The concept of the natural gradient is closely related to the FIM and KL divergence (see Appendix B.3).
Revised: The concept of the natural gradient is closely related to the FIM and KL divergence (see Appendix B.3).

Original: Since the KL divergence is intrinsic, the natural gradient is also intrinsically invariant under the parameter transformation.
Revised: Since the KL divergence is intrinsic, the natural gradient is also intrinsically invariant under the parameter transformation.

Original: By viewing the FIM as an l by l block matrix where l denotes the number of layers in a neural network, the natural gradient formula can be introduced to alleviate the gradient mismatch problem when training a QNN and updating its parameters.
Revised: By viewing the FIM as an l by l block matrix where l denotes the number of layers in a neural network, the natural gradient formula can be introduced to alleviate the gradient mismatch problem when training a QNN and updating its parameters.

Original: Lemma 

Revised: Lemma

Original: In the Riemannian manifold deﬁned by KL divergence, Fisher Information Matrix embedding for Straight-Through Estimator is written by:
Revised: In the Riemannian manifold deﬁned by KL divergence, Fisher Information Matrix embedding for Straight-Through Estimator is written by:

Original: [Equation]
Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: Furthermore, F − 1 ( ˆ θ ) can be expressed as
Revised: Furthermore, F − 1 ( ˆ θ ) can be expressed as

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:  −

Original: [Equation]
Revised: [Equation]

Original: where vec (cid:16) ∂L∂
Revised: where vec (cid:16) ∂L∂

Original: ˆ W i (cid:17) can be represented by the gradient error ∂L∂ a i .
Revised: ˆ W i (cid:17) can be represented by the gradient error ∂L∂ a i .

Original: Proof .
Revised: Proof .

Original: The proofs are the combination between Theorem 1 and Appendix C.1.
Revised: The proofs are the combination between STE and Appendix C.1.

Original: (cid:3)
Revised: (cid:3)

Original: Considering that the gradient propagation needs to span over the quantized neurons and graphs, we still need to use STE(see Theorem 1) to update the gradient of QNNs in the learning procedure:
Revised: Considering that the gradient propagation needs to span over the quantized neurons and graphs [Equation] we still need to use STE to update the gradient of QNNs in the learning procedure:

Original: [Equation]
Revised: [Equation]

Original: where ⊗ denotes the Kronecker product 4 .
Revised: where ⊗ denotes the Kronecker product 1 .

Original: Relying on the KL divergence, we indicate the parameter space as the Riemannian manifold rather than the Euclidean space in the quantization procedure.
Revised: Relying on the KL divergence, we indicate the parameter space as the Riemannian manifold rather than the Euclidean space in the quantization procedure.

Original: For neural networks with a scale of one million or more parameters, the time complexity of inverting the FIM, a component of natural gradients, is O ( n 3 ) (Povey et al., 2014).
Revised: For neural networks with a scale of one million or more parameters, the time complexity of inverting the FIM, a component of natural gradients, is O ( n 3 ) (Povey et al., 2014).

Original: Previously, there were some works to calculate the natural gradient efﬁciently, e.g. Roux et al.
Revised: Previously, there were some works to calculate the natural gradient efﬁciently, e.g. Roux et al.

Original: (Roux et al., 2008) decomposed the FIM into multiple diagonal blocks, where each diagonal block is approximated by a low-rank matrix.
Revised: (Roux et al., 2008) decomposed the FIM into multiple diagonal blocks, where each diagonal block is approximated by a low-rank matrix.

Original: Bastian et al.
Revised: Bastian et al.

Original: (Bastian et al., 2011) also used the idea of diagonal blocks by constructing a diagonal block that corresponds to a weight matrix.
Revised: (Bastian et al., 2011) also used the idea of diagonal blocks by constructing a diagonal block that corresponds to a weight matrix.

Original: Martens et al.
Revised: Martens et al.

Original: (Martens & Grosse,2015) proposed to approximate the FIM through the Kronecker product of two smaller matrices to improve computational efﬁciency.
Revised: (Martens & Grosse, proposed to approximate the FIM through the Kronecker product of two smaller matrices to improve computational efﬁciency.

Original: Even so, FIM embedding for STE with complex decomposition methods is still not suitable for large-scale tasks, and the computational amount is large compared to the standard STE.
Revised: Even so, FIM embedding for STE with complex decomposition methods is still not suitable for large-scale tasks, and the computational amount is large compared to the standard STE.

Original: 3. W EAK C URVATURE M ETRIC E MBEDDING FOR STE 3.3. I NTRODUCTION OF W EAK C URVATURE M ANIFOLD
Revised: 3. W EAK C URVATURE M ETRIC E MBEDDING FOR STE 3.2. I NTRODUCTION OF W EAK C URVATURE M ANIFOLD

Original: We can use the Euclidean coordinates to calculate the natural gradient on the manifold because of the local homeomorphism of a manifold (Wald, 2010).
Revised: We can use the Euclidean coordinates to calculate the natural gradient on the manifold because of the local homeomorphism of a manifold (Wald, 2010).

Original: Note that the homeomorphic mapping φ U satisﬁes
Revised: Note that the homeomorphic mapping φ U satisﬁes

Original: U ∈ M
Revised: U ∈ M

Original: (cid:55)→ φ U ( U ) ∈ R n .
Revised: (cid:55)→ φ U ( U ) ∈ R n .

Original: For any point x ∈ U , we can deﬁne φ U ( x ) as the Euclidean coordinate absolutely.
Revised: For any point x ∈ U , we can deﬁne φ U ( x ) as the Euclidean coordinate absolutely.

Original: The FIM embedding for STE poses a signiﬁcant challenge for computing.
Revised: The FIM embedding for STE poses a signiﬁcant challenge for computing.

Original: Intuitively, the computing F − 1 ( ˆ θ ) attached to the natural gradient will take on massive computation, which can only be numerically estimated.
Revised: Intuitively, the computing F − 1 ( ˆ θ ) attached to the natural gradient will take on massive computation, which can only be numerically estimated.

Original: The inversion is mostly unrealistic when deep neural networks are very redundant, with tens of thousands of neural connections.
Revised: The inversion is mostly unrealistic when deep neural networks are very redundant, with tens of thousands of neural connections.

Original: Inspired by information geometry (Amari, 2016) and mirror descent (Bubeck, 2015), we propose a weak curvature manifold with the weak curvature metric, which implies that the gradient mismatch problem can be alleviated by embedding the weak curvature manifold into the STE, while keeping a low computational complexity.
Revised: Inspired by information geometry (Amari, 2016) and mirror descent (Bubeck, 2015), we propose a weak curvature manifold with the weak curvature metric, which implies that the gradient mismatch problem can be alleviated by embedding the weak curvature manifold into the STE, while keeping a low computational complexity.

Original: Geometrically, the Riemannian manifold is nearly ﬂat, where the Riemannian metric is an approximation of the Euclidean metric.
Revised: Geometrically, the Riemannian manifold is nearly ﬂat, where the Riemannian metric is an approximation of the Euclidean metric.

Original: In practice, we develop a linearized Riemannian metric from the Euclidean metric δ ij , which is systematically deﬁned in general relativity (Wald, 2010).
Revised: In practice, we develop a linearized Riemannian metric from the Euclidean metric δ ij , which is systematically deﬁned in general relativity (Wald, 2010).

Original: Deﬁnition 2 (Weak Curvature Metric) Let a linearized Riemannian metric deviates from the Euclidean metric.
Revised: Deﬁnition 1 (Weak Curvature Metric) Let a linearized Riemannian metric deviates from the Euclidean metric.

Original: The deviated metric (cid:15) ij ( ˆ θ ) is much smaller than 1 in global Euclidean coordinate system of δ ij ,
Revised: The deviated metric (cid:15) ij ( ˆ θ ) is much smaller than 1 in global Euclidean coordinate system of δ ij ,

Original: [Equation]
Revised: [Equation]

Original:  where [Equation]
Revised: , where (cid:12)(cid:12) (cid:15) ij (cid:12)(cid:12) (cid:28) 1 .

Original: It is an adequate deﬁnition of “weakness” in this context and ensures G ij ( ˆ θ ) to be a positive-deﬁnite metric.
Revised: It is an adequate deﬁnition of “weakness” in this context and ensures G ij ( ˆ θ ) to be a positive-deﬁnite metric.

Original: 3.3. H YPERBOLIC D IVERGENCE
Revised: 3.2. H YPERBOLIC D IVERGENCE

Original: In practice, we can determine a unique geodesic through exponential map exp θ i ( τ θ ) that maps τ θ back to M where τ is a small constant, by deﬁning τ θ ∈ T θ i M as the tangent vector.
Revised: In practice, we can determine a unique geodesic through exponential map exp θ i ( τ θ ) that maps τ θ back to M where τ is a small constant, by deﬁning τ θ ∈ T θ i M as the tangent vector.

Original: Note that the deﬁnition of exponential map is developed by (Wald, 2010; Helgason, 2001).
Revised: Note that the deﬁnition of exponential map is developed by (Wald, 2010; Helgason, 2001).

Original: Deﬁnition 3 (Exponential Map) Let M be a Riemannian manifold, for the tangent vector v ∈ T x M in a point x ∈ M where T x M is the tangent space, there is a unique geodesic γ v ( t ) locally that satisﬁes γ v (0) =
Revised: Deﬁnition 2 (Exponential Map) Let M be a Riemannian manifold, for the tangent vector v ∈ T x M in a point x ∈ M where T x M is the tangent space, there is a unique geodesic γ v ( t ) locally that satisﬁes γ v (0) =

Original: x and γ (cid:48) v (0)
Revised: x and γ (cid:48) v (0)

Original: = v .
Revised: = v .

Original: The exponential map exp x : T x M (cid:55)→ M corresponding to γ v ( t ) is deﬁned as exp x
Revised: The exponential map exp x : T x M (cid:55)→ M corresponding to γ v ( t ) is deﬁned as exp x

Original: ( v ) =
Revised: ( v ) =

Original: γ v (1) .
Revised: γ v (1) .

Original: When constraining exp x to a neighbourhood U , this mapping is one-to-one.
Revised: When constraining exp x to a neighbourhood U , this mapping is one-to-one.

Original: Empirically, the exponential map exp θ i ( τ θ ) is to map a tangent vector τ θ in the tangent bundle to the point where the arc length from point
Revised: Empirically, the exponential map exp θ i ( τ θ ) is to map a tangent vector τ θ in the tangent bundle to the point where the arc length from point

Original: θ
Revised: θ

Original: i is equal to | τ θ | on geodesic with the initial condition ( θ i , τ θ ) .
Revised: i is equal to | τ θ | on geodesic with the initial condition ( θ i , τ θ ) .

Original: In order to make exp( τ θ ) and exp( − τ θ ) have the same effect in the training process, we symmetrize them and derive a convex function 
Revised: In order to make exp( τ θ ) and exp( − τ θ ) have the same effect in the training process, we symmetrize them and derive a convex function [Equation]

Original: [Equation]
Revised: [Equation]

Original: Geometrically, the convex function with geometric structure is introduced into the Bregman divergence (see Appendix B.2) (Bregman, 1967), and we obtain a novel Hyperbolic divergence that satisﬁes the criteria of divergence (see Appendix B.1).
Revised: Geometrically, the convex function with geometric structure is introduced into the Bregman divergence (see Appendix B.2) (Bregman, 1967), and we obtain a novel Hyperbolic divergence that satisﬁes the criteria of divergence (see Appendix B.1).

Original: Deﬁnition 4 (Hyperbolic Divergence)
Revised: Deﬁnition 3 (Hyperbolic Divergence)

Original: For a convex function ψ deﬁned by Eq.
Revised: For a convex function ψ deﬁned by Eq.

Original: 14, the Hyperbolic divergence between θ (cid:48) and θ is
Revised: 12, the Hyperbolic divergence between θ (cid:48) and θ is

Original: [Equation]
Revised: [Equation]

Original: 3.3. T HE G RADIENT F LOW IN W EAK C URVATURE M ANIFOLD
Revised: 3.2. T HE G RADIENT F LOW IN W EAK C URVATURE M ANIFOLD

Original: Lemma 2 Let d θ → 0 , the square of an inﬁnitesimal distance deﬁned by Hyperbolic divergence is
Revised: Lemma 2 Let d θ → 0 , the square of an inﬁnitesimal distance deﬁned by Hyperbolic divergence is

Original: [Equation]
Revised: [Equation]

Original: Furthermore, the deﬁned metric (cid:2) δ ij − tanh( τ θ ) tanh( τ θ ) (cid:62) (cid:3) can be consistent with the weak curvature metric.
Revised: Furthermore, the deﬁned metric (cid:2) δ ij − tanh( τ θ ) tanh( τ θ ) (cid:62) (cid:3) can be consistent with the weak curvature metric.

Original: Proof .
Revised: Proof .

Original: The proofs can be found in Appendix B.4.
Revised: The proofs can be found in Appendix B.4.

Original: Empirically, we can give a small constant τ to satisfy the weak curvature metric based on Deﬁnition 2.
Revised: Empirically, we can give a small constant τ to satisfy the weak curvature metric based on Deﬁnition 1.

Original: Comparing Eq.
Revised: Comparing Eq.

Original: 4 and Eq. 13, we can deduce the deviated metric (cid:15) 
Revised: 4 and Eq. 11, we can deduce the deviated metric (cid:15) ij = − (cid:2) tanh

Revised: ( τ θ ) tanh( τ θ ) (cid:62) (cid:3) ij .

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: In the Riemannian manifold deﬁned by Hyperbolic divergence, a weak curvature metric embedding for Straight-Through Estimator is approximated by:
Revised: In the Riemannian manifold deﬁned by Hyperbolic divergence, a weak curvature metric embedding for Straight-Through Estimator is approximated by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof .
Revised: Proof .

Original: The proofs can be found in Appendix C.2.
Revised: The proofs can be found in Appendix C.2.

Original: [Equation]
Revised: [Equation]

Original: 3. H IERARCHICAL E MBEDDINGS OF THE W EAK C URVATURE M ETRIC
Revised: 3. H IERARCHICAL E MBEDDINGS OF THE W EAK C URVATURE M ETRIC

Original: When embedding a weak curvature manifold, there is an unnatural gap between the calculation of weak curvature metric and the layer-by-layer gradient update in back-propagation.
Revised: When embedding a weak curvature manifold [Equation] there is an unnatural gap between the calculation of weak curvature metric and the layer-by-layer gradient update in back-propagation.

Original: Speciﬁcally, the metric is not a block diagonal matrix (each block corresponds to a layer of the neural network). [Table] [Equation] [Table]
Revised: Speciﬁcally, the metric is not a block diagonal matrix (each block corresponds to a layer of the neural network)

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: For the process of the quantization
Revised: For the process of the quantization

Original: [Equation]
Revised: [Equation]

Original: we have the hierarchical embeddings for STE through decoupling operation:
Revised: we have the hierarchical embeddings for STE through decoupling operation:

Original: [Equation]
Revised: [Equation]

Original:  In the ﬁrst line of the above formula, we can put τ 2 i outside because the elements of the deviated metric is very small based on Lemma 2.
Revised: [Equation] In the ﬁrst line of the above formula [Equation] we can put τ 2 i outside because the elements of the deviated metric is very small based on Lemma 2.

Original: In the second line, we use a normalized formula to uniformly represent τ 2 i : 
Revised: In the second line [Equation] we use a normalized formula to uniformly represent τ 2 i : τ

Original: τ 2 i = χ/ tr (cid:2) vec(tanh
Revised: [Equation] / tr (cid:2) vec(tanh

Original: ( W i ))
Revised: ( W i ))

Original: vec(tanh
Revised: vec(tanh

Original: ( W i ))
Revised: ( W i ))

Original: (cid:62) (cid:3) where χ is a constant factor and the matrix (cid:2)
Revised: (cid:62) (cid:3) where χ is a constant factor and the matrix (cid:2)

Original: vec(tanh
Revised: vec(tanh

Original: ( W i ))
Revised: ( W i ))

Original: vec(tanh
Revised: vec(tanh

Original: ( W i ))
Revised: ( W i ))

Original: (cid:62) (cid:3) is re-expressed by P i .
Revised: (cid:62) (cid:3) is re-expressed by P i .

Original: By limiting the value range of full-precision weights to [ − 1 , +1] , we obtain the independent weak curvature metric embedding in each layer.
Revised: By limiting the value range of full-precision weights to [ − 1 , +1] , we obtain the independent weak curvature metric embedding in each layer.

Original: The algorithm of this weak curvature gradient with STE is illustrated in Appendix E.
Revised: The algorithm of this weak curvature gradient with STE is illustrated in Appendix E.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In this section, we implement experiments to demonstrate the effectiveness of our proposed methods on benchmark datasets that are CIFAR10/100 and ImageNet mainly here.
Revised: In this section, we implement experiments to demonstrate the effectiveness of our proposed methods on benchmark datasets that are CIFAR10/100 and ImageNet mainly here.

Original: Intuitively, experiments on CIFAR10/100 and ImageNet are ablation studies to validate the advantages of weak curvature manifold embeddings.
Revised: Intuitively, experiments on CIFAR10/100 and ImageNet are ablation studies to validate the advantages of weak curvature manifold embeddings.

Original: Comparisons with other training-based quantization on quantitative indicates will be carried out on CIFAR10 and ImageNet.
Revised: Comparisons with other training-based quantization on quantitative indicates will be carried out on CIFAR10 and ImageNet.

Original: All experiments are conducted with PyTorch.
Revised: All experiments are conducted with PyTorch.

Revised: [Table]

Original: 4. E XPERIMENT S ETUP
Revised: 4. E XPERIMENT S ETUP

Original: We use a weight decay of 1e-4, a batch size of 128, and SGD optimization method with nesterov momentum of 0.9.
Revised: We use a weight decay of 1e-4, a batch size of [Table] and SGD optimization method with nesterov momentum of 0.9.

Original: For CIFAR, we set total training epochs as 200 where the learning strategy is lowered by 10 times at epoch 80, 150, and 190, with the initial0.1.
Revised: For CIFAR, we set total training epochs as 200 where the learning strategy is lowered by 10 times at epoch 80, 150, and 190, with the initial

Revised: [Table]

Original: For ImageNet, we set total training epochs as 50 where the learning strategy is lowered by 10 times at epoch 15, 30, and 45, with the initial 0.1.
Revised: For ImageNet, we set total training epochs as 50 where the learning strategy is lowered by 10 times at epoch 15, [Table] and 45, with the initial 0.1.

Original: Note that we set χ as the reciprocal of output channel in a layer.
Revised: Note that we set χ as the reciprocal of output channel in a layer.

Original: All experiments are conducted for 5 times, and the statistics of the last 10/5 epochs’ test accuracy are reported as a fair comparison. 
Revised: All experiments are conducted for 5 times [Table] and the statistics of the last 10/5 epochs’ test accuracy are reported as a fair comparison. [Table]

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: In order to illustrate the superiority of the weak curvature gradient (WCG), we train QNNs with our gradient from scratch compared to other gradients, i.e. Dorefa (Zhou et al., 2016), MultiFCG (Chen[Table] [Table] et al., 2019) and FCGrad (Chen et al., 2019).
Revised: In order to illustrate the superiority of the weak curvature gradient (WCG), we train QNNs with our gradient from scratch compared to other gradients, i.e. Dorefa (Zhou et al., 2016), MultiFCG (Chen et al., 2019) and FCGrad (Chen et al., 2019).

Original: We choose 1-bit weight quantization for all layers in the networks, which means that each layer contains just − 1 or +1 .
Revised: We choose 1-bit weight quantization for all layers in the networks, which means that each layer contains just − 1 or +1 .

Original: Note that the initialization is Xavier (Glorot & Bengio, 2010).
Revised: Note that the initialization is Xavier (Glorot & Bengio, 2010).

Original: Speciﬁcally, we experiment ResNet20/32/44 models (He et al., 2016) on CIFAR10, ResNet56/110 models (He et al., 2016) on CIFAR100 and ResNet18 model
Revised: Speciﬁcally, we experiment ResNet20/32/44 models (He et al., 2016) on CIFAR10, ResNet56/110 models (He et al., 2016) on CIFAR100 and ResNet18 model

Original: (He et al., 2016) on ImageNet.
Revised: (He et al., 2016) on ImageNet.

Original: The accuracy of full-precision baseline is reported by (Chen et al., 2019).
Revised: The accuracy of full-precision baseline is reported by (Chen et al., 2019).

Original: The results of ablation studies on CIFAR10, CIFAR100 and ImageNet are shown in Table 1, Table 2 and Table 3 respectively.
Revised: The results of ablation studies on CIFAR10, CIFAR100 and ImageNet are shown in Table 1, Table 2 and Table 3 respectively.

Original: Various quantized models applying weak curvature gradients show signiﬁcant improvement in challenging classiﬁcation tasks.
Revised: Various quantized models applying weak curvature gradients show signiﬁcant improvement in challenging classiﬁcation tasks.

Original: For fairness, we use the same forward quantization process and optimizations in experiments where the only difference is the gradients of the backward process.
Revised: For fairness, we use the same forward quantization process and optimizations in experiments where the only difference is the gradients of the backward process.

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 4. C ONVERGENCE AND S TABILITY A NALYSIS
Revised: 4. C ONVERGENCE AND S TABILITY A NALYSIS

Original: In this experiment, we compare the convergence and stability of Dorefa and WCG during the training process.
Revised: In this experiment, we compare the convergence and stability of Dorefa and WCG during the training process.

Original: As Figure 2 shows, the quantized model trained with WCG has better performance and more stable convergence than the quantized model trained with Dorefa, both for training and testing.
Revised: As Figure 2 shows, the quantized model trained with WCG has better performance and more stable convergence than the quantized model trained with Dorefa, both for training and testing.

Original: Here, we state that better performance means larger mean, and more stable convergence means smaller variance.
Revised: Here, we state that better performance means larger mean, and more stable convergence means smaller variance.

Original: As for why WCG performs better, we conjecture that WCG has increased the
Revised: As for why WCG performs better, we conjecture that WCG has increased the

Original: Fisher information of the original quantized model compared to other methods, although WCG does not use additional neural network information for learning like MultiFCG or FCGrad.
Revised: Fisher information of the original quantized model compared to other methods, although WCG does not use additional neural network information for learning like MultiFCG or FCGrad.

Original: Moreover, this is where the advantage of manifold learning lies.
Revised: Moreover, this is where the advantage of manifold learning lies.

Original: 4. C OMPARISONS WITH O THER T RAINING - BASED Q UANTIZATION
Revised: 4. C OMPARISONS WITH O THER T RAINING - BASED Q UANTIZATION

Original: In this experiment, our ManiQuant is initialized with the trained full-precision model [Table] and continues to use WCG to learn the quantized model.
Revised: In this experiment, our ManiQuant is initialized with the trained full-precision model, and continues to use WCG to learn the quantized model.

Original: ManiQuant focuses on the weak curvature manifold embeddings for STE, without any extra training tricks and the deformation of the quantization.
Revised: ManiQuant focuses on the weak curvature manifold embeddings for STE, without any extra training tricks and the deformation of the quantization.

Original: We list the comparison results of 1-bit weight and activation quantization in Table 4.
Revised: We list the comparison results of 1-bit weight and activation quantization in Table 4.

Original: More experimental results can be found in Appendix D [Table] [Table]
Revised: More experimental results can be found in Appendix D.

Original: 4. T RAINING T IME A NALYSIS
Revised: 4. T RAINING T IME A NALYSIS

Original: Our hardware environment is conducted with an Intel(R) Xeon(R)
Revised: Our hardware environment is conducted with an Intel(R) Xeon(R)

Original: Silver 4214 CPU(2.20GHz),
Revised: Silver 4214 CPU(2.20GHz),

Original: GeForce GTX 1080Ti GPU.
Revised: GeForce GTX 1080Ti GPU.

Original: We test the training time per iteration as for MetaQuant using MultiFCG, ManiQuant using WCG, and Dorefa with ResNet20 in CIFAR10.
Revised: We test the training time per iteration as for MetaQuant using MultiFCG, ManiQuant using WCG, and Dorefa with ResNet20 in CIFAR10.

Original: For ﬁnishing one iteration of training, ManiQuant costs 18.44s and MetaQuant costs 25.34s while Dorefa uses 17.81s . 
Revised: For ﬁnishing one iteration of training, ManiQuant costs 18.44s and MetaQuant costs 25.34s while Dorefa uses 17.81s . [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Empirically, training-based quantization methods tend to use Straight-Through Estimator to penetrate the non-differentiable quantization.
Revised: Empirically, training-based quantization methods tend to use Straight-Through Estimator to penetrate the non-differentiable quantization.

Original: However, we throw out the gradient mismatch problem in training quantized neural networks and attribute it to Cram´er-Rao Lower Bound of the variance of
Revised: However, we throw out the gradient mismatch problem in training quantized neural networks.

Original: Straight-Through Estimator. In this paper, we introduce manifold learning to assist Straight-Through Estimator penetrating the non-differentiable quantization, which can be considered to alleviate the gradient mismatch problem.
Revised: In this paper, we introduce manifold learning to assist StraightThrough Estimator penetrating the non-differentiable quantization, which can be considered to alleviate the gradient mismatch problem.

Original: Speciﬁcally, we embed the Riemannian manifold into the StraightThrough Estimator to increase Fisher information.
Revised: Speciﬁcally, we embed the Riemannian manifold into the Straight-Through Estimator to increase Fisher information.

Original: On the one hand, we try to embed Fisher Information Matrix for Straight-Through Estimator as the natural gradient deﬁned by KL divergence, but encounter a complicated calculation dilemma.
Revised: On the one hand, we try to embed Fisher Information Matrix for Straight-Through Estimator as the natural gradient deﬁned by KL divergence, but encounter a complicated calculation dilemma.

Original: On the other hand, we propose a weak curvature manifold deﬁned by Hyperbolic divergence to jump out the complicated calculation dilemma and embed weak curvature metric for Straight-Through Estimator as the weak curvature gradient.
Revised: On the other hand, we propose a weak curvature manifold deﬁned by Hyperbolic divergence to jump out the complicated calculation dilemma and embed weak curvature metric for Straight-Through Estimator as the weak curvature gradient [Table]

Original: A S TRAIGHT -T HROUGH E STIMATOR IN 1- BIT Q UANTIZATION
Revised: A S TRAIGHT -T HROUGH E STIMATOR IN 1- BIT Q UANTIZATION

Original: The key of the proof is to divide the loss function into two parts based on conditional probabilities
Revised: The key of the proof is to divide the loss function into two parts based on conditional probabilities

Original: P ( a > (cid:15) | a ) and 1 − P ( a > (cid:15) | a ) :
Revised: P ( a > (cid:15) | a ) and 1 − P ( a > (cid:15) | a ) :

Original: [Equation]
Revised: [Equation]

Original: where we divide the conditional probability P ( a > (cid:15) | a ) into two parts | a | ≤ 1 and | a | > 1 based on the distribution of the noise source:
Revised: where we divide the conditional probability P ( a > (cid:15) | a ) into two parts | a | ≤ 1 and | a | > 1 based on the distribution of the noise source:

Original: [Equation]
Revised: [Equation]

Original: Further, ignoring high-order derivative of a single neuron ( ˆ a = ± 1 ) that typically has only a small impact on the total loss function, the loss function can be approximated by Taylor expansion w.r.t.
Revised: Further, ignoring high-order derivative of a single neuron ( ˆ a = ± 1 ) that typically has only a small impact on the total loss function, the loss function can be approximated by Taylor expansion w.r.t.

Original: ˆ a =
Revised: ˆ a =

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: a
Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Revised: [Equation]

Revised: a

Original: [Equation]
Revised: [Equation]

Original: a
Revised: (cid:19) [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Revised: ˆ a =

Revised: (cid:19) .

Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 19 can be simply expressed as
Revised: 17 can be simply expressed as

Original: [Equation]
Revised: [Equation]

Original: where ∂L/∂a is an unbiased estimator of ∂L/∂ ˆ a with constrains.

Original: B D IVERGENCE B. D EFINITION OF D IVERGENCE IN A M ANIFOLD
Revised: B D IVERGENCE B. D EFINITION OF D IVERGENCE IN A M ANIFOLD

Original: D [ P : Q ] is called a divergence when it satisﬁes the following criteria: 1) D
Revised: D [ P : Q ] is called a divergence when it satisﬁes the following criteria: 1) D

Original: [ P : Q ] ≥ 0 . 2) D 
Revised: [ P : Q ] ≥ 0 . 2) D [ P : Q ]

Original: [ P : Q ] = 0 when and only when P = Q . 3) When P and Q are sufﬁciently close, by denoting their coordinates by ξ P and ξ Q = ξ P + d ξ , the Taylor expansion of D is written as[Equation]
Revised: = 0 when and only when P = Q . 3) When P and Q are sufﬁciently close, by denoting their coordinates by ξ P and ξ Q = ξ P + d ξ , the Taylor expansion of D is written as

Original: [Equation]
Revised: [Equation]

Original: and Riemannian metric G ij is positive-deﬁnite, depending on ξ P .
Revised: and Riemannian metric G ij is positive-deﬁnite, depending on ξ P .

Original: B. B REGMAN D IVERGENCE
Revised: B. B REGMAN D IVERGENCE

Original: Bregman divergence D B [ ξ : ξ (cid:48) ] is deﬁned as the difference between a convex function ψ ( ξ ) and its tangent hyperplane z = ψ ( ξ (cid:48) )+( ξ
Revised: Bregman divergence D B [ ξ : ξ (cid:48) ] is deﬁned as the difference between a convex function ψ ( ξ ) and its tangent hyperplane z = ψ ( ξ (cid:48) )

Original: [Equation] ξ (cid:48) ) 
Revised: +( ξ − ξ (cid:48) ) ∇ ψ ( ξ (cid:48) ) , depending on the Taylor expansion at the point ξ (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: ψ ( ξ (cid:48) ) , depending on the Taylor expansion at the point ξ (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: B. KL D IVERGENCE AND F ISHER I NFORMATION M ATRIX
Revised: B. KL D IVERGENCE AND F ISHER I NFORMATION M ATRIX

Original: The KL divergence can be deﬁned between p ( x | θ ) and p ( x | θ (cid:48) ) :
Revised: The KL divergence can be deﬁned between p ( x | θ ) and p ( x | θ (cid:48) ) :

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst derivative is:
Revised: The ﬁrst derivative is:

Original: [Equation]
Revised: [Equation]

Original: The second derivative is:
Revised: The second derivative is:

Original: [Equation]
Revised: [Equation]

Original: We deduce the Taylor expansion of the KL divergence at θ = θ (cid:48) :
Revised: We deduce the Taylor expansion of the KL divergence at θ = θ (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: The Taylor expansion of the KL divergence at θ = θ (cid:48) is related to the Fisher Information Matrix
Revised: The Taylor expansion of the KL divergence at θ = θ (cid:48) is related to the Fisher Information Matrix

Original: [Equation]
Revised: [Equation]

Original: B. H YPERBOLIC D IVERGENCE AND W EAK C URVATURE M ETRIC
Revised: B. H YPERBOLIC D IVERGENCE AND W EAK C URVATURE M ETRIC

Original: The ﬁrst derivative of the Hyperbolic divergence is:
Revised: The ﬁrst derivative of the Hyperbolic divergence is:

Original: [Equation]
Revised: [Equation]

Original: The second derivative of the Hyperbolic divergence is:
Revised: The second derivative of the Hyperbolic divergence is:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We deduce the Taylor expansion of the Hyperbolic divergence at θ = θ
Revised: We deduce the Taylor expansion of the Hyperbolic divergence at θ = θ

Original: [Equation]
Revised: [Equation]

Original: [Equation] 
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: D ESCENT D IRECTION IN A R IEMANNIAN M ANIFOLD C. P ROOF OF L EMMA
Revised: D ESCENT D IRECTION IN A R IEMANNIAN M ANIFOLD C. P ROOF OF L EMMA

Original: We would to know in which direction minimizes the loss function with the constraint of the KL divergence, so that we do the minimization:
Revised: We would to know in which direction minimizes the loss function with the constraint of the KL divergence, so that we do the minimization:

Original: [Equation]
Revised: [Equation]

Original: where c is the constant.
Revised: where c is the constant.

Original: The loss function descends along the manifold with constant speed, regardless the curvature.
Revised: The loss function descends along the manifold with constant speed, regardless the curvature.

Original: Now, we write the minimization in Lagrangian form.
Revised: Now, we write the minimization in Lagrangian form.

Original: Combined with Appendix B.3, the KL divergence can be approximated by its second order Taylor expansion.
Revised: Combined with Appendix B.3, the KL divergence can be approximated by its second order Taylor expansion.

Original: Approximating L ( θ + d θ ) with it ﬁrst order
Revised: Approximating L ( θ + d θ ) with it ﬁrst order

Original: Taylor expansion, we get:
Revised: Taylor expansion, we get:

Original: [Equation]
Revised: [Equation]

Original: To solve this minimization, we set its derivative w.r.t.
Revised: To solve this minimization, we set its derivative w.r.t.

Original: d θ to zero:
Revised: d θ to zero:

Original: [Equation]
Revised: [Equation]

Original: where a constant factor 2 /λ can be absorbed into learning rate.
Revised: where a constant factor 2 /λ can be absorbed into learning rate.

Original: Up to now, we get the optimal descent direction, i.e., the opposite direction of gradient which takes into account the local curvature deﬁned by F − 1 .
Revised: Up to now, we get the optimal descent direction, i.e., the opposite direction of gradient which takes into account the local curvature deﬁned by F − 1 .

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: Now we can deduce the steepest descent direction while taking into account the weak curvature manifold deﬁned by the Hyperbolic divergence.
Revised: Now we can deduce the steepest descent direction while taking into account the weak curvature manifold deﬁned by the Hyperbolic divergence.

Original: With the constraint of Hyperbolic divergence in a constant c , we do the minimization of the loss function L ( θ ) in Lagrangian form:
Revised: With the constraint of Hyperbolic divergence in a constant c , we do the minimization of the loss function L ( θ ) in Lagrangian form:

Original: [Equation]
Revised: [Equation]

Original: To solve the above minimization, we set its derivative with respect to d θ to zero and obtain the opposite direction of gradients in weak curvature manifold:
Revised: To solve the above minimization, we set its derivative with respect to d θ to zero and obtain the opposite direction of gradients in weak curvature manifold:

Original: [Equation]
Revised: [Equation]

Original: Where a constant factor 2 /λ can be absorbed into learning rate.
Revised: Where a constant factor 2 /λ can be absorbed into learning rate.

Original: The last line of the above formula is true because each element in (cid:2) tanh( τ θ ) tanh( τ θ ) (cid:62) (cid:3) is much smaller than 1 to satisfy the weak curvature metric.
Revised: The last line of the above formula is true because each element in (cid:2) tanh( τ θ ) tanh( τ θ ) (cid:62) (cid:3) is much smaller than 1 to satisfy the weak curvature metric.

Original: Speciﬁcally, (cid:2)
Revised: Speciﬁcally, (cid:2)

Original: I − tanh( τ θ ) tanh
Revised: I − tanh( τ θ ) tanh

Original: ( τ θ ) (cid:62) (cid:3) is a strictly diagonally-dominant matrix which satisﬁes that the inverse matrix must exist.
Revised: ( τ θ ) (cid:62) (cid:3) is a strictly diagonally-dominant matrix which satisﬁes that the inverse matrix must exist.

Original: D C OMPARISONS WITH O THER T RAINING - BASED Q UANTIZATION
Revised: D C OMPARISONS WITH O THER T RAINING - BASED Q UANTIZATION

Original: We ﬁrst list the comparison results of 1-bit weight quantization in Table 5.
Revised: We ﬁrst list the comparison results of 1-bit weight quantization in Table 5.

Original: [Table]
Revised: [Table]

Original: Next, we extend the bit width to more than 1-bit
Revised: Next, we extend the bit width to more than 1-bit

Original: [Table]
Revised: [Table]

Original: and the quantization function of k-bit used is
Revised: and the quantization function of k-bit used is

Original: [Equation]
Revised: [Equation]

Original: We quantize the weights of all layers
Revised: We quantize the weights of all layers

Original: [Table]
Revised: [Table]

Original: and activations of all layers except the ﬁrst layer,
Revised: and activations of all layers except the ﬁrst layer,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e, the image itself.
Revised: e, the image itself.

Original: Note that the QNN is initialized by the trained full-precision model.
Revised: Note that the QNN is initialized by the trained full-precision model.

Original: The comparison results are listed in Table 6.
Revised: The comparison results are listed in Table 6.

Original: [Table]
Revised: [Table]

Original: E T HE A LGORITHM D ESIGN
Revised: E T HE A LGORITHM D ESIGN

Original: training a quantized neural network.
Revised: training a quantized neural network.

Original: Note that we use the gradient measured by the Hyperbolic divergence here.
Revised: Note that we use the gradient measured by the Hyperbolic divergence here.

Original: Norm( ∗ ) speciﬁes how to normalize the activations and BackNorm( ∗ ) speciﬁes how to backpropagate through the normalization, which is consistent with (Courbariaux et al., 2016).
Revised: Norm( ∗ ) speciﬁes how to normalize the activations and BackNorm( ∗ ) speciﬁes how to backpropagate through the normalization, which is consistent with (Courbariaux et al., 2016).

Original: Note that mat( ∗ ) is the reverse operator of vec( ∗ ) , which turns the column vector back into a matrix.
Revised: Note that mat( ∗ ) is the reverse operator of vec( ∗ ) , which turns the column vector back into a matrix.

Original: Input: A minibatch of inputs and targets ( x = a 0 , y ) , θ mapped to ( W 1 , W 2 , . . .
Revised: Input: A minibatch of inputs and targets ( x = a 0 , y ) , θ mapped to ( W 1 , W 2 , . . .

Original: , W l ) , ˆ θ mapped to (cid:16) ˆ W 1 , ˆ
Revised: , W l ) , ˆ θ mapped to (cid:16) ˆ W 1 , ˆ

Original: W
Revised: W

Original: , ˆ W l (cid:17) , a nonlinear function f , a constant factor χ and a learning rate η .
Revised: , ˆ W l (cid:17) , a nonlinear function f , a constant factor χ and a learning rate η .

Original: Output: The updated discrete parameters ˆ θ .
Revised: Output: The updated discrete parameters ˆ θ .

Original: 1: { Forward propagation } 2: for i = 1; i ≤ l ; i + + do 3:
Revised: 1: { Forward propagation } 2: for i = 1; i ≤ l ; i + + do 3:

Original: Compute ˆ W i =
Revised: Compute ˆ W i =

Original: Q
Revised: Q

Original: ( W i ) ; 4:
Revised: ( W i ) ; 4:

Original: Compute s i
Revised: Compute s i

Original: =
Revised: =

Original: ˆ W i ˆ a
Revised: ˆ W i ˆ a

Original: i − 1 ; 5:
Revised: i − 1 ; 5:

Original: Compute ¯ a i
Revised: Compute ¯ a i

Original: =
Revised: =

Original: Norm ( f (cid:12) s i ) ; 6:
Revised: Norm ( f (cid:12) s i ) ; 6:

Original: Compute ˆ a i = Q ( ¯ a i ) ; 7: end for 8: { Loss derivative } 9: Compute ∇ a l L = ∂L
Revised: Compute ˆ a i = Q ( ¯ a i ) ; 7: end for 8: { Loss derivative } 9: Compute ∇ a l L = ∂L

Original: ( y , x ) ∂
Revised: ( y , x ) ∂

Original: z (cid:12)(cid:12) z
Revised: z (cid:12)(cid:12) z

Original: = ˆ a l I | ¯ a l |≤ 1 ; 10: { Backward propagation } 11: for i =
Revised: = ˆ a l I | ¯ a l |≤ 1 ; 10: { Backward propagation } 11: for i =

Original: l ; i ≥ 1; i − − do 12:
Revised: l ; i ≥ 1; i − − do 12:

Original: Compute ∇ s i L = BackNorm( ∇ a i L (cid:12) f (cid:48)
Revised: Compute ∇ s i L = BackNorm( ∇ a i L (cid:12) f (cid:48)

Original: ( s i )) ; 13:
Revised: ( s i )) ; 13:

Original: Compute P i = vec (tanh ( W i )) vec (tanh ( W i )) (cid:62) 14:
Revised: Compute P i = vec (tanh ( W i )) vec (tanh ( W i )) (cid:62) 14:

Original: Compute ˜ ∇ W i L
Revised: Compute ˜ ∇ W i L

Original: = mat (cid:16)(cid:16)
Revised: = mat (cid:16)(cid:16)

Original: I + χ · P i tr[ P i ] (cid:17) vec (cid:0) ∇ s i L ˆ a (cid:62) i − 1 (cid:1)(cid:17) ; 15:
Revised: I + χ · P i tr[ P i ] (cid:17) vec (cid:0) ∇ s i L ˆ a (cid:62) i − 1 (cid:1)(cid:17) ; 15:

Original: Compute ∇ a i − 1 L = ˆ W (cid:62) i ∇ s i L ; 16: end for 17: { The parameters update } 18: for i = 1; i ≤ l ; i + + do 19:
Revised: Compute ∇ a i − 1 L = ˆ W (cid:62) i ∇ s i L ; 16: end for 17: { The parameters update } 18: for i = 1; i ≤ l ; i + + do 19:

Original: Update W i =
Revised: Update W i =

Original: W
Revised: W

Original: − η
Revised: − η

Original: ∇ W i L I | W i |≤ 1 ; 20: end for
Revised: ∇ W i L I | W i |≤ 1 ; 20: end for