Quantized convolutional neural networks through the lens of partial differential equations
Abstract
Quantization of Convolutional Neural Networks (CNNs) is a common approach to ease the computational burden involved in the deployment of CNNs.
However, ﬁxed-point arithmetic is not natural to the type of computations involved in neural networks.
In our work, we consider symmetric and stable variants of common
CNNs for image classiﬁcation, and Graph Convolutional Networks (GCNs) for graph node-classiﬁcation.
We demonstrate through several experiments that the property of forward stability preserves the action of a network under different quantization rates, allowing stable quantized networks to behave similarly to their non-quantized counterparts while using fewer parameters.
We also ﬁnd that at times, stability aids in improving accuracy.
These properties are of particular interest for sensitive, resource-constrained or real-time applications.
Introduction
Deep neural networks have demonstrated superiority in solving many real world problems.
In particular, CNNs are an effective approach for processing structured high-dimensional data.
Consequently, the demand for the deployment of CNNs on resource-constrained devices is constantly increasing, where at the same time, CNNs grow larger than ever.
CNNs still face critical challenges despite this success and their predictions can be highly sensitive to perturbations of the input [32, 18].
Because they often require extremely deep and wide architectures they often impose a high computational cost and make deployment on resource-constrained devices prohibitive, especially for mission-critical applications such as autonomous driving.
Several authors have established a direct link between CNNs and partial differential equations (PDEs) [12, 8, 21].
The connection is two-fold.
First, CNNs ﬁlter input features with multiple layers, employing both elementwise non-linearities and afﬁne transformations, which can be seen as linear combinations of the ﬁnite difference discretizations of spatial derivative operators [35, 15, 14].
Furthermore, the popular residual network architectures (ResNets) [23] can be interpreted as a discrete time integration of a non-linear ODE using the forward Euler scheme.
Another possibility is to view the network as a continuous function [10, 35, 20, 17].
This conceptual framework gives rise to many interesting questions such as: what is the importance of the forward stability of CNNs, and can we create analogues to stable time integration through Courant-Friedrichs-Lewy conditions, to make
CNNs more robust to errors?
Speciﬁcally, in this work we consider round-off and quantization errors.
Quantization methods enable neural networks to carry out computations with ﬁxed-point operations rather than ﬂoating-point arithmetic.
This contributes to the efﬁciency of the networks and reduces their memory footprint, often at the cost of accuracy [11].
Accordingly we consider a per-layer, uniform and static (same number of bits for every layer) quantization-aware training method.
In our work we wish to examine the concept of quantized CNNs by considering that forward time integration in PDEs inevitably generates error at every step.
To satisfy forward stability, this error must decay from time step to time step, such that the integration error is bounded, resulting in a discrete approximation close to the real continuous solution of the PDE.
Therefore, a signiﬁcant contribution of our work is the promotion of stability in quantized CNNs via small changes to common architectures.
We examine the behaviour of quantization under symmetric and stable, heat equation-like CNNs [21, 1, 2].
We show that the quantization process produces signiﬁcantly lighterweight networks, with minimal loss of accuracy, and study the consistency of quantization methods in CNNs, a key attribute in the construction of quantized variants of CNNs.
To this end, we measure the aforementioned similarity using symmetric (potentially stable) and non-symmetric (potentially unstable) networks.
Our experiments indicate that symmetric and stable networks achieve better consistency and that quantized networks are expected to perform better and more in line with their full-precision counterparts when stable architectures are used, rather than unstable architectures .
Neural networks are known to be susceptible to noise in their inputs [19].
Similarly, the quantization of the activation maps inevitably adds noise to the input of every layer.
While the quantization error is not chosen speciﬁcally for a given input, it does appear in every layer.
The introduction of such errors throughout the layers of the network resembles the approximation and round-off errors in time integration of PDEs which require forward stability to converge.
To relieve this phenomenon, we follow the approach of constructing stable architectures that are more robust against such errors [21].
We call a discrete forward propagation stable if it prevents any perturbation from growing as it propagates through time steps.
More explicitly, an N -layered network is stable if there exists some
M > 0 such that
[Equation]
where x i and ˜ x
i are the true and perturbed feature maps of the i -th layer, respectively.
The Jacobian of x N with respect to x i can then be bounded proportionally to M , and accordingly, any stable architecture needs to have a bounded Jacobian [35].
See Appendix A for a more precise formulation.
Stable and quantized residual networks
To promote stability in CNNs and to prevent the ampliﬁcation of quantization errors, we would like the quantized network to behave similarly (in terms of its activations) to a non-quantized instance of the same network.
The symmetric variant of ResNet, together with the activation quantization operator is given as follows:
[Equation]
where Q b denotes the quantization operation.
We consider the same weights K between the two architectures, whether quantized or not, and also assume that the absolute quantization error for any scalar is bounded by some δ .
We analyze the propagation of the quantization error and show that it is the Jacobian matrix of the ResNet block that multiplies the error at every iteration and propagates the previous error into the next block.
Assuming that σ is non-decreasing, it means that K (cid:62) j σ (cid:48) () K j is positive semi-deﬁnite, and with a proper choice of K j and h , we can force ρ ( J j ) < 1 , so that the error decays.
To ensure this forward stability we must set h <
( L (cid:107) K j (cid:107) 22 ) − 1 for every layer j in the network, where L is the upper bound for σ (cid:48)
This is generally possible in ResNets only if we use the symmetric variant in of ResNet.
We measure the quantization error by
[Equation]
and reduce this MSE to maximize the accuracy of the quantized network, as shown in [4]. [Table]
Stable and quantized GCNs
We also study the importance of stability for quantized Graph Convolution Networks (GCNs), which can be thought of as a PDE discretized on unstructured grids [14].
Speciﬁcally, we use the diffusive
PDE-GCN architecture formulated in [13], which utilizes symmetric operators, only on unstructured graphs:
[Equation]
Here, x j are the features deﬁned over the nodes of a graph, K j is a learnt 1 × 1 convolution operator and S j is either learnt (e.g., as in [14]) or pre-deﬁned spatial operation (e.g., the graph Laplacian), both for the j -th layer of the network.
Here, the forward stability is guaranteed in the continuous case by the symmetry of the operator [13].
Refer to Appendix A for a more precise formulation.
We refer to a network governed by the dynamics in Eq.
(4) as PDE-GCN D (sym.).
Analogously, we deﬁne the non-symmetric residual layer:
[Equation]
Where K j 1 and K j 2 are distinct learnt 1 × 1 convolution operators.
This formulation does not necessarily yield a symmetric operator, thus, we denote by such a network by PDE-GCN D (non-sym.).
The quantization for Eq.
(4) and (5) is applied to the weights and before each convolution operator.
Image classiﬁcation using stable and quantized networks
We evaluate our symmetric architectures under quantization by testing them using ResNet-34, ResNet56 [23] and MobileNetV2 [36].
We conduct several experiments using CIFAR-10/100 and present their results in Tab. 1 and 2.
Symmetric networks achieve similar accuracy as non-symmetric networks, while using approximately half of the parameters, a signiﬁcant saving obtained as a by-product of promoting stability.
Furthermore, we quantify the notion of stability by comparing the behaviour of symmetric and non-symmetric networks under 4-bit quantization.
In addition to the 4-bit quantized network, we relax the quantization of the activation maps, allowing them to use the full 32-bit precision, and we measure the divergence between corresponding activations in both runs of the same network - one at 4 bits and the other at 32 bits.
This divergence is summarized as the per-entry MSE between the activation maps throughout the two networks.
Tables 1 and 2 show that the divergence of the two runs is greater in the non-symmetric (original) network variants, as predicted by the theoretical analysis.
This is illustrated in Fig.
1, where we plot the same MSE difference along the layers of each network.
Semi-supervised node-classiﬁcation with quantized GCNs
We employ PDE-GCN D (sym.) and PDE-GCN D (non-sym.)
from Eq.
(4)-(5) for semi-supervised node-classiﬁcation on the Cora, CiteSeer and PubMed data-sets.
Namely, we measure the impact of symmetry on the network’s accuracy, as well as its ability to maintain similar behaviour with respect to activation similarity under activation quantization of 8 and 4 bits.
In all experiments, we use 32 layers, with 64 hidden channels for Cora, and 256 hidden channels for CiteSeer and PubMed.
The training and evaluation procedure we used is identical to [28].
Details on initialization and
[Table] [Figure] [Table] hyper-parameter tuning, are identical to [13].
The results provided in Tab.
3 reveal two beneﬁts of a symmetric (stable) formulation over a non-symmetric (unstable) one.
First, it is apparent that that the former results in better accuracy, often by over 2% , while using almost half the number of parameters.
In addition, the action of the network is better preserved under quantization using the symmetric formulation.
We note that the symmetric formulation in Eq.
(4) is a sub-set of the non-symmetric counterpart in Eq.
Therefore, theoretically, both networks can achieve identical expressiveness.
However, as demonstrated in Tab. 3, this was not observed in our experiments.
We attribute this gap to the smoother optimization process of the stable PDE-GCN D (sym.). [Table]
Summary
In this work, we explored quantized neural networks from the perspective of PDEs and demonstrated that stability preserves the action of a network under different quantization rates.
We ﬁnd that at times, stability aids to improve accuracy.
These properties are of particular interest for resource-constrained, low-power or real-time applications like autonomous driving.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[No] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Continuous neural networks and stable ResNet architectures
The general goal of supervised machine learning is to model a function f : R n × R p → R m and train its parameters θ ∈ R p such that
[Equation]
(6) for input-output pairs { ( y i , c i ) } si =1 from a certain data set Y × C .
Typical tasks include regression or classiﬁcation and the function f can be viewed as an interpolation of the true function that we want the machine to learn.
In learning tasks involving images and videos, the learnt function f commonly includes a CNN that ﬁlters the input data.
Among the many architectures we focus on feed-forward networks of a common type called by ResNets [23].
In the simplest case, the forward propagation of a given example y through an N -layer ResNet can be written as
[Equation]
The layer function F consists spatial convolution operators parameterized by the weights θ 1 , . .
, θ N , and non-linear element-wise activation functions.
The parameter h > 0 in this notation serves as a time step, arising from a discretization of a continuous network, as we show in the next section.
The classical ResNet layer reads
[Equation]
where K 1 , K 2 are two different convolution operators parameterized by θ , and σ is a non-linear activation function, like σ ( x ) =
max { x, 0 } , known as the Rectiﬁed Linear Unit (ReLU) function.
The term F open denotes an opening layer that typically outputs a n c -channel image x 0 , where n c is greater than the number of input channels, which is typically 3 for RGB images, while possibly also reducing the size of the image.
Usually, the opening layer reads
[Equation]
where K open is a convolution operator parameterized by θ open that widens the number of channels from this point onward in the network.
Continuous neural networks have recently been suggested as an abstract generalization of the more common discrete network, where the network is viewed as a discretized instance of a continuous
ODE or PDE.
As shown by [8, 12, 21, 10], Eq.
(7) (or a ResNet) is essentially a forward Euler discretization of a continuous non-linear ODE
[Equation]
where [0 , T ] is an artiﬁcial time interval related to the depth of the network.
Relating spatial convolution ﬁlters with differential operators, the authors of [35] propose a layer function representation F that renders Eq.
(10) similar to a parabolic diffusion-like PDE
[Equation]
For example, when K represents a discrete gradient operator and σ ( x ) = x , we obtain the heat equation under this treatment.
The approach of Eq.
(11) is natural, as similar developments have led to several breakthroughs in image processing, including optical ﬂow models for motion estimation [24], non-linear anisotropic diffusion models for image denoising [33, 34], and variational methods for image segmentation [3, 7].
To best balance the network’s representational abilities with its computational cost, a “bottleneck” structure is used commonly, exploiting a different number of input and output channels in K , as in Eq.
Generally speaking, the training of the neural network model consists of ﬁnding parameters θ such that Eq.
(6) is approximately satisﬁed for examples from a training data set.
The same should also hold for examples from a validation data set, which is not used to optimize the parameters.
The training objective is commonly modeled as an expected loss to be minimized, denoted as
[Equation]
, where
[Equation]
Here, ( y 1 , c 1 ) , . . .
, ( y s , c s ) ∈
Y × C are the training data, f ( y , θ ) includes the action of the neural network, but may also contain other layers, e.g., fully-connected layers, opening and connective layers, and softmax transformations in classiﬁcation.
R is a regularization term.
The optimization problem in Eq.
(12) is typically solved with gradient-based non-linear optimization techniques [6].
Due to the large-scale and stochastic nature of the learning problem, it is common to use stochastic approximation schemes such as variants of stochastic gradient descent (SGD) like Adam [27].
These methods perform iterations using gradient information from randomly chosen subsets of the data.
B Stable and quantized residual networks
As discussed earlier, our goal is to promote stability in CNNs, to prevent the ampliﬁcation of quantization errors.
To obtain that, we would like the quantized network to behave similarly (in terms of its activations) to a non-quantized instance of the same network.
The symmetric variant of ResNet in Eq.
(8) and (11), together with the activation quantization operator in Eq.
(18) is given as follows:
[Equation]
where Q b denotes the quantization operation in Eq.
Let us denote the feature maps of the non-quantized network by ˆ x j (i.e., assuming q b ( x ) = x in Eq.
(18)), and let the error between corresponding activations be denoted by η j = x
j − ˆ x j .
We consider the same weights K between the two architectures, whether quantized or not, and also assume that the absolute quantization error for any scalar is bounded by some δ .
We analyze the propagation of the quantization error η j +1 as a function of η j .
We start unwrapping the block in Eq.
First, we subtract ˆ x j +1 from both sides, and replace the outer quantization with the error term η j 1 .
[Equation]
Next, we remove the other instance of Q b and add another error term η j 2 :
[Equation]
Remembering that x j = ˆ x j + η j , and using the ﬁrst-order approximation of σ , we note that σ [Equation] K j x j )
≈
σ ( K j ˆ x j ) + σ (cid:48)
( K j ˆ x j ) K j η j .
Therefore, we have
[Equation]
The key ingredient of the analysis above is that η j 1 and η j 2 are ﬁxed and bounded for every layer.
On the other hand, it is the Jacobian matrix of the block J j =
I − h K (cid:62)
j σ (cid:48)
( K j ˆ x j ) K j that multiplies η j at every iteration and propagates the previous error into the next block.
Assuming that σ is non-decreasing, it means that K (cid:62) j σ (cid:48) () K j is positive semi-deﬁnite, and with a proper choice of K j and h , we can force ρ ( J j ) < 1 , so that the error decays.
To ensure this forward stability we must set h <
( L (cid:107) K j (cid:107) 22 ) − 1 for every layer j in the network, where L is the upper bound for σ (cid:48)
This is generally possible in ResNets only if we use the symmetric variant in Eq.
[35, 40] also achieved a similar stability result, but not in the context of quantization.
C Quantization-aware training
As mentioned before, in this paper we focus on quantized neural networks, and in particular, on the scenario of quantization-aware training.
We restrict the values of the weights to a smaller set, so that after training, the calculation of a prediction by the network can be carried out in ﬁxed-point integer arithmetic.
The intermediate hidden layers are quantized (rounded) as well.
Even though the quantization involves a discontinuous rounding function, the discrete weights can be optimized using gradient-based methods, also known as quantization-aware training schemes [22, 39].
During the forward pass, both the weights and hidden activation maps are quantized, while during gradient calculation in the backward pass, derivative information is passed through the rounding function, whose exact derivative is zero.
This method is known as the Straight Through Estimator (STE) [5].
We note that the gradient-based optimization for the weights is applied in ﬂoating-point arithmetic as we describe next, but during inference, the weights and activations are quantized, and all operations are performed using integers only.
We now present the details of the quantization scheme that we use, based on [30].
First, we deﬁne the pointwise quantization operator:
[Equation]
where t is a real-valued scalar in one of the ranges , respectively.
b is the number of bits that are used to represent t as integer at inference time (during [Figure] training, t and q b ( t ) are real-valued).
During the forward pass, we force each quantized value to one of the ranges above by applying a clipping function to the quantized weights and activations before applying Eq.
[Equation]
Here, w, w b are the real-valued and quantized weight tensors, x, x b are the real-valued and quantized input tensors, and α w , α x are their associated clipping parameters (also called scales), respectively.
An example of Eq.
(18) applied to a weight tensor using 4-bit signed quantization, is given in Fig.
During training, we iterate on the ﬂoating-point values of the weights w , while both the weights and activation maps are quantized in the forward pass (i.e., x b and w b are passed through the network).
The STE [5] is used to compute the gradient in the backward pass, where the derivative of q b is ignored and we use the derivatives w.r.t w b in the SGD optimization to update w iteratively.
The quantization scheme in Eq.
(18) involves the clipping parameters α w and α x , also called scales, that are used to translate the true value of each integer in the network to its ﬂoating-point value.
This way, at inference time, two integers from different layers can be multiplied in a meaningful way that is faithful to the original ﬂoating-point values [31, 26].
These scales also control the quantization error, and need to be chosen according to the values propagating through the network and the bit allocation b , whether the weights or the activations.
Alternatively, the works [30, 16], which we follow here, introduced an effective gradient-based optimization to ﬁnd the clipping values α x , α w for each layer.
Given Eq.
(18) the gradients w.r.t α w , and α x can be approximated using the STE [30, 16].
For the activation maps, for example, this resolves to:
[Equation]
This enables the quantized network to be trained end-to-end manner with backpropagation.
To further improve the optimization, [30] normalize the weights before quantization:
[Equation]
Here, µ and σ are the mean and standard deviation of the weight tensor, respectively, and (cid:15) =
10 − 6 .
D Stable channel and resolution changing layers
The classical CNN architecture typically applies several layers like Eq.
However, to obtain better representational capabilities for the network, the number of channels typically increases every few layers.
In CNNs, this is typically accompanied by a down-sampling operation performed by a pooling layer or a strided convolution.
In such cases, the residual equation Eq.
(7) cannot be used, since the update F ( x j , θ j ) does not match x j in terms of dimensions.
For this reason, ResNets typically include 3-4 steps like
[Equation]
throughout the network to match the changing resolution and number of channels.
These layers are harder to control than Eq.
(7) and (11) if one wishes to ensure stability.
In this work we are interested in testing and demonstrating the theoretical property described in Sec. 2 empirically, using fully stable networks (except for the very ﬁrst and last layers).
To this end, when increasing the channel space from n c in to n c out for x j +1 and x j respectively, we simply concatenate the output of the step with the necessary channels from the current iteration to maintain the same dimensions.
That is, we apply the following:
[Equation]
where ( x j ) 1: n cout − n cin are the ﬁrst n c out − n c in channels of x j .
This assumes that each of the channel changing steps satisﬁes n c in ≤ n c out ≤ 2 n c in , which is quite common in CNNs.
This way, we only use the symmetric dynamics as in Eq.
(11) throughout the network, which is guaranteed to be stable for a proper choice of parameters.
Finally, we do not apply strides as part of Eq.
(22), and to reduce the channel resolution we simply apply average pooling following Eq.
(22), which is a stable, parameter-less, operation.
E A stable variant of MobileNetV
In addition to standard residual networks, we also consider the popular MobileNet family of lightweight CNN architectures [36, 25], which are the most common architectures for edge devices, achieving very nice results while requiring modest computational resources to deploy.
These architectures utilize the “inverse bottleneck” structure, where the channel space throughout the network is relatively small, but in every step it is expanded and reduced by 1 × 1 convolutions, with “depthwise” 3 × 3 convolutions in the middle expanded channel space.
The depthwise convolutions apply 3 ×kernels on each channel without mixing between the channels, and hence they are less expensive than 1 × 1 convolutions.
The general structure of MobileNetV2 [36] reads:
[Equation]
(23) where K 31 × 1 , K 11 × 1 are two different learnable 1 × 1 convolution ﬁlters, and K 2 dw is a learnable depthwise 3 × 3 convolution matrix.
Due to the relatively small channel space used throughout the architecture, the quantization of this network to low bit-rates is challenging.
The MobileNetV2 architecture in Eq.
(23) can also be seen as a residual network, and is generally unstable.
Because of its inverse bottleneck structure, it does not ﬁt the structure of Eq.
(7) and Eq.
To test the importance of stability under quantization for such networks as well, we deﬁne the stable MobileNetV2 variant:
[Equation]
where now, the separable convolution operator K 2 dw K 11 × 1 takes the role of the single operator K in
Eq.
Going between Eq.
(23) and (24) we lost one non-linearity, but as we show later, this has marginal inﬂuence on the accuracy of the network.
We also apply the depthwise operation twice, but this is inexpensive compared to 1 × 1 convolutions.
Changing the resolutions and channel space is done the same way as described in section D.
F Settings and data-sets
Our code is implemented in PyTorch, and all experiments are conducted on an Nvidia RTX-3090 with 24GB of memory.
Below, we elaborate on the different data-sets explored throughout the numerical experiments. [Figure]
CIFAR-10/100.
The CIFAR-10/100 image classiﬁcation data-sets [29] each consist of 60k natural images of size 32 × 32 where each image is assigned to one of ten categories (for CIFAR-10) or one hundred categories (for CIFAR-100).
The data-set includes 50K training examples and 10K test examples.
We derive a validation set for training by holding out 10% of the training data and report accuracy metrics on the test data.
Node-classiﬁcation data-sets.
Lastly, we use graph neural networks on three citation network node classiﬁcation data-sets: Cora, CiteSeer and PubMed [37].
For each data-set, we use the standard train/validation/test split as in [38], with 20 nodes per class for training, 500 validation nodes and 1,000 testing nodes.
We use the same training scheme as in [9].