Original: O N THE U NIVERSAL A PPROXIMABILITY AND C OMPLEXITY B OUNDS OF Q UANTIZED R E LU N EURAL N ETWORKS
Revised: O N THE U NIVERSAL A PPROXIMABILITY AND C OMPLEXITY B OUNDS OF Q UANTIZED R E LU N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Compression is a key step to deploy large neural networks on resource-constrained platforms.
Revised: Compression is a key step to deploy large neural networks on resource-constrained platforms.

Original: As a popular compression technique, quantization constrains the number of distinct weight values and thus reducing the number of bits required to represent and store each weight.
Revised: As a popular compression technique, quantization constrains the number of distinct weight values and thus reducing the number of bits required to represent and store each weight.

Original: In this paper, we study the representation power of quantized neural networks.
Revised: In this paper, we study the representation power of quantized neural networks.

Original: First, we prove the universal approximability of quantized ReLU networks on a wide class of functions.
Revised: First, we prove the universal approximability of quantized ReLU networks on a wide class of functions.

Original: Then we provide upper bounds on the number of weights and the memory size for a given approximation error bound and the bit-width of weights for function-independent and functiondependent structures.
Revised: Then we provide upper bounds on the number of weights and the memory size for a given approximation error bound and the bit-width of weights for function-independent and functiondependent structures.

Original: Our results reveal that, to attain an approximation error bound of (cid:15) , the number of weights needed by a quantized network is no more than O (cid:0) log 5 (1 /(cid:15) )
Revised: Our results reveal that, to attain an approximation error bound of (cid:15) , the number of weights needed by a quantized network is no more than O (cid:0) log 5 (1 /(cid:15) )

Original: (cid:1) times that of an unquantized network.
Revised: (cid:1) times that of an unquantized network.

Original: This overhead is of much lower order than the lower bound of the number of weights needed for the error bound, supporting the empirical success of various quantization techniques.
Revised: This overhead is of much lower order than the lower bound of the number of weights needed for the error bound, supporting the empirical success of various quantization techniques.

Original: To the best of our knowledge, this is the ﬁrst in-depth study on the complexity bounds of quantized neural networks.
Revised: To the best of our knowledge, this is the ﬁrst in-depth study on the complexity bounds of quantized neural networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Various deep neural networks deliver state-of-the-art performance on many tasks such as object recognition and natural language processing using new learning strategies and architectures (Chen et al., 2017;
Revised: Various deep neural networks deliver state-of-the-art performance on many tasks such as object recognition and natural language processing using new learning strategies and architectures (Chen et al., 2017;

Original: He et al., 2016; Kumar et al., 2016; Ioffe & Szegedy, 2015; Vaswani et al., 2017).
Revised: He et al., 2016; Kumar et al., 2016; Ioffe & Szegedy, 2015; Vaswani et al., 2017).

Original: Their prevalence has extended to embedded or mobile devices for edge intelligence, where security, reliability or latency constraints refrain the networks from running on servers or in clouds.
Revised: Their prevalence has extended to embedded or mobile devices for edge intelligence, where security, reliability or latency constraints refrain the networks from running on servers or in clouds.

Original: However, large network sizes with the associated expensive computation and memory consumption make edge intelligence even more challenging (Cheng et al., 2018; Sandler et al., 2018).
Revised: However, large network sizes with the associated expensive computation and memory consumption make edge intelligence even more challenging (Cheng et al., 2018; Sandler et al., 2018).

Original: In response, as will be more detailed in Section 2, substantial effort has been made to reduce the memory consumption of neural networks while minimizing the accuracy loss.
Revised: In response, as will be more detailed in Section 2, substantial effort has been made to reduce the memory consumption of neural networks while minimizing the accuracy loss.

Original: The memory consumption of neural networks can be reduced by either directly reducing the number of weights or decreasing the number of bits (bit-width) needed to represent and store each weight, which can be employed on top of each other (Choi et al., 2016).
Revised: The memory consumption of neural networks can be reduced by either directly reducing the number of weights or decreasing the number of bits (bit-width) needed to represent and store each weight, which can be employed on top of each other (Choi et al., 2016).

Original: The number of weights can be reduced by pruning (Han et al., 2015b), weight sparsifying (Liu et al., 2015), structured sparsity learning (Wen et al., 2016) and low rank approximation (Denton et al., 2014).
Revised: The number of weights can be reduced by pruning (Han et al., 2015b), weight sparsifying (Liu et al., 2015), structured sparsity learning (Wen et al., 2016) and low rank approximation (Denton et al., 2014).

Original: The bit-width is reduced by quantization that maps data to a smaller set of distinct levels (Sze et al., 2017).
Revised: The bit-width is reduced by quantization that maps data to a smaller set of distinct levels (Sze et al., 2017).

Original: Note that while quantization may stand for linear quantization only (Li et al., 2017; Gysel et al., 2016) or nonlinear quantization only (Han et al., 2015a; Choi et al., 2016) in different works, our discussion will cover both cases.
Revised: Note that while quantization may stand for linear quantization only (Li et al., 2017; Gysel et al., 2016) or nonlinear quantization only (Han et al., 2015a; Choi et al., 2016) in different works, our discussion will cover both cases.

Original: However, as of today quantization is still only empirically shown to be robust and effective to compress various neural network architectures (Hubara et al., 2016; Zhou et al., 2017b; Zhuang et al., 2017).
Revised: However, as of today quantization is still only empirically shown to be robust and effective to compress various neural network architectures (Hubara et al., 2016; Zhou et al., 2017b; Zhuang et al., 2017).

Original: Its theoretical foundation still remains mostly missing.
Revised: Its theoretical foundation still remains mostly missing.

Original: Speciﬁcally, many important questions remain unanswered.
Revised: Speciﬁcally, many important questions remain unanswered.

Original: For example:
Revised: For example:

Original: • Why even binarized networks, those most extremely quantized with bit-width down to one, still work well in some cases?
Revised: • Why even binarized networks, those most extremely quantized with bit-width down to one, still work well in some cases?

Original: • To what extent will quantization decrease the expressive power of a network?
Revised: • To what extent will quantization decrease the expressive power of a network?

Original: Alternatively, what is the overhead induced by weight quantization in order to maintain the same accuracy?
Revised: Alternatively, what is the overhead induced by weight quantization in order to maintain the same accuracy?

Original: In this paper, we provide some insights into these questions from a theoretical perspective.
Revised: In this paper, we provide some insights into these questions from a theoretical perspective.

Original: We focus on ReLU networks, which is among the most widely used in deep neural networks (Xu et al., 2015).
Revised: We focus on ReLU networks, which is among the most widely used in deep neural networks (Xu et al., 2015).

Original: We follow the idea from Yarotsky (2017) to prove the complexity bound by constructing a network.
Revised: We follow the idea from Yarotsky (2017) to prove the complexity bound by constructing a network, but with new and additional construction components essential for quantized networks.

Original: Speciﬁcally, given the number of distinct weight values λ  , we construct a network that can achieve an arbitrarily small error bound (cid:15) to prove the universal approximability .
Revised: Speciﬁcally, given the number of distinct weight values λ and a target function f , we construct a network that can approximate f with an arbitrarily small error bound (cid:15) to prove the universal approximability.

Original: The memory size of this network then naturally serves as an upper bound for the minimal network size.
Revised: The memory size of this network then naturally serves as an upper bound for the minimal network size.

Original: When constructing the network to approximate a target function, we derive the bound in two cases, function-dependent structure and function-independent structure.
Revised: The high level idea of our approach is to replace basic units in an unquantized network with quantized sub-networks 1 that approximate these basic units.

Original: In the case of functionindependent structure, the topology of the network is ﬁxed for any function considered. In contrast, the topology of function-dependent structure can be adapted to a speciﬁc target function and thus enables a tighter bound. We consider two types of quantization, linear quantization and nonlinear quantization (Sze et al., 2017).
Revised: For example, we can approximate a connection with any weight in an unquantized network by a quantized sub-network that only uses a ﬁnite number of given weight values. Even though the approximation of a single unit can be made arbitrarily accurate in principle with unlimited resources (such as increased network depth), in practice, there exists some inevitable residual error at every approximation, all of which could propagate throughout the entire network.

Original: Linear quantization does mapping with a same distance between contiguous quantization levels and is usually implemented by storing weights as ﬁxed-point numbers with reduced bit-width (Li et al., 2017; Gysel et al., 2016). Nonlinear quantization maps the data to quantization levels that are not uniformly distributed and can be either preselected or learned from training.
Revised: The challenge becomes, however, how to mathematically prove that we can still achieve the end-to-end arbitrary small error bound even if these unavoidable residual errors caused by quantization can be propagated throughout the entire network. This paper ﬁnds a solution to solve the above challenge.

Original: Then the weights are stored using lossless binary coding (the index to  a lookup table) instead of  the actual values (Han et al., 2015a; Choi et al., 2016). The most important difference between linear quantization and nonlinear quantization is that the latter does not require weights to be low precision values.
Revised: In doing so, we have to propose a number of new ideas to solve related challenges, including judiciously choosing the proper ﬁnite weight values, constructing the approximation sub-networks as efﬁcient as possible (to have a tight upper bound), and striking a good balance among the complexities of different approximation steps.

Original: Based on the bounds derived, we compare them with the available results on unquantized neural networks and discuss its implications.
Revised: Based on the bounds derived, we compare them with the available results on unquantized neural networks and discuss its implications.

Original: In particular, the main contributions of this paper include:
Revised: In particular, the main contributions of this paper include:

Original: • We prove that even the most extremely quantized ReLU networks using two distinct weight values are capable of representing a wide class of functions with arbitrary accuracy.
Revised: • We prove that even the most extremely quantized ReLU networks using two distinct weight values are capable of representing a wide class of functions with arbitrary accuracy.

Original: • Given the number of distinct weights and the desired approximation error bound, we provide upper bounds on the number of weights and the memory size.
Revised: • Given the number of distinct weights and the desired approximation error bound, we provide upper bounds on the number of weights and the memory size.

Original: We further show that our upper bounds have good tightness by comparing them with the lower bound of unquantized ReLU networks established in the literature.
Revised: We further show that our upper bounds have good tightness by comparing them with the lower bound of unquantized ReLU networks established in the literature.

Original: • We show that, to attain the same approximation error bound (cid:15) , the number of weights needed by a quantized network is no more than O (cid:0) log 5 (1 /(cid:15) )
Revised: • We show that, to attain the same approximation error bound (cid:15) , the number of weights needed by a quantized network is no more than O (cid:0) log 5 (1 /(cid:15) )

Original: (cid:1) times that of an unquantized network.
Revised: (cid:1) times that of an unquantized network.

Original: This overhead is of much lower  compared with even the lower bound of the number of weights needed for the error bound.
Revised: This overhead is of much lower order compared with even the lower bound of the number of weights needed for the error bound.

Original: This partially explains why many state-ofthe-art quantization schemes work well in practice.
Revised: This partially explains why many state-ofthe-art quantization schemes work well in practice.

Original: • We demonstratethat how a theoretical complexity bound can be used to estimate an optimal bit-width, which in turn enables the best cost-effectiveness for a given task.
Revised: • We demonstrate how a theoretical complexity bound can be used to estimate an optimal bit-width, which in turn enables the best cost-effectiveness for a given task.

Original: The remainder of the paper is organized as follows.
Revised: The remainder of the paper is organized as follows.

Original: Section 2 reviews related works.
Revised: Section 2 reviews related works.

Original: Section 3 lays down the models and assumptions of our analysis.
Revised: Section 3 lays down the models and assumptions of our analysis.

Original: We prove the universal approximability and the upper bounds with function-independent structure in Section 4 and extend it to function-dependent structure in Section 5.
Revised: We prove the universal approximability and the upper bounds with function-independent structure in Section 4 and extend it to function-dependent structure in Section 5.

Original: We analyze the bound-based optimal bit-width in Section 6.
Revised: We analyze the bound-based optimal bit-width in Section 6.

Original: Finally, Section 7 discusses the results and gets back to the questions raised above.
Revised: Finally, Section 7 discusses the results and gets back to the questions raised above.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Quantized Neural Networks: There are rich literatures on how to obtain quantized networks, either by linear quantization or nonlinear quantization (Zhou et al., 2017a;
Revised: Quantized Neural Networks: There are rich literatures on how to obtain quantized networks, either by linear quantization or nonlinear quantization (Zhou et al., 2017a;

Original: Leng et al., 2017; Shayar et al., 2017).
Revised: Leng et al., 2017; Shayar et al., 2017).

Revised: Linear quantization does mapping with a same distance between contiguous quantization levels and is usually implemented by storing weights as ﬁxed-point numbers with reduced bit-width (Li et al., 2017; Gysel et al., 2016).

Revised: Nonlinear quantization maps the data to quantization levels that are not uniformly distributed and can be either preselected or learned from training.

Revised: Then the weights are stored using lossless binary coding (the index to a lookup table) instead of the actual values (Han et al., 2015a; Choi et al., 2016).

Original: It is reported that a pruned AlexNet can be quantized to eight bits and ﬁve bits in convolutional layers and fully connected layers, respectively, without any loss of accuracy.
Revised: It is reported that a pruned AlexNet can be quantized to eight bits and ﬁve bits in convolutional layers and fully connected layers, respectively, without any loss of accuracy.

Original: Similar results are also observed in LENET-300-100, LENET-5, and VGG-16 (Han et al., 2015a).
Revised: Similar results are also observed in LENET-300-100, LENET-5, and VGG-16 (Han et al., 2015a).

Original: One may argue that some of these benchmark networks are known to have redundancy.
Revised: One may argue that some of these benchmark networks are known to have redundancy.

Original: However, recent works show that quantization works well even on networks that are designed to be extremely small and compact.
Revised: However, recent works show that quantization works well even on networks that are designed to be extremely small and compact.

Original: SqueezeNet, which is a state-of-the-art compact network, can be quantized to 8-bit while preserving the original accuracy (Gysel et al., 2016; Iandola et al., 2016).
Revised: SqueezeNet, which is a state-of-the-art compact network, can be quantized to 8-bit while preserving the original accuracy (Gysel et al., 2016; Iandola et al., 2016).

Original: There are some representative works that can achieve little accuracy loss on ImageNet classiﬁcation even using binary or ternary weights (Courbariaux et al., 2015; Rastegari et al., 2016; Li et al., 2016; Zhu et al., 2016).
Revised: There are some representative works that can achieve little accuracy loss on ImageNet classiﬁcation even using binary or ternary weights (Courbariaux et al., 2015; Rastegari et al., 2016; Li et al., 2016; Zhu et al., 2016).

Original: More aggressively, some works also reduce the precision of activations, e.g. (Hubara et al., 2016; Rastegari et al., 2016;
Revised: More aggressively, some works also reduce the precision of activations, e.g. (Hubara et al., 2016; Rastegari et al., 2016;

Original: Faraone et al., 2018).
Revised: Faraone et al., 2018).

Original: Although the classiﬁcation accuracy loss can be minimized, the universal approximation property is apparently lost, as with limited output precision the network cannot achieve arbitrary accuracy.
Revised: Although the classiﬁcation accuracy loss can be minimized, the universal approximation property is apparently lost, as with limited output precision the network cannot achieve arbitrary accuracy.

Original: Accordingly, we do not include them in the discussion of this paper.
Revised: Accordingly, we do not include them in the discussion of this paper.

Original: The limit of quantization is still unknown while the state-of-the-art keeps getting updated.
Revised: The limit of quantization is still unknown while the state-of-the-art keeps getting updated.

Original: For example, VGGis quantized to 3-bit while maintaining the original accuracy (Leng et al., 2017).
Revised: For example, VGG-16 is quantized to 3-bit while maintaining the original accuracy (Leng et al., 2017).

Original: Motivated by the great empirical success, the training of quantized neural networks has been analyzed theoretically, but not the network capacity (Li et al., 2017; Choi et al., 2016).
Revised: Motivated by the great empirical success, the training of quantized neural networks has been analyzed theoretically, but not the network capacity (Li et al., 2017; Choi et al., 2016).

Original: Universal Approximability and Complexity Bounds: The universal approximability of ReLU networks is proved in Mhaskar & Micchelli (1992) and revisited in (Sonoda & Murata, 2017).
Revised: Universal Approximability and Complexity Bounds: The universal approximability of ReLU networks is proved in Mhaskar & Micchelli (1992) and revisited in (Sonoda & Murata, 2017).

Original: Recently,
Revised: Recently,

Original: Hanin (2017) discusses the expressive power of ReLU networks with bounded width and proves that a ReLU network with width d +1 can approximate any continuous convex function of d variables arbitrarily well.
Revised: Hanin (2017) discusses the expressive power of ReLU networks with bounded width and proves that a ReLU network with width d +1 can approximate any continuous convex function of d variables arbitrarily well.

Original: Shaham et al.
Revised: Shaham et al.

Original: (2016) construct a sparsely-connected depth-4 ReLU network and prove its error bound.
Revised: (2016) construct a sparsely-connected depth-4 ReLU network and prove its error bound.

Original: Liang & Srikant (2016) prove that, for a large class of piecewise smooth functions, a network with ReLU and binary step units can provide an error bound (cid:15) with O (1 /(cid:15) ) layers and
Revised: Liang & Srikant (2016) prove that, for a large class of piecewise smooth functions, a network with ReLU and binary step units can provide an error bound (cid:15) with O (1 /(cid:15) ) layers and

Original: O ( poly log(1 /(cid:15) )) neurons.
Revised: O ( poly log(1 /(cid:15) )) neurons.

Original: The universal approximation property of low displacement rank (LDR) neural networks has been proved by Zhao et al.
Revised: The universal approximation property of low displacement rank (LDR) neural networks has been proved by Zhao et al.

Original: (2017) under a mild condition on the displacement operator, which is the result of another effective technique of neural network compression.
Revised: (2017) under a mild condition on the displacement operator, which is the result of another effective technique of neural network compression.

Original: In this work, we use the bound-by-construction idea for unquantized networks from Yarotsky (2017), but with new and additional construction components essential for quantized networks.

Original: Moreover, we prove the error bound of the constructed network with the new error terms imposed by quantization constraints.

Original:  M ODELS AND A SSUMPTIONS
Revised:  M ODELS AND A SSUMPTIONS

Original: Throughout this paper, we deﬁne ReLU networks as feedforward neural networks with the ReLU activation function σ ( x ) = max (0 , x ) .
Revised: Throughout this paper, we deﬁne ReLU networks as feedforward neural networks with the ReLU activation function σ ( x ) = max (0 , x ) .

Original: The ReLU network considered includes multiple input units, a number of hidden units, and one output unit.
Revised: The ReLU network considered includes multiple input units, a number of hidden units, and one output unit.

Original: Without loss of generality, each unit can only connect to units in the next layer.
Revised: Without loss of generality, each unit can only connect to units in the next layer.

Original: Our conclusions on ReLU networks can be extended to any other networks that use piecewise linear activation functions with ﬁnite breakpoints such as leaky ReLU and ReLU6 immediately, as one can replace a ReLU network by an equivalent one using these activation functions while only increasing the number of units and weights by constant factors (Yarotsky, 2017).
Revised: Our conclusions on ReLU networks can be extended to any other networks that use piecewise linear activation functions with ﬁnite breakpoints such as leaky ReLU and ReLU6 immediately, as one can replace a ReLU network by an equivalent one using these activation functions while only increasing the number of units and weights by constant factors (Yarotsky, 2017).

Original: We denote the  number of distinct weight values as λ ( λ  ≥ 2 ), for both linear quantization and nonlinear quantization.
Revised: We denote the ﬁnite number of distinct weight values as λ ( λ ∈ Z + and λ ≥ 2 ), for both linear quantization and nonlinear quantization.

Revised: For linear quantization, without loss of generality, we assume the ﬁnite number of distinct weight values are given as {− 1 , 1 λ , 2 λ , . . .

Revised: , λ − 1 λ } , where { 1 λ , 2 λ , . . .

Revised: , λ − 1 λ } are uniformly spaced (hence called “linear”) in (0 , 1) and − 1 is used to obtain the negative weight values.

Revised: For nonlinear quantization, we assume the ﬁnite number of distinct weight values are not constrained to any speciﬁc values, i.e., they can take any values as needed.

Original: To store each weight, we only need log( λ ) 1 bits to encode the index, i.e. the bit-width is log( λ ) .
Revised: To store each weight, we only need log( λ ) 2 bits to encode the index, i.e. the bit-width is log( λ ) .

Original: The overhead to store sparse structures can be ignored because it varies depending on the implementation and can be easily reduced to the same order as the weight storage using techniques such as compressed sparse row (CSR) for nonlinear quantization.
Revised: The overhead to store sparse structures can be ignored because it varies depending on the implementation and can be easily reduced to the same order as the weight storage using techniques such as compressed sparse row (CSR) for nonlinear quantization.

Original: The number of bits needed to store the codebook can also be ignored because it has lower order of complexity.
Revised: The number of bits needed to store the codebook can also be ignored because it has lower order of complexity.

Original: We consider any function f in the Sobolev space: f ∈ W n, ∞ ([0 , 1] d ) and
Revised: We consider any function f in the Sobolev space: f ∈ W n, ∞ ([0 , 1] d ) and

Original: [Equation]
Revised: [Equation]

Original: The space W n, ∞ consists of all locally integrable function f : Ω → R such that D n f ∈ L ∞ (Ω) , where | n | ≤ n and Ω is an open set in R d .
Revised: The space W n, ∞ consists of all locally integrable function f : Ω → R such that D n f ∈ L ∞ (Ω) , where | n | ≤ n and Ω is an open set in R d .

Original: We denote this function space as F d,n in this paper.
Revised: We denote this function space as F d,n in this paper.

Original: Note that we only assume weak derivatives up to order n exist where n can be as small as 1 where the function is non-differentiable.
Revised: Note that we only assume weak derivatives up to order n exist where n can be as small as 1 where the function is non-differentiable.

Original: We also only assume the Lipschitz constant to be no greater than 1 for the simplicity of the derivation.
Revised: We also only assume the Lipschitz constant to be no greater than 1 for the simplicity of the derivation.

Original: When the Lipschitz constant is bigger than 1, as long as it is bounded, the whole ﬂow of the proof remains the same though the bound expression will scale accordingly.
Revised: When the Lipschitz constant is bigger than 1, as long as it is bounded, the whole ﬂow of the proof remains the same though the bound expression will scale accordingly.

Revised: When constructing the network to approximate any target function f , we consider two scenarios for deriving the bounds.

Revised: The ﬁrst scenario is called function-dependent structure, where the constructed network topology and their associated weights are all affected by the choice of the target function.

Revised: In contrast, the second scenario is called function-independent structure, where the constructed network topology is independent of the choice of the target function in f ∈ F d,n with a given (cid:15) .

Revised: The principle behind these design choices (the network topology constructions and the choice of weights) is to achieve a tight upper bound as much as possible.

Original: One might consider that we can transform an unquantized network within the error bound to a quantized one in a straightforward way by approximating every continuous-value weight with a combination of discrete weights with arbitrary accuracy.
Revised: One might consider that we can transform an unquantized network within the error bound to a quantized one in a straightforward way by approximating every continuous-value weight with a combination of discrete weights with arbitrary accuracy.

Original: However, the complexity of such approximation (number of discrete weights needed) depends on the distribution of those continuous-value weights (e.g., their min and max), which may vary depending on the training data and network structure and a closed-form expression for the upper bounds is not possible.
Revised: However, the complexity of such approximation (number of discrete weights needed) depends on the distribution of those continuous-value weights (e.g., their min and max), which may vary depending on the training data and network structure and a closed-form expression for the upper bounds is not possible.

Original: As such, a more elegant approach is needed.
Revised: As such, a more elegant approach is needed.

Original: Below we will establish a constructive approach which allows us to bound the approximation analytically.
Revised: Below we will establish a constructive approach which allows us to bound the approximation analytically.

Original: Throughout this paper, we use ”sub-network” to denote a network that is used as a part of the ﬁnal network that approximates the target function.

Original:  F UNCTION - INDEPENDENT S TRUCTURE
Revised:  F UNCTION - INDEPENDENT S TRUCTURE

Original: We start our analysis with function-independent structure, where the network topology is ﬁxed for any function considered.
Revised: We start our analysis with function-independent structure, where the network topology is ﬁxed for any f ∈ F d,n and a given (cid:15) .

Original: We ﬁrst present the approximation of some basic functions by subnetworks in Section 4.1.
Revised: We ﬁrst present the approximation of some basic functions by subnetworks in Section 4.1.

Original: We then present the sub-network that approximates any weight in Section 4.2, and ﬁnally the approximation of general functions and our main results are in Section 4.3.
Revised: We then present the sub-network that approximates any weight in Section 4.2, and ﬁnally the approximation of general functions and our main results are in Section 4.3.

Original: 4. A PPROXIMATION OF SQUARING / MULTIPLICATION
Revised: 4. A PPROXIMATION OF SQUARING / MULTIPLICATION

Original: Proposition 1.
Revised: Proposition 1.

Revised: Denote the design parameter that determines the approximation error bound as r .

Original: Let f rs be a ReLU sub-network with only two weight values 12 and − 12 .
Revised: Let f rs be a ReLU sub-network with only two weight values 12 and − 12 .

Original: The function f s ( x ) = x 2 on the segment [0 , 1] can be approximated by f rs , such that (i) if x = 0 , f rs ( x ) = 0 ; (ii) the approximation error (cid:15) s ≤ 2 − 2( r +1) ; (iii) the depth is O ( r ) ; (iv) the width is a constant; (v) the number of weight is O ( r ) .
Revised: The function f s ( x ) = xon the segment [0 , 1] can be approximated by f rs , such that (i) if x = 0 , f rs ( x ) = 0 ; (ii) the approximation error (cid:15) s ≤ 2 − 2( r +1) ; (iii) the depth is O ( r ) ; (iv) the width is a constant; (v) the number of weight is O ( r ) .

Original: The proof and the details of the sub-network constructed are included in Appendix A.1.
Revised: The proof and the details of the sub-network constructed are included in Appendix A.1.

Original: Once the approximation to squaring function is obtained, we get Proposition 2 by the fact that 2 xy = ( x + y ) 2 − x 2 − y 2 .
Revised: Once the approximation to squaring function is obtained, we get Proposition 2 by the fact that 2 xy = ( x + y ) 2 − x 2 − y 2 .

Original: Proposition 2.
Revised: Proposition 2.

Revised: Denote the design parameter that determines the approximation error bound as r .

Original: Given x ∈ [ − 1 , 1] , y ∈ [ − 1 , 1] , and only two weight values 12 and − 12 , there is a ReLU sub-network with two input units that implements a function × (cid:48) :
Revised: Given x ∈ [ − 1 , 1] , y ∈ [ − 1 , 1] , and only two weight values 12 and − 12 , there is a ReLU sub-network with two input units that implements a function × (cid:48) :

Original: R 2 (cid:55)→ R , such that (i) if x = 0 or y = 0 , then × (cid:48)
Revised: R 2 (cid:55)→ R , such that (i) if x = 0 or y = 0 , then × (cid:48)

Original: ( x, y ) = 0 ; (ii) for any x , y , the error (cid:15) × (cid:48)
Revised: ( x, y ) = 0 ; (ii) for any x , y , the error (cid:15) × (cid:48)

Original: = 
Revised: = | × (cid:48)

Original: |× (cid:48) ( x, y ) − xy | ≤ 6 · 2 − 2( r +1) ; (iii) the depth is O ( r ) ; (iv) the width is a constant; (v) the number of weights is O ( r ) .
Revised: ( x, y ) − xy | ≤ 6 · 2 − 2( r +1) ; (iii) the depth is O ( r ) ; (iv) the width is a constant; (v) the number of weights is O ( r ) .

Original: Proof.
Revised: Proof.

Original: Build three sub-networks f rs as described in Proposition 1 and let
Revised: Build three sub-networks f rs as described in Proposition 1 and let

Original: [Equation]
Revised: [Equation]

Original: Then the statement (i) is followed by property (i) of Proposition 1.
Revised: Then the statement (i) is followed by property (i) of Proposition 1.

Original: Using the error bound in Proposition 1 and Equation (2), we get the error bound of × (cid:48) :
Revised: Using the error bound in Proposition 1 and Equation (2), we get the error bound of × (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: Since a sub-network B abs that computes σ ( x ) + σ ( − x ) can be constructed to get the absolute value of x trivially, we can construct × (cid:48)
Revised: Since a sub-network B abs that computes σ ( x ) + σ ( − x ) can be constructed to get the absolute value of x trivially, we can construct × (cid:48)

Original: ( x, y ) as a linear combination of three parallel f rs and feed them with | x | 2 , | y | 2 , and | x + y | 2 .
Revised: ( x, y ) as a linear combination of three parallel f rs and feed them with | x | 2 , | y | 2 , and | x + y | 2 .

Original: Then claims of statement (iii), (iv), and (v) are also obtained.
Revised: Then claims of statement (iii), (iv), and (v) are also obtained.

Original: 4. A PPROXIMATION OF WEIGHTS
Revised: 4. A PPROXIMATION OF WEIGHTS

Original: Proposition 3.
Revised: Proposition 3.

Revised: Denote the design parameter that determines the approximation error bound as t .

Original: A connection with any weight w ∈ [ − 1 , 1] can be approximated by a ReLU subnetwork that has only λ ≥ 2 distinct weights, such that (i) the sub-network is equivalent to a connection with weight w (cid:48) while the approximation error is bounded by 2 − t i.e., | w (cid:48) − w | < 2 − t ;  the depth  O [Table]
Revised: A connection with any weight w ∈ [ − 1 , 1] can be approximated by a ReLU sub-network that has only λ ≥ 2 distinct weights, such that (i) the sub-network is equivalent to a connection with weight w (cid:48) while the approximation error is bounded by 2 − t i.e., | w (cid:48) − w | < 2 − t ; (ii) the depth is O (cid:16) λt 1 λ − 1 (cid:17) ; (iii) the width is O ( t ) ; (iv) the number of weights is O (cid:16) λt 1 λ − 1 +1 (cid:17) .

Original: [Table]

Original: the width is O

Original: number

Original: [Table]

Original: [Table]

Original: Proof.
Revised: Proof.

Original: Consider that we need a weight w to feed the input x to a unit in the next layer as wx .
Revised: Consider that we need a weight w to feed the input x to a unit in the next layer as wx .

Original: With a limited number of distinct weight values, we can construct the weight we need by cascade and combination.
Revised: With a limited number of distinct weight values, we can construct the weight we need by cascade and combination.

Original: For clarity, we ﬁrst consider w ≥ 0 and x ≥ 0 , and relax these assumptions later.
Revised: For clarity, we ﬁrst consider w ≥ 0 and x ≥ 0 , and relax these assumptions later.

Original: The connections with w = 0 can be seen as an empty sub-network while w = 1 can be easily implemented by 4 units with weight 12 .
Revised: The connections with w = 0 can be seen as an empty sub-network while w = 1 can be easily implemented by 4 units with weight 12 .

Original: Now we show how to represent all integral multiples of 2 − t from 2 − t to 1 − 2 − t , which will lead to the statement (i) by choosing the nearest one from w as w (cid:48) .
Revised: Now we show how to represent all integral multiples of 2 − t from 2 − t to 1 − 2 − t , which will lead to the statement (i) by choosing the nearest one from w as w (cid:48) .

Original: Without loss of generality, we assume t 1 λ − 1 is an integer.
Revised: Without loss of generality, we assume t 1 λ − 1 is an integer.

Original: We use λ weights that include − 12 and W :
Revised: We use λ weights that include − 1 2 and W :

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst construct all w from W c which is deﬁned as
Revised: We ﬁrst construct all w from W c which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Similar to a numeral system with radix equal to t 1 λ − 1 , any w i ∈ W c can be obtained by concatenating weights from W while every weights in W is used no greater than t 1 λ − 1 − 1 times.
Revised: Similar to a numeral system with radix equal to t 1 λ − 1 , any w i ∈ W c can be obtained by concatenating weights from W while every weights in W is used no greater than t 1 λ − 1 − 1 times.

Original: After that, all integral multiples of 2 − t from 2 − t to 1 − 2 − t can be represented by a binary expansion on W c .
Revised: After that, all integral multiples of 2 − t from 2 − t to 1 − 2 − t can be represented by a binary expansion on W c .

Original: Note that connections in the last layer for binary expansion use weight 12 , thus additional 2 − 1 is multiplied to scale the resolution from 2 − ( t − 1) to 2 − t .
Revised: Note that connections in the last layer for binary expansion use weight 12 , thus additional 2 − 1 is multiplied to scale the resolution from 2 − ( t − 1) to 2 − t .

Original: Since for any weight in W c we need to concatenate no more than λ (cid:16) t 1 λ − 1 − 1 (cid:17) weights in a straight line, the sub-network has no greater than λ (cid:16) t 1 λ − 1 − 1 (cid:17) + 1 layers, and no greater than 4 tλ (cid:16) t 1 λ − 1 − 1 (cid:17) + 8 t + 4 weights.
Revised: Since for any weight in W c we need to concatenate no more than λ (cid:16) t 1 λ − 1 − 1 (cid:17) weights in a straight line, the sub-network has no greater than λ (cid:16) t 1 λ − 1 − 1 (cid:17) + 1 layers, and no greater than 4 tλ (cid:16) t 1 λ − 1 − 1 (cid:17) + 8 t + 4 weights.

Original: We now relax the assumption w ≥ 0 .
Revised: We now relax the assumption w ≥ 0 .

Original: When w < 0 , the sub-network can be constructed as w = | w | , while we use − 12 instead of 12 in the last layer.
Revised: When w < 0 , the sub-network can be constructed as w = | w | , while we use − 12 instead of 12 in the last layer.

Original: To relax the assumption x ≥ 0 , we can make a duplication of the sub-network.
Revised: To relax the assumption x ≥ 0 , we can make a duplication of the sub-network.

Original: Let all the weights in the ﬁrst layer of the sub-network be 12 for one and − 12 for the other.
Revised: Let all the weights in the ﬁrst layer of the sub-network be 12 for one and − 12 for the other.

Original: Here we are utilizing the gate property of ReLU.
Revised: Here we are utilizing the gate property of ReLU.

Original: In this way, one sub-network is activated only when x > 0 and the other is activated only when x < 0 .
Revised: In this way, one sub-network is activated only when x > 0 and the other is activated only when x < 0 .

Original: The sign of the output can be adjusted by ﬂipping the sign of weights in the last layer.
Revised: The sign of the output can be adjusted by ﬂipping the sign of weights in the last layer.

Original: Note that the conﬁguration of the sub-network is solely determined by w and works for any input x .
Revised: Note that the conﬁguration of the sub-network is solely determined by w and works for any input x .

Original: The efﬁciency of the weight approximation is critical to the overall complexity.
Revised: The efﬁciency of the weight approximation is critical to the overall complexity.

Original: Compared with the weight selection as { 2 − 1 , 2 − t 1 λ − 1 , 2 − t 2 λ − 1 , . . .
Revised: Compared with the weight selection as { 2 − 1 , 2 − t 1 λ − 1 , 2 − t 2 λ − 1 , . . .

Original: , 2 − t ( λ − 2)
Revised: , 2 − t ( λ − 2)

Original: λ − 1 } , our approximation reduces the number of weights by a factor of t λ − 2 λ − 1 .
Revised: λ − 1 } , our approximation reduces the number of weights by a factor of t λ − 2 λ − 1 .

Original: 4. A PPROXIMATION OF GENERAL FUNCTIONS
Revised: 4. A PPROXIMATION OF GENERAL FUNCTIONS

Original: With the help of Proposition 2 and Proposition 3, we are able to prove the upper bound for general functions.
Revised: With the help of Proposition 2 and Proposition 3, we are able to prove the upper bound for general functions.

Original: Theorem 1.
Revised: Theorem 1.

Original: For any f ∈ F d,n , given λ distinct weights, there is a ReLU network with ﬁxed structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is
Revised: For any f ∈ F d,n , given λ distinct weights, there is a ReLU network with ﬁxed structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is

Original: O (cid:16) λ log 1 λ − 1 (1 /(cid:15) )
Revised: O (cid:16) λ log 1 λ − 1 (1 /(cid:15) )

Original: + log (1 /(cid:15) ) (cid:17) ; (ii) the number of weights is O (cid:16) λ log 1 λ − 1 +1 (1 /(cid:15) )
Revised: + log (1 /(cid:15) ) (cid:17) ; (ii) the number of weights is O (cid:16) λ log 1 λ − 1 +1 (1 /(cid:15) )

Original: (1 /(cid:15) ) dn (cid:17) ; (iii) the number of bits needed to store the network is O (cid:16) λ log ( λ ) log 1 λ − 1 +1 (1 /(cid:15) )
Revised: (1 /(cid:15) ) dn (cid:17) ; (iii) the number of bits needed to store the network is O (cid:16) λ log ( λ ) log 1 λ − 1 +1 (1 /(cid:15) )

Original: (1 /(cid:15) ) dn (cid:17) .
Revised: (1 /(cid:15) ) dn (cid:17) .

Original: The complete proof and the network constructed can be found in Appendix A.2.
Revised: The complete proof and the network constructed can be found in Appendix A.2.

Original: We ﬁrst approximate f by f 2 using the Taylor polynomial of order n − 1 and prove the approximation error bound.
Revised: We ﬁrst approximate f by f 2 using the Taylor polynomial of order n − 1 and prove the approximation error bound.

Original: Note that even when f is non-differentiable (only ﬁrst order weak derivative exists), the Taylor polynomial of order 0 at x = m N can still be used, which takes the form of T m =
Revised: Note that even when f is non-differentiable (only ﬁrst order weak derivative exists), the Taylor polynomial of order 0 at x = m N can still be used, which takes the form of P m =

Original: f ( m N ) .
Revised: f ( m N ) .

Original: Then we approximate f 2 by a ReLU network that is denoted as f (cid:48) with bounded error.
Revised: Then we approximate f 2 by a ReLU network that is denoted as f (cid:48) with bounded error.

Original: After that, we present the  ReLUimplementation the network  f (cid:48) and the complexity.
Revised: After that, we present the quantized ReLU network that implements f (cid:48) and the complexity.

Original: The discussion above focuses on nonlinear quantization which is a more general case compared to linear quantization.
Revised: The discussion above focuses on nonlinear quantization which is a more general case compared to linear quantization.

Original: For linear quantization, which strictly determines the available weight values once λ is given, we can use the same proof for nonlinear quantization except for a different subnetwork for weight approximation with width t and depth t log λ
Revised: For linear quantization, which strictly determines the available weight values once λ is given, we can use the same proof for nonlinear quantization except for a different subnetwork for weight approximation with width t and depth t log λ

Original: +1.
Revised: +1.

Original: Here we give the theorem and the proof is included in Appendix A.3.
Revised: Here we give the theorem and the proof is included in Appendix A.3.

Original: Theorem 2.
Revised: Theorem 2.

Original: For any f ∈ F d,n , given weight maximum precision 1 λ , there is a ReLU network with ﬁxed structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is
Revised: For any f ∈ F d,n , given weight maximum precision 1 λ , there is a ReLU network with ﬁxed structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is

Original: O (log (1 /(cid:15) )) ; (ii) the number of weights is O (cid:16)(cid:16) log (1 /(cid:15) )
Revised: O (log (1 /(cid:15) )) ; (ii) the number of weights is O (cid:16)(cid:16) log (1 /(cid:15) )

Original: + log 2 (1 /(cid:15) ) log λ (cid:17) (1 /(cid:15) )
Revised: + log 2 (1 /(cid:15) ) log λ (cid:17) (1 /(cid:15) )

Original: dn (cid:17) ; (iii) the number of bits needed to store the network is O (cid:16)(cid:0) log( λ ) log (1 /(cid:15) )
Revised: dn (cid:17) ; (iii) the number of bits needed to store the network is O (cid:16)(cid:0) log( λ ) log (1 /(cid:15) )

Original: + log 2 (1 /(cid:15) ) (cid:1) (1 /(cid:15) )
Revised: + log 2 (1 /(cid:15) ) (cid:1) (1 /(cid:15) )

Original: dn (cid:17) .
Revised: dn (cid:17) .

Original:  F UNCTION - DEPENDENT S TRUCTURE
Revised:  F UNCTION - DEPENDENT S TRUCTURE

Original: The network complexity can be reduced if the network topology can be set according to a speciﬁc target function, i.e. function-dependent structure.
Revised: The network complexity can be reduced if the network topology can be set according to a speciﬁc target function, i.e. function-dependent structure.

Original: In this section, we provide an upper bound for function-dependent structure when d = 1 and n = 1 , which is asymptotically better than that of a ﬁxed structure.
Revised: In this section, we provide an upper bound for function-dependent structure when d = 1 and n = 1 , which is asymptotically better than that of a ﬁxed structure.

Original: Speciﬁcally, we ﬁrst deﬁne an approximation to f ( x ) as (cid:101) f
Revised: Speciﬁcally, we ﬁrst deﬁne an approximation to f ( x ) as (cid:101) f

Original: ( x ) that has special properties to match the peculiarity of quantized networks.
Revised: ( x ) that has special properties to match the peculiarity of quantized networks.

Original: Then we use piecewise linear interpolation and “cached” functions (Yarotsky, 2017) to approximate (cid:101) f
Revised: Then we use piecewise linear interpolation and “cached” functions (Yarotsky, 2017) to approximate (cid:101) f

Original: ( x ) by a ReLU network.
Revised: ( x ) by a ReLU network.

Original: 5. F UNCTION T RANSFORMATION
Revised: 5. F UNCTION T RANSFORMATION

Original: While simply using piecewise linear interpolation at the scale of (cid:15) can satisfy the error bound with
Revised: While simply using piecewise linear interpolation at the scale of (cid:15) can satisfy the error bound with

Original: O (1 /(cid:15) ) weights, the complexity can be reduced by ﬁrst doing interpolation at a coarser scale and then ﬁll the details in the intervals to make the error go down to (cid:15) .
Revised: O (1 /(cid:15) ) weights, the complexity can be reduced by ﬁrst doing interpolation at a coarser scale and then ﬁll the details in the intervals to make the error go down to (cid:15) .

Original: By assigning a “cached” function to every interval depending on speciﬁc function and proper scaling, the number of weights is reduced to O (cid:0)(cid:0) log − 1 (1 /(cid:15) )
Revised: By assigning a “cached” function to every interval depending on speciﬁc function and proper scaling, the number of weights is reduced to O (cid:0)(cid:0) log − 1 (1 /(cid:15) )

Original: (cid:1) 1 /(cid:15) (cid:1) when there is no constraint on weight values (Yarotsky, 2017).
Revised: (cid:1) 1 /(cid:15) (cid:1) when there is no constraint on weight values (Yarotsky, 2017).

Original: The key difﬁculty in applying this approach to quantized ReLU networks is that the required linear interpolation at iT exactly where i = 1 , 2 , · · · , T is not feasible because of the constraint on weight selection.
Revised: The key difﬁculty in applying this approach to quantized ReLU networks is that the required linear interpolation at iT exactly where i = 1 , 2 , · · · , T is not feasible because of the constraint on weight selection.

Original: To this end, we transform f ( x ) to (cid:101) f
Revised: To this end, we transform f ( x ) to (cid:101) f

Original: ( x ) such that the approximation error is bounded; the Lipschitz constant is preserved; (cid:101) f (cid:0) iT (cid:1) are reachable for the network under the constraints of weight selection without increasing the requirement on weight precision.
Revised: ( x ) such that the approximation error is bounded; the Lipschitz constant is preserved; (cid:101) f (cid:0) iT (cid:1) are reachable for the network under the constraints of weight selection without increasing the requirement on weight precision.

Original: Then we can apply the interpolation and cached function method on (cid:101) f
Revised: Then we can apply the interpolation and cached function method on (cid:101) f

Original: ( x ) and ﬁnally approximate f ( x ) with a quantized ReLU network.
Revised: ( x ) and ﬁnally approximate f ( x ) with a quantized ReLU network.

Original: Formally, we get the following proposition and the proof can be found in Appendix A.4.
Revised: Formally, we get the following proposition and the proof can be found in Appendix A.4.

Original: Proposition 4.
Revised: Proposition 4.

Original: For any f ∈ F 1 , 1 ,  there exists a function (cid:101) f ( x ) such that (i) (cid:101) f
Revised: For any f ∈ F 1 , 1 , t ∈ Z + , and T ∈ Z + , there exists a function (cid:101) f ( x ) such that (i) (cid:101) f

Original: ( x ) is a continuous function with Lipschitz constant 1; (ii) (cid:101)
Revised: ( x ) is a continuous function with Lipschitz constant 1; (ii) (cid:101)

Original: f ( iT ) =
Revised: f ( iT ) =

Original: (cid:6) Tf (cid:0) iT (cid:1) / 2 − t (cid:7) 2 − t T ; (iii) | (cid:101) f
Revised: (cid:6) Tf (cid:0) iT (cid:1) / 2 − t (cid:7) 2 − t T ; (iii) | (cid:101) f

Original: ( x ) − f ( x ) | < 2 − t T .
Revised: ( x ) − f ( x ) | < 2 − t T .

Original: 5. A PPROXIMATION BY R E LU N ETWORKS
Revised: 5. A PPROXIMATION BY R E LU N ETWORKS

Original: With the help of Proposition 4 and the weight construction method described in Section 4.2, we are able to apply the interpolation and cached function approach.
Revised: With the help of Proposition 4 and the weight construction method described in Section 4.2, we are able to apply the interpolation and cached function approach.

Original: Denoting the output of the network as f (cid:48)(cid:48) ( x ) , we have | f ( x ) − f (cid:48)(cid:48)
Revised: Denoting the output of the network as f (cid:48)(cid:48) ( x ) , we have | f ( x ) − f (cid:48)(cid:48)

Original: ( x ) | = | f
Revised: ( x ) | = | f

Original: ( x ) − (cid:101) f
Revised: ( x ) − (cid:101) f

Original: ( x ) | + | (cid:101) f
Revised: ( x ) | + | (cid:101) f

Original: ( x ) − f (cid:48)(cid:48)
Revised: ( x ) − f (cid:48)(cid:48)

Original: ( x ) | ≤ (cid:15) by choosing appropriate hyperparameters which are detailed in Appendix A.5 and the network complexity is obtained accordingly.
Revised: ( x ) | ≤ (cid:15) by choosing appropriate hyperparameters which are detailed in Appendix A.5 and the network complexity is obtained accordingly.

Original: Theorem 3.
Revised: Theorem 3.

Original: For any f ∈ F 1 , 1 , given λ distinct weights, there is a ReLU network with function-dependent structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is O (cid:16) λ (log log (1 /(cid:15) ))
Revised: For any f ∈ F 1 , 1 , given λ distinct weights, there is a ReLU network with function-dependent structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is O (cid:16) λ (log log (1 /(cid:15) ))

Original: 1 λ − 1 + log (1 /(cid:15) ) (cid:17) ; (ii) the number of weights is
Revised: 1 λ − 1 + log (1 /(cid:15) ) (cid:17) ; (ii) the number of weights is

Original: O (cid:16) λ (log log (1 /(cid:15) ))
Revised: O (cid:16) λ (log log (1 /(cid:15) ))

Original: 1 λ − 1 +1 + (1 /(cid:15) ) (cid:17) (iii) the number of bits needed to store the network is
Revised: 1 λ − 1 +1 + (1 /(cid:15) ) (cid:17) (iii) the number of bits needed to store the network is

Original: O (cid:16) log λ (cid:16) λ (log log [Table] [Table]
Revised: O (cid:16) log λ (cid:16) λ (log log (1 /(cid:15) ))

Original: [Table] λ [Table] + 
Revised: 1 λ − 1 +1 + (1 /(cid:15) )

Original: [Table]

Original: + (

Original: [Table]

Original: [Table]

Original: ) (cid:17)(cid:17) .
Revised: (cid:17)(cid:17) .

Original: Using the different weight construction approach as in the case of function-independent structure, we have the result for linear quantization:
Revised: Using the different weight construction approach as in the case of function-independent structure, we have the result for linear quantization:

Original: Theorem 4.
Revised: Theorem 4.

Original: For any f ∈ F 1 , 1 , given weight maximum precision 1 λ , there is a ReLU network with function-dependent structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is O (log (1 /(cid:15) )) ; (ii) the number of weights is O (1 /(cid:15) ) ; (iii) the number of bits needed to store the network is O (log( λ ) /(cid:15) ) .
Revised: For any f ∈ F 1 , 1 , given weight maximum precision 1 λ , there is a ReLU network with function-dependent structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is O (log (1 /(cid:15) )) ; (ii) the number of weights is O (1 /(cid:15) ) ; (iii) the number of bits needed to store the network is O (log( λ ) /(cid:15) ) .

Original:  B OUND - BASED O PTIMAL B IT - WIDTH
Revised:  B OUND - BASED O PTIMAL B IT - WIDTH

Original: In this section, we ﬁrst introduce the optimal bit-width problem and then show how a theoretical bound could potentially be used to estimate the optimal bit-width of a neural network.
Revised: In this section, we ﬁrst introduce the optimal bit-width problem and then show how a theoretical bound could potentially be used to estimate the optimal bit-width of a neural network.

Original: Because of the natural need and desire of comparison with competitive approaches, most quantization techniques are evaluated on some popular reference networks, without modiﬁcation of the[Figure] network topology.
Revised: Because of the natural need and desire of comparison with competitive approaches, most quantization techniques are evaluated on some popular reference networks, without modiﬁcation of the network topology.

Original: On the one hand, the advancement of lossless quantization almost stalls at a bit-width between two and six (Han et al., 2015a; Choi et al., 2016;
Revised: On the one hand, the advancement of lossless quantization almost stalls at a bit-width between two and six (Han et al., 2015a; Choi et al., 2016;

Original: Sze et al., 2017; Blott et al., 2017; Su et al., 2018; Faraone et al., 2018).
Revised: Sze et al., 2017; Blott et al., 2017; Su et al., 2018; Faraone et al., 2018).

Original: A speciﬁc bit-width depends on the compactness of the reference network and the difﬁculty of the task.
Revised: A speciﬁc bit-width depends on the compactness of the reference network and the difﬁculty of the task.

Original: On the other hand, the design space, especially the different combinations of topology and bit-width, is largely underexplored because of the complexity, resulting in sub-optimal results.
Revised: On the other hand, the design space, especially the different combinations of topology and bit-width, is largely underexplored because of the complexity, resulting in sub-optimal results.

Original: A recent work by Su et al.
Revised: A recent work by Su et al.

Original: (2018) empirically validates the beneﬁt of exploring ﬂexible network topology during quantization.
Revised: (2018) empirically validates the beneﬁt of exploring ﬂexible network topology during quantization.

Original: That work adds a simple variable of network expanding ratio, and shows that a bit-width of four achieves the best cost-accuracy trade-off among limited options in { 1 , 2 , 4 , 8 , 16 , 32 } .
Revised: That work adds a simple variable of network expanding ratio, and shows that a bit-width of four achieves the best cost-accuracy trade-off among limited options in { 1 , 2 , 4 , 8 , 16 , 32 } .

Original: Some recent effort on using reinforcement learning to optimize the network hyper-parameters (He et al., 2018) could potentially be used to address this issue.
Revised: Some recent effort on using reinforcement learning to optimize the network hyper-parameters (He et al., 2018) could potentially be used to address this issue.

Original: But the current design space is still limited to a single variable per layer (such as the pruning ratio based on a reference network).
Revised: But the current design space is still limited to a single variable per layer (such as the pruning ratio based on a reference network).

Original: How to estimate an optimal bit-width for a target task without training could be an interesting research direction in the future.
Revised: How to estimate an optimal bit-width for a target task without training could be an interesting research direction in the future.

Original: The memory bound expression as derived in this paper helps us to determine whether there is an optimal λ that would lead to the lowest bound and most compact network (which can be translated to computation cost in a fully connected structure) for a given target function.
Revised: The memory bound expression as derived in this paper helps us to determine whether there is an optimal λ that would lead to the lowest bound and most compact network (which can be translated to computation cost in a fully connected structure) for a given target function.

Original: For example, by dropping the lower-order term and ignoring the rounding operator, our memory bound can be simpliﬁed as
Revised: For example, by dropping the lower-order term and ignoring the rounding operator, our memory bound can be simpliﬁed as

Original: [Equation]
Revised: [Equation]

Original: where θ 1 is a constant determined by (cid:15), n, and d .
Revised: where θ 1 is a constant determined by (cid:15), n, and d .

Original: We can ﬁnd an optimal λ that minimizes M ( λ ) :
Revised: We can ﬁnd an optimal λ that minimizes M ( λ ) :

Original: [Equation]
Revised: [Equation]

Original: As is detailed in Appendix B, we prove that there exists one and only one local minimum (hence global minimum) in the range of [2 , ∞ ) whenever (cid:15) <
Revised: As is detailed in Appendix B, we prove that there exists one and only one local minimum (hence global minimum) in the range of [2 , ∞ ) whenever (cid:15) <

Original: We also show that λ opt is determined by log (cid:0) 3 n 2 d /(cid:15) (cid:1) , which can be easily dominated by d .
Revised: We also show that λ opt is determined by log (cid:0) 3 n 2 d /(cid:15) (cid:1) , which can be easily dominated by d .

Original: Based on such results, we quantitatively evaluate the derivative of M ( λ ) , and based on which the optimal bit-width log( λ opt ) under various settings in Figure 1a and Figure 1b, respectively.
Revised: Based on such results, we quantitatively evaluate the derivative of M ( λ ) , and based on which the optimal bit-width log( λ opt ) under various settings in Figure 1a and Figure 1b, respectively.

Original: In Figure 1b, we also mark the input dimension of a few image data sets.
Revised: In Figure 1b, we also mark the input dimension of a few image data sets.

Original: It is apparent to see that the optimal bit width derived from M ( λ ) is dominated by d and lies between one and four for a wide range of input size.
Revised: It is apparent to see that the optimal bit width derived from M ( λ ) is dominated by d and lies between one and four for a wide range of input size.

Original: This observation is consistent with most existing empirical research results, hence showing the potential power of our theoretical bound derivation.
Revised: This observation is consistent with most existing empirical research results, hence showing the potential power of our theoretical bound derivation.

Original: Since the bounds are derived for fully connected networks and depend on the construction approach, the interesting proximity between log( λ opt ) and the empirical results can not be viewed as a strict theoretical explanation.
Revised: Since the bounds are derived for fully connected networks and depend on the construction approach, the interesting proximity between log( λ opt ) and the empirical results cannot be viewed as a strict theoretical explanation.

Original: Regardless, we show that the complexity bound may be a viable approach to understand the optimal bit-width problem, thus potentially accelerating the hyper-parameter optimization of deep neural networks.
Revised: Regardless, we show that the complexity bound may be a viable approach to understand the optimal bit-width problem, thus potentially accelerating the hyper-parameter optimization of deep neural networks.

Original: We defer such a thorough investigation of the optimal bit-width or optimal hybrid bit-width conﬁguration across the network to our future work. 
Revised: We defer such a thorough investigation of the optimal bit-width or optimal hybrid bit-width conﬁguration across the network to our future work. [Figure]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this section, we further discuss the bound of nonlinear quantization with a function-independent structure as the generality of nonlinear quantization.
Revised: In this section we further discuss the bound of nonlinear quantization with a function-independent structure as the generality of nonlinear quantization.

Original: The availability of unquantized functionindependent structures in literature also makes it an excellent reference for comparison.
Revised: The availability of unquantized functionindependent structures in literature also makes it an excellent reference for comparison

Original: Comparison with the Upper Bound: The quality of an upper bound lies on its tightness.
Revised: Comparison with the Upper Bound: The quality of an upper bound lies on its tightness.

Original: Compared with the most recent work on unquantized ReLU networks (Yarotsky, 2017), where the upper bound on the number of weights to attain an approximation error (cid:15) is given by O (cid:16) log(1 /(cid:15) )
Revised: Compared with the most recent work on unquantized ReLU networks (Yarotsky, 2017), where the upper bound on the number of weights to attain an approximation error (cid:15) is given by O (cid:16) log(1 /(cid:15) )

Original: (1 /(cid:15) ) dn (cid:17) , our result for a quantized ReLU network is given by O (cid:16) λ (cid:16) log 1 λ − 1 +1 (1 /(cid:15) )
Revised: (1 /(cid:15) ) dn (cid:17) , our result for a quantized ReLU network is given by O (cid:16) λ (cid:16) log 1 λ − 1 +1 (1 /(cid:15) )

Original: (cid:17) (1 /(cid:15) )
Revised: (cid:17) (1 /(cid:15) )

Original: d n (cid:17) , which translates to an increase by a factor of λ (cid:16) log 1 λ − 1 (1 /(cid:15) )
Revised: dn (cid:17) , which translates to an increase by a factor of λ (cid:16) log 1 λ − 1 (1 /(cid:15) )

Original: (cid:17) .
Revised: (cid:17) .

Original: Loosely speaking, this term reﬂects the loss of expressive power because of weight quantization, which decreases quickly as λ increases.
Revised: Loosely speaking, this term reﬂects the loss of expressive power because of weight quantization, which decreases quickly as λ increases.

Original: Comparison with the Lower Bound: We also compare our bound with the lower bound of the number of weights needed to attain an error bound of (cid:15) to have a better understanding on the tightness of the bound.
Revised: Comparison with the Lower Bound: We also compare our bound with the lower bound of the number of weights needed to attain an error bound of (cid:15) to have a better understanding on the tightness of the bound.

Original: We use the lower bound for unquantized ReLU networks from (Yarotsky, 2017), as it is also a natural lower bound for quantized ReLU networks.
Revised: We use the lower bound for unquantized ReLU networks from (Yarotsky, 2017), as it is also a natural lower bound for quantized ReLU networks.

Original: Under the same growth rate of depth, the lower bound is given by Ω(log − 3 (1 /(cid:15) )
Revised: Under the same growth rate of depth, the lower bound is given by Ω(log − 3 (1 /(cid:15) )

Original: (1 /(cid:15) )
Revised: (1 /(cid:15) )

Original: d/n ) , while our upper bound is, within a polylog factor when λ is a constant, O ( λ log 1 λ − 1 +1 (1 /(cid:15) )
Revised: d/n ) , while our upper bound is, within a polylog factor when λ is a constant, O ( λ log 1 λ − 1 +1 (1 /(cid:15) )

Original: (1 /(cid:15) )
Revised: (1 /(cid:15) )

Original: d/n ) .
Revised: d/n ) .

Original: The comparison validates the good tightness of our upper bound.
Revised: The comparison validates the good tightness of our upper bound.

Original: The Upper Bound of Overhead: More importantly, the above comparison yields an upper bound on the possible overhead induced by quantization.
Revised: The Upper Bound of Overhead: More importantly, the above comparison yields an upper bound on the possible overhead induced by quantization.

Original: By comparing the expressions of two bounds while treating λ as a constant, we can show that, to attain the same approximation error bound (cid:15) , the number of weights needed by a quantized ReLU network is no more than O (log 5 (1 /(cid:15) ))
Revised: By comparing the expressions of two bounds while treating λ as a constant, we can show that, to attain the same approximation error bound (cid:15) , the number of weights needed by a quantized ReLU network is no more than O (log 5 (1 /(cid:15) ))

Original: times that needed by an unquantized ReLU network.
Revised: times that needed by an unquantized ReLU network.

Original: Note that this factor is of much lower order than the lower bound Ω(log − 3 (1 /(cid:15) )
Revised: Note that this factor is of much lower order than the lower bound Ω(log − 3 (1 /(cid:15) )

Original: (1 /(cid:15) )
Revised: (1 /(cid:15) )

Original: d/n ) .
Revised: d/n ) .

Original: This little overhead introduced by weight quantization explains in part the empirical success on network compression and acceleration by quantization and also answers in part the questions as raised in Section 1.
Revised: This little overhead introduced by weight quantization explains in part the empirical success on network compression and acceleration by quantization and also answers in part the questions as raised in Section 1.

Original: Given the signiﬁcant beneﬁts of quantization in term of memory and computation efﬁciency, we anticipate that the use of quantization networks will continue to grow, especially on resource-limited platforms.
Revised: Given the signiﬁcant beneﬁts of quantization in term of memory and computation efﬁciency, we anticipate that the use of quantization networks will continue to grow, especially on resource-limited platforms.

Original: Future Work: There remain many other avenues for future investigation.
Revised: Future Work: There remain many other avenues for future investigation.

Original: For example, although we derived the ﬁrst upper bound of quantized neural networks, the lower bound is still missing.
Revised: For example, although we derived the ﬁrst upper bound of quantized neural networks, the lower bound is still missing.

Original: If a tight lower bound of the network size is established, it could be combined with the upper bound to give a much better estimation of required resources and the optimal bit-width.
Revised: If a tight lower bound of the network size is established, it could be combined with the upper bound to give a much better estimation of required resources and the optimal bit-width.

Original: We believe the trends associated with the bounds can also be useful and deserve some further investigation.
Revised: We believe the trends associated with the bounds can also be useful and deserve some further investigation.

Original: For example, the trend may help hardware designers in their early stage of design exploration without the need of lengthy training.
Revised: For example, the trend may help hardware designers in their early stage of design exploration without the need of lengthy training.

Original: While we assume a uniform bit-width across all layers, another area of research is to allow different bit-widths in different layers, which could achieve better efﬁciency and potentially provide theoretical justiﬁcations on the emerging trend of hybrid quantization (Zhang et al., 2017;
Revised: While we assume a uniform bit-width across all layers, another area of research is to allow different bit-widths in different layers, which could achieve better efﬁciency and potentially provide theoretical justiﬁcations on the emerging trend of hybrid quantization (Zhang et al., 2017;

Original: Wang et al., 2018)
Revised: Wang et al., 2018).

Original: A P ROOFS A. T HE PROOF OF P ROPOSITION
Revised: A P ROOFS A. T HE PROOF OF P ROPOSITION

Original: Proposition 1.
Revised: Proposition 1.

Revised: Denote the design parameter that determines the approximation error bound as r .

Revised: Let f rs be a ReLU sub-network with only two weight values 12 and − 12 .

Original: The function f s ( x ) = x 2 on the segment [0 , 1] can be approximated bya ReLU sub-network f rswith only two weight values 12 and − 12 , such that (i) if x = 0 , f rs ( x ) = 0 ; (ii) the approximation error (cid:15) s ≤ 2 − 2( r +1) ; (iii) the depth is O ( r ) ; (iv) the width is a constant; (v) the number of weight is O ( r ) .
Revised: The function f s ( x ) = xon the segment [0 , 1] can be approximated by f rs , such that (i) if x = 0 , f rs ( x ) = 0 ; (ii) the approximation error (cid:15) s ≤ 2 − 2( r +1) ; (iii) the depth is O ( r ) ; (iv) the width is a constant; (v) the number of weight is O ( r ) .

Original: Proof.
Revised: Proof.

Original: For f s ( x ) = x 2 , let f rs be the piecewise linear interpolation of f with 2 r + 1 uniformly distributed breakpoints k 2 r , k = 0 , . .
Revised: For f s ( x ) = x 2 , let f rs be the piecewise linear interpolation of f with 2 r + 1 uniformly distributed breakpoints k 2 r , k = 0 , . .

Original: , 2 r .
Revised: , 2 r .

Original: We have f rs (cid:0) k 2 r (cid:1) =
Revised: We have f rs (cid:0) k 2 r (cid:1) =

Original: (cid:0) k 2 r (cid:1) 2 , k = 0 , . . .
Revised: (cid:0) k 2 r (cid:1) 2 , k = 0 , . . .

Original: , 2 r and the approximation error (cid:15) s =
Revised: , 2 r and the approximation error (cid:15) s =

Original: || f
Revised: || f

Original: rs ( x ) − f s ( x ) || ∞ ≤ 2 − 2( r +1) .
Revised: rs ( x ) − f s ( x ) || ∞ ≤ 2 − 2( r +1) .

Original: We can use the function g : [0 , 1] (cid:55)→ [0 , 1] to obtain f rs : 
Revised: We can use the function g : [0 , 1] (cid:55)→ [0 , 1] to obtain f rs : g (

Original: [Equation]
Revised: [Equation]

Original:  where g ◦ i ( x ) is the i -th iterate of g ( x ) .
Revised: = [Equation] [Equation] 9) where g ◦ i ( x ) is the i -th iterate of g ( x ) .

Original: Since g ( x ) can be implemented by a ReLU network as g ( x ) = 2 σ ( x ) − 4 σ ( x − 12 ) , g ◦ r ( x ) can be obtained by concatenating such implementation of g ( x ) for r times.
Revised: Since g ( x ) can be implemented by a ReLU sub-network as g ( x ) = 2 σ ( x ) − 4 σ ( x − 12 ) , g ◦ r ( x ) can be obtained by concatenating such implementation of g ( x ) for r times.

Original: Now, to implement f rs ( x ) based on g ◦ r ( x ) , all we need are weights { 2 − 2 , 2 − 4 , · · · , 2 − 2( r − 1) , 2 − 2 r } , which can be easily constructed with additional 2 r layers and the weight 12 . 
Revised: Now, to implement f rs ( x ) based on g ◦ r ( x ) , all we need are weights { 2 − 2 , 2 − 4 , · · · , 2 − 2( r − 1) , 2 − 2 r } , which can be easily constructed with additional 2 r layers and the weight 12 . Note that a straightforward implementation will have to scale g ◦ i ( x ) separately (multiply by different number of 12 ) before subtract them from x because each g ◦ i ( x ) have a different coefﬁcient. Then the width of the network will be Θ( r ) . Here we use a “pre-scale” method to reduce the network width from Θ( r ) to a constant.

Original: Then, f rs ( x ) can be implemented by a ReLU sub-network  as in Figure 2.
Revised: The network constructed is shown in Figure 2. The one-layer subnetwork that implements g ( x ) and the one-layer sub-network that scales the input by 4 are denoted as B g and B m respectively. Some units are copied to compensate the scaling caused by 12 . The intermediate results g ◦ i ( x ) are computed by the concatenation of B g at the ( i +1) -th layer.

Original: A connection from or to a block indicates connections to  all units in the block.
Revised: The ﬁrst B m takes x as input and multiply it by 4 . The output of i -th B m is subtracted by g ◦ i ( x ) and then fed to the next B m to be multiplied by 4 again. There are r layers of B m and all g ◦ i ( x ) are scaled by 2 2( r − i ) respectively.

Original: Some units are copied to compensate the scaling caused byand to facilitate the construction of higher level networks.
Revised: As a result, we obtain 2 2 r x − (cid:80) ri =1 2 2( r − i ) g ◦ i ( x ) after the last B m . Then it is scaled by 2 − 2 r in the later 2 r layers to get f rs ( x ) . In this way, we make all g ◦ i ( x ) sharing the same scaling link and a constant width can be achieved.

Original: A. T HE PROOF OF T HEOREM
Revised: A. T HE PROOF OF T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: For any f ∈ F d,n , given λ distinct weights, there is a ReLU network with ﬁxed structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is
Revised: For any f ∈ F d,n , given λ distinct weights, there is a ReLU network with ﬁxed structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is

Original: O (cid:16) λ log 1 λ − 1 (1 /(cid:15) )
Revised: O (cid:16) λ log 1 λ − 1 (1 /(cid:15) )

Original: + log (1 /(cid:15) ) (cid:17) ; (ii) the number of weights is O (cid:16) λ log 1 λ − 1 +1 (1 /(cid:15) )
Revised: + log (1 /(cid:15) ) (cid:17) ; (ii) the number of weights is O (cid:16) λ log 1 λ − 1 +1 (1 /(cid:15) )

Original: (1 /(cid:15) ) dn (cid:17) ; (iii) the number of bits needed to store the network is O (cid:16) λ log ( λ ) log 1 λ − 1 +1 (1 /(cid:15) )
Revised: (1 /(cid:15) ) dn (cid:17) ; (iii) the number of bits needed to store the network is O (cid:16) λ log ( λ ) log 1 λ − 1 +1 (1 /(cid:15) )

Original: (1 /(cid:15) ) dn (cid:17) .
Revised: (1 /(cid:15) ) dn (cid:17) .

Original: Proof.
Revised: Proof.

Original: The proof is composed of four steps.
Revised: The proof is composed of four steps.

Original: We ﬁrst approximate f by f 2 using the Taylor polynomial of order n − 1 and prove the approximation error bound.
Revised: We ﬁrst approximate f by f 2 using the Taylor polynomial of order n − 1 and prove the approximation error bound.

Original: Note that even when f is nondifferentiable (only ﬁrst order weak derivative exist), the Taylor polynomial of order 0 at x = m N can still be used, which takes the form of T m =
Revised: Note that even when f is nondifferentiable (only ﬁrst order weak derivative exist), the Taylor polynomial of order 0 at x = m N can still be used, which takes the form of P m =

Original: f ( m N ) .
Revised: f ( m N ) .

Original: Then we approximate f 2 by a ReLU network that is denoted as f (cid:48) with bounded error.
Revised: Then we approximate f 2 by a ReLU network that is denoted as f (cid:48) with bounded error.

Original: After that, we present the  ReLU implementation the network f (cid:48) and the complexity of the network.
Revised: After that, we present the quantized ReLU network that implements the network f (cid:48) and the complexity of the network.

Original: We use a partition of unity on [0 , 1] d : (cid:80) m ψ m ( x ) ≡ 1 , x ∈ [0 , 1] d where m = ( m 1 , · · · , m d ) ∈ { 0 , 1 , · · · , N } d , and h ( x ) is deﬁned as follows:
Revised: We use a partition of unity on [0 , 1] d : (cid:80) m ψ m ( x ) ≡ 1 , x ∈ [0 , 1] d where m = ( m 1 , · · · , m d ) ∈ { 0 , 1 , · · · , N } d , and h ( x ) is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where N is a constant and
Revised: where N is a constant and

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Note that supp ψ m ⊂ { x : (cid:12)(cid:12) x k − m k N (cid:12)(cid:12) <
Revised: Note that supp ψ m ⊂ { x : (cid:12)(cid:12) x k − m k N (cid:12)(cid:12) <

Original: 1 N ∀ k } .
Revised: 1 N ∀ k } .

Original: For all m , we have the order n − 1 Taylor polynomial for the function f at x = m N as
Revised: For all m , we have the order n − 1 Taylor polynomial for the function f at x = m N as

Original: [Equation]
Revised: [Equation]

Original: To get a more realizable approximation for quantized networks, we deﬁne T (cid:48) m
Revised: To get a more realizable approximation for quantized networks, we deﬁne P (cid:48) m

Original: ( x ) = (cid:80) n : | n | <n β m , n (cid:0) x − m N (cid:1) n where β m , n is D n f n ! (cid:12)(cid:12) (cid:12) x
Revised: ( x ) = (cid:80) n : | n | <n β m , n (cid:0) x − m N (cid:1) n where β m , n is D n f n ! (cid:12)(cid:12)(cid:12) x

Original: = m N rounded to the nearest integral multiple of 1 n (cid:0) dN (cid:1) n −| n | .
Revised: = m N rounded to the nearest integral multiple of 1 n (cid:0) dN (cid:1) n −| n | .

Original: Then we get an approximation to f using T (cid:48) m and ψ m as f 2 (cid:44) (cid:80) m ∈{ 0 , ··· ,N } d ψ m T (cid:48) m .
Revised: Then we get an approximation to f using P (cid:48) m and ψ m as f 2 (cid:44) (cid:80) m ∈{ 0 , ··· ,N } d ψ m P (cid:48) m .

Original: Then the approximation error of f 2 is bounded by Equation (13).
Revised: Then the approximation error of f 2 is bounded by Equation (13).

Original: [Equation]
Revised: [Equation]

Original: The second step follows ψ m ( x ) = 0 when x / ∈ supp ψ m .
Revised: The second step follows ψ m ( x ) = 0 when x / ∈ supp ψ m .

Original: In the third step we turn the sum to multiplication, because for any x there are up to 2 d terms ψ m ( x ) that are not equal to zero.
Revised: In the third step we turn the sum to multiplication, because for any x there are up to 2 d terms ψ m ( x ) that are not equal to zero.

Original: The fourth step uses a Lagrange’s form of the Taylor remainder.
Revised: The fourth step uses a Lagrange’s form of the Taylor remainder.

Original: The ﬁfth step follows different round precision of β m , n in different order and the fact that the number of terms with order i is not greater than d i .
Revised: The ﬁfth step follows different round precision of β m , n in different order and the fact that the number of terms with order i is not greater than d i .

Original: We rewrite f 2 as
Revised: We rewrite f 2 as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Note that β m , n is a constant and thus f 2 is a linear combination of at most d n ( N + 1) d terms of f m , n ( x ) .
Revised: Note that β m , n is a constant and thus f 2 is a linear combination of at most d n ( N + 1) d terms of f m , n ( x ) .

Original: Note that when d = 1 , the number of terms should be n ( N +1) d instead; but for simplicity of presentation we loosely use the same expression as they are on the same order.
Revised: Note that when d = 1 , the number of terms should be n ( N +1) d instead; but for simplicity of presentation we loosely use the same expression as they are on the same order.

Original: We deﬁne an approximation to f m , n ( x ) as f (cid:48) m , n ( x ) .
Revised: We deﬁne an approximation to f m , n ( x ) as f (cid:48) m , n ( x ) .

Original: The only difference between f m , n ( x ) and f (cid:48) m , n ( x ) is that all multiplication operations are approximated by × (cid:48) as discussed in Proposition 2.
Revised: The only difference between f m , n ( x ) and f (cid:48) m , n ( x ) is that all multiplication operations are approximated by × (cid:48) as discussed in Proposition 2.

Original: Consider that if we construct our function × (cid:48) with | × (cid:48) ( x, y ) − xy | < (cid:15) × (cid:48)
Revised: Consider that if we construct our function × (cid:48) with | × (cid:48) ( x, y ) − xy | < (cid:15) × (cid:48)

Original: = 2 − 2( r +1) , then
Revised: = 2 − 2( r +1) , then

Original: [Equation]
Revised: [Equation]

Original: Applying Equation (16) to | f (cid:48) m , n ( x ) − f m , n ( x ) | repeatedly, we bound it to Equation (17). 
Revised: Applying Equation (16) to | f (cid:48) m , n ( x ) − f m , n ( x ) | repeatedly, we bound it to Equation (17). [Equation] =

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we deﬁne our approximation to f ( x ) as f (cid:48)
Revised: we deﬁne our approximation to f ( x ) as f (cid:48)

Original: ( x ) :
Revised: ( x ) :

Original: [Equation]
Revised: [Equation]

Original: Using Equation (17), we get the error bound of the approximation to f 2 ( x ) as in Equation (19).
Revised: Using Equation (17) [Equation] we get the error bound of the approximation to f 2 ( x ) as in Equation (19)

Original: [Equation]
Revised: [Equation]

Original: The second line follows again the support property and statement (i) of Proposition 2.
Revised: The second line follows again the support property and statement (i) of Proposition 2.

Original: The third line uses the bound | β m , n | ≤ 1 .
Revised: The third line uses the bound | β m , n | ≤ 1 .

Original: The fourth line is obtained by inserting Equation (17).
Revised: The fourth line is obtained by inserting Equation (17).

Original: Then the ﬁnal approximation error bound is as follows:
Revised: Then the ﬁnal approximation error bound is as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Using statement (ii) of Proposition 2 and choosing r as r = log(6 N n ( d + n − 1))− 1 , the approximation error turns to
Revised: Using statement (ii) of Proposition 2 and choosing r as r = log(6 N n ( d + n − 1))− 1 , the approximation error turns to

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Table]

Original: Therefore, for any f ∈ F d,n and (cid:15) ∈ (0 , 1) , there is a ReLU network f (cid:48) that approximate f with error bound (cid:15) if we choose N ≥ (cid:0) 3 · 2 d d n (cid:14) (cid:15) ) 1 n .
Revised: Therefore, for any f ∈ F d,n and (cid:15) ∈ (0 , 1) , there is a ReLU network f (cid:48) that approximate f with error bound (cid:15) if we choose N ≥ (cid:0) 3 · 2 d d n (cid:14) (cid:15) ) 1 n .

Original: We now present the construction of the network for f (cid:48) ( x ) .
Revised: We now present the construction of the network for f (cid:48) ( x ) .

Original: If every f (cid:48) m , n ( x ) can be computed by a sub-network, then f (cid:48) ( x ) is simply a weighted sum of all outputs of f (cid:48) m , n ( x ) .
Revised: If every f (cid:48) m , n ( x ) can be computed by a sub-network, then f (cid:48) ( x ) is simply a weighted sum of all outputs of f (cid:48) m , n ( x ) .

Original: By Proposition 3, we can implement the needed weights β m , n by choosing t = log nN n d n .
Revised: By Proposition 3, we can implement the needed weights β m , n by choosing t = log nN n d n .

Original: Then we simplify the task to constructing f (cid:48) m , n ( x ) .
Revised: Then we simplify the task to constructing f (cid:48) m , n ( x ) .

Original: × (cid:48) can be implemented as discussed in Proposition 2.
Revised: × (cid:48) can be implemented as discussed in Proposition 2.

Original: For h ( (cid:0) 3 N (cid:0) x
Revised: For h ( (cid:0) 3 N (cid:0) x

Original: i − m i N (cid:1)(cid:1) , noticing that h ( x ) =
Revised: i − m i N (cid:1)(cid:1) , noticing that h ( x ) =

Original: h ( − x ) , we can ﬁrst compute | x i − m i N | as σ ( x i − m i N ) + σ ( m i N − x i ) and then scale it to 3 N (cid:0) | x i − m i N | (cid:1) .
Revised: h ( − x ) , we can ﬁrst compute | x i − m i N | as σ ( x i − m i N ) + σ ( m i N − x i ) and then scale it to 3 N (cid:0) | x i − m i N | (cid:1) .

Original: The implementation of h ( x ) can thus be simpliﬁed as 1 − σ ( x − 1) + σ ( x − 2) since the input is nonnegative.
Revised: The implementation of h ( x ) can thus be simpliﬁed as 1 − σ ( x − 1) + σ ( x − 2) since the input is nonnegative.

Original: Furthermore, by choosing N as cd 2 where c ∈ N and c > 1 , 1 N is an integral multiple of 1 n (cid:0) dN (cid:1) n if n > 1 .
Revised: Furthermore, by choosing N as cd 2 where c ∈ N and c > 1 , 1 N is an integral multiple of 1 n (cid:0) dN (cid:1) n if n > 1 .

Original: When n = 1 , 1 N is an integral multiple of (cid:16) 1 n (cid:0) dN (cid:1) n (cid:17) 2 .
Revised: When n = 1 , 1 N is an integral multiple of (cid:16) 1 n (cid:0) dN (cid:1) n (cid:17) 2 .

Original: As discussed in Proposition 3, we build a weight construction network B w in the way that all integral multiples of the minimal precision can be obtained.
Revised: As discussed in Proposition 3, we build a weight construction network B w in the way that all integral multiples of the minimal precision can be obtained.

Original: Therefore, all m i N can be obtained in the same way as β m , [Table]
Revised: Therefore, all m i N can be obtained in the same way as β m , n , except that we need to concatenate two weight construction sub-networks.

Original: Now we analyze the complexity of the network.
Revised: Now we analyze the complexity of the network.

Original: The implementation of f (cid:48) ( x ) is shown in Figure 3.
Revised: The implementation of f (cid:48) ( x ) is shown in Figure 3.

Original: The function and size of blocks are listed in Table 1.
Revised: The function and size of blocks are listed in Table 1.

Original: Then we are able to obtain the complexity of the network.
Revised: Then we are able to obtain the complexity of the network.

Original: While we can write the complexity of the network in an explicit expression, here we use the O notation for clarity.
Revised: While we can write the complexity of the network in an explicit expression, here we use the O notation for clarity.

Original: Let N d , N w , N b be the depth, the number of weights, and the number of bits required respectively.
Revised: Let N d , N w , N b be the depth, the number of weights, and the number of bits required respectively.

Original: The weight construction blocks B w have the highest order of number of weights and we have N w =
Revised: The weight construction blocks B w have the highest order of number of weights and we have N w =

Original: O (cid:16) λt 1 λ − 1 +1 N d (cid:17) .
Revised: O (cid:16) λt 1 λ − 1 +1 N d (cid:17) .

Original: Meanwhile, we get N d = O (cid:16) λt 1 λ − 1 + log N (cid:17) .
Revised: Meanwhile, we get N d = O (cid:16) λt 1 λ − 1 + log N (cid:17) .

Original: Inserting t = log nN n d n and N = O (cid:16) (1 /(cid:15) )
Revised: Inserting t = log nN n d n and N = O (cid:16) (1 /(cid:15) )

Original: 1 n (cid:17) , we get N d
Revised: 1 n (cid:17) , we get N d

Original: = O (cid:16) λ log 1 λ − 1 (1 /(cid:15) )
Revised: = O (cid:16) λ log 1 λ − 1 (1 /(cid:15) )

Original: + log (1 /(cid:15) ) (cid:17) and
Revised: + log (1 /(cid:15) ) (cid:17) and

Original: N w = O (cid:16) λ log 1 λ − 1 +1 (1 /(cid:15) )
Revised: N w = O (cid:16) λ log 1 λ − 1 +1 (1 /(cid:15) )

Original: (1 /(cid:15) ) dn (cid:17) .
Revised: (1 /(cid:15) ) dn (cid:17) .

Original: Multiplying N w by log λ , N b is obtained.
Revised: Multiplying N w by log λ , N b is obtained.

Original: This concludes the proof of Theorem 1.
Revised: This concludes the proof of Theorem 1.

Original: A. T HE PROOF OF T HEOREM
Revised: A. T HE PROOF OF T HEOREM

Original: Theorem 2.
Revised: Theorem 2.

Original: For any f ∈ F d,n , given weight maximum precision 1 λ , there is a ReLU network with ﬁxed structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is
Revised: For any f ∈ F d,n , given weight maximum precision 1 λ , there is a ReLU network with ﬁxed structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is

Original: O (log (1 /(cid:15) )) ; (ii) the number of weights is O (cid:16)(cid:16) log (1 /(cid:15) )
Revised: O (log (1 /(cid:15) )) ; (ii) the number of weights is O (cid:16)(cid:16) log (1 /(cid:15) )

Original: + log 2 (1 /(cid:15) ) log λ (cid:17) (1 /(cid:15) )
Revised: + log 2 (1 /(cid:15) ) log λ (cid:17) (1 /(cid:15) )

Original: dn (cid:17) ; (iii) the number of bits needed to store the network is O (cid:16)(cid:0) log( λ ) log (1 /(cid:15) )
Revised: dn (cid:17) ; (iii) the number of bits needed to store the network is O (cid:16)(cid:0) log( λ ) log (1 /(cid:15) )

Original: + log 2 (1 /(cid:15) ) (cid:1) (1 /(cid:15) )
Revised: + log 2 (1 /(cid:15) ) (cid:1) (1 /(cid:15) )

Original: dn (cid:17) .
Revised: dn (cid:17) .

Original: With λ distinct values, a linearly quantized network has a minimal resolution of 1 λ .
Revised: With λ distinct values, a linearly quantized network has a minimal resolution of 1 λ .

Original: The proof for the approximability of linear quantization can be done in the same way as Theorem 1 except for a different sub-network for weight approximation.
Revised: The proof for the approximability of linear quantization can be done in the same way as Theorem 1 except for a different sub-network for weight approximation.

Original: We still construct W c in Proposition 3 ﬁrst and any weight value from W c can be obtained by multiply at most t log λ weights.
Revised: We still construct W c in Proposition 3 ﬁrst and any weight value from W c can be obtained by multiply at most t log λ weights.

Original: Thus the width and depth of the weight approximation network will be t and t log λ + 1 respectively.
Revised: Thus the width and depth of the weight approximation network will be t and t log λ + 1 respectively.

Original: Updating the B w in Table 1, we obtain the complexity accordingly.
Revised: Updating the B w in Table 1, we obtain the complexity accordingly.

Original: A. T HE PROOF OF P ROPOSITION
Revised: A. T HE PROOF OF P ROPOSITION

Original: Proposition 4.
Revised: Proposition 4.

Original: For any f ∈ F 1 , 1 , the (cid:101) f( x ) deﬁned above has the following properties.
Revised: For any f ∈ F 1 , 1 , t ∈ Z + , and T ∈ Z + , there exists a function (cid:101) f

Original:  (i) (cid:101) f
Revised: ( x ) such that (i) (cid:101) f

Original: ( x ) is a continuous function with Lipschitz constant 
Revised: ( x ) is a continuous function with Lipschitz constant 1; (ii) (cid:101)

Original: [Equation]
Revised: f ( iT ) = (cid:6) Tf (cid:0) iT (cid:1) / 2 − t (cid:7) 2 − t T ; (iii) | (cid:101) f ( x ) − f ( x ) | < 2 − t T .

Original: Proof.
Revised: Proof.

Original: We ﬁrst divide [0 , 1] into T intervals uniformly and deﬁne f
Revised: We ﬁrst divide [0 , 1] into T intervals uniformly and deﬁne f

Original: + ( x ) as
Revised: + ( x ) as

Original: [Equation]
Revised: [Equation]

Original: T

Original: [Equation] [Equation] where t ∈ N Note that f + ( iT ) =
Revised: Note that f + ( iT ) =

Original: (cid:6) Tf (cid:0) iT (cid:1) / 2 − t (cid:7) 2 − t T and d f + ( x ) d x = d f ( x ) d x on ( iT , i +1 T )
Revised: (cid:6) Tf (cid:0) iT (cid:1) / 2 − t (cid:7) 2 − t T and d f + ( x ) d x = d f ( x ) d x on ( iT , i +1 T )

Original: where , i = 1 , 2 , · · · , T .
Revised: where , i = 1 , 2 , · · · , T .

Original: Then, we deﬁne (cid:101) f ( x ) :
Revised: Then, we deﬁne (cid:101) f ( x ) :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  f + ( x ) is parallel with f ( x ) on any given interval ( iT , i +1 T ] and is shifted to let f + ( iT ) be an integral multiple of 2 − t T .
Revised: 24) f + ( x ) is parallel with f ( x ) on any given interval ( iT , i +1 T ] and is shifted to let f + ( iT ) be an integral multiple of 2 − t T .

Original: (cid:101) f
Revised: (cid:101) f

Original: ( iT ) is f ( iT ) rounded up to an integral multiple of 2 − t T .
Revised: ( iT ) is f ( iT ) rounded up to an integral multiple of 2 − t T .

Original: Meanwhile, (cid:101) f
Revised: Meanwhile, (cid:101) f

Original: ( x ) approaches f ( x ) with ﬁxed slope whenever they are not equal.
Revised: ( x ) approaches f ( x ) with ﬁxed slope whenever they are not equal.

Original: An example is shown in Figure 4.
Revised: An example is shown in Figure 4.

Original: Statement (i) follows the deﬁnition of (cid:101) f
Revised: Statement (i) follows the deﬁnition of (cid:101) f

Original: ( x ) directly.
Revised: ( x ) directly.

Original: We give a proof for statement (ii) by contradiction and induction.
Revised: We give a proof for statement (ii) by contradiction and induction.

Original: If (cid:101) f
Revised: If (cid:101) f

Original: ( iT ) =
Revised: ( iT ) =

Original: f
Revised: f

Original: + ( iT ) , then (cid:101) f
Revised: + ( iT ) , then (cid:101) f

Original: ( x ) =
Revised: ( x ) =

Original: f
Revised: f

Original: + ( x ) on ( iT , i +1 T ] .
Revised: + ( x ) on ( iT , i +1 T ] .

Original: The proof idea for (cid:101) f
Revised: The proof idea for (cid:101) f

Original: ( iT ) >
Revised: ( iT ) >

Original: f + ( iT ) and (cid:101) f
Revised: f + ( iT ) and (cid:101) f

Original: ( iT ) <
Revised: ( iT ) <

Original: f + ( iT ) are basically the same thus we present the proof of the latter for brevity.
Revised: f + ( iT ) are basically the same thus we present the proof of the latter for brevity.

Original: We ﬁrst assume (cid:101) f
Revised: We ﬁrst assume (cid:101) f

Original: ( i T ) =
Revised: ( i T ) =

Original: (cid:6)
Revised: (cid:6)

Original: Tf (cid:0) i T (cid:1) /
Revised: Tf (cid:0) i T (cid:1) /

Original: 2 − t (cid:7) 2 − t T , which is satisﬁed by Equation (23) when i = 0 .
Revised: 2 − t (cid:7) 2 − t T , which is satisﬁed by Equation (23) when i = 0 .

Original: Since (cid:101) f
Revised: Since (cid:101) f

Original: ( x ) =
Revised: ( x ) =

Original: f
Revised: f

Original: + ( x ) after the ﬁrst intersection on ( iT , i +1 T ] , if (cid:101) f
Revised: + ( x ) after the ﬁrst intersection on ( iT , i +1 T ] , if (cid:101) f

Original: ( x ) and f + ( x ) have no intersection on ( iT , i +1 T ] , then (cid:101) f
Revised: ( x ) and f + ( x ) have no intersection on ( iT , i +1 T ] , then (cid:101) f

Original: ( i +1 T ) <
Revised: ( i +1 T ) <

Original: f
Revised: f

Original: + ( i +1 T ) because (cid:101) f
Revised: + ( i +1 T ) because (cid:101) f

Original: ( iT ) <
Revised: ( iT ) <

Original: f + ( iT ) .
Revised: f + ( iT ) .

Original: Meanwhile, we have d (cid:101) f
Revised: Meanwhile, we have d (cid:101) f

Original: ( x ) d x = 1 , and(cid:101) f (cid:18) i + T (cid:19) = (cid:101)
Revised: ( x ) d x = 1 , and

Original: f (cid:18)

Original: [Equation]
Revised: [Equation]

Original: This contradicts the fact that (cid:101) f
Revised: This contradicts the fact that (cid:101) f

Original: ( i +1 T ) <
Revised: ( i +1 T ) <

Original: f + ( i +1 T ) .
Revised: f + ( i +1 T ) .

Original: Thus (cid:101) f
Revised: Thus (cid:101) f

Original: ( x ) and f + ( x ) intersect on (0 , iT ] and in turn guarantee that (cid:101) f
Revised: ( x ) and f + ( x ) intersect on (0 , i T ] and in turn guarantee that (cid:101) f

Original: ( i +1 T )
Revised: ( i +1 T )

Original: = (cid:6)
Revised: = (cid:6)

Original: Tf (cid:0) i +1 T (cid:1) / 2 − t (cid:7) 2 − t T .
Revised: Tf (cid:0) i +1 T (cid:1) / 2 − t (cid:7) 2 − t T .

Original: By induction, we prove that (cid:101) f
Revised: By induction, we prove that (cid:101) f

Original: ( x ) and f + ( x ) intersect on every interval ( iT , i +1 T ] .
Revised: ( x ) and f + ( x ) intersect on every interval ( iT , i +1 T ] .

Original: This implies statement (ii).
Revised: This implies statement (ii).

Original: Now we prove the statement (iii).
Revised: Now we prove the statement (iii).

Original: Note that we have 0 ≤ (cid:101)
Revised: Note that we have 0 ≤ (cid:101)

Original: f ( iT ) − f ( iT ) ≤ 2 − t T by statement (ii).
Revised: f ( iT ) − f ( iT ) ≤ 2 − t T by statement (ii).

Original: In every interval ( iT , i +1 T ) , (cid:101) f
Revised: In every interval ( iT , i +1 T ) , (cid:101) f

Original: ( x ) =
Revised: ( x ) =

Original: f
Revised: f

Original: + ( x ) after the their intersection.
Revised: + ( x ) after the their intersection.

Original: Therefore we have 0 ≤ | f + ( x ) − f ( x ) | ≤ 2 − t T by Equation (22).
Revised: Therefore we have 0 ≤ | f + ( x ) − f ( x ) | ≤ 2 − t T by Equation (22).

Original: Before the intersection, if (cid:101) f (0) <
Revised: Before the intersection, if (cid:101) f (0) <

Original: f
Revised: f

Original: + (0) , d( (cid:101) f
Revised: + (0) , d( (cid:101) f

Original: ( x ) − f + ( x ))
Revised: ( x ) − f + ( x ))

Original: d x ≥ 0 .
Revised: d x ≥ 0 .

Original: Since d f + ( x ) d x = d f ( x ) d x on ( iT , i +1 T ) , we have d( (cid:101) f
Revised: Since d f + ( x ) d x = d f ( x ) d x on ( iT , i +1 T ) , we have d( (cid:101) f

Original: ( x ) − f ( x ))
Revised: ( x ) − f ( x ))

Original: d x ≥ 0 , thus 0 ≤ (cid:101) f
Revised: d x ≥ 0 , thus 0 ≤ (cid:101) f

Original: ( iT ) − f ( iT ) ≤ (cid:101) f
Revised: ( iT ) − f ( iT ) ≤ (cid:101) f

Original: ( x ) − f ( x ) ≤ f + ( x ) − f ( x ) ≤ 2 − t T .
Revised: ( x ) − f ( x ) ≤ f + ( x ) − f ( x ) ≤ 2 − t T .

Original: If (cid:101) f ( iT ) ≥
Revised: If (cid:101) f ( iT ) ≥

Original: f
Revised: f

Original: + ( iT ) , apply the same logic and we obtain 0 ≤f + ( x ) − f ( x ) ≤ (cid:101)
Revised: + ( iT ) , apply the same logic and we obtain 0 ≤

Original: f ( x ) − f ( x ) ≤ (cid:101) f
Revised: [Equation] ≤ (cid:101) f

Original: ( iT ) − f ( iT ) ≤ 2 − t T .
Revised: ( iT ) − f ( iT ) ≤ 2 − t T .

Original: This implies statement (iii) and concludes the proof.
Revised: This implies statement (iii) and concludes the proof.

Original: A. T HE PROOF OF T HEOREM
Revised: A. T HE PROOF OF T HEOREM

Original: Theorem 3.
Revised: Theorem 3.

Original: For any f ∈ F 1 , 1 , given λ distinct weights, there is a ReLU network with function-dependent structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is O (cid:16) λ (log log (1 /(cid:15) ))
Revised: For any f ∈ F 1 , 1 , given λ distinct weights, there is a ReLU network with function-dependent structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is O (cid:16) λ (log log (1 /(cid:15) ))

Original: 1 λ − 1 + log (1 /(cid:15) ) (cid:17) ; (ii) the number of weights is
Revised: 1 λ − 1 + log (1 /(cid:15) ) (cid:17) ; (ii) the number of weights is

Original: O (cid:16) λ (log log (1 /(cid:15) )) 
Revised: O (cid:16) λ (log log (1 /(cid:15) )) 1 λ − +

Original: 1 λ − 1 +1 + (1 /(cid:15) ) (cid:17) (iii) the number of bits needed to store the network is
Revised: + (1 /(cid:15) ) (cid:17) (iii) the number of bits needed to store the network is

Original: O (cid:16) log λ (cid:16) λ (log log (1 /(cid:15) ))
Revised: O (cid:16) log λ (cid:16) λ (log log ( (cid:15) ))

Original: 1 λ − 1+1 + (1 /(cid:15) )
Revised: 1 λ − 1 + + ( /(cid:15) )

Original: (cid:17)(cid:17) .
Revised: (cid:17)(cid:17) .

Original: Proof.
Revised: Proof.

Original: We ﬁrst transform f to f (cid:48)(cid:48) with Proposition 4.
Revised: We ﬁrst transform f to f (cid:48)(cid:48) with Proposition 4.

Original: Then we apply the interpolation and cached function method from [35] while using the weight construction method described in Proposition 3.
Revised: Then we apply the interpolation and cached function method from [35] while using the weight construction method described in Proposition 3.

Original: Denoting the output of the network as f (cid:48)(cid:48) ( x ) , we have | f ( x ) − f (cid:48)(cid:48)
Revised: Denoting the output of the network as f (cid:48)(cid:48) ( x ) , we have | f ( x ) − f (cid:48)(cid:48)

Original: ( x ) | = | f
Revised: ( x ) | = | f

Original: ( x ) − (cid:101) f
Revised: ( x ) − (cid:101) f

Original: ( x ) | + | (cid:101) f
Revised: ( x ) | + | (cid:101) f

Original: ( x ) − f (cid:48)(cid:48)
Revised: ( x ) − f (cid:48)(cid:48)

Original: ( x ) | ≤ (cid:15) by choosing the hyper-parameters as m =  12 log (1 /(cid:15) )  , t =  log m  , δ = 18 m ,
Revised: ( x ) | ≤ (cid:15) by choosing the hyper-parameters as m = (cid:100) 12 log (1 /(cid:15) ) (cid:101) , t = (cid:100) log m (cid:101) , δ = 18 m ,

Original: T =  8 (cid:15) log(1 /(cid:15) )  .
Revised: T = (cid:100) 8 (cid:15) [Equation] (cid:15) ) (cid:101) .

Original: The approximation network is shown in Figure 5.
Revised: The approximation network is shown in Figure 5.

Original: The sizes of blocks are given in Table 2 where f T is the uniform linear interpolation function of f with T − 1 breakpoints, f ∗ is the sum of the selected cached functions, Φ( x ) is a ﬁltering function.
Revised: The sizes of blocks are given in Table 2 where f T is the uniform linear interpolation function of f with T − 1 breakpoints, f ∗ is the sum of the selected cached functions, Φ( x ) is a ﬁltering function.

Original: The inputs connections to B f ∗ 1 and the connections inside B m have higher order to the number of weights than others.
Revised: The inputs connections to B f ∗ 1 and the connections inside B m have higher order to the number of weights than others.

Original: Then the complexity can be obtained accordingly.
Revised: Then the complexity can be obtained accordingly.

Original: A. T HE PROOF OF T HEOREM
Revised: A. T HE PROOF OF T HEOREM

Original: Theorem 4.
Revised: Theorem 4.

Original: For any f ∈ F 1 , 1 , given weight maximum precision 1 λ , there is a ReLU network with function-dependent structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the depth is O (log (1 /(cid:15) )) ; (ii) the number of weights is O (1 /(cid:15) ) ; (iii) the number ofbits needed to store the network is O (log( λ ) /(cid:15) ) .
Revised: For any f ∈ F 1 , 1 , given weight maximum precision 1 λ , there is a ReLU network with function-dependent structure that can approximate f with any error (cid:15) ∈ (0 , 1) , such that (i) the For example, arrows between blocks can represent more than one connection in the network and there are short-cut connections that allow scaling only part of the inputs.

Original: [Table]
Revised: [Table]

Revised: depth is O (log (1 /(cid:15) )) ; (ii) the number of weights is O (1 /(cid:15) ) ; (iii) the number of bits needed to store the network is O (log( λ ) /(cid:15) ) .

Original: Proof.
Revised: Proof.

Original: The proof for the approximability of linear quantization can be done in the same way as
Revised: The proof for the approximability of linear quantization can be done in the same way as

Original: Theorem 3 except for a different sub-network for weight approximation.
Revised: Theorem 3 except for a different sub-network for weight approximation.

Original: We still construct W c in
Revised: We still construct W c in

Original: Proposition 3 ﬁrst and any weight value from W c can be obtained by multiply at most t log λ weights.
Revised: Proposition 3 ﬁrst and any weight value from W c can be obtained by multiply at most t log λ weights.

Original: Thus the width and depth of the weight approximation network will be t and t log λ + 1 respectively.
Revised: Thus the width and depth of the weight approximation network will be t and t log λ + 1 respectively.

Original: Updating the B δ and B m in Table [Table] we obtain the complexity accordingly [Table]
Revised: Updating the B δ and B m in Table 2, we obtain the complexity accordingly.

Original: B T HE EXISTENCE OF AN OPTIMAL BIT - WIDTH
Revised: B T HE EXISTENCE OF AN OPTIMAL BIT - WIDTH

Original: In this section [Table] we show that there exists one and only one local minimum on [2 , ∞ ) for M ( λ )  .
Revised: In this section, we prove the statement in Section 6 that there exists one and only one local minimum (hence global minimum) for M ( λ ) in the range of [2 , ∞ ) whenever (cid:15) < 12 .

Original: Denote log (cid:0) 3 n 2 d /(cid:15) (cid:1) as θ 2 and we get the derivative of M ( λ ) as:
Revised: Denote log (cid:0) 3 n 2 d /(cid:15) (cid:1) as θ 2 and we get the derivative of M ( λ ) as:

Original: [Equation]
Revised: [Equation]

Original: Let M s ( λ ) =
Revised: Let M s ( λ ) =

Original: log( λ ) + 1 ln2 − ln
Revised: log( λ ) + 1 ln2 − ln

Original: ( θ 2 ) λ log( λ ) ( λ − 1) 2 .
Revised: ( θ 2 ) λ log( λ ) ( λ − 1) 2 .

Original: Since θ λλ − 1> 0 , we have sgn( M s ) = sgn (cid:18) dM dλ (cid:19) .
Revised: Since θ λλ − 1> 0 , we have sgn( M s ) = sgn (cid:18) dM dλ (cid:19) .

Original: We have dM sdλ = λ
Revised: We have dM s

Original: + 1 − λ + log( λ ) + λ log( λ ) ( λ − 1)> 0 , ∀ λ ≥(27)
Revised: [Equation]

Original: and M s (2) = 1 + 1 ln2 − 2 ln( θ 2 ) < 0  and lim λ →∞ M s ( λ ) = ∞ .
Revised: M s (2) = 1+ 1 ln2 − 2 ln( θ 2 ) < 0 given (cid:15) < 12 , and lim λ →∞ M s ( λ ) = ∞ .

Original: It is clear that there exist a λ opt  such that sgn( M s ( λ )) < 0 on [2 , λ opt ) and sgn( M s ( λ )) > 0 on ( λ opt , ∞ ) .
Revised: It is clear that there exist a λ opt determined by θ 2 such that sgn( M s ( λ )) = − 1 on [2 , λ opt ) and sgn( M s ( λ )) = 1 on ( λ opt , ∞ ) .

Original: Remember that sgn( M s ) = sgn (cid:18) dM dλ (cid:19) , then λ
Revised: Remember that sgn( M s ) = sgn (cid:18) dM dλ (cid:19) , then λ

Original: = λ opt is the one and only one local minimum of M ( λ ) on [2 , ∞ ) .
Revised: = λ opt is the one and only one local minimum of M ( λ ) on [2 , ∞ ) .