VC DIMENSION OF PARTIALLY QUANTIZED NEURAL NETWORKS IN THE OVERPARAMETRIZED REGIME
A BSTRACT
Vapnik-Chervonenkis
VC
) theory has so far been unable to explain the small generalization error of overparametrized neural networks
Indeed, existing applications of
VC theory to large networks obtain upper bounds on VC dimension that are proportional to the number of weights and for a large class of networks, these upper bound are known to be tight.
In this work, we focus on a subclass of partially quantized networks that we refer to as hyperplane arrangement neural networks (HANNs).
Using a sample compression analysis we show that HANNs can have VC dimension signiﬁcantly smaller than the number of weights while being highly expressive.
In particular, empirical risk minimization over HANNs in the overparametrized regime achieves the minimax rate for classiﬁcation with
Lipschitz posterior class probability.
We further demonstrate the expressivity of
HANNs empirically.
On a panel of 121 UCI datasets, overparametrized HANNs match the performance of state-of-the-art full-precision models
I NTRODUCTION
Neural networks have become an indispensable tool for machine learning practitioners, owing to their impressive performance especially in vision and natural language processing (Goodfellow et al., 2016).
In practice, neural networks are often applied in the overparametrized regime and are capable of ﬁtting even random labels (Zhang et al., 2021).
Evidently, these overparametrized models perform well on real world data despite their ability to grossly overﬁt, a phenomenon that has been dubbed “the generalization puzzle” (Nagarajan & Kolter, 2019).
Toward solving this puzzle, several research directions have ﬂourished and offer potential explanations, including implicit regularization (Chizat & Bach, 2020), interpolation (Chatterji & Long, 2021), and benign overﬁtting (Bartlett et al., 2020).
So far, VC theory has not been able to explain the puzzle, because existing bounds on the VC dimensions of neural networks are on the order of the number of weights (Maass, 1994; Bartlett et al., 2019).
It remains unknown whether there exist neural network architectures capable of modeling rich set of classﬁers with low VC dimension.
The focus of this work is on a class of neural networks with threshold activation that we refer to as hyperplane arrangement neural networks (HANNs).
Using the theory of sample compression schemes (Littlestone & Warmuth, 1986), we show that HANNs can have VC dimension that is signiﬁcantly smaller than the number of parameters.
Furthermore, we apply this result to show that
HANNs have high expressivity by proving that HANN classiﬁers achieve minimax-optimality when the data has Lipschitz posterior class probability in an overparametrized setting.
We benchmark the empirical performance of HANNs on a panel of 121 UCI datasets, following several recent neural network and neural tangent kernel works (Klambauer et al., 2017; Wu et al., 2018; Arora et al., 2019; Shankar et al., 2020).
In particular, Klambauer et al.
(2017) showed that, using a properly chosen activation, overparametrized neural networks perform competitively compared to classical shallow methods on this panel of datasets.
Our experiments show that HANNs, a partially-quantized model, match the classiﬁcation accuracy of the self-normalizing neural network
(Klambauer et al., 2017) and the dendritic neural network (Wu et al., 2018), both of which are fullprecision models.
1. R ELATED WORK
VC dimensions of neural networks.
The VC dimension (Vapnik & Chervonenkis, 1971) is a combinatorial measure of the complexity of a concept class, i.e., a set of classiﬁers.
The Fundamental
Theorem of Statistical Learning (Shalev-Shwartz & Ben-David, 2014, Theorem 6.8) states that a concept class has ﬁnite VC-dimension if and only if it is probably approximately correct (PAC) learnable, where the VC-dimension is tightly related to the number of samples required for PAC learning.
For threshold networks, Cover (1965); Baum & Haussler (1989) showed a VC-dimension upper bounded of O ( w log w ) , where w is the number of parameters.
Maass (1994) obtained a matching lower bound attained by a network architecture with two hidden layers.
More recently, Bartlett et al.
(2019) obtained the upper and lower bounds O ( w(cid:96) log w ) and Ω( w(cid:96) log( w/(cid:96) )) respectively for the case when the activation is piecewise linear, where (cid:96) is the number of layers.
These lower bounds are achieved by somewhat unconventional network architectures.
The architectures we consider exclude these and thus we are able to achieve a smaller upper bound on the VC dimensions.
The generalization puzzle.
In practice, neural networks that achieve state-of-the-art performance use signiﬁcantly more parameters than samples, a phenomenon that cannot be explained by classical
VC theory if the VC dimension ≥ number of weights.
This has been dubbed the generalization puzzle (Zhang et al., 2021).
To explain the puzzle, researchers have pursued new directions including margin-based bounds (Neyshabur et al., 2017; Bartlett et al., 2017), PAC-Bayes bounds (Dziugaite & Roy, 2017), and implicit bias of optimization methods (Gunasekar et al., 2018; Chizat & Bach, 2020).
We refer the reader to the recent article by Bartlett et al.
(2021) for a comprehensive coverage of this growing literature.
The generalization puzzle is not speciﬁc to deep learning.
For instance, AdaBoost has been observed to continue to decrease the test error while the VC dimension grows linearly with the number of boosting rounds (Schapire, 2013).
Other learning algorithms that exhibits similarly surprising behavior include random forests (Wyner et al., 2017) and kernel methods (Belkin et al., 2018).
Minimax-optimality.
Whereas VC theory is distribution-independent, minimax theory is concerned with the question of optimal estimation/classiﬁcation under distributional assumptions 1 .
A minimax optimality result shows that the expected excess classiﬁcation error goes to zero at the fastest rate possible, as the sample size tend to inﬁnity.
For neural networks, this often involves a hyperparameter selection scheme in terms of the sample size.
Farag ´ o & Lugosi (1993) show minimax-optimality of (underparametrized) neural networks for learning to classify under certain assumptions on the Fourier transform of the data distribution.
Schmidt-Hieber (2020) shows minimax-optimality of s - sparse neural networks for regression over
H¨older classes, where at most s = O ( n log n ) network weights are nonzero, and n = the number of training samples.
Kim et al.
(2021) extends the results of Schmidt-Hieber (2020) to the classiﬁcation setting, remarking that effective optimization under sparsity constraint is lacking.
Kohler & Langer (2020) and Langer (2021) proved minimax-optimality without the sparsity assumption, however in an underparametrized setting.
To the best of our knowledge, our result is the ﬁrst to establish minimax optimality of overparametrized neural networks without a sparsity assumption.
(Partially) quantized neural networks.
Quantizing some of the weights and/or activations of neural networks has the potential to reduce the high computational burden of neural networks at test time (Qin et al., 2020).
Many works have focused on the efﬁcient training of quantized neural networks to close the performance gap with full-precision architectures (Hubara et al., 2017; Rastegari et al., 2016; Lin et al., 2017).
Several works have observed that quantization of the activations, rather than of the weights, leads to a larger accuracy gap (Cai et al., 2017; Mishra et al., 2018;
Kim et al., 2019).
Towards explaining this phenomenon, researchers have focused on understanding the so-called coarse gradient , a term coined by Yin et al.
(2019), often used in training QNNs as a surrogate for the usual gradient.
One commonly used heuristic is the straight-through-estimator (STE) ﬁrst introduced in an online course by Hinton et al.
Theory supporting the STE heuristic has recently been studied in Li et al.
(2017) and Yin et al.
QNNs have also been analyzed from other theoretical angles, including mean-ﬁeld theory (Blumenfeld et al., 2019), memory capacity (Vershynin, 2020), Boolean function representation capacity (Baldi & Vershynin, 2019) and adversarial robustness (Lin et al., 2018).
Of particular relevance to our work, Maass (1994) constructed an example of a QNN architecture with VC dimension on the order of the number of weights in the network.
In contrast, our work shows that there exist QNN architectures with much smaller VC dimensions.
Sample compression schemes.
Many concept classes with geometrically structured decision regions, such as axis-parallel rectangles, can be trained on a properly chosen size σ subset of an arbitrarily large training dataset without affecting the result.
Such a concept class is said to admit a sample compression schemes of size σ , a notion introduced by Littlestone & Warmuth (1986) who showed that the VC dimension of the class is upper bounded by O ( σ ) .
Furthermore, the authors posed the Sample Compression Conjecture .
See Moran & Yehudayoff (2016) for the best known partial result and an extensive review of research in this area.
Besides the conjecture, sample compression schemes have also been applied to other long-standing problems in learning theory (Hanneke et al., 2019; Bousquet et al., 2020;
Ashtiani et al., 2020).
To the best of our knowledge, our work is the ﬁrst to apply sample compression schemes to neural networks.
N OTATIONS
The set of real numbers is denoted R .
The unit interval is denoted [0 , 1] .
For an integer k ≥ 1 , let [ k ] = { 1 , . . .
, k } .
We use X to denote the feature space, which in this work will either be R d or [0 , 1] d where d ≥ 1 is the ambient dimension/number of features.
Denote by I { input } the indicator function which returns 1 if input is true and 0 otherwise.
The sign function is given by σ sgn ( t )
=
I { t ≥ 0 }− I { t < 0 } .
For vector inputs, σ sgn applies entry-wise.
The set of labels for binary classiﬁcation is denoted B := {± 1 } .
Joint distributions on X × B are denoted by P , where X, Y ∼ P denotes a random instance-label pair distributed according to P .
Let f : X → B be a binary classiﬁer.
The risk with respect to P is denoted by R P ( f ) :
= P ( f ( X ) (cid:54) = Y ) .
For an integer n ≥ 1 , the empirical risk is the random variable ˆ R P,n ( f ) :
= 1 n (cid:80) ni
=
I { f ( X i ) (cid:54) =
Y i } , where ( X 1 , Y 1 ) , . . .
, ( X n , Y n ) ∼ P are i.i.d.
The Bayes risk inf f : X→ B R P ( f ) with respect to P is denoted by R ∗ P .
Let f, g : { 1 , 2 , . . . }
→ R ≥ 0 be nonnegative functions on the natural numbers.
We write f (cid:16) g if there exists α, β > 0 such that for all n = 1 , 2 , . . .
we have αg ( n ) ≤ f ( n ) ≤ βg ( n ) .
H YPERPLANE ARRANGEMENT NEURAL NETWORKS
A hyperplane H in R d is speciﬁed by its normal vector w ∈ R d and bias b ∈ R .
The mapping x (cid:55)→ σ sgn ( w (cid:62) x + b ) indicates the side of H that x lies on, and hence induces a partition of R d into two halfspaces.
A set of k ≥ 1 hyperplanes is referred to as a k - hyperplane arrangement , and speciﬁed by a matrix of normal vectors and a vector of offsets:
W = [ w 1 · · · w k ] ∈ R d × k and b = [ b 1 , . . .
, b k ] (cid:62) .
Let q W ,b ( x ) :
= σ sgn ( W (cid:62) x + b ) for all x ∈ R d .
The vector q W ,b ( x ) ∈ B k is called a sign vector and the set of all realizable sign vectors is denoted S W ,b := { q W ,b ( x ) : x ∈ R d } .
Each sign vector s ∈ S W ,b uniquely deﬁnes a set { x ∈ R d : q W ,b ( x ) = s } known as a cell of the hyperplane arrangement.
The set of all cells forms a partition of R d .
For an example, see Figure 1-left.
A classical result in the theory of hyperplane arrangement due to Buck (1943) gives the following tight upper bound on the number of distinct sign patterns/cells:
[Equation]
See Fukuda (2015) Theorem 10.1 for a simple proof.
A hyperplane arrangement classiﬁer assigns a binary label y ∈ B to a point x ∈ R d solely based on the sign vector q W ,b ( x ) .
H
H
H+++ -++ --+
+-+
++- +-- -+-
B 1 B 2 B
Y + + + + + + − + + − + − + − − + − + + − − + − + − − + − + - - - + + +
Deﬁnition 3.1.
Let B X be the set of all functions from X to B .
A concept class C over X is a subset of B X .
Fix r, k positive integers, r ≤ min { d, k } .
Let Bool k be the set of all Boolean functions B k
→ B .
The hyperplane arrangement classiﬁer class is the concept class, denoted HAC ( d, r, k ) , over R d deﬁned by
[Equation]
See Figure 2 for a graphical representation of HAC ( d, r, k ) .
When the set of Boolean functions is realized by a neural network, we refer to the resulting classiﬁer as a hyperplane arrangement neural network (HANN).
Remark 3.2 .
Consider a ﬁxed hyperplane arrangement W , b and Boolean function h ∈ Bool k .
When performing prediction with the classifer h ◦ q W ,b , the feature vector x is mapped to a sign vector to which h is applied.
Thus, we do not need to know how h behaves outside of S W ,b .
The restriction of h to S W ,b is a partially deﬁned Boolean function or a lookup table .
[Figure]
Latent
[Figure]
Boolean
[Figure]
h : B k
→ B
B 1 B 2 B 3 Y − − − − − − + + − + − + ...
+ + + −
Output
Y h
[Figure]
Linear
Threshold
Remark 3.3 .
The hidden layer of width r in Figure 2 allows the user to impose the restriction that the hyperplane arrangement classiﬁer depends only on r relevant features, which can be either learned or deﬁned by data preprocessing.
When r = d , no restriction is imposed.
In this case, the input layer is directly connected to the Boolean layer.
This is consistent with Deﬁnition 3.1 where the rank constraint rank( W ) ≤ r becomes trivial.
Our next goal is to upper bound the VC dimension of HAC ( d, r, k ) .
Deﬁnition 3.4 (VC-dimension) .
Let C
⊆
B X be a concept class over X .
A set S := { x 1 , . . .
, x n } ⊆
X is shattered by C if for all sequences ( y 1 , . . .
, y n ) ∈ B n , there exists f ∈ C such that f ( x i ) =
y i for all i ∈ [ n ] .
The VC-dimension of C is deﬁned as
VC ( C ) =
sup {| S | : S ⊆ X , S is shattered by C} .
The VC-dimension has many far-reaching consequences in learning theory and, in particular, classiﬁcation.
One of these consequences is a sufﬁcient (in fact also necessary) condition for uniform convergence in the sense of the following well-known theorem.
See Shalev-Shwartz & Ben-David (2014) Theorem 6.8.
Theorem 3.5.
Let C be a concept class over X .
There exists a constant C > 0 such that for all joint distributions P on X × B and all f ∈ C , we have | ˆ R P,n ( f ) − R P ( f ) | ≤ C (cid:112) ( VC ( C ) + log(1 /δ ))
/n with probability at least 1 − δ with respect to the draw of ( X 1 , Y 1 ) , . . .
, ( X n , Y n ) .
The above VC bound is useless in the overparametrized setting if VC ( C ) =
Θ( # of weights ) because # of weights > n and therefore the VC bound does not vanish.
We now present our main result: an upper bound on the VC dimension of HAC ( d, r, k ) .
Theorem 3.6.
Let d, r, k ≥ 1 be integers and HAC ( d, r, k ) be deﬁned as in Deﬁnition 3.1.
Then
[Equation]
In the next section, we will prove this result using a sample compression scheme.
Before proceeding, we comment on the signiﬁcance of the result.
Remark 3.7 .
Since (cid:0) k ≤ r (cid:1) =
O ( k r ) , we have VC ( HAC ( d, r, k ))
=
O ( k r + dk log k ) which only involves the input dimension d and the width of the ﬁrst two hidden layers r and k .
For constant d and r ≥ 2 , this reduces to VC ( HAC ( d, r, k ))
=
O ( k r ) .
In particular, the number of weights used by an architecture to implement the Boolean function h does not affect the VC dimension at all and can be even inﬁnitely wide such as in Neural Tangent Kernels (Jacot et al., 2018).
For instance, Mukherjee & Basu (2017) Lemma 2.1 states that a 1-hidden layer neural network with
ReLU activation can model any k -input Boolean function if the hidden layer has width ≥ 2 k .
Note that this network uses ≥ k 2 k weights, and k 2 k (cid:29) k
r for ﬁxed r and k large.
Baldi & Vershynin (2019) study implementation of Boolean functions using threshold networks.
A consequence of their Theorem 9.3 is that a 2-hidden layer network with widths ≥ c 2 k/ 2 / √ k can implement all k input Boolean functions, where c is a constant not depending on k .
This requires ≥ c 2 2 k /k weights which again is exponentially larger than k r .
Furthermore, this lower bound on the weights is also necessary as k → ∞ .
Remark 3.8 .
Using the memorization approach of Vershynin (2020); Rajput et al.
(2021), it is also possible to implement HAC ( d, r, k ) with far fewer weights than in Remark 3.7.
This can be done by applying Theorem 1 of Rajput et al.
(2021) to memorize the (cid:0) k ≤ r (cid:1) distinct sign vectors in R k .
A SAMPLE COMPRESSION SCHEME
In this section, we will construct a sample compression scheme for HAC ( d, r, k ) .
As alluded to in the
Related Work section, the size of a sample compression scheme upper bounds the VC-dimension of a concept class, which will be applied to prove Theorem 3.6.
We ﬁrst recall the deﬁnition of sample compression schemes with side information introduced in Littlestone & Warmuth (1986).
Deﬁnition 4.1.
Let C be a concept class.
A length n sequence { ( x i , y i ) ∈
X × B } i ∈ [ n ] is C - labelled if there exists f ∈ C such that f ( x i ) =
y i for all i ∈ [ n ] .
Denote by L C ( n ) the set of C -labelled sequences of length at most n .
Denote by L C ( ∞ ) the set of all C -labelled sequences of ﬁnite length.
The concept class C over X has an m -sample compression scheme with s -bits of side information if there exists a pair of maps ( ρ, κ ) where
[Equation]
such that for all C -labelled sequences S :
= { ( x i , y i ) } i ∈ [ n ] , we have ρ ( κ ( S ))
( x i ) =
y i for all i ∈ [ n ] .
The size of the sample compression scheme is size ( ρ, κ ) :
= m + s .
Intuitively, κ and ρ can be thought of as the compression and the reconstruction maps, respectively.
The compression map κ keeps m elements from the training set and s bits of additional information, which ρ uses to reconstruct a classiﬁer that correctly labels the uncompressed training set.
The main result of this section is:
Theorem 4.2.
HAC ( d, r, k ) has a sample compression scheme ( ρ, κ ) of size
[Equation]
We remark that both the hyperplane arrangement ( W , b ) and the Boolean function h contribute to the number of parameters/weights.
The rest of this section will work toward the proof of Theorem 4.2.
The following result states that a C -labelled sequence can be labelled by a hyperplane arrangement classiﬁer of a special form.
Proposition 4.3.
Let { ( x i , y i ) } i ∈
[ n ] be HAC ( d, r, k ) -labelled.
Then there exist V =
[ v 1 · · · v k ] ∈
R d × k , c ∈ R k and h ∈ Bool k
such that for all i ∈ [ n ] , we have 1)
y
i
= h ( σ sgn ( V (cid:62) x i + c )) , 2) rank( V ) ≤ r and 3) | v (cid:62) j x i + c j | ≥ 1 for all i ∈ [ n ] , j ∈ [ k ] .
The proof, given in Appendix A.1, is similar to showing the existence of a max-margin separating hyperplane for a linearly separable dataset.
Deﬁnition 4.4.
Let I be a ﬁnite set and let a i ∈ R n for each i ∈
I .
Let A =
{ a i } i ∈ I .
A conical combination of A is a linear combination (cid:80) i ∈
I
λ i a i where the weights λ i ∈ R ≥ 0 are nonnegative.
The conical hull of A , denoted coni ( A ) , is the set of all conical combinations of A , i.e., coni ( { a i }
i ∈ I ) :
= (cid:8)(cid:80) i ∈ I
λ i a i : λ i ∈ R ≥ 0 , ∀ i ∈ I (cid:9) .
The result below follows easily from the Carath ´ edory’s theorem for the conical hull (Lov ´ asz &
Plummer, 2009).
For the sake of completeness, we included the proof in Appendix A.2.
Proposition 4.5.
Let a 1 , . .
. , a m ∈ R n and b 1 , . . .
, b m ∈ R .
For each subset I ⊆
[ m ] , deﬁne
P I := { x ∈ R n : a (cid:62) i x ≤ b i ∀ i ∈ I } .
Suppose that P [ m ] is nonempty.
Then 1) min x ∈P I 12 (cid:107) x
(cid:107)has a unique minimizer, denoted by x ∗ I below, and 2) there exists a subset J ⊆
[ m ] such that | J | = min { m, n } and for all
I ⊆
[ m ] with J ⊆
I , we have x ∗ [ m ] =
x
∗ I .
Proof of Theorem 4.2.
Let ( x i , y i ) be HAC ( d, r, k ) -realizable, and V , c and h be as in Proposition 4.3.
For each i ∈ [ n ] , deﬁne the Boolean vectors s i := σ sgn ( V (cid:62) x i + c ) ∈ {± 1 } k and s ij = σ sgn ( v (cid:62) j x i + c j ) denote the j -th entry of s i .
Note that s ij ( v (cid:62) j x i + c j ) =
| v (cid:62) j x i + c j | ≥ 1 .
We ﬁrst outline the steps of the proof:
Using a subset of the samples { ( x i (cid:96) , y i (cid:96) ) :
(cid:96) ∈
[ d ( k +1)] } with additional k ( d +1)(1+ (cid:100) log 2 k (cid:101) ) bits of side information { ( s i (cid:96) j (cid:96) , j (cid:96) ) :
(cid:96) ∈
[ d ( k + 1)] } , we can reconstruct W , b
such that σ sgn ( W (cid:62) x
i + b )
=
s i for all i ∈ [ n ] .
Using an additional subset of samples { ( x ι (cid:96) , y ι (cid:96) ) : (cid:96) = 1 , . .
, (cid:0) k ≤ r (cid:1) } in conjunction with the W , b reconstructed in the previous step, we can ﬁnd
g
∈ Bool k
such
that
g
( s i )
=
h
( s i for all
i .
Now, consider the set
[Equation]
Note that P is a convex polyhedron in ( d + 1) k -dimensional space.
Let ( W , b ) be the minimum norm element of P .
Note that σ sgn ( W (cid:62) x
i + b ) =
σ sgn
( V (cid:62) x i + c ) =
s i by construction.
By Proposition 4.5, there exists a set of tuples
[Equation]
where
[Equation] [Equation]
such that W , b is also the minimum norm element of
[Equation]
To encode the deﬁning equations of P (cid:48) , we need to store samples { ( x i (cid:96) , y i (cid:96) ) } d ( k +1) (cid:96) = and side information { ( s i (cid:96)
[Equation]
[Equation]
Note that each s i (cid:96) j (cid:96) requires 1 bit while each j (cid:96) ∈
[ k ] requires (cid:100) log 2 k (cid:101) bits.
In total, encoding P (cid:48) requires storing d ( k + 1) samples and d ( k + 1)(1 + (cid:100) log 2 k (cid:101) ) of bits.
To reconstruct g ∈ Bool k that agrees with h on all the samples, it sufﬁces to know h when restricted to { s i } ni =1 .
Since { s i } ni =1 is a subset of S W ,b , we have by eq.
(1) that |{ s i } ni | ≤ (cid:0) k ≤ r (cid:1) .
Thus, { s i } ni =1 has at most (cid:0) k ≤ r (cid:1) unique elements.
Let (cid:110) s ι (cid:96) : (cid:96) = 1 , . . .
, (cid:0) k ≤ r (cid:1)(cid:111) be a set containing all such unique elements.
Thus, we store samples {
[Equation]
Using W , b as deﬁned above, we have s ι (cid:96) =
σ sgn ( W (cid:62) x
ι (cid:96) + b ) .
Now, simply choose g such that g ( s ι (cid:96) )
=
y ι (cid:96) for all (cid:96) = 1 , . . .
, (cid:0) k ≤ r (cid:1) .
To summarize, we formally deﬁne the compression and reconstruction functions ( κ, ρ ) .
Let κ take the full sample { ( x i , y i ) } ni =1 and output the subsample (and side information) in eq.
(2) and eq.
The reconstruction function ρ ﬁrst constructs W , b using eq.
Next, ρ constructs g using W , b and the samples of eq.
Now, the following result together with the sample compression scheme for HAC ( d, r, k ) we constructed imply Theorem 3.6 from the previous section.
Theorem 4.6 (Littlestone & Warmuth (1986)) .
If C has sample compression scheme ( ρ, κ ) , then
VC ( C ) ≤ 8 · size ( ρ, κ ) .
Remark 4.7 .
Note that the reconstruction function ρ is not permutation-invariant.
Furthermore, the overall sample compression scheme ρ, κ is not stable in the sense of Hanneke & Kontorovich (2021).
In general, sample compression schemes with permutation-invariant ρ (Floyd & Warmuth, 1995) and stable sample compression schemes (Hanneke & Kontorovich, 2021) enjoy tighter generalization bounds compared to ordinary sample compression schemes.
We leave as an open question whether
HAC ( d, r, k ) has such specialized compression schemes.
M INIMAX - OPTIMALITY FOR LEARNING L IPSCHITZ CLASS
In this section, we show that empirical risk minimization (ERM) with respect to the 0-1 loss over
HAC ( d, r, k ) , for properly chosen r and k , is minimax optimal for classiﬁcation where the posterior class probability function is L -Lipschitz, for ﬁxed L > 0 .
Furthermore, the choices for r and k is such that the associated HANN, the neural network realization of HAC ( d, r, k ) , is overparametrized for the Boolean function implementations discussed in Remark 3.7.
Below, let X ∈ [0 , 1] d and Y ∈ B be the random variables corresponding to a sample and label jointly distributed according to P .
Write η P ( x ) :
= P ( Y = 1 | X = x ) for the posterior class probability function.
Let Σ( L, [0 , 1] d ) denote the class of L -Lipschitz functions f : [0 , 1] d → R , i.e.,
[Equation]
The following minimax lower bound result 2 concerns classiﬁcation when η P is L -Lipschitz:
Theorem 5.1 (Audibert & Tsybakov (2007)) .
There exists a constant C > 0 such that
[Equation]
The inﬁmum above is taken over all possible learning algorithms ˜ f n , i.e., mappings from ( X ×
B ) n to Borel measurable functions X
→ B .
When ˆ f n is an empirical risk minimizer (ERM) over
HAC ( d, r, k ) where d = r for k = n 1 d +2 , this minimax rate is achieved.
Theorem 5.2.
Let d ≥ 1 be ﬁxed.
Let ˆ f n be an ERM over HAC ( d, d, k ) where k = k ( n ) (cid:16) n 1 d +1 .
Then there exists a constant C (cid:48) such that
[Equation]
Proof sketch (see Appendix A.3 for full proof).
We ﬁrst show that the histogram classiﬁer over the standard partition of [0 , 1] d into smaller cubes is an element of C :=
HAC ( d, d, k ) , thus reducing the problem to proving minimax-optimality of the histogram classiﬁer.
Previous work Gy¨orﬁ et al.
Theorem 4.3 established this for the histogram regressor .
The analogous result for the histogram classiﬁer , to the best of our knowledge, has not appeared in the literature and thus is included for completeness.
The neural network implementation of HAC ( d, d, k ) where k (cid:16) n 1 / ( d +2) in Theorem 5.2 can be overparametrized.
Using either the 1- or the 2-hidden layer neural network implementations of
Boolean functions as in Remark 3.7, the resulting HANN is overparametrized and has number of weights either ≥ k 2 k or ≥ c 2 2 k /k respectively.
Both k 2 k and c 2 2 k /k (cid:29) n exponentially while
VC ( HAC ( d, d, k ))
=
o ( n ) .
E MPIRICAL RESULTS
In this section, we discuss experimental results of using HANNs for classifying synthetic and real datasets.
Our implementation uses TensorFlow (Abadi et al., 2016) with the Larq (Geiger & Team, 2020) library for training neural networks with threshold activations.
Note that Theorem 5.2 holds for ERM with respect to the 0-1 loss over HANNs, which is intractable in practice.
Furthermore, our theory is for binary classiﬁcation, while some of the datasets in the experiments are multiclass.
Synthetic datasets.
We apply a HANN (model speciﬁcation shown in Figure 3-top left) to the
MOONS synthetic dataset with two classes with the hinge loss.
The heuristic for training networks with threshold activation can signiﬁcantly affect the performance (Kim et al., 2019).
We consider two of the most popular heuristics: the straight-through-estimator (SteSign) and the SwishSign, introduced by Hubara et al.
(2017) and Darabi et al.
(2019), respectively.
Below, we use SwishSign since it reliably leads to higher validation accuracy (Figure 3bottom left), consistent with the ﬁnding of Darabi et al.
[Figure]
The width of the hidden layer is 2 10 = 1024 .
Thus, by Mukherjee & Basu (2017) Lemma 2.1, the model can assign labels to the bolded cells arbitrarily without changing the training loss.
Nevertheless, the optimization appears to be biased toward a topologically simpler classiﬁer.
This behavior is consistently reproducible.
See Figure 7.
Real-world datasets.
Klambauer et al.
(2017) introduced self-normalizing neural networks (SNN) which were shown to outperform other neural networks on a panel of 121 UCI datasets.
Subsequently, Wu et al.
(2018) proposed the dendritic neural network architecture, which further improved classiﬁcation performance on this panel of datasets.
Following their works, we evaluate the performance of HANNs on the 121 UCI datasets.
A crucial hyperparameter for HANN is the number of hyperplanes k .
We ran the experiments with k ∈ { 15 , 100 } to test the impact on accuracy.
The Boolean function h is implemented as a 1-hidden layer residual network (He et al., 2016) of width 1000 .
The logistic loss is used.
We use the same train, validation, and test sets as in Klambauer et al.
The reported accuracies on the held-out test set are based on the model with the highest validation accuracy.
The models will be referred to as HANN15 and HANN100 , respectively.
The results are shown in Figure 4.
The accuracies of SNN and DENN are obtained from Table A1 in the supplemental materials of Wu et al.
Full details for the training and accuracy tables can be found in the appendix.
[Figure]
[Table]
[Figure]
p l a n e s i n H A
NN
[Figure]
The HANN15 model (top row of Figure 4) already achieves median accuracy within 1.5% of both
SNN and DENN.
With the larger HANN100 model (bottom row), the gap is reduced to zero.
The largest training set in this panel of datasets has size 77904 .
The HANN15 and HANN100 models use ≈ 10 4 and 10 5 weights, respectively.
By comparison, the average numbers of weights used by SNN and DENN are both ≥ 5 ∗ 10 5 .
Details on these estimates are included in Appendix C.
Thus, all three models considered here are overparametrized.
D ISCUSSION
We have introduced an architecture for which the VC theorem can be used to prove minimaxoptimality of ERM over HANNs in an overparametrized setting with Lipschitz posterior.
To our knowledge, this is the ﬁrst time VC theory has been used to analyze the performance of a neural network in the overparametrized regime.
Furthermore, the same architecture leads to state-of-the-art performance over a benchmark collection of unstructured datasets.
To the best of our knowledge, no existing theoretical bound for overparametrized NNs yields meaningful results.
Yet there is immense interest in understanding what aspects of deep NNs can explain their performance, even if the bounds aren’t yet small (Bartlett et al., 2017; Neyshabur et al., 2017;
Jiang et al., 2019).
Our work shows that the compressibility of the network, as reﬂected by the sample compression scheme, is a useful avenue, and one that has not previously been explored – ours is the ﬁrst work applying sample compression to NNs.
This seems likely to open the door to further analysis of quantized NNs.
A CKNOWLEDGEMENTS
The authors were supported in part by the National Science Foundation under awardsand 2008074, and by the Department of Defense, Defense Threat Reduction Agency under award
HDTRA1-20-2-0002.
R EPRODUCIBILITY S TATEMENT
All code for downloading and parsing the data, training the models, and generating plots in this manuscript are available at https://github.com/YutongWangUMich/HANN .
Complete proofs for all novel results are included in the main article or in an appendix
A P ROOFS A. P ROOF OF P ROPOSITION 4.
By deﬁnition, there exists h ∈ Bool k ,
W ∈ R d × k of rank at most r , and b ∈ R k such that y i
= h ( σ sgn ( W (cid:62) x
i + b )) .
Now, let j ∈ [ k ] be ﬁxed.
Since | w (cid:62) j x i + b j | ≥ 0 for all i ∈ [ n ] , there exists a small perturbation ˜ c
j of b j such that | w (cid:62) j x i + ˜ c j | > 0 for all i ∈ [ n ] .
Now, let λ j :=
min i ∈ [ n ] | w (cid:62) j x i + ˜ c
j | which is positive.
Deﬁne v j := w j /λ j and c j = ˜ c
j /λ j , we have | v (cid:62)
j x i + c
j | ≥ 1 for all i ∈ [ n ] , as desired.
Note that rank( V ) = rank( W ) .
(cid:3)
A. P ROOF OF P ROPOSITION 4.
If m ≤ n , then the result holds trivially by letting J = [ m ] .
Below, suppose that m > n .
Let g i ( x ) = a (cid:62) i x
− b i for each i ∈ [ m ] and f ( x ) = 12 (cid:107) x (cid:107) 22 .
Then ∇ f ( x ) = x and ∇ g
i ( x ) = a i .
By deﬁnition, x ∗ I is a minimizer of
[Equation]
∈ I, which is a convex optimization with strongly convex objective.
Thus, the minimizer x ∗ I is unique and furthermore is the unique element x of R n satisfying the KKT conditions:
[Equation]
nonnegative
[Equation]
Thus, x ∗ I can be equivalently characterized as the unique element of x ∈ R n satisfying x ∈ P I and − x ∈ coni ( { a i } i ∈ I ) .
In particular, x ∗ [ m ] ∈ P [ m ] and − x ∗ [ m ] ∈ coni ( { a i } i ∈ [ m ] ) .
By the Carath´edory’s theorem for the conical hull (Lov´asz & Plummer, 2009), there exists I ⊆
[ m ] such that | I | = n and − x ∗ [ m ] ∈ coni ( { a i } i ∈ I ) .
Thus, for any J ⊆ [ m ] such that I ⊆ J , we have − x ∗ [ m ] ∈ coni ( { a i } i ∈ J ) .
Furthermore, J ⊆ [ m ] implies P J ⊇ P [ m ] .
In particular, x ∗ [ m ] ∈ P J .
Putting it all together, we have x ∗ [ m ] ∈ P J and − x ∗ [ m ] ∈ coni ( { a i } i ∈ J ) .
By the uniqueness, we have x ∗ J = x ∗ [ m ] .
(cid:3)
A. P ROOF OF T HEOREM 5.
In this proof, the constant C does not depending on n , and may change from line to line.
We ﬁx a joint distribution P such that η P ∈ Σ( L, [0 , 1] d ) throughout the proof.
Thus, the notation for risks will omit the P in their subscript, e.g., we write ˆ R n ( f ) instead of ˆ R P,n ( f ) and R ∗ instead of
R ∗ P .
Below, let β > α > 0 be constants such that αdn 1 / ( d +2) ≤ k ≤ βdn 1 / ( d +2) .
Let ˜ k :
=
(cid:100) k/d (cid:101) .
Let R 1 , R 2 , . . .
, R ˜ k
d denote the hypercubes of side length (cid:96) = 1 / ˜
k forming a partition of [0 , 1] d .
For each i ∈ [˜ k d ] , let R − i := { x ∈ R
i : η P ( x ) < 1 / 2 } and R + i := { x ∈ R i : η P ( x ) ≥ 1 / 2 } .
Let ˜ f : [0 , 1] d → B be the classiﬁer such that
[Equation]
In other words, ˜ f classiﬁes all x ∈ R
i
as +1 if and only if P ( Y = 1 | X ∈ R i ) ≥ 1 / 2 .
This is commonly referred to as the histogram classiﬁer (Gy ¨ orﬁ et al., 2006).
It is easy to see that
[Equation]
For the remainder of this proof, we write “ (cid:80) i ” to mean “ (cid:80) i ∈
[˜ k d ] ”.
Thus,
R
[Equation]
=
[Equation]
, (cid:90)
R i η P
[Equation]
x [Equation] x ) (cid:27) . Next, we note that ˜ f ∈ HAC ( d, d, k ) .
To see this, let j ∈ [ d ] .
Take H j 1 , . .
, H j (˜ k − 1)
⊆
R d to be the hyperplanes perpendicular to the j -th coordinate where, for each (cid:96) ∈
[˜ k ] , H j(cid:96) intersects the j -th coordinate axis at (cid:96)/ ˜ k .
Consider the hyperplane arrangement consisting of all { H j(cid:96) } j ∈
[ d ] ,(cid:96) ∈
[˜ k − 1] and let { C 1 , C 2 , . . .
} be its cells.
Then { C 1 ∩ [0 , 1] d , C 2 ∩ [0 , 1] d , . . .
= {R 1 , . .
, R ˜ k d } is the partition of [0 , 1] d by 1 / ˜ k side length hypercubes.
See Figure 5.
Let W be the matrix of normal vectors and b be the vector of offsets representing this hyperplane arrangement, which requires d (˜ k − 1) =
d ( (cid:100) k/d (cid:101) − 1) ≤ d ( k/d ) =
k hyperplanes.
Since ˜ f is constant on R i , there exists a Boolean function
h ∈ Bool k
such that h ◦ q W ,b | [0 , 1] d = ˜ f .
From this, we conclude that ˜ f ∈ HAC ( d, d, k ) .
Thus ˆ R n ( ˆ f n ) − ˆ R n ( ˜ f ) ≤ 0 and so
[Equation]
[Figure]
We now bound Terms 1 and 2 using the uniform deviation bound.
From Theorem 3.6, we know that there exists a constant C independent of n such that
[Equation]
Thus, by Theorem 3.5 with δ = 1 / (2 n ) and a union bound, with probability at least 1 − 1 /n
[Equation]
for some C > 0 .
Next, we focus on Term 3.
Recall that
[Equation]
and that
[Equation]
Fix some i ∈
[ k d ] .
Our goal now is to bound the difference between the i -th summands in the above expressions for R ( ˜ f ) and R ∗ :
[Equation]
First, consider the case that
[Equation]
We claim that there must exist x 0 ∈ R i such that η P ( x 0 ) ≤ 1 / 2 .
Suppose η P ( x ) > 1 / 2 for all x ∈ R i .
Then η P ( x ) > 1 / 2 > 1 − η P ( x ) .
Since η P ( x ) is continuous, this would contradict eq.
Continue assuming eq.
(7), we further divide into two subcases: (1) η P ( x ) ≤ 1 / 2 for all x ∈ R i , and (2) there exists some x 1 ∈ R i such that η P ( x 1 ) > 1 / 2 .
Under subcase (1), min { η P ( x ) , 1 − η P ( x ) } =
η P ( x ) for all x ∈ R i in which case eq.
(6) = 0 .
Under subcase (2), since η P ( x 0 ) ≤ 1 / 2 < η P ( x 1 ) , we know by the intermediate value theorem that there must exist x (cid:
) ∈
R i such that η P ( x (cid:48) )
= 1 / 2 .
Now,
[Equation] [Equation]
∵ Deﬁnition of R ±
i
[Equation]
(cid:100)
[Equation] [Equation]
Thus, under assumption eq.
(7), we have proven that eq.
(6) ≤ 2 Ld 3 / 2 /k .
For the other assumption, i.e., the minimum in eq.
(7) is attained by (cid:82) R i 1 − η P ( x ) dP ( x ) , a completely analogous argument again shows that eq.
(6) ≤ 2 Ld 3 / 2 /k .
Putting it all together, we have
[Equation]
We have shown that, with probability at least 1 − 1 /n ,
[Equation]
k .
Using αdn 1 / ( d +2) ≤ k ≤ βdn 1 / ( d +2) , we have with probably at least 1 − 1 /n that
R ( ˆ f n ) − R ∗ ≤ C (cid:114) k
d + log( n ) n + 2 Ld 3 /k ≤ C (cid:114) ( βd ) d n d/ ( d +2)
+ log( n ) n + 2 Ld 3 /αdn 1 / ( d +2) ≤ C (cid:32)(cid:114) n d/ ( d +2) n + n − 1 / ( d +2) (cid:33) ∵ log( n ) =
o ( n 1 /d +2 ) =
C (cid:16)(cid:112) n − 2 / ( d +2)
+ n − 1 / ( d +2) (cid:17) ≤ Cn − 1 d
+2 .
Taking expectation, we have E
[ R ( ˆ f n )] − R ∗ ≤ (1 − 1 /n )
Cn − 1 d
+
+ 1 /n · 1 ≤ Cn − 1 d +2 .
(cid:3)
B T RAINING DETAILS
Data preprocessing.
The pooled training and validation data is centered and standardized using the StandardScaler function from sklearn.
The transformation is also applied to the test data, using the centers and scaling from the pooled training and validation data:
[Equation]
If the feature dimension and training sample size are both > 50 , then the data is dimension reduced to 50 principal component features: if min(X_train_valid.shape) > 50: pca =
PCA(n_components = 50).fit(X_train_valid)
[Equation]
test
[Equation]
Note that this is equivalent to freezing the weights between the Input and the Latent layer in Figure 2.
Validation and test accuracy.
Every 10 epochs, the validation accuracy during the past 10 epochs are averaged.
A smoothed validation accuracy is calculated as follows:
[Equation]
description:
[Equation]
The predicted test labels is based on the snapshot of the model at the highest smoothed validation accuracy, at the end once max epochs is reached.
Heuristic for coarse gradient of the threshold function.
We use the SwishSign from the Larq library (Geiger & Team, 2020).
[Equation] [Equation]
Dropout.
During training, dropout is applied to the Boolean output of the threshold function, i.e, the variables B 1 , B 2 , . . .
, B k in Figure 2.
This improves generalization by preventing the training accuracy from reaching 100% .
# from tensorflow.keras.layers import Dense, Dropout hyperplane_enc =
Dense(n_hyperplanes, activation = qtz)(inputs) hyperplane_enc = Dropout(dropout_rate)(hyperplane_enc)
Implementation of the Boolean function.
For the Boolean function h , we use a 1-hidden layer residual network (He et al., 2016) with 1000 hidden nodes: # from tensorflow.keras.layers import Dense, Add # output_dim = num_classes n_hidden = 1000 hidden = Dense(n_hidden, activation="relu")(hyperplane_enc) out_hidden = Dense(output_dim, activation = "linear")(hidden) out_skip = Dense(output_dim, activation = "linear")(hyperplane_enc) outputs =
Add()([out_skip,out_hidden])
Hyperparameters.
HANN15 is trained with a hyperparameter grid of size 3 where only the dropout rate is tuned.
The hyperparameters are summarized in Table 2.
The model with the highest smoothed validation accuracy is chosen.
The model HANN15 is trained with the following hyperparameters:
[Table]
For HANN100 , we only used 1 set of hyperparameters.
[Table]
C P ARAMETER COUNTS
The widest part of HANN15 and HANN100 models are the weights mapping from B k ( k = number of hyperplanes) to R 1000 (1000 = number of hidden layer of the boolean function) where k ∈ { 15 , 100 } .
Thus, the two HANN models use ≥ 15 × 1000 ≥ 10 4 and ≥ 100 × 1000 = 10weights, respectively.
The weight count estimates for the Self-normalized Neural Network (SNN) and Dendritic Neural
Network (DENN) use the formula (# layers − 1) × (# neurons per layer ) 2 .
For the Self-normalized Neural Network (SNN), average number of layers = 10.8, and the number of neurons per layers ≥ 256 , found on page 7 and Table A4 of Klambauer et al.
(2017), respectively.
The number of weights is ≥ (10 − 1) ∗ (256 2 )
= 655 , 360 weights.
The parameters for the dendritic neural network (DENN) is found in the public GitHub repository xiangwenliu/DENN of Wu et al.
(2018) which lists number of layers = 3 and number of neurons per layer = 512, found on line 41 and 52 of train uci.py , respectively.
The number of weights is ≥ (3 − 1) ∗ (512 2 ) = 524 , 288 weights.
D A DDITIONAL PLOTS
Multiclass hinge versus cross-entropy loss.
Figure 6 shows the accuracy differences when the
Weston-Watkins hinge loss is used.
Compared to the results shown in Figure 4, the performance for
HANN100 is slightly worse and the performance for HANN15 is slightly better.
[Figure]
Implicit bias for low complexity decision boundary.
In Figure 7, we show additional results ran with the same setting for the MOONS synthetic dataset as in the Empirical Results section.
From the perspective of the training loss, the label assignment in the bold-boundary regions is irrelevant.
Nevertheless, the optimization consistently appears to be biased toward the geometrically simpler classiﬁer, despite the capacity for ﬁtting complex classiﬁers
[Figure]
E T ABLE OF ACCURACIES
Below is the table of accuracies used to make Figure 4.
The last column “HANN100trn” records the training accuracy at the epoch of the highest validation accuracy.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
cardiotocography-3clases 90.
chess
[Table]
chess
[Table]
congressional-voting 61.
conn-bench-sonar-mines-rocks 78.
conn-bench-vowel-deterding 89. connect78.
contrac 52.
credit-approval 81.
cylinder-bands 69.
dermatology 98.
echocardiogram 84.
ecoli 86.
energy-y 93.
energy-y 89.
fertility 92. ﬂags 39.
glass 77.
haberman-survival 72.
hayes-roth 71.
heart-cleveland 53.
heart-hungarian 72.
heart-switzerland 45.
heart-va 36.
hepatitis 82.
hill-valley 66.
horse-colic 80.
ilpd-indian-liver 70.
image-segmentation 87.
ionosphere 89.
iris 100.
led-display 73.
Continued on next page
DSName
HANN HANN
SNN
DENN
HANN100trn lenses 50.
letter 81.
libras 64.
low-res-spect 86.
lung-cancer 37.
lymphography 89.
magic 86.
mammographic 81.
miniboone 90.
molec-biol-promoter 73.
molec-biol-splice 79.
monks65. monks66. monks54.
mushroom 100. musk77.
musk97.
nursery 99.
oocytes-merluccius-nucleus-4d 86.
oocytes-merluccius-states-2f 92.
oocytes-trisopterus-nucleus-2f 81.
oocytes-trisopterus-states-5b 93.
optical 93.
ozone 96.
page-blocks 96.
parkinsons 87.
pendigits 94. pima 71.
pittsburg-bridges-MATERIAL 88.
pittsburg-bridges-REL-L 76.
pittsburg-bridges-SPAN 60.
pittsburg-bridges-T-OR-D 84.
pittsburg-bridges-TYPE 65.
planning 66.
plant-margin 50.
plant-shape 39.
plant-texture 51.
post-operative 40.
primary-tumor 54.
ringnorm 90.
seeds 92.
semeion 74.
soybean 77.
spambase 93.
spect 62.
spectf 91.
statlog-australian-credit 65.
statlog-german-credit 72.
statlog-heart 85.
statlog-image 95.
statlog-landsat 87.
statlog-shuttle 99.
statlog-vehicle 78.
steel-plates 73.
synthetic-control 94.
Continued on next page
[Table] [Table]