L EARNING B ACKPROPAGATION -F REE D EEP A RCHITECTURES WITH K ERNELS
Anonymous authors
Paper under double-blind review
A BSTRACT
One can substitute each neuron in any neural network with a kernel machine and obtain a counterpart powered by kernel machines.
The new network inherits the expressive power and architecture of the original but works in a more intuitive way since each node enjoys the simple interpretation as a hyperplane (in a reproducing kernel Hilbert space).
Further, using the kernel multilayer perceptron as an example, we prove that in classiﬁcation and under certain losses, an optimal representation that minimizes the risk of the network can be characterized for each hidden layer.
This result removes the need of backpropagation in learning the model and can be generalized to any feedforward kernel network.
Moreover, unlike backpropagation, which turns models into black boxes, the optimal hidden representation enjoys an intuitive geometric interpretation, making the dynamics of learning in a deep kernel network transparent.
Empirical results are provided to complement our theory.
I NTRODUCTION
Any neural network (NN) can be turned into a kernel network (KN) by replacing each artiﬁcial neuron (McCulloch & Pitts, 1943), i.e., learning machine of the form f ( x )
=
σ ( w (cid:62) x + b ) , with a kernel machine, i.e., learning machine of the form f ( x ) =
(cid:104) w, φ ( x ) (cid:105)
+ b with kernel function k
( x, y )
= (cid:104) φ ( x ) , φ ( y ) (cid:105) .
This combination of connectionism and kernel method enables the learning of hierarchical, distributed representations with kernels.
In terms of training, similar to NN, KN can be trained with backpropagation (BP) (Rumelhart et al., 1986).
In the context of supervised learning, the need for BP in learning a deep architecture is caused by the fact that there is no explicit target information to tune the hidden layers (Rumelhart et al., 1986).
Moreover, BP is usually computationally intensive and can suffer from vanishing gradient.
And most importantly, BP results in hidden representations that are notoriously difﬁcult to interpret or assess, turning deep architectures into “black boxes”.
The main theoretical contribution of this paper is the following:
Employing the simplest feedforward, fully-connected KN as an example, we prove that in classiﬁcation and under certain losses, the optimal representation for each hidden layer that minimizes the risk of the network can be explicitly characterized.
This result removes the need for BP and makes it possible to train the network in a feedforward, layer-wise fashion.
And the same idea can be generalized to other feedforward KNs.
The layer-wise learning algorithm gives the same optimality guarantee as BP in the sense that it minimizes the risk.
But the former is much faster and evidently less susceptible to vanishing gradient.
Moreover, the quality of learning in the hidden layers can be directly assessed during or after training, providing more information about the model to the user.
For practitioners, this enables completely new model selection paradigms.
For example, the bad performance of the network can now be traced to a certain layer, allowing the user to debug the layers individually.
Most importantly, the optimal representation for each hidden layer enjoys an intuitive geometric interpretation, making the learning dynamics in a deep KN more transparent than that in a deep NN.
A simple acceleration method that utilizes the “sparse” nature of the optimal hidden representations is proposed to further reduce computational complexity.
Empirical results on several computer vision benchmarks are provided to demonstrate the competence of the models and the effectiveness of the greedy learning algorithm.
[Figure]
layer-wise optimality drifting away from network-optimality. Consider a two-layer network and let T 1 , T 2 be the target function of the ﬁrst second layer, respectively
If the creates error illustrated far away
F ROM N EURAL TO K ERNEL N ETWORKS
In this section we discuss how to build a KN using a given NN.
First, the generic approach is described in Fig.
1a.
Note that KN inherits the expressive power of the original NN since a kernel machine is a universal function approximator under mild conditions (Park & Sandberg, 1991; Micchelli et al., 2006) and the two models share the same architecture.
However, KN works in a more intuitive way since each node is a simple linear model in a reproducing kernel Hilbert space (RKHS).
We now concretely deﬁne the KN equivalent of an l -layer Multilayer Perceptron (MLP), which we shall refer to as the kernel MLP (kMLP).
1 Given a random sample ( x n , y n ) Nn =1 , where ( x n , y n ) ∈
X 1 × Y ⊂ R d 0 × R , denote ( x n ) Nn =1 as S and ( y n ) Nn =1 as Y S for convenience.
For i = 1 , 2 , . . .
, l , consider kernel k ( i ) :
X i × X i → R , X i ⊂ R d i − 1 (for i > 1 , d i − 1 is determined by the width of the i − 1 th layer).
k ( i ) ( x, y ) =
(cid:10) φ ( i ) ( x ) , φ ( i ) ( y ) (cid:11)
H i , where φ ( i ) is a mapping into RKHS H i .
For i ≥ 1 , the i th layer in a kMLP, denoted F ( i ) , is an array of d i kernel machines: F ( i ) :
X i →
R d i , F ( i ) =
( f ( i ) 1 , f ( i ) 2 , . . .
, f ( i ) d i ) , a d i -tuple.
Let F (0) be the identity map on R d 0 , each f ( i ) j : X i →
R is a hyperplane in H i : f ( i ) j
( x ) = (cid:68) w ( i ) j , φ ( i ) (cid:0) F ( i − 1) ◦· · ·◦ F (0) ( x ) (cid:1)(cid:69) H i
+ b ( i ) j , w ( i )
j ∈ H
i , b ( i ) j ∈ R .
In practice, f ( i ) j
( x ) is usually implemented as (cid:80)
Nn =1 α
( i ) nj k
( i ) (cid:0) F ( i − 1) ◦ · · · ◦ F (0) ( x ) , F ( i − 1) ◦ · · · ◦
F (0) ( x n ) (cid:1) +
b ( i ) j , where the α ( i ) nj , b ( i ) j ∈ R are the learnable parameters.
The set of mappings { F ( (cid:96) ) ◦ · · · ◦ F (1) : α ( i ) nj , b ( i ) j ∈ R for all admissible n, j, i } deﬁnes an l -layer kMLP.
In the rest of this paper, we shall restrict our discussions to this kMLP.
A SSUMPTIONS AND N OTATIONS
We now specify the assumptions that we impose on all kernels considered in this paper.
First, we consider real, continuous, symmetric kernels only and we call a kernel positive semideﬁnite (PSD) or positive deﬁnite (PD) if for any S , the kernel matrix deﬁned as ( G ) mn
= k
( x m , x n ) is PSD or PD, respectively.
We shall always assume that any kernel considered is at least PSD and
that k ( i ) ( x, x ) =
c < + ∞ for all x ∈ X i and inf x,y ∈
X
i k
( i ) ( x, y )
= a > −∞ .
It is straightforward to check using Cauchy-Schwarz inequality that the ﬁrst condition implies max x,
y ∈
X
i k
( i ) ( x, y )
=
c .
For each ﬁxed x ∈ X i , we assume that k ( i ) ( x, y ) , as a function of y , is L ( i ) x -Lipschitz with respect to the Euclidean metric on X i .
Let sup x ∈ X i L ( i ) x
= L ( i ) , which we assume to be ﬁnite.
The following notations will be used whenever convenient: We use the shorthand F (1) ( S ) for ( F (1) ( x n ))
Nn =1 .
For i = 2 , 3 , . . .
, l , F ( i ) ( S ) :
=
F ( i ) ◦
F ( i − 1) ◦ · · · ◦ F (1) ( S ) .
Throughout this paper, notations such as F ( i ) can either be used to denote a set of functions or a speciﬁc function in some set depending on the context.
Also, when there is no confusion, we shall suppress the dependency of any loss function on the example for brevity, i.e., for a loss function (cid:96) , instead of writing (cid:96) ( f ( x ) , y ) , we shall write (cid:96) ( f ) .
A L AYER - WISE L EARNING A LGORITHM
To simplify discussion, we shall restrict ourselves to binary classiﬁcation ( Y = { +1 , − 1 } ) and directly give the result on classiﬁcation with more than two classes in the end.
A generalization to regression is left as future work.
Again, we only focus on kMLP although the idea can be directly generalized to all feedforward KNs.
We now discuss the layer-wise learning algorithm, beginning by addressing the difﬁculties with training a deep architecture layer-by-layer.
4. F UNDAMENTAL D IFFICULTIES
There are two fundamental difﬁculties with learning a deep architecture layer-wise.
First, the hidden layers do not have supervision (labels) to learn from.
And it depends on BP to propagate supervision from the output backward (Rumelhart et al., 1986).
We shall prove that for kMLP, one can characterize the optimal target representation for each hidden layer, which induces a risk for that layer.
The target is optimal in the sense that it minimizes the risk of the subsequent layer and eventually that of the network if all layers succeed in learning their optimal representations.
This optimal representation deﬁnes what we call “layer-wise optimality”.
The other difﬁculty with layer-wise learning is that for any given hidden layer, when the upstream layers create error, layer-wise optimality may not coincide with “network-wise optimality”, i.e., the solution of this layer that eventually leads to the composed solution that minimizes the risk of the network in this suboptimal case.
Indeed, when a hidden layer creates error, the objective of any layer after it becomes learning a solution that is a compromise between one that is close to the layer-wise optimality and one that prevents the error from the “bad” layer before it from “getting through” easily.
And the best compromise is the network-wise optimality.
The two solutions may not coincide, as shown in the toy example in Fig.
1b.
Clearly, we would like to always learn the network-wise optimality at each layer, but the learner is blind to it if it is only allowed to work on one layer at a time.
By decomposing the overall error of the network into error at each layer, we prove that in fact, network-wise optimality is learnable for each hidden layer even in a purely layer-wise fashion and that the proposed layer-wise algorithm learns network-wise optimality at each layer.
4. T HE O PTIMAL H IDDEN R EPRESENTATIONS
We now address the ﬁrst difﬁculty in layer-wise learning.
The basic idea is ﬁrst described in Section 4.2.1.
Then we provide technical results in Section 4.2.2 and Section 4.2.3 to ﬁll in the details.
B ASIC I DEA
Given a deep architecture F := F ( l ) ◦ · · · ◦ F (1) and a loss function (cid:96) l deﬁned for this network which induces a risk that we wish to minimize: R l = E (cid:96) l ( F ) .
BP views this problem in the following way:
R l is a function of F .
The learner tries to ﬁnd an F that minimizes R l using the random sample S with labels Y S according to some learning paradigm such as Empirical Risk Minimization (ERM) or Structural Risk Minimization (SRM) (Vapnik, 2000; Shalev-Shwartz & Ben-David, 2014).
S is considered as ﬁxed in the sense that it cannot be adjusted by the learner.
Alternatively, one can view R l as a function of F ( l ) and the learner tries to ﬁnd an F ( l ) minimizing
R l using random sample S l − 1 with labels Y S according to some learning paradigm, where S l − 1 :=
F ( l − 1) ◦· · ·◦ F (1) ( S ) 3 .
The advantage is that the learner has the freedom to learn both the function F ( l ) and the random sample S l − 1 .
And since S l − 1 determines the decision of the learning paradigm, which then determines R l , R l is now essentially a function of both F ( l ) and S l − 1 : R l = E (cid:96) l ( F ( l ) , S l − 1 ) .
The key result is that independently of the actual learning of F ( l ) , one can characterize the sufﬁcient condition on S l − 1 under which R l , as a function of S l − 1 , is minimized, as we shall prove.
In other words, the “global minimum” of R l w.r.t.
S l − 1 can be explicitly identiﬁed prior to any training.
This gives the optimal S l − 1 , which we denote as S (cid:63)l − 1 .
Moreover, the characterization of S (cid:63)l − 1 gives rise to a new loss function (cid:96) l − 1 and thus also a new risk R l − 1 that is a function of F ( l − 1) ◦ · · · ◦ F (1) .
Consequently, the same reasoning would allow us to deduce S (cid:63)l − 2 before the learner learns F ( l − 1) .
And this analysis can be applied to each layer, eventually leading to a greedy learning algorithm that sequentially learns F (1) , F (2) ◦ F (1) ∗ , . .
. , F ( l ) ◦
F ( l − 1) ∗ ◦· · ·◦ F (1) ∗ , in that order, where the asterisk on the superscript indicates that the corresponding layer has been learned and frozen.
The layer-wise learning algorithm provides a framework that enjoys great ﬂexibility.
To be speciﬁc, one could stop the above analysis at any layer i , then learn layers i + 1 , . . .
, l in a greedy fashion but still learn layers 1 , . . .
, i together with BP.
Thus, it is easy to see that BP can be brought under this framework as a special case.
Nevertheless, in later text, we shall stay on the one end of the spectrum where each layer is learned individually for clarity.
We now present the formal results that give the optimal hidden representations.
By the reasoning above, the analysis starts from the last hidden layer (layer l − 1 ) and proceeds backward.
4.2. F ORMAL R ESULTS : S (cid:63)l −
To begin with, we need to approximate the true classiﬁcation error R l since it is not computable.
To this end, we ﬁrst review a well-known complexity measure.
Deﬁnition 4.1 ( Gaussian complexity (Bartlett & Mendelson, 2002) ) .
Let P be a probability distribution on a metric space X and suppose x 1 , . . .
, x N are independent random elements distributed as
P .
Let F be a set of functions mapping from X into R .
Deﬁne
[Equation]
where g 1 , . .
. , g N are independent standard normal random variables.
The Gaussian complexity of
F is G N ( F ) =
E ˆ G N ( F )
Intuitively, Gaussian complexity quantiﬁes how well elements in a given function class can be correlated with a noise sequence of length N , i.e., the g n (Bartlett & Mendelson, 2002).
Based on this complexity measure, we have the following bound on the expected classiﬁcation error.
Theorem 4.2.
(Bartlett & Mendelson, 2002)
For each F mapping S into X l − 1 , let F l,A = (cid:110)
f : x (cid:55)→ (cid:68) w, φ ( l ) (cid:0) F ( x ) (cid:1)(cid:69) H l + b (cid:12)(cid:12)(cid:12) (cid:107) w (cid:107) H l ≤ A, b ∈ R (cid:111) .
Fix A, γ > 0 , with probability at least 1 − δ and for any N ∈ N , every function f ( l ) in F l,A satisﬁes
[Equation]
where ˆ R l ( f ( l ) )
= 1 N (cid:80) N n =1 max(0 , 1 − y n f ( l ) ( x n ) /γ ) , the empirical hinge loss.
Given the assumptions on k ( l ) , for any F , we have
[Equation]
Without loss of generality, we shall set hyperparameter γ = 1 .
We now characterize S (cid:63)l − 1 .
Note that for a given f ( l ) , A = (cid:13)(cid:13) w f ( l ) (cid:13)(cid:13)
H l is the smallest nonnegative real number such that f ( l ) ∈ F l,A
and it is immediate that this gives the tightest bound in Theorem 4.2.
Let κ = 1 N (cid:80)
Nn =1 1 { y n =+ } .
Lemma 4.3 ( optimal S l − 1 ) .
Given a learning paradigm minimizing ˆ R l ( f ( l ) )
+ τ (cid:13)(cid:13) w f ( l ) (cid:13)(cid:13)
H l using representation S l − 1 = F ( S ) , where τ is any positive constant satisfying τ < (cid:112) 2( c − a ) min( κ, 1 − κ ) .
Denote as S (cid:63)l − 1 any representation satisfying k ( l ) ( F ( x + ) , F ( x − ))
=
a and k ( l ) ( F ( x ) , F ( x (cid:48) ))
=
c (1) for all pairs of x + , x − from distinct classes in S and all pairs of x, x (cid:48) from the same class.
Suppose the learning paradigm returns f ( l ) (cid:63) under this representation.
Let S ◦ l − 1 be another representation under which the learning paradigm returns f ( l ) ◦ .
If f ( l ) ◦ achieves zero hinge loss on at least one example from each class, then for any N ∈ N , ˆ R l ( f ( l ) (cid:63) )
+ τ (cid:13)(cid:13) w f ( l ) (cid:63) (cid:13)(cid:13)
H l ≤ ˆ R l ( f ( l ) ◦ ) + τ (cid:13)(cid:13) w f ( l ) ◦
(cid:13)(cid:13)
H l .
The optimal representation S (cid:63)l − 1 , characterized by Eq. 1, enjoys a straightforward geometric interpretation: Examples from distinct classes are as distant as possible in the RKHS whereas examples from the same class are as concentrated as possible (see proof (C) of Lemma 4.3 for a rigorous justiﬁcation).
Intuitively, it is easy to see that such a representation is the “easiest” for the classiﬁer.
The conditions in Eq.
1 can be concisely summarized in an ideal kernel matrix G (cid:63) deﬁned as
[Equation]
And to have the l − 1 th layer learn S (cid:63)l − 1 , it sufﬁces to train it to minimize some dissimilarity measure between G (cid:63) and the kernel matrix computed from k ( l ) and F ( l − 1) ( S ) , which we denote G l − 1 .
Empirical alignment (Cristianini et al., 2002), L 1 and L 2 distances between matrices can all serve as the dissimilarity measure.
To simplify discussion, we let the dissimilarity measure be the L 1 distance
[Equation]
This speciﬁes
ˆ R l − 1 ( F ( l − 1) ) as the sample mean of ( (cid:96) l − 1 ( F ( l − 1) , ( x m , y m ) , ( x n , y n )))
Nm,n =1 and
R l − 1 as the expectation of (cid:96) l − 1 over ( X 1 , Y ) × ( X 1 , Y ) .
Note that due to the boundedness assumption on k ( l ) , (cid:96) l − 1 ≤ 2 max( | c | , | a | ) .
4.2. F ORMAL R ESULTS : S (cid:63)l − 2 , . . . , S (cid:63)
[Equation]
Similar to Section 4.2.2, we ﬁrst need to approximate R l − 1 .
Lemma 4.4.
For j = 1 , 2 , . . .
, d l − 1 , let f ( l − 1) j ∈ F l − 1 , where F l − 1 is a given hypothesis space.
There exists an absolute constant C > 0 such that for any N ∈ N , with probability at least 1 − δ ,
[Equation]
We are now in a position to characterize S (cid:63)l − 2 .
For the following lemma only, we further assume that k ( l − 1) ( x, y ) , as a function of ( x, y ) , depends only on and strictly decreases in (cid:107) x − y (cid:107) 2 for all x,
y ∈ X l − 1 with k ( l − 1) ( x, y ) > a , and that the inﬁmum inf x,
y ∈
X l − 1 k ( l − 1)
( x, y ) =
a is attained in
X l − 1 at all x, y with (cid:107) x − y (cid:107) 2 ≥ η .
Also assume that inf x,y ∈
X l − 1 ; (cid:107) x
− y (cid:107) 2 <η (cid:12)(cid:12) ∂k ( l − 1) ( x, y ) /∂
(cid:107) x − y (cid:107) 2 (cid:12)(cid:12) =
ι ( l − 1) is deﬁned and is positive.
Consider an F mapping S into X l − 1 , let F l − 1 ,A = (cid:110)
f : x (cid:55)→ (cid:68) w, φ ( l − 1) (cid:0) F ( x ) (cid:1)(cid:69)
H l − 1 + b (cid:12)(cid:12)(cid:12) (cid:107) w (cid:107) H l − 1 ≤ A, b ∈ R (cid:111) .
For a given F ( l − 1) = (cid:16) f ( l − 1), . . .
, f ( l − 1) d l − 1 (cid:17) , it is immediate that
A = max j (cid:13)(cid:13)(cid:13) w f ( l − 1) j
(cid:13)(cid:13)(cid:13)
H l − 1 is the smallest nonnegative real number such that f ( l − 1) j ∈ F l − 1 ,A for all j , giving the tightest bound in Lemma 4.4 (recall the bound on G N ( F l − 1 ,A ) in Theorem 4.2).
Let ψ = (cid:80) Nm,n =1
y m (cid:54) =
y n } /N 2 .
Lemma 4.5 ( optimal S l − 2 ) .
Given a learning paradigm minimizing
[Equation]
τ max j (cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13) H l − 1 using representation S l − 2 = F ( S ) , where τ is any positive constant satisfying τ < (cid:112) 2 d l − 1 ( c − a ) ψι ( l − 1) .
Denote as S (cid:63)l − 2 any representation satisfying k ( l − 1) ( F ( x + ) , F ( x − ))
=
a and k ( l − 1) ( F ( x ) , F ( x (cid:48) ))
=
c for all pairs of x + , x − from distinct classes in S and all pairs of x, x (cid:48) from the same class.
Suppose the learning paradigm returns F ( l − 1) (cid:63) =
(cid:16) f ( l − 1) (cid:63), . . .
, f ( l − 1) (cid:63) d l − 1 (cid:17) under this representation.
Let
S ◦ l − 2 be another representation under which the learning paradigm returns F ( l − 1) ◦ .
If F ( l − 1) ◦ achieves zero loss on at least one pair of examples from distinct classes, then for any N ∈ N , ˆ R
[Equation]
−
[Equation]
F
[Equation]
Applying this analysis to the rest of the hidden layers, it is evident that each of these layers should be trained to minimize the difference between G (cid:63) and its own kernel matrix.
Generalizing to classiﬁcation with more than two classes requires no change to the algorithm since the deﬁnition of G (cid:63) is agnostic to the number of classes involved in the classiﬁcation task.
Also note that the sufﬁciency of expanding the kernel machines of each layer on the training sample (see Section 2) for the learning objectives in Lemma 4.3 and Lemma 4.5 is trivially justiﬁable since the generalized representer theorem directly applies (Sch ¨ olkopf et al., 2001).
Now since the optimal representation is consistent across layers, the dynamics of layer-wise learning in a kMLP is clear: The network maps the random sample sequentially through layers, with each layer trying to map examples from distinct classes as far as possible in the RKHS while keeping examples from the same class in a cluster as concentrated as possible.
In other words, each layer learns a more separable representation of the sample.
Eventually, the output layer works as a classiﬁer on the ﬁnal representation and since the representation would be “simple” after the mappings of the lower layers, the learned decision boundary would generalize better to unseen data.
4. F INDING N ETWORK - WISE O PTIMALITY VIA L AYER - WISE L EARNING
We now discuss how to learn the network-wise optimality for each layer by working with only one layer at a time.
A rigorous description of the problem of layer-wise optimality drifting away from network-wise optimality and the search for a solution begins with the following bound on the total error of any two consecutive layers in a kMLP.
Lemma 4.6.
For any i = 2 , . . .
, l , let the target function and the approximation function be T i , F ( i ) ∗ :
X i → R d i , respectively.
Let (cid:15) i =
(cid:13)(cid:13) F ( i ) ∗ − T i (cid:13)(cid:13) s :
= sup x ∈ X
i (cid:13)(cid:13) F ( i ) ∗ ( x ) − T
i
( x ) (cid:13)(cid:13) 2 , we have (cid:13)(cid:13)(cid:13) F
[Equation]
i ) ∗
[Equation]
F
[Equation]
i − 1) ∗
[Equation]
1 (cid:13)(cid:13)(cid:13) s ≤ (cid:15)
[Equation]
+
[Equation]
(cid:15) i −(cid:118) (cid:117)(cid:117)(cid:116) 2 L ( i )
d i (cid:88)
j = (cid:13)(cid:13)(cid:13) w f ( i ) ∗ j (cid:13)(cid:13)(cid:13)
H i .
By applying the above bound sequentially from the input layer to the output, we can decompose the error of an arbitrary kMLP into the error of the layers.
This result gives a formal description of the problem: The hypothesis with the minimal norm minimizes the propagated error from upstream, but evidently, this hypothesis is not necessarily close to the layer-wise optimality T i .
Moreover, this bound also provides the insight needed for learning the network-wise optimality independently at each layer:
Searching for the network-wise optimality amounts to sequentially minimizing the r.h.s.
of Eq.
2 from the input layer forward.
And it is easy to see that the solution that minimizes ˆ R i ( F ( i ) )
+ τ max j (cid:13)(cid:13)(cid:13)
w f
( i ) j (cid:13)(cid:13)(cid:13)
H
i , where τ > 0 is a hyperparameter, provides a good approximate to the minimizer of the r.h.s.
of Eq.
2 if, of course, τ is chosen well.
This learning problem is solvable independently at each layer and the optimality of the optimal hidden representations are still guaranteed by Lemma 4.3 and Lemma 4.5.
Note that for BP, one usually also needs to heuristically tune the regularization coefﬁcient for weights as a hyperparameter.
4. A CCELERATING THE U PPER L AYERS
There is a natural method to accelerate the upper layers (all but the input layer): The optimal representation F ( S ) is sparse in the sense that φ ( F ( x m )) =
φ ( F ( x n )) if y m =
y n and φ ( F ( x m ))
(cid:54) =
φ ( F ( x n )) if y m (cid:54) =
y n (see the proof (C) of Lemma 4.3).
Since a kernel machine built on this representation of the given sample is a function in the RKHS that is contained in the span of the image of the sample, retaining only one example from each class would result in exactly the same hypothesis space because trivially, we have { (cid:80) Nn
=
α n φ ( F ( x n )) | α n ∈ R } =
{ α + φ ( F ( x + ))
+ α − φ ( F ( x − ))
| α + , α − ∈ R } for arbitrary x + , x − in S when the representation F ( S ) is optimal.
Thus, after training a given layer, depending on how close the actual kernel matrix is to the ideal one, one can (even randomly) discard a large portion of centers for kernel machines of the next layer to speed up the training of it without sacriﬁcing performance.
As we will later show in the experiments, randomly keeping a fraction of the training sample as centers for upper layers produces performance comparable to or better than that obtained with using the entire training set.
R ELATED W ORKS
The idea of combining connectionism with the classical kernel method was initiated by Cho & Saul (2009).
In their work, an “arc cosine” kernel was so deﬁned as to imitate the computations performed by a one-layer MLP.
Zhuang et al.
(2011) extended the idea to arbitrary kernels with a focus on MKL, using an architecture very similar to a two-layer MLP with neurons substituted by kernel machines.
As a further generalization, Zhang et al.
(2017) independently proposed kMLP and the KN equivalent of CNN.
However, they did not extend the idea to any arbitrary NN.
Scardapane et al.
(2017) proposed to reparameterize each nonlinearity in an NN with a kernel expansion, resulting in a network similar to KN but is trained with BP.
There are other works aiming at building “deep” kernels using approaches that are different in spirit from those above.
Wilson et al.
(2016) proposed to learn the covariance matrix of a Gaussian process using an NN in order to make the kernel “adaptive”.
This idea also underlies the now standard approach of combining a deep NN with an SVM for classiﬁcation, which was ﬁrst explored by Huang & LeCun (2006) and Tang (2013).
Note that such an interpretation can be given to KNs as well, as we point out in Appendix B.5.
Mairal et al.
(2014) proposed to learn hierarchical representations by learning mappings of kernels designed to be invariant to irrelevant variations in images.
Much works have been done to improve or substitute BP in learning a deep architecture.
Most aim at improving the classical method, working as add-ons for BP.
The most notable ones are perhaps the popular unsupervised greedy pre-training techniques proposed by Hinton et al.
(2006) and Bengio et al.
To the best of our knowledge, among the works that try to completely substitute
BP, none provides a comparable optimality guarantee in theory as that given by BP.
Fahlman &
Lebiere (1990) pioneered the idea of greedily learn the architecture of an NN.
In their work, each new node is added to maximize the correlation between its output and the residual error signal.
Several authors explored the idea of approximating the error signals propagated by BP locally at each layer or each node (Bengio, 2014; Carreira-Perpinan & Wang, 2014; Lee et al., 2015; Balduzzi et al., 2015;
Jaderberg et al., 2016).
Kulkarni & Karande (2017) proposed to train NN layer-wise using an ideal kernel matrix that is a special case of that in our work.
And no theoretical results were provided to justify its optimality for NN.
Zhou & Feng (2017) proposed a BP-free deep architecture based on decision trees, but the idea is very different from our work.
Raghu et al.
(2017) attempted to quantify the quality of hidden representations toward learning more interpretable deep architectures, sharing a motivation similar to ours.
E XPERIMENTS
We compared kMLP learned using the proposed greedy algorithm with other popular deep architectures including MLP, Deep Belief Network (DBN) (Hinton & Salakhutdinov, 2006) and Stacked
Autoencoder (SAE) (Vincent et al., 2010), with the last two trained using a combination of unsupervised greedy pre-training and standard BP (Hinton et al., 2006; Bengio et al., 2007).
Note that we only focused on comparing with the standard, generic architectures because kMLP, as the KN equivalent of MLP, does not have a specialized architecture or features designed for speciﬁc application domains. Several optimization and training techniques were applied to the MLPs to boost performance. These
[Table] [Table]
include Adam (Kingma & Ba, 2014), RMSProp (Tieleman & Hinton, 2012), dropout (Srivastava et al., 2014) and batch normalization (BN) (Ioffe & Szegedy, 2015). kMLP accelerated using the proposed method (kMLP FAST ) was also compared. For these models, we randomly retained a subset of the centers of each upper layer before its training.
As for the benchmarks used, rectangles, rectangles-image and convex are binary classiﬁcation datasets, mnist (10k) and mnist (10k) rotated are variants of MNIST (Larochelle et al., 2007; LeCun et al., 2010).
To further test the proposed layer-wise learning algorithm and the acceleration method, we compared greedily-trained kMLP with
MLP and kMLP trained using BP (Zhang et al., 2017). Two popular acceleration methods for kernel machines were also compared, including using a parametric representation (i.e., for each node in a kMLP, f ( x ) =
k ( w, x ) , w learnable) (kMLP PARAM ) and using random Fourier features (kMLP RFF ) (Rahimi & Recht, 2008).
These comparisons were done using the standard MNIST (LeCun et al., 2010).
More details for the experiments can be found in Appendix A.
From Table 1a, we see that the performance of kMLP is on par with some of the most popular and most mature deep architectures.
In particular, the greedily-trained kMLPs compared favorably with their direct NN equivalents, i.e., the MLPs, even though neither batch normalization nor dropout was used for the former.
These results also validate our earlier theoretical results on the layer-wise learning algorithm, showing that it indeed has the potential to be a substitute for BP.
Results in Table 1b further demonstrate the effectiveness of the greedy learning scheme.
For both the single-hidden-layer and the two-hidden-layer kMLPs, the layer-wise algorithm consistently outperformed BP.
It is worth noting that the proposed acceleration trick, despite being extremely simple, is clearly very effective and even produced models outperforming the original ones.
This shows that kMLP together with the greedy learning scheme can be of practical interest even when dealing with the massive data sets in today’s machine learning.
Last but not least, we argue that it is the practical aspects that makes the greedy learning framework promising.
Namely, this framework of learning makes deep architectures more transparent and intuitive, which can serve as a tentative step toward more interpretable, easy-to-understand models with strong expressive power.
Also, new design paradigms are now possible under the layer-wise framework.
For example, each layer can now be “debugged” individually.
Moreover, since learning becomes increasingly simple for the upper layers as the representations become more and more well-behaved, these layers are usually very easy to set up and also converge very fast during training.
A PPENDIX A E XPERIMENTAL S ETUP
The ﬁrst data set, known as rectangles , has 1000 training images, 200 validation images andtest images.
The learning machine is required to tell if a rectangle contained in an image has a larger width or length.
The location of the rectangle is random.
The border of the rectangle has pixel value 255 and pixels in the rest of an image all have value 0.
The second data set, rectangles-image , is the same with rectangles except that the inside and outside of the rectangle are replaced by an image patch, respectively.
rectangles-image has 10000 training images, 2000 validation images andtest images.
The third data set, convex , consists of images in which there are white regions (pixel value 255) on black (pixel value 0) background.
The learning machine needs to distinguish if the region is convex.
This data set has 6000 training images, 2000 validation images and 50000 test images.
The fourth data set contains 10000 training images, 2000 validation images andtest images taken from MNIST.
The ﬁfth is the same as the fourth except that the digits have been randomly rotated.
Sample images from the data sets are given in Fig.
For actual training and testing, the pixel values were normalized to [0, 1].
For detailed descriptions of the data sets, see (Larochelle et al., 2007).
The experimental setup for the greedily-trained kMLPs is as follows, kMLP-1 corresponds to a one-hidden-layer kMLP with the ﬁrst layer consisting of 15 to 150 kernel machines using the same
Gaussian kernel ( k ( x, y ) =
e −(cid:107) x − y (cid:107) 2 /σ 2 ) and the second layer being a single or ten (depending on the number of classes) kernel machines using another Gaussian kernel.
Note that the Gaussian kernel does not satisfy the condition that the inﬁmum a is attained (see the extra assumptions before
Lemma 4.5), but for practical purposes, it sufﬁces to set the corresponding entries of the ideal kernel matrix to some small value. For all of our experiments, we set ( G (cid:63) ) mn = 1 if y m =
y n andotherwise.
Hyperparameters were selected using the validation set.
The validation set was then used in ﬁnal training only for early-stopping based on validation error.
For the standard MNIST, the last 5000 training examples were held out as validation set.
For other datasets, see (Larochelle et al., 2007).
kMLP-1 FAST is the same kMLP for which we accelerated by randomly choosing a fraction of the training set as centers for the second layer after the ﬁrst had been trained.
The kMLP-2 and kMLP-2 FAST are the two-hidden-layer kMLPs, the second hidden layers of which contained 15 to 150 kernel machines.
We used Adam (Kingma & Ba, 2014) as the optimization algorithm of the layer-wise scheme.
Settings of all the kMLPs trained with BP can be found in (Zhang et al., 2017).
We compared kMLP with a one/two-hidden-layer MLP (MLP-1/MLP-2), a one/three-hidden-layer
DBN (DBN-1/DBN-3) and a three-hidden-layer SAE (SAE-3).
For these models, hyperparameters were also selected using the validation set.
For the MLPs, the sizes of the hidden layers were chosen from the interval [25, 700]; for DBN-3 and SAE-3, the sizes of the three hidden layers varied in intervals [500, 3000], [500, 4000] and [1000, 6000], respectively.
DBN-1 used a much larger hidden layer than DBN-3 to obtain comparable performance.
A simple calculation shows that the total numbers of parameters in the kMLPs were fewer than those in the corresponding DBNs and SAEs by orders of magnitude in all experiments.
Like in the training for the kMLPs, the validation set were also reserved for early-stopping in ﬁnal training.
The DBNs and SAEs had been pre-trained unsupervisedly before the supervised training phase, following the algorithms described in (Hinton et al., 2006; Bengio et al., 2007).
More detailed settings for these models were reported in (Larochelle et al., 2007).
A PPENDIX B F URTHER A NALYSIS
In this section, we provide some further analysis on kMLP and the layer-wise learning algorithm.
Namely, in Appendix B.1, we give a bound on the Gaussian complexity of an l -layer kMLP, which describes the intrinsic model complexity of kMLP.
In particular, the bound describes the relationship between the depth/width of the model and the complexity of its hypothesis space, providing useful information for model selection.
In Appendix B.2, we give a constructive result stating that the dissimilarity measure being optimized at each hidden layer will not increase as training proceeds from the input layer to the output.
This also implies that a deeper kMLP performs at least as well as its shallower counterparts in minimizing any loss function they are trained on.
In Appendix B.3, a result similar to Lemma 4.3 is provided, stating that the characterization for the optimal representation can be made much simpler if one uses a more restricted learning paradigm.
In fact, in contrast to
Lemma 4.3, both necessary and sufﬁcient conditions can be determined under the more restricted setting.
In Appendix B.4, we provide a standard, generic method to estimate the Lipschitz constant of a continuously differentiable kernel, as this quantity has been repeatedly used in many of our results in this paper.
In Appendix B.5, we state some advantages of kMLP over classical kernel machines.
In particular, empirical results are provided in Appendix B.5.1, in which a two-layer kMLP consistently outperforms the classical Support Vector Machine (SVM) (Cortes & Vapnik, 1995) as well as several SVMs enhanced by Multiple Kernel Learning (MKL) algorithms (Bach et al., 2004;
G ¨ onen & Alpaydın, 2011).
B. G AUSSIAN C OMPLEXITY OF K MLP
We ﬁrst give a result on the Gaussian complexity of a two-layer kMLP.
Lemma B.1.
Given kernel k : X 2 × X 2 → R , where X 2 ⊂ R d 1 .
Let F 2 = { f : X 2 → R , f ( x ) =
(cid:80)
[Equation]
[Equation] ν , x ) + b | α = ( α 1 , . . .
, α m ) ∈ R m , (cid:107) α (cid:107) 1 ≤ A, b ∈ R } , where the x ν are arbitrary examples from X 2 .
Consider F 1 = { ( f 1 , . .
, f d 1 ) :
X 1 → X 2 | f j ∈ Ω } , where Ω is a given hypothesis space that is closed under negation, i.e., if f ∈ Ω , then − f ∈ Ω .
Deﬁne F 2 ◦ F 1 = { h : x (cid:55)→ (cid:80) mν
=1 α ν k ( F ( x ν ) , F ( x )) + b | (cid:107) α (cid:107) 1 ≤ A, b ∈ R , F ∈ F 1 } .
If the range of some element in Ω contains 0 , we have
[Equation]
The above result can be easily generalized to kMLP with an arbitrary number of layers.
Lemma B.2.
Given an l -layer kMLP, for each f ( i ) j ( x ) =
(cid:80) mν
=1 α ( i ) νj k ( i ) ( F ( i − 1) ( x ν ) , F ( i − 1) ( x )) + b , α ( i ) j :=
( α ( i ) 1 j , . . .
, α ( i ) mj ) ∈ R m , b ∈ R , assume (cid:107) α
( i ) j (cid:107) 1 ≤
A i and let d l = 1 .
Denote the class of functions implemented by this kMLP as F l , we have
[Equation]
Proof.
It is trivial to check that the hypothesis space of each layer is closed under negation and that there exists a function in each of these hypothesis spaces whose range contains 0 .
Then the result follows from repeatedly applying Lemma B.1.
B. N ONINCREASING L OSS A CROSS L AYERS
Lemma B.3.
For i ≥ 2 , assume k
( i ) is PD and ﬁx layers 1 , 2 , . .
, i − 1 at arbitrary states
F (1) , F (2) , . .
, F ( i − 1) .
Let the loss functions (cid:96) i , (cid:96) i − 1 be the same up to their domains, and denote both (cid:96) i and (cid:96) i − 1 as (cid:96) .
Suppose layer i is trained with a gradient-based algorithm to minimize the loss (cid:96) ( F ( i ) ) .
Denote the state of layer i after training by F ( i ) ∗ .
If d i − 1 =
d i , there is an initialization for F ( i ) such that
[Equation]
Calculation for this initialization is speciﬁed in the proof.
For i = 1 , under the further assumption that X 1 ⊂ R d 0 and d 0 = d 1 , Eq.
becomes (cid:96) ( F (1) ∗ ) ≤ (cid:96) ( F (0) ) , where F (0) is the identity map on X 1 .
Remark B.3.1.
For the greedily-trained kMLP, Lemma B.3 applies to the hidden layers and implicitly requires that k ( i +1)
=
k ( i ) since the loss function for layer i , when viewed as a function of F ( i ) , takes the form (cid:96) ( k ( i +1) ( F ( i ) )) and can be rewritten as (cid:96) k
( i +1)
( F ( i ) ) .
Similarly, (cid:96) ( k ( i ) ( F ( i − 1) ))
is (cid:96) k
( i ) ( F ( i − 1) ) .
Since Lemma B.3 assumes (cid:96) to be the same across layers (otherwise it does not make sense to compare between layers), this forces (cid:96) k
( i +1) =
(cid:96) k
( i ) .
Then it follows that k ( i +1)
=
k ( i ) .
Further, if k ( i +1) and k ( i ) have the property that k ( x, y )
=
k (¯ x, ¯ y ) , where ¯ x, ¯ y denote the images of x, y under an embedding of R p into R q ( p ≤ q ) deﬁned by the identity map onto a p -dimensional subspace of R q , then the condition d i − 1 =
d i can be relaxed to d i − 1 ≤ d i .
This lemma states that for a given kMLP, when it has been trained upto and including the i th hidden layer, the i + 1 th hidden layer can be initialized in such a way that the value of its loss function will be lower than or equal to that of the i th hidden layer after training.
In particular, the actual hidden representation “converges” to the optimal represetation as training proceeds across layers.
On the other hand, when comparing two kMLPs, this result implies that the deeper kMLP will not perform worse in minimizing the loss function than its shallower counterpart.
In deep learning literature, results analogous to Lemma B.3 generally state that in the hypothesis space of a NN with more layers, there exists a hypothesis that approximates the target function nontrivially better than any hypothesis in that of another shallower network (Sun et al., 2016).
Such an existence result for kMLP can be easily deduced from the earlier bound on its Gaussian complexity (see Lemma B.2).
However, these proofs of existence do not guarantee that such a hypothesis can always be found through learning in practice, whereas Lemma B.3 is constructive in this regard.
Nevertheless, one should note that Lemma B.3 does not address the risk R = E (cid:96) .
Instead, it serves as a handy result that guarantees fast convergence of upper layers during training in practice.
B. S IMPLER O PTIMAL R EPRESENTATION UNDER M ORE R ESTRICTED L EARNING P ARADIGMS
The following lemma states that if we are willing to settle with a more restricted learning paradigm, the necessary and sufﬁcient condition that guarantees the optimality of a representation can be characterized and is simpler than that described in Lemma 4.3.
The setup for this lemma is the same as that of Lemma 4.3 except that the assumption that the numbers of examples from the two classes are equal is not needed.
Lemma B.4.
Consider a learning paradigm that minimizes ˆ R l ( f ( l ) )
+ τ G N ( F l,A ) using representation S l − 1 = F ( S ) under the constraint that it returns an f ( l ) (cid:63) ∈ F l,A with ˆ R l ( f ( l ) (cid:63) )
= 0 , where τ is any positive constant.
For any N ∈ N , ˆ R l ( f ( l ) (cid:63) )
+ τ G N ( F l,A ) is minimized over all linearly separable representations if and only if the representation F ( S ) satisﬁes [Equation] ) , F
[Equation]
x [Equation] = a for all pairs of x + , x − from distinct classes in S .
B. L IPSCHITZ C ONSTANT FOR C ONTINUOUSLY D IFFERENTIABLE K ERNELS
In general, for a continuously differentiable function f : R → R with derivative f (cid:48) and any a, b ∈ R , a < b , we have
[Equation]
This simple result can be used to bound the Lipschitz constant of a continuously differentiable kernel.
For example, for Gaussian kernel k : X × X
→ R , X ⊂ R , k
( x, y )
=
e − ( x − y ) 2 /σ 2 , we have ∂k ( x, y ) /∂y
= 2( x − y )
k
( x, y )
/σ 2 .
Hence for each ﬁxed x ∈ X , k ( x, y ) is Lipschitz in y with
Lipschitz constant bounded by sup y ∈ X (cid:12)(cid:12)(cid:12) 2( x − y ) k
( x, y )
/σ 2 (cid:12)(cid:12)(cid:12) .
In practice, X is always compact and can be a rather small subspace of some Euclidean space after normalization of data, hence this would provide a reasonable approximation to the Lipschitz constant of Gaussian kernel.
B. C OMPARING K MLP WITH C LASSICAL K ERNEL M ACHINES
There are mainly two issues with classical kernel machines besides their usually high computational complexity.
First, despite the fact that under mild conditions, they are capable of universal function approximation (Park & Sandberg, 1991; Micchelli et al., 2006) and that they enjoy a very solid mathematical foundation (Aronszajn, 1950), kernel machines are unable to learn multiple levels of distributed representations (Bengio et al., 2013), yet learning representations of this nature is considered to be crucial for complicated artiﬁcial intelligence (AI) tasks such as computer vision, natural language processing, etc.
(Bengio, 2009; LeCun et al., 2015).
Second, in practice, performance of a kernel machine is usually highly dependent on the choice of kernel since it governs the quality of the accessible hypothesis space.
But few rules or good heuristics exist for this topic due to its extremely task-dependent nature.
Existing solutions such as MKL (Bach et al., 2004; G¨onen &
Alpaydın, 2011) view the task of learning an ideal kernel for the given problem to be separate from the problem itself, necessitating either designing an ad hoc kernel or ﬁtting an extra trainable model on a set of generic base kernels, complicating training.
kMLP learns distributed, hierarchical representations because it inherits the architecture of MLP.
To be speciﬁc, ﬁrst, we see easily that the hidden activation of each layer, i.e., F ( i ) ( x ) ⊂
R d i , is a distributed representation (Hinton, 1984; Bengio et al., 2013).
Indeed, just like in an MLP, each layer of a kMLP consists of an array of identical computing units (kernel machines) that can be activated independently.
Further, since each layer in a kMLP is built on top of the previous layer in exactly the same way as how the layers are composed in an MLP, the hidden representations are hierarchical (Bengio et al., 2013).
Second, kMLP naturally combines the problem of learning an ideal kernel for a given task and the problem of learning the parameters of its kernel machines to accomplish that task.
To be speciﬁc, kMLP performs nonparametric kernel learning alongside learning to perform the given task.
Indeed, for kMLP, to build the network one only needs generic kernels, but each layer F ( i ) can be viewed as a part of a kernel of the form k
( i +1)
( F ( i ) ( x ) , F ( i ) ( y )) .
The fact that each F ( i ) is learnable makes this kernel “adaptive”, mitigating to some extent any limitation of the ﬁxed generic kernel k ( i +1) .
The training of layer i makes this adaptive kernel optimal as a constituent part of layer i + 1 for the task the network was trained for.
And it is always a valid kernel if the generic kernel k ( i +1) is.
Note that this interpretation has been given in a different context by Huang & LeCun (2006) and Bengio et al.
, we include it here only for completeness.
B.5. E MPIRICAL R ESULTS
We now compare a single-hidden-layer kMLP using simple, generic kernels with SVMs enhanced by
MKL algorithms that used signiﬁcantly more kernels to demonstrate the ability of kMLP to automatically learn task-speciﬁc kernels out of standard ones.
The standard SVM and seven other SVMs enhanced by popular MKL methods were compared (Zhuang et al., 2011), including the classical convex MKL (Lanckriet et al., 2004) with kernels learned using the extended level method proposed in (Xu et al., 2009)
(MKL LEVEL ); MKL with L p norm regularization over kernel weights (Kloft et al., 2011) ( L p MKL), for which the cutting plane algorithm with second order Taylor approximation of
L p was adopted; Generalized MKL in (Varma & Babu, 2009) (GMKL), for which the target kernel class was the Hadamard product of single Gaussian kernel deﬁned on each dimension; Inﬁnite Kernel
Learning in (Gehler & Nowozin, 2008) (IKL) with MKL LEVEL as the embedded optimizer for kernel weights; 2-layer Multilayer Kernel Machine in (Cho & Saul, 2009) (MKM); 2-Layer MKL (2LMKL) and Inﬁnite 2-Layer MKL in (Zhuang et al., 2011) (2LMKL INF ).
Eleven binary classiﬁcation data sets that have been widely used in MKL literature were split evenly for training and test and were all normalized to zero mean and unit variance prior to training.runs with identical settings but random weight initializations were repeated for each model.
For each repetition, a new training-test split was selected randomly.
For kMLP, all results were achieved using a greedily-trained, one-hidden-layer model with the number of kernel machines ranging from 3 to 10 on the ﬁrst layer for different data sets.
The second layer was a single kernel machine.
All kernel machines within one layer used the same Gaussian kernel, and the two kernels on the two layers differed only in kernel width σ .
All hyperparameters were chosen via 5 -fold cross-validation.
As for the other models compared, for each data set, SVM used a Gaussian [Table]
kernel. For the MKL algorithms, the base kernels contained Gaussian kernels with 10 different widths on all features and on each single feature and polynomial kernels of degree 1 to 3 on all features and on each single feature.
For 2LMKL INF , one Gaussian kernel was added to the base kernels at each iteration.
Each base kernel matrix was normalized to unit trace.
For L p MKL, p was selected from { 2 , 3 , 4 } .
For MKM, the degree parameter was chosen from { 0 , 1 , 2 } .
All hyperparameters were selected via 5 -fold cross-validation.
From Table 2, kMLP compares favorably with other models, which validates our claim that kMLP learns its own kernels nonparametrically hence can work well even without excessive kernel parameterization.
Performance difference among models can be small for some data sets, which is expected since they are all rather small in size and not too challenging.
Nevertheless, it is worth noting that only 2 Gaussian kernels were used for kMLP, whereas all other models except for SVM used signiﬁcantly more kernels.
A PPENDIX C P ROOFS
Proof of Lemma 4.3 .
Throughout this proof we shall drop the layer index l for brevity.
Given that the representation satisﬁes Eq. 1, the idea is to ﬁrst collect enough information about the returned f (cid:63) =
( w (cid:63) , b (cid:63) )
such that we can compute ˆ R ( f (cid:63) )
+ τ (cid:107) w (cid:63) (cid:107) H and then show that for any other F (cid:48) ( S ) satisfying the condition in the lemma, suppose the learning paradigm returns f (cid:48)
= ( w (cid:48) , b (cid:48) )
∈ F A (cid:48) , then ˆ R ( f (cid:48) )
+ τ (cid:107) w (cid:48) (cid:107) H ≥
ˆ R ( f (cid:63) )
+ τ (cid:107) w (cid:63) (cid:107) H .
We now start the formal proof.
First, note that in the optimal representation, i.e., an F ( S ) such that Eq.
1 holds, it is easy to see that (cid:107) φ ( F ( x − ))
− φ ( F ( x + ))
(cid:107) H is maximized over all representations for all x − , x + .
Moreover, note that given the representation is optimal, we have φ ( F ( x ))
=
φ ( F ( x (cid:48) ))
if y = y (cid:48) and φ ( F ( x )) (cid:54) =
φ ( F ( x (cid:48) ))
if y (cid:54) =
y (cid:48) :
Indeed, by Cauchy-Schwarz inequality, for all x, x (cid:48) ∈ S , k ( F ( x ) , F ( x (cid:48) )) = (cid:104) φ ( F ( x )) , φ ( F ( x (cid:48) ))
(cid:105) H ≤ (cid:107) φ ( F ( x ))
(cid:107) H (cid:107) φ ( F ( x (cid:48) ))
(cid:107) H and the equality holds if and only if φ ( F ( x ))
=
pφ ( F ( x (cid:48) )) for some real constant p .
Using the assumption on k , namely, that (cid:107) φ ( F ( x )) (cid:107) H = √ c for all F ( x ) , we further conclude that the equality holds if and only if p = 1 .
And the second half of the claim follows simply from c > a .
Thus, all examples from the + and − class can be viewed as one vector φ ( F ( x + )) and φ ( F ( x − )) , respectively.
The returned hyperplane f (cid:63) cannot pass both F ( x + ) and F ( x − ) , i.e., f (cid:63) ( F ( x + ))
=
and f (cid:63) ( F ( x − ))
= 0 cannot happen simultaneously since if so, ﬁrst subtract b (cid:63) , rotate while keeping (cid:107) w (cid:63) (cid:107) H unchanged and add some suitable b (cid:48) to get a new f (cid:48) such that f (cid:48)
( F ( x − ))
< 0 and f (cid:48)
( F ( x + )) > 0 , then it is easy to see that
ˆ R ( f (cid:48) )
+ τ (cid:107) w (cid:48)
(cid:107) H < ˆ R ( f (cid:63) )
+ τ (cid:107) w (cid:63) (cid:107) H .
But by construction of the learning paradigm, this is not possible.
Now suppose the learning paradigm returns an f (cid:63) such that
[Equation]
First note that for an arbitrary θ F,w (cid:63) , ζ is less than or equal to 2 since one can always adjust b (cid:63) such that y + f (cid:63) ( F ( x + )) =
y − f (cid:63) ( F ( x − )) without changing ζ and hence having a larger ζ will not further reduce ˆ R ( f (cid:63) ) , which is 0 when ζ = 2 , but will result in a larger (cid:107) w (cid:63) (cid:107) H according to
Eq. 4.
On the other hand, θ F,w (cid:63) must be 0 since this gives the largest ζ with the smallest (cid:107) w (cid:63) (cid:107) H .
Indeed, if the returned f (cid:63) does not satisfy θ F,w (cid:63) = 0 , one could always shift, rotate while keeping (cid:107) w (cid:63) (cid:107) H ﬁxed and then shift the hyperplane back to produce another f (cid:48) with θ F,w (cid:48)
=
and this f (cid:48) results in a larger ζ if ζ < 2 or the same ζ if ζ = 2 but a smaller (cid:107) w (cid:48)
(cid:107) H by rescaling.
Hence ˆ R ( f (cid:48) )
+ τ (cid:107) w (cid:48)
(cid:107) H < ˆ R ( f (cid:63) )
+ τ (cid:107) w (cid:63) (cid:107) H
but again, this is impossible.
Together with what we have shown earlier, we conclude that 2 ≥ ζ > 0 .
Then for some t ∈ R , we have
[Equation]
First note that we can choose t freely while keeping w (cid:63) ﬁxed by changing b (cid:63) .
If κ = 1 / 2 , we have ˆ R ( f (cid:63) ) =

[Equation]
Evidently, the last two cases both result in ˆ R ( f (cid:63) ) > 1 − ζ/ 2 hence f (cid:63) must produce a t in [ ζ − 1 , 1] and ˆ R ( f (cid:63) )
= 1 − ζ/ 2 .
Now, when κ (cid:54) = 1 / 2 , ﬁrst observe that
if 1 ≥ t ≥ ζ − 1 ,
[Equation]
If κ > 1 / 2 , ˆ R ( f (cid:63) ) decreases in t hence t must be 1 for f (cid:63) , which implies ˆ R ( f (cid:63) )
= (1 − κ )(2 − ζ ) .
Similarly, if κ < 1 / 2 , t = ζ − 1 and hence ˆ R ( f (cid:63) ) = κ (2 − ζ ) .
Now suppose t ≥ 1 , ˆ R ( f (cid:63) ) =
(1 − κ )(1 + t − ζ ) , which increases in t and hence t = 1 and ˆ R ( f (cid:63) ) =
(1 − κ )(2 − ζ ) .
If κ < 1 / 2 , since (1 − κ )(2 − ζ ) > κ (2 − ζ ) , this combination of κ and t contradicts the optimality assumption of f (cid:63) .
If t ≤ ζ − 1 , ˆ R ( f (cid:63) ) = κ (1 − t ) = κ (2 − ζ ) , where the second equality is because ˆ R ( f (cid:63) ) decreases in t .
Again, κ > 1 / 2 leads to a contradiction.
Combining all cases, we have
[Equation]
which, by the assumption on τ , strictly decreases in ζ over (0 , 2] .
Hence the returned f (cid:63) must satisfy ζ = 2 , which implies ˆ R ( f (cid:63) )
=
and we have
[Equation]
Now, for any other F (cid:48) ( S ) , suppose the learning paradigm returns f (cid:48) .
Let x w (cid:48)
+ , x w (cid:48) − be the pair of examples with the largest f (cid:48)
( F [Equation]
( x + ))
− f (cid:48) ( F (cid:48) ( x − )) .
We have y
[Equation]
Then
[Equation]
where we have used the assumption that there exists x − , x + with y −
[Equation]
x −
[Equation]
x
[Equation] ≥ [Equation]
This proves the desired result.
Lemma C.1.
Suppose f 1 ∈ F 1 , . . .
, f d ∈ F d are elements from sets of real-valued functions deﬁned on all of X 1 , X 2 , ..., X m , where X j ⊂ R d for all j , F ⊂ F 1 × · · · × F d .
For f ∈ F , deﬁne ω ◦ f :
X 1 × · · · × X m × Y → R as ( x 1 , . .
, x m , y ) (cid:55)→ ω ( f 1 ( x 1 ) , . . .
, f d ( x 1 ) , f 1 ( x 2 ) , . . .
, f d ( x m ) , y ) , where ω : R md × Y → R + ∪ { 0 } is bounded and L -Lipschitz for each y ∈ Y with respect to the
Euclidean metric on R md .
Let ω ◦ F = { ω ◦ f : f ∈ F} .
Denote the Gaussian complexity of F i on
X j as G jN ( F i ) , if the F i are closed under negation, i.e., for all i , if f ∈ F i , then − f ∈ F i , we have
[Equation]
In particular, for all j , if the x jn upon which the Gaussian complexities of the F i are evaluated are sets of i.i.d.
random elements with the same distribution, we have G 1 N ( F i ) = · · · = G mN ( F i ) :
= G N ( F i ) for all i and Eq.
5 becomes
[Equation]
This lemma is a generalization of a result on the Gaussian complexity of Lipschitz functions on R k from (Bartlett & Mendelson, 2002).
And the technique used in the following proof is also adapted from there.
Proof.
For the sake of brevity, we prove the case where m = 2 .
The general case uses exactly the same technique except that the notations would be more cumbersome.
Let F be indexed by A .
Without loss of generality, assume |A| < ∞ .
Deﬁne
[Equation]
where α ∈ A , the ( x n , x (cid:48) n ) are a sample of size N from X 1 × X 2 and g 1 , .
. . , g N , g 1 , 1 , . .
. , g 2 N,d are i.i.d.
standard normal random variables.
Let arbitrary α,
β ∈
A be given, deﬁne (cid:107) X
α − X β (cid:107)
= E ( X α − X β ) 2 , where the expectation is taken over the g n .
Deﬁne (cid:107) Y α − Y β (cid:107) 22 similarly and we have
[Equation]
By Slepian’s lemma (Pisier, 1999) and since the F i are closed under negation,
[Equation]
Taking the expectation of the x n and x (cid:48) n on both sides, we have
[Equation]
Proof of Lemma 4.4 .
Normalize (cid:96) l − 1 to [0 , 1] by dividing 2 max( | c | , | a | ) .
Then the loss function becomes
[Equation]
For each ﬁxed ( G (cid:63) ) mn ,
[Equation]
[Table]
[Equation] ≤ [Equation] Hence (cid:96) l − 1 is L ( l ) / max( | c | , | a | ) -Lipschitz in ( F ( l − 1) ( x m ) , F ( l − 1) ( x n )) with respect to the Euclidean metric on R 2 d l − 1 for each ( G (cid:63) ) mn .
The result follows from Lemma C.1 with m = 2 , d = d l − 1 and Corollary 15 in (Bartlett & Mendelson, 2002).
Proof of Lemma 4.5 .
This proof uses essentially the same idea as that of Lemma 4.3.
Due to the complete dependence of k ( l ) ( x, y ) on (cid:107) x − y (cid:107) 2 , we can rewrite k ( l ) (cid:16) F ( l − 1) ( F ( x m )) , F ( l − 1) ( F ( x n )) (cid:17) as h ( l ) (cid:16)(cid:13)(cid:13) F ( l − 1) ( F ( x m ))
− F ( l − 1) ( F ( x n ))
(cid:13)(cid:13) 2 (cid:17) for some h ( l ) .
Deﬁne µ ( l − 1) mn =
(cid:107) φ ( l − 1)
( F ( x m )) − φ ( l − 1) ( F ( x n ))
(cid:107) H l − 1 , we have
[Equation]
=
N
N (cid:88) m,n = (cid:12)(cid:12)(cid:12)(cid:12) h ( l ) (cid:16)(cid:13)(cid:13) F ( l [Table] 1) ( F ( x m ))
− F ( l
[Table]
1) ( F [Table] n )) (cid:13)(cid:13) 2 (cid:17) − ( G (cid:63) ) mn (cid:12)(cid:12)(cid:12)(cid:12) =
N
N (cid:88) m,n = (cid:12) (cid:12) (cid:12)(cid:12) h [Table] l ) (cid:16) (cid:118) (cid:117) (cid:117) (cid:116)
[Table]
l
[Table]
j
[Table]
(cid:0) f
[Table]
( F
[Table]
)) − f
[Table]
[Table]
( F
[Table]
[Table]
−
[Table]
mn [Table] = N
[Table]
N
[Table]
m,n
[Table]
(cid:12)(cid:12)(cid:12)(cid:12) h
[Table]
l ) (cid:16)(cid:118)(cid:117)(cid:117)(cid:116) d l
[Table]
1 (cid:88)
j = (cid:0) µ ( l − 1) mn (cid:107) w f
[Table]
− 1) j (cid:107) H l − 1 cos θ F,w f ( [Table] − 1) j (cid:1) 2 (cid:17) − ( G (cid:63) ) mn (cid:12)(cid:12)(cid:12)(cid:12) . Given that the representation is optimal, we have
[Equation]
The returned F ( l − 1) (cid:63) must satisfy
[Equation]
The ﬁrst observation is trivial since if otherwise, one can always reduce the largest (cid:107) w f ( l −
[Equation]
to obtain equality to η , this gives the same ˆ R l − 1 ( F ( l − 1) (cid:63) ) with a smaller τ max j (cid:107) w f ( l − 1) (cid:63) j (cid:107) H l − 1 .
Note that if during shrinking the largest (cid:107) w f ( l − 1) (cid:63) j (cid:107) H l − 1 , this element ceases to be the largest among all j , we shall continue the process with the new largest instead.
To see the rest of the claim, note that for the largest of the (cid:107) w f ( l − 1) (cid:63) j (cid:107) H l − 1 , we must have θ F,w f ( l − 1) (cid:63) j = 0 since if not, one could shift and rotate the hyperplane and again obtain
the same ˆ R l − 1 ( F ( l − 1) (cid:63) ) with a smaller τ max j (cid:107) w
[Equation]
Reducing the largest (cid:107) w f
[Equation]
[Equation]
(cid:107) H l
[Equation]
1 and increasing the second largest by scaling, one would get a smaller risk.
It is immediate that the minimal risk (w.r.t.
only the ﬁrst and second largest (cid:107) w f ( l − 1) (cid:63) j (cid:107) H l − 1 ) is attained when the largest and the second largest (cid:107) w f ( l − 1) (cid:63) j (cid:107) H l −are equal.
Then a similar argument as before gives θ F,w f ( l − 1) (cid:63) j = 0 for both of them.
The rest of the claim follows via repeatedly applying this argument to all the (cid:107) w f ( l − 1) (cid:63) j (cid:107)
[Equation]
Deﬁne
[Equation]
Then we have
[Equation]
As we have shown, (cid:112) d l − 1 | f (cid:63) ( F ( x + )) − f (cid:63) ( F ( x − ))
| ∈ [0 , η ] .
Let λ = | f
(cid:63) ( F ( x + ))
− f (cid:63) ( F ( x − ))
| and differentiate the r.h.s.
of the above equation w.r.t.
λ and using the assumption on τ , we have
[Equation]
Hence the overall risk decreases in λ over [0 , η/ (cid:112) d l − 1 ] , which implies that the returned F ( l − 1) (cid:63) must have λ =
η/ (cid:112) d l − 1 and
[Equation]
Now for any other representation S (cid:48)
l − 2 =
F (cid:48) ( S ) , suppose the learning paradigm returns F ( l − 1) (cid:48) .
Assume without loss of generality that the largest f ( l − 1) (cid:48)
j ( F (cid:48) ( x + )) − f ( l − 1) (cid:48)
j ( F (cid:48) ( x − )) over all j , x + and x − is attained at j = 1 and write w (cid:48) =
w f ( l − 1) (cid:48), f (cid:48)
=
f ( l − 1) (cid:48)for convenience.
Let x w (cid:48)
+ , x w (cid:48) − be the pair with the largest f (cid:48)
( F (cid:48)
( x + ))
− f (cid:48) ( F (cid:48) ( x − )) .
Note that the assumption on F (cid:48) implies that f (cid:48)
( F (cid:48)
( x + ))
− f (cid:48) ( F (cid:48) ( x − ))
≥ η/ (cid:112) d l − 1 .
Then we have
[Equation]
proving the lemma.
Proof of Lemma 4.6 .
First, it is trivial that the so-deﬁned s metric is indeed a metric.
In particular, it satisﬁes the triangle inequality.
For i = 2 , . . .
, l ,
[Equation]
≤ sup
[Equation]
= sup
[Equation]
i
[Equation]
≤
[Equation]
[Table]
which proves the lemma
[Table]
Proof of Lemma B.1 .
Since Ω and F 2 are both closed under negation, we have
[Equation]
Taking expectation with respect to the x n ﬁnishes the proof.
Proof of Lemma B.3 .
For i ≥ 1 , given the assumption that d i − 1 =
d i , we ﬁrst show that one can ﬁnd an initialization for layer i , denoted F ( i ) ◦ , such that F ( i ) ◦ ( S )
=
F ( i − 1) ( S ) .
Without loss of generality, assume i = 1 , then it amounts to showing that a set of parameters can be found for the kernel machines f (1)1 , . . .
, f (1) d 1 of F (1) such that when these kernel machines are initialized with this certain set of parameters, this layer agrees with the identity function when restricted to S .
Recall that f ( i ) j
( x ) = (cid:80) Nn
=1 α ( i ) jn k
( i ) ( x n , x ) and { α ( i )
jn ∈ R | n = 1 , 2 , . .
, N } is the set of trainable parameters for this kernel machine.
Solve equation AG =
D for A , where A is the d 0 × N matrix of parameters deﬁned as ( A ) nm = α (1) nm ,
G is the N × N kernel matrix deﬁned as ( G ) nm = k (1) ( x n , x m ) , D is a d 0 × N matrix whose n th column is the n th example x n = ( x n 1 , x n 2 , . . .
, x nd 0 ) (cid:62) for n = 1 , 2 , . .
N .
It is straightforward that the set of parameters A initializes F (1) to the desired F (1) ◦ .
Moreover, A can always be solved in closed-form by inverting G since k (1) being PD guarantees that G is invertible.
Combining the above result and the deﬁning property of gradient of the loss function with respect to the parameters, i.e., that gradient points at the direction of the greatest rate of increase of the loss function, we then have
[Equation]
Proof of Lemma B.4 .
In this proof we shall again drop the layer index l for brevity.
Identify f ∈ F A using w, b upto R -scalar multiplication.
Fix an F such that φ ( F ( S )) is linearly separable (otherwise the learning paradigm is not applicable).
Let x + , x − be the closest pair of examples from distinct classes in H .
For a given f = ( w, b ) , let x w − , x w + be a pair of examples from different classes that are closest to f in H in the sense that f ( F ( x w + )) , f ( F ( x w − )) are the smallest in absolute value among all x + , x − , respectively.
Denote the canonical form of f with respect to φ ( F ( S )) with equality attained by at least one of x w + , x w − as f F = ( w F , b F ) , i.e., f F satisﬁes yf F ( x ) ≥ 1 for all examples in ( x n , y n ) Nn =1 and for at least one of x w − , x w + , the equality is attained.
Deﬁne A f F :=
inf b F ∈ R (cid:107) w F (cid:107) H .
Suppose the algorithm returns f (cid:63)
=
( w (cid:63) , b (cid:63) ) and recall that A is the smallest such that f (cid:63) ∈ F A , then for all f = ( w, b ) ∈
F A such that ˆ R ( f ) = 0 ,
[Equation]
And it is easy to see that the last equality holds if and only if b F is such that y w − f F ( x w − ) =
y w + f F ( x w + ) .
Then f F ∈ F A .
Since ˆ R ( f F ) =
and we have the freedom to choose b F such that the last equality holds, by construction of the learning paradigm and the minimality of A , we have
[Equation]
In canonical form, for any f F ∈ F A , choose b F such that (cid:107) w F (cid:107) H =
A f F , then
[Equation]
H cos θ F,w .
By construction of the learning paradigm, f (cid:63) must make the equality hold and satisfy θ F,w (cid:63) = 0 .
Hence we have
A =
[Equation]
which proves that, as a function of F , A achieves its minimum if and only if F maximizes (cid:13)(cid:13)
φ ( F ( x + )) − φ
F ( x − ))
(cid:13)(cid:13) H .
Since
[Equation]
where we have used the assumption on k , namely, that k ( x, x ) = (cid:104) φ ( x ) , φ ( x ) (cid:105) H
= (cid:107) φ ( x ) (cid:107) 2 H
=
c , for all x .
It immediately follows that any minimizer F of A must minimize k ( F ( x + ) , F ( x − )) for all pairs of examples from opposite classes.
This proves the desired result.