M ODEL -E FFICIENT D EEP L EARNING WITH K ERNELIZED C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We investigate the possibility of using the embeddings produced by a lightweight network more effectively with a nonlinear classiﬁcation layer.
Although conventional deep networks use an abundance of nonlinearity for representation (embedding) learning, they almost universally use a linear classiﬁer on the learned embeddings.
This could be suboptimal for a network with a limited-capacity backbone since better nonlinear classiﬁers could exist in the same embedding vector space.
We advocate a nonlinear kernelized classiﬁcation layer for deep networks to tackle this problem.
We theoretically show that our classiﬁcation layer optimizes over all possible radial kernel functions on the space of embeddings to learn an optimal nonlinear classiﬁer.
We then demonstrate the usefulness of this layer in learning more model-efﬁcient classiﬁers in a number of computer vision and natural language processing tasks
I NTRODUCTION
A traditional classiﬁcation deep network consists of two parts: a representation learner that maps an input to a vector-valued representation called the embedding, and a classiﬁer that classiﬁes this embedding into the correct class.
For example, in the text classiﬁcation setting, the input text may be sent through a transformer encoder network with CLS-pooling (the representation learner) to obtain an embedding vector for the text (Devlin et al., 2018).
A fully-connected layer (the classiﬁer) is then operated on this embedding.
A classiﬁer learned in a usual fully connected layer with the softmax loss is linear in the space of embeddings.
Therefore, the representation learner has to learn linearly-separable embeddings to do well in the classiﬁcation task.
When the representation learner backbone is unable to learn linearly separable embeddings to satisfactorily solve a given classiﬁcation task, the usual ﬁx is to use a deeper and/or wider backbone to generate better embeddings (Turc et al., 2019;
He et al., 2016).
However, bigger backbones demand more resources in terms of compute, memory, and model storage during both training and inference.
Therefore, it is natural to ask whether it is instead possible to use embeddings produced by a capacity-limited network more effectively by using a more sophisticated classiﬁcation layer.
In this work we address this problem by proposing a nonlinear, kernelized classiﬁcation layer.
This classiﬁcation layer ﬁnds an optimal nonlinear classiﬁer for the embeddings by mapping them into a
Reproducing Kernel Hilbert Space (RKHS) that optimally separates them into different classes.
We borrow the key idea from kernel methods in the classic machine learning literature (Cortes & Vapnik, 1995; Sch¨olkopf & Smola, 2002): instead of running a classiﬁer directly on the embeddings, they are ﬁrst mapped to much higher dimensional vectors in an RKHS using a positive deﬁnite kernel function.
A linear classiﬁer is then run on this high-dimensional RKHS.
Since the dimensionality of the embeddings is dramatically increased via this mapping, a linear classiﬁer in the RKHS corresponds to a powerful nonlinear classiﬁer in the original embedding space.
Such a classiﬁer can therefore utilize even linearly inseparable embeddings to satisfactorily solve a classiﬁcation task as demonstrated in Figure 1.
A common issue with traditional kernel methods is the choice of the kernel function used to obtain the RKHS mapping.
While a collection of well-known kernels such as the linear kernel, polynomial kernels, and the Gaussian RBF kernel is available, it is often unclear which kernel would work best for a given problem.
We tackle this issue by sweeping over all possible kernel functions within the deep network itself to ﬁnd the optimal one via backpropagation and stochastic gradient descent.
[Figure]
There is a wealth of literature on making deep learning models more efﬁcient by using techniques such as pruning, quantization, low-rank factorization, and distillation (Cheng et al., 2017; Deng et al., 2020).
We approach the model efﬁciency from an orthogonal angle by asking whether it is possible to utilize embeddings produced by a given representation learner more effectively by doing nonlinear classiﬁcation.
Therefore, our approach is complementary to these existing model compression techniques.
In summary, we propose a method for model-efﬁcient classiﬁcation in a deep network with three contributions: (i) we introduce a kernelized classiﬁcation layer with built-in kernel learning that can utilize a given representation learning model more effectively.
(ii) we theoretically establish that it is possible to optimize, within the deep network itself, over all possible kernel functions we care about to ﬁnd the best RKHS that optimally separates embeddings.
(iii) we empirically show that the kernelized classiﬁcation layer is a viable alternative to using a larger backbone to improve the classiﬁcation accuracy in a number of computer vision and natural language processing tasks.
R ELATED W ORK
There have been several explorations of loss functions other than the usual softmax crossentropy loss in CNN settings, specially in the open-set classiﬁcation setting (Wen et al., 2019; Cevikalp &
Saglamlar, 2021; Deng et al., 2019; Wang et al., 2018).
An example is the Center Loss (Wen et al., 2019), which encourages low intra-class variance in the feature vectors.
Other methods such as
Deng et al.
(2019) achieve higher performance by leaving additional margins in the softmax loss.
Our work differs from these since we work in the closed-set classiﬁcation setting and employ an automatically learned kernel to obtain nonlinear classiﬁcation boundaries.
Second order pooling methods (Lin et al., 2015; Li et al., 2017; Wang et al., 2020) propose a way to perform nonlinear classiﬁcation in the embedding space.
In has been shown that second order pooling is equivalent to using a second-degree polynomial kernel (Gao et al., 2016;
Cai et al., 2017).
Cui et al.
(2017) extended second-order pooling to higher orders while learning the coefﬁcients of higher order interactions.
Their method requires explicit calculation of feature maps, which they tackle using Fast Fourier Transforms.
Mahmoudi et al.
(2020), use the kernel trick in the dense layer along with the polynomial kernel.
Our work differs from the above works in that we never calculate explicit feature maps and we theoretically show that our method learns over the space of all possible positive deﬁnite kernels on the hyper-sphere, which includes all polynomial kernels.
Some methods focus on extending the linear convolution in CNNs to a nonlinear operation.
Convolutional kernel networks (Mairal et al., 2014) provide a kernel approximation to interpret convolutions.
Zoumpourlis et al.
(2017) used Volterra series approximations to extend convolutions to a nonlinear operation.
Wang et al.
(2019b) proposed a kernelized version of the convolution operation and demonstrated that it can learn more complicated features than the usual convolution.
Our work differs from theirs in a number of ways: Some kernels used in their work, such as the L p -norm kernels, are not positive deﬁnite (Berg et al., 1984) and therefore do not represent a valid RKHS mapping (Aronszajn, 1950).
In contrast, we strictly work with positive deﬁnite kernels, which represent valid mappings to an RKHS.
Furthermore, learning of hyperparameters of pre-deﬁned kernels advocated in their work is principally different from the kernel learning method presented in this paper – we theoretically show that our method optimizes over the space of all radial positive deﬁnite kernels on the unit sphere to ﬁnd the best kernel, instead of limiting the optimization to the hyperparameters of pre-deﬁned kernels.
Prior to the dominance of deep learning methods, picking the right kernel for a given problem has been studied extensively in works such as Howley & Madden (2005); Ali & Smith-Miles (2006);
Jayasumana et al.
(2014); G¨onen et al.
In particular, Multiple Kernel Learning (MKL) approaches (G¨onen et al., 2011; Varma & Ray, 2007) were popular in conjunction with SVM.
Unfortunately, these methods scale poorly with the size of the training dataset.
In this work, we automatically learn the kernel within a deep network.
This not only allows automatic representation learning, but also scales well for large training sets.
Kernels have also been considered for deep learning to reduce the memory footprint of CNNs.
This was accomplished by achieving an endto-end training of a Fastfood kernel layer (Yang et al., 2015), which uses approximations of kernel functions using Fastfood transforms (Le et al., 2013).
Other related methods involving both kernels and deep learning include stochastic kernel machines (Dai et al., 2015), deep SimNets (Cohen et al., 2015), scalable deep kernels (Wilson et al., 2015), KerNET (Lauriola et al., 2020), and deep belief network based work of Le et al.
N ONLINEAR S OFTMAX C LASSIFICATION
In this section we discuss the usual classiﬁcation inside a deep network and its nonlinear extension.
Let us consider a classiﬁcation problem with a training set { ( x i , y i ) } Ni =1 , where each x i ∈ X ,
each y i ∈
[ L ] .
= { 1 , 2 , . . .
, L } , X is a nonempty set, L is the number of labels, and N is the number of training examples.
For instance, each training datum ( x i , y i ) can be an image with its class label.
A deep neural network that solves this task has two components: a representation learner and a classiﬁer .
In the case of image classiﬁcation, the representation learner consists of modules such as convolution layers, max-pooling layers, and fully-connected layers.
The classiﬁer is the last fullyconnected layer operating on the learned representations (embeddings).
This layer is endowed with a loss function during training.
Let r (Θ) :
X → R d denote the representation learner, where d is the dimensionality of the embeddings and Θ represents all the parameters in this part of the network.
The classiﬁer is characterized by a function g (Ω) : R d → [ L ] , where Ω denotes all the parameters in the last layer of the network.
Usually, Ω consists of weight vectors w 1 , w 2 , . . .
, w L with each w j ∈ R d , and bias terms b 1 , b 2 , . . .
, b L with each b j ∈ R .
The function g (Ω) then takes the form:
[Equation]
where z = r (Θ) ( x ) ∈ R
d is the embedding for input the x .
Note that we have dropped the additive bias term b j , with no loss of generality, to keep the notation uncluttered.
During inference, the deep network’s class prediction ˆ y ∗ for an input x ∗ is the composite of these two functions: ˆ y ∗ = (cid:0) g (Ω) ◦ r (Θ) (cid:1) ( x ∗ ) .
Although conceptually there are two components of the deep network, their parameters Θ and Ω are learned jointly during training.
The de facto standard way of training a classiﬁcation network is minimizing the softmax loss applied to the classiﬁcation layer.
The softmax loss is the combination of the softmax function and the cross-entropy loss.
More speciﬁcally, for a single training example ( x, y ) with the embedding z = r (Θ) ( x ) , the softmax loss is calculated as:
[Equation]
Note that the classiﬁer g (Ω) trained is this manner is completely linear in R d , the space of the embeddings z s, as is evident from (1).
From the classic knowledge in kernel methods, we are aware that more powerful nonlinear classiﬁers on R d can be obtained using the kernel trick.
The key idea here is to ﬁrst map the embeddings z s into a high-dimensional RKHS H and perform classiﬁcation there.
Although the classiﬁcation is linear in the high-dimensional H , it is nonlinear in the original embedding space R d .
Let φ : R d → H represent this RKHS embedding.
Performing classiﬁcation in H is then equivalent to training the neural network with the following modiﬁed version of the softmax loss:
[Equation]
where (cid:104) .
(cid:105)
H denotes the inner product in the Hilbert space H .
The key difference between (2) and (3) is that the dot products between w j s and z have been replaced with the inner products between φ ( w j ) s and φ ( z ) .
The more general notion of inner product is used instead of dot product because the Hilbert space H can be inﬁnite dimensional.
For a network trained with this nonlinear softmax function, predictions can be obtained using a modiﬁed version of the predictor:
[Equation]
Note that the Hilbert space embeddings φ ( . )
s can be very-high, even inﬁnite, dimensional.
Therefore, computing and storing them can be problematic.
We can use the kernel trick from classic machine learning (Sch¨olkopf & Smola, 2002; Shawe-Taylor & Cristianini, 2004) to overcome this problem: explicit computation of φ ( . )
s can be avoided by directly evaluating the inner product between them using a kernel function k : R d × R d → R .
That is:
[Equation]
Intuitively, mapping d -dimensional embeddings into a much higher dimensional RKHS using a kernel would help in ﬁnding complex, nonlinear patterns in the embeddings that the usual softmax classiﬁcation is unable to ﬁnd due to its linear nature.
We therefore expect kernelized classiﬁcation to use embeddings provided by a given representation learner more effectively.
K ERNELS ON THE U NIT S PHERE
It was shown in the previous section that, given a kernel function on the embedding space, we can obtain a nonlinear classiﬁer in the last layer of a deep network by modifying the softmax loss function during training and the predictor during inference.
However, only positive deﬁnite kernels allow this trick (Aronszajn, 1950; Berg et al., 1984).
There are various choices for kernel functions in the classic machine learning literature.
Some popular choices include the polynomial kernel, the Gaussian RBF kernel (squared exponential kernel), and the Laplacian kernel.
Nevertheless, it is often difﬁcult to decide the optimal kernel for a given problem.
Furthermore, many of the kernels have hyperparameters that need to be manually tuned.
The generally accepted solution to this problem in classic kernel methods is the MKL framework (G¨onen et al., 2011), where a better kernel is learned as a linear combination of some pre-deﬁned kernels.
Unfortunately, like SVM,
MKL methods do not scale well with the train set size.
Furthermore, usual MKL provides no guarantee to explore all possible kernel functions to ﬁnd the optimal one.
In this section, we present some theoretical results that will pave the way to deﬁne a neural network layer that can automatically learn the optimal kernel from data.
By formulating kernel learning as a neural network layer, we inherit the desirable properties of deep learning, including scalability and automatic representation learning.
Importantly, we show that our method can sweep over the entire space of positive deﬁnite kernels applicable to our problem setting to ﬁnd the best kernel.
We start the discussion with the following deﬁnition of positive deﬁnite kernels (Berg et al., 1984).
Deﬁnition 4.1.
Let U be a nonempty set.
A function k : ( U × U ) → R is called a positive deﬁnite kernel if k ( u, v ) =
k ( v, u ) for all u, v ∈ U and (cid:80) Nj =1 (cid:80) Ni =
c
i c j k ( u i , u j ) ≥ 0 , for all N ∈
N , { u 1 , . . . , u N } ⊆
U and { c 1 , . . .
, c N } ⊆ R .
Properties of positive deﬁnite kernels have been studied extensively in mathematics (Berg et al., 1984).
The following summarizes some important closure properties of this class of functions.
Proposition 4.2.
The family of all positive deﬁnite kernels on a given nonempty set forms a convex cone that is closed under pointwise multiplication and pointwise convergence.
Proof.
To intuitively understand this result, it is helpful to recall that the geometry of the family of the all positive deﬁnite kernels on a given nonempty set is closely related to that of the space of the d × d symmetric positive deﬁnite matrices, which forms a convex cone.
The formal proof of this proposition can be found in Remark 3.1.11 and Theorem 3.1.12 of Berg et al.
To simplify the problem setting, we assume that both the embeddings z s and the weight vectors w j s are L 2 -normalized.
Not only this simpliﬁes the mathematics, but also it is a practice in use for stabilizing the training of neural networks (Yi et al., 2019; Liu et al., 2017; Hoffer et al., 2018).
Due to this assumption, we are interested in positive deﬁnite kernels on the unit sphere in R d .
From now on, we use S n , where n = d − 1 , to denote this space.
We also restrict our discussion to radial kernels on S n .
Radial kernels, kernels that only depend on the distance between the two input points, have the desirable property of translation invariance.
Furthermore, all the commonly used kernels on S n , such as the linear kernel, the polynomial kernel, the Gaussian RBF kernel, and
the
Laplacian kernel are radial kernels.
The following theorem, origins of which can be traced back to
Schoenberg (1942), fully characterizes radial kernels on S n .
Theorem 4.3.
A radial kernel k : S n × S n → R is positive deﬁnite for any n if and only if it admits a unique series representation of the form
[Equation]
where each α m ≥ 0 , (cid:80) ∞ m = − 2 α m < ∞ , and (cid:74) .
(cid:75) depicts the Iversion bracket.
Proof.
The kernel k 1 : S n × S n → [ − 1 , 1] : k 1 ( u , v ) = (cid:104) u , v (cid:105) is positive deﬁnite on S n for any n since (cid:80) j (cid:80)
i c i c j (cid:104) u i , u j (cid:105) =
(cid:107) (cid:80)
i
c i u i (cid:107) 2 ≥ 0 .
Therefore, from the closure properties in Proposition 4.2, the kernel k m : ( u , v ) (cid:55)→ (cid:104) u , v (cid:105) m is also positive deﬁnite for any m ∈ N .
Furthermore, k m is positive deﬁnite for m = 0 since (cid:80) j (cid:80) i c
i c j (cid:104) u
i , u j (cid:105)
= (cid:107) (cid:80)
i
c
i (cid:107) 2 ≥ 0 .
Let us now consider the two sequences of kernels s odd =
k 1 , k 3 , . . . ,
k 2 m +1 , . .
and s even =
k 2 , k 4 , . .
k 2 m , . .
Since − 1 ≤ (cid:104) u , v (cid:105) ≤ 1 , it is clear that s odd and s even converge pointwise to k odd ( u , v ) =
(cid:74) (cid:104) u , v (cid:105) =
1 (cid:75) − (cid:74) (cid:104) u , v (cid:105) =
− 1 (cid:75) and k even ( u , v ) =
(cid:74) (cid:104) u , v (cid:105) ∈ {− 1 , 1 } (cid:75) , respectively.
From the last closure property of Proposition 4.2, both k odd and k even are positive deﬁnite on S n .
Invoking Proposition 4.2 again, we conclude that any ﬁnite conic combination of the kernels k even , k odd , k 0 , k 1 , . . .
is positive deﬁnite on S n for any n .
This completes the forward direction of the proof.
The proof of the converse is found in Chapter 5 of Berg et al.
Equipped with a complete characterization of the positive deﬁnite radial kernels on S n , we now discuss how we can combine this result with the nonlinear softmax formulation in § 3 to automatically learn the best kernel classiﬁer within a deep network.
T HE K ERNELIZED C LASSIFICATION L AYER
We now introduce a kernelized classiﬁcation layer that acts as a drop-in replacement for the usual softmax classiﬁcation layer in a deep network.
This new layer classiﬁes embeddings in a highdimensional RKHS while automatically choosing the optimal positive deﬁnite kernel that enables the mapping into the RKHS.
As a result, we do not have to hand-pick a kernel or its hyperparameters.
Our nonlinear classiﬁcation layer can utilize embeddings more efﬁciently than a softmax classiﬁer, which is linear in the embedding space.
We also show that the kernelized classiﬁcation layers comes with negligible added cost during both training and inference.
5. M ECHANICS OF THE L AYER
Our classiﬁcation layer is parameterized by the usual weight vectors: w 1 , w 2 , . .
, w L , and some additional learnable coefﬁcients: α − 2 , α − 1 , . .
. , α M , where M ∈ N and each α m ≥ 0 .
During training, this classiﬁer maps embeddings z s into a high-dimensional RKHS H opt that optimally separates embeddings belonging to different classes, and learns a linear classiﬁer in H opt .
During inference, the classiﬁer maps embeddings of previously unseen inputs to the RKHS it learned during training and performs classiﬁcation in that space.
This is achieved by using the nonlinear softmax loss deﬁned in (3) during training and the nonlinear predictor deﬁned in (4) during testing, with the inner product in H given by: (cid:104) φ ( w ) , φ ( z ) (cid:105)
H = (cid:104) φ ( w ) , φ ( z ) (cid:105)
H opt =
k opt ( w , z ) , where k opt ( ., . ) is the reproducing kernel of H opt .
The optimal RKHS H opt for a given classiﬁcation problem is learned by ﬁnding the optimal kernel k opt during training as discussed in the following.
Theorem 4.3 states that any positive deﬁnite radial kernel on S n admits the series representation shown in (6).
Therefore, the optimal kernel k opt must also have such a series representation.
We approximate this series with a ﬁnite summation by cutting off the terms beyond the order M :
[Equation]
where, k even , k odd , k 0 , k 1 , . .
, k M have meanings deﬁned § 4 and α − 2 , α − 1 , . . .
, α M ≥ 0 .
Using Proposition 4.2 and the discussion in the proof of Theorem 4.3, one can easily verify that this approximation does not violate the positive deﬁniteness of k opt .
A rigorous analysis of the accuracy of this approximation is provided in Appendix E.
With this, k opt is learned automatically from data by making the coefﬁcients α − 2 , α − 1 , . . .
, α M s learnable parameters of the classiﬁcation layer.
Let α =
[ α − 2 , α − 1 , . . .
, α M ] T .
The gradient of the loss function with respect to α can be calculated via the backpropagation algorithm using (3) and (7).
Therefore, it can be optimized along with w 1 , w 2 , . . .
, w L during the gradient descent based optimization of the network.
This procedure is equivalent to automatically ﬁnding the RKHS that optimally separates the embeddings belonging to different classes.
The constraint α − 2 , α − 1 , . .
, α M ≥ 0 in (7) can be imposed with a ReLU function (see Appendix D.5 for more discussion).
As shown later in § 7.3, the exact value of M is not critical as long as it is sufﬁciently large.
This is because, as discussed in the proof of Theorem 4.3, the higher order terms that are truncated approach either k odd or k even , both of which are already included in the ﬁnite summation.
On the other hand, if the terms beyond some order M (cid:48) < M are not important, the network can automatically learn to make the corresponding α coefﬁcients vanish.
We observed that M = 10 works well enough in practice and stick to this number in all our experiments.
Importantly, the kernelized classiﬁcation layer described above can pass on the loss gradients to its inputs.
Therefore, the kernelized classiﬁcation layer is fully compatible with end-to-end training and can act as a drop-in replacement for an existing softmax classiﬁcation layer.
5. A DDITIONAL C OMPLEXITY
Since we propose kernelized classiﬁcation as a replacement for the usual softmax classiﬁcation to improve model efﬁciency, one might wonder about the added cost of the kernelized classiﬁcation layer.
It uses ( M + 3) extra learnable parameters.
During both training and inference, the added computational complexity is O
( M + d ) per datum, assuming a commonly-available constant-time operation for taking powers.
Additional memory footprint is O ( M ) during training to account for cached gradients.
Note that M = 10 and d ranges from 64 to 768 in our experiments.
Therefore, the kernelized classiﬁcation comes with a negligible added cost over the softmax classiﬁcation in terms of compute, memory, and trained model storage.
E XPERIMENTS
We now present experimental evaluation of our method.
For all experiments, the main baseline is the standard softmax classiﬁer.
Where appropriate, we show three additional baseline results based on the linear kernel, second order pooling (Lin et al., 2015), and kervolutional networks (Wang et al., 2019b).
Note that the focus of our experiments is to demonstrate the beneﬁts of kernelized classiﬁcation in efﬁciently utilizing embeddings learned with various representation learners, not to claim state-of-the-art results on already well-explored benchmark datasets.
Details about our experimental setup is in Appendix A.
6. S YNTHETIC D ATA
We ﬁrst evaluate the kernelized classiﬁcation layer as an isolated unit by demonstrating its capabilities to learn nonlinear patterns on S n .
We train a softmax classiﬁer and our kernelized classiﬁer on the synthetic dataset described in Appendix B. Results on the test set are shown in Table 1.
We also report the theoretical maximum accuracy, the accuracy of the Bayes optimal classiﬁer.
The accuracy of our kernelized classiﬁcation layer signiﬁcantly outperforms the baseline and gets close to theoretical best.
This can be attributed to the layer’s capabilities to learn nonlinear patterns on the sphere by embedding the data into an RKHS that optimally separates the classes.
We visualize the outcomes of the classiﬁers in Figure 1.
Note that the softmax classiﬁer can only separate cap-like regions on S 2 , this is a result of its being a linear classiﬁer with respect to the embeddings.
Our kernelized classiﬁer, on the other hand, can do a more complex nonlinear separation of the embeddings.
[Table] [Table]
6. I MAGE C LASSIFICATION
We now report results on CIFAR-10 and CIFAR-100 real world image benchmarks 1 (Krizhevsky, 2009).
To demonstrate better model-efﬁciency with the kernelized classiﬁcation, we experiment with several CIFAR-ResNet architectures (He et al., 2016) with increasing model capacity.
We consider four different baselines: (1) Softmax: the standard softmax loss, (2) LIN: normalized embeddings and weights with only the linear kernel along with a learnable coefﬁcient.
This is similar to the approach discussed in Hoffer et al.
(2018), but with additional freedom to learn the weight vectors, (3) SOP: second order pooling (Lin et al., 2015), which is also equivalent to Mahmoudi et al.
(2020) with a second degree polynomial, and (4) KERVO: kervolutional networks (Wang et al., 2019b) with the best out of the Gaussian RBF kernel and the polynomial kernel.
Increased model-efﬁciency obtained with the kernelized classiﬁer is evident from the accuracies summarized in Table 3.
For example, the same accuracy of ResNet-56 (540K parameters) with the standard softmax classiﬁer can be obtained with a much smaller ResNet-32 (300K parameters) when the kernelized classiﬁer is used.
Our method signiﬁcantly outperforms the other baselines as well.
This shows the beneﬁts of optimizing over the entire space of positive deﬁnite kernels instead of restricting ourselves to linear methods or pre-deﬁned kernels.
[Table]
6. N ATURAL L ANGUAGE U NDERSTANDING
In this section
[Table]
we show the beneﬁts of the kernelized classiﬁcation layer in solving four different text classiﬁcation tasks in the GLUE benchmark (Wang et al., 2019a).
We use mask-LM pretrained
BERT models of different capacities (Turc et al., 2019) and ﬁnetune them on each classiﬁcation task.
Note that we do not use distillation and directly ﬁnetune the models with the dataset labels.
Since detailed analyses on GLUE test datasets are not allowed (Wang et al., 2019a), we tune hyperparameters on subsets of train sets and report results on the validation sets in Table 4.
More details about the experiment setup are provided in Appendix A.
Table 4 provides evidence that kernelized classiﬁcation layers helps in extracting more gains out of a given representation learner.
For example, across all the datasets, BERT-Mini (11.3M parameters) with kernelized classiﬁcation can get similar results as the BERT-Small (29.1M parameters) with softmax classiﬁcation.
Therefore, using the nonlinear, kernelized classiﬁer is an effective alternative to using a bigger backbone for increasing classiﬁcation performance.
6. K NOWLEDGE D ISTILLATION
We now evaluate our method in the distillation setting to show that it is complementary to existing model compression techniques.
We used the CIFAR-10 and CIFAR-100 datasets, the softmax CIFAR ResNet-56 models from Tables 3 as the teacher models, and the LeNet-5 network (Lecun et al., 1998) as the student model.
Note that it is straightforward to utilize the kernelized classiﬁcation layer in the distillation setting described in Hinton et al. (2015) by replacing usual logits with their these
[Table]
kernelized counterparts.
We use the cross-entropy loss with the teacher scores with the temperature set to 20 in all cases.
Results are shown in Table 2.
Signiﬁcant gains are observed with the kernelized classiﬁcation layer.
This can be attributed to the layer’s capabilities to approximate complex teacher probabilities even with weak embeddings due to the nonlinear classiﬁer.
6. A CTIVE L EARNING
Active learning focuses on reducing human annotation costs by selecting a subset of images to label that are more likely to yield the best model (Settles, 2009).
We used different sampling methods such as random, margin (Lewis & Gale, 1994; Scheffer et al., 2001), and k-center (Sener & Savarese, 2017; Wolf, 2011) to generate subsets of various sizes.
The setup is detailed in Appendix C. As shown in Table 5, our results on random subsets outperform the softmax ResNet-56 models on margin and k-center based subsets, and we achieve even better results using improved sampling methods.
This demonstrates that the kernelized classiﬁcation layer can produce better models in limited-data settings as well.
A BLATION S TUDIES
We now present ablation experiments using CIFAR-100 and the ResNet-56 backbone
. 7. K ERNEL L EARNING
To investigate the beneﬁts of automatic kernel learning compared to using a pre-deﬁned kernel in the kernelized classiﬁcation layer, we compare our kernel learning method with two pre-deﬁned kernels in the kernelized classiﬁcation layer: the polynomial kernel of order 10 and the Gaussian
RBF kernel.
We also show results with an MKL baseline with linear, 2nd order, and Gaussian
[Table]
-ctr rnd mgn k-ctr 58.
kernels.
Results are shown in Table 6.
It is evident that sweeping over all possible kernels in light of Theorem 4.3 to ﬁnd the best kernel yields signiﬁcant practical beneﬁts.
This is not surprising because the polynomial and Gaussian kernels are members of the large search space of kernels that we optimize over in our method.
[Table] [Table]
7. E MBEDDING N ORMALIZATION
As discussed previously, we L 2 -normalize both embeddings and weights in the classiﬁcation layer.
We study the effect of this normalization for the baseline softmax loss in Table 7.
Note the setting where both embeddings and weights are normalized with an appropriate temperature T is equivalent to the cosine softmax loss (Liu et al., 2017;
Chen et al., 2019; Wang et al., 2020).
Note also that the
LIN baseline considered in Tables 3 and 4 uses normalized embeddings and weights with an automatically learned T , same as the fourth row in Table 7.
In our softmax baselines we use normalized embeddings since it works consistently better than the unnormalized version.
7. S ENSITIVITY TO THE O RDER OF THE A PPROXIMATION
As discussed in § 5.1, intuitively, the order M of the approximation should not matter as long as it is large enough.
We verify this in Figure 2 with the CIFAR-100 dataset, where we show the changes in accuracy with increasing M .
We use M = 10 in all our experiments.
[Equation]
ELP ?
One could wonder whether more fully-connected layers at the end of the network would improve the classiﬁcation.
To address this, we added an additional fully-connected layer with d units to
ResNet-56.
The accuracy improved only marginally from 71 .
23 to 71 .
29 , as opposed to 74 .
10 with our method.
This observation is in-line with the discussion in Rendle et al.
(2020): MLP scorers are somewhat difﬁcult to train.
An explanation for this could be the common observation
that, although
MLPs can theoretically approximate any function, learning one from data is difﬁcult.
This has motivated inductive-bias based models such as CNNs and Transformers.
Kernelized classiﬁcation can also be viewed as a way of presenting an inductive bias motivated by the classic kernel method theory.
Note also that, unlike the kernelized classiﬁcation layer, added MLP layers come with a signiﬁcant increase in the model’s computational and memory complexity.
7. D IFFERENT L OSS F UNCTIONS , A CTIVATION FOR C OEFFICIENT , ETC .
We also provide a number of other ablations studies on different loss functions, effects of embedding rectiﬁcation, kernel coefﬁcient activation, and other settings in Appendix D.
C ONCLUSION
We presented a kernelized classiﬁcation layer for deep neural networks aiming to extract the best possible classiﬁer with embeddings produced by a given representation learner.
This classiﬁcation layer classiﬁes embeddings in a high dimensional RKHS while automatically learning the optimal kernel that enables this high-dimensional mapping.
We showed that a classiﬁcation network with a lightweight representation learning backbone can be made more effective by replacing the usual softmax classiﬁer with the kernelized classiﬁer.
We showed consistent and substantial accuracy improvements in image classiﬁcation, natural language understanding, distillation, and active learning settings.
These accuracy improvements strongly support the usefulness of kernelized classiﬁcation layer in ﬁnding nonlinear patterns in the embeddings.
E THICS S TATEMENT
This work concerns mathematical and empirical analysis of deep learning based classiﬁcation techniques with applications in image recognition and natural language understanding.
We do not foresee our work having undue societal effects.
Our work does not explicitly consider issues of fairness in classiﬁcation, which is an important yet under-studied dimension.
We do not foresee our techniques as unduly amplifying biases in existing algorithms.
A E XPERIMENTAL S ETUP
In all experiments, we use M = 10 for the kernelized classiﬁcation layer.
As discussed in § 5.1 and
Appendix D.5, we use ReLU activation and a weight decay of 0 . 0001 on the α parameter vector.
This is the same amount of weight decay used in the other parts of the network, when applicable.
The α parameter vector is initialized with all ones.
For all image classiﬁcation experiments, we use SGD with 0.9 momentum, linear learning rate warmup (Goyal et al., 2017), cosine learning rate decay (Loshchilov & Hutter, 2017), and decide the base learning by cross validation.
When a better learning rate schedule is available for the baseline (e.g. the CIFAR schedule in He et al. 2016), we experimented with both that and our schedule and report the best accuracy of the two.
The maximum number of epochs was 450 in all cases.
Minibatch size was 128 for the synthetic and CIFAR datasets and 64 other datasets with larger images used in Appendix D.2.
We used the standard CIFAR data augmentation method in He et al.for CIFAR-10 and CIFAR-100 datasets, and the Imagenet data augmentation in the same paper for other image datasets.
Some ResNet models use ReLU activation on embeddings.
We also remove this activation to utilize the full surface of S n .
The same is done to the baseline models to enable a fair comparison (see AppendixD.4 for more details).
For the natural language understanding tasks in § 6.3, we use the publicly-available, MLMpretrained BERT/Small-BERT models from TensorFlow Hub (Turc et al., 2019).
Usual CLS pooling is done at the end of the Transformer encoder to obtain an embedding for each input sentence/sentence pair.
We use the AdamW optimizer with linear ramp-up and decay as is standard for BERT model ﬁnetuning.
Note that we do not distill the ﬁnal task from a bigger model and directly ﬁnetune with the one-hot labels in the original datasets.
Hyperparameter search followed the procedure described in Turc et al.
Since detailed analysis of different methods on GLUE test tests are not allowed, we tune hyperparameters on a 10% subset of the train set and report results on the validation sets.
Once the hyperparamers are decided, the full train set is used to train the ﬁnal model.
Since GLUE validation sets are small, we report the average accuracy over 5 different runs for each method.
We do not however notice signiﬁcant variance in accuracy across different runs.
B S YNTHETIC D ATA G ENERATION
We generated the binary classiﬁcation dataset used in § 6.1 using a mixture of Gaussians, inspired by the blue-orange dataset in Hastie et al.
More speciﬁcally, we ﬁrst generated 10 cluster centers for each class by sampling from an isotropic Gaussian distribution with covariance 0 .
5 I 3 and mean [1 , 0 ,
] T for the blue class and [0 , 1 , 0] T for the orange class.
We then generated 5,000 train observations for each class using the following method: for each observation, we uniform-randomly picked a cluster center of the corresponding class and then generated a sample from an isotropic
Gaussian distribution centered at that cluster center with covariance 0 .
02 I 3 .
All the observations were projected on to S 2 by L 2 -normalizing them.
The test set was generated in the same manner using the same cluster centers as the train set.
C A CTIVE L EARNING
The goal of the active learning experiment is to show that the kernelized classiﬁcation layer can produce accurate models even with less data.
In order to study this, we produce subsets of datasets under various budgets using several sampling techniques, and evaluate the models trained on this.
The simplest one is random sampling, where images are selected randomly under a given budget.
Other methods include margin (Lewis & Gale, 1994; Scheffer et al., 2001), and k-center (Sener &
Savarese, 2017; Wolf, 2011) where class prediction scores and embeddings from the images are used in the subset selection.
We do not rely on the actual labels of the images in the dataset during the subset selection, since active learning is driven toward reducing label annotation costs.
We used a 10% random subset of the original CIFAR-100 dataset with labels to ﬁrst learn an initial seed model, which was then used to generate predictions and embeddings.
Note that only embeddings and class prediction scores from this initial seed model are used in subset selection, and we do not access the original class labels of the images.
We use a batch setting where we do not incrementally update the model after selecting every image, and we directly select entire subsets under a given budget.
In all our experiments, we used the CIFAR ResNet-56 model.
The learning rate, batch size and the number of epochs are provided in Appendix A.
The embeddings are of dimension 64.
For the k-center method, we need distances between the embeddings, and we used cosine distances computed using fast similarity search of (Guo et al., 2020).
D A DDITIONAL E XPERIMENTS
We report a number of additional experimental results in this section.
D. A DDITIONAL B ACKBONES
Image classiﬁcation results with VGG-16 and DenseNet-40-12 backbones are reported in Table 8.
Since the original VGG-16 is designed for 224 × 224 images, we used a modiﬁed CIFAR version with 256 dimensional hidden size at the end.
[Table]
D. I MAGE C LASSIFICATION T RANSFER L EARNING
Here, we evaluate our method in a image classiﬁcation transfer learning setting.
To this end, we take a ResNet-50 network pre-trained on the Imagenet ILSVRC 2012 classiﬁcation dataset (Deng et al., 2009) and ﬁnetune it on Oxford-IIIT Pets (Parkhi et al., 2012) and Stanford Cars (Krause et al., 2013) datasets.
For each dataset, we use the train/test splits provided by the standard Tensorﬂow Datasets implementation (Authors).
Results are summarized in Table 9.
Note that the KERVO baseline is not possible in this setting as it involves changes to the backbone network.
On both datasets, kernelized classiﬁcation layer results in signiﬁcant gains over the baselines.
This is intuitive to understand since the embeddings learned from the source task (Imagenet) might not linearly separate the new classes in the target task.
We can therefore beneﬁt from a nonlinear classiﬁer in the transfer learning setting.
[Table]
D. D IFFERENT L OSS F UNCTIONS
To evaluate the kernelized classiﬁcation layer under a loss function other than the cross-entory loss, we report CIFAR-10/100 results with the squared loss (Hui & Belkin, 2020) in Table 10.
Note that the application of squared loss to the kernelized classiﬁcation layer’s outputs is straightforward since it outputs logits in the usual sense.
D. E FFECT OF E MBEDDING R ECTIFICATION
As discussed previously, different to the usual image classiﬁcation networks in He et al.
(2016), we remove the ReLU activation from the embeddings.
This is to utilize the full surface of S n without restricting ourselves to only the nonnegative orthant.
As is evident from Table 11, removing ReLU has only a marginal effect on the standard softmax baseline.
It is however an important factor for our method.
We consistently used embeddings without the ReLU activation in all our experiments in the previous sections.
[Table] [Table]
D. E FFECT OF ACTIVATION ON THE KERNEL COEFFICIENTS
Following the discussion in § 5.
[Table]
the constraint α − 2 , α − 1 , . .
, α M ≥ 0 is important to preserve the positive deﬁniteness of k opt .
This can be imposed by using α = ReLU( α (cid:48) ) , where α (cid:48) is the learnable parameter vector.
However, ReLU has no upper-bound and allowing the scale of α to grow unboundedly causes issues in optimization: Assume we have an instantiation α 0 of the vector α .
By replacing α 0 with λ α 0 , where λ > [Table] we scale all the inner product terms in (3) and (4) by the same λ .
As a result
[Table]
we improve the loss of the already correctly classiﬁed training examples
[Table]
but without making any effective change to the predictor.
Therefore, under this setting, once the majority of the training examples are correctly classiﬁed, the neural network can easily improve the loss just by increasing the norm of α , which is not useful.
We therefore advocate an L 2 -regularization term on α when ReLU activation is used.
Alternatively, one could also use α = sigmoid( α (cid:48) ) or α = softmax( α (cid:48) ) , both of which not only guarantee α − 2 , α − 1 , . .
, α M ≥ 0 , but also produce bounded α .
Therefore, no regularization on α is needed for these options.
The softmax activation here should not be confused with the softmax loss discussed in § 3.
The usage of the softmax activation in this context is similar to that in the self-attention literature (Vaswani et al., 2017), where it is used to normalize the coefﬁcients of a linear combination.
We experimented with these different activations on α (cid:48) and summerized the results in Table 12.
We used a weight decay of 0.0001 on the coefﬁcient vector whenever ReLU activation is used.
Although sigmoid and softmax activations eliminate the need for weight decay, they put a hard constraint on | (cid:104) φ ( w ) , φ ( f ) (cid:105) H | .
To overcome this limitation, it is helpful to use a temperature hyperparameter in (3), where each inner product is divided by T before taking the exponential.
We used a temperature of 0.1 and 0.005, with sigmoid and softmax , respectively.
Although sigmoid gives the best performance in Table 12, we occasionally observed optimization issues with it, which could be due to the vanishing gradient issue associated with this activation function.
We therefore stick to ReLU in all other experiments.
We however note that, in most cases, competitive results can be obtained with softmax as well, when used with a temperature of 0.005.
It is also interesting to note that using no activation function on α (cid:48) causes frequent divergence in training.
This is consistent with the theory: The summation in (7) is not guaranteed to be positive deﬁnite when a m s are allowed to be negative (see Proposition 4.2).
Therefore, the theory of kernelized classiﬁcation is not valid in this case.
[Table]
E A PPROXIMATION E RROR B OUNDS
In this section, we analyze error bounds for the approximation in Eq.
We start by proving the following theorem, which establishes a rigorous upper bound for the approximation error.
Theorem E.1.
Let k : S n × S n → R be any positive deﬁnite radial kernel on S n with the series expansion
k ( u , v ) =
α − 2 k even ( u , v ) + α − 1 k odd ( u , v ) + (cid:80) ∞
{+m =
+}
α m (cid:104) u , v (cid:105) m , and k M : S n ×
S n → R be its M th partial sum.
Deﬁne ψ : [ − 1 , 1] → R as ψ ( x ) :
= (cid:80) ∞
{+m =
+}
α m x m .
Then the approximation error bound for the partial sum of the kernel is given by
[Equation]
for all ( u , v ) ∈ S n × S n .
Proof.
Since k is positive deﬁnite, from Theorem 4.3, it has a series expansion of the form:
[Equation]
[Equation]
+
[Equation]
+ ∞ (cid:88) m
= α m (cid:104)
[Equation]
(cid:105)
[Equation]
Note that ψ is an analytic function.
Furthermore,
[Equation]
where ψ M ( x ) := (cid:80)
{+Mm =
+}
α m x m .
Therefore, ψ M ( x ) is the M th order Maclaurin polynomial approximation of ψ ( x ) .
Since ψ ( x ) is analytic, and therefore inﬁnitely differentiable, we can obtain the
Lagrange form of the approximation error as:
[Equation]
for some ξ ∈ ( − x, x ) , for all x ∈ [ − 1 , 1] , where ψ ( M +1) is the ( M + 1) th order derivative of ψ .
It follows that, for all ( u , v ) ∈ S n × S n ,
[Equation]
The above theorem states that the absolute error made by cutting off the terms beyond order M is less than or equal to the maximum absolute value of the ( M + 1) th order derivative of k , attenuated by a factor of 1 / ( M + 1)! .
To put this into context, since we use M = 10 , the attenuation factor is around 2 .
5 × 10 − 8 .
Therefore, to make the error signiﬁcant, the 11 th order derivative of the kernel would have to be very high, suggesting a kernel function with abrupt changes looking almost like discontinuities.
Since such functions are unlikely to be useful to learn a generalizable model, we believe that the error caused by this approximation is indeed negligible.
Note also that, we set k M to be the Maclaurin polynomial in the proof of Theorem E.1 to make the derivations easier.
However, since we have the freedom to learn the coefﬁcients of the summation, it is theoretically possible to approximate k even better.
In particular, even when k has abrupt changes causing a non-negligible error in the Maclaurin approximation, it will be possible to capture some of the residuals using the kernels k odd and k even in k M ’s expansion.
This is because the higher order kernels reach one of these kernels in the limit as discussed in the proof of Theorem 4.3.
F C OMPARISON TO N ON - PARAMETRIC K ERNEL M ETHODS
We use a parametric model with kernels in our method.
This is in contrast to the more popular usage of kernels with non-parametric models, such as support vector machines and Gaussian processes.
Non-parametric models are usually more ﬂexible.
They are also more interpretable than deep network-based parametric models.
However, unfortunately, non-parametric models scale poorly with the train set size.
For example, the number of support vectors grow linearly with the train set size (Steinwart, 2003), and Gaussian processes methods scale with the cube of the train set size, or linearly after some optimizations (Wilson & Nickisch, 2015).
In contrast, parametric models, such as the one proposed in this work, scale well with the training set size since they use a constant number of parameters regardless of the number of training examples.
In fact, this is one of the main reasons why deep learning methods have become extremely popular in the recent years.
Non-parametric models also allow faster inference, making them suitable for developing models for compute-limited scenarios, which is the primary focus of this work.