A CTIVE L EARNING FOR C ONVOLUTIONAL N EURAL N ETWORKS : A C ORE -S ET A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
Convolutional neural networks (CNNs) have been successfully applied to many recognition and learning tasks using a universal recipe; training a deep model on a very large dataset of supervised examples.
However, this approach is rather restrictive in practice since collecting a large set of labeled images is very expensive.
One way to ease this problem is coming up with smart ways for choosing images to be labelled from a very large collection (i.e. active learning).
Our empirical study suggests that many of the active learning heuristics in the literature are not effective when applied to CNNs in batch setting.
Inspired by these limitations, we deﬁne the problem of active learning as core-set selection , i.e. choosing set of points such that a model learned over the selected subset is competitive for the remaining data points.
We further present a theoretical result characterizing the performance of any selected subset using the geometry of the datapoints.
As an active learning algorithm, we choose the subset which is expected to yield best result according to our characterization.
Our experiments show that the proposed method signiﬁcantly outperforms existing approaches in image classiﬁcation experiments by a large margin.
INTRODUCTION
Deep convolutional neural networks (CNNs) have shown unprecedented success in many areas of research in computer vision and pattern recognition, such as image classiﬁcation, object detection, and scene segmentation.
Although CNNs are universally successful in many tasks, they have a major drawback; they need a very large amount of labeled data to be able to learn their large number of parameters.
More importantly, it is almost always better to have more data since the accuracy of CNNs is often not saturated with increasing dataset size.
Hence, there is a constant desire to collect more and more data.
Although this a desired behavior from an algorithmic perspective (higher representative power is typically better), labeling a dataset is a time consuming and an expensive task.
These practical considerations raise a critical question: “what is the optimal way to choose data points to label such that the highest accuracy can be obtained given a ﬁxed labeling budget.”
Active learning is one of the common paradigms to address this question.
The goal of active learning is to ﬁnd effective ways to choose data points to label, from a pool of unlabeled data points, in order to maximize the accuracy.
Although it is not possible to obtain a universally good active learning strategy (Dasgupta, 2004), there exist many heuristics (Settles, 2010) which have been proven to be effective in practice.
Active learning is typically an iterative process in which a model is learned at each iteration and a set of points is chosen to be labelled from a pool of unlabelled points using these aforementioned heuristics.
We experiment with many of these heuristics in this paper and ﬁnd them not effective when applied to CNNs.
We argue that the main factor behind this ineffectiveness is the correlation caused via batch acquisition/sampling.
In the classical setting, the active learning algorithms typically choose a single point at each iteration; however, this is not feasible for CNNs since i) a single point is likely to have no statistically signiﬁcant impact on the accuracy due to the local optimization methods, and ii) each iteration requires a full training until convergence which makes it intractable to query labels one-by-one.
Hence, it is necessary to query labels for a large subset at each iteration and it results in correlated samples even for moderately small subset sizes.
In order to tailor an active learning method for the batch sampling case, we decided to deﬁne the active learning as core-set selection problem.
Core-set selection problem aims to ﬁnd a small subset given a large labeled dataset such that a model learned over the small subset is competitive over the whole dataset.
Since we have no labels available, we perform the core-set selection without using the labels.
In order to attack the unlabeled core-set problem for CNNs, we provide a rigorous bound between an average loss over any given subset of the dataset and the remaining data points via the geometry of the data points.
As an active learning algorithm, we try to choose a subset such that this bound is minimized.
Moreover, minimization of this bound turns out to be equivalent to the k-Center problem (Wolf, 2011) and we adopt an efﬁcient approximate solution to this combinatorial optimization problem.
We further study the behavior of our proposed algorithm empirically for the problem of image classiﬁcation using three different datasets.
Our empirical analysis demonstrates state-of-the-art performance by a large margin.
R ELATED W ORK
We discuss the related work in the following categories separately.
Brieﬂy, our work is different from existing approaches in that i ) it deﬁnes the active learning problem as core-set selection, ii ) we consider both fully supervised and weakly supervised cases, and iii ) we rigorously address the core-set selection problem directly for CNNs with no extra assumption.
Active Learning Active learning has been widely studied and most of the early work can be found in the classical survey of Settles (2010).
It covers acquisition functions such as information theoretical methods (MacKay, 1992), ensemble approaches (McCallumzy & Nigamy, 1998;
Freund et al., 1997) and uncertainty based methods (Tong & Koller, 2001; Joshi et al., 2009; Li & Guo, 2013).
Bayesian active learning methods typically use a non-parametric model like Gaussian process to estimate the expected improvement by each query (Kapoor et al., 2007) or the expected error after a set of queries (Roy & McCallum, 2001).
These approaches are not directly applicable to large
CNNs since they do not scale to large-scale datasets.
A recent approach by Gal & Ghahramani (2016) shows an equivalence between dropout and approximate Bayesian inference enabling the application of Bayesian methods to deep learning.
Although Bayesian active learning has been shown to be effective for small datasets (Gal et al., 2017), our empirical analysis suggests that they do not scale to large-scale datasets because of batch sampling.
One important class is that of uncertainty based methods, which try to ﬁnd hard examples using heuristics like highest entropy (Joshi et al., 2009), and geometric distance to decision boundaries (Tong & Koller, 2001; Brinker, 2003).
Our empirical analysis ﬁnd them not to be effective for CNNs.
There are recent optimization based approaches which can trade-off uncertainty and diversity to obtain a diverse set of hard examples.
Elhamifar et al.
(Elhamifar et al., 2013) design a discrete optimization problem for this purpose and use its convex surrogate.
However, the algorithm uses n 2 variables where n is the number of data points.
Hence, it does not scale to large datasets.
There are also many discrete optimization based active learning algorithms designed for the speciﬁc class of machine learning algorithms like k-nearest neighbors and naive Bayes (Wei et al., 2015).
Even in the algorithm agnostic case, one can design a set-cover algorithm to cover the hypothesis space using sub-modularity (Guillory & Bilmes, 2010; Golovin & Krause, 2011).
Our algorithm can be considered to be in this class; however, we do not use any uncertainty information.
Our algorithm is also the ﬁrst one which is applied to the CNNs.
Most similar to ours is (Joshiy et al., 2010), using a similar optimization problem.
However, they offer no theoretical justiﬁcation or analysis.
Moreover, their algorithm is also not experimented with CNNs.
Recently, a discrete optimization based method (Berlind & Urner, 2015) which is similar to ours has been presented for k-NN type algorithms in the domain shift setting.
Although our theoretical analysis borrows some techniques from them, their results are only valid for k-NNs.
Active learning algorithms for CNNs are also recently presented in (Wang et al., 2016; Stark et al., 2015).
Wang et al.
(2016) propose an heuristic based algorithm which directly assigns labels to the data points with high conﬁdence and queries labels for the ones with low conﬁdence.
Moreover,
Stark et al.
(2015) speciﬁcally targets recognizing CAPTCHA images.
Although their results are promising for CAPTCHA recognition, their method is not effective for image classiﬁcation.
We discuss limitations of both approaches in Section 5.
Core-Set Selection
The closest literature to our work is the problem of core-set selection since we deﬁne active learning as a core-set selection problem.
This problem considers a fully labeled dataset and tries to choose a subset of it such that the model trained on the selected subset will perform as closely as possible to the model trained on the entire dataset.
For speciﬁc learning algorithms, there are methods like core-sets for SVM (Tsang et al., 2005) and core-sets for k-Means and k-Medians (Har-Peled & Kushal, 2005).
However, we are not aware of such a method for CNNs.
The most similar algorithm to ours is the unsupervised subset selection algorithm in (Wei et al., 2013).
It uses a facility location problem to ﬁnd a diverse cover for the dataset.
Our algorithm differs in that it uses a slightly different formulation of facility location problem.
Instead of the min-sum, we use the minimax (Wolf, 2011) form.
More importantly, we apply this algorithm for the ﬁrst time to the problem of active learning and provide theoretical guarantees for CNNs.
Weakly
-Supervised
Deep Learning Our paper is also related to semi-supervised deep learning since we experiment the active learning both in the fully-supervised and weakly-supervised scheme.
One of the early weakly-supervised convolutional neural network algorithms was Ladder networks (Rasmus et al., 2015).
Recently, we have seen adversarial methods which can learn a data distribution as a result of a two-player non-cooperative game (Salimans et al., 2016;
Goodfellow et al., 2014;
Radford et al., 2015).
These methods are further extended to feature learning (Dumoulin et al., 2016; Donahue et al., 2016).
We use Ladder networks in our experiments; however, our method is agnostic to the weakly-supervised learning algorithm choice and can utilize any model.
P ROBLEM D EFINITION
In this section, we formally deﬁne the problem of active learning in the batch setting and set up the notation for the rest of the paper.
We are interested in a C class classiﬁcation problem deﬁned over a compact space X and a label space Y = { 1 , . . .
, C } .
We also consider a loss function l ( · , · ; w ) :
X × Y → R parametrized over the hypothesis class ( w ), e.g. parameters of the deep learning algorithm.
We further assume class-speciﬁc regression functions η c ( x ) = p ( y = c
| x ) to be λ η -Lipschitz continuous for all c .
We consider a large collection of data points which are sampled i.i.d.
over the space Z = X
× Y as { x i , y i } i ∈ [ n ] ∼ p Z where [ n ] = { 1 , . . . , n } .
We further consider an initial pool of data-points chosen uniformly at random as s
= { s 0 ( j ) ∈
[ n ] } j ∈ [ m ] .
An active learning algorithm only has access to { x i } i ∈ [ n ] and { y s ( j ) } j ∈ [ m ] .
In other words, it can only see the labels of the points in the initial sub-sampled pool.
It is also given a budget b of queries to ask an oracle, and a learning algorithm
A s which outputs a set of parameters w given a labelled set s .
The active learning with a pool problem can simply be deﬁned as
[Equation]
In other words, an active learning algorithm can choose b extra points and get them labelled by an oracle to minimize the future expected loss.
There are a few differences between our formulation and the classical deﬁnition of active learning.
Classical methods consider the case in which the budget is 1 ( b = 1 ) but a single point has negligible effect in a deep learning regime hence we consider the batch case.
It is also very common to consider multiple rounds of this game.
We also follow the multiple round formulation with a myopic approach by solving the single round of labelling as;
[Equation]
We only discuss the ﬁrst iteration where k = 0 for brevity although we apply it over multiple rounds.
At each iteration, an active learning algorithm has two stages: 1.
identifying a set of data-points and presenting them to an oracle to be labelled, and 2.
training a classiﬁer using both the new and the previously labeled data-points.
The second stage (training the classiﬁer) can be done in a fully or weakly-supervised manner.
Fully-supervised is the case where training the classiﬁer is done using only the labeled data-points.
Weakly-supervised is the case where training also utilizes the points which are not labelled yet.
Although the existing literature only focuses on the active learning for fully-supervised models, we consider both cases and experiment on both.
M ETHOD 4. A CTIVE L EARNING AS A S ET C OVER
In the classical active learning setting [Equation] the algorithm acquires labels one by one by querying an oracle (i.e. b = 1 ).
Unfortunately, this is not feasible when training CNNs since i ) a single point will not have a statistically signiﬁcant impact on the model due to the local optimization algorithms.
ii ) it is infeasible to train as many models as number of points since many practical problem of interest is very large-scale and each iteration is computationally expensive.
Hence, we focus on the batch active learning problem in which the active learning algorithm choose a moderately large set of points to be labelled by an oracle at each iteration.
In order to design an active learning strategy which is effective in batch setting
[Equation]
we consider the following upper bound of the active learning loss we formally deﬁned in (1):
[Equation]
[Table]
[Equation]
The quantity we are interested in is the population risk of the model learned using a small labelled subset ( s ).
The population risk is controlled by the training error of the model on the labelled subset, the generalization error over the full dataset ( [ n ] ) and a term we deﬁne as the core-set loss .
Core-set loss is simply the difference between average empirical loss over the set of points which have labels for and the average empirical loss over the entire dataset including unlabelled points.
Empirically, it is widely observed that the CNNs are highly expressive leading to very low training error and they typically generalize well for various visual problems.
Hence, the critical part for active learning is the core-set loss.
Following this observation, we re-deﬁne the active learning problem as:
[Equation] [Equation]
Informally, given the initial labelled set ( s 0 ) and the budget ( b ), we are trying to ﬁnd a set of points to query labels ( s 1 ) such that when we learn a model, the performance of the model on the labelled subset and that on the whole dataset will be as close as possible.
4. C ORE -S ETS FOR CNN S
The optimization objective we deﬁne in (4) is not directly computable since we do not have access to all the labels (i.e. [ n ] \ ( s 0 ∪ s 1 ) is unlabelled).
Hence, in this section we give an upper bound for this objective function which we can optimize.
We start with presenting this bound for any Lipschitz loss function and then show that loss functions of CNNs with ReLu non-linearities are Lipschitz.
We also rely on the zero training error assumption.
Although the zero training error is not an entirely realistic assumption, our experiments ﬁnd the resulting upper bound very effective.
Moreover, zero training error can be enforced by converting average loss into maximal loss (Shalev-Shwartz & Wexler, 2016).
We state the following proposition;
Proposition 1.
Given n i.i.d.
samples drawn from p Z as { x i , y i } i ∈ [ n ] , and set of points s .
If loss function l ( · , y, w ) is λ l -Lipschitz continuous for all y, w and bounded by L , regression function is λ η -Lipschitz, s is δ s cover of { x i , y i } i ∈ [ n ] , and l ( x s ( j ) , y s ( j ) ; A S ) =
∀ j ∈ [ m ] ; with probability at least 1 − γ ,
[Equation]
[Figure]
We visualize this proposition in Figure 1 and defer its proof to the appendix.
In this proposition, “a set s is a δ cover of a set s (cid:63) ” means a set of balls with radius δ centered at each member of s can cover the entire s (cid:63) .
Informally, this proposition suggests that we can bound the core-set loss with covering radius and a term which goes to zero with rate depends solely on n .
This is an interesting result since this bound does not depend on the number of labelled points.
In other words, a provided label does not help the core-set loss unless it decreases the covering radius.
In order to show that this bound applies to CNNs, we prove the Lipschitz-continuity of the loss function of a CNN with the following lemma where max-pool and restricted linear units are the non-linearities and the loss is deﬁned as the l 2 distance between the desired probabilities and the soft-max outputs.
Lemma 1.
A convolutional neural network with n c convolutional (with max-pool and ReLU) and n fc fully connected layers deﬁned over C classes with loss function deﬁned as the 2-norm between the softmax output and class probability is (cid:16) √ C − 1 C
[Equation]
fc (cid:17) -Lipschitz.
Here, α is the maximum sum of input weights per neuron (see appendix for formal deﬁnition).
Although it is in general unbounded, it can be made arbitrarily small without changing the loss function behavior (i.e. keeping the label of any data point s unchanged).
We defer the proof to the appendix and conclude that CNNs enjoy the bound we presented in Proposition 1.
In order to computationally perform active learning, we use this upper bound.
In other words, the practical problem of interest becomes min s 1 : | s 1 ≤ b | δ s 0 ∪ s 1 .
This problem is equivalent to the k-Center problem (also called min-max facility location problem)
(Wolf, 2011).
In the next section, we explain how we solve the k-Center problem in practice using a greedy approximation.
4. S OLVING THE K -C ENTER P ROBLEM
Input: data x i , existing pool s 0 and a budget b
Initialize s = srepeat u = arg max i ∈ [ n ] \ s min j ∈ s ∆( x i , x j ) s
= s ∪ { u } until | s | =
b + | s 0 | return s \ s
We have so far provided an upper bound for the loss function of the core-set selection problem and showed that minimizing it is equivalent to the k-Center problem (minimax facility location (Wolf, 2011)) which can intuitively be deﬁned as follows; choose b center points such that the largest distance between a data point and its nearest center is minimized.
Formally, we are trying to solve:
[Equation]
Unfortunately this problem is NP-Hard (Cook et al., 1998).
However, it is possible to obtain a 2 − OPT solution efﬁciently using a greedy approach shown in Algorithm 1.
If OPT = min s 1 max i min j ∈ s 1 ∪ s 0 ∆( x i , x j ) , the greedy algorithm shown in Algorithm 1 is proven to have a solution ( s 1 ) such that; max i min j ∈ s 1 ∪ s 0 ∆( x i , x j ) ≤ 2 × OPT .
Although the greedy algorithm gives a good initialization, in practice we can improve the 2 − OPT solution by iteratively querying upper bounds on the optimal value.
In other words, we can design an algorithm which decides if OPT ≤ δ .
In order to do so, we deﬁne a mixed integer program (MIP) parametrized by δ such that its feasibility indicates min s 1 max i min
j ∈ s 1 ∪ s 0 ∆( x i , x j ) ≤ δ .
A straight-forward algorithm would be to use this MIP as a sub-routine and performing a binary search between the result of the greedy algorithm and its half since the optimal solution is guaranteed to be included in that range.
While constructing this MIP, we also try to handle one of the weaknesses of k-Center algorithm, namely robustness.
To make the k-Center problem robust, we assume an upper limit on the number of outliers Ξ such that our algorithm can choose not to cover at most Ξ unsupervised data points.
This mixed integer program can be written as:
[Equation]
In this formulation, u i is 1 if the i th data point is chosen as center, ω i,j is 1 if the i th point is covered by the j th , point and ξ i,j is 1 if the i th point is an outlier and covered by the j th point without the δ constraint, and 0 otherwise.
And, variables are binary as u i , ω i,j , ξ i,j ∈ { 0 , 1 } .
We further visualize these variables in a diagram in Figure 2, and give the details of the method in Algorithm 2.
Algorithm 2 Robust k-Center
Input: data x i , existing pool s 0 , budget b and outlier bound Ξ
Initialize s g = k-Center-Greedy( x i , s 0 , b )
[Equation]
∆
[Equation]
[Equation]
lb =
[Equation] [Equation]
=
[Equation]
repeatif Feasible ( b, s 0 , lb + ub 2 , Ξ) then
[Equation]
else
[Equation]
end if until ub = lb return { i s.t.
u
i = 1 }
4. I MPLEMENTATION D ETAILS
One of the most important design choices is the distance metric ∆( · , · ) .
We use the l 2 distance between activations of the ﬁnal fully-connected layer as the distance.
For weakly-supervised learning, we used Ladder networks (Rasmus et al., 2015) and for all experiments we used VGG-16 (Simonyan & Zisserman, 2014) as the CNN architecture.
We optimized all models using RMSProp with a learning rate of 1e − 3 using Tensorﬂow (Abadi et al., 2016).
We train CNNs from scratch after each iteration.
While implementing our algorithm, we used the Gurobi (Inc., 2016) framework for checking feasibility of the MIP deﬁned in (6).
As an upper bound on number of outliers, we used Ξ = 1e − 4 × n where n is the number of unlabelled points.
[Figure] [Figure]
i
[Figure]
t
i
[Figure]
E XPERIMENTAL R ESULTS
We tested our algorithm on the problem of classiﬁcation using three different datasets.
We performed experiments on CIFAR (Krizhevsky & Hinton, 2009) and Caltech-256 (Grifﬁn et al., 2007) datasets for image classiﬁcation and on SVHN(Netzer et al., 2011) dataset for digit classiﬁcation.
CIFAR (Krizhevsky & Hinton, 2009) dataset has two tasks; one coarse-grained over 10 classes and one ﬁne-grained over 100 classes.
We performed experiments on both.
We compare our method with the following baselines: i ) Random:
Choosing the points to be labelled uniformly at random from the unlabelled pool.
ii ) Best Empirical Uncertainty:
Following the empirical setup in (Gal et al., 2017), we perform active learning using max-entropy, BALD and
Variation Ratios treating soft-max outputs as probabilities.
We only report the best performing one for each dataset since they perform similar to each other.
iii )
Bayesian Active Learning: We perform Monte Carlo dropout to obtain improved uncertainty measures and report only the best performing acquisition function among max-entropy, BALD and Variation Ratios for each dataset.
iv )
Best Oracle Uncertainty:
We also report a best performing oracle algorithm which uses the label information for entire dataset.
We replace the uncertainty with l ( x i , y i , A s 0 ) for all unlabelled examples.
We sample the queries from the normalized form of this function by setting the probability of choosing the i th point to be queried as p i
=
l ( x i ,y i ,A s 0 ) (cid:80) j l ( x j ,y j ,A s 0 ) .
v ) CEAL (Wang et al., 2016):
CEAL (Wang et al., 2016) is a weakly-supervised active learning method proposed speciﬁcally for CNNs.
we include it in the weakly-supervised analysis.
We conducted experiments on active learning for fully-supervised models as well as active learning for weakly-supervised models.
In our experiments, we start with small set of images sampled uniformly at random from the dataset as an initial pool.
The weakly-supervised model has access to labeled examples as well as unlabelled examples.
The fully-supervised model only has access to the labeled data points.
We run all experiments with ﬁve random initializations of the initial pool of labeled points and use the average classiﬁcation accuracy as a metric.
We plot the accuracy vs the number of labeled points.
We also plot error bars as three standard deviations.
We run the query algorithm iteratively; in other words, we solve the discrete optimization problem min s k +1 : | s k +1 |≤ b E x ,y ∼ p
Z
[ l ( x , y ;
A s 0 ∪ ..., s k +1 )]
for each point on the accuracy vs number of labelled examples graph.
We present the results in Figures 3 and 4.
Figures 3 and 4 suggests that our algorithm outperforms all other baselines in all experiments; for the case of weakly-supervised models, by a large margin.
We believe the effectiveness of our approach in the weakly-supervised case is due to the better feature learning.
Weakly-supervised models provide better feature spaces resulting in accurate geometries.
Since our method is geometric, it performs signiﬁcantly better with better feature spaces.
We also observed that our algorithm is less effective in CIFAR-100 and Caltech-256 when compared with CIFAR-10 and SVHN.
This can easily be explained using our theoretical analysis.
Our bound over the core-set loss scales with the number of classes, hence it is better to have fewer classes.
[Figure] [Figure]
[Table]
[Figure]
Our results suggest that both oracle uncertainty information and Bayesian estimation of uncertainty is helpful; however, they are still not effective in the batch setting.
We believe this is due to the correlation in the queried labels as a consequence of active learning in batch setting.
We further investigate this with a qualitative analysis via tSNE (Maaten & Hinton, 2008) embeddings.
We compute embeddings for all points using the features which are learned using the labelled examples and visualize the points sampled by our method as well as the oracle uncertainty.
This visualization suggests that due to the correlation among samples, uncertainty based methods fail to cover the large portion of the space conﬁrming our hypothesis
Optimality of the k-Center Solution: Our proposed method uses the greedy 2-OPT solution for the k-Center problem as an initialization and checks the feasibility of a mixed integer program (MIP).
We use LP-relaxation of the deﬁned MIP and use branch-and-bound to obtain integer solutions.
The utility obtained by solving this expensive MIP should be investigated.
We compare the average run-time of MIP 1 with the run-time of 2-OPT solution in Table 1.
We also compare the accuracy obtained with optimal k-Center solution and the 2-OPT solution in Figure 6 on CIFAR-100 dataset.
As shown in the Table 1; although the run-time of MIP is not polynomial in worst-case, in practice it converges in a tractable amount of time for a dataset of 50k images.
Hence, our algorithm can easily be applied in practice.
Figure 6 suggests a small but signiﬁcant drop in the accuracy when the 2-OPT solution is used.
Hence, we conclude that unless the scale of the dataset is too restrictive, using our proposed optimal solver is desired.
Even with the accuracy drop, our active learning strategy using 2-OPT solution still outperforms the other baselines.
Hence, we can conclude that our algorithm can scale to any dataset size with small accuracy drop even if solving MIP is not feasible.
C ONCLUSION
We study the active learning problem for CNNs.
Our empirical analysis showed that classical uncertainty based methods have limited applicability to the CNNs due to the correlations caused by batch sampling.
We re-formulate the active learning problem as core-set selection and study the core-set problem for CNNs.
We further validated our algorithm using an extensive empirical study.
Empirical results on three datasets showed state-of-the-art performance by a large margin.
A P ROOF FOR L EMMA
Proof.
We will start with showing that softmax function deﬁned over C class is √ C − 1 C -Lipschitz continuous.
It is easy to show that for any differentiable function f : R n → R m , [Equation] x
[Equation] R n where (cid:107) J (cid:107) ∗ F = max x (cid:107) J (cid:107) F and J is the Jacobian matrix of f .
Softmax function is deﬁned as
[Equation]
For brevity, we will denote f i ( x ) as f i .
The Jacobian matrix will be,
[Equation]

Now, Frobenius norm of above matrix will be,
[Equation]
It is straightforward to show that f i = 1 C is the optimal solution for (cid:107) J (cid:107) ∗ F = max x (cid:107) J (cid:107) F Hence, putting f i = 1 C in above equation , we get (cid:107) J (cid:107) ∗ F = √ C − 1 C .
Now, consider two inputs x and ˜x , such that their representation at layer d is x d and ˜x d .
Let’s consider any convolution or fully-connected layer as x dj =
(cid:80) i w di,j x d − 1 i .
If we assume, (cid:80) i | w i,j | ≤ α ∀ i, j, d , for any convolutional or fully connected layer, we can state:
[Equation]
On the other hand, using | a − b | ≤ | max(0 , a ) − max(0 , a ) | and the fact that max pool layer can be written as a convolutional layer such that only one weight is 1 and others are 0 , we can state for
ReLU and max-pool layers,
[Equation]
Combining with the Lipschitz constant of soft-max layer,
[Equation]
B P ROOF FOR P ROPOSITION
Before starting our proof, we state the Claim 1 from Berlind & Urner (2015).
Fix some p, p (cid:48) ∈ [0 , 1] and y (cid:48) ∈ { 0 , 1 } .
Then,
[Equation]
Proof.
We will start our proof with bounding E y i ∼ η ( x i )
[ l ( x i , y i , A s )] .
We have a condition which states that there exists and x j in δ ball around x i such that x j has 0 loss.
E
[Equation]
A
[Equation]
k
[Equation]
C
[Equation]
A
[Equation]
e
[Equation]
With abuse of notation, we represent { y i = k } ∼ η k ( x i ) with y
i ∼ η k ( x i ) .
We use Claim 1 in ( d ) , and Lipschitz property of regression function and bound of loss in ( d ) .
Then, we can further bound the remaining term as;
[Equation]
where last step is coming from the fact that the trained classiﬁer assumed to have 0 loss over training points.
If we combine them,
[Equation]
We further use the Hoeffding’s Bound and conclude that with probability at least 1 − γ ,
[Equation]