B ETTER O PTIMIZATION C AN R EDUCE S AMPLE C OMPLEXITY : A CTIVE S EMI -S UPERVISED L EARNING VIA C ONVERGENCE R ATE C ONTROL
Anonymous authors
Paper under double-blind review
A BSTRACT
Reducing the sample complexity associated with deep learning (DL) remains one of the most important problems in both theory and practice since its advent.
Semisupervised learning (SSL) tackles this task by leveraging unlabeled instances which are usually more accessible than their labeled counterparts.
Active learning (AL) directly seeks to reduce the sample complexity by training a classiﬁcation network and querying unlabeled instances to be annotated by a human-in-the-loop.
Under relatively strict settings, it has been shown that both SSL and AL can theoretically achieve the same performance of fully-supervised learning (SL) using far less labeled samples.
While empirical works have shown that SSL can attain this beneﬁt in practice, DL-based AL algorithms have yet to show their success to the extent achieved by SSL.
Given the accessible pool of unlabeled instances in pool-based AL, we argue that the annotation efﬁciency brought by AL algorithms that seek diversity on labeled samples can be improved upon when using SSL as the training scheme.
Equipped with a few theoretical insights, we designed an AL algorithm that rather focuses on controlling the convergence rate of a classiﬁcation network by actively querying instances to improve the rate of convergence upon inclusion to the labeled set.
We name this AL scheme convergence rate control (CRC), and our experiments show that a deep neural network trained using a combination of CRC and a recently proposed SSL algorithm can quickly achieve high performance using far less labeled samples than SL.
In contrast to a few works combining independently developed AL and SSL (ASSL) algorithms, our method is a natural ﬁt to ASSL, and we hope our work can catalyze research combining AL and SSL as opposed to an exclusion of either.
I NTRODUCTION
The data-hungry nature of supervised deep learning (DL) algorithms has spurred interest in active learning (AL), where a model can interact with a dedicated annotator and request unlabeled instances to be labeled.
In the pool-based AL setting, a model initially has access to a set of unlabeled samples and can query instances which need be labeled for training.
Under certain conditions on the task,
AL can provably achieve up to exponential improvement in sample complexity and thus has great potential for reducing the number of labeled instances required to achieve high accuracy.
This is especially important when the annotation task is extremely costly, for example, in medical imaging where only highly-specialized experts can diagnose a subject’s condition.
Active learning algorithms have been extensively explored, with various formulations including uncertainty-based sampling (Wang & Shang, 2014), aligning the labeled and unlabeled distributions (Gissin & Shalev-Shwartz, 2019) with connections to domain adaptation (Ben-David et al., 2010), and coreset (Sener & Savarese, 2018).
Furthermore, there is no standard method in modeling a deep neural network’s (DNN) uncertainty, and uncertainty-based AL has its own variants ranging from utilizing Bayesian networks (Kirsch et al., 2019) to using a model’s predictive conﬁdence (Wang & Shang, 2014).
This ambiguous characterization of how much information a sample’s label carries also motivated AL algorithms based on maximizing the expected change of a classiﬁcation model (Huang et al., 2016; Ash et al., 2020).
While AL comes with optimistic potentials, most algorithms outperform random sampling (passive learning) by only a small margin, with follow-up works (Gissin & Shalev-Shwartz,
; Sener & Savarese, 2018; Ducoffe & Precioso, 2018) reporting worse performance of certain AL algorithms than random sampling due to their dependency on speciﬁc model architectures or dataset characteristics.
Furthermore, the performance of AL algorithms are usually reported by training a model using supervised learning (SL) on the queried labeled data despite the availability of unlabeled data in pool-based AL.
Semi-supervised learning (SSL) has recently shown impressive performance with a small number of labeled instances, and its most premature variant known as pseudo-labeling
(Lee, 2013) has been combined with AL algorithms (Wang et al., 2017).
One recent work (Song et al., 2019) uses a rather modern SSL algorithm (Berthelot et al., 2019) and shows the strength of combining AL with SSL which we name ASSL.
However, their AL algorithm is not designed speciﬁcally considering the ASSL setting.
In this work, we propose a novel query strategy which naturally blends in with SSL, and show that our AL algorithm can rapidly achieve the high performance of fully-supervised algorithms using fewer labeled data.
Our algorithm is inspired by recent developments in DNN theory, namely the neural tangent kernel (NTK)
(Jacot et al., 2018).
Experimental comparisons with diversity-seeking strategies and an algorithm with an objective similar to ours demonstrate how labeling instances based on our objective helps SSL attain high performance in a sample-efﬁcient manner.
B ACKGROUND T HEORY 2. S AMPLE E FFICIENCIES OF A CTIVE L EARNING AND S EMI -S UPERVISED L EARNING
Here we informally describe some theoretical results describing the superiority of AL to passive learning and SSL to SL in terms of labeled sample complexity, that is, the number of labeled instances that can be used to attain (cid:15) -classiﬁcation error.
Because pool-based AL subsumes unlabeled instances, it makes much more sense to perform SSL on the readily available unlabeled instances when training a classiﬁcation network between each query iteration (Alg. 1).
This section describes how AL and SSL algorithms can consider different objectives to attain higher accuracy in the ASSL setting.
It is well known that SL can ﬁnd an (cid:15) -optimal classiﬁer from a sufﬁciently rich class of hypotheses (e.g. classiﬁers which can be realized by deep neural networks) using ˜Θ (cid:0) 1 (cid:15) (cid:1) i.i.d samples for the separable case and ˜ Θ (cid:0) 1 (cid:15) 2 (cid:1) in the agnostic case (Massart & N´ed´elec, 2006; Vapnik & Izmailov, 2015) 1 .
In contrast, actively adding instances to the (labeled) training set can sometimes improve the sample complexity sub-exponentially from O (1 /(cid:15) ) to O ( poly log 1 /(cid:15) )
(Balcan et al., 2010).
When unlabeled instances are drawn i.i.d., Balcan & Urner (2016) showed how labeling samples selected via binary search over ˜ O
(1 /(cid:15) ) unlabeled instances can achieve exponential improvement over passive learning.
G¨opfert et al.
(2019) constructed a few examples which show how an SSL algorithm can use unlabeled instances to signiﬁcantly improve the labeled sample complexity for a rather restricted class of data distributions.
A corollary of one such example is that using O (log 1 /(cid:15) ) labeled samples and O (1 /(cid:15) 2 ) unlabeled samples can be used to obtain (cid:15) -error.
Considering a target error (cid:15)
≈ 6% achieved by fully-supervised learning in CIFAR10 using O (1 /(cid:15) )
≈ 50 , 000 labeled samples, a training set with only O ( poly log(1 /(cid:15) ))
≈ 250 labeled samples and leaving the remaining O (1 /(cid:15) )
≈ 49 , 750 samples unlabeled is analogous to the aforementioned sample complexities achievable in the respective AL and SSL settings, although their assumptions may not be satisﬁed.
Instead of focusing on the same objective for both AL and SSL to attain the potential exponential sample complexity improvements above, we suggest using an AL algorithm that queries instances to help optimization.
Speciﬁcally, we would like an AL algorithm to construct a labeled dataset such that a classiﬁer f ( T ) trained for T iterations rapidly minimizes the optimization error | f ( T ) − f ( ∞ ) | , and formulate an SSL objective L n ( f ) to guide the target classiﬁer f ( ∞ ) ∈ arg
min
f L n ( f ) towards a solution that minimizes | f ( ∞ ) − f ∗ | , where f ∗ ∈ arg min f E
[ L n ( f )] is the optimal classiﬁer and L n is the empirical loss computed over n samples.
This perspective spans an online setting where less training iterations implies better generalization, i.e. f ( ∞ ) =
f ∗ , and AL focusing on | f ( T ) − f ( ∞ ) | effectively minimize the optimization-generalization error | f ( T ) − f ∗ | . Together, the resulting model would ultimately minimize the generalization error’s upper bound: | f ( T ) − f ∗ | ≤ | f ( T ) − f ( ∞ ) | + | f ( ∞ ) − f ∗ | .
The next section shows how we can carefully design labeled and unlabeled sets such that a classiﬁer trained on this construction enjoys a fast rate of convergence by employing a recently developed tool known as the NTK.
2. N EURAL T ANGENT K ERNEL
The neural tangent kernel (NTK) (Jacot et al., 2018) is a theoretical tool developed to understand how overparametrized networks can be well-optimized and generalize to unseen data.
In particular, the training dynamics of DNNs parameterized by θ t at any iteration t follows a closed form expression
[Equation]
when minimizing the mean-squared error (MSE) loss using gradient descent, where ˆ K ( t ) ( X , X ) = (cid:2) ∇ θ f θ t ( x ) T ∇ θ f θ t ( x (cid:48) )
(cid:3) ( x,x (cid:48) )
∈X×X is the empirical NTK (Arora et al., 2019), where each element ∇ θ f θ t ( x ) T ∇ θ f θ t ( x (cid:48) )
∈ R C × C is a matrix block and C is the number of classes.
If we let the width of all layers grow to inﬁnity, a remarkable property holds in which the training dynamics can be characterized using a matrix independent of time, namely the NTK K :
[Equation]
with probability 1 over random initialization 2 .
For any network of ﬁnite width d trained for t iterations, the relation (2) holds with probability ≥ 1 − δ d , where δ d → 0 as d → ∞ .
Assuming a network that satisﬁes the differential equation, the transient solution is given by
[Equation]
for some constants c i and eigenvalue/vector pairs ( λ i , v i ) of the NTK K ( X , X ) .
Notice how the
NTK, and in turn the convergence rate, depends on the set of instances
X it is computed over.
This sparks the intuition that the rate of convergence can be controlled by carefully designing a labeled set.
A CTIVE L EARNING VIA C ONVERGENCE R ATE C ONTROL 3. C ONVERGENCE R ATE C ONTROL
Inspired by the aforesaid NTK’s characterization of the model’s training dynamics when supervised on the dataset it is computed over, we propose to move unlabeled instances X ∗ u selected from a 2 Formally, the dynamics df θt ( X ) dt converges to the RHS in probability as the network’s width grows to inﬁnity initialized He initialization. pool of unlabeled samples X U that maximize the rate of convergence of the SGD dynamics to the labeled set X L , hence the name convergence rate control (CRC).
While Eq.
3 is a solution to the time-inhomogeneous ODE in Eq.
2, the performance of inﬁnitely-wide networks fall behind their ﬁnite-width counterparts.
We thus replace the eigen-pair ( λ, v ) of the true NTK K with its empirical
NTK ˆ K ( T ) in Eq.
1 using a model trained for T epochs until its validation accuracy saturates.
Maximizing the eigenvalues of the NTK is a multi-objective problem, and CRC takes a worst-case approach to select unlabeled instances that maximize the smallest eigenvalue of the NTK:
[Equation]
Each block ˆ K ( T ) ( x, x (cid:48) )
∈ R C × C is replaced by its trace computed as (cid:80)
c
ˆ K ( T ) cc ( x, x (cid:48) ) , which we numerically found to be an accurate replacement to the block matrices in computing the minimum eigenvalue, to reduce both computational and memory complexities.
In general, the eigenvalue distribution is not necessarily supported over positive values, and we observed that considering only positive eigenvalues consistently results in higher performance than when considering both negative and positive eigenvalues.
CRC thus scores unlabeled instances as the minimum over positive eigenvalues.
The solution to Eq.
4 can then be found using dedicated GPU-implementations available in popular DL frameworks such as PyTorch.
This algorithm is detailed in Alg. 2.
A labeled sample’s inﬂuence on a DNN’s predictive behavior is abstruse.
Intuitively, a labeled sample may be most informative when it is near a region separating different classes; however, this does not guarantee that a DNN learns such a decision boundary effectively.
Furthermore, it is unclear whether a sample in some other region could better enhance a classiﬁer’s performance when added to the labeled set, especially when considering SSL algorithms which complicate DL analysis.
Contrary to the above intuition, Rebufﬁ et al.
(2019) demonstrates how a DNN trained with a SSL objective (Laine & Aila, 2017) learns a decision boundary using only 4 manually selected pivot points not on the class-separating region.
Figure 1 illustrates how the classiﬁer’s decision boundary progressively forms near low-density regions as CRC enlarges the labeled set. 3. T HEORETICAL A NALYSIS
This section theoretically analyzes the effect of estimating the dynamics using the empirical NTK to construct a labeled set for least squares regression (cid:96) ( θ ) = 12 (cid:107) f θ ( X ) − Y(cid:107) 22 .
We begin with a deviation bound of the empirical NTK from (Arora et al., 2019).
Theorem 1 (Paraphrased Theorem 3.1 in (Arora et al., 2019)) . Consider a fully-connected network with ReLU nonlinearities of width d and depth L . At initialization, taking the form speciﬁed in (Arora et al., 2019), the following holds with probability ≥ 1 − δ [Equation]
The above statement shows a general case in which the empirical NTK can be expressed as a linear combination of the true NTK and a noise matrix parametrized by (cid:15) > 0 , as long as the empirical
NTK does not change abruptly.
A similar result holds for ResNets and CNNs given appropriate conditions (Lipschitz, smooth, non-polynomial, analytic) on non-linearities and initialization (Du et al., 2019), where it has also been shown that ˆ K ( t ) provably stays close to K (0) throughout training.
The next proposition gives a tractable approximation of the true NTK’s (unknown) eigenvalue as a neighborhood of the empirical NTK’s (known) eigenvalue.
Proposition 1 (CRC approximation error due to empirical NTK) .
Suppose K = ˆ K ( t ) + N ( t ) with a symmetric noise matrix of bounded spectral norm ρ (cid:0) N ( t ) (cid:1) < (cid:15) .
Then, the true NTK’s eigenvalues lies in an (cid:15) -neighborhood of the empirical NTK’s eigenvalues with respect to (cid:107)·(cid:107) ∞ , i.e. λ ( K ) ∈ B (cid:15) (cid:16) λ (cid:16)
ˆ K ( t ) (cid:17)(cid:17) , where the eigenvalues λ ( · ) are listed monotonically.
This statement has nice implications on optimizing DNNs.
Consider Theorem 5.1 in (Du et al., 2019) that states running gradient descent with learning rate η
[Equation]
O
[Equation]
(cid:17) on leastsquares regression using a fully-connected network of sufﬁcient width converges linearly at rate 1 − ηλ min ( K ( X L , X L )).
Two remarks follow:
Remark 1 (Better conditioning of learning rates) .
The sufﬁcient condition on learning rate is proportional to the (unknown) minimum eigenvalue of the NTK.
This implies that the increased minimum eigenvalue allows for a larger set of admissable learning rate schedules.
In (Athiwaratkun et al., 2019), it has been shown that SSL algorithms take larger steps throughout training, continually exploring parameters.
Since CRC permits larger learning rates, tuning the step-size may be easier and the larger steps taken by SSL algorithms can still converge when the step-size is over-estimated.
Remark 2 (Direct estimate of sufﬁcient learning rate and convergence rate guarantees) .
Both the admissable learning rates and convergence rate are unknown.
Invoking Proposition 1 after selecting
X L using CRC with ˆ K ( T ) , we can set
[Equation]
(cid:19) which would guarantee linear convergence with the rate deviating at most by the spectral norm (cid:15) of the noise matrix.
3. P RACTICAL C ONSIDERATIONS
Summing the network’s gradients over all layers for each instance pair is infeasible for modern network architectures, and we compare one realization of using all parameters with the ﬁnal layer’s to compute the NTK.
A comparison of one realization using the NTK computed over all layers vs. using just the ﬁnal layer averaged over 3 trials (Tab. 1) shows how using the ﬁnal layer performs just as well as when all layers were used but with much faster ( × 100 ) queries.
If we assume that the shallow layers saturate more quickly than deeper layers, justiﬁed by the ‘lower layers learn ﬁrst’ phenomenon (Raghu et al., 2017), the NTK estimate computed at the network’s last layer is a prediction of the training dynamics corresponding to a single layer perceptron attached to a (slowly-changing) feature extractor.
In this perspective, the ﬁnal accuracy is mostly affected by the ﬁnal layer which determines how high-level features are recognized, and designing a labeled set using the NTK estimated over the last layer would sufﬁciently capture the model’s training dynamics.
[Table]
One reason why estimating the training dynamics over all layers could be worse than using the last is that the former requires a network trained in the next SSL phase to learn a similar low-level representation to that obtained in the previous SSL phase.
However, it was observed by Raghu et al.
(2017) that two randomly initialized networks trained on the same dataset have similar ﬁnal layers but extremely different intermediate representations.
Because our networks are retrained from scratch between each query iteration, it is highly likely that the next network will have very different intermediate representations, and these middle layers would provide noisy estimates as to how much the model’s functional representation would be affected by the enlarged dataset.
Since we are seeking to modify a dataset in order to amplify a model’s performance, it is reasonable that we enlarge the labeled set based on how much the additional samples would affect the model’s ﬁnal layers.
Had we designed the experiments such that a model is continuously trained from its previous SSL phase, computing the NTK over all layers may have been beneﬁcial at the expense of computation.
3. M YOPIC VS . B ATCH -M ODE A CTIVE L EARNING
Myopic AL policies refer to algorithms that score unlabeled instances based on their individual importance.
Considering how an expert annotator ‘in-the-loop’ must label queried samples, it is desirable that an AL algorithm queries large batches based on their collective importance.
Myopic policies usually degrade in performance as the query size Q increases, as they may repetitively sample identical instances (Kirsch et al., 2019), and it has been of great interest in designing a computationally efﬁcient batch-AL algorithm.
CRC (Alg. 2) is a batch-mode AL algorithm
as it computes λ min over groups of unlabeled instances X u .
To cope with the combinatorial complexity associated with the outer-maximization in Eq. 4, we randomly sample candidate groups X u without for G do s empirical NTK computed ﬁnal layers
∪ · · · ∪ X ( t ∗ Q G ) u replacement. This randomization signiﬁcantly reduces the search space, potentially yielding highly sub-optimal solutions, and we introduce a hyperparameter G such that G divides Q , and search over candidate groups |X u
| = G , selecting Q/G groups at each query step.
The group size controls whether the outer maximization is solved more accurately ( G = 1 ) without considering the collective signiﬁcance of a group, or the inner-minimization incorporates the collective importance of all queried samples ( G = Q ) at the expense of a substantially smaller search space.
This trade-off is established in Tab. [-
-]2 when Q = 100 , averaged over three realizations for G = 2 , 5 , 10 , 100 .
Our relatively small group-sizes make the cubic complexity of the inner-minimization negligible, and a larger group size is consequently faster due to less ( Q → Q/G ) eigen-decomposition operations and inefﬁcient batch-gradient computations in autograd packages.
All experiments hereon use G = Q/considering the speed vs. performance trade-off. [Table] [Table] =84.72 ±
[Table]
87.11 ±
[Table]
[Table]
±
[Table]
[Table]
±
[Table]
89.44 ±
[Table]
[Table]
=82.55 ±
[Table]
[Table]
3. R ELATION TO G RADIENT - BASED M ETHODS
The proposed algorithm can be interpreted as measuring the importance of unlabeled instances by how much they would change a model’s parameters.
Two AL algorithms are similar in this manner in that they seek to label instances which affect the model the most: expected gradient length (EGL) (Huang et al., 2016) and
BADGE (Ash et al., 2020).
EGL ﬁrst computes the gradient of the cross-entropy loss over all unlabeled instances, replacing the ground truth target with an expectation over a posterior distribution determined by the trained model, then selects instances with the greatest norm.
BADGE uses an embedding of the loss’ gradients using predicted classes, computed over the parameters in just the ﬁnal layer, and uses kmeans++ to enforce diversity.
Nonetheless, both algorithms only estimate the beneﬁt due to candidate unlabeled samples and do not measure the collective inﬂuence of the labeled set.
E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Our SSL implementation is primarily based on different publicly-available PyTorch implementations of FixMatch (Sohn et al., 2020) for CIFAR10 3 and CIFAR100 4 , as well as mean teacher (Tarvainen & Valpola, 2017) for SVHN 5 that performed best at the time of this work. We made modiﬁcations as
[Table] suggested in (Oliver et al., 2018).
In particular, we used the WRN-28-2 and WRN-28-10 architectures for SVHN/CIFAR10 and CIFAR100, respectively, further motivated by the derivation of NTK as a network in its inﬁnite width limit.
Mean teacher (MT) enforces consistency between the model trained using SGD and an exponential moving average (teacher) of the model’s weights.
FixMatch is a variant of MixMatch (Berthelot et al., 2019), replacing standard augmentations with more powerful augmentation techniques when computing the SSL loss.
In short, FixMatch enforces consistency between a classiﬁcation model’s predictions on perturbed inputs which effectively yields a more robust model with small Jacobian norm (Athiwaratkun et al., 2019) in junction with other SSL algorithms including MT.
Furthermore, the hyperparameters in FixMatch were kept the same across different dataset sizes in their original report, making our experimental comparisons convenient.
The initial pool of labeled instances signiﬁcantly affected the downstream SSL task and subsequent
AL queries, especially for uncertainty-based algorithms (see section 4.2).
It is practical to assume that a model would initially have access to a small pool of labeled images with an equal number of classes.
Thus, we randomly sampled a balanced number of images per class in the ﬁrst query iteration and use the AL algorithms in subsequent queries without enforcing balanced classes.
Between each query step, the classiﬁcation network was trained until either its validation accuracy does not improve for 50/100/ ∞ epochs or 350/600/512 epochs is reached for CIFAR10/100/SVHN.
All experiments are repeated 3 times with mean ± std performances reported unless stated otherwise.
Our implementation will be open-sourced upon publication.
4. B ASELINE A LGORITHMS
Both AL and SSL algorithms demand much more time than standard supervised learning, and an extensive comparison of all ASSL combinations is beyond our compute availability.
We resort to comparing the proposed algorithm with 3 other AL algorithms, 2 of which scores unlabeled instances based on how much they reduce a model’s uncertainty, and the other because of its similarity to CRC.
Uncertainty-based AL (Wang & Shang, 2014) is one of the simplest DL-based
AL algorithm that selects the unlabeled instance with lowest conﬁdence in its prediction: X ∗ u = arg min { x ∈X U } max y f θ,y ( x ) . In the same work, Wang & Shang (2014) proposed to also use entropy as a measure of the model’s uncertainty in unlabeled samples.
EGL (Huang et al., 2016) retrieves instances that modify a model the most: X ∗ u = arg max { x ∈X U } E y ∼ ˆ p ( · ; x,θ ) (cid:2) (cid:107)∇ θ H ( y, f θ ( x ))
(cid:107) 2 (cid:3) where ˆ p ( y ; x i , θ ) =
f θ,y ( x i ) is the model’s conﬁdence in class y and H is the cross entropy function.
4. A CTIVE S EMI -S UPERVISED L EARNING
Here we report the sample efﬁciencies obtained by different SSL algorithms when the labeled set is constructed using various AL schemes in Tabs. 3 and 4.
For reference, the implementations used for each experiment achieve average accuracies of SVHN (100/class): 93.52%, CIFAR10 (25/class): 87.33, and CIFAR100 (40/class): 48.96% with balanced sampling. Note, however, that SSL settings consider labeled sets (and consequently, unlabeled sets) with equal numbers of images per class , putting SSL settings to an unrealistic advantage at the same number of labeled images. Different levels of class-imbalance result in larger performance variance, which is especially critical in the low-data regime.
One difﬁculty arising from the ASSL setting is class imbalance.
In contrast to standard AL settings that consider much larger query sizes, the small query size in ASSL signiﬁcantly impacts the class distribution, resulting in occasional performance drop for certain query strategies.
Entropy has repeatedly achieved state-of-the-art performance (Gissin & Shalev-Shwartz, 2019; Wang et al., 2017;
Ash et al., 2020) in AL settings despite its simplicity, and our experiments show the same holds for ASSL.
Conforming with our motivation in section 2.1, CRC performed comparable or better
[Table]
than entropy although it does not explicitly enforce diversity on the labeled set.
Furthermore, while both CRC and EGL scores instances by estimating how much they would affect the model’s training dynamics, only CRC outperformed random sampling.
Our results suggest that actively enlarging the labeled set can deﬁnitively reduce the labeled sample complexity in the SSL setting, without having to construct a labeled set consisting of images representative of their class.
Existing DL-based AL algorithms have performed comparably to random sampling when using large query sizes, and our experiments show the same for entropy and CRC with Q = 100 in the ASSL setting.
A proper AL algorithm should achieve higher performance at the same number of labeled samples if the query size is small since it would be taking more query steps in order to claim its superiority to random sampling; however, we observed that only CRC satisﬁes this property.
EGL’s low performance had been reported on image classiﬁcation tasks (Gissin & Shalev-Shwartz, 2019;
Sener & Savarese, 2018; Ducoffe & Precioso, 2018) under the AL setting.
Our experimental design of retraining networks from scratch may have had negative inﬂuence on EGL, similar to how it affected the NTK estimate over all layers.
We attribute conﬁdence’s low performance to the poor calibration of DNNs (Guo et al., 2017), inadequately capturing the model’s uncertainty on unlabeled instances.
A successful application of conﬁdence-based query may require incorporating calibration while training classiﬁcation networks or before performing such queries.
Ideally, an AL algorithm should have a nice transferrability property, constructing labeled sets that are useful to train other networks. This obviously holds for random sampling, as the queries do not depend on the model’s weights. To validate the transferrability of dataset constructions of AL algorithms, we used a randomly initialized model to query only the initial pool, trained the model using a different realization of initialization, and as done previously for other experiments, queried additional labels using the model trained using SSL.
As shown in Fig.
2, however, such initial queries adversely affected all subsequent queries.
While entropy and CRC overcome this initial disadvantage after a few query iterations, other baselines tend to under-perform random sampling due to this initial disadvantage.
This phenomenon has not been reported in previous works, and whether this is an artifact of our ASSL setting or the architecture/hyperparameters used deserves future notice.
4. C ONVERGENCE R ATE AND T IME H ORIZON P REDICTION
The proposed CRC algorithm is designed to query unlabeled instances that improve the rate of convergence a DNN in the following SSL phase.
To see if our algorithm achieves this purpose, we plot the empirical NTK’s minimum eigenvalues used to score unlabeled instance groups against the number of training epochs until the test loss reaches its global minimum in Fig.
3 (a) to show how the minimum eigenvalues are predictive of the time horizon.
Global minimum here refers to the value for which the loss never falls below until training completes.
We additionally plot how the minimum eigenvalues’ distribution progresses with query steps, where Fig.
3 (b) shows that the eigenvalue distribution concentrates towards 0 as the labeled set grows, implying slower convergence rates.
A similar observation was made by Nitanda & Suzuki (2020) where the authors showed how the true NTK’s eigenvalues concentrate towards zero as the training set size increases, whereas our plot is over a union of labeled and queried groups.
This ﬁgure illustrates how all candidate datasets will inevitably have eigenvalue distributions concentrated near zero, but the enlarged dataset chosen by CRC will have the fastest convergence rate among all considered candidates.
C ONCLUSION
This work motivated the combination of AL with SSL, using the former for optimization and latter objective to restrict the hypothesis space to those that generalize well.
We then proposed an AL algorithm inspired by the NTK, and demonstrated how controlling the convergence rate of the training dynamics can signiﬁcantly improve sample efﬁciency.
To the best of our knowledge, this work is the ﬁrst practical application of NTK outside of supervised learning, where it has been shown that the true NTK is inferior to its DNN counterpart.
In contrast to most DL-based AL experiments which use relatively small networks, this work demonstrates that DL can beneﬁt from AL using modern architectures.
Our algorithm outperformed uncertainty-based AL algorithms which enforce diversity and an AL algorithm with a similar goal of maximizing the model’s change.
The proposed algorithm’s superiority was also demonstrated in the batch-AL setting, where an optional hyperparameter can be used to control the trade-off between computational complexity and performance.
Our CRC algorithm is developed under the assumption that a well-optimized network f ( ∞ ) induced by the SSL objective is a good approximation of a model f ∗ that generalizes well.
Incorporating the interplay between AL and SSL seems promising in further reducing the sample complexity, designing either an SSL algorithm best ﬁt for some ﬁxed AL algorithm or a joint ASSL scheme.
For example,
CRC provides an estimate on the number of epochs required for convergence, and a learning rate schedule best ﬁt for the estimated time horizon could be used (Ge et al., 2019).
The proposed CRC algorithm can be extended in several directions, applying generalization guarantees characterized by the NTK to an online (streaming) setting where unlabeled instances cannot be stored.
Another exciting direction to pursue would be to use the True NTK (Arora et al., 2019; Novak et al., 2020) to estimate the training dynamics.
While CRC is motivated by controlling the transient solution using an estimated proxy of the training dynamics, the true NTK would enable an exact computation of the ODE solution.
Of utmost importance would be that this could allow AL queries to be performed ofﬂine, that is, without having to train a model between each query iteration, alleviating the long training times associated with ASSL. A A PPENDIX A. E XPERIMENTAL DETAILS
All experiments described samples equal numbers of images per class on the ﬁrst query step.
Those described as “using random sampling” uses the Random (passive) AL scheme, i.e. sampling uniformly at random, and “randomly initialized models” refers to transferrability, where the ﬁrst query step is performed using a randomly initialized model (e.g. conﬁdence of a randomly initialized model when using the “Conﬁdence” AL scheme), then performing SSL with another instance of random initialization.
When random sampling was performed, we ran several experiments and used the best models at query step 0, since we observed (similar to transferrability experiments) that the ﬁrst query step may highly inﬂuence subsequent steps, affecting the performance of different AL algorithms resulting in an unfair comparison.
This is why all experiments with ﬁrst query being random sampling (query step 0) excluded the ﬁrst query step, as they all used identical models when performing the ﬁrst actual AL query (query step 1).
All subsequent queries were performed using the trained model obtained from the previous SSL step, regardless of transferrability.
Taking “Conﬁdence” on CIFAR as an example for the AL scheme, a model trained using SSL at query step 1 is loaded to query the next 50 (when Q = 50 ) labeled samples for query step 2.
Architectures used for SVHN/CIFAR10/CIFAR100 are WRN-28-2/WRN-28-2/WRN-28-8, where the last is a known requirement due to more classes.
SVHN used leaky ReLU with negative slope 0.1 for Mean Teacher, and the others used ReLU activations.
“One epoch” is deﬁned as the number of SGD steps between each validation, which we set as 1024 training iterations (far more frequent than the default parameters for Mean Teacher on SVHN).
All other conﬁgurations were as set in the original implementations.
A. P ROOF OF P ROPOSITION
Proof.
Let N ( t ) =
K − ˆ K ( t ) and denote λ ( · ) the eigenvalues of some matrix of size n , listed in non-increasing order.
Since K and ˆ K ( t ) are Hermitian by deﬁnition, Weyl’s inequality gives λ i
[Equation]
ˆ K
[Equation]
+
[Equation]
N
[Equation]
[Equation]
≤
[Equation]
This implies
[Equation]
} and hence, | λ
[Equation]
where (cid:107) · (cid:107) 2 is the spectral norm.