L OW -B UDGET A CTIVE L EARNING VIA W ASSERSTEIN D ISTANCE : A N I NTEGER P ROGRAMMING A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
Active learning is the process of training a model with limited labeled data by selecting a core subset of an unlabeled data pool to label.
The large scale of data sets used in deep learning forces most sample selection strategies to employ efﬁcient heuristics.
This paper introduces an integer optimization problem for selecting a core set that minimizes the discrete Wasserstein distance from the unlabeled pool.
We demonstrate that this problem can be tractably solved with a Generalized Benders Decomposition algorithm.
Our strategy uses high-quality latent features that can be obtained by unsupervised learning on the unlabeled pool.
Numerical results on several data sets show that our optimization approach is competitive with baselines and particularly outperforms them in the low budget regime where less than one percent of the data set is labeled.
I NTRODUCTION
Although deep learning demonstrates superb accuracy on supervised learning tasks, most deep learning methods require massive data sets that can be expensive to label.
To address this challenge, active learning is a labeling-efﬁcient supervised learning paradigm when given a large unlabeled data set (Cohn et al., 1994; Settles, 2009).
Active learning uses a ﬁnite budget to select and label a subset of a data pool for downstream supervised learning.
By iteratively training a model on a current labeled set and labeling new points, this paradigm yields models that use a fraction of the data to achieve nearly the same accuracy as classiﬁers with unlimited labeling budgets.
The task of selecting points to label is an integer optimization problem of determining the most representative subset of the unlabeled data (Sener & Savarese, 2017).
However, to estimate the value of labeling a point, active learning strategies require informative features from the unlabeled data.
Integer optimization models can also grow impractically quickly with the data set size.
Nonetheless, optimally selecting which points to label can reduce operational costs, especially for applications such as medical imaging where labeling requires time-consuming human expert labor (Rajchl et al., 2016), or domain adaptation where data pools are hard to obtain (Saenko et al., 2010).
In this paper, we introduce an optimization framework for active learning with small labeling budgets.
To select points, we minimize the Wasserstein distance between the unlabeled set and the set to be labeled.
We prove that this distance bounds the difference between training with a ﬁnite versus an unlimited labeling budget.
Our optimization problem admits a Generalized Benders Decomposition solution algorithm wherein we instead solve sub-problems that are orders of magnitude smaller (Geoffrion, 1972).
Our algorithm guarantees convergence to a globally optimal solution and can be further accelerated with customized constraints.
Finally, to compute informative features, we use unsupervised learning on the unlabeled pool.
We evaluate our framework on four visual data sets with different feature learning protocols and show strong performance versus existing selection methods.
In particular, we outperform baselines in the very low budget regime by a large margin.
Our overall contributions are as follows: (1) We derive a new deterministic bound on the difference in training loss between using the full data set versus a subset via the discrete Wasserstein distance.
This further bounds the overall expected risk.
(2) We propose active learning by minimizing the
Wasserstein distance.
We develop a globally convergent customized algorithm for this problem using Generalized Benders Decomposition.
(3) We consider low budget active learning for image classiﬁcation and domain adaptation where up to 400 images can be labeled and show that optimally selecting the points to label in this regime improves over heuristic-based methods.
[Figure]
P ROPOSED M ODEL
We ﬁrst set up the active learning problem and develop the main theory motivating our active learning strategy illustrated in Figure 1.
All of our proofs can be found in Appendix A.
2. P RELIMINARIES OF A CTIVE L EARNING
Consider a C -class classiﬁcation problem over features x ∈ X and labels y ∈ Y :
= { 1 , · · · , C } , where ( X , (cid:107)·(cid:107) )
and ( Y , |·| ) are metric spaces.
With a data distribution and a loss function (cid:96) ( x , y ; w ) :
X × Y → R parametrized by w , our goal is to minimize the expected risk min w E [ (cid:96) ( x , y ; w )] .
In practice, we use a labeled data set { ( x i , y i ) } Ni =1 of N samples to minimize the empirical risk.
In active learning, we instead have features D =
{ x i } Ni =1 and a labeling oracle Ω : X → Y .
Let π ∈ { 0 , 1 } N indicate points to label given a budget B .
That is, we may call the oracle B times to create a labeled core set C ( π ) :
= { ( x j , Ω( x j ) } Bj =1 where x i ∈ C
( π ) is labeled iff π i = 1 .
An active learning strategy is an algorithm to optimize C ( π ) .
We omit π in the notation for C when obvious.
Active learning strategies can be categorized into uncertainty-based, representation-based, or hybrid.
Uncertainty methods determine images to label by using measures of the uncertainty of a classiﬁer (Li & Guo, 2013; Roth & Small, 2006; Wang & Shang, 2014).
Representation methods select a diverse core set of examples spread over the feature space (Sener & Savarese, 2017; Contardo et al., 2017; Yu et al., 2006).
Uncertainty estimates and latent features are commonly obtained by training a preliminary classiﬁer for feature extraction with a previously labeled data set (Sinha et al., 2019; Shui et al., 2020).
Other approaches also include pre-clustering (Nguyen & Smeulders, 2004) or few-shot learning (Woodward & Finn, 2017).
Sener & Savarese (2017) bound the expected risk by a generalization error of training with an unlimited number of oracle calls, the empirical risk of training with a core set of B points, and a core set loss for training with all points versus only the core set.
This bound, slightly revised, is
[Equation]
[Table]
The generalization bound does not depend on the core set.
Furthermore for small B , the empirical risk is always negligibly small.
Thus from Sener & Savarese (2017), we can focus on an active learning strategy that speciﬁcally minimizes the core set loss.
2. A CTIVE L EARNING VIA W ASSERSTEIN D ISTANCE
As a function of C , the core set loss resembles a distribution matching loss on the empirical risk with C versus D .
This motivates an active learning strategy of minimizing a distance between distributions over these two sets (Shui et al., 2020).
We consider minimizing the discrete Wasserstein distance (Villani, 2008).
Let D x
=
[ (cid:107) x
i − x i (cid:48) (cid:107) ] N,Ni =1 ,i (cid:48)
=1 be a distance matrix over features in D .
The discrete Wasserstein distance between C and D is denoted by W ( C ( π ) , D ) and deﬁned as
[Equation]
where (cid:104) A , B (cid:105) =
Trace ( A (cid:62) B ) is the Frobenius inner product between real matrices A and B .
We then formulate our active learning strategy as the following optimization problem:
[Equation]
We may also consider alternative distance functions such as divergences.
However, since the Wasserstein distance minimizes a transport between distributions, it can lead to better coverage of the unlabeled set (e.g., Shui et al.
(2020) demonstrate numerical examples).
Furthermore, we compute the discrete Wasserstein distance by a linear program, meaning that our active learning strategy can be written as a mixed integer linear program (MILP).
Finally, the Wasserstein distance induces a new bound on the expected risk in training.
Speciﬁcally, we show below that the Wasserstein distance directly upper bounds the core set loss under the assumption of a Lipschitz loss function.
Theorem 1.
Fix w constant.
For any ε > 0 , if (cid:96) ( x , y ; w ) is K -Lipschitz continuous over ( X ×
Y , (cid:107) x (cid:107) + ε | y | ) , then 1 N (cid:80) Ni =1 (cid:96) ( x i , Ω( x i ); w ) − 1 B (cid:80) Bj =1 (cid:96) ( x j , Ω( x j ); w ) ≤ KW ( C , D ) + εKC.
The second term in the bound vanishes for small ε .
Moreover, Theorem 1 assumes Lipschitz continuity of the model for ﬁxed parameters, which is standard in the active learning literature (Sener & Savarese, 2017) and relatively mild.
We provide a detailed discussion on the assumption in Appendix B. Finally, the bound in Theorem 1 directly substitutes into the bound of Sener & Savarese (2017), meaning that minimizing the Wasserstein distance intuitively bounds the expected risk.
M INIMIZING W ASSERSTEIN D ISTANCE VIA I NTEGER P ROGRAMMING
We now present our Generalized Benders Decomposition (GBD) algorithm for solving problem (3).
GBD is an iterative framework for large-scale non-convex constrained optimization (Geoffrion, 1972).
We provide a general review of GBD in Appendix C and refer to Rahmaniani et al.
(2017) for a recent survey.
In this section, we ﬁrst reformulate problem (3), summarize the main steps of our algorithm, and show three desirable properties: (i) it has an intuitive interpretation of iteratively using sub-gradients of the Wasserstein distance as constraints; (ii) it signiﬁcantly reduces the size of the original optimization problem; and (iii) it converges to an optimal solution.
Finally, we propose several customization techniques to accelerate our GBD algorithm.
3. M INIMIZING W ASSERSTEIN D ISTANCE WITH GBD
Problem (3), which selects an optimal core set, can be re-written as the following MILP:
[Equation]
Let us deﬁne G :=
{ Γ ≥ 0 | Γ1 = 1 /N } and P :=
{ π ∈ { 0 , 1 } N | π T 1 = B } .
Problem (4) is equivalent to the following Wasserstein-Master Problem (W-MP):
[Equation]
Although W-MP contains an inﬁnite number of constraints controlled by the dual variable λ , it can also be re-written as a single constraint by replacing the right-hand-side with the Lagrangian L ( π ) of the inner optimization problem.
Furthermore, L ( π ) is equivalent to the Wasserstein distance:
[Equation]
Main Steps of the Algorithm.
Instead of the semi-inﬁnite problem W-MP (5), we consider a ﬁnite set of constraints Λ ⊂ R N and solve a Wasserstein-Relaxed Master Problem (W-RMP( Λ )):
[Equation]
Let (ˆ η, ˆ π ) be an optimal solution to W-RMP( Λ ) and let ( η ∗ , π ∗ ) be an optimal solution to W-MP.
Since W-RMP is a relaxation, ˆ η ≤ η ∗ lower bounds the optimal value to W-MP (and thus also to problem (4)).
Furthermore, L (ˆ π ) =
W ( C (ˆ π ) , D ) ≥ W ( C ( π ∗ ) , D ) =
η ∗ upper bounds the optimal value to W-MP.
By iteratively adding new constraints to Λ , we can make W-RMP( Λ ) a tighter approximation of W-MP, and consequently tighten the upper and lower bounds.
In GBD, we initialize a selection ˆ π 0 ∈ P and Λ = ∅ .
We repeat at each iteration t ∈ { 0 , 1 , 2 , . . . }
Given a solution (ˆ η t , ˆ π t ) , solve the Lagrangian L (ˆ π t ) to obtain a primal-dual transport (ˆ Γ t , ˆ λ t ) .
Update Λ ← Λ ∪ { ˆ λ t } and solve W-RMP( Λ ) to obtain a solution (ˆ η t +1 , ˆ π t +1 ) .
We terminate when W ( C (ˆ π t ) , D ) − ˆ η t < ε is less than a tolerance ε , or up to a runtime limit.
We omit t in the notation when it is obvious.
Interpretation.
The Lagrangian is equivalent to computing a Wasserstein distance.
Furthermore, the dual variables of the Wasserstein equality constraints are sub-gradients of problem (4) (Rahmaniani et al., 2017).
In each iteration, we compute W ( C (ˆ π ) , D ) and add a new constraint to W-RMP( Λ ).
Intuitively, each constraint is a new lower bound on η using the sub-gradient with respect to ˆ π .
Solving Smaller Problems.
The original problem (4) contains N 2 + N variables and 2 N +constraints where N = |D| , making it intractable for most deep learning data sets (e.g., N = 50 ,for CIFAR-10).
W-RMP( Λ ) only contains N + 1 variables and | Λ | constraints (i.e., the number of constraints equals the number of iterations).
In our experiments, we run for approximatelyiterations, meaning W-RMP( Λ ) is orders of magnitude smaller than (4).
Although we must also solve the Lagrangian sub-problem in each iteration, computing Wasserstein distances is a wellstudied problem with efﬁcient algorithms and approximations (Bonneel et al., 2011; Cuturi, 2013).
We discuss our overall reduced complexity in more detail in Appendix D.3.
Convergence.
Finally, because the objectives and constraints of W-MP are linear and separable in Γ and π
(see Appendix C), GBD converges to an optimal solution in ﬁnite time.
Corollary 1.
Let Λ 0 = ∅ and ˆ π 0 ∈ P .
Suppose in each iteration t ∈ { 0 , 1 , 2 , . . .
} , we compute W ( C (ˆ π t ) , D ) to obtain a dual variable ˆ λ t and compute W-RMP( Λ ← Λ ∪ { ˆ λ t } ) to obtain (ˆ η t +1 , ˆ π t +1 ) .
Then for any ε > 0 , there exists a ﬁnite t ∗ for which W ( C (ˆ π t ∗ ) , D ) − ˆ η t ∗ < ε .
3. A CCELERATING THE A LGORITHM
Since W-RMP( Λ ) is a relaxation of W-MP, we can accelerate the algorithm with techniques to tighten this relaxation.
We propose two families of constraints which can be added to W-RMP( Λ ).
Enhanced Optimality Cuts (EOC).
A given inequality is referred to as an optimality cut for an optimization problem if the optimal solution is guaranteed to satisfy the inequality.
Introducing optimality cuts as additional constraints to an optimization problem will not change the optimal solution, but it may lead to a smaller feasible set.
We augment W-RMP( Λ ) with “Enhanced Optimality
Cuts” (EOC) in addition to the Lagrangian constraints at each iteration of the algorithm.
Proposition 1.
For all ˆ π ∈ P , let (ˆ Γ , ˆ λ , ˆ µ ) be optimal primal-dual solutions to W ( C (ˆ π ) , D ) in problems (1) and (2).
Let D i,i (cid:48) be elements of D x .
The following are optimality cuts for W-MP:
Lower Bound:
[Equation]
Triangle Inequality:
[Equation]
Dual Inequality:
[Equation]
Inequality (6) is derived from a lower bound on the Wasserstein distance between two probability distributions.
This is a general relationship between η and π and can be included as an additional constraint to W-RMP( Λ ) at the onset of the algorithm.
Inequalities (7) and (8) bound η and π with respect to a given ˆ π .
In each iteration, when we obtain a new ˆ π , we can construct a Triangle and
Dual Inequality, respectively, and add them to W-RMP( Λ ) along with the Lagrangian constraints.
From Proposition 1, an optimal solution to W-MP satisﬁes the inequalities (6)–(8), meaning that including them as additional constraints to W-RMP( Λ ) will still ensure that the optimal solution to
W-MP is feasible for W-RMP( Λ ) for any Λ (and therefore also optimal once Λ is sufﬁciently large).
That is, we preserve convergence from Corollary 1, while tightening the relaxation, which reduces the search space and consequently, the overall number of iterations of our algorithm.
Pruning Constraints (P).
Augmenting W-RMP( Λ ) with constraints that are not valid optimality cuts may potentially remove the global optimal solution of W-MP from the relaxed problem.
Nonetheless, such constraints may empirically improve the algorithm by aggressively removing unnecessary regions of the relaxed feasible set (Rahmaniani et al., 2017).
One simple approach is to use trust region-type constraints that prune neighbourhoods of good versus bad solutions (van Ackooij et al., 2016).
We introduce a simple set of constraints that remove search neighbourhoods around core sets with high Wasserstein distances and encourage searching near neighbourhoods around core sets with lower Wasserstein distances.
We detail these constraints in Appendix D.1.
In Appendix D.2, we provide a full algorithm description and Python pseudo-code.
Finally, note that the techniques proposed here are not exhaustive and more can be used to improve the algorithm.
C OMPARISON WITH R ELATED L ITERATURE
In this section, we compare our optimization problem and active learning strategy with the related literature in active learning, minimizing Wasserstein distances, self-supervised learning,
GBD.
Active Learning.
Our approach is closely related to Greedy k -centers (Sener & Savarese, 2017), which develops a core set bound similar to Theorem 1 but using the optimal solution to a k -center facility location problem rather than the Wasserstein distance (Wolf, 2011).
Their bound is probabilistic whereas ours is deterministic.
Furthermore since facility location is a large MILP, Sener &
Savarese (2017) use a greedy approximation to select points.
This sub-optimality can incur significant opportunity cost in settings where every point must be carefully selected so as to not waste a limited budget.
Here, GBD solves our MILP with an optimality guarantee if given sufﬁcient runtime.
Another related method is Wasserstein Adversarial Active learing (WAAL)
(Shui et al., 2020), which also proposes a probabilistic Wasserstein distance core set bound.
Their bound is proved with techniques from domain adaptation and requires a Probabilistic-Lipschitz assumption on the data distribution itself, whereas our bound uses only linear duality and a weaker assumption: Lipschitz continuity in the loss function.
To select points, Shui et al.
(2020) use a semi-supervised adversarial regularizer to estimate the Wasserstein distance and a greedy heuristic to select points that minimize the regularizer.
Our algorithm does not need to learn a custom feature space and can instead be used with any features.
Furthermore, we directly optimize the Wasserstein distance and provide a theoretical optimality gap.
Numerical results show that optimization outperforms greedy estimation in the low-budget regime and yields a small improvement even in high-budget settings.
Minimizing Wasserstein Distance.
Problem (3) has been previously studied in active learning (Ducoffe, 2018; Shui et al., 2020) and in other applications such as scenario reduction for stochastic optimization (Heitsch & R¨omisch, 2003; Rujeerapaiboon et al., 2018).
The related literature concedes that MILP models for this problem have difﬁculty scaling to large data sets, necessitating heuristics.
For instance in scenario reduction, Heitsch & R¨omisch (2003) and Rujeerapaiboon et al.
(2018) equate problem (3) to the k -Medoids Cluster Center optimization problem.
They employ the k -medoids heuristic and an approximation algorithm, respectively, on small data sets containing approximately 1 , 000 samples.
In contrast, our GBD algorithm provides an optimality guarantee and as we show in numerical experiments, scales gracefully on data sets with 75 ,samples.
We review k -medoids and scenario reduction in Appendix E.
Self-supervised Learning.
Active learning selection strategies improve with useful features x ∈
X .
Recently, self-supervised learning has shown tremendous promise in extracting representations for semi-supervised tasks.
For example, a model pre-trained with SimCLRv2 on ImageNet (Deng et al., 2009) can be ﬁnetuned with labels from 1% of the data to achieve impressive classiﬁcation accuracy (Chen et al., 2020a;b;c).
In contrast, the standard practice in classical active learning is to train a classiﬁer for feature extraction with previously labeled data (Sener & Savarese, 2017;
Sinha et al., 2019; Shui et al., 2020), with the previous papers requiring labeling up to 20% of the data to achieve comparable performance to self-supervised pre-training with supervised ﬁnetuning.
Our numerical experiments verify that self-supervised pre-training can accelerate active learning to competitive accuracy with orders of magnitude less labeled data.
Chen et al.
(2020c) use random selection after self-supervised learning.
With enough points, current active learning strategies achieve similar bottleneck performances (Sim´eoni et al., 2019).
However, the heuristic nature of these strategies leave opportunities in the low budget regime.
Numerical results show that optimization in this regime speciﬁcally yields large gains over heuristic baselines.
Generalized Benders Decomposition (GBD).
This framework is commonly used in non-convex and integer constrained optimization (Geoffrion, 1972), since it instead solves sub-problems while often providing an optimality guarantee.
However, the vanilla GBD algorithm is typically slow to converge for large problems.
As a result, the integer programming literature proposes improvement techniques such as multiple different types of constraints to tighten the relaxations and heuristics to direct the search (Rahmaniani et al., 2017).
These improvements are often customized for the speciﬁc optimization problem being solved.
In our case, we propose enhanced optimality cuts in
Section 3.2 that leverage properties of the Wasserstein distance.
Further, our pruning constraints are simple techniques that can be used in GBD for arbitrary binary MILPs.
E XPERIMENTS
In this section, we ﬁrst evaluate our algorithm on solving problem (4) by comparing against the standard heuristic from the scenario reduction literature (Heitsch & R ¨ omisch, 2003).
We then evaluate our algorithm on the following two active learning tasks: (i) low budget image classiﬁcation where we select images based on features obtained via self-supervised learning and (ii) domain adaptation where we select images in an unlabeled target pool using features obtained via unsupervised domain adaptation over a source data pool.
For both tasks, we outperform or remain competitive with the best performing baselines on all data sets and budget ranges.
In ablations, we ﬁrst observe that our approach is effective with different embedding methods (e.g., self-supervised, domain adversarial, and classical active learning where features are obtained from previous rounds); in each case, we yield improvements over baselines.
Second, the acceleration techniques proposed in Section 3.2 help optimize problem (4) faster and often improve active learning.
Finally, from a wall-clock analysis, running GBD for more iterations yields better selections, but even early stopping after 20 minutes is sufﬁcient for high-quality solutions.
In this section, we only summarize the main results.
We leave detailed analyses for all experiments in
Appendix E. Further in Appendix E.7, we include additional experiments including further ablations and qualitative evaluation of the selections and t -SNE visualizations of the feature space.
5. E VALUATION P ROTOCOL AND B ASELINES
We evaluate active learning in classiﬁcation on three data sets: STL-10 (Coates et al., 2011), CIFAR10 (Krizhevsky, 2009), and SVHN (Netzer et al., 2011).
Each data set contains 5 , 000 , 50 , 000 , and 73 , 257 images, respectively, showcasing when the number of potential core sets ranges from small to large.
We initialize with no labeled points and ﬁrst pre-train our model (i.e., feature encoder + head) using the self-supervised method SimCLR (Chen et al., 2020a;b).
We then employ active learning over sequential rounds.
In each round, we use feature vectors obtained from the encoder output to select and label B new images and then train the classiﬁer with supervised learning.
Our domain adaptation experiments use the Ofﬁce-31 data set (Saenko et al., 2010), which contains 4 , 652 images from three domains, Amazon (A), Web (W), and DSLR (D).
We use active learning to label points from an initially unlabeled target set to maximize accuracy over the target domain.
We follow the same sequential procedure as above except with a different feature encoding step, f -DAL, which is an unsupervised domain adversarial learning method (Acuna et al., 2021).
[Table] [Table]
[Table]
We use a ResNet-18 (He et al., 2016) feature encoder for STL-10, ResNet-34 for CIFAR-10 and
SVHN, and Resnet-50 for Ofﬁce-31.
Our head is a two-layer MLP.
The downstream classiﬁer is the pre-trained encoder (whose weights are frozen) with a new head in each round.
Supervised learning is performed with cross entropy loss.
We compute Wasserstein distances using the Python
Optimal Transport library (Flamary et al., 2021) and solve W-RMP( Λ ) with the COIN-OR/CBC
Solver (Forrest & Lougee-Heimer, 2005).
We run our algorithm up to a time limit of 3 hours (and increase to 6 hours for experiments on SVHN).
Finally, we implement two variants of the GBD algorithm without the pruning constraints (Wass. + EOC) and with them (Wass.
+ EOC + P).
See
Appendix E for details on training and optimizer settings.
We compare against the following baselines: (i) Random Selection; (ii) Least Conﬁdence (Settles, 2009;
Shen et al., 2017; Gal et al., 2017); (iii) Maximum Entropy (Settles, 2012); (iv) k
-Medoids
Cluster Centers (Heitsch & R ¨ omisch, 2003); (v) Greedy k -centers
(Sener & Savarese, 2017); (vi)
Wasserstein Adversarial Active Learning (WAAL) (Shui et al., 2020).
The second and third baselines are uncertainty-based approaches using the trained classiﬁer in each round, the fourth and ﬁfth are representation-based approaches that leverage unsupervised pre-trained features, and the last is a greedy strategy that approximates the Wasserstein distance with an adversarial regularizer in training.
All baselines use the same self-supervised pre-training and supervised learning steps as us.
5. M AIN R ESULTS
[Table]
Objective Function Value (see Appendix E.2 for more results).
k -Medoids is a well-known heuristic for problem (4) (Heitsch & R ¨ omisch, 2003; Rujeerapaiboon et al., 2018).
Table 1 compares objective function values obtained by our algorithm versus k -medoids for STL-10, CIFAR10, and SVHN.
We especially outperform k -medoids for
B ≥ 120 where the selection problem is large and harder to solve.
This gap also widens for larger data sets; for instance on SVHN, we almost always achieve around 2 × improvement in objective function value.
Table 2 further compares k -medoids versus Wass.
+ Enhanced Optimality Cuts (EOC) on CIFAR-10 where we force early termination.
Even after reducing our algorithm’s runtime to three minutes, we still achieve high-quality solutions to the optimization problem.
Image Classiﬁcation (Appendix E.3).
Figure 2 summarizes our methods versus baselines on each data set.
For each data set and number of points, our approaches are either competitive with or outperform the best baseline.
Table 3 highlights the extremely low budget regime ( B ≤ 40) where every point must be carefully selected.
For instance, if we can only label B = 10 points, poorly selecting even one point wastes 10% of the budget.
Here, our best methods typically outperform the baselines on all three data sets often by large margins (e.g., ≥ 4 .
9% on STL-10 at B = 40 , ≥ 9 .
1% on CIFAR-10 at B = 20 , and ≥ 5 .
2% on SVHN at B = 40 ).
There is no dominant baseline over all budget ranges and data sets: k -medoids is effective for
B ≤ 40 or smaller data sets (STL-10) but worse with bigger budgets and data sets (CIFAR-10,
SVHN), whereas Random and Greedy k -centers improve in the latter.
Furthermore while SimCLRpretraining improves all baselines compared to classical active learning papers (Sener & Savarese,
Our best models outperform baselines for most budgets
[Table] [Table]
2017; Shui et al., 2020), representation methods (Greedy k -centers, k -medoids) generally outperform uncertainty methods (WAAL, Least Conﬁdence, Maximum Entropy) in this setup because classiﬁcation is now easier, which means that it is more useful to obtain representative samples of each class than uncertain examples.
By optimizing for a diverse core set selection, we outperform with the best baseline for nearly all budgets and data sets.
Domain Adaptation (Appendix E.4).
Figure to
Amazon, Web to Amazon, and DSLR to Web; we leave the other source-target combinations in
Appendix E.4.
As before, our approach either outperforms or is competitive with the best baseline for all budgets and source-target pairs, the baselines are non-dominating over each other, and representation methods such as ours beneﬁt from strong features and model pre-training.
Thus, our
[Table]
framework can be plugged into any classiﬁcation task as long as there is a good feature space for optimization; for example, here we use unsupervised domain adaptation (Acuna et al., 2021)
Representations from classical active learning (Appendix E.5).
Table 4 ablates the effect of using features from unsupervised pre-training by presenting accuracy on CIFAR-10 and SVHN under classical active learning.
We implement the exact setup of and compare against WAAL (Shui et al., 2020) which was speciﬁcally designed for the classical setting.
Although the optimization problem is harder than in the low-budget setting due to having less useful features and solving for larger B , our approach is still competitive and can outperform WAAL sometimes by up to 3% .
This suggests that our algorithm is also effective in classical active learning.
Acceleration Techniques.
Figure 4 (left) plots the upper and lower bounds for the ﬁrst 50 iterations of the vanilla algorithm Wass.
, Wass.
+ EOC, and Wass.
+ EOC + P (i.e. including Pruning constraints) for CIFAR-10 at B = 10 .
The lower bounds for Wass.
are negative, meaning the vanilla algorithm requires a large number of iterations to improve the bounds.
Wass.
+ EOC
and
Wass.
+ EOC + P both improve the lower bounds.
At the same time, Wass.
+ EOC + P yields better upper bounds early in the algorithm, but is restricted to a local minimum whereas Wass.
+
EOC eventually improves over the heuristic.
Thus, Wass. requires many iterations to converge, necessitating our cuts and pruning techniques.
Wall-clock Analysis (Appendix E.6).
Our algorithm is effective even without convergence.
Figure 4 (right) plots accuracy on CIFAR-10 for Wass.
+ EOC where we terminate at different runtimes and use the incumbent solution.
Longer runtimes typically yield better results, especially for B ≤(i.e.
, a sub-optimal selection here wastes ≥ 2 .
5% of the budget).
However at higher budgets, setting the runtime as low as 20 minutes is sufﬁcient to obtain high-quality selections, since the marginal value of each new labeled point decreases as the labeled set increases.
Note that the baselines typically require between several seconds to minutes to make selections, meaning our algorithm is on average slower to generate selections.
Nonetheless, the time cost of selection is still less than model training time, and for some applications, less than human labeling time (Rajchl et al., 2016), meaning a slightly longer selection time is generally acceptable if we can yield better selections.
D ISCUSSION
When the budget for labeling images is limited, every point must be carefully selected.
In this work, we propose an integer optimization problem for selecting a core set of data to label.
We prove that our problem upper bounds the expected risk, and we develop a GBD algorithm for solving our problem.
Under relevant features, formulating a MILP ensures that we may ﬁnd globally optimal core set selections, in contrast to most selection strategies that serve as efﬁcient but ad-hoc heuristics.
Our numerical results show that our algorithm is an effective minimizer of the Wasserstein distance compared to heuristics.
Finally, we show over four image data sets that our optimization is competitive with current baselines in active learning, especially when the labeling budget is low.
A PPENDIX A P ROOFS OF S TATEMENTS
Proof of Theorem 1.
Let D y
=
[ | Ω( x i ) − Ω( x (cid:48)
i ) | ] N,Ni =1 ,i (cid:48)
=1 be a distance matrix between the labels of points in D .
Let L K be the set of K -Lipschitz functions.
Using the Kantorovich-Rubenstein
Duality (Villani, 2008), we bound the core set loss by a Wasserstein distance
[Equation]
The ﬁrst line introduces probability masses and upper bounds the objective with a supremum over all K -Lipschitz functions ˜ (cid:96)
∈
L K .
The second line replaces ˜ (cid:96) with vectors µ and λ that satisfy the
Lipschitz property.
The ﬁnal line takes the dual of the linear program.
Next note that | Ω( x i ) − Ω( x i (cid:48) ) | ≤ C , meaning that for any feasible Γ to the Wasserstein problem, we have the following inequality:
[Equation]
The last equality follows from the constraints in the Wasserstein distance primal program implying 1 T Γ1 = 1 .
We now rewrite the objective of the linear program to
[Equation]
Since the second term does not depend on Γ , we remove it from the optimization objective and complete the proof.
Proof of Corollary 1.
The proof follows from Lemma 2 (see Appendix C), which states that if the objectives and constraints of the optimization problem are (i) separable in the decision variables and (ii) linear, then the algorithm will converge.
Problem (4) satisﬁes the requirements of this lemma, meaning that our algorithm terminates in ﬁnite iterations.
Proof of Proposition 1.
Let ( η ∗ , π ∗ ) be the optimal solution to W-MP and let Γ ∗ be the optimal primal solution to W ( C ( π ∗ ) , D ) .
To prove that each inequality is an optimality cut, we show that they are met by ( η ∗ , π ∗ ) .
We ﬁrst prove the Lower Bound:
[Equation]
The second line follows from expanding the trace and noting Γ ∗ T 1 = π ∗ /B .
The inequality follows from taking the minimum over all
D
i,i (cid:48) and the last line follows from Γ ∗ T 1 = π ∗ /B .
To prove the Triangle Inequality, note that the Wasserstein distance is a distance metric and then satisﬁes a triangle inequality by deﬁnition.
That is, for any ˆ π ∈ P ,
[Equation]
The right-hand-side of the inequality is a Wasserstein distance which can be lower bounded similar to before:
[Equation]
 
[Equation]
The key difference in this sequence from the previous is that we only consider indices i (cid:48) where x i (cid:48) ∈ C (ˆ π ) .
Finally, recall that η ∗ ≥ W ( C ( π ∗ ) , D ) .
Substituting these two inequalities and rearranging (9) completes the proof
[Equation]
Finally
[Equation]
to prove the Dual Inequality
[Equation]
note that
[Equation]
We lower bound the second term by obtaining an upper bound on the Wasserstein distance:
[Equation]
i = max i (cid:48) ∈{ 1 ,...,N } { D i,i (cid:48) }
B ˆ π i where the upper bound now follows from taking a maximum.
Subtracting from both sides lower bounds the second term in (10).
Finally, we lower bound ( C ( π ∗ ) , D ) by considering the Wasserstein distance dual program
[Equation]
Note that the optimal dual variables (ˆ λ , ˆ µ ) for W ( C (ˆ π ) , D ) are feasible solutions for the Wasserstein dual problem W ( C ( π ∗ ) , D ) and thus, also a lower bound on the Wasserstein distance.
Substituting the corresponding objective value into (10) completes the proof.
B O N THE L IPSCHITZ C ONTINUITY A SSUMPTION
Our main result in Theorem 1 states that the Wasserstein distance upper bounds the core set loss if the training loss function (cid:96) ( x , y ; w ) is K -Lipschitz continuous over the data set space X × Y for ﬁxed model parameters w .
Lipschitz continuity assumptions are commonplace in the development of active learning theory (Sener & Savarese, 2017; Shui et al., 2020).
We show below that our assumption is relatively mild and can sometimes be satisﬁed in practice.
Our assumption builds from Sener & Savarese (2017) who require their loss function (cid:96) ( x , y ; w ) to be
Lipschitz continuous over the input features X only.
They show that convolutional neural networks trained using Mean-Squared Error (MSE) loss meet their assumption.
Although we require the loss function to be Lipschitz in both the input and output, we can extend the result of Sener & Savarese (2017) to show that convolutional neural networks are Lipschitz in the input-output space as well.
We ﬁrst state the Lemma of Sener & Savarese (2017) before proving our corollary.
Lemma 1 (Sener & Savarese (2017)) .
Consider the C -class classiﬁcation problem over feature
X and one-hot-encoded label ˆ Y := { e 1 , . .
, e C } spaces.
Let F : X → ˆ Y be a convolutional neural network with n c convolutional-maxpool-ReLU layers and n f fully connected layers.
Then, there exists a constant α > 0 such that the MSE loss function (cid:96) ( x , y ; w ) :
= (cid:107) F ( x ; w ) − y (cid:107) 2 is ( √ C − 1 α n c + n f /C )
-Lipschitz continuous in x for ﬁxed y and w .
While the lemma states that (cid:107) F ( x ; w ) − y (cid:107) 2 is Lipschitz continuous, the proof further implies that by composition, the neural network F ( x ; w ) is also Lipschitz continuous with the same constant.
Corollary 2.
Consider the metric space ( X × Y , (cid:107) x (cid:107) + ε | y | ) using features X and labels Y = { 1 , . .
, C } .
Let e ( y ) denote the one-hot encoding of y .
The MSE loss function (cid:96) ( x , y ; w ) :
= (cid:107) F ( x ; w ) − e ( y ) (cid:107) 22 is Lipschitz continuous in ( x , y ) for ﬁxed w .
Proof.
The proof is a straightforward extension of Lemma 1.
Consider two pairs ( x , e ( y )) and ( x (cid:48) , e ( y (cid:48) )) .
We bound with the following steps:
[Equation]
The ﬁrst two inequalities follow from applying the Reverse Triangle Inequality and then the Triangle
Inequality.
The third inequality follows from Lemma 1 and | y − y (cid:48) | ≥ (cid:107) e ( y ) − e
( y (cid:48) )
(cid:107) 2 .
The ﬁnal inequality follows from taking the maximum of the coefﬁcients.
Our result holds for MSE loss, speciﬁcally.
In practice, we use the Cross-Entropy loss, which does not satisfy this Lipschitz continuity assumption.
Nonetheless, the empirical effectiveness of our strategy highlights its usefulness.
C B ACKGROUND ON G ENERALIZED B ENDERS D ECOMPOSITION
Generalized Benders Decomposition (GBD) is an iterative algorithm for solving non-convex constrained optimization problems (Geoffrion, 1972).
We summarize the general steps of this algorithm below and refer the reader to Rahmaniani et al.
(2017) for a recent survey on this methodology.
Let x ∈ X ⊆
R
d 1 and y ∈ Y ⊆
R d 2 be variables and consider the problem
[Equation]
where f ( x , y ) : R d 1 × R d 2 → R and g ( x , y ) :
R d 1 × R d 2 → R m are convex functions but X and Y may be non-convex sets.
This problem is equivalent to the following bi-level Master Problem (MP):
[Equation]
where η is a slack variable which when minimized, is equal to the optimal value of problem (11).
This MP can also be re-written with a single constraint by replacing the right-hand-side of the inequality with the Lagrangian L ( y ) :
= sup λ ∈ R m inf x ∈X { f ( x , y ) + λ T g ( x , y ) } .
Although MP contains an inﬁnite number of constraints, we can instead solve a Relaxed Master
Problem (RMP):
[Equation]
where Λ ⊂ R m consists of a ﬁnite set of dual variables.
Let ( η ∗ , x ∗ , y ∗ ) be an optimal solution to
MP and let (ˆ η, ˆ y ) be an optimal solution to RMP.
Then, ˆ η ≤ η ∗ presents a lower bound to MP (and therefore problem (11)).
Furthermore, L (ˆ y ) ≥ f ( x ∗ , y ∗ ) gives an upper bound.
Finally, let ˆ x be the minimizer of L (ˆ y ) .
Then (ˆ x , ˆ y ) is a feasible solution achieving the above upper bound.
Carefully selecting Λ will yield tighter relaxations to MP (Floudas & Pardalos, 2013).
GBD proposes iteratively growing Λ and solving increasingly larger RMPs until the upper and lower bounds converge.
We initialize with a feasible ˆ y ∈ Y and Λ = ∅ .
Then in each iteration t ∈ { 0 , 1 , 2 , . . .
} , we:
Given a solution (ˆ η t , ˆ y t ) , solve the Lagrangian L (ˆ y t ) to obtain primal-dual solutions (ˆ x t , ˆ λ t ) .
Update Λ ← Λ ∪ { ˆ λ t } and solve RMP to obtain a solution (ˆ η t +1 , ˆ y t +1 ) .
The algorithm terminates when the tolerance L (ˆ y t ) − ˆ η t ≤ ε is below a threshold ε > 0 .
For many problems, the above algorithm will converge in a ﬁnite number of iterations.
This is referred to as Property (P) in the literature (Geoffrion, 1972; Floudas & Pardalos, 2013).
Lemma 2 (Geoffrion (1972)) .
Consider the non-convex optimization problem (11).
If f ( x , y ) =
f 1 ( x ) + f 2 ( y ) and g ( x , y ) =
g 1 ( x ) + g 2 ( y ) are separable, linear functions, then for any ε > 0 ,
Generalized Benders Decomposition terminates in a ﬁnite number of iterations.
Even so, the number of iterations may be exceedingly long.
As a result, the modern literature in
GBD explores new constraints and heuristics to quickly tighten the relaxation of RMP versus MP.
Often, these techniques are customized to the structure of the speciﬁc optimization problem (11) being solved.
In Section 3.2, we propose new custom constraints to improve the algorithm.
D D ETAILS ON OUR G ENERALIZED B ENDERS D ECOMPOSITION A LGORITHM FOR M INIMIZING W ASSERSTEIN D ISTANCE
In this section, we provide further details on the GBD algorithm used to solve problem (4).
To complement the Enhanced Optimality Cuts in Section 3.2, we ﬁrst discuss an additional set of pruning constraints to improve our algorithm.
We then detail the full steps of the algorithm and provide
Python-style pseudo-code.
D. P RUNING C ONSTRAINTS
We ﬁrst deﬁne the pruning constraints used to improve our GBD algorithm.
These constraints complement the Enhanced Optimality Cuts introduced in Section 3.2 as an additional set of constraints
1: Input: budget B ; initial core set ˆ
; constraint sets bound B = bound B
; maximum limit T 2: repeat 3:
Solve W ( C (ˆ π ) , D ) : ˆ Γ ← arg min Problem (1), (ˆ λ , ˆ µ ) ← arg max Problem (2).
4: Λ ← Λ ∪ { (ˆ π , ˆ Γ , ˆ λ , ˆ µ ) } //
Update GBD constraint set 5: if W ( C (ˆ π ) , D ) ≤ B then 6:
B ← min( B, W ( C (ˆ π ) , D )) // Update upper bound on optimal value 7:
P + ← P + ∪ { ˆ π } // Update Near Neighbourhood set 8: else 9:
P − ← P − ∪ { ˆ π } //
Update Prune Neighbourhood set 10: end if 11:
Solve W-RMP( Λ ): ( [Equation]
, ˆ
[Equation]
) ←
[Equation]
Lower Bound, Eq.
Triangle Ineq.
, Eq.
[Equation] Γ [Equation] [Equation]
Dual Ineq.
, Eq.
[Equation]
[Equation]
π
[Equation] [Equation]
P + π
[Equation]
B, ∀
[Equation]
P − 12: B ← max( B, ˆ η ) // Update lower bound on optimal value 13: until T hours or B − B < ε // Exit at timeout or convergence used to accelerate the algorithm.
However, these pruning constraints do not preserve the convergence criteria of Corollary 1, since there is no guarantee that the optimal solution will satisfy them.
Instead, these constraints are heuristics that our numerical results empirically show to improve the active learning strategy.
Let P + and P − denote two sets of good and bad candidate core sets ˆ π , respectively, and let β + , β − ∈ (0 , 1) denote two hyperparameters.
In each iteration, we obtain a candidate ˆ π and a corresponding upper bound W ( C (ˆ π ) , D ) to the W-MP.
If this upper bound is a new incumbent (i.e., is lower than the previous upper bounds)
then we update P + ∪ { ˆ π } and limit the search to within a Hamming ball of size β + B around ˆ π .
Otherwise, we update P − ∪ { ˆ π } and remove a Hamming ball of size β − B around ˆ π from the search space.
These two pruning techniques can be described by the following linear constraints: ∀ ˆ π ∈ P + , π T ˆ π ≥ β + B and ∀ ˆ π ∈ P − , π T ˆ π ≤ β − B .
D. A LGORITHM S TEPS AND P SEUDO - CODE
Algorithm 1 details the mathematical steps of the GBD algorithm and Figure 5 provides a Pythonstyle pseudocode snippet.
We formulate our algorithm in a loop by iteratively solving our optimization problem and adding new constraints.
The main dependency of this optimization algorithm lies in an off-the-shelf Wasserstein distance computation library (Flamary et al., 2021) and a MILP solver (Forrest & Lougee-Heimer, 2005).
Note that rather than running GBD for a ﬁxed number of iterations, Algorithm 1 performs iterations until a maximum time limit T or until the upper B and lower bounds B converge under a tolerance ε .
Furthermore in the implementation of the algorithm, we may also select hyperparameters for the MILP solver being used to solve each instance of WRMP( Λ ).
For our experiments, we only tune the time limit and the minimum optimality gap.
That is, we set the solver for W-RMP( Λ ) to timeout in 180 seconds or until the solution it ﬁnds is within 10% of optimality.
[Table]
= ...
[Table]
of
[Table] [Table]
= ...
[Table] [Table]
# Calculate distance matrix arcs = pairwise_distances(features) # Define W-RMP MIP model and variables prob =
mip.
Model() pis
=
[prob.add_var(var_type=mip.
BINARY)
for i in range(N)
] eta = prob.add_var() # Set objective and constraints prob.objective = mip.minimize(eta) prob += mip.xsum(pis) =
=
B # If we labeled a point in an earlier round, then it should be 1 for i in prev_selected: prob += pis[i]
==
# GBD loop lb, ub, tol, max_time = -inf, inf, 1e-3,while lb < ub - tol
and current_time() < max_time:
# Run MIP solver to populate pis, eta prob.optimize() # Compute Wasserstein distance and dual variables with POT pis_x =
[p.x for p in pis] w_hat, info = ot.emd2(pis_x / N, ones(N) / N, arcs, log=
True) lambda_hat, mu_hat = info[’u’], info[’v’]
# Update bounds lb, ub = max(lb, eta.x), min(ub, w_hat)
# Add Benders cut prob += eta >=
w_hat + mip.xsum( lambda_hat[i] * (pis[i] - pis.x[i]) /
B for i in range(N))
# Triangle inequality cut min_Di = ...
# Compute coefficients according to Proposition 1 prob += eta
+ w_hat >= mip.xsum( min_Di[i] * pis[i] / B for i in range(N))
# Dual inequality cut max_Di = ...
# Compute coefficients according to Proposition 1 prob += eta >=
w_hat + mip.xsum( (lambda_hat[i] - max_Di[i]) * pis[i] / B - mu_hat[i] / N for i in range(N))
# Pruning constraints if ub ==
w_hat: prob += B * beta_plus <= mip.xsum( pis[i] * pis.x[i] for i in range(N)) else:prob += B * beta_minus >= mip.xsum
( pis[i] * pis.x[i] for i in range(N))
# Output return where(pis.x == 1) # All selected indices so far
D. R EDUCING THE C OMPUTATIONAL C OMPLEXITY
Recall that the original problem (4) is a MILP with N 2 + N variables and 2 N + 1 constraints.
This problem is too large to solve on standard computers when N ≥ 50 , 000 , which is typical for deep learning data sets.
Instead in Algorithm 1, we repeatedly solve two smaller sub-problems: (i) solving W-RMP( Λ ) to obtain selections ˆ π and (ii) computing the Wasserstein distance W ( C (ˆ π ) , D ) to obtain dual variables ˆ λ .
W-RMP( Λ ) is a MILP with N + 1 variables and | Λ | + 1 constraints.
Incorporating the Enhanced
Optimality Cuts and the Pruning Constraints leads to a MILP with N + 1 variables and 5 | Λ | +constraints.
From our experiments, | Λ | is typically on the order of 100 , meaning that this MILP contains several orders of magnitude fewer variables and constraints than problem (4).
We can ﬁnd an optimal or nearly optimal solution to this MILP in less than 180 seconds.
Before computing the discrete Wasserstein distance W ( C (ˆ π ) , D ) , we ﬁrst need the distance matrix of features D x .
For Euclidean distance, computing this has complexity O ( dN 2 ) where d is the feature dimension.
Note that this is a one-time cost and that once we construct the distance matrix, we can store it in memory for the remainder of the algorithm.
To compute W ( C ( ˆ π ) , D ) in each iteration, we use an off-the-shelf solver, POT
(Flamary et al., 2021).
POT applies the Network Simplex Algorithm (Tarjan, 1997), which is an algorithm for solving network ﬂow problems over graphs.
Because W ( C (ˆ π ) , D ) is a minimum ﬂow over a graph with 2 N nodes and N 2 edges, our complexity is O ( N 3 (log N ) 2 ) .
When B is much smaller than N , we can often simply approximate ˆ λ .
We ﬁrst remove nodes corresponding to ˆ π i = 0 to rewrite W ( C (ˆ π ) , D ) as an equivalent network ﬂow with N + B nodes and NB edges, resulting in complexity O
( N 2 (log N ) 2 ) .
However, from (2), this problem returns dual variables corresponding only to the B points for which ˆ π i = 1 .
We can ﬁnd the remaining dual variables via a c-transform (Peyr ´ e et al., 2019), which requires solving a minimization problem over an N -length array for each remaining variable.
We refer the reader to Peyr ´ e et al.
for details.
This approach does not guarantee an optimal ˆ λ , but we ﬁnd that it works well in practice.
E C OMPLETE N UMERICAL R ESULTS
This section expands the numerical results summarized in the main paper.
Below, we summarize the main points:
• Section E.1 details the hyperparameters that were used in our GBD algorithm and optimization solver.
We also describe model choices and training settings used in our image classiﬁcation and domain adaptation experiments.
• Section E.2 describes our experiments on optimizing problem (4).
We ﬁrst summarize the related scenario reduction literature and highlight the k -Medoids Cluster Center algorithm baseline.
We then expand the numerical results from the main paper to show that for generating high-quality solutions to problem (4), our approach typically outperforms k -medoids and sometimes by up to 2 × .
Furthermore, these improvements remain even when we limit our algorithm to comparable runtimes with k -medoids.
• Section E.3 provides the full table of values for active learning experiments on the three image classiﬁcation data sets.
In addition to the results in the main paper, this section shows that our approach is consistently effective even up to budgets of B = 180 .
• Section E.4 provides the full table of values for active learning with domain adaptation experiments.
We evaluate all six source-target combinations on the Ofﬁce-31 data set to show our approach either outperforms or competitive with the best baseline.
To further compare against k -medoids, we perform statistical t -tests in a head-to-head setting.
Our approach statistically outperforms k -centers for more budgets and data sets.
• Section E.5 details our ablation on classical active learning without self -supervised features.
We ﬁrst describe the experiment setup and then expand on the results in the main paper, showing that our algorithm is effective even without self-supervised features and in higher budgets. [Table] [Table]
• Section E.6 provides the full table of selection runtimes.
Most baselines require at most a few minutes to select points.
We further discuss the downstream effect of our algorithm having longer runtimes in practice.
• Section E.7 highlights two additional results not discussed in the main paper.
We ﬁrst ablate the value of customizing the base metric in the Wasserstein distance by showing that Cosine distance yields signiﬁcant improvements over Euclidean distance on the CIFAR10 data set.
We then provide a qualitative analysis of our selections by plotting t -SNE visualizations of the feature space as well as samples of images selected.
In general, our visualizations show that we can better cover the feature space than baselines
E. M ETHODS
We ﬁrst detail hyperparameters for the optimization algorithm, methods for the image classiﬁcation experiments, and methods for the domain adaptation experiments.
All experiments are performed on computers with a single NVIDIA
V
GPU card and 50 GB CPU memory.
Our optimization problem is solved on CPU.
Algorithm Settings.
When formulating the Wasserstein distance problem, we use Cosine distance (i.e., l 2 normalize our features) as the base metric for the image classiﬁcation experiments.
Cosine distance is a better choice than Euclidean distance here because the features are obtained with
SimCLR, which involves maximizing a Cosine distance between different images; that is, the optimization and feature generation use the same base metric.
We include ablation tests in Appendix E. comparing Cosine versus Euclidean distance on CIFAR-10.
We test both Cosine and Euclidean distance for the domain adaptation experiments.
These experiments use features obtained via f -DAL, which is not necessarily tied to Cosine distance.
We implement the optimization problem using the COIN-OR/CBC Solver (Forrest & LougeeHeimer, 2005).
Since it is crucial to run as many iterations as possible to obtain better solutions, we set the solver optimality gap tolerance to 10% and a time limit of 180 seconds to solve W-RMP( Λ ) in each iteration.
We also warm-start our algorithm with the solution to the k -centers core set, i.e., ˆ π
=
ˆ π k -centers , for image classiﬁcation and k -centers or k -medoids for domain adaptation.
We run Algorithm 1 with ε = 10 − 3 and T = 3 hours for STL-10, CIFAR-10, and Ofﬁce-31, and
T = 6 hours for SVHN, which contains a larger data set.
Finally for Wass.
+ EOC + P, we set ( β + , β − ) =
These are only optimization parameters and can be tuned by evaluating the objective function value of the optimization problem.
Image Classiﬁcation.
Table 5 details the differences in setup for each of the three data sets used in the image classiﬁcation experiments.
For each experiment, we use a ResNet feature encoder with a two-layer MLP (Linear-BatchNorm-ReLU-Linear) head.
The ﬁrst linear layer has a 64 -dimension output in order to reduce the dimensions of the feature space.
We use a 64 -dimension second layer in the SimCLR pre-training step and a 10 -dimension second layer during supervised learning.
The choice of speciﬁc ResNet encoder was made based on the computational requirement of training on a single GPU under the required batch sizes.
For each experiment, we use the default test sets to evaluate our strategies.
In the SimCLR representation learning step, we train for 800 epochs with an initial learning rate of 0 .
1 decaying to 0 .
We implement SimCLR following the steps of Chen et al.
(2020c) with the following random data augmentations in their default PyTorch settings: resized crops, horizontal ﬂips, color jitter, and grayscale.
For SVHN, we omit the horizontal ﬂips and increase the scale of the crops to (0 . 8 , 1) , because this yielded better representations in our experiments.
In the supervised learning step, we train with cross entropy loss, random crops and horizontal ﬂips, and a batch size of 128 with a constant learning rate of 0 .
002 for 1600 epochs.
Domain Adaptation.
Table 6 details the differences in setup for each of the tasks in the domain adaptation experiment.
We use a ResNet-50 + one-layer bottleneck (Linear-BatchNorm-ReLUDropout) encoder and a two-layer classiﬁcation head (Linear-ReLU-Dropout-Linear).
The ResNet50 is initialized with Imagenet-pretrained weights.
The bottleneck sets the feature space dimension to 1 , 000 .
The second layer of the classiﬁcation head reduces the dimension to 31 for the number of classes.
For each experiment, we randomly partition the target data set into 70% for training and 30% for testing.
We perform the representation learning step using the full labeled source data set and the unlabeled target training data set.
We then perform the supervised learning step using only labeled target training data.
The representation learning is performed via the unsupervised domain adaptation algorithm, f -DAL (Acuna et al., 2021).
We train for 20 epochs with 1 , 000 iterations per epoch.
We use a learning rate of 0 .
004 decaying stepwise and a batch size of 32 , which are the default parameters of Acuna et al.
In the supervised learning step, we train with cross entropy loss with the same learning rate and batch size.
E. O BJECTIVE F UNCTION V ALUE
We evaluate GBD on minimizing the Wasserstein distance by comparing it to the k -Medoids Cluster
Center algorithm from scenario reduction for stochastic optimization.
We ﬁrst brieﬂy summarize the scenario reduction problem.
We then expand the numerical results in the main paper, showing that for large problem sizes, our algorithm yields better minimizers of the Wasserstein distance.
Background.
Consider a general stochastic optimization problem
[Equation]
where f ( x , w ) is an objective function, w ∈ W is the variable to optimize and x ∼ P is a random variable, here referred to as a scenario.
Given a data set D :=
{ ˆ x i } Ni =1 of N scenarios, we can solve the Sample Average Approximation (SAA), which corresponds to minimizing the empirical expected value.
Machine learning is a special case of the above problem where f ( x , w ) corresponds to the loss function, w to neural network weights, and x to training data.
SAA may grow difﬁcult to solve with large data sets, especially if W is non-convex or contains integral constraints.
Here, scenario reduction is the problem of formulating an approximation of the
SAA by using only a subset C :
=
{ ˆ x j } Bj =1 ⊂ D of B scenarios.
Heitsch & R¨omisch (2003) consider constructing this subset by minimizing the Wasserstein distance between C and D , i.e., by solving problem (4).
When N is large, Heitsch & R¨omisch (2003) propose the k -Medoids Cluster Center algorithm as a heuristic.
Although the algorithm is computationally easy to implement, making it desirable in practice, recently Rujeerapaiboon et al.
(2018) show that k -medoids can potentially be a poor approximation in the worst case, and also introduce a new approximation algorithm.
Finally, we note that the scenario reduction literature rarely considers problem sizes of the scale in our active learning tasks, with numerical experiments in most cases ranging up to 10 3 .
In contrast, the SVHN data set requires N ≤ 75 , 000 . [Table] [Table]
Comparing the Minimum Wasserstein Distance.
Our GBD algorithm directly solves the integer optimization problem (4) rather than a heuristic or approximation algorithm.
We compare against k -medoids on minimizing the Wasserstein distance for the three image classiﬁcation data sets.
Table 7 presents the complete results of our method versus k -medoids for each data set.
We run the solver for three hours for STL-10 and CIFAR-10 and for six hours for SVHN.
Although the algorithm has not yet converged, for each data set, our approach is competitive for all settings (except for B ≤ 20 at CIFAR-10), and especially outperforms k -medoids for B ≥ 120 .
Finally, note that for SVHN, we reduce the objective function value by over half compared to k -medoids.
To further ablate our method, we also compare different runtimes of CIFAR-10 with k -medoids in
Table 8.
Even down to three minutes, we often get a better minimizer than k -medoids.
Note from
Table 12 that k -medoids requires on average several minutes to compute a selection.
This suggests that our algorithm may have more general use cases outside of active learning, such as an alternative to k -medoids in scenario reduction.
E. I MAGE C LASSIFICATION
Table 9 lists the means ± standard deviations of accuracies shown in Figure 2.
Our models consistently outperform the baselines for both STL-10 and SVHN, with the exception of the ﬁrst round in STL-10.
On CIFAR-10, our models outperform in the ﬁrst two rounds, but afterwards remain competitive with k -centers (i.e., less than 2% difference in accuracy).
On SVHN, our models similarly outperform in the ﬁrst four rounds at which point k -centers catches up and the two remain competitive for the rest of training.
Note that as the budget increases to B = 180 points, all of the methods converge to a ﬁnal accuracy. [Table]
E. D OMAIN A DAPTATION
Table 10 lists the means of for all of the different source- combinations of the Ofﬁce-31 data set.
We summarize the results below.
Amazon as target data set.
For both DSLR to Amazon and Web to Amazon, our framework dominates all baselines over the entire budget range.
These two domain combinations are the two hardest problems, as the unlabeled target data set, Amazon, is much larger than the source data sets and the test accuracy on Amazon is always lower than 80% .
DSLR and Web as target data set.
For the other source-target combinations that have a smaller and “easier” target domain, our framework still outperforms at the extremely low budget regime
B ≤ 20 and then remains competitive with k -centers at the higher budgets.
It is worth noting that when DSLR or Web are the target domains, the top test accuracies are near 100% , meaning there is little room for variation.
To further compare our framework against k -centers on domain adaptation, Table 11 shows the head-to-head accuracies of our model versus the baseline with t -tests to compare for statistical signiﬁcance.
The results here further validate our analysis on Table 10, as we are statistically better than k -centers for 15 out of a total of 42 tests and k -centers statistically outperforms us for only six tests.
More speciﬁcally, we outperform k -centers on ﬁve out of the seven budgets for both DSLR to
Amazon and Web to Amazon, which is the hardest domain to learn, especially since the source data sets are much smaller than the target.
Note that the remaining source-target combinations are easy problems that nominally yield high accuracies with unsupervised domain adaptation, meaning that [Table] the marginal value of labeling points is much less for these tasks.
For Web to DSLR and DSLR to
Web, we are statistically better at smaller budgets whereas k -centers is better for medium budgets
[Table]
Tests where Wass. + EOC + P statistically outperforms k -centers are marked in green and tests where k -centers statistically outperforms Wass. + EOC + P are marked in red
[Table]
Here, we ablate the effect of feature generation by removing unsupervised pre-training and considering classical batch-mode active learning (Sener & Savarese, 2017; Sinha et al., 2019; Shui et al., 2020).
Our baseline, WAAL, is an alternative active learning strategy that also minimizes the
Wasserstein distance but rather by using an adversarial model to estimate the dual distance (Shui et al., 2020).
WAAL is designed speciﬁcally for classical active learning where self-supervised features are not available.
In this experiment, we follow their exact setup to demonstrate (i) that optimization is effective over a greedy approach such as WAAL even in the high-budget setting, and (ii) that optimization is still effective when given weaker features via supervised learning as opposed to powerful self-supervised features.
Methods.
We initialize our labeled data set with a small randomly selected subset.
Then in each round, we ﬁrst train the classiﬁer using supervised learning on the currently labeled pool and then use the ﬁnal layer of the trained classiﬁer to obtain feature embeddings.
These embeddings are then used to solve our optimization problem and select the next B points to label.
We use the available code of Shui et al.
(2020) to implement WAAL without changing any of their parameters.
We use their recommended model, training, and algorithm hyper-parameter settings.
In order to adapt our optimization problem to the high-budget setting, we make one modiﬁcation to our algorithm.
Rather than solving for the entire labeling budget in one problem, we instead [Table] randomly partition the data set into two subsets and solve two optimization problems each for half of the labeling budget respectively.
For example, if we have a previously labeled set of 2 , 000 images, an unlabeled pool of 48 , 000 , and a budget of B = 1 , 000 we randomly split the problem into two sets of 1 , 000 labeled images, 24 , 000 unlabeled images [Table] and B = 500 budgets.
This ensures that our optimization problem remains manageable in computational size.
Furthermore, we implement
Wass.
+ EOC + P, set ( β + , β − ) =
(0 . 6 , 0 . 9) , and a total time limit of T = 4 hours for each problem.
Results.
From Table 4, for both data sets and at all budgets, our approach outperforms the baseline by up to 3% .
Recall that WAAL (i) learns a feature embedding that estimates the Wasserstein distance via dual regularization and (ii) uses a greedy uncertainty strategy to select points that minimize the Wasserstein distance of selected points.
In contrast, we use standard embeddings and formulate an integer program that to select points that minimize the Wasserstein distance.
Our approach can provably obtain optimal solutions.
Our performance highlights the value of optimization via improvement in classiﬁcation accuracy.
Furthermore, we conclude that our optimization problem can be tractably solved in higher budget settings.
E. W ALL - CLOCK A NALYSIS
Table 12 presents the average runtime for each active learning baseline.
The runtimes do not include the ﬁrst cost of generating features, which occurs for each method except for Random.
Nearly every baseline requires only a few seconds to make queries.
The k -medoids baseline usually requires several minutes due to the cost of computing the distance matrices between points.
In contrast, we encourage running our method for at least 20 minutes and up to several hours.
A potential concern for the use of our method in active learning is the long run time (i.e., requiring several hours) in contrast to baselines that typically require on the order of seconds.
In general, combinatorial optimization frameworks for deep learning usually scale poorly with data set size.
For example, the nominal MILP (4) does not even ﬁt into memory for large data sets such as CIFARand SVHN.
GBD simpliﬁes our problem by permitting a variable runtime and easier computational burdern, while still preserving global optimality.
Ultimately, we emphasize that the time cost of selection can be less than model training time and in some applications, less than human labeling time itself (Rajchl et al., 2016).
Furthermore, the selection is entirely automated and ofﬂine from training.
Thus, we suggest that it is more practical to ensure the best possible selection regardless of runtime in order to minimize the real human labor of labeling.
E. A DDITIONAL A BLATIONS
Customizing the Wasserstein distance.
Figure 6 plots accuracy on CIFAR-10 using Wass.
+ EOC with Cosine versus Euclidean distance.
The Euclidean alternative outperforms Cosine distance only when B = 10 whereas Cosine distance outperforms the alternative for higher budgets.
Using Cosine distance yields better downstream accuracy here because the representation learning step, SimCLR, also maximizes a Cosine distance between different images.
Qualitative Analysis.
Figure 7 shows t -SNE visualizations of the latent space in the ﬁrst and second round for STL-10, CIFAR-10, and SVHN, respectively.
In each case, our selection strategy provides a better coverage of the entire latent space by selecting points closer to cluster centers and covering most clusters.
All of the baselines leave large holes in certain regions of the latent space.
Figures 8, 9, and 10 show a sample set of labeled images by the ﬁnal round for each data set.
In each case, our model covers every class with a labeled image within the ﬁrst two rounds.
For example with STL-10, the ﬁrst round labels two images of dogs instead of a cat, and so the second round corrects for this issue by identifying and labeling two cats.
We observe the same trend with airplanes and cars for CIFAR-10.
Furthermore, we select images from nearly every class in each round.
Figure 8 compares our method versus k -centers, k -medoids, and Random for STL-10.
Compared to the baselines, we identify and label examples from more classes in the ﬁrst rounds.
While we capture labeled examples of all classes by the second round, the k -centers baseline does not label a dog until the third round.
Note that in the ﬁrst round, the Random baseline includes three images of dogs, meaning that this baseline at small budgets cannot accurately select a diverse set of points.
In particular in the ﬁrst rounds, our approach selects unobstructed images that capture the silhouette of each class.
For example in the ﬁrst round we label unobostructed full examples of horse, bird, dog, car, deer, and airplane, whereas the k -centers baseline includes unobstructed bird, deer, airplane, and horse, the k -medoids baseline includes only an unobstructed dog and car, and the Random baseline includes an unobstructed ship, dog, and truck.
Finally, the pictures selected by all three non-random strategies appear brighter and sharper on average than the Random baseline.
Figure 9 compares our method versus the three baselines for CIFAR-10.
The samples here display similar trends to the STL-10 example: (i) we more quickly label a diverse set of classes, (ii) our approach identiﬁes unobstructed, clear images in the ﬁrst rounds, and (iii) the non-random alternatives select brighter images more often.
Speciﬁcally, we cover all classes within the second round; in contrast, the k -medoids baseline requires three rounds to cover all classes.
Furthermore in the ﬁrst round, we obtain unobstructed images of a dog, horse, deer, frog, truck, and airplane, while the k -centers baseline obtains an unobstructed airplane, bird, truck, and horse, and the k -medoids baseline obtains an unobstructed frog, horse, truck.
Finally, Figure 10 compares our method versus the three baselines for SVHN.
Here in the ﬁrst round, our model more often selects different classes (i.e., 0, 1, 2, 3, 4, 5, 7, 9) versus the baselines.
Furthermore, in the early rounds, our images cover a variety of different backgrounds including blue, red, and yellow.