D ISCREPANCY -B ASED A CTIVE L EARNING FOR D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The goal of the paper is to design active learning strategies which lead to domain adaptation under an assumption of Lipschitz functions.
Building on previous work by Mansour et al.
(2009) we adapt the concept of discrepancy distance between source and target distributions to restrict the maximization over the hypothesis class to a localized class of functions which are performing accurate labeling on the source domain.
We derive generalization error bounds for such active learning strategies in terms of Rademacher average and localized discrepancy for general loss functions which satisfy a regularity condition.
A practical Kmedoids algorithm that can address the case of large data set is inferred from the theoretical bounds.
Our numerical experiments show that the proposed algorithm is competitive against other state-of-the-art active learning techniques in the context of domain adaptation, in particular on large data sets of around one hundred thousand images.
I NTRODUCTION
Machine learning models trained on a labeled data set from a source domain may fail to generalize on new target domains of interest (Saenko et al., 2010).
This issue, which can be caused by domain shift , can be handled when no target labels are available through unsupervised domain adaptation methods (Ganin et al., 2016).
Using a small sample of labeled target data can, besides, greatly improve the model performances (Motiian et al., 2017).
Acquiring such new labels is often expensive (Settles, 2010) and one seeks to query as few labels as possible.
This explains why strategies of optimal labels acquisition, referred as active learning (Cohn et al., 1994) seem very promising for domain adaptation (Su et al., 2020).
Active learning is a challenging task and a broad literature exists.
On the one hand, some active learning methods introduce heuristic approaches which provide the benefit of using practical algorithm based on simple criteria.
For instance, the spatial coverage of the target domain (Hu et al., 2010;
Bod´o et al., 2011) or the minimization of target uncertainties (RayChaudhuri & Hamey, 1995; Gal et al., 2017) are considered, as well as combination of these heuristics (Wei et al., 2015; Kaushal et al., 2019).
However finding the proper heuristics is not straightforward and previous methods do not link their query strategy with the target risk (Viering et al., 2019).
On the other hand, active learning methods based on distribution matching aim at minimizing a distribution distance between the labeled set and the unlabeled one (Balcan et al., 2009; Wang & Ye, 2015; Viering et al., 2019).
These methods provide theoretical guarantees on the target risk through generalization bounds.
However the computation of the distances is either not scalable to large-scale data sets (Balcan et al., 2009;
Wang & Ye, 2015; Viering et al., 2019) or based on adversarial training (Su et al., 2020; Shui et al., 2020) which involves complex hyper-parameter calibration (Kurach et al., 2019).
In this work, we propose to address the issue of active learning for general loss functions under domain shift through a distribution matching approach based on discrepancy minimization (Mansour et al., 2009).
In Section 2, we derive theoretical results by adopting a localized discrepancy distance (Zhang et al., 2020) between the labeled and unlabeled empirical distributions.
This localized discrepancy is defined as the plain discrepancy considered on a hypothesis space restricted to hypotheses close to the labeling function on the labeled data set.
This distance has the benefit to focus only on relevant candidates for approximating the labeling function and thus provides tighter bound of the target risk under some given assumptions (Zhang et al., 2020).
Based on this distance, we provide a generalization bound of the target risk involving pairwise distances between sample points (Theorem 1).
Inspired by this generalization error bound, we propose in Section 3 an accelerated K-medoids query algorithm which scales to large data set.
In Section 4, we present the related works and analytically show that our proposed approach displays tighter theoretical control of the target risk than the one provided by recent active learning methods.
We finally present in Section 5 the benefit of the proposed approach on several empirical regression and classification active learning problems in the context of domain adaptation.
D ISCREPANCY B ASED A CTIVE L EARNING
Setup and theory develop in this section mainly focus on regression tasks.
Section 4.3 presents how the algorithm derived from the theoretical results can be extended to classification tasks.
2. S ETUP AND DEFINITIONS
Given two subsets X ⊂ R p and Y ⊂ R q and d : X ×X → R + a distance on X , we denote the source data set S = { x 1 , ..., x m } ∈ X m and the target data set T = { x ′ 1 , ..., x ′ n } ∈ X n .
We consider the domain shift setting where the respective data sets S and T are drawn according to two different distributions Q and P on X .
We consider a loss function L : Y ×
Y
→ R + and a hypothesis space
H of k -Lipschitz functions from X to Y .
We denote by L D ( h, h ′ )
=
E x ∼ D
[ L ( h ( x ) , h ′ ( x ))]
the average loss (or risk) over any distribution D on X between two hypotheses h, h ′ ∈ H .
We also define the expected Rademacher complexity of H for the distribution P as:
[Equation]
with σ i drawn according to U the uniform distribution on {− 1 , 1 } .
We consider a labeling function for each distribution: f Q : X → Y and f P : X → Y .
For adaptation to be possible, the two labeling functions are supposed to be close (Mansour et al., 2009).
We finally consider the single-shot batch active learning framework (Viering et al., 2019) where all queried data are picked in one single batch of fixed budget of K > 0 queries.
In this framework, an active learning algorithm takes as inputs the source data set S along with its corresponding recorded labels as well as the target data set T .
The algorithm then returns a batch of K queried target data denoted T K ⊂ T .
The corresponding labels for T K are then recorded through an oracle and used along with the source labeled data to fit an hypothesis h ∈ H .
We denote by L K = S ∪ T K the labeled data set.
The goal is to select the K target data to label in order to minimize the target risk of h : L P ( h, f P ) .
2. L OCALIZED DISCREPANCY
To formulate the problem of active learning under domain shift as a distribution matching problem, one needs to consider a measure of divergence between distributions.
Recent interest focuses on the discrepancy (Mansour et al., 2009) which proves to be useful for domain adaptation (Cortes &
Mohri, 2014; Zhang et al., 2019) and is recently used on the active learning setting (Viering et al., 2019).
However this metric, defined as a maximal difference between domain losses over the whole hypothesis space, is relatively conservative as it includes hypotheses that the learner might not ever consider as candidates for the labeling function (Cortes et al., 2019c; Zhang et al., 2020).
Based on this consideration, we introduce a localized discrepancy (Zhang et al., 2020) to restrict the measure of divergence between domains on a set of relevant candidate hypotheses for approximating the labeling function:
Definition 1.
Localized Discrepancy.
Let K > 0 be the number of queries and ϵ ≥ 0 .
Let T K be a queried batch of size K , the empirical distributions of L K = S ∪ T K and T are respectively denoted (cid:98) Q K and (cid:98) P .
Let H be a hypothesis space and L a loss function.
The localized discrepancy is defined as:
[Equation]
with H Kϵ
= { h ∈ H ; L ( h ( x ) , f Q ( x ))
≤ ϵ ∀ x ∈ L K }
The localized space H Kϵ includes hypotheses ”consistent” with f Q on the labeled data, i.e. hypotheses fitting the labeled data with an error below ϵ .
The parameter ϵ drives the size of H Kϵ .
Obviously, ϵ needs to be high enough to ensure that H Kϵ is not an empty subset.
Under appropriate assumptions on ϵ , a preliminary result is an empirical target risk bound for the localized discrepancy:
Proposition 1.
Let K > 0 be the number of queries and H a hypothesis space.
Let (cid:98) P and (cid:98) Q K be the empirical distributions of the respective sets T and L K = S ∪ T K of respective size n and m + K .
We assume that L is a symmetric, µ -Lipschitz and bounded loss function verifying the triangular inequality.
We denote by M the bound of L .
Let η H be the ideal maximal error on S ∪ T :
[Equation]
Then, for any ϵ ≥ η H , any hypothesis h ∈ H Kϵ and any δ > 0 , the following generalization bound holds with at least probability
1- δ :
[Equation]
This bound share a similar form than the one derived in Cortes et al. (2019c) for the discrepancy.
The ideal maximal error η H characterizes the difficulty of the adaptation problem as well as the ability to learn the labeling functions with the considered hypothesis set H .
If f P and f Q significantly differ on S ∪ T , η H will be high and the proposed bound will lack of informativeness.
We then follow the common domain adaptation assumption that the difference between the two functions is small (Mansour et al., 2009; Zhang et al., 2020).
2. M AIN RESULTS : GENERALIZATION BOUNDS FOR ACTIVE LEARNING
Considering the previous bound (Proposition 1) it appears that a natural way of choosing the K queries in an active learning perspective is to pick the target data minimizing the localized discrepancy.
Unfortunately this is a difficult problem for an arbitrary functional space H , since it leads to compute a maximum over the set space H Kϵ .
Our main idea is then to further bound the localized discrepancy with a computable criterion:
Theorem 1.
Let K > 0 be the number of queries, H a hypothesis space of k -Lipschitz functions and ϵ ≥ η H .
Let L K = S ∪ T K be the labeled set and T the target set drawn according to P .
We assume that L is a symmetric, µ -Lipschitz and bounded loss function verifying the triangular inequality.
We define M such that L ( y, y ′ )
≤ M for any y, y ′ ∈ Y . For any hypothesis h ∈ H Kϵ and any δ > 0 , the following generalization bound holds with at least probability
1- δ :
[Equation]
With d ( x ′ , L K ) =
min x ∈ L K d ( x ′ , x ) and d ( x ′ , x ) the distance from x to x ′ .
Visual insights to understand Theorem 1 are presented in Figure 1 : the main idea is to approximate the maximal hypotheses h, h ′ ∈ H
Kϵ returning the localized discrepancy by the k -Lipschitz envelope of the labeling function f Q , consistent with f Q on the labeled points, i.e. at most ϵ close to f Q on these points.
Indeed, for any target point x ′ ∈ T , the gap between h, h ′ on x ′ : L ( h ′ ( x ′ ) , h ( x ′ )) is upper bounded by twice the distance from x ′ to its closest labeled point times the Lipschitz constants of L, h and h ′ plus the error on the labeled point :
L ( h ′ ( x ′ ) , h ( x ′ )) ≤ 2 kµ d ( x ′ , L K ) + 2 ϵ .
The generalization bound of Theorem 1 highlights the trade-off that exists between the Lipschitz constant k of the hypothesis space H and the parameters η H and ϵ .
To reduce the term 2 ϵ + η H in order to obtain tighter controls over the target risk, one needs to consider a more complex set of hypothesis and thus to increase the Lipschitz constant k .
Then, to benefit from the theoretical guarantees of this bound, a careful choice of hypothesis set have to be made (cf Section 4.3).
The important benefit of the derived bound is to bring out the bounding criterion (cid:80) x
∈T d ( x ′ , L K ) which is independent of the hypothesis complexity (characterized by k ) and the loss function, it only involves pairwise distances between sample points.
As this criterion is computable and depends on the queried batch T K , we can then propose an active learning strategy.
D ISCREPANCY -B ASED A LGORITHM
Theorem 1 directly implies that selecting the K queries minimizing (cid:80) x
∈T d ( x ′ , L K ) leads to minimize an upper bound of the target risk.
We can thus propose an algorithm dedicated to active learning which provides theoretical guarantees on the target risk.
Seeking the K queries minimizing (cid:80) x
∈T d ( x ′ , L K ) corresponds to solve a K-medoids problem (Kaufmann & Rousseeuw, 1987).
Notice however that it does not consist of a K-medoids performed directly on the target domain as the source data are already labeled and considered as medoids.
The presented algorithm only differs in the initialization process, where the distance between each target and its nearest source neighbour needs to be computed.
Several algorithms exist to solve or approximate the K-medoids (Kaufman & Rousseeuw, 2009), (Ng & Han, 2002), (Park & Jun, 2009).
Here we use the greedy version of the algorithm. It can be shown that the gain of the greedy algorithm (the amount of decrease of the criterion by selecting K points) is at least a ( 1 − 1 /e )-approximation of the optimal gain (cf Appendix).
It is well known that K-medoids algorithms suffer from computational burdens or memory issues on large and moderately large data sets ( ∼ 100 K data) (Newling & Fleuret, 2017).
Indeed they require to compute huge pairwise distance matrix between the source and target samples as well as between targets.
Precisely, the greedy K-medoids algorithm presents a complexity of O ( p ( nm + n 2 ) + Kn 2 ) and a memory usage of O ( nm + n 2 ) with m, n, K the respective size of S , T and T K .
p is the dimension of X .
To handle this issue, we propose an adaptation of the K-medoids greedy algorithm with better scalability (Algorithm 1).
This algorithm performs the following steps:
1) Computation of the distance to the closest source for each target points via a KD-trees random forest algorithm (Silpa-Anan & Hartley, 2008) of complexity O ( T ( m + pn ) log( m ) 2 ) with T the number of trees.
Medoids initialization using the greedy algorithm on a random target batch of B samples with complexity O (( K + p ) B 2 ) .
An iterative algorithm on the model of (Park & Jun, 2009), combining assignment of each target point to its closest medoid ( O ( Kpn ) ) and medoid update inside each cluster.
The medoid update for each cluster is done through an original Branch-and-Bound algorithm (Land & Doig, 2010) which estimates the criterion by iteration over mini-batch, target points for which criterion is bigger than a statistical threshold are left aside.
Thus, the number of pairwise
distances to compute is reduced at each iteration until the maximal iteration number is reached or all target points are left aside.
Under some assumptions, we can show that the complexity of the update for all cluster is O ( n 3 / 2 K − 1 / 2 ) (the proof is given in the supplementary material).
The overall complexity is then O ( T ( m + pn ) log( m )
+ ( K + p ) B 2 + Kpn + pn 3 / 2 K − 1 / 2 ) which provides reasonable computational time for moderately large data set ( n, m ∼ 10 5 and p ∼ 10 3 ).
Empirical comparison of computational time is also provided in the supplementary material.
1: { d ( x ′ , S ) } x ′ ∈T ← KDT-Forest-Nearest-Neighbour ( S , T ) # Compute distances to source 2: T K = { x ∗ i } i =1 .. K ← Kmedois-Greedy ( T , B, K, { d ( x ′ , S ) } x ′ ∈T )
# Initialize medoids 3: C i ← { x ′ ∈ T | d ( x ′ , x ∗ i ) =
min j =1 ..K d ( x ′ , x ∗ j ) ∧ d ( x ′ , S ) } # Assign targets to closest cluster 4: while any medoid x ∗ i can be updated do 5: for i from 1 to K do 6: x ∗ i = argmin x ′ ∈ C
i (cid:80)
[Table]
[Table]
d (
[Table] [Table]
← Branch-and-Bound ( C i )
# Update medoid 7: end for 8:
For all x ′ ∈ T , compute d ( x ′ , x ∗ i ) for all updated medoids and reassign x ′ to the closest C i 9: end while
R ELATED W ORK AND D ISCUSSION 4. R ELATED WORK
Active learning as distribution matching.
Active learning methods based on distribution matching aim at reducing the gap between the distributions of the labeled sample and the unlabeled one with a minimal query budget.
Several metrics are used to measure the gap between distributions as the Transductive Rademacher Complexity (Gu & Han, 2012), the MMD (Wang & Ye, 2015; Kim et al., 2016;
Viering et al., 2019) the Disagreement Coefficient (Hanneke, 2007; Balcan et al., 2009;
Beygelzimer et al., 2009; Cortes et al., 2019a;b; 2020), the H -divergence (Sinha et al., 2019; Gissin & Shalev-Shwartz, 2019; Su et al., 2020) or the Wasserstein distance (Shui et al., 2020).
To the best of our knowledge, only one paper deals with the discrepancy for active learning (Viering et al., 2019).
The authors consider the discrepancy on the space of RKHS hypotheses with PSD kernels and provide an explicit way of computing the discrepancy using eigen-value analysis.
However, the corresponding algorithm encounters computational burden and could hardly be applied on large sets.
K-medoids for active learning.
Many active learning methods use a K-medoids algorithm as an heuristic measure of representativeness (Lin et al., 2009; Gomes & Krause, 2010; Wei et al., 2013;
Iyer & Bilmes, 2013; Zheng et al., 2014).
The K-medoids is in general computed on a smaller set of selected targets with the higher uncertainties (Wei et al., 2015; Kaushal et al., 2019).
In this present work, we provide theoretical insights for this algorithm by highlighting the link with discrepancy minimization.
Active learning for domain adaptation.
Our work is related to the recent advances on active learning for domain adaptation as we also consider the domain shift hypothesis (Rai et al., 2010; Saha et al., 2011; Deng et al., 2018;
Su et al., 2020).
These works use in general the output of a domain classifier to measure the informativeness of target samples.
In our work, we consider instead the distance to the source sample to capture informative target data.
Lipschitz consistent functions for active learning.
In the context of function optimization, some methods consider the set of Lipschitz or locally Lipschitz functions consistent with the observations (Valko et al., 2013; Grill et al., 2015; Malherbe & Vayatis, 2017).
We use similar functions to approximate the maximal hypotheses returning the localized discrepancy.
Notice that the goal of the previous papers differ from ours as they aim at finding the maximum of the labeling function.
4. C OMPARISON WITH EXISTING GENERALIZATION BOUNDS FOR ACTIVE LEARNING
Some previous works on active learning also propose theoretical bound on the target risk.
In this section, we will compare them with our derived bound of equation (4) in order to relate our contribution with existing works.
As their framework is the pure active learning setting which differs from the setting consider here, we make the comparisons under the simplifying assumptions:
f = f P = f Q , f ∈ H and ϵ = 0 .
In this case, our bound is written:
[Equation]
The paper by Sener & Savarese (2018) proposes the K-centers algorithm for active learning based on an easily computable criterion offering theoretical guarantees.
For a regression loss L and under the previous assumptions, it controls the target risk as follows:
[Equation]
We directly observe that this bound is looser than the one of equation (5).
Indeed, in our case the target risk is controlled with the mean of distances between unlabeled points and the labeled set whereas K-centers considers the maximum of these distances.
Notice however that both algorithms use greedy selection approximation. Thus, in some cases, the queries from K-centers may lead to a smaller bound than the ones from K-medoids.
A generalization bound for active learning involving the Wasserstein distance W 1 has also been proposed (Shui et al., 2020).
An adaptation of this bound for the proposed scenario with the aforementioned assumptions could be written as follows:
[Equation]
Where W [Equation] ( (cid:98) Q [Equation]
= arg
[Equation]
K (cid:80) x ′ ∼ (cid:98) P γ xx ′ d ( x, x ′ ) with Γ = { γ ∈ R n × ( m + K ) ; γ 1 = 1 n 1 ; γ T 1 = 1 m + K 1 } .
Thus, in observing that d ( x, x ′ ) ≥ d ( x ′ , L K ) for any x ′ ∈ T and x ∈ L K we can show that our presented bound of equation (5) is tighter than the one above.
4. D ISCUSSION ABOUT THE ASSUMPTIONS AND LIMITATIONS
One crucial point of the present work is the setting of the ϵ parameter of the hypothesis set H Kϵ .
In practice, the parameter ϵ is determined by the hypothesis set and the training algorithm that the learner considers.
For instance, if the learner uses over-parameterized hypotheses overfitted on the labeled data set, the parameter ϵ will be small because for any h , h ( x ) ≃ f Q ( x ) on L K .
This could leads to ϵ <
η H and the bound would not be valid anymore.
This highlights the trade-off between fitting the source data and generalizing to the target domain.
As, in practice, the parameter η H is hard to estimate, choosing larger ϵ (by regularizing the hypotheses) is safer, but will lead to larger bounds.
To deal with this difficulty, we use, in our experiments, a set of neural networks H regularized through weight clipping.
The value of the clipping parameter is directly linked to the Lipschitz constant of H .
By selecting an adequate clipping parameter and network architecture, we ensure that H is complex enough to learn the task on the source domain but sufficiently regularized to avoid over-fitting (and thus avoid ϵ < η H ).
For this purpose, we select the architecture and the clipping parameter through validation on the source labeled data.
Regularity assumptions on the loss function L are essentially verified by norms as the L p which are common losses for regression problems.
However, they are are not verified by most classification losses.
In fact, classification loss as the cross-entropy is bounded between 0 and 1 and can not increase linearly with the distance to the closest labeled point.
In this context, considering target points far away from sources as informative points is not efficient.
In fact, the most interesting points are the ones in the margin between classes (Balcan et al., 2007).
Thus, in order to focus the K-medoids selection in the margin, and thus extent the proposed algorithm to classification task, we propose an improved version of our algorithm, the Weighted K-medoids (K-medoids+W) .
This algorithm performs the K-medoids algorithm with a weighted criterion.
To compute the weights, we consider the Best-vs-Second-Best (BVSB) criterion (Joshi et al., 2009) which is the difference between the probabilities of the best class and the second best class, given by a hypothesis pre-trained on the source data set.
The Weighted K-medoids optimization can be written as follows:
[Equation]
E XPERIMENTS
We choose to compare the performances of our algorithm to classical active learning methods on regression and classification problems in a domain shift context.
We consider the single-batch active learning setting (Viering et al., 2019) where all queries are taken at the same time in one batch.
We compare the results obtained on the target domain for different query and training methods.
The experiments have been run on a ( 2 . 7 GHz, 16 G RAM) computer.
The source code is provided on
GitHub 1 .
We use the open source code of the corresponding authors for BADGE (Ash et al., 2019) and the implementations from ADAPT 2 (de Mathelin et al., 2021) for the domain adaptation methods.
5. C OMPETITORS
We compare the proposed approach with the following query methods : Random Sampling ;
Kmeans (Hu et al., 2010); K-centers (Sener & Savarese, 2018); Diversity (Jain & Grauman, 2016);
QBC (RayChaudhuri & Hamey, 1995); BVSB (Joshi et al., 2009); AADA (Su et al., 2020) : an hybrid active learning method for domain adaptation using a combination of entropy measure from a classifier and the outputs of a domain discriminator; BADGE (Ash et al., 2019): an hybrid deep active learning method optimizing for both uncertainty and diversity.
CLUE (Prabhu et al., 2020) an active domain adaptation strategy that select instances that are both uncertain and diverse.
We select four different training methods Uniform Weighting ;
Balanced Weighting :
Assign balanced total weight between source and target instances; TrAdaBoost (Pardoe & Stone, 2010) :
Transfer learning regression method based on a reverse boosting principle; Adversarial training : unsupervised features transformation on the model of DANN (Ganin et al., 2016).
To make a fair comparison between the different query strategies, we use for all experiments, the same set of training hypotheses H .
We define H as the set of neural networks composed of two fully connected hidden layers of 100 neurons, with ReLU activations and projection constraints on the layer norms ( < 1 . ).
We use the Adam optimizer (Kingma & Ba, 2015).
The network architecture is defined to be complex enough that the network provides a good approximation of the labeling function on the source domain in all experiments.
Besides, for each experiment, fine-tuning of the optimization hyper-parameters (epochs, batch sizes...) is performed using only source labeled data.
We assume that the architecture and the resulting hyper-parameters will still be appropriate after adding the queried target data to the training set (see Section 4.3).
Finally, for distance-based algorithm, we consider the L 1 distance computed in the penultimate layer of a network pre-trained on sources.
We use an ensemble of 10 models in QBC and the greedy version of K-centers.
5. S UPERCONDUCTIVITY DATA SET
As there is very few public data sets for domain adaptation with regression tasks (Teshima et al., 2020), we choose an UCI data set with a reasonable amount of instances and split it in different domains using the setup of (Pardoe & Stone, 2010).
We choose Superconductivity (Hamidieh, 2018;
Dua & Graff, 2017) which is composed of features extracted from superconductors chemical formula.
The task consists in predicting their critical temperature.
Experimental setup: The data set is divided in four domains following (Pardoe & Stone, 2010) : low ( l ), middle-low ( ml ), middle-high ( mh ) and high ( h ) of around 4000 instances and 166 features.
We use a learning rate of 0 .
001 , a number of epochs of 100 , a batch size of 128 and the mean squared error as loss function.
We conduct an experiment for the 12 pairs of domains.
We vary K from 5 to 300 and repeated each experiment 8 times with different random seeds.
We report the mean absolute error (MAE) on the target unlabeled data for all experiments when K = 20 in Table 1.
We present the MAE evolution for the adaptation from mh to h in Figure 2.
Results: We observe on Figure 2 that, for any K > 0 , the K-medoids algorithm presents the lowest
MAE on the target data for the three different training methods.
In particular we observe a significant performance gain of K-medoids when using TrAdaBoost which provides the lowest MAE for the majority of fixed budget K compared to other training methods.
These observations are confirmed on
Table 1 where we observe that K-medoids presents the lowest MAE in 11 experiments over 12 .
We
[Table]
also observe here that methods based on spatial consideration as K-medoids, K-means and K-centers select more informative target points than the uncertainty based method QBC.
This comes from the fact that, in batch mode, QBC is selecting close target points with similar uncertainty level.
Finally,
K-medoids outperforms K-means because it takes into account the distance to source points and then queries less redundant information
[Table]
( mh → h experiment).
Evolution of the MAE in function of the budget K for three different training methods and six query methods.
5. O FFICE DATA SET
The office data set (Saenko et al., 2010) consists in pictures of office items coming from different domains: amazon or webcam.
The task is a multi-classification problem with 31 classes (chairs, printers, ...).
The goal is to use data from the amazon domain where labels are easily available to learn a good model on the webcam domain where a few labels are chosen using active learning methods.
Experimental setup: We consider the adaptation from ”amazon” with 2817 labeled images to ”webcam” with 795 unlabeled images.
We use, as input features, the outputs of the ResNetnetwork (He et al., 2016) pretrained on ImageNet.
We vary K from 5 to 300 , repeating each experiment 8 times.
The learning rate is 0 .
001 , the number of epochs 60 and the batch size 128 .
Results: Figure 3.A presents the results obtained.
We observe that the K-medoids+W algorithm provides the best performances for almost any K and in particular for small values of K .
We then present the visualization of the two first components of the PCA transform on Figure 4.
We observe that the K-medoids+W algorithm queries points at the center of the target distribution but at a reasonable distance from the sources.
The Random and K-means algorithms select a representative subset of the target distribution but without taking into account the sources and therefore query redundant information.
K-centers selects data far from the source domain but which are less representative of the the target distribution.
5. D IGITS DATA SET
We consider the experiment proposed in (Ganin et al., 2016) where a synthetic digits data set: SYNTH is used to learn a classification task for a data set of real digits pictures: SVHN (Street-View House
Number)
(Netzer et al., 2011).
Both data sets are composed of around 65 k images of size 28 × 28 .
Experimental setup: To handle the large number of data, we use the accelerated K-medoids algorithm (Algorithm 1) with T = 50 trees and a initial batch size of B = 5000 .
We use the KD-trees random forest nearest neighbour algorithm in Diversity and K-centers to approximate the distance of each target data to the source data set.
We consider two kinds of input features: the ones obtained with the convulational part of a Lenet (LeCun et al., 1998) trained with the source labeled data and the ones coming from the same network but trained with adversarial training following the model of
DANN (Ganin et al., 2016).
In both cases, the network is pre-trained on 30 epochs with a batch size of 128 and a learning rate of 0 .
001 , for the adversarial training the trade-off parameter λ is set to 0 .following
the setup of (Su et al., 2020).
After the query process, a Balance Weighting training is performed with the source and target labeled data using the same optimization hyper-parameters than before.
Experiments are conducted 8 times for K between 10 and 500 .
Results: Figure 3.B and 3.C correspond to the evolution of accuracy with respect to K for the experiments conducted with the features obtained respectively without and with adversarial training.
We observe that, for any K > 0 , K-medoids+W provides improved results over other query strategies in both cases.
This highlights the ability of the method to select informative target points in a variety of scenarios.
C ONCLUSION AND F UTURE W ORK
This work introduces a novel active learning approach based on a localized discrepancy between the labeled and unlabeled distributions.
We provide both theoretical guarantees of this approach and an active learning algorithm scaling to large data sets.
Several experiments show very competitive results of the proposed approach.
Future work will focus on considering a more appropriate distance on the input space, giving more importance to relevant features with respect to the task.
R EPRODUCIBILITY S TATEMENT
To help the reproducibility of the results presented in this work, the source code of the experiments is available in an anonymous repository at https://github.com/AnonymousAccount0/dbal.
Furthermore, the presented methods and the majority of the competitors have been implemented with ”pythonic” objects which implement a fit and predict methods.
Thus, the code can easily be used on other data sets than the ones considered in this work.
Finally, notebooks are provided in the repository to enable a rapid access to the methods and the possibility to try for different hyper-parameters.
Appendix
We recall here the notations and definitions used in the following:
• X ⊂ R p and Y ⊂ R are the respective input and output subsets.
• d : X × X
→ R + is a distance on X .
• P and Q are two distributions on X . • T = { x ′ 1 , ..., x ′ n } ∈ X n is the unlabeled target data set and S
= { x 1 , ..., x m } ∈
X m the labeled source data set drawn respectively from P and Q .
• L :
Y × Y → R + is a loss function.
• H is a hypothesis set of k -Lipschitz functions from X to Y .
• E x ∼ D
[ L ( h ( x ) , h ′ ( x ))] is the average loss (or risk) over any distribution D on X between two hypotheses h, h ′ ∈ H .
• R n ( H ) =
E { x ′ i } i ∼ P (cid:20) E { σ i } i ∼ U
(cid:20) sup h ∈ H 1 n n (cid:80) i
=
σ i h ( x ′ i ) (cid:21)(cid:21) , is the expected Rademacher complexity of H , with U the uniform distribution on {− 1 , 1 } .
• f Q : X → Y is the source labeling function.
• f P : X → Y is the target labeling function.
• K > 0 is the number of queries • T K ⊂ T with |T K | =
K is a queried batch or subset.
• L K = S
∪ T K is the labeled data set.
• (cid:98) Q , (cid:98) Q K and (cid:98) P are the respective empirical distributions on X of S , S ∪ T K and T .
• H Kϵ = { h ∈ H ; L ( h ( x ) , f Q ( x ))
≤ ϵ ∀ x ∈ L K } is the localized hypothesis space .
• disc H Kϵ ( (cid:98) Q K , (cid:98) P ) =
max h,h ′ ∈
H Kϵ |L (cid:98) Q K ( h, h ′ )
− L (cid:98) P ( h, h ′ ) | is the localized discrepancy between (cid:98) Q K and (cid:98) P .
A P ROOF OF P ROPOSITION
Proposition 1.
Let K > 0 be the number of queries and H a hypothesis space.
Let (cid:98) P and (cid:98) Q K be the empirical distributions of the respective sets T and L K = S ∪ T K of respective size n and m + K .
We assume that L is a symmetric, µ -Lipschitz and bounded loss function verifying the triangular inequality.
We denote by M the bound of L .
Let η H be the ideal maximal error on S ∪ T :
[Equation]
Then, for any ϵ ≥ η H , any hypothesis h ∈ H Kϵ and any δ > 0 , the following generalization bound holds with at least probability
1- δ :
[Equation]
Proof.
Let’s consider h ∈ H .
According to (Mohri et al., 2018)
we have for any δ > 0 , with probability at least 1 − δ :
[Equation]
Besides, we have for any h, h 0 ∈ H Kϵ :
[Equation]
As the inequality is true for any h 0 ∈ H Kϵ
[Equation]
we have in particular:
[Equation]
We notice that, for any h 0 ∈ H Kϵ :
[Equation]
From which we deduce that,
[Equation]
Let’s now consider h ∗ ∈ H , such that:
[Equation]
By assumption we have η H ≤ ϵ , and thus:
[Equation]
This implies that for any x ∈ S ∪ T :
[Equation]
In particular, L ( h ∗ ( x ) , f Q ( x )) ≤ ϵ for any x ∈ L K , which implies that h ∗ is in H Kϵ .
We then deduce that:
[Equation]
Thus we conclude that for any h ∈ H Kϵ and any δ > 0 , we have with probability at least 1 − δ :
[Equation]
B P ROOF OF T HEOREM
Theorem 1.
Let K > 0 be the number of queries, H a hypothesis space of k -Lipschitz functions and ϵ ≥ η H .
Let L K = S ∪ T K be the labeled set and T the target set drawn according to P .
We assume that L is a symmetric, µ -Lipschitz and bounded loss function verifying the triangular inequality.
We denote by M the bound of L .
For any hypothesis h ∈ H Kϵ and any δ > 0 , the following generalization bound holds with at least probability
1- δ :
[Equation]
With d ( x ′ , L K ) =
min x ∈ L K d ( x ′ , x )
Proof.
Let ϵ ≥ η H and 0 ≤ K ≤ n .
For all h, h ′ ∈ H Kϵ , for all x ′ ∈ T and for all x ∈ L K we have:
[Equation]
[Equation]
The two first inequalities come from the triangular inequality, the others from the lipschitzness of h, h ′ and definition of H Kϵ .
As the above inequality is true for any x ∈ L K , we have in particular for any x ′ ∈ T :
[Equation]
Leading to:
[Equation]
We then deduce the following, for all h, h ′ ∈ H
Kϵ :
[Equation]
Finally, according to Proposition 1, we have for all h,[-h (cid:
-]
h ′ ∈ H K ϵ :
[Equation]
We present in this section the approximation error of the relaxation between the bounds of Proposition 1 and Theorem 1.
We will show that, with the assumption of Theorem 1 and in the case L =
L 1 , we have, for any labeled set L K :
[Equation]
With k f the Lipschitz constant of the source labeling function f Q . We assume that k > k f . For this purpose we will look for two hypotheses h, h ′ ∈ H
Kϵ verifying: [Equation] Let’s define k ′ = k − k f , and h, h ′ :
X → Y such that, for any x ∈ X :
[Equation]
We will now prove that h, h ′ are in H Kϵ :
Let’s consider x 1 , x 2 ∈ X , we define:
[Equation]
Assuming without restriction that d ( x 1 , ˜ x 1 ) >
d ( x 2 , ˜ x 2 ) , we have according to assumption (2):
[Equation]
Using the triangular inequality and the fact that d ( x 1 , ˜ x 1 ) ≤ ( x 1 , ˜ x 2 ) by definition of ˜ x 1 .
Using a similar development we can prove the k -Lipschitzness of h ′ .
Let’s now consider x ∈ L K , we have
[Equation]
In the same way L ( h ′ ( x ) , f ( x {+)) = 
 and we have h, h ′ ∈ H sϵ .+}
Furthermore, we have:
[Equation]
Thus, disc
[Equation]
From which we conclude.
D C OMPARISON WITH OTHER ACTIVE LEARNING BOUNDS ( CF SECTION 4.2)
In this section we assume that f = f P = f Q and f ∈ H with H a set of k -Lipschitz functions.
D. K- CENTER BOUNDS
Sener and Savarese (Sener & Savarese, 2018) propose the following generalization bounds for ϵ =and
h ∈ H Kϵ :
[Equation]
which leads to:
[Equation]
With δ = max x ′ ∈T d ( x ′ , L K ) , C the class number and λ µ the Lipschitz constant of a class-specific regression function.
λ l is the Lipschitz constant of the loss function l verifying l : ( x, h ) → l ( x, f ( x ) , h ) =
L ( h ( x ) , f ( x )) .
If we consider a regression problem, we can drop the term corresponding to the class-specific function and we have:
[Equation]
Let’s now consider h ∈ H and x, x ′ ∈ X , we have for k -Lipschitz f :
[Equation]
The two first inequalities are obtained with triangular inequalities, then we use Lipschitz assumptions on h, f and L .
Thus, we have λ l = 2 µk from which we deduce:
[Equation]
D. W ASSERSTEIN BOUNDS
To adapt the bound from (Shui et al., 2020)
Corollary 1 to our setting, we identify the distributions (cid:98) D and (cid:98) Q from Shui et al.
(2020) with respectively the distributions (cid:98) P and (cid:98) Q K .
We then have the following generalization bound:
[Equation]
Notice that the term corresponding to the labeling function ”decay property” in the bound of Shui et al.
(2020) is null when considering a Lipschitz labeling function ( f ∈ H ).
Thus for ϵ = 0 and for any h ∈ H Kϵ , we have:
[Equation]
With γ ∗ = arg min γ ∈ Γ (cid:80) x ′ ∈T (cid:80) x ∈ L K γ x ′ x d ( x ′ , x ) and
Γ = { γ ∈ R n × ( m + K ) ; γ 1 = 1 n 1 ; γ T 1 = 1 m + K 1 } .
Thus, in observing that:
[Equation]
We deduce that our presented bound of Theorem 1 is also tighter than the one proposed in (Shui et al., 2020) in the case of ϵ = [-
-]0 and k -Lipschitz f and h .
E A LGORITHMS ( CF S ECTION 3) E. K- MEDOIDS
1: Input
2: Output: 3: T ← { x ′ s ..., x ′ s B } { x ′ s 1 , ... x ′ s B } randomly
T replacement.
4: Initialize i do 8: x i ← min x ∈T (cid:80) x
∈T min (cid:16) d xx ′ d x S
← For update d d x [-← (cid:113) 
-]d 11: end do 10:
C ← 1 Bi (cid:80) d ∈D x
( d − C x )13: end for 14: x ∗ ← arg min x ∈ ˜ C C x 15: t ← min( t, C x ∗ + 2 σ x ∗ √ Bi ) 16:
˜ C ← (cid:110) x ∈ ˜ C ; C x − 2 σ x √ Bi < t (cid:111) 17: end for 18: x ∗ ← arg min x ∈ ˜ C C x
E. A PPROXIMATION BOUND FOR THE GREEDY ALGORITHM
This section is dedicated to the proof of the bound for the greedy K-medoids algorithm which is expressed as follows: The gain of selecting K new medoids with the greedy algorithm is an (1 − 1 /e ) -approximation of the optimal gain. Let T K be a batch of K target points selected with the greedy algorithm.
The gain is the difference between the initial objective and the final objective after selecting T K :
[Equation]
With d ( x ′ , S )
=
min x ∈S d ( x ′ , x )
We will prove now that the gain is monotone submodular.
Let A ⊂ B ⊂ T and x ∈ T \ B ,
We denote T A
= { x ′ ∈ T ; d ( x ′ , x ) <
d ( x ′ , A ) } and T B = { x ′ ∈ T ; d ( x ′ , x )
<
d ( x ′ , B ) } .
As A ⊂ B , we have d ( x ′ , A ) ≥ d ( x ′ , B ) for any x ′ ∈ T .
Thus,
T B ⊂ T A (42)
Besides, for any x ′ ∈ T ,
[Equation]
We deduce then that,
[Equation]
Then, the gain is submodular.
Besides, as A ⊂ B it appears clearly that,
[Equation]
A )
[Equation]
B ) (45) and the gain is monotone.
Finally, by noticing that the gain is always positive and according to Nemhauser et al.
, we conclude that the gain of selecting K new medoids with the greedy algorithm is an (1 − 1 /e ) - approximation of the optimal gain.
E. C OMPLEXITY C OMPUTATION
In the following, a distance computation is considered to be done in O ( p ) .
KD-Trees Random Forest : Each of the T trees is built by splitting one sample, at each root, at the median of a random feature until the leaf-sizes are ∼ log( m ) .
The median computation for each root with m r data is in O ( m r log( m r )) .
Thus the overall complexity to build one tree is O ( (cid:80) Mi =0 2 − i m log(2 − i m )2 i ) =
O ( m log( m ) (cid:80) Mi =0 1) with M ∼ log( m/ log( m )) which becomes O ( m log( m ) 2 ) .
Then, each target is assigned to a leaf by performing O (log( m )) computations, inside the assigned leaf all distance computations are done in O ( p log( m )) .
Thus an approximate nearest neighbour is given for all targets with a complexity O
( T ( m + pn ) log( m ) 2 ) .
Medoids Initialization : Using the greedy algorithm, O ( pB 2 ) distance computations are first done, then, for all targets, a sum over the target data is computed at each of the K steps.
Thus the complexity is O
(( K + p ) B 2 ) .
Assignation to the Closest Medoid : The distance between all target and the medoids is computed in O ( Kpn ) .
4. Branch & Bound Medoid Computation
B & B algorithm (Algorithm 3)
takes as input one cluster C of n c unlabeled data from T .
It also takes a batch size B and the previous cluster medoid criterion C ∗ which is used as an initial threshold.
The use of the initialization C ∗ may accelerate the algorithm, in the following we do not take into account this initialization, i.e we consider that C ∗ = + ∞ .
Besides, to use notations consistent with the common notations in statistics, we will denote the batch size B by n ( B ≡ n ).
Notice that it is redundant with the size of the unlabeled data set T .
An explicit mention will be made, if n does not refer to the batch size.
Definitions and notations: Let’s consider one cluster C ⊂ T with n c data.
We consider the uniform norm as underlying distance d , defined for all x i , x j ∈ C as d ( x i , x j ) = max (cid:16) | x (1) i − x (1) j | , ..., | x ( p ) i − x ( p ) j | (cid:17) with x i =
( x (1) i , ..., x ( p ) i )
∈ R p .
Computing the complexity of the B & B algorithm for any distribution of the x i would be too difficult.
We make here the simplifying assumption that the x i in C are uniformly distributed on the hyper-cube C of edge size 2 and centered on x ∗ = (0 , ..., 0)
∈ R p
We define for any i ∈ [ | 1 , n c | ] and any j ∈ [ | 1 , n c | ] , the variables Z ij =
d ( x i , x j ) .
We also define Z ∗ j = d ( x ∗ , x j ) for any j ∈ [ | 1 , n c | ] .
We suppose that for any i ∈ [ | 1 , n c | ] ,
Z i 1 , ..., Z in c are iid and that for any j ∈ [ | 1 , n c | ] the Z ij are independents.
We define, for any i ∈ [ | 1 , n c | ] , the mean µ i = E
[ Z i 0 ]
= 12 p (cid:82) x ∈ C d ( x i , x ) and the variance σ 2 i =
Var [ Z i 0 ]
=2 p (cid:82) x ∈ C ( d ( x i , x ) − µ i ) 2 .
We consider a first batch of distance computations of size n < n C .
We define for any i ∈ [ | 1 , n c | ] the empirical mean (cid:98) µ i = 1 n (cid:80) nj =1 Z ij and the empirical variance (cid:98) σ 2 i = 1 n − 1 (cid:80) nj =
( Z ij − (cid:98) µ i ) 2 .
We denote by µ ∗ and σ ∗ 2 the mean and variance of
Z ∗ 0 and (cid:98) µ ∗ and (cid:98) σ ∗ 2 their respective empirical estimator.
We first observe that µ i and σ i are finite for any i ∈ [ | 1 , n c | ] as the x i are uniformly distributed on the hyper-cube centered in x ∗ ∈ C .
We also notice that Z ij ∈ [0 , 2] for any i, j ∈ [ | 1 , n c | ] .
Preliminary results: We make the assumption that p >> 1 .
We aim at giving bounds for any µ i and σ i .
We admit the intuitive results that for any x i =
( x (1) i , ..., x ( p ) i ) ∈ C and for any j ∈ [ | 1 , n c | ] , µ i ≥ µ j → 0 i and σ i ≥ σ j → 0 i with µ j → 0 i and σ j → 0 i the mean and variance of the variable d ( x j → 0 i , . )
with x j → 0 i
=
( x (1) i , ..., x ( j − 1) i , 0 , x ( j +1) i , ..., x ( p ) i ) .
We consider indeed that the more x i is close to the center of the hyper-cube smaller is µ i and σ i .
Considering this fact, we have, for any i ∈ [ | 1 , n c | ] :
[Equation] [Equation]
Besides, for any 0 ≤ r ≤ 1 the sample density in the elementary surface between the balls centered on x ∗ and of respective radius r + d r and r is pr p − 1 d r .
Thus, we can notice that
Z ∗ 0 follows a beta distribution of parameters α = p and β = 1 , from which we deduce that:
[Equation]
We further notice that for any 0 ≤ a ≤ 1 and for any i ∈ [ | 1 , n c | ] such that d ( x ∗ , x i ) =
a we have µ i ≥ µ a with µ a the criterion of the sample x a = ( a, 0 , ..., 0)
∈ R p .
To compute µ a = 12 p (cid:82) x ∈ C d ( x a , x ) , we split the integral on three parts: d ( x a , x ) ≤ 1 − a , 1 − a ≤ d ( x a , x ) ≤ 1 and 1 ≤ d ( x a , x ) ≤ 1 + a : µ a =
[Equation]
[Equation] [Equation] Using the simplifying approximation pp +1 ≃ p − 1 p ≃ 1 for p >> 1 .
Thus for any 0 ≤ a ≤and for any i ∈ [ | 1 , n c | ] such that d ( x ∗ , x i ) =
a we have:
[Equation]
An upper bound of the σ 2 i is given by the variance σ 2 c of the variable d ( x c , . ) with x c = (1 , ..., 1) which corresponds to one corner of the hyper-cube C :
[Equation]
From which we deduce that for any i ∈ [ | 1 , n c | ] :
[Equation]
To simplify the calculations, we make the approximation (cid:98) σ i ≃ σ i for any i ∈ [ | 1 , n c | ] , which is relevant for sufficiently large n as E [ Z i 04 ] < + ∞ .
Probability of rejecting all optimal medoid candidates: Let ϵ > 0 be an approximation factor of µ ∗ .
The goal of the Branch & Bound algorithm is to identify one sample x i ∈ C such that µ i ≤ µ ∗ (1 + ϵ ) with less distance computations as possible.
The process consists in removing all candidates x i such that (cid:98) µ i − 2 (cid:98) σ i √ n > (cid:98) µ i ∗ + 2 (cid:98) σ i ∗ √ n with (cid:98) µ i ∗ the current minimal empirical mean.
We aim now at computing the probability of rejecting all optimal medoid candidates x i verifying µ i ≤ µ ∗ (1 + ϵ ) during the B & B process.
For this, we define B ϵ ( µ ∗ ) =
{ i ∈ [ | 1 , n c | ] ; µ i ≤ µ ∗ (1 + ϵ ) } the index set of optimal medoid candidates and B cϵ ( µ ∗ ) =
{ i ∈ [ | 1 , n c | ] ; µ i > µ ∗ (1 + ϵ ) } the index set of sub-optimal medoid candidates.
We assume that
B & B returns an optimal candidate if at least one sample of B ϵ ( µ ∗ ) is kept after the first batch computation.
We define the two following probabilities:
[Equation]
We can observe that the probability of rejecting all optimal medoid candidates is upper bounded by: (1 − P 2 ) + P 2 P 1 considering the approximation (cid:98) σ i ≃ σ i and the fact that σ i ≥ σ ∗ for any i ∈ [ | 1 , n c | ] .
We now define, for i ∈ B cϵ ( µ ∗ ) :
[Equation]
Leading to:
[Equation]
On the other hand, according to the Bennett’s inequality from (Maurer & Pontil, 2009;
Hoeffding, 1994) we have, for any i ∈ B cϵ ( µ ∗ ) and for any δ > 0 :
[Equation]
Notice that we apply the inequality to the Z ij / 2 .
Then, considering the fact that σ i ≤ 2 σ ∗ and that µ i > µ ∗ (1 + ϵ ) we have:
[Equation]
Leading to:
[Equation]
Let’s consider δ > 0 such that the following equality holds:
[Equation]
Thus:
[Equation]
With:
[Equation]
We then set:
[Equation]
We have for the δ defines above and for any i ∈ B cϵ ( µ ∗ ) :
[Equation]
Thus, the probability P 1 can be upper bounded as follows:
[Equation]
Considering the fact that |B cϵ ( µ ∗ ) | < n c . 1 − (1 − δ ) n c is the probability of getting at least one success for the binomial law of parameters ( n c , δ ) .
We are now looking for an upper bound of P 2 .
We observe that, at least the index i such that x i = x
∗ is in B ϵ ( µ ∗ ) .
Besides, as x ∗ is the center of the hyper-cube, Z ∗ j is in [0 , 1] for any j ∈ [ | 1 , n c | ] .
We can then apply the Bennett’s inequality to the Z ∗ j , and for any γ > 0 we have:
[Equation]
We set:
[Equation]
We then have:
[Equation]
Leading to:
[Equation]
Finally the probability of rejecting all optimal candidates is upper bounded by 1 − (1 − δ ) n c + γ .
To give an order of magnitude of this probability, we consider the scenario where n c = 10 5 , p = 100 , n = √ n c and ϵ = 0 .
Then we have: δ ≃ 3 .
6 10 − 8 and γ ≃ 7 . 7 10 − 5 , which leads to a probability of rejection around 3 .
6 10 − 3 .
Thus, in this case, there is at least a probability 0 .
995 that B & B returns a medoid candidate with a criterion less than 1 .
the optimal.
Complexity computation: We are now looking for the number of distance computations performed by B & B. For this, we need to compute the number of x i kept at each batch.
An upper bound of this number is given by the number of x i verifying (cid:98) µ i ≤ µ ∗ (1 + ϵ/ 4) + 8 σ ∗ √ n .
We further assume that the previous upper bound can be approximated by the number of x i verifying µ i ≤ µ ∗ (1 + ϵ/ 4) + 8 σ ∗ √ n
We have shown in the preliminary results that for any i ∈ [ | 1 , n c | ] , µ i ≥ 1 + a 2 4 with a = d ( x i , x ∗ ) .
Thus an upper bound of the number of candidates x i kept after the first batch is given by the number of x i in the ball of radius a with a verifying:
[Equation]
Using the approximation µ ∗ = pp +1 ≃ 1 .
Besides, as σ ∗ =
O (1 /p ) we can suppose that for sufficiently large p and large n , 8 σ ∗ √ n ≤ 34 ϵ (For instance with n c , n, p and ϵ considered previously, we have 8 σ ∗ √ n ≃ 0 .
005 and 34 ϵ ≃ 0 .
Thus:
[Equation]
Finally the number of candidates kept after the first batch is in O (cid:0) n c ϵ p/ 2 (cid:1) which is very small (for the values of n c , p and ϵ considered previously, we have n c ϵ p/ 2 ≃ 10 − 60 ).
If we consider a batch size of O (cid:0) √ n c (cid:1) , the number of distance computations after the first batch is negligible behind O (cid:0) n c √ n c (cid:1) .
We then conclude that the complexity of a medoid computation in one cluster is in
O (cid:0) p n c √ n c (cid:1) as each of the K cluster has approximately n c ≃ n/K samples (with n the sample size of T ), the overall complexity of B & B is in O (cid:0) p n 3 / 2 K − 1 / 2 (cid:1) .
F E MPIRICAL C OMPLEXITIES
This section presents the empirical time computation recorded for K-medoids Greedy , Accelerated
K-medoids , K-centers and K-centers + KD-Trees which corresponds to the K-centers algorithm with initialization of the nearest source neighbour distances computed through the KD-Trees Random
Forest algorithm.
The experiments are conducted on the Digits data set.
The experiments are run on a (2.7GHz, 16G RAM) computer using Python 3 . 8 .
The scikit-learn (Pedregosa et al., 2011) implementation of the pairwise euclidean distance is used.
No parallel computing is performed.
The parameters are set to K = 100 , T = 50 for the KD-Trees Random Forest algorithm and an initial batch size B = 5000 for the Accelerated K-medoids algorithm.
The euclidean distance is used as base distance d .
The results are reported on Figure 5.
The evolution of computational time is a function of the size of the source and target samples ( m and n ).
We first observe on Figure 5 that the K-medoids Greedy algorithm encounters computational burden for samples larger than 10 k instances.
For n = m = 60 k, the K-centers algorithm encounters a similar issue due to the computation of the distance matrix between the source and target data sets.
Using the KD-Trees Random Forest algorithm decreases in this case the computational time by a factor of 5 .
We also notice that the accelerated K-medoids algorithm has similar complexity performance to K-medoids for n, m ≤ 5000 which is the maximum size of the initial batch.
Then, the computation time of the algorithm increases slightly from n, m = 5 k to n, m = 60 k while remaining at an acceptable level.
For n, m = 60 k the complexities of the accelerated K-medoids and the K-centers + KD-Trees are almost similar but with a level of performance in favor of the accelerated K-medoids (see Section 5.4).
We also present the evolution of the objective function of the different algorithm in function of the number of queries K for n = m = 60 k
(cf Figure 6). We observe that the objective of the K-center algorithm is higher than the one of the K-medoids algorithms. We also observe that K-medoids Greedy provide slightly smaller objectives than Kmedoids Accelerated.
G E XPERIMENTS G. S UPERCONDUCTIVITY
Setup The UCI data set ”Superconductivity” (Hamidieh, 2018; Dua & Graff, 2017) is composed of features extracted from the chemical formula of several superconductors along with their critical temperature.
There is two kind of features: some features correspond to the chemical element number’s in the superconductor chemical formula, others are statistical features derived from the chemical formula as the mean and variance of the atomic mass.
We use the setup of (Pardoe & Stone, 2010) to divide the data set in separate domains.
We select an input feature with a moderate correlation factor with the output ( ∼ 0 . 3 ).
We then sort the set according to this feature and split it in four parts: low (l), middle-low (ml), middle-high (mh), high (h).
Each part defining a domain with around 4000 instances and 166 features.
The considered feature is then withdrawn from the data set.
A standard scaling preprocessing is performed using the source data on the input statistical features and the output feature.
A max scaling is performed on features corresponding to the chemical element number’s.
A visualization of the first components of the PCA as well as the output distribution is provided in Figure 7).
Standard deviation Table 2 presents the standard deviation of the MAEs obtained on the 8 repetitions of the 12 experiments with the Balanced Weighting training and K = 20 .
We observe that K-medoids provides the smallest standard deviations in the majority of the experiments.
K-medoids is indeed a deterministic algorithm and thus selects a determined batch of target points.
Besides, K-medoids selects the target batch to label in a distribution matching perspective, i.e. it produces a training set with a distribution close to the one of the testing set (cf Section 2).
This can explain why the training is more stable with the training set provided by K-medoids. [Table] [Table]
(high to low) analysis An interesting fact to notice in the results of the superconductivity experiments (for which a recap is given in Table 3) is the asymmetry of results between the low-to-high experiment and the high-to-low experiment.
Indeed, the results on the latter experiment are worse than the ones of the former, except for the Random and K-medoids algorithms
[Table]
It first appears, in Figure 8, that this difference in performance for the low-to-high experiment compared to the high-to-low experiment comes from the difference in the output distributions of the low and high domains.
Indeed, the output distribution of the high domain is made of one mode concentrated in the low temperature while the output distribution of low is made of two modes, one in the low and the other in the high temperature.
Thus, the model trained with the low domain data will generalize better to the high domain since the model has seen the full range of temperature.
In the inverse problem (high-to-low), the model has only seen low temperatures and thus poorly predict a part of the low domain.
Then, for this difficult experiment (high-to-low), the difference in performance between the different query algorithms lies mainly in the number of data with high temperature in the low domain that are queried (see Figure 9).
It appears that K-medoids and Random query more of this type of data, as they are highly concentrated and relatively far from the sources (see Figure 10, 11).
G. O FFICE
[Figure]
[Figure]