W ILDLY U NSUPERVISED D OMAIN A DAPTATION AND I TS P OWERFUL AND E FFICIENT S OLUTION
Anonymous authors
Paper under double-blind review
A BSTRACT
In unsupervised domain adaptation (UDA), classiﬁers for the target domain (TD) are trained with clean labeled data from the source domain (SD) and unlabeled data from TD.
However, in the wild, it is hard to acquire a large amount of perfectly clean labeled data in
SD given limited budget.
Hence, we consider a new, more realistic and more challenging problem setting, where classiﬁers have to be trained with noisy labeled data from SD and unlabeled data from TD—we name it wildly UDA (WUDA).
We show that WUDA ruins all UDA methods if taking no care of label noise in SD, and to this end, we propose a
Butterﬂy framework, a powerful and efﬁcient solution to WUDA.
Butterﬂy maintains four models (e.g., deep networks) simultaneously, where two take care of all adaptations (i.e., noisy-to-clean, labeled-to-unlabeled, and SD-to-TD-distributional) and then the other two can focus on classiﬁcation in TD.
As a consequence, Butterﬂy possesses all the conceptually necessary components for solving WUDA.
Experiments demonstrate that under WUDA,
Butterﬂy signiﬁcantly outperforms existing baseline methods.
I NTRODUCTION
Domain adaptation (DA) aims to learn a discriminative classiﬁer in the presence of a shift between training data in source domain and test data in target domain (Ben-David et al., 2010; Ganin and Lempitsky, 2015;
Xiao and Guo, 2015; Zhang et al., 2015; 2013).
Currently, DA can be divided into three categories: supervised
DA (Tzeng et al., 2015), semi-supervised DA (Guo and Xiao, 2012) and unsupervised DA (UDA) (Saito et al., 2017).
When the number of labeled data is few in target domain, supervised DA is also known as few-shot
DA (Motiian et al., 2017).
Since unlabeled data in target domain can be easily obtained, UDA exhibits the greatest potential in the real world (Ganin and Lempitsky, 2015; Ganin et al., 2016;
Gong et al., 2012; 2016;
Long et al., 2015; Saito et al., 2017; 2018).
UDA methods train with clean labeled data in source domain (i.e., clean source data) and unlabeled data in target domain (i.e., unlabeled target data) to obtain classiﬁers for the target domain (TD), which mainly consist of three orthogonal techniques: integral probability metrics (IPM) (Ghifary et al., 2017;
Gong et al., 2016; Gretton et al., 2012; Lee and Raginsky, 2018; Long et al., 2015), adversarial training (Ganin et al., 2016;
Gong et al., 2018; Hoffman et al., 2018;
Li et al., 2018;
Saito et al., 2018; Tzeng et al., 2017) and pseudo labeling (Saito et al., 2017).
Compared to IPM- and adversarial-training-based methods, the pseudolabeling-based method (i.e., asymmetric tri-training domain adaptation (ATDA) (Saito et al., 2017)) can construct a high-quality target-speciﬁc representation, providing a better classiﬁcation performance.
However, in the wild, the data volume of source domain tends to be large.
To avoid the expensive labeling cost, labeled data in source domain normally come from amateur annotators or the Internet (Lee et al., 2018;
Schroff et al., 2011; Tommasi and Tuytelaars, 2014).
This brings us a new, more realistic and more challenging problem, wildy unsupervised domain adaptation (abbreviated as WUDA, Figure 1).
This adaptation aims to
P s P s
P x t
P s
Existing UDA
T w o - s t e p a pp r
o ac h
T w o - s t e p a p p r
o ac h
Label corruption approach: Butterfly
L a b e l c o rr ec ti on
E x i ti n g U D
A transfer knowledge from noisy labeled data in source domain ( (cid:101) P s , i e., noisy source data) to unlabeled target data ( P x t ).
Unfortunately, existing UDA methods share an implicit assumption that there are no noisy source data .
Namely, these methods focus on transferring knowledge from clean source data ( P s ) to unlabeled target data ( P x t ).
Therefore, these methods cannot well handle the WUDA.
To validate this fact, we empirically reveal the deﬁciency of existing UDA methods (Figure 2).
To improve these methods, a straightforward solution is a two-step approach.
In Figure 1, we can ﬁrst use label-noise algorithms to train a model on noisy source data, then leverage this trained model to assign pseudo labels for noisy source data.
Via UDA methods, we can transfer knowledge from pseudo-labeled source data ( ˆ P s ) to unlabeled target data ( P x t ).
Nonetheless, pseudo-labeled source data are still noisy, and such two-step approach may not eliminate noise effects.
To circumvent the issue of two-step approach, we present a robust one-step approach called Butterﬂy .
In high level, Butterﬂy directly transfers knowledge from (cid:101) P s to P x t , and uses the transferred knowledge to construct target-speciﬁc representations.
In low level, Butterﬂy maintains four networks dividing two branches (Figure 3): Two networks in Branch-I are jointly trained on noisy source data and pseudo-labeled target data (data in mixture domain (MD)); while two networks in Branch-II are trained on pseudo-labeled target data.
The reason why Butterﬂy can be robust takes root in the dual-checking principle (DCP): Butterﬂy checks high-correctness data out, from not only the data in MD but also the pseudo-labeled target data.
After crosspropagating these high-correctness data, Butterﬂy can obtain high-quality domain-invariant representations (DIR) and target-speciﬁc representations (TSR) simultaneously in an iterative manner.
If we only check data in MD (i.e., single checking), the error existed in pseudo-labeled target data will accumulate, leading to low-quality DIR and TSR.
We conduct experiments on simulated WUDA tasks, including 4 MNIST-to-SYND tasks, 4 SYND-to-MNIST tasks and 24 human-sentiment tasks.
Besides, we conduct experiments on 3 real-world WUDA tasks.
Empirical results demonstrate that Butterﬂy can robustly transfer knowledge from noisy source data to unlabeled target data.
Meanwhile, Butterﬂy performs much better than existing UDA methods when source domain (SD) suffers the extreme (e.g., 45% ) noise.
W ILDLY U NSUPERVISED D OMAIN A DAPTATION
In this section, we ﬁrst deﬁne a new, more realistic and more challenging setting called wildly unsupervised domain adaptation (WUDA) , and explain the nature of WUDA.
Then, we empirically show that representative
UDA methods cannot handle WUDA well, which motivates us to propose Butterﬂy (see Section 3).
Deﬁnition 1 (Wildly Unsupervised Domain Adaptation) .
Let X t be a multivariate random variable deﬁned on the space X with respective a probability density p x t , ( X s , Y s ) be a multivariate random variable deﬁned
[Figure]
B on the space X × C with respective a probability density ˜ p s , where C = { 1 , . . .
, K } .
Let p x s be the marginal probability density of ˜ p s .
Given i.i.d.
data ˜ D s
= { ( x si , ˜ y si ) } n s i
=1 and D t
= { x ti } n t
i
=1 drawn from ˜ P s and
P x t , in wildly unsupervised domain adaptation, we aim to train with ˜ D s and D t to accurately annotate data drawn from P x t , where p x s (cid:54) =
p x t .
Remark 1.
In Deﬁnition 1, ˜ D s is noisy source data, D t is unlabeled target data, and ˜ P s and P x t are two probability measures corresponding to densities ˜
p s ( x s , ˜ y s ) and p x t ( x t ) .
Nature of WUDA.
Speciﬁcally, there are ﬁve distributions involved in WUDA setting: 1) a marginal distribution on source data, i.e., p x s in Deﬁnition 1; 2) a marginal distribution on target data, i.e., p x t in
Deﬁnition 1; 3) an incorrect conditional distribution of label given x s , q ( y s | x s ) ; 4) a correct conditional distribution of label given x s , p ( y s | x s ) and 5) a correct conditional distribution of label given x t , p ( y t | x t ) .
Based on Deﬁnition 1 and Appendix A.2, noisy source data ˜ D s are drawn from ˜ p
s ( x s , y s )
=
p x s ( x s )((1 − ρ ) p ( y s | x s )
+ ρq ( y s | x s )) , where ρ is the noise rate in source data.
Namely, source data ˜ D
s are mixture of correct source data from p x s ( x s ) p ( y s | x s ) and incorrect data from p x s ( x s ) q ( y s | x s ) .
Target data D t are drawn from p x t .
In WUDA setting, we aim to train a classiﬁer with ˜ D s and D t .
This classiﬁer is expected to accurately annotate data from p x t , i.e., to accurately simulate distribution 5).
WUDA ruins representative UDA methods.
We take a simple example to illustrate why WUDA ruins representative UDA methods.
We corrupt source data using symmetry ﬂipping (Patrini et al., 2017) and pair ﬂipping (Han et al., 2018) (Appendix B).
Namely, the corrupted source data ( ˜ D s in Deﬁnition 1) are drawn from ˜ P s whose density is ˜ p
s ( x s , y s )
=
p x s ( x s )((1 − ρ ) p ( y s | x s )
+ ρq ( y s | x s )) .
We draw the target data D t from P x t whose density is p x t .
To instantiate source and target data, we leverage MNIST and SYND (Figurein Appendix C), respectively.
Thus, we ﬁrst construct two WUDA tasks with symmetry-ﬂip noise: corrupted SYND → MNIST ( S → M ) and corrupted MNIST → SYND ( M → S ).
In Figure 2 (a), we report target-domain accuracy of representative UDA methods, when the noise rate ρ of SD changes from 5% to 70% .
It is clear that target-domain accuracy of these representative UDA methods drops quickly when ρ increases.
This means that WUDA ruins representative
UDA methods.
Then, we construct another two WUDA tasks with pair-ﬂip noise.
In Figure 2 (b), we report target-domain accuracy, when the noise rate ρ of SD changes from 5% to 45% .
Again, WUDA still ruins representative UDA methods.
Note that pair-ﬂip noise is much harder than symmetry-ﬂip noise, and its noise rate cannot be over 50% in practice (Han et al., 2018).
[Figure]
Data
F
F F t F t However, the proposed Butterﬂy network (abbreviated as B-Net, Figure 3) performs robustly when ρ increases (blue lines in Figure 2).
In following sections, we will introduce Butterﬂy framework, and explain why
Butterﬂy achieves better target-domain accuracy consistently.
B UTTERFLY : T OWARDS ROBUST ONE - STEP APPROACH
To realize a robust WUDA approach, we propose a Butterﬂy framework (a one-step approach, Algorithm 1), which trains four networks dividing into two branches (Figure 3).
By using DCP, Branch-I checks which data is correct in MD; while Branch-II checks which pseudo-labeled target data is correct.
To ensure these checked data highly-correct, we apply the small-loss trick based on memorization effects of deep learning (Arpit et al., 2017).
After cross-propagating these checked data (Bengio, 2014), Butterﬂy can obtain high-quality DIR and
TSR simultaneously in an iterative manner.
Loss function in Butterﬂy.
Four networks trained by Butterﬂy share the same loss function but with different inputs.
[Equation]
where n is the batch size, and F represents a network (e.g., F 1 , F 2 , F t 1 and F t 2 ).
D =
{ ( x i , ˇ y i ) } ni =1 is a mini-batch for training a network, where { x i , ˇ y i } ni =1 could be data in MD or TD (Figure 3), and θ represents parameters of F and u =
[ u 1 , ..., u n ] T is an n -by- 1 vector whose elements equal 0 or 1 .
(cid:96) ( · , · ) is the cross-entropy loss.
For two networks in Branch-I, following Saito et al.
(2017), we also add a regularizer | θ Tf 11 θ f 21 | in their loss functions, where θ f 11 and θ f 21 are weights of the ﬁrst fully-connect layer of F 1 and F 2 .
With this regularizer, F 1 and F 2 will learn from different features.
Nature of the loss L .
In loss function L , we have n instances: ( x i , ˇ y i ) , where i = 1 , 2 , . . . , n .
For the i th instance, we will compute its cross-entropy loss (i.e., (cid:96) ( F ( x i ) , ˇ y i ) ), and we will denote this instance as “selected” if u i = 1 .
Thus, the nature of L is actually the average value of cross-entropy loss of these
“selected” instances. Note that, we need to set a constrain to prevent (cid:80) ni =1 u i = 0 in L , which means that we need to select at least one instance to compute L . Training procedure of Butterﬂy.
For two networks in each branch, they will ﬁrst check high-correctness data out and then cross update their parameters using these data.
Based on loss function deﬁned in Eq.
(1), entire training procedures of Butterﬂy are shown in Algorithm 1.
First, we initialize training data for two branches ( ˜ D for Branch-I and ˜ D lt for Branch-II), four networks ( F 1 , F 2 , F t 1 and F t 2 ) and the number of pseudo labels (line 2 ).
In the ﬁrst epoch ( T = 1 ), following Saito et al. (2017), ˜ D lt is the same with ˜ D s (i.e., we use noisy source data as pseudo-labeled target data) since we cannot annotate pseudo labels for target data when T = 1 .
After mini-batch ˇ D is fetched from ˜ D (line 4 ), F 1 and F 2 check high-correctness data out and update their parameters (lines 5 ) using Algorithm 2.
Using similar procedures, F t 1 and F t 2 also update their parameters using Algorithm 2 (lines 6 - 7 ).
In each epoch, after N max mini-batch updating, we randomly select n lt unlabeled target data and assign them pseudo labels using F 1 and F 2 (lines 8 ).
Following Saito et al.
(2017), the Labeling function in Algorithm(line 8 ) assigns pseudo labels for unlabeled target data, when predictions of F 1 and F 2 agree and at least one of them is conﬁdent about their predictions (probability above 0 .
9 or 0 . 95 ).
Using this function, we can obtain the pseudo-labeled target data
˜ D lt for training Branch-II in the next epoch.
Then, we merge ˜ D lt and ˜ D
s to be ˜ D for training Branch-I in the next epoch (line 9 ).
Finally, we update n lt , R ( T ) and R t ( T ) in lines 10 - 11 according to Saito et al.
and Han et al.
Note that R ( T ) is a piecewise-deﬁned linear function.
Namely, when T ≥ T k , R ( T ) = 1 − τ ; when T ≤ T k , R ( T ) = 1 − T/T k × τ .
In Algorithm 1, we use τ to represent the noise rate (i.e., the ratio of data with incorrect labels) in MD and use τ t to represent the noise rate in TD.
However, in WUDA, we cannot obtain the ground-truth τ and τ t .
Thus, we regard τ and τ t as hyper-parameters.
Note that τ and τ t are robustly set to 0 .
4 and 0 .
in experiments.
Checking process in Butterﬂy. In Algorithm 2, we ﬁrst obtain four inputs: 1) networks F 1 and F 2 , and 2) a mini-batch D , and 3) learning rate η and 4) remember rate α (line 1). Then, we will obtain the best u 1 by solving a minimization problem (line 2). L represents the loss function deﬁned in Eq.
θ 1 represents the parameters of the network F 1 .
Similarly we will obtain the best u 2 (line 3).
θ 2 represents the parameters of the network F 2 .
Next, θ 1 and θ 2 are updated using gradient descent, where the gradients are computed using a given optimizer (lines 4-5).
Finally, we substitute the updated θ 1 into F 1 and the updated θ 2 into F 2 and output F 1 and F 2 (line 6).
Solution to minimization problems in Algorithm 2.
In line 2 or 3 in Algorithm 2, we need to solve a minimization problem: min u (cid:48) : 1 u (cid:48) >
α | D | L ( θ, u (cid:48) ; F, D ) and return the best u (cid:48) as u ( u 1 in line 2 and u 2 in line 3 ).
In this paragraph, we will show how to quickly solve this problem using a sorting algorithm.
Recall the nature of the loss L , we know L is the average value of cross-entropy losses of “selected” instances, and 1 u (cid:48) is the number of these “selected” instances.
Thus this minimization problem is equivalent to “given a ﬁxed F ( F 1 or F 2 ) and n instances in D , how to select at least k instances such that L is minimized”, where k = (cid:100) α | D |(cid:101) .
To solve this problem, we ﬁrst use a sorting algorithm (top_k function in TensorFlow) to sort these n instances according to their cross-entropy losses (cid:96) ( F 1 ( x i ) , ˇ y i ) .
Then, we select k instance with the smallest cross-entropy losses.
Finally, let u i of these k instances be 1 and u i of the other instances be 0 , and we can get the best u =
[ u 1 , . .
u n ] .
The average value of cross-entropy losses of these k instances is the minimized value of L ( θ, u (cid:48) ; F, D ) under the constrain 1 u (cid:48) >
α | D | .  B UTTERFLY VS . T WO - STEP A PPROACH This section analyzes why Butterﬂy is better than two-step approach using theoretical results in Appendix D.
Practitioner may safely skip it.
Following Ben-David et al.
(2010), we derive an upper bound of target-domain risk for WUDA.
Compared to existing UDA bounds, the WUDA bound has two extra terms (see Eq.
(6)): ∆ s (noise effect from source data), and ∆ t (noise effects from pseudo labels of target data).
We will use ∆ s and ∆ t to show why Butterﬂy (a one-step approach) can eliminate noise effects while two-step methods cannot. [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Two-step approach (a compromise solution).
To reduce noise effects, a straightforward solution is twostep approach.
In the ﬁrst step, we can train a classiﬁer with noisy source data using Co-teaching (Han et al., 2018) and use this classiﬁer to annotate pseudo labels for source data.
In the second step, we use ATDA (Saito et al., 2017) to train a target-domain classiﬁer with pseudo-label-source data and target data.
However, two-step approach may not reduce noise effects ∆ s (i.e., not alleviating noise effects from source data).
In two-step approach, after using Co-teaching, ∆ s will become pseudo-label-source effects ∆ (cid:48) s
(see
Eq.
The ﬁrst part of ∆ (cid:48)
s may be less than that of ∆ s due to Co-teaching, but the second term of ∆ (cid:48)
s may be higher than that of ∆ s since Co-teaching does not consider to minimize it.
Thus, it is hard to say whether ∆ (cid:48) s <
∆ s .
This means that, the two-step approach may not really reduce noise effects ∆ s .
Besides, two-step approach does not take care of eliminating ∆ t explicitly.
Based on above analysis, we can ﬁnd that a two-step approach cannot eliminate ∆ s and ∆ t .
One-step approach (Butterﬂy).
To eliminate noise effects ∆ =
∆ s + ∆ t , Butterﬂy aims to select correct data simultaneously from noisy source data and pseudo-labeled target data (see Section 3).
Let ρ s 01 be the probability that incorrect data is selected from noisy source data, and ρ t 01 be the probability that incorrect data is selected from pseudo-labeled target data.
Theorem 3 shows that ∆ → 0 if ρ s 01 → 0 and ρ t 01 → 0 .
Since
Butterﬂy can select correct data with a high probability (i.e., ρ s 01 → 0 and ρ t 01 → 0 ), noise effects will be eliminated ( ∆ → 0 ). [Figure]
E XPERIMENTS
Simulated WUDA tasks.
We verify the effectiveness of our approach on three benchmark datasets (vision and text), including MNIST , SYN-DIGITS (SYND) and human-sentiment analysis (i.e., Amazon products reviews on book , dvd , electronics and kitchen ).
They are used to construct 14 basic tasks: MNIST → SYND ( M → S ), SYND → MNIST ( S → M ), book → dvd ( B → D ), book → electronics ( B → E ), . . . , and kitchen → electronics ( K → E ).
These tasks are often used for evaluation of UDA methods (Ganin et al., 2016;
Saito et al., 2017; 2018).
Since all source datasets are clean, we corrupt source data using symmetry ﬂipping (Patrini et al., 2017) and pair ﬂipping (Han et al., 2018) (Appendix B) with noise rate ρ chosen from { 0 . 2 , 0 . 45 } .
So, for each basic task, we have four kinds of noisy source data:
Pair- 45% (P 45 ), Pair- 20% (P 20 ), Symmetry- 45% (S 45 ), Symmetry- 20% (S 20 ).
Namely, we evaluate the performance of each method using 32 simulated
WUDA tasks: 8 digit tasks and 24 human-sentiment tasks.
Note that the human-sentiment task is a binary classiﬁcation problem, so pair ﬂipping is equal to symmetry ﬂipping.
Thus, we only have 24 human-sentiment tasks.
Results on human-sentiment tasks are reported in Appendix E.
Real-world WUDA tasks.
We also verify the efﬁcacy of our approach on “cross-dataset benchmark” including Bing , Caltech256 , Imagenet and SUN (Tommasi and Tuytelaars, 2014).
In this benchmark, Bing ,
Caltech256 , Imagenet and SUN contain common 40 classes.
Since Bing dataset was formed by collecting images retrieved by Bing image search, it contains rich noisy data, with presence of multiple objects in the same image and caricaturization (Tommasi and Tuytelaars, 2014).
We use Bing as noisy source data, and
Caltech256 , Imagenet and SUN as unlabeled target data, which can form three real-world WUDA tasks.
Besides, we also test our method on two common UDA tasks ( MNIST → SVHN and SVHN → MNIST ) to help compare with other UDA-based papers in literature.
Baselines.
We realize Butterﬂy using four networks (abbreviated as B-Net) and compare B-Net with following baselines: 1) ATDA: representative pseudo label based UDA method (Saito et al., 2017); 2) deep adaptation networks (DAN): representative IPM based UDA method (Long et al., 2015); 3) DANN: representative adversiral training based UDA method (Ganin et al., 2016); 4) TCL: an existing robust UDA method; 5)
Co teaching+ATDA
(Co+ATDA): a two-step method (see Section 4); 6) Co teaching+TCL (Co+TCL): a two-step method.
Implementation details are demonstrated in Appendix F.
Results on simulated WUDA (including 8 tasks).
Table 1 reports the accuracy on the unlabled target data (i.e., target-domain accuracy) in 8 tasks.
As can be seen, average target-domain accuracy of B-Net is higher than those of all baselines.
On S 20 case (the easiest case), most methods work well.
ATDA has a satisfactory performance although it does not consider the noise effects explicitly.
Then, when facing harder cases (i.e.,
P 20 and P 45 ), ATDA fails to transfer useful knowledge from noisy source data to unlabeled target data.
When facing the hardest cases (i.e., M → S with P 45 and S 45 ), DANN has higher accuracy than DAN and ATDA have.
However, when facing the easiest cases (i.e., S → M with P 20 and S 20 ), the performance of DANN is worse than that of DAN and ATDA.
Although two-step method Co+ATDA (or Co+TCL) outperforms ATDA (or TCL) in all 8 tasks, it cannot beat one-step method: B-Net in terms of average target-domain accuracy.
This result is an evidence for the claim in Section 4.
In the task S
→ M with P 20 , Co+ATDA outperforms all methods (slightly higher than B-Net), since pseudo-labeled source data are almost correct.
[Figure]
[Table]
Figures 4 and 5 show the target-domain accuracy vs. number of epochs among ATDA, Co+ATDA and
B-Net.
Besides, we show the accuracy of ATDA trained with clean source data (ATDA-TCS) as a reference point.
When accuracy of one method is close to that of ATDA-TCS (red dash line), this method successfully eliminates noise effects.
From our observations, it is clear that B-Net is very close to ATDA-TCS in 7 out of 8 tasks (except for S → M task with P 45 , Figure 4-(d)), which is an evidence that Butterﬂy can eliminate noise effects.
Since P 45 case is the hardest one, it is reasonable that B-Net cannot perfectly eliminate noise effects.
An interesting phenomenon is that, B-Net outperforms ATDA-TCS in 2 M → S tasks (Figure 5-(a), (c)).
This means that B-Net transfers more useful knowledge (from noisy source data to unlabeled target data) even than ATDA-TCS (from clean source data to unlabeled target data).
Results on real-world WUDA (including 3 tasks).
Table 2 reports the target-domain accuracy for 3 tasks.
B-Net enjoys the best performance on all tasks.
It should be noted that, in both Bing → Caltech256 and
Bing → ImageNet tasks, ATDA is slightly worse than B-Net.
However, in Bing → SUN task, ATDA is much worse than B-Net.
The reason is that the DIR between Bing and SUN are more affected by noisy source data.
This phenomenon is also observed when comparing DANN and TCL.
Compared to Co+ATDA, ATDA is slightly better than Co+ATDA.
This abnormal phenomenon can be explained using ∆ (see Section 4), after using Co-teaching to assign pseudo labels for noisy source data, the second term in ∆ s may increase, which results in that ∆ increases, i.e., noise effects actually increase.
This phenomenon is an evidence that a two-step method may not really reduce noise effects. Results on two UDA tasks. Table 3 reports the target-domain accuracy for 2 common UDA tasks.
The target-domain accuracy of DAN, DANN and ATDA are results reported by Saito et al.
Although the target-domain accuracy of B-Net is not always higher than that of ATDA, B-Net still has satisfactory performance on UDA problem.
Note that, since B-Net is a WUDA method, it is reasonable that B-Net is not the state-of-the-art UDA method.
Ablation study.
Finally, we conduct thorough experiments to show the contribution of individual components in B-Net.
We report average target-domain accuracy on 32 simulated WUDA tasks ( 8 digit and [Table] [Table]
[Table]
human-sentiment WUDA tasks) and 3 real-world WUDA tasks.
We consider following baselines: 1) B w/o
C: train B-Net by Algorithm 1, without adding | θ Tf 11 θ f 21 | into the loss function of B-Net. 2) DCP-D: realize
DCP via Decoupling (Malach and Shalev-Shwartz, 2017) to check data in MD and TD.
3) DCP-M: realize
DCP via MentorNet (Jiang et al., 2018) to check data in MD and TD.
4) B-Net-S: train B-Net where the check is turned on for Source data in MD. 5) B-Net-T: train B-Net where the check is turned on for Target data in TD.
6) B-Net-ST: train B-Net where the checks are turned on for Source data in MD and Target data in TD.
7) B-Net-M: train B-Net where the check is turned on for all data in MD.
Note that in the full B-Net, the checks are turned on for all data in MD and Target data in TD.
Comparing B-Net with B w/o C reveals whether the constraint | θ Tf 11 θ f 21 | takes effects.
Comparing B-Net with DCP-D and DCP-M shows whether realizing DCP via Co-teaching is the optimal way.
Comparing
B-Net with B-Net-S, B-Net-T, B-Net-ST and B-Net-M reveals whether DCP is necessary.
Table 4 reports average target-domain accuracy of above baselines and B-Net.
As can be seen, 1) B-Net beneﬁts from adding the constraint to the loss function L ; 2) realizing DCP by Co-teaching is better than using Decoupling or
MentorNet; and 3) DCP is necessary since accuracy of B-Net is higher than those of B-Net-S, B-Net-T,
B-Net-ST and B-Net-M.
C ONCLUSIONS
This paper opens a new problem called wildly unsupervised domain adaptation (WUDA).
However, existing
UDA methods cannot handle WUDA well.
To address this problem, we propose a robust one-step approach called Butterﬂy .
Butterﬂy maintains four deep networks simultaneously: Two take care of all adaptations; while the other two can focus on classiﬁcation in target domain.
We compare Butterﬂy with existing UDA methods on 32 simulated and 3 real-world WUDA tasks.
Empirical results demonstrate that Butterﬂy can robustly transfer knowledge from noisy source data to unlabeled target data.
In future, we will extend our
Butterﬂy framework to address open-set UDA when source domain contains noisy data.
A R EVIEW OF GENERATION OF NOISY LABELS
This section presents a review on two label-noise generation processes.
A. T RANSITION MATRIX
We assume that there is a clean multivariate random variable ( X s , Y s ) deﬁned on X × Y with a probability density
p s ( x s , y s ) , where Y = { 1 , ..., K } is a label set with K labels.
However, samples of ( X s , Y s ) cannot be directly obtained and we only can observe noisy source data from the multivariate random variable ( X s , ˜ Y s ) deﬁned on X × Y with a probability density ˜
p s ( x s , ˜ y s ) .
˜ p
s ( x s , ˜ y s ) is generated by a transition probability Pr ( ˜ Y s =
j | Y s = i ) , i.e., the ﬂip rate from a clean label i to a noisy label j .
When we generate ˜ p s ( x s , ˜ y s ) using Q , we often assume that (cid:80) Ky s =1 p s ( x s , y s )
=
(cid:80) K
˜ y s
=1 ˜ p s ( x s , ˜ y s ) ,
i.e., the class conditional noise (Liu and Tao, 2016).
All these transition probabilities are summarized into a transition matrix Q , where
Q ij =
Pr ( ˜ Y s =
j | Y s = i ) .
The transition matrix Q is easily estimated in certain situations (Liu and Tao, 2016).
However, in more complex situations, such as clothing1M dataset (Xiao et al., 2015), noisy source data is directly generated by selecting data from a pool, which mixes correct data (data with correct labels) and incorrect data (data with incorrect labels).
Namely, how the correct label i is corrupted to j ( i (cid:54) =
j ) is unclear.
A. S AMPLE SELECTION
Formally, there is a multivariate random variable ( X s , Y s , V s ) deﬁned on X ×Y ×V with a probability density p po s ( x s , y s , v s ) , where V = { 0 , 1 } and V s = 1 means “correct” and V s = 0 means “incorrect”.
Nonetheless, samples from ( X s , Y s , V s ) cannot be obtained and we can only observe ( X s , ˜ Y s ) from a distribution with the following density.
[Equation]
where p po V s ( v s )
= (cid:82) X (cid:80)
Ky s =1 p po s ( x s , y s , v s )
dx s .
The density in Eq.
(2) means that we lost the information from V s .
If we uniformly select samples drawn from ˜ p
s ( x s , ˜ y s ) ,
the noise rate of these samples is p po V s (0) .
It is clear that the multivariate random variable ( X s , Y s | V s = 1) is the clean multivariate random variable ( X s , Y s ) deﬁned in Appendix A.1.
Then, q s ( x s , y s ) is used to describe the density of incorrect multivariate random variable ( X s , Y s | V s = 0) .
Using p s ( x s , y s ) and q s ( x s , y s ) , ˜ p s ( x s , ˜ y s ) can be expressed by the following equation.
[Equation]
where ρ = p po V s (0)
Here, we do not assume (cid:80)
Ky s =1 p s ( x s , y s )
= (cid:80)
Ky s =
1 q s ( x s , y s ) .
To reduce noise effects from incorrect data, scholars aim to recover the information of V s , i.e., to select correct data from data drawn from ˜ p
s ( x s , ˜ y
s )
(Han et al., 2018; Jiang et al., 2018; Malach and Shalev-Shwartz, 2017).
[Figure]
B T RANSITION MATRIX Q
Precise deﬁnitions of Symmetry ﬂipping and Pair ﬂipping are presented below, where ρ is the noise rate and
K is the number of labels.
[Equation]

[Equation]
ρ
[Equation] [Equation]
 .
Following Han et al.
; Jiang et al.
(2018), we can corrupt clean-label datasets manually using the noise transition matrix Q .
C D ATASETS VISUALIZATION
Figure 6 shows datasets: MNIST and SYND .
Figure 7 shows datasets: Bing , Caltech256 , Imagenet and SUN (taking “horse” as the common class).
[Figure]
D T HEORETICAL ANALYSIS
This section presents some interesting theoretical ﬁndings related to WUDA problem.
We use following notations in this section: 1) a space X ⊂ R d and Y = { 1 , 2 , . .
, K } as a label set; 2) ˜
p s ( x s , ˜ y s ) , p s ( x s , y s ) and q s ( x s , y s ) represent densities of noisy, correct and incorrect multivariate random variables (m.r.v.) deﬁned on X × Y , respectively 1 , and ˜ p x s ( x s ) , p x s ( x s ) and q x s ( x s ) are their marginal densities; and 3) p x t ( x t ) represents density of m.r.v.
x t deﬁned on X ; and 4) we use (cid:96) ( h ( x ) , h (cid:48)
( x )) to represent loss function between two labelling functions; and 5) we use ˜ R s ( h ) =
E ˜ p
s ( x s , ˜ y s )
[ (cid:96) ( h ( x s ) , ˜ y s )] and R s ( h ) =
E p s ( x s ,y s )
[ (cid:96) ( h ( x s ) , y s )] to represent expected risks on the noisy and correct m.r.v.; and 6) we use ˜ R s ( h, h (cid:48) ) =
E ˜ p
xs ( x s ) [ (cid:96) ( h ( x s ) , h (cid:48) ( x s ))] ,
R s ( h, h (cid:48) )
=
E p xs ( x s ) [ (cid:96) ( h ( x s ) , h (cid:48) ( x s ))] and R t ( h, h (cid:48) )
=
E p xt
( x t ) [ (cid:96) ( h ( x t ) , h (cid:48) ( x t ))] to represent expected discrepancy between two labelling functions h, h (cid:48) under different marginal densities; 7) the ground-truth and pseudo labeling function of the target domain are denoted by f t ( x t ) and ˜ f t ( x t ) .
D. WUDA RUINS UDA METHODS
Theoretically, we analyze why existing UDA methods cannot well transfer useful knowledge from noisy source data
˜ D
s to unlabelled target data D t directly.
We ﬁrst present a theorem to show relations between
R s ( h ) and ˜ R s ( h ) .
Theorem 1.
For any labelling function h : X → Y , if ˜ p s ( x s , ˜ y s ) is generated by a transition matrix Q as demonstrated in Appendix A.1, we have
[Equation]
where (cid:96) ( h ( x s ))
=
[ (cid:96) ( h ( x s ) , 1) , ..., (cid:96) ( h ( x s ) , K )] T and η ( x s )
=
[ p Y s | X s (1 | x s ) , ..., p Y s |
X s ( K | x s )] T .
If ˜ p s ( x s , ˜ y s ) is generated by sample selection as described in in Appendix A.2, we have
[Equation]
where η q ( x s ) =
[ q Y s | X s (1 | x s ) , ..., q Y s |
X s ( K | x s )] T .
Remark 2.
In Eq.
(5), E q xs ( x s )
[ η T q ( x s ) (cid:96) ( h ( x s ))] represents the expected risk on the incorrect m.r.v..
To ensure to obtain useful knowledge from ˜ P s , we need to avoid ˜ R s ( h )
≈ E q xs ( x s )
[ η T q ( x s ) (cid:96) ( h ( x s ))] .
Speciﬁcally, we assume: there is a constant 0 < M s < ∞ such that E q xs ( x s )
[ η T q ( x s ) (cid:96) ( h ( x s ))] ≤ R s ( h ) + M s .
Theorem 1 shows that the expected risk
˜ R s ( h ) only equals R s ( h ) when two cases happen: 1) Q
= I and ρ = 0 and 2) some special combinations (e.g., special p x s , q x s , Q , η and (cid:96) ) to make the second term in Eq.
(4) equal zero or to make the second term in Eq.
(5) equal ρR s ( h ) .
Case 1) means that data in source domain is clean, which is not real in the wild.
Case 2) almost never happens, since it is hard to ﬁnd such special combinations when p x s , q x s , Q and η are unknown.
Thus, ˜ R s ( h ) has an essential difference with R s ( h ) .
Then, following proof skills in Ben-David et al.
(2010), we derive the upper bound of R t ( h ) as follows.
Theorem 2.
For any labelling function h : X → Y , we have
[Equation]
˜ f t ) − R s
( h, ˜ f t ) | (cid:124) (cid:123)(cid:122) (cid:125) (
[Table]
s + | R t ( h, f t ) − R t ( h, ˜ f t ) | (cid:124) (cid:123)(cid:122) (cid:125) ( v ) noise effects from target ∆ t .
Remark 3.
To ensure that we can gain useful knowledge from ˜ f t ( x t ) , we assume: there is a constant 0 <
M t < ∞ such that E q xs ( x ) [ (cid:96) ( h ( x ) , ˜ f t ( x ))]
≤ R s ( h, ˜ f t )
+ M t and E q xt ( x ) [ (cid:96) ( h ( x ) , ˜ f t ( x ))]
≤ R t ( h, f t )+ M t , where q x t ( x ) =
p x t ( x )1 A ( x ) /P
x t ( A ) and A = { x : ˜ f t ( x ) (cid:54) =
f t ( x ) } .
It is clear that the upper bound of R t ( h, f t ) , shown in Eq.
(6), has 5 components.
However, existing UDA methods only focus on minimizing ( i ) + ( ii ) (Ganin et al., 2016; Ghifary et al., 2017;
Long et al., 2015) or ( i ) + ( ii ) + ( iii ) (Saito et al., 2017), which ignores terms ( iv ) and ( v ) (i.e., ∆ =
∆ s + ∆ t ).
Thus, in theory, existing UDA methods cannot handle wildly unsupervised domain adaptation well.
D. T WO - STEP APPROACH IS A COMPROMISE SOLUTION
To reduce noise effects, a straightforward solution is two-step approach.
In the ﬁrst step, we can train a classiﬁer with noisy source data using Co-teaching (Han et al., 2018) and use this classiﬁer to annotate pseudo labels for source data.
In the second step, we use ATDA (Saito et al., 2017) to train a target-domain classiﬁer with pseudo-label-source and target data.
Nonetheless, the pseudo-labeled source data is still noisy.
Let labels of noisy source data ˜
y s be replaced with pseudo labels ˜
y (cid:48) s after pre-processing.
Noise effects ∆ will become pseudo-label effects ∆ p as follows.
[Equation]
where ˜ R (cid:48) s ( h ) and ˜ R (cid:48)
s ( h, ˜ f t ) correspond to ˜ R s ( h ) and
˜ R s ( h, ˜ f t ) in ∆ s .
It is clear that the difference between ∆ p and ∆ is ∆ (cid:48) s − ∆ s .
The ﬁrst term in ∆ (cid:48) s may be less than that in ∆ s due to Co-teaching, but the second term in ∆ (cid:48)
s may be higher than that in ∆ s since Co-teaching does not consider to minimize it.
Thus, it is hard to say whether ∆ (cid:48) s <
∆ s (i.e., ∆ p < ∆ ).
This means that two-step approach may not really reduce noise effects.
D. W HY DOES B UTTERFLY CAN ELIMINATE NOISE EFFECT ?
To eliminate noise effects ∆ , we aim to select correct data simultaneously from noisy source data and pseudo-labeled target data.
In theory, we prove that noise effects will be eliminated if we can select correct data with a high probability.
Let ρ s 01 represent the probability that incorrect data is selected from noisy source data, and ρ t 01 represent the probability that incorrect data is selected from pseudo-labeled target data.
Theorem 3 shows that ∆ → 0 if ρ s 01 → 0 and ρ t 01 → 0 and presents a new upper bound of R t ( h, f t ) .
Theorem 3.
Given two m.r.v.
( X s , Y s , U s ) deﬁned on X × Y × V and ( X t , U t ) deﬁned on X × V , under the assumptions in Remarks 2 and 3, ∀ (cid:15) ∈ (0 , 1) , there are δ s and δ t , if E p (cid:48) xt ( x t ) [ (cid:96) ( h ( x t ) , f t ( x t ))]
≤
R t ( h, f t ) + ρ s 01 M t , ρ s 01 < δ s and ρ t 01 <
δ t , for any labeling function h , we will have
[Equation]
Moreover, if ρ s 01 ≤ δ s and ρ t 01 ≤ δ t , we will have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where p (cid:48) x t ( x ) =
p x t ( x )1 B ( x ) /P
x t ( B ) , ˜ R po s ( h, u s ) =
(1 − ρ u s ) − 1 E ˜ p po s ( x s ,y s ,u s )
[ u s (cid:96) ( h ( x s ) , y s )] , ˜ R po t ( h, ˜ f t , u t )
= (1 − ρ u t ) − 1 E ˜ p po t ( x t ,u t ) [ u t (cid:96) ( h ( x t ) , ˜ f t ( x t ))] , ˜ R po s ( h, ˜ f t , u s ) = (1 − ρ u s ) −
E ˜ p po s ( x s ,y s ,u s ) [ u s (cid:96) ( h ( x s ) , ˜ f t ( x s ))] , ˜ p po s ( x s , y s , u s ) is the density of ( X s , Y s , U s ) , ˜ p po t ( x t , u t ) is the density of ( X t , U t ) , ρ u s = (cid:82) X (cid:80)
Ky s =1 ˜
p po s ( x s , y s , 0) dx s < 1 , ρ u t =
(cid:82) X ˜ p po t ( x t , 0)
dx t < 1 , B = X /A
and V = { 0 , 1 } .
Remark 4.
In Appendix H.3.1, we give precise deﬁnitions of ρ s 01 and ρ t 01 and demonstrate the meaning of E p (cid:48) xt
( x t ) [ (cid:96) ( h ( x t ) , f t ( x t ))] ≤ R t
( h, f t ) + ρ s 01 M t
(Remark 5).
Data drawn from the distribution of ( X s , Y s , U s ) can be regarded as a pool that mixes the selected ( u s = 1 ) and unselected ( u s = 0 ) noisy source data.
Data drawn from the distribution of ( X t , U t ) can be regarded as a pool that mixes the selected ( u t = 1 ) and unselected ( u t = 0 ) pseudo-labeled target data.
Theoremshows that if selected data have a high probability to be correct ones ( ρ s 01 → 0 and ρ t 01 → 0 ), then ∆ s and ∆ t approach 0 , meaning that noise effects are eliminated.
This motivates us to ﬁnd a reliable way to select correct data from noisy source data and pseudo-labeled target data and build up a one-step approach for WUDA.
Why Butterﬂy?
Guided by Theorem 3, a robust approach should check high-correctness data out (meaning ρ s 01 → 0 and ρ t 01 → 0 ).
This checking process will make ( iv ) and ( v ) , 2 (cid:15) + 2 (cid:15) , become 0 .
Then, we can obtain gradients of ˜ R po s ( h, u s ) , ˜ R s ( h, ˜ f t , u s ) and ˜ R po t ( h, ˜ f t , u t ) w.r.t.
parameters of h and use these gradients to minimize them, which minimizes ( i ) and ( ii ) as ( i ) + ( ii ) ≤ ˜ R po s ( h, u s )
+ ˜ R
s ( h, ˜ f t , u s )
+ ˜ R po t ( h, ˜ f t , u t ) .
Note that ( iii ) cannot be directly minimized since we cannot pinpoint clean source data.
However, following
Saito et al.
(2017), we can indirectly minimize ( iii ) via minimizing ˜ R po s ( h, u s ) +
˜ R po s ( h, ˜ f t , u s ) , as ( iii ) ≤
R s ( h, ˜ f t ) + R s ( h ) ≤ ˜ R po s ( h, u s ) + ˜ R po s ( h, ˜ f t , u s ) + 2 (cid:15) , where the last inequality follows Eq.
This means that a robust approach guided by Theorem 3 can minimize all terms in the right side of inequality in Eq.
To realize this robust approach, we propose a Butterﬂy framework (Algorithm 1), which trains four networks dividing into two branches (Figure 3).
By using dual-checking principle, Branch-I checks which data is correct in the mixture domain; while Branch-II checks which pseudo-labeled target data is correct.
To ensure these checked data highly-correct, we apply the small-loss trick based on memorization effects of deep learning Arpit et al.
After cross-propagating these checked data Bengio (2014), Butterﬂy can obtain high-quality DIR and TSR simultaneously in an iterative manner.
Theoretically, Branch-I minimizes ( i ) + ( ii ) + ( iii ) + ( iv ) ; while Branch-II minimizes ( ii ) + ( v ) .
This means that Butterﬂy can minimize all terms in the right side of inequality in Eq.
Note that empirical estimators of ˜ R po s ( h, u s ) , ˜ R po t ( h, ˜ f t , u t ) and ˜ R po s ( h, ˜ f t , u s ) (in Theorem 3) can be expressed using the loss function of Butterﬂy (see Eq.
Relations to Co-teaching.
As Butterﬂy is related to Co-teaching, we discuss their major differences here.
Although Co-teaching (Han et al., 2018) applies the small-loss trick and the cross-update technique to train deep networks against noisy data, it can only deal with one-domain problem instead cross-domain problem.
Besides, we argue that Butterﬂy is not a simple mixtrue of Co-teaching and ATDA for two reasons.
First, network structure of Butterﬂy is different with that of ATDA and Co-teaching: Butterﬂy maintains four networks; while ATDA maintains three and Co-teaching maintains two.
We cannot simply combine ADTA and Co-teaching to derive Butterﬂy.
Second, we have justiﬁed that the sequential mixture of Co-teaching and ATDA (i.e., two-step method) cannot eliminate noise effects caused by noisy source data (see Section 4).
Speciﬁcally, two-step methods only take care of part of noise effects but Butterﬂy takes care of the whole noise effects.
Thus, Butterﬂy is the ﬁrst method to eliminate noise effects rather than alleviate it.
Relations to TCL.
Recently, transferable curriculum learning (TCL) is a robust UDA method to handle noise Shu et al.
TCL uses small-loss trick to train the domain-adversarial neural network (DANN)
Ganin et al.
However, TCL can only minimize ( i ) + ( ii ) + ( iv ) , while Butterﬂy can minimize all terms in the right side of Eq.
E R ESULTS ON HUMAN - SENTIMENT WUDA TASKS
Tables 5 and 6 report the target-domain accuracy of each method for 24 human-sentiment WUDA tasks.
For these tasks, B-Net has the highest average accuracy.
It should be noted that two-step method does not always perform better than existing UDA methods, such as for 20% -noise situation.
The main reason is Co-teaching performs poorly when pinpointing clean source data from noisy source data.
Another observation is that noise effects is not eliminated like classiﬁcation results on digit WUDA tasks.
The main reason is that these datasets provide ﬁxed features and we cannot extract better features in the training process.
However, in digit
WUDA tasks, we can gradually obtain better features for each domain and ﬁnally eliminate noise effects.
[Table] [Table]
F E XPERIMENTAL SETTINGS F. N ETWORK STRUCTURE AND OPTIMIZER
We implement all methods on Python 3.6 with a NIVIDIA P100 GPU.
We use MomentumSGD for optimization in digit and real-world tasks, and set the momentum as 0 . 9 .
We use Adagrad for optimization in human-sentiment tasks because of sparsity of review data (Saito et al., 2017).
F 1 , F 2 , F t 1 and F t 2 are 6-layer
CNN ( 3 convolutional layers and 3 fully-connected layers) for digit tasks; and are 3-layer neural networks ( 3 fully-connected layers) for human-sentiment tasks; and are 4 -layer neural networks ( 4 fully-connected layers) for real-world tasks.
The ReLU active function is used as avtivation function of these networks.
Besides, dropout and batch normalization are also used.
The network topology is shown in Figures 8, 9 and 10.
As deep networks are highly nonconvex, even with the same network and optimization method, different initializations can lead to different local optimal.
Thus, following Malach and Shalev-Shwartz (2017), we also take four networks with the same architecture but different initializations as four classiﬁers. [Figure] [Figure]
[Figure]
E XPERIMENTAL SETUP
For all 35 WUDA tasks and 2 UDA tasks, T k is set to 5 , T max is set to 30 .
Learning rate is set to 0 .
01 for simulated tasks and 0 .
05 for real-world WUDA tasks and UDA tasks, γ t is set to 0 .
05 for simulated tasks and 0 .
02 for real-world WUDA tasks and UDA tasks.
Conﬁdence level of labelling function in line 8 of Algorithm 1 is set to 0 .
95 for 8 digit tasks and 2 UDA tasks, and 0 .
9 for 24 human-sentiment tasks and 0 .
8 for real-world WUDA tasks.
γ is set to 0 .
4 for digit tasks, 0 .
1 for human-sentiment tasks, 0 . 2 for real-world WUDA tasks and 0 .
1 for UDA tasks. n lt,max is set to 15 , 000 for digit tasks and UDA tasks, 500 for human-sentiment tasks and 4000 for real-world WUDA tasks.
N max is set to 1000 for digit tasks and UDA tasks, and 200 for human-sentiment and real-world tasks.
Batch size is set to 128 for digit, real-world WUDA tasks and UDA tasks, and 24 for human-sentiment tasks.
Penalty parameter is set to 0 .
01 for digit, real-world WUDA tasks and UDA tasks, and 0 .
001 for human-sentiment tasks.
To fairly compare all methods, they have the same network structure.
Namely, ATDA, DAN, DANN, TCL and
B-Net adopt the same network structure for each dataset.
Note that DANN and TCL use the same structure for their discriminate networks.
All experiments are repeated ten times and we report the average accuracy value and standard deviation (STD) of accuracy values of ten experiments.
F. L INKS TO DATASETS
Digit datasets ( MNIST and SYN Digit (SYND) ) can be downloaded from ofﬁcial code of ATDA.
The link is https://github.com/ksaito-ut/atda .
Sentiment datasets ( Amazon products reviews ) can be downloaded from the ofﬁcial code of marginalized Stacked Denoising Autoencoder (mSDA).
The link is https://www.cse.wustl.edu/~mchen/ code/mSDA.tar .
Real-world datasets ( BCIS ) can be downloaded from the website of the project “A Testbed for Cross-Dataset
Analysis”: https://sites.google.com/site/crossdataset/home/files ("setup DENSE decaf7", 1.3GB, decaf7 features).
G R UNNING TIME
Table 7 shows the average running time of each method on the task SYND → MNIST .
Although B-Net trains four networks, its running time is still comparable to most baselines.
[Table]
H P ROOFS
This section provides proofs of theorems demonstrated in the supplementary.
H. P ROOF OF T HEOREM
Proof.
We will ﬁst prove Eq.
(4) (Case 1 ) and then prove Eq.
(5) (Case 2 ).
Case 1 .
According to deﬁnition of ˜ R s ( h ) , we have
[Equation]
(10) where (cid:96) ( h ( x s ))
=
[ (cid:96) ( h ( x s ) , 1) , ..., (cid:96) ( h ( x s ) , K )] T and ˜ η ( x s ) =
[˜ p ˜
Y s | X s (1 | x s ) , . . .
, ˜
p
˜ Y s |
X s ( K | x s )] T .
According to deﬁnition of the transition matrix Q , we know that
[Equation] [Equation]
where η ( x s ) =
[ p Y s | X s (1 | x s ) , ... , p Y s |
X s ( K | x s )] T .
Substituting Eq.
(11) into Eq.
(10), we have
[Equation]
Hence, Case 1 is proved.
Case 2 .
According to deﬁnition of ˜ R s ( h ) and Eq.
(3), we have
[Equation]
[Equation]
Let η q ( x s ) =
[ q Y s | X s (1 | x s ) , ..., q Y s |
X s
[Equation]
K | x s )]
T , we have
[Equation]
Hence, Case 2 is proved.
H. P ROOF OF T HEOREM
Proof.
For any labelling function h , we have
[Equation]
Since we do not know f t
[Equation]
we substitute following equations into Eq.
[Equation]
Then
[Equation]
we have
[Equation]
Hence, this theorem is proved
[Equation]
OF T HEOREMH.3. P RELIMINARY
Before stating the proof, we ﬁrst present a random variable below.
Let ( X t , V t ) be a m.r.v.
deﬁned on X × V with respective a density p po t ( x t , v t ) , where V = { 0 , 1 } .
V t can be regarded as perfect-selection random variables .
Namely, V t = 1 means f t ( x t ) =
˜ f t ( x t ) and V t =means f t ( x t ) (cid:54) =
˜ f t ( x t ) .
Let p po V t ( v t ) be the marginal density of p po t ( x t , v t ) .
It is clear that, higher value of p po V t ( V t = 1) means that ˜ f t is more like f t .
In following, we use 1 − ρ v t to represent p po V t ( V t = 1) .
Then, we will show 1) relation between ( X s , Y s , V s ) and ( X s , Y s , U s ) , 2) relation between ( X t , V t ) and ( X t , U t ) and deﬁnitions of ρ s 01 and ρ t 01 .
Based on ( X t , V t ) and ( X s , Y s , V s ) deﬁned in Appendix A.2, the densities of ( X s , Y s , U s ) and ( X t , U t ) can be expressed as follows.
[Equation]
where ρ sji =
Pr ( V s =
j | U s =
i ) represents the probability of the event:
V s =
j given U s =
i , ρ tji =
Pr ( V t =
j | U t = i ) represents the probability of the event:
V t = j given U t =
i ( i, j = 0 , 1 ).
Since p s ( x s , y s ) = p po X s ,Y s |
V s ( x s , y s | 1) , q
[Equation]
, y
[Equation]
) = p
[Equation] [Equation]
) , p
[Equation]
|
[Equation]
x t ) /P
x t
[Equation]
A ) =
q x t ( x t ) and p po X t |
[Equation]
t
[Equation]
x t | 1) =
[Equation]
x
[Equation]
x
[Equation]
) /P
x
[Equation]
p
[Equation]
x t ( x
[Equation]
( A = { x : ˜ f t ( x )
[Equation]
f t
[Equation]
x ) } , B = X /A ), we have
[Equation]
Next, we give a lemma to show relation between ˜ R po s ( h, u s ) and R s ( h ) .
Lemma 1.
Given the multivariate random variable ( X s , Y s , U s ) with the probability ˜ p po s ( x s , y s , u s ) and
Eq.
(14), we have
[Equation]
Proof.
According to deﬁnition of ˜ R po s ( h, u s ) in Theorem 3, we have
[Equation]
where ( a ) is based on the deﬁnition of ρ u s and Eq.
Thus, we have
[Equation]
This lemma is proved.
Similar with Lemma 1, we can obtain
[Equation]
Then
[Equation]
we give another lemma to show relation between ˜ R po t ( h
Lemma 2.
Given the multivariate random variable ( X t , U t ) with the probability ˜ p po s ( x t , u t ) and Eq.
[Equation]
if E p
[Equation]
t ) [ (cid:96) ( h ( x t ) , f t ( x t ))] ≤ R t
( h, f t )
+ ρ
[Equation]
01 M t , then we have
[Equation]
Proof.
According to deﬁnition of ˜ R po t ( h
[Equation]
˜ f t
[Equation]
u t ) in Theorem
[Equation]
we have
[Equation]
where ( a ) is based on the deﬁnition of ρ u s and Eq.
(14) and ( b ) is based on the deﬁnition of V t ( f t ( x t ) =
˜ f t ( x t ) when V t = 1 ).
Since E p (cid:48) xt ( x t ) [ (cid:96) ( h ( x t ) , f t ( x t ))] ≤ R t ( h, f t ) + ρ s 01 M t , we have
[Equation] [Equation]
Thus, we have |
[Equation]
This lemma is proved.
Remark 5.
In Lemma 2, the assumption E p (cid:48) xt ( x t ) [ (cid:96) ( h ( x t ) , f t ( x t ))] ≤ R t ( h, f t ) + ρ s 01 M t means that the expect risk restricted in B (i.e., E p (cid:48) xt ( x t ) [ (cid:96) ( h ( x t ) , f t ( x t ))] ) can represent the true risk R t ( h, f t ) when ρ s 01 is small, where B = { x : ˜ f t ( x ) =
f t ( x ) } .
In Butterﬂy, it is equivalent to that pseudo labels provided by noisy source data are more useful if we can select more correct data from noisy source data.
If this assumption fails, we cannot gain useful knowledge from ˜
f t even when we can perfectly select correct data from pseudo-labeled target data ( ρ t 01 = 0 ).
Inequalities (16), (17) and (18) show that if we can avoid to annotate incorrect data as “correct” ( ρ s 01 = 0 and ρ t 01 = 0 ), we have ˜ R po s ( h, u s )
=
R s ( h ) , ˜ R po s ( h, ˜ f t , u t ) = R s ( h, ˜ f t ) and ˜ R po t ( h, ˜ f t , u t ) =
R t ( h, f t ) .
Nonetheless, ρ s 01 and ρ t 01 never equal 0 , and E q s ( x s ,y s )
[ (cid:96) ( h ( x ) , y )] , E q xs ( x s ) [ (cid:96) ( h ( x s ) , ˜ f t ( x s ))] and
E q xt ( x t ) [ (cid:96) ( h ( x t ) , ˜ f t ( x t ))] may equal + ∞ for some h .
In next section, we prove that, under the assumption in Remarks 2 and 3, ˜ R po s ( h, u s )
→ R s ( h ) , ˜ R po s ( h, ˜ f t , u t ) → R s ( h, ˜ f t ) and ˜ R po t ( h, ˜
f
t , u t ) → R t ( h, f t ) if ρ s 01 → 0 and ρ t 01 → 0 .
Moreover, we give a new upper bound of R t ( h, f t ) .
H.3.
P ROOF OF T HEOREM
Now, we prove Theorem 3 as follows.
Proof.
We ﬁrst prove upper bounds of | ˜ R po s ( h, u s ) − R s ( h ) | , | ˜ R po s ( h, ˜ f t , u t ) − R s
( h, ˜ f t ) | and | ˜ R po t ( h, ˜ f t , u t ) −
R t ( h, f t ) | under assumptions in Theorem 3.
Based on Lemma 1, we have
[Equation]
Similar, we have
[Equation]
Since M s and M t are positive constants, it is clear that ˜ R po s ( h, u s )
→ R s ( h ) , ˜ R po s ( h, ˜ f t , u t ) →
R s ( h, ˜ f t ) and ˜ R po t ( h, ˜ f t , u t ) → R t ( h, f t ) when ρ s 01 → 0 and ρ t 01 → 0 .
Speciﬁcally, ∀ (cid:15) ∈ (0 , 1) , let δ t =
(cid:15)/M t and δ s =
(cid:15)/ max { M s , ρ t 11 M t } .
When ρ s 01 < δ s and ρ t 01 <
δ t , we have
[Equation]
Hence, we prove the Eq.
In following, we give a new upper bound of R t ( h, f t ) .
Call back to Theorem 2, we replace 1)
˜ R s ( h ) with ˜ R po s ( h, u s ) , 2) ˜
R s ( h, ˜ f t ) with ˜ R po s ( h, ˜ f t , u t ) , 3) R s ( h, ˜ f t ) with ˜ R po t ( h, ˜ f t , u t ) .
Then, we have
[Equation]
Let ρ s 01 ≤ δ s and ρ t 01 ≤ δ t , we have
[Equation]
[Equation] [Equation]
[Figure]
[Table]
[Equation]
[Figure]
[Equation]
[Table]
target ∆
[Equation]
Hence, we prove this theorem
[Equation]