Invariance Learning based on Label Hierarchy
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Deep Neural Networks inherit biased correlations embedded in training data andhence may fail to predict desired labels on unseen domains (or environments),which have different distributions from the domain to provide training data.
Invariance Learning (IL) has been developed recently to overcome this shortcoming;using training data in many domains, IL estimates such a predictor that is invariant to a change of domain.
However, the requirement of training data in multipledomains is a strong restriction of using IL, since it demands expensive annotation.
We propose a novel IL framework to overcome this problem.
Assuming the availability of data from multiple domains for a classiﬁcation task at a higher level, forwhich the labeling cost is lower, we estimate an invariant predictor for the targetclassiﬁcation task with training data gathered in a single domain.
Additionally, wepropose two cross-validation methods for selecting hyperparameters of invarianceregularization, which has not been addressed properly in existing IL methods.
The effectiveness of the proposed framework, including the cross-validation, isdemonstrated empirically.
Theoretical analysis reveals that our framework canestimate the desirable invariant predictor with a hyperparameter ﬁxed correctly,and that such a preferable hyperparameter is chosen by the proposed CV methodsunder some conditions.
Introduction
Training data used in machine learning may contain features that are spuriously correlated to the labels of data.
Deep Neural Networks (DNNs) often learn such biased correlations embedded in training data and hence may fail to predict desired labels of test data generated by a different distributionfrom one to provide training data.
In classiﬁcation of animal images, DNNs tend to misclassify cowson sandy beaches, since most training pictures are taken in green pastures and DNNs inherit contextinformation in training [3, 9].
Another example is learning from medical data.
Systems trained withdata collected in one hospital do not generalize well to other hospitals; DNNs unintentionally extractenvironmental factors speciﬁc to a particular hospital in training [16–18].
Invariance Learning (IL) is a rapidly developed approach to overcome the issue of biased correlation,which is caused by some bias in the distribution of a training dataset [10–15, 21, 33–36, 54].
In thispaper, we use the term domain to specify the bias in the distribution of a dataset.
IL is thus amethod for removing the inﬂuence of domain shits.
Using training data from multiple domains, ILestimates a predictor invariant to the change of domains, aiming at keeping good performance inunseen domains as well as in the training domains.
While the IL approach has attracted much attention, requiring training data from multiple domainsmay hinder wide applications in practice; preparing training data in many domains often involvesexpensive data annotation.
In real-world data, labels may be missing [42–45, 56–58] or incomplete[46–48].
Such data with incomplete annotation are not directly applicable to the standard IL meth- ods; they must be re-annotated accurately, often at great ﬁnancial or human expense.
The high costdrives a strong need to establish a new IL framework without or with lower annotation costs.
[Figure]
To mitigate the problem of annotation cost,we propose a novel IL framework for the situation where the training data of target classiﬁcation is given in only one domain, whilethe task of higher label hierarchy , whichneeds lower annotation cost, has data frommultiple domains.
Here, the task of higherlabel hierarchy means a classiﬁcation taskwith coarser labels than those of the target classiﬁcation.
Figure 1 shows an example of label hierarchy.
Consider the case where a target classiﬁcation has 300 labels 50 { bird 1 ; :: bird 100 ; snake 1 ; :::; snake 100 ; turtle 1 ; :::; turtle 100 } corresponding to 300 species.
Then, thebinary labels { bird ; reptile } are an example of labels in the higher hierarchy.
The decrease in thenumber of classes reduces annotation time per image.
Moreover, annotation of the binary labelsdoes not require any expert knowledge, while annotation of the original 300 labels would requiresexpert knowledge about birds, snakes, and turtles.
Hence, the new IL framework signiﬁcantly reduces the annotation cost in comparison with previous IL methods; we need exhausting annotationof 300 classes only for one domain and just binary labels for other domains.
Another important issue in IL is hyperparameter selection.
Most IL methods involve some hyperparameters to balance the classiﬁcation accuracy and the degree of invariance.
As [21] and [19]point out, in the literature of IL, the best performances of invariance had often been achieved byselecting the hyperparameters using test data from unseen domains.
Moreover, [19] numericallydemonstrated that, without using test data, simple methods of hyperparameter selection fail to ﬁnd apreferable hyperparameter.
It demonstrates a strong need for establishing an appropriate method ofhyperparameter selection for IL.
We propose two methods of cross-validation (CV) for hyperparameter selection in our new IL framework.
Since we assume training data of a single domain for the target task, it is impossible to estimate the deviation of the risks over the domains.
Our CV methods mitigate the difﬁculty by usingadditional data from multiple domains in the higher label hierarchy.
Theoretical analysis proves thatour methods select a hyperparameter correctly under some conditions.
The main contributions of this paper are as follows:
• We establish a novel framework of invariance learning, which estimates an invariant predictorfrom a single domain data, assuming additional data from multiple domains for a classiﬁcationtask at a higher level.• Under the framework, we propose two methods of cross-validation for selecting hyperparameterswithout accessing any samples from unseen target domains.•
Experimental studies verify that the proposed framework extracts an invariant predictor moreeffectively than other existing methods.•
We mathematically prove that our framework can estimate a correct invariant predictor with ahyperparameter ﬁxed correctly and that such a preferable hyperparameter is selected by the proposed CV methods under some settings.
Invariance Learning based on Label Hierarchy
Notations Throughout this paper, the space of input features and ﬁnite class labels are denotedby X and Y , respectively.
For given predictor f : X → Y and random variable ( X; Y ) on
X × Y with its probability P X,Y , R ( X,Y ) ( f ) denotes the risk of f on ( X; Y ) ; i:e: , R ( X,Y ) ( f ) :
= 84 R l ( f ( x ) ; y ) dP X,Y , where l : Y × Y → R is a loss function.
For m ∈ N > 0 , [ m ] denotes the set 85 { 1 ; :::; m } .
For a ﬁnite set A , | A | ∈ N denotes the number of elements in A .
2.1 Review of Invariance Learning
Following [10], to formulate the out-of-distribution (o.o.d.) generalization, we assume that the jointdistribution of data ( X e ; Y e ) depends on the domain (or environment)
e ∈ E , and consider the dependence of a predictor f on the domain variable e .
Suppose we are given training datasets 90 D e := {
[Equation]
x ei
[Equation]
y ei ) } n e i =1 ∼ P X e ,Y e i.i.d.
from multiple domains E tr ⊂ E .
The ﬁnal goal of theo.o.d.
problem is to predict a desired label Y e ∈ Y from X e ∈ X for larger target domains E ⊃ E tr .
To discuss the o.o.d.
performance, [10] introduced the o.o.d.
risk
[Equation]
where R e ( f ) :
= R ( X e ,Y e ) ( f ) .
This is the worst case risk over E , including unseen domains E \E tr .
To solve (1), [10] estimates a predictor f that performs well in unseen domains E \ E tr as well astraining domains E tr , namely a predictor invariant to a change of domains.
The invariant predictorf is composed of two maps Φ and w ; that is, f = w ◦ Φ holds.
A feature map Φ : X → H , which isoften called an invariance , realizes a feature of x ∈ X in the feature space H with biased correlationsin x removed.
A predictor w : H → Y estimates the label of feature Φ( x ) .
The estimation of aninvariant predictor is implemented by solving the following optimization problem:
[Equation]
where I tr is the set of invariances captured by S e ∈E tr D e :
[Equation]
This invariance is based on conditional independence as [14, 15, 40], while [10, 11] use a differenttype invariance based on argmin w R e ( w ◦ Φ) instead of P ( Y e | Φ( X e )) .
All IL methods estimate theinvariance using the difference among E tr , assuming the availability of multiple training domains.
2.2 Invariance estimation by higher label data
[Figure]
Our goal is to make an invariant predictor from a single training domain E tr = { e ∗ } .
In this case,(2) is reduced to the empirical risk minimization
min
f R e ∗ ( f ) on e ∗ , and therefore the standard ILframework is not able to extract an invariance.
In this paper, we introduce an assumption that additional data D ead for an another task ( X e ; Z e ) ,which is in the higher label hierarchy than ( X e ; Y e ) , is available with respect to multiple domains 109 E ad ⊂ E .
The task of higher hierarchy ( X e ; Z e ) has a coarser label
Z e ∈
Z than Y e ∈ Y ; moreformally, Z e is represented as Z e = g ( Y e ) with a surjective label mapping g :
Y → Z fromthe lower to the higher level in the label hierarchy.
The example in Section 1 is formalized bya surjevtive function g as, setting Y :
= { bird 1 ; ::; bird 100 ; turtle 1 ; ::; turtle 100 ; snake 1 ; ::; snake 100 }and Z := { bird ; reptile } , g ( y ) :
= bird if y = bird
i ( i ∈ { 1 ; 2 ; :::; 100 } ) and g ( y ) :
= reptile else.
By making use of {D ead } e ∈E ad , our objective for the invariant prediction is given by
[Equation]
(3) where I ad is the set of invariances:
[Equation]
Note that (3) evaluates the risk with a single training domain while the invariances are given byadditional data of multiple domains.
2.3 Construction of objective function
Among several candidates of the loss and model design, we focus a probabilistic output case and evaluate its error by the cross entropy loss; that is, we model w by p θ : H → P Y , where P Y denotes the set of probabilities on Y and (cid:18) denotes a model parameter.
The risk is then written by
[Equation]
We aim to solve (3) by minimizing the following objective function:
[Equation]
Here, ˆ R e ∗ ( p θ ◦ Φ) denotes the empirical risk of p θ ◦
Φ on the training domain E tr = { e ∗ } evaluatedby D e ∗ : ˆ R e ∗ ( p θ ◦ Φ) :
= − 1 |D e ∗ | P ( x e ∗ ,y e ∗ ) ∈D e ∗ log p θ ( y e ∗ | Φ( x e ∗ )) .
While we can consider somevariations of invariance regularization, we adopt the one used in [10] and construct an objectivefunction as
[Equation]
[Equation]
Here, p Z|H θ : H → P Z is the linear logistic regression model same as [10], Φ is a nonlinear neuralnetwork, and ˆ R ( X e ,Z e
[Equation]
( p Z|H
[Equation]
ad
[Equation]
Φ
[Equation] [Equation]
It is not obvious if the regularization term in (5) is valid as a dependence measure of
P ( g ( Y e ) | Φ( X e )) , since it was proposed for another type of invariance based on argmin w R e ( w ◦Φ) .
The next lemma shows that these notions of invariance are the same in the current setting.
Lemma 1.
When modeling w by conditional probabilities, the following statements are equivalent:
P ( Z e | Φ( X e )) does not depend on e ⇔ argmin p Z|H (cid:18)ad R ( X e ,Z e ) ( p Z|H θ ad ◦ Φ) does not depend on e ,where model p Z|H θ ad in the right hand side runs over all probability densities.
While our objective function (5) is similar to the ones in [10, 21] in that they are composed of anempirical risk and an invariance regularization, the correctness has not been fully discussed so far.
In Section 4, we will mathematically prove the correctness of (5) under some settings.
Hyperparameter selection method
3.1 Hyperparameter Selection in Invarance Learning
The objective function (5) has a hyperparameter (cid:21) to select, as is often the case with IL methods.
Thehyperparameter selection in IL has special difﬁculty, however; because the o.o.d.
problem needs topredict Y e on unseen domains, (cid:21) must be chosen without accessing any data in such unseen domains.
It was reported in [19, 21] that the success of IL methods depends strongly on the careful choice ofhyperparameters, and some of the results even used data from unseen domains in the choice.
[19]reported also experimental results of various IL methods with two CV methods,
training-domainvalidation (Tr-CV) and leave-one-domain-out validation (LOD-CV), and showed that the CV methods failed to select preferable hyperparameters.
In the Colored MNIST experiment, for example, theaccuracy of Invariant Risk Minimization [10] is 52 : 0% at best, which is about a random guess level.
The failure of the CV methods is caused by the improper design of the objective function for CV;they do not simulate the o.o.d.
risk, which is the maximum risk over the domains.
Tr-CV splitsdata in each training domain into training and validation subsets, and takes the sum of the validatedrisks over the training domains.
Obviously, this is not an estimate of the o.o.d.
risk.
LOD-CVholds out one domain among the training domains in turn and validates models with the averageof the validated risks over the held-out domains.
Again, this average does not correspond to theo.o.d.
risk.
In summary, the problem we need to solve is answering the following question: how canwe construct an evaluation function of the o.o.d.
risk from validation data?
In the sequel, we willpropose two methods of CV, which are summarized in Algorithm 1.
3.2 Method I: using data of higher level task
We divide each of D e ∗ ; D e 1 ad ; :::; D e n ad into K parts where |E ad | = n , and use the k -th sample{D e ∗ [ k ] ; D e 1 ad, [ k ] ; :::; D e n ad, [ k ] } and the rest {D e ∗ [ − k ] ; D e 1 ad, [ − k ] ; :::; D e n ad, [ − k ] } for validation and training,respectively.
To approximate the o.o.d.
risk of the trained predictor p θ (cid:21) [ − k ] ◦
Φ λ [ − k ] , we wish to estimate
R e
( p θ (cid:21) [ − k ] ◦ Φ λ [ − k ] ) for e ∈ E ad ∪ { e ∗ } by the validation set.
For e ∗ , we use the standardempirical estimate ˆ R e ∗ [ k ] ( p θ (cid:21) [ − k ] ◦ Φ λ [ − k ] ) .
For e ∈ E ad , we substitute unavailable Y e with Z e and useˆ R ( X e ,Z e ) [ k ] ( p θ (cid:21) [ − k ] ◦ Φ λ [ − k ] ) :
= 1 |D ead; [ k ] |
P ( x e ,z e ) ∈D ead; [ k ] − log p θ (cid:21) [ − k ] ( z e | Φ λ [ − k ] ( x e )) .
Require:
Require: : ˆ P e ( z (cid:8)(cid:8) , → )
← |D ead;z (cid:8)(cid:8) , → | |D ead | , where D ead,z (cid:8)(cid:8) , → :=
( x; z ) ∈ D ead | z
=
z (cid:8)(cid:8) , → } for all e ∈ E
[Table]
z (cid:8)(cid:8)
[Table]
Z
[Table]
for
[Table]
do
[Table]
for to
[Table]
Learn (cid:18)
[Table]
[Table] [Table]
→ .
6: for e ∈ E ad do 7:
[Equation]
( (cid:21)
[Equation]
←
[Equation]
Risk estimation on
[Equation]
8: if CORRECTION then 9:
[Equation]
// Correction term addition.
10: end if 11: end for 12: ˆ R o.o.d.k ( (cid:21) ) ←
[Equation]
// o.o.d.
risk estimation.
13: end for 14: ˆ
[Equation]
( (cid:21) ) //
Final o.o.d.
risk estimation.
15: end for 16: Select (cid:21) ∗ :=
argmin
λ ∈ Λ ˆ R o.o.d.
( (cid:21) )
3.3 Method II: using correction term
Method I can be improved by correcting the replacement R e
= R
( X e ,Y e ) with R ( X e ,Z e ) for e ∈
E ad .
We use the following theorem for the correction:
Theorem 2.
Let Z (cid:8)(cid:8) , → := (cid:8) z ∈ Z (cid:12)(cid:12) | g − 1 ( z ) | > 1 (cid:9) .
For any map Φ : X → H , p θ : H → P Y , andrandom variable ( X; Y ) on X × Y , the following equality holds:
[Equation]
o :
Here, R ( X,Y ) | z (cid:8)(cid:8) , → ( p θ ◦ Φ
[Equation]
(cid:8)(cid:8) , → where
P ( X,Y ) | g ( Y )=
z (cid:8)(cid:8) , → denotes the conditional distribution of ( X; Y ) given the event g ( Y )
=
z (cid:8)(cid:8) , → , andp θ
[Equation]
g
[Equation]
The proof is given in Appendix A.
The theorem shows that, to estimate the correction term, we needto estimate (i) P ( g ( Y e )
=
z (cid:8)(cid:8) , → ) and (ii) R ( X e ,Y e ) | z (cid:8)(cid:8) , → ( p θ (cid:21) [ − k ] ◦ Φ λ [ − k ] ) for every z (cid:8)(cid:8) , → ∈ Z (cid:8)(cid:8) , → .(i) is naturally estimated even on e ∈ E ad : ˆ P ( Z e = z (cid:8) (cid:8) , → ) :=
|D
e ad;z (cid:8)(cid:8) , → | |D ead | , where D ead,z (cid:8)(cid:8) , → := { ( x; z ) ∈ D ead | z
= z (cid:8)(cid:8) , → } .
(ii) is not easily estimable; while a direct simulation of the integration R dP ( X e ,Y e ) | g ( Y e )
=
z (cid:8)(cid:8) , → demands data from ( X e ; Y e ) ∼ P X e ,Y e , our available data D ead on e ∈ E ad is from P X e ,g ( Y e ) , not from P X e ,Y e .
To solve the non-availability of data from P X e ,Y e , we use the training data D e ∗ ∼ P X e ∗ ,Y e ∗ instead.
Namely, (ii) is estimated by
[Equation]
where D e ∗ [ k ] ,z (cid:8)(cid:8) , → := n ( x; y ) ∈
D e ∗ [ k ] | g ( y ) =
z (cid:8)(cid:8) , → o ⊂ D e ∗ [ k ] : In Algorithm 1, the above risk estimate isabbreviated by ˆ R e ∗ | z (cid:8)(cid:8) , → [ k ] ( (cid:21) ) for notation simplicity.
Theoretical analysis
Throughout this section, to avoid discussing the non-trivial effects of nonlinear Φ , we focus on thesimpliﬁed case of variable selections, where the feature map Φ is chosen from the projections of x to a subset of its components.
For example, Φ may be Φ( x 1 ; x 2 ; x 3 ) =
( x 1 ; x 3 ) when x is threedimensional.
This type of IL appears practically in causal inference [14, 13] and regression [40].
Let X :=
X 1 × X 2 where X 1 := R n 1 and X 2 := R n 2 with n 1 ; n 2 ∈ N .
For a projection Φ , letΦ i denote the X
i -component of Φ ( i = 1 ; 2 ).
If Φ has a X 2 -component, we write ImΦ 2 ̸ = ∅ .
Forsimplicity of analysis, the domain set E is deﬁned by all the probability distributions with the ﬁxedmarginal distribution P X
I 1 ,Y I of ( X 1 ; Y ) ; namely,
[Equation]
In this case, for any e ∈ E the variable ( X e ; Y e ) satisﬁes (i)
P
Y e |
Φ X 1 ( X e ) equals to P Y I
|
X I 1 , and(ii)
the marginal distribution P Φ X 1 ( X ) of the invariant feature Φ X 1 ( X ) equals to P X I 1 .
The abovesetting and deﬁnition persist through Section 4.
4.1 Theoretical analysis of our objective function
The following theorem ensures that, neglecting estimations and under some conditions, a minimumof our objective function (5) with careful hyperparameter choice also minimizes the o.o.d.
risk (1):
Theorem 3 (o.o.d. optimality of our objective function) .
Under the setting ( ※ ), additionally assumethat the following condition holds:
(A) For any variable selection Φ with ImΦ 2 ̸ = ∅ , there exist two domains { e 1 ; e 2 } ⊂ E ad such that
[Equation]
Y
[Equation]
) | Φ( X e 2 )) .
Then, there exists (cid:21) ∗ ∈ R such that any minimizer ( (cid:18) † ; (cid:18) † ad ; Φ † ) of (5),
[Equation] [Equation]
∈ argmin
[Equation]
is o.o.d.
optimal, i.e.,
[Equation]
where models p θ and p Z|H θ ad in min θ,θ ad , Φ run all the probability density functions, and Φ runs allthe variable selections.
The gradient ∇ θ ad should be understood as the functional derivative on thespace of probability density functions.
For the proof, see Appendix B. Condition (A) means that E ad has sufﬁcient variation to capture thedesirable invariance Φ X 1 .
4.2 Theoretical analysis of our cross validation methods
In Sections 3.2 and 3.3, we approximate R ( X e ,Y e ) using label Z e of higher level.
While the approximation is not exact, we will prove that the proposed CV methods still select a correct hyperparameterunder some conditions.
We will also elucidate the difference of the two CV methods.
Given hyperparameter (cid:21) , minimizing (5) over the model yields the feature map (variable selection) denoted byΦ λ : X → R n (cid:21) ( n λ ≤ n 1 + n 2 ).
For simplicity of theoretical analysis, we assume that the minimization of (5) achieves perfectly the conditional probability density function of P Y e ∗ | Φ (cid:21) ( X e ∗ ) , denotedby p ∗ ,λ ( y | Φ λ ( x )) .
Then, neglecting estimation errors, the approximated o.o.d.
risk of p ∗ ,λ ◦ Φ λ usedin Methods I and II are represented by the following R
I ( (cid:21) ) and R II ( (cid:21) ) , respectively:
[Equation]
We have the following theoretical justiﬁcation of our CV methods: the chosen (cid:21) gives a minimizerof the correct CV criterion.
For the proofs, see Appendices C and D.
Theorem 4 (Correctness of Method I) .
Assume that the following conditions (i) and (ii) hold: i
Among a set Λ of hyperparameter candidates, there exists
(cid:21)
I
∈ Λ
such that
Φ λ
I
= Φ X 1 .
(ii) Let p e ∗ be the probability density function of P X e ∗ ,g ( Y e ∗ ) .
Then, for any (cid:21) with ImΦ λ 2 ̸ = ∅ ,there is e λ ∈ E ad such that
[Equation]
Here, " ∈ R > 0 is some sufﬁcient small positive real number (that is, 0 < " ≪ 1 ) and (cid:12) :=
H ( Y e ∗ | Φ X 1 ( X e ∗ )) is the conditional entropy of (Φ X 1 ( X e ∗ ) ; Y e ∗ ) .
Then, under the setting ( ※ ), we have
[Equation]
Theorem 5 (Correctness of Method II) .
Assume that, in addition to (i) in Theorem 4, the followingcondition (iii) holds:(ii)’ for any (cid:21) with ImΦ λ 2 ̸ = ∅ , there is e λ ∈ E ad such that
[Equation]
− " holds with probability 1.
Here, " is some sufﬁciently small positive real number and
[Equation]
Then, under the setting ( ※ ), we have
[Equation]
R II ( (cid:21) )
⊂ argmin λ ∈ Λ R o.o.d.
( p ∗ ,λ ◦ Φ λ ) :
The conditions (ii) and (ii)’ impose that, for at least one e λ ∈ E ad , the two domains e λ and e ∗ aredifferent in the following meaning.
if (cid:21) fails to remove domain-speciﬁc factors ( i:e: , ImΦ λ 2 ̸ = ∅ ),for some e λ ∈ E ad , ( x; z ) ∼ P X e(cid:21) ,g ( Y e(cid:21) )
yields low p e ∗ ( z | Φ λ ( x )) with high probability.
On theother hand, ( x; z ) ∼ P X e ∗ ,g ( Y e ∗ ) yields high p e ∗ (
z | Φ λ ( x )) with high probability: that is, e ∗ and e λare different .
The theoretical analysis shows, while Method I is simpler to implement than Method II, Method IIis more applicable.
Noting that (cid:12) ≥ (cid:12) λ and hence, e − β − " ≤ e − β (cid:21) − " , the condition (ii)’ is milderthan (ii).
Recalling that (ii) and (ii)’ impose the discrepancy between E ad and e ∗ as discussed inthe last paragraph, relaxation of conditions from (ii) to (ii)’
implies that method II can be appliedeven when domains E ad and e ∗ have smaller discrepancy than the condition for Method I. Thedifference of these two methods will be demonstrated in Section 6.
The real-world feasibility of theassumptions (ii) and (ii)’ are discussed in Appendix F.
Related work
Fine-tuning The proposed framework uses additional data from multiple domains as well as thetraining data for the target task.
It is relevant to Transfer learning (TL) [22–24] and meta-learning[30, 41], which realize fast and accurate learning for a new target task based on a model pre-trainedwith additional data sets.
For example, after initial learning with a large data set, ﬁne tune [22, 23]re-trains the model with the target task, while frozen feature [24] ﬁxes the pre-trained model andtunes a head network.
Although they show advantages in many learning problems, they may notwork effectively in the current setting; in the ﬁne-tuning with the target task ( X e ∗ ; Y e ∗ ) , the modeltends to learn biased correlation in the data set and does not generalize to unseen domains.
Someﬁne-tuning methods will be compared with the proposed approach in Section 6.
Domain adaptation by deep feature learning
Domain adaptation strategies by deep featurelearning [25–27, 31, 32] assume that we can access input data on a test domain in advance, andtry to obtain data representation
Φ( X e ) that follows the same distribution for the training and testdomains.
While the strategies lead to high predictive performance on a test domain similar to atraining domain, such Φ does not function by discarding environmental factors from X e ∈ X asnoted in [10].
Experimental comparisons will be shown in Section 6.
Distribution robust methods [59–61] and other debias methods [62, 63] are discussed in Appendix.
Experiments
We study the effectiveness of the proposed framework and CVs through experiments, comparingthem with several existing methods: empirical risk minimization (ERM), ﬁne-tuning methods, and
[Table]
TDV selects (cid:21) that yields the highest performance on e 2 .
The best scores are
[Table]
deep domain adaptation strategies.
For ﬁne-tuning, we employ two typical methods of transferlearning: ﬁne tune (FT) and frozen feature (FF) [22–24].
As a deep domain adaptation technique, weadopt the state-of-the-art method DSAN [31].
We also compare our two CVs (CVI and CVII) withconventional CVs: training-domain validation (Tr-CV) and leave-one-domain-out cross-validation(LOD-CV) [19].
We have two hyperparameters to be selected by CV.
In the training with (5), we set(cid:21) :
= (cid:21) before when the training epoch is less than a certain threshold t , and (cid:21) :
= (cid:21) after if the epochis larger than t .
It is known that these two hyperparameters are critical for IL methods to achievegood results.
From a set of candidates, each of the CV methods selects a pair ( t; (cid:21) after ) .
To know thebest possible performance among the candidates, we also apply the test-domain validation (TDV)[19], which selects the hyperparameters with the unseen test domain, and thus is not applicable inpractical situations.
Additional experiments and experimental details can be found in Appendices Hand I, respectively.
The code is available in Supplementary Material.
Colored MNIST
We apply our framework to Colored MNIST [10] with Y =
[10] and Z :=
We aim to predict Y e ∈ Y from digit image data X e ∈ R 2 × 24 × 24 .
The label Y e is changedrandomly to one of the rest uniformly with a probability of 25% .
All digits in images are coloredred or green.
The domain e ∈ [0 ; 1] controls the color of digits; the digits Y e > 4 and Y e ≤ 4 arecolored in red and green, respectively, with probability e .
In training, D e ∗ ∼ P X 0 : 1 ,Y 0 : 1 is drawnwith sample size n e ∗ = 5000 , and in testing, Y e is predicted from X e for e 2 := 0 : 9 .
Regardingthe higher level Z e , the task is to predict Z e = 0 for X e in 0 − 4 and Z e
= 1 for 5 − 9 (that is,g ( Y e ) =
if Y e > 4 and else, g ( Y e ) = 0 )
The label Z e is swapped randomly with 25% .
We set 274 E ad = { 0 : 1 ; 0 : 3 ; 0 : 5 ; 0 : 7 ; 0 : 9 } with n e = 5000 for each e ∈ E ad .
We model Φ by a 3 -layer neuralnet.
Setting the maximum epoch 500 and (cid:21) before := 1 : 0 , we select ( t; (cid:21) after ) from 4 × 7 candidateswith t ∈ { 0 ; 100 ; 200 ; 300 } ; (cid:21) after ∈ { 10 0 ; 10 1 ; :::; 10 6 } by each of the CVs.
Table 1 shows the test accuracies for 2000 random samples in the domain e 2 .
The results, togetherwith additional ones in Appendices H.1 and H.2, demonstrate that the proposed methods signiﬁcantly outperform the others for e 2 .
Among the two proposed methods, CV II yields higher testaccuracies on e 2 .
Table 2 shows the accuracy gain of each CV from TDV with the same data sets fordomain e 2 .
These results, together with Appendices H.1 and H.2, concur with the theory in Section4 suggesting that CVII succeeds in wider situations, resulting in smaller errors.
ImageNet To see the performance of the proposed methods for more practical data, they are applied to the ImageNet [53] with its label re-annotated imitating BREEDS [52], which proposes a
[Figure]
(cid:17795)(cid:17795)(cid:17795)
[Figure]
method for re-annotating ImageNet to create ano.o.d.
benchmark.
The target task here is to predictlabels Y e ∈ Y of images X e ∈ R 3 × 224 × 224 .
Weconduct three experiments with |Y| = 3 ; 7 ; 17 .
Foreach experiment, we prepare image datasets in different two manners e 1 and e 2 .
2, 4 and 8 classes out of 3,7, 17 classes, respectively, are composed of differentsubtypes between e 1 and e 2 ; for example, the imagesof class bird in e 1 are composed of ruffed grouse andindigo bunting, and the bird images on e 2 are composed of albatross and water ouzel (Figure 2).
In detail, show Appendix I.3.
In training, D e ∗ ∼ P
X
e 1 ,Y is drawn, and in testing, Y e is predicted from X e one 2 .
The coarser label Z e is binary (that is, Z = [2] ), and the sample in the higher level D ead of( X e ; Z e ) is drawn from both e 1 and e 2 .
Here, D e 1 ad is the same as D e ∗ but with labels re-annotatedby g .
We model Φ by ResNet50 [29].
Setting the maximum epoch 32 and (cid:21) before := 0 : 1 , we select( t; (cid:21) after ) from 3 × 4 candidates with t ∈ { 10 ; 20 ; 30 } ; (cid:21) after ∈ { 0 ; 1 ; 10 ; 100 } by each of the CVs.
Table 1 shows the test accuracies on e 2 .
We can see that the proposed framework succeeded incapturing the predictor invariant to the change of e , while the other methods failed.
Table 2, whichshows the difference between accuracies by TDV and each CV for the same data set with e 2 , verifythat CVI and II selects (cid:21) with the smallest error.
Comparison of two CV methods To highlight the difference between the proposed two CVs, wecompare them regarding the discrepancy between the additional domains of higher level E ad and thedomain for training of the target task e ∗ .
We used synthesized data with X = R 2 , Y =
[10] and 309 Z :=
[2] , preparing ten distributions { N i } 10 i =1 on R 2 , which include a domain-speciﬁc factor in thesecond component depending on e ∈ Z (see Appendix I.2 for explicit representations of { N i } 10 i =1 ).
The task is to predict the distribution label i ∈ { 1 ; : : : ; 10 } .
Setting e ∗ :
= 20 with n e ∗ = 60000 ,the test task is to predict the label for domain e = − 20 .
Regarding the task with label of higherlevel, we use g ( y ) =
if y > 4 and g ( y ) = 0 else.
We draw D ead ∼ P X e ,Z e ( n e = 20000 ) from 314 E ad = { e ad ; 40 } , where e ad ranges from − 9 to 1 .
As e ad increases, e ad approaches to e ∗ .
Themodel Φ is a 3 -layer neural net.
We set the maximum epoch 500 and t = 0 , and select (cid:21) after from 4candidates (cid:21) after ∈ { 0 ; 0 : 001 ; 80 ; 100 } by each CV method.
e ad = −e ad = −e ad = −e ad = −e ad = −e ad = −e ad = −e ad = −e ad = −e ad =e ad =
[Table]
Table 3 shows the test accuracy on e = − e ∗ with 2000 random samples ( x; y ) ∼ P X − e ∗ ,Y − e ∗ .
Fromthe results, we can see that CVII tends to select better hyperparameters than CVI, especially in thecase where the variation among the domains is smaller as e ad approaches to e ∗ .
This accords withthe theoretical results in Theorems 4 and 5, which show that CVII ﬁnds a correct hyperparameter insmaller discrepancy between E ad and e ∗ than CVI.
Conclusion
We have proposed a new framework of invariance learning: assuming the availability of datasetsfor another relevant task in higher label hierarchy, we obtain an invariant predictor for the targetclassiﬁcation task using training data in a single domain.
We have also proposed two CV methods forhyperparameter selection, which has been an outstanding problem of previous methods for invariantlearning.
Theoretical analysis has revealed correctness of our methods, including CVs, and theexperimental results have demonstrated the effectiveness of the proposed framework and CVs.
Limitations and Potential societal impact In Theorem 3, it is ensured that our objective functiongives a minimizer of the o.o.d.
risk, only if at least two domains have different distributions.
Ingeneral, different domains do not have necessarily different distributions.
Judging the discrepancyis a further important problem.
As a positive impact, our method will enable us to estimate predictorsthat don’t use discriminatory factors such as gender.
There may be some negative aspects in that ourmethod removes important information for a prediction as well as unnecessary ones.
Checklist
For all authors...(
a)
Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes](
b
) Did you describe the limitations of your work?
[Yes] In Section 1, we note that data with targetclassiﬁcation label needs from at least one domain.
In Section 4, we theoretically reveal theconditions for our objective function and CV methods to succeed.
(c)
Did you discuss any potential negative societal impacts of your work
[Yes
] It
is included in
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] They are included in Appen-
dices.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes] We put the code in the supple-
mental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were cho-
sen)?
[Yes] They are speciﬁed in Section 6 and Appendix I.4.
(c) Did you report error bars (
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
A Proof of Theorem
[Equation]
[Equation]
By the deﬁnition of p (cid:18) ( y | Φ( x ) ; g ( Y ) =
z ) in Theorem 2, p (cid:18) (
[Equation]
| Φ( x )) p
[Equation]
( g ( y ) | Φ( x )) =
p (cid:18) ( y | Φ( x ) ; g ( Y ) =
z ) holds,where z = g ( y ) .
Therefore, we obtain(6)
= −
[Equation]
→
[Equation] [Equation]
Noting that, for any z , → ∈ Z − Z (cid:26)(cid:26) , → and y :
= g − 1 ( z , → ) 1 , p (cid:18) ( y | Φ( x ) ; g ( Y )
=
z , → ) = 1 holds, we can see thatlog p (cid:18) ( y | Φ( x ) ; g ( Y )
=
z , → ) = 0 .
The second term in the last line thus equals to zero, which concludes theproof.
□
B Proof of Theorem
Before Proving Theorem 3, we recap the problem setting of variable selection for invariance learning discussed 499 in the ﬁrst paragraph of Section 4.
Let X :=
X 1 × X 2 where X 1 := R n 1 and X 2 := R n 2 with n 1 ; n 2 ∈ N , so that X = R n with n = n 1 + n 2 .
For the class of feature map Φ , we consider the projections of X to a subset of variables ( x 1 ; : : : ; x n ) .
We 502 have thus 2 n candidates of feature maps, and choose the best one in optimizing (5).
Recall that Φ i denote the
X i -component of Φ ( i = 1 ; 2 ) and ImΦ 2 ̸ = ∅ means that the range of Φ has a X 2 -component.
We assume that ( X I 1 ; Y I ) on X 1 × Y gives the invariant feature X I 1 and invariant predictor P Y I
|
X
I 1 .
Notethat the invariant features are given by X 1 only, and the invariance learning is expected to ﬁnd the projection 506 to X 1 as Φ .
Also, we assume that the domains E are composed of all the possible distributions of X 2 with the 507 marginal distribution of X 1 × Y being the same as that of ( X I 1 ; Y I ) .
Namely, we introduce the class
T all
[Equation]
random variable on
[Equation]
o for all the domains, and often specify a domain directly by a random variable ( X; Y ) in T all .
The training data for the target task is given by a single domain ( X ∗ ; Y ∗ ) ∈ T all .
For the domains of theadditional task with higher class labels, we consider a subclass T ad ⊂ T all .
For a projection Φ : X → R n Φ1 z , → ∈ Z − Z (cid:26)(cid:26) , → implies that | g − 1 ( z , → ) | = 1 and therefore, g − 1 ( z , → ) is determined uniquely.
Note that there is no chance that | g − 1 ( z , → ) | = 0 by the surjectivity of g .
with its range n Φ variables, let p ∗ ; Φ : R n Φ → P Y denote the conditional probability density functions (p.d.f.)of P ( Y ∗ | Φ( X ∗ )) .
With a slight abuse of notation, for any probability P (cid:18) on X × Y and a projection Φ , thedensity function of the conditional distribution P (cid:18) ( Y | Φ( X )) is denoted by p (cid:18) ◦ Φ .
Throughout this section, the projection to the components of X 1 is denoted by Φ X 1 , which is the desiredprojection to give the invariant predictor.
From the condition of T all , for the projection Φ X 1 , the conditionalprobability P Y | Φ X 1 ( X ) for any random variable ( X; Y ) ∈ T all is the same; namely, letting p I : X 1 → P Ydenote the conditional p.d.f.
of the invariant predictor P Y I
|
X I 1 , we have
[Equation]
for any ( X e ; Y e ) ∈ T all , where p e is the conditional p.d.f.
of P Y e | Φ X 1 ( X e ) .
We restate Theorem 3 as follows.
Theorem 6.
Under the above assumptions and notations, assume that the following condition holds:(
A)
For
any projection
Φ with
ImΦ
2 ̸ = ∅
there exist ( X e 1 ;
Y
e
; ( X
e 2 ; Y e 2 ) ∈ T
ad such that
P ( g ( Y e 1 ) | Φ( X e 1 ))
̸
= P ( g ( Y e 2 )
| Φ( X e 2 )) .
Then, there exists (cid:21) ∗ ∈ R such that a minimizer ( (cid:18) † ; (cid:18) † ad ; Φ † ) of the objective function
[Equation]
is o.o.d.
optimial, i.e.,
[Equation]
where p (cid:18) and p Z|H (cid:18) ad in min (cid:18);(cid:18) ad ; Φ run all the p.d.f.s, and Φ runs all the variable selections.
The gradient ∇ (cid:18) adshould be understood as the functional derivative on the space of p.d.f.
Before proving Theorem 6, we prepare one lemma, which asserts that, if ImΦ 2 ̸ = ∅ , at least one domain in 528 T ad has non-trivial gradient:
Lemma 7.
[Equation]
Proof.
It sufﬁces to prove that, for any projection Φ with ImΦ 2 ̸ = ∅ and p Z|H (cid:18) ad , there is ( X e ; Y e ) ∈ T ad suchthat ∥∇ ˆ (cid:18) ad = (cid:18) ad R ( X e ;g ( Y e ))
( p Z|H ˆ (cid:18) ad ◦ Φ) ∥ 2 ̸ = 0 .
We prove this by contradiction.
Suppose that there exist aprojection Φ with ImΦ 2 ̸ = ∅ and p Z|H (cid:18) ad which satisfy
[Equation]
From Assumption (A), take ( X e 1 ; Y e 1 ) and ( X e 2 ; Y e 2 ) in T ad such that P ( g ( Y e 1 ) | Φ( X e 1 )) ̸ =
P ( g ( Y e 2 ) | Φ( X e 2 )) .
Note that the risk is deﬁned by the cross-entropy loss:
[Equation]
It is well known that this is minimized in the space of probability distributions if and only if p Z|H ˆ (cid:18) ad equals to
P ( Y e | Φ( X e )) .
From ∥∇ ˆ (cid:18) ad = (cid:18) ad R ( X e ;g ( Y e ))
( p Z|H ˆ (cid:18) ad ◦ Φ) ∥ 2 = 0 for ( X e 1 ; Y e 1 ) and ( X e 2 ; Y e 2 ) , we canconclude that p Z|H (cid:18) ad should equal to both of P ( g ( Y e 1 )
| Φ( X e 1 )) and P ( g ( Y e 2 ) | Φ( X e 2 )) .
This contradictswith the assumption P ( g ( Y e 1 )
| Φ( X e 1 ))
̸
= P ( g ( Y e 2 )
| Φ( X e 2 )) .
Proof
of Theorem 6
Let Φ id denote the identity map of X .
Deﬁne the constants C 1 , C 2 , and C 3 by
[Equation]
where H ( Y I |
X I 1 ) and H ( Y ∗ | X ∗ ) denote the conditional entropy.
Note that C 3 is well-deﬁned because of thepositivity result of Lemma 7.
Take (cid:21) ∗ such that (cid:21) ∗ > C 3 .
For notational simplicity, the objective function (9) is denoted by O ( (cid:18); (cid:18) ad ; Φ) ;namely,
O ( (cid:18); (cid:18) ad ; Φ) :
= R ( X ∗ ;Y ∗ ) ( p (cid:18)
[Equation]
[Equation]
We prove the theorem in three steps.
[Equation]
proof of Step
We will prove p I ∈ argmin p : X→P Y R o:o:d: ( p ) .
From the deﬁnition
[Equation]
p
[Equation] [Equation]
: ( p ) holds if and only if
[Equation]
for any p (cid:18) :
X → P Y .
Note that, as discussed before Theorem 6, for any ( X e ; Y e ) ∈ T all , we have
P
[Equation]
e | X
[Equation]
= P Y I |
X
I 1 .
Then, it sufﬁces to prove that for any p (cid:18) there exists ( X e ′ ;
Y e ′ ) ∈ T all such that
[Equation]
Deﬁne ( X e ′ ; Y e ′ ) ∈
T all such that its distribution is the direct product P X
I 1 ;Y I ⊗ P X e ′ 2 , where P X e ′ 2 is anarbitrary distribution on X 2 .
In this case, the left hand side of (10) is given by
[Equation]
We can see that, for any x 2 ∈ X 2 , the inequality
[Equation]
holds, since the minimum of the cross entropy loss is attained at the conditional p.d.f.
p I .
Integrating thisinequality with P X e ′ 2 , we have
[Equation] [Equation]
Eqs.
(11) and (12) show (10), from which the assertion is obtained by − R log p
I ( Y I | X
[Equation] [Equation]
dP
[Equation]
Y
[Equation]
=
H
[Equation]
[Table] [Table]
of Step
[Table]
[Equation]
≥
[Equation]
On the other hand, by taking Φ =
Φ I , we obtain
[Equation]
Since p Z|H ˆ
[Equation]
ad ◦ Φ I = p
I ( g ( Y I ) |
X I 1 ) does not depend on (cid:18) ad , the gradient is zero, and therefore
[Equation]
We thus obtain
[Equation]
which completes the proof.
Step
If
[Equation]
p
[Equation]
Φ † ) ∈
[Equation]
(cid:18)
[Equation]
(cid:18) ad ; Φ O ( (cid:18); (cid:18) ad ; Φ) , then R o
:o:d: ( p (cid:18) †
[Equation]
Φ
[Equation]
= H
[Equation]
| X I 1 ) .
proof of Step
569 From Step 1, we have H ( Y I |
X I 1 ) ≤ R o:o:d: ( p (cid:18) † ◦ Φ † ) .
We will probe the converse inequality.
From Step 2, we have ImΦ † 2 = ∅ .
This tells R o:o:d: ( p (cid:18) † ◦ Φ † )
=
R e ∗ ( p (cid:18) † ◦ Φ † ) , since P X 1 ;Y are the samefor all elements in T all .
Therefore,
[Equation]
Final
step for
the
proof
of Theorem
For ( (cid:18) † ; (cid:18) † ad ; Φ † ) ∈ argmin (cid:18);(cid:18) ad ; Φ O ( (cid:18); (cid:18) ad ; Φ) , Step 1 and Step 3 show
[Equation]
which completes the proof.
C Proof of Theorem 4
Before the proof, let us rearrange some notations introduced in Section 4.2.
Notations are the same as in 577 Appendix B. Recall that we assume, given hyperparameter (cid:21) , the minimization of (5) achieves the globaloptimum perfectly, which yields the projection (variable selection)
Φ (cid:21) ( x ) : X → R n (cid:21) ( n (cid:21) ≤ n 1 + n 2 ) andthe conditional p.d.f.
of P Y e ∗ | Φ (cid:21) ( X e ∗ ) , denoted by p ∗ ;(cid:21) ( y | Φ (cid:21) ( x )) .
The X 1 and X 2 components of Φ (cid:21) ( X ) aredenoted by Φ (cid:21) 1 ( X ) and Φ (cid:21) 2 ( X ) , respectively.
For (cid:21) ∈ Λ and the training variable ( X ∗ ; Y ∗ ) for the target task, the conditional p.d.f.
of P ( Y ∗ | Φ (cid:21) ( X ∗ ))
given the selected variables is denoted by p ∗ ;(cid:21) : R n (cid:21) → P Y .
Deﬁne R o:o:d: ( (cid:21) ) and R I ( (cid:21) ) by
[Equation]
o:o:d: ( (cid:21) ) :=
[Equation] [Equation]
respectively.
We restate Theorem 4 with some notation arrangements:
Theorem 8.
Under the above notations, assume the following two conditions:
(I) there is (cid:21) I ∈
Λ such that Φ (cid:21)
I =
Φ X 1 , where Φ X 1 is the projection to the X 1 -components.(II)
Let p ∗ be the p.d.f of P X ∗ ;g ( Y ∗ ) .
For any (cid:21) with ImΦ (cid:21) 2 ̸ = ∅ , there is ( X e (cid:21) ;
Y e (cid:21) )
∈ T ad such that
[Equation]
;g
[Equation]
satisﬁes p
[Equation]
− " with probability 1 in P X e(cid:21)
[Equation]
g ( Y e(cid:21) ) :
Here, " ∈ R > 0 is a sufﬁciently small positive real number (that is, 0 < " ≪ 1 ) and (cid:12) :
= H ( Y ∗ | ( X ∗ 1 )) is the conditional entropy of (( X ∗ 1 ) ; Y ∗ ) .
Then, we have
[Equation]
To prove Theorem 8, we prepare three lemmas, in which the notations are the same as in Theorem 8 and 588 conditions (I) and (II) in Theorem 8 are also imposed.
Lemma 9.
(cid:21) I ∈ argmin (cid:21) ∈
Λ R o:o:d: ( (cid:21) ) .
Lemma 10.
If ˆ (cid:21) ∈ argmin (cid:21) ∈ Λ R I ( (cid:21) ) , then ImΦ ˆ (cid:21) 2 = ∅ .
Lemma 11.
If ˆ (cid:21) ∈ Λ satisﬁes ImΦ ˆ (cid:21) 2 = ∅ , then R I (ˆ (cid:21) )
=
R o:o:d: (ˆ (cid:21) ) .we prove Theorem 8 based on the above lemmas, before proving them.
proof of Theorem 8 .
Take ˆ (cid:21) ∈ argmin R
I ( (cid:21) ) .
Then, ImΦ ˆ (cid:21) 2 = ∅ holds by Lemma 10 and therefore, R I (ˆ (cid:21) )
=
R o:o:d: (ˆ (cid:21) ) holdsby Lemma 11.
Moreover, R o:o:d: ( ˆ (cid:21) ) ≥ R o:o:d: ( (cid:21) I ) holds by Lemma 9 and R o:o:
d: ( (cid:21) I ) =
R I ( (cid:21) I ) holdsby Lemma 11 (since Φ (cid:21) I is the projection onto X 1 , ImΦ (cid:21) I 2 = ∅ ).
By the assumption ˆ (cid:21) ∈ argmin (cid:21) ∈ Λ R
I ( (cid:21) ) ,
R I ( (cid:21) I ) ≥
R I (ˆ (cid:21) ) holds.
Arranging these inequalities, we obtain
[Equation]
13) in which the inequalities must be equalities.
Hence, we obtain R o:o
d: (ˆ (cid:21) ) =
R o:o:d: ( (cid:21) I ) .
Because (cid:21) I achievesthe minimum of R o:o:d: (Lemma 9), so does ˆ (cid:21) , which concludes the proof.
□
Since Lemma 9 is proven in the proof of Theorem 6 (especially, the proof in Step 1), we may prove the others.
proof of Lemma 10 .
Let us prove the contraposition of Lemma 10.
Take ˆ (cid:21) ∈ Λ with ImΦ ˆ (cid:21) 2 ̸ = ∅ .
To prove that ˆ (cid:21) =
∈ argmin R I ( (cid:21) ) ,we may prove that R I (ˆ (cid:21) )
>
R I ( (cid:21) I ) since (cid:21) I ∈ Λ (Assumption (I) in the statement).
It then sufﬁces to provethe following:there exists ( ¯ X
¯ Y
∈ T ad such that
[Equation]
From Condition (II), we can take ( X e ˆ (cid:21) ; Y e ˆ (cid:21) )
∈ T ad such that
[Equation]
probability
[Equation]
To prove (14), we prepare one supplementary inequality:
Supplementary Inequality Z
[Equation]
This inequality can be easily seen; from the way of taking e ˆ (cid:21) , we have
[Equation]
with probability 1 with respect to ( x; z )
[Equation]
P
[Equation]
) , and thus the integration proves the inequality.
Proof of Inequality (14) .
610 It follows from the above supplementary inequality that
Z
[Equation]
Since Φ (cid:21) I =
Φ X 1 by Condition (I), the discussion at (8) tells that R I ( (cid:21) I ) =
H ( Y I | X
[Equation] [Equation]
= H ( Y ∗ | X
[Equation]
1 ) ,which concludes (14) and the proof.
Proof of Lemma 11 .
614 Take ˆ (cid:21) ∈ Λ that satisﬁes ImΦ ˆ (cid:21) 2 = ∅ .
Then, P Φ ˆ (cid:21) ( X ) ;Y =
P Φ ˆ (cid:21) ( X I ) ;Y I holds for any ( X; Y ) ∈ T all because of
P X 1 ;Y =
P X
I 1 ;Y , and therefore, R ( X;g ( Y ))
( p ∗ ; ˆ (cid:21) ◦ Φ ˆ (cid:21) )
=
R ( X I ;g ( Y I ))
( p ∗ ; ˆ (cid:21) ◦ Φ ˆ (cid:21) ) and R ( X ∗ ;Y ∗ ) ( p ∗
[Equation]
ˆ (cid:21) ◦ Φ ˆ (cid:21) )
=
R ( X I ;Y I ) ( p ∗ ; ˆ (cid:21) ◦ Φ ˆ (cid:21) ) hold.
These two equalities lead the following equality:
[Equation]
It follows from Theorem 2 that
[Equation]
≥ R ( X I ;g ( Y I )
[Equation]
( p ∗
[Equation]
ˆ (cid:21)
[Equation]
Φ ˆ (cid:21)
[Equation]
Therefore, from (16), we have R I (ˆ (cid:21) )
=
R ( X I ;Y I ) ( p ∗ ; ˆ (cid:21) ◦
Φ ˆ (cid:21) ) .
Since P Φ ˆ (cid:21) ( X ) ;Y are the same for any elementsin T all , we obtain
[Equation]
which concludes the proof.
D Proof of Theorem 5
We restate Theorem 5 with some notation arrangements:
Theorem 12.
Notations are the same as in the statement of Theorem 8.
Deﬁne R II ( (cid:21) )
by
[Equation]
where the correction term D (cid:21) ( Y ) is deﬁned by
[Equation]
o
In addition to the condition (I), assume the following condition (II)’:
(II)’
Let p ∗ be the p.d.f.
of P X ∗ ;g ( Y ∗ ) .
For any (cid:21) with ImΦ (cid:21) 2 ̸ = ∅ , there is ( X
[Equation]
; Y
[Equation]
) ∈ T ad such that
[Equation]
Here, " is some positive real number and
[Equation] [Equation]
Then, we have
[Equation]
Note that, in D (cid:21) ( Y ) , although the random variable Y is given by ( X; Y ) ∈ T all , the marginal distributions of 629 Y s are the same by the assumption of T all .
Thus, hereafter, we use D (cid:21) for the notation, and
D (cid:21) =
X
[Equation]
Note also that (cid:12) (cid:21) =
H ( Y ∗ | X ∗ 1 ) − D
cid:21) .
We ﬁrst show lemmas before the proof of the theorem.
Lemma 13.
If ˆ (cid:21) ∈ argmin (cid:21) ∈ Λ R II ( (cid:21) ) , then ImΦ ˆ (cid:21) 2 = ∅ .
Lemma 14.
If ˆ (cid:21) ∈ Λ satisﬁes ImΦ ˆ (cid:21) 2 = ∅ , then R II (ˆ (cid:21) )
=
R o:o:d: (ˆ (cid:21) ) .proof of Theorem 12 634 Combining the above two lemmas and Lemma 9, we can derive the required assertion in essentially the samemanner as in the proof of Theorem 8.proof of Lemma 13 .
Let us prove the contraposition of Lemma 13.
Take ˆ (cid:21) ∈ Λ with ImΦ ˆ (cid:21) 2 ̸ = ∅ .
To prove that ˆ (cid:21) = ∈ argmin R II ( (cid:21) ) ,we may prove
that R II (ˆ (cid:21) ) > R II ( (cid:21) I ) since (cid:21) I ∈ Λ (Assumption (I) in the statement).
To show this, it sufﬁcesto prove the following statement:there is ( ¯ X; ¯ Y ) ∈ T ad such that R ( ¯ X;g (¯ Y ))
(ˆ (cid:21) )
+ D ˆ (cid:21) >
R II ( (cid:21) I ) : (17)
Take ( X e ˆ (cid:21) ; Y e ˆ (cid:21) )
∈ T ad as in Condition (II)’.
Then, in the same way as the proof of Lemma 10, we have the following inequality: Z
[Equation]
n
[Equation]
which leads us to obtain
[Equation]
On the other hand, for any ( X; Y ) ∈ T all the marginal distribution of ( Y; Φ I ( X )) is the same as that of( Y ∗ ; X ∗ 1 ) .
Noting that R ( X;g ( Y )) ( p ∗ ;(cid:21) I ◦ Φ (cid:21) I )
+ D (cid:21) I depends only on ( Y; X 1 ) , we have
[Equation]
Now, Lemma 2 implies
[Equation]
From (18), (19), and (20), we thus have
[Equation]
which shows (17) and completes the proof.proof of Lemma 14 .
Take ˆ (cid:21) ∈ Λ such that ImΦ ˆ (cid:21) 2 = ∅ .
It follows from ImΦ ˆ (cid:21) 2 = ∅ that P Φ ˆ (cid:21) ( X ) ;
Y =
P Φ ˆ (cid:21) ( X ∗ ) ;Y ∗ holds for all( X; Y ) ∈ T all .
Therefore,
R o
[Equation]
ˆ
[Equation]
Likewise, from the condition of ˆ (cid:21) , the deﬁnition of R II (ˆ (cid:21) ) involves the same distribution for ( Y; Φ ˆ (cid:21) ( X )) , andthus
[Equation]
In a similar way to the proof of Lemma 13, Theorem 2 tells
[Equation]
This completes the proof.
E Sufﬁcient Conditions of (ii) and (ii)’
In the section, we reveal sufﬁcient conditions of e ∗ for there to exist ( X e (cid:21) ; Y e (cid:21) )
that satisﬁes (ii) and (ii)’ in
Theorems 4 and 5, respectively.
Theorem 15.
Notations are the same as in Theorem 8.
Assume that ( X ∗ ; Y ∗ ) satisﬁes the following condition:(A2)
For a sufﬁciently small " ≪ 1 , any (cid:21) with ImΦ (cid:21) 2 ̸ = ∅ , any a ∈ ImΦ (cid:21) 1 , and any b ∈ Y , there existsc ( (cid:21); a; b ) 2 such that
[Equation]
Then, for any (cid:21) with ImΦ (cid:21) 2 ̸ = ∅ , there exists ( X e (cid:21) ; Y e (cid:21) )
∈ T all such that the inequality in Theorem 8 (ii)holds.
Remark .
The condition (A2) means that, in the domain e = e ∗ , the affection of domain-speciﬁc factors ( = X 2 )to the response variable Y e ∗ is large; indeed, the inequality in (A2) means that, if (cid:21) fails to remove domainspeciﬁc factors ( i:e: , ImΦ (cid:21) 2 ̸ = ∅ ), we can control the probability of Y e ∗ =
b by selecting c for any b ∈ Y .
Notealso that the inequality (A2) is a lower bound of the likelihood, while the condition in (ii), Theorem 8, is an 665 upper bound of the likelihood.
Although imposing an upper bound might look reasonable to reﬂect non-ﬁttingof the projection Φ (cid:21) , Theorem 15 shows that we can use lower bound as a sufﬁcient condition.
Proof.
Fix (cid:21) with ImΦ (cid:21) 2 ̸ = ∅ .
Take ( ¯ X; ¯ Y ) ∈ T all such that its probability measure corresponds to ¯
P X
| Y;X 1 × P Y I ;
X I 1 ; where ¯ P X 2 | Y;X 1 is deﬁned by, setting ˆ c (ˆ (cid:21); a; b ) by
[Equation]
¯
[Equation]
(ˆ (cid:21);a;b ) :
Here, for c ∈ X 2 , the probability measure (cid:14) X 2 = c on X 2 denotes a Dirac 668 measure at c ∈ X 2 .
Before proving Theorem 15, we prepare the following inequalities: Supplementary Inequality 1 .
[Equation]
To see the fact, take b ∗
∈
Y
such that g ( b ∗ ) ̸
= g ( b ) 3 .
Then, by the condition (ii) of Theorem 5 and ImΦ ˆ (cid:21) 2 ̸ = ∅ , there exists c (ˆ (cid:21); a; b ) ∈
X 2 such that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Therefore,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
By the
Proof of Theorem
We may prove that P ¯ X; ¯ Y ( A )
= 1 where (
[Equation]
Then,
P
[Equation]
holds where A ( x 1 ;y ) :
= { x 2 ∈ X 2 | (( x 1 ; x 2 ) ; y
) ∈
X × Y} .
By the Supplementary Inequality 1,ˆ c (ˆ (cid:21); x 1 ; y
) ∈
A ( x 1 ;y ) holds and therefore, (cid:14) X
=ˆ c
(ˆ (cid:21);x 1 ;y )
( A ( x 1 ;y ) )
= 1 , which leads us to the equation 673 R dP Y I ;X I 1 ( x 1 ; y ) (cid:14) X
=
ˆ c (ˆ (cid:21);x 1 ;y )
( A ( x 1 ;y ) )
= 1 .
Theorem 16.
Notations are same as in Theorem 8 and 12.
( X ∗ ; Y ∗ ) satisﬁes the following condition:(A2)’
For a sufﬁciently small " ≪ 1 , the following statement holds:∀ (cid:21) with ImΦ (cid:21) 2 ̸ = ∅ , ∀ (cid:11) ∈ ImΦ (cid:21) 1 , ∀ b ∈ Y , ∃ c ( (cid:21); a; b ) s.t.
P ( Y ∗ =
b | Φ (cid:21) 1 ( X ∗ ) = a; Φ (cid:21) 2 ( X ∗ )
=
c ) ≥ (1 − e − (cid:12) (cid:21) )
+ " .
Then, ∀ (cid:21) with ImΦ (cid:21) 2 ̸ = ∅ , there exists ( X e (cid:21) ; Y e (cid:21) )
∈ T all such that the inequality in (ii)’ holds.
The proof of Theorem 16 is essentially same as the one of Theorem 15 and therefore, we omit.
F The real-world feasibility of (ii) and (ii)’
In the subsection, we discuss the feasibility of (ii) and (ii)’, and show these conditions are not necessarily 684 strong.
First, we discuss the Condition (ii).
Since (cid:12) =
H ( Y e | Φ X 1 ( X e )) is the conditional entropy, we have
[Equation]
≤
[Equation]
≤
[Equation]
|Y| and hence
[Equation]
holds.
We can see that Condition (ii) is weak if e − (cid:12) − " approaches 1 , or if (cid:12) is small.
Recall that Φ X 1 ( X e )is the bias-removed feature of X e (digit of CMNIST, or object of ImageNet, for example).
We can thenexpect that, in many real-world settings, (cid:12) =
H ( Y e | Φ X 1 ( X e )) is often small, since the bias-removed featureΦ X 1 ( X e ) should have a large amount of information on the labels.
Condition (ii) is satisﬁed if the likelihoodp e ∗ ( z | Φ (cid:21) ( x )) evaluated at a random point ( x; z ) ∼ P X e ;g ( Y e ) is bounded by the large value e − (cid:12) − " for atleast one e ∈ E ad , so that the inequality in (ii) is likely to hold.
Noting that (ii)’ is weaker than (ii) (line 228 inthe mainbody), the feasibility of (ii)’ is concluded from one of (ii).
G Additional related works
Distributionally robust optimization
The propose work tries to minimize the worst risk among risks on 694 distributions perturbated from a training distribution [59–61].
The perturbated distributions are formalized asa small " -ball centered at the training distribution evaluated by some divergence.
In our setting, the change of 696 distributions in training and testing is necessarily small; if the background changes drastically, it is expectedthat its corresponding distribution also changes drastically.
Concerning other strategies, [62] tries to obtain a de-biased feature Φ following the independence Φ( X ) ⊥⊥ E ,seeing E as a random variable E .
Recently, [63] consider the setting where there exists some f in the model that 700 f ̸ = f
o:o:d: , where f o:o:d: is an estimator with high prediction performance on both training and test domain,and that f ( x ) =
f
o:o:d: ( x ) for sample x from training domains.
Under the setting, they deduce a upper boundof a risk on a test domain, and then, propose a method for decreasing the upper bound.
H Additional ExperimentH.1 Additional Experiments of Colored MNIST in Section 6
Although the Colored MNIST experiment in Section 6 ﬁxes its ﬂip rate to 25% , we additionally demonstrate 706 by changing its ﬂip rate among { 10% ; 15% ; 20% ; 25% } .
[Table]
[Table] [Table]
Table 4 and 5 show that, among several CV methods, our method II keeps a high predictive performance 708 regardless of ﬂipping rates.
Table 6 the difference between accuracies by TDV and each CV for the same dataset with e = 0 : 9 .
The result veriﬁes that CVII selects preferable hyperparameters with smaller errors.
H.2 Additional Experiments: Colored MNIST II
We conduct an additional Colored MNIST experiment, changing annotation and coloring rules from ones in 712 Section 6.
Setting Y =
[3] and Z :=
[2] , we aim to predict Y e from digit image data X e , which is in the threecategories 0 − 2 ( Y e = 0 ), 3 or 4 ( Y e = 1 ) and 5 − 9 ( Y e = 2 ).
The label is changed randomly to one of therest with a some probability ranging from { 10% ; 15% ; 20% ; 25% } .
The domain index e ∈ [0 : 0 ; 1 : 0] controlsthe color of the digit; for Y e = 0 ; 1 , the digit is colored in red with probability e and for Y e = 2 coloredin green with probability e .
In the experiment, D e ∗ ∼ P X 0 : 1 ;Y 0 : 1 is drawn with sample size n e ∗ = 5000 ,and Y e is predicted based on X e for e = 0 : 1 and 0 : 9 .
Regarding Z e , we consider the task where we predict
Z e = 0 for X e in 0 − 2 and Z e = 1 for 3 − 9 (that is, g (0) =
and g (1) =
g (2) = 1 ).
We obtain the ﬁnallabel by ﬂipping with some probability.
As the domain-speciﬁc factor, we color the digit red for Z e = 0 withprobability e and green for Z e = 1 with probability e .
We set E ad = { 0 : 1 ; 0 : 3 ; 0 : 5 ; 0 : 7 ; 0 : 9 } with n e = 5000for ∀ e ∈ E ad .
We model Φ by a 3 -layer neural net.
With the maximum epoch 500 , we select ( t; (cid:21) after ) from 722 3 × 10 candidates with t ∈ { 0 ; 100 ; 200 } ; (cid:21) after ∈ { 10 0 ; 10 1 ; :::; 10 9 } by each CV method.
Table 7 and 8 shows test accuracies for 2000 random samples in the domains e = 0 : 1 and e = 0 : 9 .
Theresults demonstrate that the proposed methods signiﬁcantly outperform the others for e = 0 : 9 .
Among the
[Table] [Table]
two proposed methods, CV II yields the higher test accuracy.
Table 9 shows the difference between accuracies 726 by TDV and each CV for the same data set with e = 0 : 9 .
The results verify that CVII selects preferablehyperparameters with smaller errors.
H.3 Additional Experiment: Synthesized Data
We compared the proposed method with the other approaches using synthesized data with X = R 2 , Y =
[3]and Z :=
We used distributions N 0 :
= N (0 ; 10 2 ) × N ( e; 10 2 ) , N 1 := N (30 ; 10 2 ) × N ( − 4 e; 10 2 ) and
N 2 := N ( − 30 ; 10 2 ) × N ( − e; 10 2 ) , where N ( a; b ) denotes a normal distribution with its (mean, variance)= ( a; b ) .
Given x ∼ N i , the task is to predict N i among i = 0 ; 1 ; 2 .
The aim of IL is to ignore the second 733 component of x , as it works as a domain-speciﬁc factor.
Given e ∗ ∈ N ≥
ranging from 0 to 50 , each experimentdraws D e ∗ ∼ P X e ∗ ;Y e ∗ with its sample size n e ∗ = 2000 , and then predicts Y − e ∗ from X − e ∗ .
Setting g byg (0)
=
and g (1) =
g (2) = 1 , we draw D ead ∼ P X e ;Z e from E ad
= {− 100 ; − 50 ; 0 ; 50 ; 100 } with itssample size n e = 1000 ( ∀ e ∈ E ad ).
We model Φ by a 3 -layer neural net.
Setting the maximum epoch 500 , weselect ( t; (cid:21) after ) from 3 × 5 candidates with t ∈ { 0 ; 100 ; 200 } and (cid:21) after ∈ { 10 0 ; 10 1 ; :::; 10 4 } by each ofthe CV methods.
Table 10 shows the test accuracy of the estimates for e = − e ∗ over 2000 random samples ( x; y ) ∼
P X − e ∗ ;Y − e ∗ .
When e ∗ = 0 and 5 , the domain bias of training ( e ∗ ) are similar to the one of test ( − e ∗ ) ,and hence, the ﬁne-tuning methods yield high performances, which may use biased correlation.
As e ∗ increases, the difference between the training ( e ∗ ) and test ( − e ∗ ) distributions becomes larger, and the previousmethods fail to achieve high accuracy.
The proposed methods (Ours) keep higher performance than the others 744 even for large e ∗ .
Among the CV methods, our two methods (CVI, CVII) signiﬁcantly outperform the othersfor larger e ∗ .
For this data set, CVI and CVII show almost the same performance.
H.4 Additional Experiment: Bird recognition
Our method is applied to the Bird recognition problem [28], which aims to predict three labels Y e of images X e : waterbird ( Y e = 0), landbird ( Y e = 1) and no bird ( Y e = 2).
The dataset is made by combiningbackground images from the Places dataset [39] and bird images from the CUB dataset [38] in two different 750 ways E :=
{ e 1 ; e 2 } .
In domain e 1 , we prepare three types of image: landbird image with land background,waterbird image with water background, and no bird with land background (Figure 3, left).
In domain e 2 , 752 we have landbird images with water background, waterbird images with land background, and no bird withwater background (Figure 3, right).
For the sample of the target task, we used the domain e ∗ =
e 1 and generated n e ∗ = 8649 data D e ∗ ∼ P
X
e 1 ;Y e 1 .
The sample in the higher level D ead of ( X e ; Z e ) , whose label is
[Table] [Table]
landbird ( Z e = 0 ) and no landbird ( Z e = 1 ) ( i:e: , g (1) = 0 and g (0) = g (2) = 1 ), is drawn from bothe 1 and e 2 with n e 1 = n e 2 = 8649 .
Here, we use D e ∗ as D e 1 ad with labels of D e ∗ re-annotated by g .
Wemade a predictor of Y e based on X e , and evaluated the test accuracy in the two domains e = e 1 ; e 2 .
Wemodel Φ by ResNet50 [29].
Setting the maximum epoch 5 , we select ( t; (cid:21) after ) from 5 × 5 candidates with 759 t ∈ [5] ; (cid:21) after ∈ { 10 0 ; 10 1 ; :::; 10 4 } by each CV method.
[Figure] [Figure]
Table 11 shows test accuracies with 2162 random samples for e 1 and e 2 .
We can see that the proposed frame- 761 work together with CV methods succeeded in capturing the predictor invariant to the change of background,while the other methods failed.
ERM and FT show much higher accuracy for e 1 than Oracle and worst resultsfor e 2 , which implies that these methods learn biased correlation in D e ∗ .
[Table]
H.5 Additional Experiment: ImageNet
In the main body, only test accuracies on e 2 are shown.
The result adding test accuracies on the training domain 767 e 1 are as follows:
ImageNet: Y =
[3] ; Z = [2] .
[Table]
ImageNet: Y =
[7] ; Z = [2] .
[Table]
: Y = [3] ; Z = [2] .
[Table]
I Experimental Details
I.1 Visualizations of Experiment Results
Synthesized data in Appendix H.3 is visualized as in Figure 4.
Synthesized data in the CVs comparison exper- 771 iment (Section 6) is visualized as in Figure 5.
[Figure]
I.2 Explicit representation of Synthetic data used in CV comparison experiment
N 1 = N ( − 180 ; 20 2 ) × N ( − 5 e; 30 2 ) ;
N 2 = N ( − 100 ; 20 2 ) × N ( − 3 e; 30 2 ) ;
N 3 = N ( − 20 ; 20 2 ) × N ( − 1 e; 30 2 ) ;
N 4 = N (60 ; 20 2 ) × N ( − 2 e; 30 2 ) ;
N 5 = N (140 ; 20 2 ) × N ( − 4 e; 30 2 ) ;
N 6 = N ( − 140 ; 20 2 ) × N (4 e; 30 2 ) ;
N 7 = N ( − 60 ; 20 2 ) × N (2 e; 30 2 ) ;
N
8 = N (20 ; 20 2 ) × N (1 e; 30 2 ) ;
N 9 = N (100 ; 20 2 ) × N (3 e; 30 2 ) ;
N 10 = N (180 ; 20 2 ) × N (5 e; 30 2 ) :
I.3 Detail of ImageNet experiment data set
In the ImageNet experiment in Section 6, Y is set as follows:
• Y =
[3] : { bird , turtle ,snake }
• Y =
[7] : { bird , turtle , snake, cat, food, vehicle , building } ,
• Y =
[17] : { bird , turtle , snake, cat, dog , monkey, spider, butterﬂy, food , vehicle , building , shoes ,
hat, instrument, tellephone, bottle , chair } .
Images of bolded labels are composed of different species among e 1 and e 2 .
Explicitly, dataset are composed 781 as follows:
Y =
[Table]
Y =
[Table]
I.4 model architectures and optimization procedures
Through the experiment in the present paper, all models of competitors are composed of neural networks where 784 its loss function, activation function, and optimizer are cross entropy, Relu Networks and Adam [37].
In thefollowing explanation, NN with its model architecture a → h 1 → · · · h k → h n → P [ m ] means that itsinput and hidden dimensions are a and ( h 1 ; :::; h n ) respectively, and its output is probability density functions 787 on [ m ] .
NN with its model architecture a → h 1 → · · · h k → h n → b means that its input, hidden andoutput dimensions are a , ( h 1 ; :::; h n ) and b respectively.
All the experiment, we add L 2 -reguralized term toour objective function.
We add explanations of previous CV methods.
Tr-CV implements cross-validation with using only D ∗ .
In
LOD-CV, a model is learnt with excluding one of the D e ⊂ D ad from D ad , and evaluate its performance by 792 D e .
Changing the role of e ∈ E ad , and taking their mean, we evaluate ﬁnal CV-value.
I.4. Colored MNIST
We set model architecture of Φ used in our method 2 → 440 → 440 → 440 .
We set model architecture ofand ERM 2 → 440 → 440 → P [3] .
When we use FT and FF, its model architecture on pre-train phase andretraining phase are 2 → 440 → 440 → P [2] and 2 → 440 → 440 → P [3] respectively.
We set running rateand hyperparameter of L 2 -regularized term 0 : 0004 and 0 : 002 respectively.
When we use DSAN , we inheritlearning condition in the Amazon Review dataset experiment.
When training, we use batch learning.
We set 799 K = 10 of our CV method.
I.4.2 ImageNet
We set model architecture of Φ used in our method ResNet50 [29] with changing its output dimension 256 .
We set model architecture of and ERM ResNet50 [29] with changing its output P [3] .
When we use FT and
FF, its model architecture on pre-train phase and retraining phase are ResNet50 [29] with changing its output 804 dimension 2 and 3 respectively.
We set running rate and hyperparameter of L 2 -regularized term 0 : 00004 and0 : 001 respectively.
When training, we use minibatch learning with a minibatch size 56 .
We set K = 5 of each 806 CV method.
I.4.3 CV comparison experiment
We set model architecture of Φ used in our method 2 → 8 → 8 → 1 .
We set running rate and hyperparametersof L 2 -regularized term 0 : 05 and 0 : 001 respectively.
When training, we use minibatch learning with dividing
D ∗ , D e ad and D 40 into 50 equal parts respectively.
We set K = 10 of each CV method.
I
4.4 Appendix: Colored MNIST II
We set model architecture of Φ used in our method 2 → 440 → 440 → 440 .
We set model architectureof ERM 2 → 440 → 440 → P [3] .
When we use FT and FF, its model architecture on pre-train phase andretraining phase are 2 → 440 → 440 → P [2] and 2 → 440 → 440 → P [3] respectively.
We set running rateand hyperparameter of L 2 -regularized term 0 : 0004 and 0 : 002 respectively.
When we use DSAN , we inheritlearning condition in the Amazon Review dataset experiment.
When training, we use batch learning.
We set 817 K = 10 of our CV method.
I.4.5 Appendix: Synthesized Data
We set model architecture of Φ used in our method 2 → 20 → 20 → 1 .
We set model architecture of ERM 820 2 → 20 → 20 → P [3] .
When we use FT and FF, its model architecture on pre-train phase and retraining phase 821 are 2 → 20 → 20 → P [2] and 2 → 20 → 20 → P [3] respectively.
We set running rate and hyperparametersof L 2 -regularized term 0 : 0115 and 0 : 01 respectively.
When we use DSAN [31], we inherit learning conditionin the Amazon Review dataset experiment.
When training, we use batch learning.
We set K = 10 of each CV 824 method.
I.4.6 Appendix: Birds recognition
We set model architecture of Φ used in our method ResNet50 [29] with changing its output dimension 256 .
We 827 set model architecture of ERM ResNet50 [29] with changing its output P [3] .
When we use FT and FF, its modelarchitecture on pre-train phase and retraining phase are ResNet50 [29] with changing its output dimension 2 and 829 3 respectively.
We set running rate and hyperparameter of L 2 -regularized term 0 : 00004 and 0 : 001 respectively.
When training, we use minibatch learning with a minibatch size 56 .
We set K = 5 of each CV method.