Invariance Learning based on Label Hierarchy
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Deep Neural Networks inherit biased correlations embedded in training data andhence may fail to predict desired labels on unseen domains (or environments),which have different distributions from the domain to provide training data.
Invariance Learning (IL) has been developed recently to overcome this shortcoming;using training data in many domains, IL estimates such a predictor that is invariant to a change of domain.
However, the requirement of training data in multipledomains is a strong restriction of using IL, since it demands expensive annotation.
We propose a novel IL framework to overcome this problem.
Assuming the availability of data from multiple domains for a classiﬁcation task at a higher level, forwhich the labeling cost is lower, we estimate an invariant predictor for the targetclassiﬁcation task with training data gathered in a single domain.
Additionally, wepropose two cross-validation methods for selecting hyperparameters of invarianceregularization, which has not been addressed properly in existing IL methods.
The effectiveness of the proposed framework, including the cross-validation, isdemonstrated empirically.
Theoretical analysis reveals that our framework canestimate the desirable invariant predictor with a hyperparameter ﬁxed correctly,and that such a preferable hyperparameter is chosen by the proposed CV methodsunder some conditions.
Introduction
Training data used in machine learning may contain features that are spuriously correlated to the labels of data.
Deep Neural Networks (DNNs) often learn such biased correlations embedded in training data and hence may fail to predict desired labels of test data generated by a different distributionfrom one to provide training data.
In classiﬁcation of animal images, DNNs tend to misclassify cowson sandy beaches, since most training pictures are taken in green pastures and DNNs inherit contextinformation in training [3, 9].
Another example is learning from medical data.
Systems trained withdata collected in one hospital do not generalize well to other hospitals; DNNs unintentionally extractenvironmental factors speciﬁc to a particular hospital in training [16–18].
Invariance Learning (IL) is a rapidly developed approach to overcome the issue of biased correlation,which is caused by some bias in the distribution of a training dataset [10–15, 21, 33–36, 54].
In thispaper, we use the term domain to specify the bias in the distribution of a dataset.
IL is thus amethod for removing the inﬂuence of domain shits.
Using training data from multiple domains, ILestimates a predictor invariant to the change of domains, aiming at keeping good performance inunseen domains as well as in the training domains.
While the IL approach has attracted much attention, requiring training data from multiple domainsmay hinder wide applications in practice; preparing training data in many domains often involvesexpensive data annotation.
In real-world data, labels may be missing [42–45, 56–58] or incomplete[46–48].
Such data with incomplete annotation are not directly applicable to the standard IL meth- ods; they must be re-annotated accurately, often at great ﬁnancial or human expense.
The high costdrives a strong need to establish a new IL framework without or with lower annotation costs.
[Figure]
To mitigate the problem of annotation cost,we propose a novel IL framework for the situation where the training data of target classiﬁcation is given in only one domain, whilethe task of higher label hierarchy , whichneeds lower annotation cost, has data frommultiple domains.
Here, the task of higherlabel hierarchy means a classiﬁcation taskwith coarser labels than those of the target classiﬁcation.
Figure 1 shows an example of label hierarchy.
Consider the case where a target classiﬁcation has 300 labels{ bird 1 , .. bird 100 , snake 1 , ..., snake 100 , turtle 1 , ..., turtle 100 } corresponding to 300 species.
Then, thebinary labels { bird , reptile } are an example of labels in the higher hierarchy.
The decrease in thenumber of classes reduces annotation time per image.
Moreover, annotation of the binary labelsdoes not require any expert knowledge, while annotation of the original 300 labels would requiresexpert knowledge about birds, snakes, and turtles.
Hence, the new IL framework signiﬁcantly reduces the annotation cost in comparison with previous IL methods; we need exhausting annotationof 300 classes only for one domain and just binary labels for other domains.
Another important issue in IL is hyperparameter selection.
Most IL methods involve some hyperparameters to balance the classiﬁcation accuracy and the degree of invariance.
As [21] and [19]point out, in the literature of IL, the best performances of invariance had often been achieved byselecting the hyperparameters using test data from unseen domains.
Moreover, [19] numericallydemonstrated that, without using test data, simple methods of hyperparameter selection fail to ﬁnd apreferable hyperparameter.
It demonstrates a strong need for establishing an appropriate method ofhyperparameter selection for IL.
We propose two methods of cross-validation (CV) for hyperparameter selection in our new IL framework.
Since we assume training data of a single domain for the target task, it is impossible to estimate the deviation of the risks over the domains.
Our CV methods mitigate the difﬁculty by usingadditional data from multiple domains in the higher label hierarchy.
Theoretical analysis proves thatour methods select a hyperparameter correctly under some conditions.
The main contributions of this paper are as follows:
• We establish a novel framework of invariance learning, which estimates an invariant predictorfrom a single domain data, assuming additional data from multiple domains for a classiﬁcationtask at a higher level. • Under the framework, we propose two methods of cross-validation for selecting hyperparameterswithout accessing any samples from unseen target domains.
• Experimental studies verify that the proposed framework extracts an invariant predictor moreeffectively than other existing methods.•
We mathematically prove that our framework can estimate a correct invariant predictor with ahyperparameter ﬁxed correctly and that such a preferable hyperparameter is selected by the proposed CV methods under some settings.
Invariance Learning based on Label Hierarchy
Notations Throughout this paper, the space of input features and ﬁnite class labels are denotedby X and Y , respectively.
For given predictor f : X → Y and random variable ( X, Y ) on
X × Y with its probability P X,Y , R ( X,Y ) ( f ) denotes the risk of f on ( X, Y ) ; i.e. , R ( X,Y ) ( f ) :
=!
l ( f ( x ) , y ) dP X,Y , where l : Y × Y → loss m } ﬁnite set
A
| A | ∈ N number of elements in A
2.1 Review of Invariance Learning
Following [10], to formulate the out-of-distribution (o.o.d.) generalization, we assume that the jointdistribution of data ( X e , Y e ) depends on the domain (or environment)
e ∈ E , and consider the dependence of a predictor f on the domain variable e .
Suppose we are given training datasets
D
[Equation] X [Equation] Y e i.i.d.
from multiple domains E tr ⊂ E .
The ﬁnal goal of the 91 o.o.d.
problem is to predict a desired label Y e ∈ Y from X e ∈ X for larger target domains E ⊃ E tr . {+
+}
To discuss the o.o.d.
performance, [10] introduced the o.o.d.
risk
R
[Equation]
[Equation]
[Equation]
(1) where R e ( f ) :
= R ( X e ,Y e ) ( f ) .
This is the worst case risk over E , including unseen domains E \E tr .
To solve (1), [10] estimates a predictor f that performs well in unseen domains E \ E tr as well astraining domains E tr , namely a predictor invariant to a change of domains.
The invariant predictorf is composed of two maps Φ and w ; that is, f = w ◦ Φ holds.
A feature map Φ : X → H , which isoften called an invariance , realizes a feature of x ∈ X in the feature space H with biased correlations 98 in x removed.
A predictor w :
H → Y estimates the label of feature Φ ( x ) .
The estimation of aninvariant predictor is implemented by solving the following optimization problem:
[Equation]
where I tr is the set of invariances captured by
[Equation]
any
[Equation]
[Equation] This invariance is based on conditional independence as [14, 15, 40], while [10, 11] use a differenttype invariance based on argmin w R e ( w ◦ Φ ) instead of P ( Y e | Φ ( X e )) .
All IL methods estimate theinvariance using the difference among E tr , assuming the availability of multiple training domains.
2.2 Invariance estimation by higher label data
[Figure] Our goal is to make an invariant predictor from a single training domain E tr = { e ∗ } .
In this case,(2) is reduced to the empirical risk minimization
min
f R e ∗ ( f ) on e ∗ , and therefore the standard ILframework is not able to extract an invariance.
In this paper, we introduce an assumption that additional data D ead for an another task ( X e , Z e ) ,which is in the higher label hierarchy than ( X e , Y e ) , is available with respect to multiple domains
E ad ⊂ E .
The task of higher hierarchy ( X e , Z e ) has a coarser label
Z e ∈
Z than Y e ∈ Y ; more 110 formally, Z e is represented as Z e = g ( Y e ) with a surjective label mapping g :
Y → Z fromthe lower to the higher level in the label hierarchy.
The example in Section 1 is formalized bya surjevtive function g as, setting Y :
= { bird 1 , .., bird 100 , turtle 1 , .., turtle 100 , snake 1 , .., snake 100 }and Z := { bird , reptile } , g ( y ) :
= bird if y = bird
i ( i ∈ { 1 , 2 , ..., 100 } ) and g ( y ) :
= reptile else.
By making use of {D ead } e ∈ E ad , our objective for the invariant prediction is given by
[Equation]
(3) where I ad is the set of invariances:
[Equation]
[Equation]
for any
[Equation]
Note that (3) evaluates the risk with a single training domain while the invariances are given byadditional data of multiple domains.
2.3 Construction of objective function
Among several candidates of the loss and model design, we focus a probabilistic output case and evaluate its error by the cross entropy loss; that is, we model w by p θ : H → P Y , where P Y denotes the set of probabilities on
Y and θ denotes a model parameter.
The risk is then written by
[Equation]
We aim to solve (3) by minimizing the following objective function:
[Equation]
Here, ˆ R e ∗ ( p θ ◦ Φ ) denotes the empirical risk of p θ ◦ Φ on the training domain E tr = { e ∗ } evaluatedby D e ∗ : ˆ R e ∗ ( p θ ◦ Φ ) :
= − 1 |D e ∗ | ’ ( x e ∗ ,y e ∗ ) ∈
D e ∗ log p θ ( y e ∗ | Φ ( x e ∗ )) .
While we can consider somevariations of invariance regularization, we adopt the one used in [10] and construct an objectivefunction as
[Equation]
Here, p Z|H θ : H → P Z is the linear logistic regression model same as [10], Φ is a nonlinear neuralnetwork, and ˆ R ( X e ,Z e ) ( p Z|H θ ad ◦ Φ ) : = −
[Equation]
|D
[Equation]
It is not obvious if the regularization term in (5) is valid as a dependence measure of
P ( g ( Y e ) | Φ ( X e )) , since it was proposed for another type of invariance based on argmin w R e ( w ◦Φ ) .
The next lemma shows that these notions of invariance are the same in the current setting.
Lemma 1.
When modeling w by conditional probabilities, the following statements are equivalent:
P ( Z e | Φ ( X e )) does not depend on e ⇔ argmin p Z|H θ ad R ( X e ,Z e ) ( p Z|H θ ad ◦ Φ ) does not depend on e ,where model p Z|H θ ad in the right hand side runs over all probability densities.
While our objective function (5) is similar to the ones in [10, 21] in that they are composed of anempirical risk and an invariance regularization, the correctness has not been fully discussed so far.
In Section 4, we will mathematically prove the correctness of (5) under some settings.
Hyperparameter selection method
3.1 Hyperparameter Selection in Invarance Learning
The objective function (5) has a hyperparameter λ to select, as is often the case with IL methods.
Thehyperparameter selection in IL has special difﬁculty, however; because the o.o.d.
problem needs topredict Y e on unseen domains, λ must be chosen without accessing any data in such unseen domains.
It was reported in [19, 21] that the success of IL methods depends strongly on the careful choice ofhyperparameters, and some of the results even used data from unseen domains in the choice.
[19]reported also experimental results of various IL methods with two CV methods,
training-domainvalidation (Tr-CV) and leave-one-domain-out validation (LOD-CV), and showed that the CV methods failed to select preferable hyperparameters.
In the Colored MNIST experiment, for example, theaccuracy of Invariant Risk Minimization [10] is 52 .
0% at best, which is about a random guess level.
The failure of the CV methods is caused by the improper design of the objective function for CV;they do not simulate the o.o.d.
risk, which is the maximum risk over the domains.
Tr-CV splitsdata in each training domain into training and validation subsets, and takes the sum of the validatedrisks over the training domains.
Obviously, this is not an estimate of the o.o.d.
risk.
LOD-CVholds out one domain among the training domains in turn and validates models with the averageof the validated risks over the held-out domains.
Again, this average does not correspond to theo.o.d.
risk.
In summary, the problem we need to solve is answering the following question: how canwe construct an evaluation function of the o.o.d.
risk from validation data?
In the sequel, we willpropose two methods of CV, which are summarized in Algorithm 1.
3.2 Method I: using data of higher level task
We divide each of D e ∗ , D e 1 ad , ..., D e n ad into K parts where |E ad | = n , and use the k -th sample{D e ∗ [ k ] , D e 1 ad, [ k ] , ..., D e n ad, [ k ] } and the rest {D e ∗ [ − k ] , D e 1 ad, [ − k ] , ..., D e n ad, [ − k ] } for validation and training,respectively.
To approximate the o.o.d.
risk of the trained predictor p θ λ [ − k ] ◦
Φ λ [ − k ] , we wish to estimate R e ( p θ λ [ − k ] ◦ Φ λ [ − k ] ) for e ∈ E ad ∪ { e ∗ } by the validation set.
For e ∗ , we use the standardempirical estimate ˆ R e ∗ [ k ] ( p θ λ [ − k ] ◦ Φ λ [ − k ] ) .
For e ∈ E ad , we substitute unavailable Y e with Z e and useˆ R ( X e ,Z e ) [ k ] ( p θ λ [ − k ] ◦ Φ λ [ − k ] ) :
= |D ead, [ k ] | ’ ( x e ,z e ) ∈ D ead, [ k ] − log p θ λ [ − k ] ( z e | Φ λ [ − k ] ( x e )) .
Require: : Split D e ∗ , D e 1 ad , ..., D e n ad into K parts.
Set the hyperparameter candidates Λ .
Require: : ˆ P e ( z !!
# → ) ← |D ead,z !!
# → | |D ead | , where D ead,z !!
# → := { ( x, z ) ∈ D ead | z
= z !!
# → } for all e ∈ E ad and z !!
# → ∈ Z !!
# → .
1: for λ ∈ Λ do 2: for k = 1 to K do 3:
Learn θ λ [ − k ] , Φ λ [ − k ] by using D e ∗ [ − k ] , D e 1 ad, [ − k ] , ..., D e n ad, [ − k ] .
4: ˆ R e ∗ k ( λ ) ← |D e ∗ [ k ] | ’ ( x e ∗ ,y e ∗ ) ∈
D e ∗ [ k ] − log p θ λ [ − k ] ( y e ∗ | Φ λ [ − k ] ( x e ∗ ))
//Risk estimation on e ∗ .
5: ˆ R e ∗ | z !!
# → k ( λ ) ← |D e ∗ [ k ] ,z !!
# → | ’ ( x,y ) ∈
D e ∗ [ k ] ,z !!
# → − log p θ λ [ − k ] ( y | Φ λ [ − k ] ( x ) , g ( Y ) =
z !!
# → ) for z !!
# → in Z !!
# → .
6: for e ∈ E ad do 7: ˆ R ek ( λ ) ← |D ead, [ k ] | ’ ( x e ,z e ) ∈ D ead, [ k ] − log p θ λ [ − k ]
( z e | Φ λ [ − k ] ( x e )) .
Risk estimation on e .
8: if CORRECTION then 9:
[Equation] [Equation]
z
[Equation]
) · ˆ R e
[Equation]
( λ ) //
Correction term addition.
10: end if 11: end for 12: ˆ R o.o.d.k ( λ ) ← max e ∈ E ad ∪ { e ∗ } ˆ R ek ( λ ) // o.o.d.
risk estimation.
13: end for 14: ˆ R o.o.d.
( λ ) ← 1 K ’
K k =
1 ˆ R o.o.d.k ( λ ) //
Final o.o.d.
risk estimation.
15: end for 16:
Select λ ∗ :=
argmin
λ ∈ Λ ˆ R o.o.d.
( λ )
3.3 Method II: using correction term
Method I can be improved by correcting the replacement R e
= R
( X e ,Y e ) with R ( X e ,Z e ) for e ∈
E ad .
We use the following theorem for the correction:
Theorem 2.
Let Z !!
# → := $ z ∈ Z (( | g − 1 ( z ) | >% .
For any map Φ : X → H , p θ : H → P Y , and 166 random variable ( X, Y ) on
X × Y , the following equality holds:
[Equation]
Here, R ( X,Y ) | z !!
# → ( p θ ◦ Φ ) := !
−
[Equation]
p
[Equation]
z !!
# →
[Equation]
dP ( X,Y ) | g
( Y )=
z !!
# → where 168 P ( X,Y ) | g ( Y )=
z !!
# → denotes the conditional distribution of ( X, Y ) given the event g ( Y )
=
z !!
# → , andp θ ( y | Φ ( x ) , g [Equation] Y
[Equation]
= z !!
[Equation]
) :=
[Equation] [Equation]
The proof is given in Appendix A.
The theorem shows that, to estimate the correction term, we needto estimate (i) P ( g ( Y e )
=
z !!
# → ) and (ii) R
( X e ,Y e ) | z !!
# → ( p θ λ [ − k ] ◦ Φ λ [ − k ] ) for every z !!
# → ∈ Z !!
# → .(i) is naturally estimated even on e ∈ E ad :
[Equation]
|
[Equation]
ead | , where D ead,z !!
# → := { ( x, z ) ∈ D ead | z
= z !!
# → } .
(ii) is not easily estimable; while a direct simulation of the integration !
dP ( X e ,Y e ) | g
( Y e )=
z !!
# → demands data from ( X e , Y e ) ∼ P X e ,Y e , our available data D ead on e ∈ E ad is from P
X e ,g ( Y e ) , not from P X e ,Y e .
To solve the non-availability of data from P X e ,Y e , we use the training data D e ∗ ∼ P X e ∗ ,Y e ∗ instead.
Namely, (ii) is estimated by
[Equation]
where D e ∗ [ k ] ,z !!
# → :=
[Equation]
∗ [ k ] | g ( y )
=
z !!
# → * ⊂ D e ∗ [ k ] .
In Algorithm 1, the above risk estimate isabbreviated by ˆ R e ∗ | z !!
# → [ k ] ( λ ) for notation simplicity.
Theoretical analysis
Throughout this section, to avoid discussing the non-trivial effects of nonlinear Φ , we focus on thesimpliﬁed case of variable selections, where the feature map Φ is chosen from the projections of x to a subset of its components.
For example, Φ may be Φ ( x 1 , x 2 , x 3 ) =
( x 1 , x 3 ) when x is threedimensional.
This type of IL appears practically in causal inference [14, 13] and regression [40].
Let X :=
X 1 × X 2 where X 1 := R n 1 and X 2 := R n 2 with n 1 , n 2 ∈ N .
For a projection Φ , let 180 Φ i denote the X
i -component of Φ ( i = 1 , 2 ).
If Φ has a X 2 -component, we write Im Φ 2 .
= ∅ .
Forsimplicity of analysis, the domain set E is deﬁned by all the probability distributions with the ﬁxedmarginal distribution P X I 1 ,Y I of ( X 1 , Y ) ; namely,
[Equation]
In this case, for any e ∈ E the variable ( X e , Y e ) satisﬁes (i)
P Y e |
Φ X 1 ( X e ) equals to P Y I
|
X I 1 , and 184 (ii) the marginal distribution P Φ X 1 ( X ) of the invariant feature Φ X 1 ( X ) equals to P X I 1 .
The abovesetting and deﬁnition persist through Section 4.
4.1 Theoretical analysis of our objective function
The following theorem ensures that, neglecting estimations and under some conditions, a minimumof our objective function (5) with careful hyperparameter choice also minimizes the o.o.d.
risk (1):
Theorem 3 (o.o.d. optimality of our objective function) .
Under the setting ( ※ ), additionally assumethat the following condition holds:
(A) For any variable selection Φ with Im Φ 2 .
= ∅ , there exist two domains { e 1 , e 2 } ⊂
E ad such that
[Equation]
Then, there exists λ ∗ ∈ R such that any minimizer ( θ † , θ † ad , Φ † ) of (5),
[Equation]
is o.o.d.
optimal, i.e.,
[Equation]
where models p θ and p Z|H θ ad in min θ , θ ad , Φ run all the probability density functions, and Φ runs allthe variable selections.
The gradient ∇ θ ad should be understood as the functional derivative on thespace of probability density functions.
For the proof, see Appendix B. Condition (A) means that E ad has sufﬁcient variation to capture thedesirable invariance Φ X 1 .
4.2 Theoretical analysis of our cross validation methods
In Sections 3.2 and 3.3, we approximate R ( X e ,Y e ) using label Z e of higher level.
While the approximation is not exact, we will prove that the proposed CV methods still select a correct hyperparameterunder some conditions.
We will also elucidate the difference of the two CV methods.
Given hyperparameter λ , minimizing (5) over the model yields the feature map (variable selection) denoted byΦ λ : X → R n λ ( n λ ≤ n 1 + n 2 ).
For simplicity of theoretical analysis, we assume that the minimization of (5) achieves perfectly the conditional probability density function of P Y e ∗ | Φ λ ( X e ∗ ) , denotedby p ∗ , λ ( y | Φ λ ( x )) .
Then, neglecting estimation errors, the approximated o.o.d.
risk of p ∗ , λ ◦ Φ λ usedin Methods I and II are represented by the following R
I ( λ ) and R II ( λ ) , respectively:
[Equation]
We have the following theoretical justiﬁcation of our CV methods: the chosen λ gives a minimizerof the correct CV criterion.
For the proofs, see Appendices C and D.
Theorem 4 (Correctness of Method I) .
Assume that the following conditions (i) and (ii) hold:(i)
Among a set Λ of hyperparameter candidates, there exists λ I ∈ Λ such that Φ λ I =
Φ X 1 .
(ii) Let p e ∗ be the probability density function of P X e ∗ ,g ( Y e ∗ ) .
Then, for any λ with Im Φ λ 2 .
= ∅ ,there is e λ ∈ E ad such that
[Equation]
probability 1.
Here, ε ∈ R > 0 is some sufﬁcient small positive real number (that is, 0 < ε 1 1 ) and β :=
H ( Y e ∗ | Φ X 1 ( X e ∗ )) is the conditional entropy of ( Φ X 1 ( X e ∗ ) , Y e ∗ ) .
Then, under the setting ( ※ ), we have
[Equation]
Theorem 5 (Correctness of Method II) .
Assume that, in addition to (i) in Theorem 4, the followingcondition (iii) holds:(ii)’ for any λ with Im Φ λ 2 .
= ∅ , there is e λ ∈ E ad such that
[Equation]
with probability 1.
Here, ε is some sufﬁciently small positive real number and
[Equation]
Then, under the setting ( ※ ), we have
[Equation]
The conditions (ii) and (ii)’ impose that, for at least one e λ ∈ E ad , the two domains e λ and e ∗ aredifferent in the following meaning. if λ fails to remove domain-speciﬁc factors ( i.e. , Im Φ λ 2 .
= ∅ ),for some e λ ∈ E ad , ( x, z ) ∼ P X
e λ ,g ( Y e λ ) yields low p e ∗ ( z | Φ λ ( x )) with high probability.
On theother hand, ( x, z ) ∼ P X e ∗ ,g ( Y e ∗ ) yields high p e ∗ (
z | Φ λ ( x )) with high probability: that is, e ∗ and e λare different .
The theoretical analysis shows, while Method I is simpler to implement than Method II, Method IIis more applicable.
Noting that β ≥ β λ and hence, e − β − ε ≤ e − β λ − ε , the condition (ii)’ is milderthan (ii).
Recalling that (ii) and (ii)’ impose the discrepancy between E ad and e ∗ as discussed inthe last paragraph, relaxation of conditions from (ii) to (ii)’
implies that method II can be appliedeven when domains E ad and e ∗ have smaller discrepancy than the condition for Method I. Thedifference of these two methods will be demonstrated in Section 6.
The real-world feasibility of theassumptions (ii) and (ii)’ are discussed in Appendix F.
Related work
Fine-tuning The proposed framework uses additional data from multiple domains as well as thetraining data for the target task.
It is relevant to Transfer learning (TL) [22–24] and meta-learning[30, 41], which realize fast and accurate learning for a new target task based on a model pre-trainedwith additional data sets.
For example, after initial learning with a large data set, ﬁne tune [22, 23]re-trains the model with the target task, while frozen feature [24] ﬁxes the pre-trained model andtunes a head network.
Although they show advantages in many learning problems, they may notwork effectively in the current setting; in the ﬁne-tuning with the target task ( X e ∗ , Y e ∗ ) , the modeltends to learn biased correlation in the data set and does not generalize to unseen domains.
Someﬁne-tuning methods will be compared with the proposed approach in Section 6.
Domain adaptation by deep feature learning
Domain adaptation strategies by deep featurelearning [25–27, 31, 32] assume that we can access input data on a test domain in advance, andtry to obtain data representation
Φ ( X e ) that follows the same distribution for the training and testdomains.
While the strategies lead to high predictive performance on a test domain similar to atraining domain, such Φ does not function by discarding environmental factors from X e ∈ X asnoted in [10].
Experimental comparisons will be shown in Section 6.
Distributionally robust optimizations and other de-bias methods are explained in Appendix G.
Experiments
We study the effectiveness of the proposed framework and CVs through experiments, comparingthem with several existing methods: empirical risk minimization (ERM), ﬁne-tuning methods, and
[Table]
TDV selects λ that yields the highest performance on e 2 .
The best scores are bolded .
[Table]
deep domain adaptation strategies.
For ﬁne-tuning, we employ two typical methods of transferlearning: ﬁne tune (FT) and frozen feature (FF) [22–24].
As a deep domain adaptation technique, weadopt the state-of-the-art method DSAN [31].
We also compare our two CVs (CVI and CVII) withconventional CVs: training-domain validation (Tr-CV) and leave-one-domain-out cross-validation(LOD-CV) [19].
We have two hyperparameters to be selected by CV.
In the training with (5), we setλ :=
λ before when the training epoch is less than a certain threshold t , and λ :=
λ after if the epochis larger than t .
It is known that these two hyperparameters are critical for IL methods to achievegood results.
From a set of candidates, each of the CV methods selects a pair ( t, λ after ) .
To know thebest possible performance among the candidates, we also apply the test-domain validation (TDV)[19], which selects the hyperparameters with the unseen test domain, and thus is not applicable inpractical situations.
Additional experiments and experimental details can be found in Appendices Hand I, respectively.
The code is available in Supplementary Material.
Colored MNIST
We apply our framework to Colored MNIST [10] with Y =
[10] and Z :=
We aim to predict Y e ∈ Y from digit image data X e ∈ R 2 × 24 × 24 .
The label Y e is changedrandomly to one of the rest uniformly with a probability of 25% .
All digits in images are coloredred or green.
The domain e ∈ [0 , 1] controls the color of digits; the digits Y e > 4 and Y e ≤ 4 arecolored in red and green, respectively, with probability e .
In training, D e ∗ ∼ P X 0 .
1 ,Y 0 .
1 is drawnwith sample size n e ∗ = 5000 , and in testing, Y e is predicted from X e for e 2 := 0 .
Regardingthe higher level Z e , the task is to predict Z e = 0 for X e in 0 − 4 and Z e
= 1 for 5 − 9 (that is,g ( Y e ) = {+
+} if Y e > 4 and else, g ( Y e ) = 0 )
The label Z e is swapped randomly with 25% .
We set E ad = { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } with n e = 5000 for each e ∈ E ad .
We model Φ by a 3 -layer neuralnet.
Setting the maximum epoch 500 and λ before := 1 . 0 , we select ( t, λ after ) from 4 × 7 candidateswith t ∈ { 0 , 100 , 200 , 300 } , λ after ∈ { 10 0 , 10 1 , ..., 10 6 } by each of the CVs.
Table 1 shows the test accuracies for 2000 random samples in the domain e 2 .
The results, togetherwith additional ones in Appendices H.1 and H.2, demonstrate that the proposed methods signiﬁcantly outperform the others for e 2 .
Among the two proposed methods, CV II yields higher testaccuracies on e 2 .
Table 2 shows the accuracy gain of each CV from TDV with the same data sets fordomain e 2 .
These results, together with Appendices H.1 and H.2, concur with the theory in Section4 suggesting that CVII succeeds in wider situations, resulting in smaller errors.
ImageNet To see the performance of the proposed methods for more practical data, they are applied to the ImageNet [53] with its label re-annotated imitating BREEDS [52], which proposes a [Table] method for re-annotating ImageNet to create ano.o.d.
benchmark.
The target task here is to predictlabels Y e ∈ Y of images X e ∈ R 3 × 224 × 224 . Weconduct three experiments with |Y| = 3 , 7 , 17 .
For 289 each experiment, we prepare image datasets in different two manners e 1 and e 2 . 2, 4 and 8 classes out of 3,7, 17 classes, respectively, are composed of differentsubtypes between e 1 and e 2 ; for example, the imagesof class bird in e 1 are composed of ruffed grouse andindigo bunting, and the bird images on e 2 are composed of albatross and water ouzel (Figure 2).
[-Snake data containthunder snake, ringneck snake and gather snakein common with domains e 1 and e 2 ; the background of snakes does not change much overthe species. For the sample of the target task,we used the domain e ∗ = e 1 and drew n e ∗ = 
-]In detail, show Appendix I.3.
In training, D e ∗ ∼ P
X
e 1 ,Y e 1is drawn, and in testing, Y e is predicted from X e one 2 . The coarser label Z e is binary (that is, Z = [2] ), and the sample in the higher level D ead of( X e , Z e ) is drawn from both e 1 and e 2 .
Here, D e 1 ad is the same as D e ∗ but with labels re-annotatedby g .
We model Φ by ResNet50 [29].
Setting the maximum epoch 32 and λ before := 0 .
1 , we select( t, λ after ) from 3 × 4 candidates with t ∈ { 10 , 20 , 30 } , λ after ∈ { 0 , 1 , 10 , 100 } by each of the CVs.
Table 1 shows the test accuracies on e 2 .
We can see that the proposed framework succeeded incapturing the predictor invariant to the change of e , while the other methods failed.
Table 2, whichshows the difference between accuracies by TDV and each CV for the same data set with e 2 , verifythat CVI and II selects λ with the smallest error.
Comparison of two CV methods To highlight the difference between the proposed two CVs, wecompare them regarding the discrepancy between the additional domains of higher level E ad and thedomain for training of the target task e ∗ .
We used synthesized data with X = R 2 , Y =
[10] and
Z :=
[2] , preparing ten distributions { N i } 10 i =1 on R 2 , which include a domain-speciﬁc factor in thesecond component depending on e ∈ Z (see Appendix I.2 for explicit representations of { N i } 10 i =1 ).
The task is to predict the distribution label i ∈ { 1 , . .
Setting e ∗ :
= 20 with n e ∗ = 60000 ,the test task is to predict the label for domain e = − 20 .
Regarding the task with label of higherlevel, we use g ( y ) =
if y > 4 and g ( y ) = 0 else.
We draw D ead ∼ P X e ,Z e ( n e = 20000 ) from
E ad = { e ad , 40 } , where e ad ranges from − 9 to 1 .
As e ad increases, e ad approaches to e ∗ .
The 315 model Φ is a 3 -layer neural net.
We set the maximum epoch 500 and t = 0 , and select λ after from 4candidates λ after ∈ { 0 , 0 .
001 , 80 , 100 } by each CV method.
e ad = −e ad = [Equation] e ad = −e ad = −e ad = −e ad = −e ad = −e ad = −e ad = −e ad =e
[Table]
=
[Table]
Table 3 shows the test accuracy on e = − e ∗ with 2000 random samples ( x, y ) ∼ P X − e ∗ ,Y − e ∗ .
From 318 the results, we can see that CVII tends to select better hyperparameters than CVI, especially in thecase where the variation among the domains is smaller as e ad approaches to e ∗ .
This accords withthe theoretical results in Theorems 4 and 5, which show that CVII ﬁnds a correct hyperparameter insmaller discrepancy between E ad and e ∗ than CVI.
Conclusion
We have proposed a new framework of invariance learning: assuming the availability of datasetsfor another relevant task in higher label hierarchy, we obtain an invariant predictor for the targetclassiﬁcation task using training data in a single domain.
We have also proposed two CV methods forhyperparameter selection, which has been an outstanding problem of previous methods for invariantlearning.
Theoretical analysis has revealed correctness of our methods, including CVs, and theexperimental results have demonstrated the effectiveness of the proposed framework and CVs.
Limitations and Potential societal impact In Theorem 3, it is ensured that our objective functiongives a minimizer of the o.o.d.
risk, only if at least two domains have much discrepancy with respectto distributions.
In general, different domains have different distributions.
Judging the discrepancyis further important problem. As a positive impact, our method will enable us to estimate predictorsthat don’t use discriminatory factors such as gender.
There may be some negative aspects in that ourmethod removes important information for a prediction as well as unnecessary ones.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] In Section 1, we note that data with targetclassiﬁcation label needs from at least one domain.
In Section 4, we theoretically reveal theconditions for our objective function and CV methods to succeed.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] It is included inSection 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] They are included in Appendices.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[Yes] We put the code in the supple-
mental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] They are speciﬁed in Section 6 and Appendix I.4.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments mul- tiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] They are included in Appendix I.4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable infor
[Table]