AdaME: Adaptive learning of multisource adaptation ensembles
Abstract
We present a new adaptive algorithm to build multisource domain adaptation neural networks ensembles.
Since the standard convex combination ensembles cannot succeed in this scenario, we present a learnable domain-weighted combination and new learning guarantees based on the deep boosting algorithm.
We introduce and analyze a new algorithm, A DA ME, for this scenario and show that it beneﬁts from favorable theoretical guarantees, is risk-averse and reduces the worst-case mismatch between the inference and training distributions.
We also report the results of several experiments demonstrating its performance in the FM O W-WILDS dataset.
Introduction
Generalization in supervised machine learning dramatically deteriorates as a function of the divergence between the training and test distributions.
Transferring knowledge from multiple source domains to a target domain is appealing but performance decays due to domain shift (Torralba and
Efros, 2011).
In this paper, we are focusing on minimizing the impact of domain shift between the source and target domains via multi-source domain adaptation (MSA) where the labeled data may be collected from multiple sources with different distributions.
The goal is to do well even on the worst-case subpopulation (e.g., a standard model performing poorly on under-represented demographics).
We assume that the target domain can typically be viewed as a combination of the source domains, that is a mixture of their joint distributions, or it may be close to such mixtures.
We focus on how the learner can adaptively combine relatively accurate predictors for each source domain to derive an accurate ensemble predictor for any new mixture target domain.
MSA was ﬁrst theoretically studied by Mansour et al.
(2008) and subsequently by Hoffman et al.
Many methods have been proposed to address different challenges: distribution-weighted combination via conditional probabilities (Cortes et al., 2020a), semi-supervised domain adaptation (Saito et al., 2019) or multisource boosting (Cortes et al., 2021a).
Recently, a common approach to transfer the task knowledge to the unlabeled target domain is an auxiliary feature alignment loss (Guo et al., 2018; Zhao et al., 2020).
Existing ensembling solutions have focused on combining domain-speciﬁc models via a distributionweighted combining rule (Hoffman et al., 2012;
Cortes et al., 2020a) which beneﬁted from favorable theoretical guarantees (Hoffman et al., 2018).
A successful approach to the combination of source predictors is the so-called Q -ensembles (Cortes et al., 2021a).
These are convex combinations g (1,⋅) hhhg (2,⋅) hhh g ( p ,⋅) h ph ph pf ( x ) … … weighted by a domain classiﬁer Q , that is, Q ( k | x ) is the conditional probability of domain k given input point x .
Our contributions are as follows.
(1) We extend the idea of example-dependent distribution-weighted
Q -ensembles to a learnable gating mechanism.
While a distribution-weighted combination with a domain classiﬁer is reported to perform well (Cortes et al., 2020b, 2021a), low accuracy due to domain overlap can affect the performance of the domain-speciﬁc predictors; (2) We extend the work in (Cortes et al., 2021a) and present tighter theoretical guarantees based on deep boosting (Kuznetsov et al., 2014); (3) We generalize the idea of adaptively learning ensembles (i.e., A DA N ET by Cortes et al.
(2017b))
to the MSA case while maintaining the agnostic loss principle from (Cortes et al., 2021a).
This reduces the worst-case mismatch between the test and training distributions;
Our approach 2.1 Learning scenario
Let X denote the input space and Y = {− 1 , +1 } the output space associated to binary classiﬁcation.
We consider a scenario where the learner receives labeled samples from p source domains, each deﬁned by a distribution D k over X × Y , k ∈ [ p ] .
For any function f : X → R and distribution D over X × Y , let L ( D , f ) be the expected loss of f , that is L
( D , f )
=
E ( x,y ) ∼ D [ (cid:96) ( f ( x ) , y )] , where (cid:96) is the binary loss (cid:96) ( f ( x ) , y )
=[-
-]
I ( yf ( x ) ≤ 0) .
For any k ∈ [ p ] , let H k denote a hypothesis set of functions mapping from X to [ − 1 , +1] , | H k | = N k .
The objective of the learner is to ﬁnd a predictor f that is accurate for any target distribution D λ that is a mixture of the source distributions, where λ may be in a subset Λ of the simplex.
Thus, D λ can be written as D λ
= (cid:80) pk
=1 λ k D k , with λ = ( λ 1 , . . .
, λ p ) ≥ 0 and (cid:80) pk
=1 λ k = 1 .
To come up with a predictor f , the learner seeks an ensemble of functions from the base classes H k .
The natural solution of a convex combinations (i.e., (cid:80) t α t h t ) would lead to a poor solution for some distributions (Proposition 1 in Cortes et al.
(2021b)).
Instead, it is possible to consider a combination of base predictors that takes into consideration the conditional probability of domain k given x , or Q ( k | x ) .
In practice, Q ( k | x ) can be approximated by training a dedicated classiﬁer that maps the inputs X to the respective domain k ∈ [ p ] .
In this case, the model becomes a domain-weighted ensemble where the learning process makes each base predictor domain-independent (Cortes et al., 2020b, 2021b).
In this paper, we propose to extend the Q -ensemble mechanism with a function g that combines the discriminative effect of Q ( k |· ) and the performance of h k across domains, ∀ k (cid:48) ∈
[ p ] , such that k (cid:54) =
k
(cid:48) .
Function g aims at mitigating two issues: (1) facilitate the combination of multiple sources in the base predictors and use transfer learning; (2) alleviate the detrimental effect of Q ( k |· ) to the stability of the α ’s when the loss of the discriminators for a domain k is zero, and the α for cross-domain inﬂuence would be forced to be really large.
In the general case, this function g can be seen as a gating mechanism deﬁning the problem as a mixture of experts (MoE) framework where each expert is trained with a base predictor of a domain.
This function g is learned during training to serve as a gating function for a multi-source adaptation ensemble model.
In this context, we consider the following form for the ensembles of base predictors:
[Equation]
where g ( l, x ) refers to the gating function for domain l given x , h l,r ∈ H lr and α l,r ≥ 0 , (cid:80) pl =1 (cid:80) N l r =1 α l,r = 1 .
For any k ∈ [ p ] , deﬁne the family J k as follows:
J k = (cid:8) g ( k, · ) h : h ∈
H k , g ∈ G (cid:9) and H k
= (cid:8)(cid:83) N k r =1 H kr (cid:9) .
Then, the family of ensemble functions F that we consider can be deﬁned as F = conv( (cid:83) pk =1 J k ) .
2.2 Learning guarantees
For any λ ∈ ∆ , let D λ be the distribution deﬁned by D λ
= (cid:80) pk
=1 λ k (cid:98) D k , where (cid:98) D k is the empirical distribution associated to an i.i.d.
sample S k drawn from D k .
D λ is distinct from the distribution associated to D λ .
We denote by S k
= (( x k, 1 , y k, 1 ) , . . .
, ( x k,m k , y k,m k )) ∈ (cid:0) X
× Y (cid:1) m k the labeled sample of size m k received from source k , which is drawn i.i.d.
from D k .
Theorem 1.
Fix ρ > 0 .
Then, for any δ > 0 , with probability at least 1 − δ over the draw of a sample
S =
( S 1 , . . .
, S p ) ∼ D m 1 1 ⊗ · · · ⊗ D m p p , the following inequality holds for all ensemble functions f = (cid:80) Tt
{+=
+}
α t g ( k t , · ) h t ∈ F and all λ ∈ ∆ :
[Equation]
where N p = (cid:80) pl =1 N l and C ( ρ, N p , λ k , m k , δ / p )
= (cid:101) O
[Equation]
[Equation]
The bound of the theorem can be generalized to hold uniformly for all ρ ∈ (0 , 1] , at the price of an additional term of the form (cid:112) log log 2 (2 /ρ ) /m
k using standard techniques (Koltchinskii and
Panchenko, 2002).
The proof of Theorem 1 can be found in Appendix A.
This is the result of applying the deep boosting learning bounds (Kuznetsov et al., 2014) to our framework using the set of p multidomain ensembles in function class J k , k ∈ [ p ] , as the sub-families composing the base classiﬁer set.
The learning bounds in Eq.
2 represent an improvement to the guarantees presented by Cortes et al.
(2021b). Our algorithm beneﬁts from the mixture weight assigned to each sub-family, instead of depending on the worst-case domain Rademacher complexity in Eq. 5 of Theorem 1 in (Cortes et al., 2021b). 2.3 Ensemble learning
A DA ME seeks to ﬁnd a function f = (cid:80)
Tt =
1 α t g ( k t , · ) h t ∈ F that directly minimizes the datadependent generalization bound of Theorem 1.
For the k -th domain, the following objective function for an ensemble f is deﬁned for any α l,j ≥ 0 :
[Equation]
where Φ is a convex, increasing and differentiable function such as the logistic function, and
R l,r
= R m k
( H lr ) is the regularization term dependent on the Rademacher complexity of a base predictor r in domain l .
Given the agnostic loss (Mohri et al., 2019) of a predictor f leads to the following objective function:
[Equation]
We will consider the case where the set Λ coincides with the full simplex, that is Λ = ∆ .
F can then be expressed more simply as F = max pk =1 F k .
Since a convex function composed with an afﬁne function is convex and a sum of convex functions is convex, F is convex as the maximum of a set of convex functions.
2.4 Domain-weighted ensemble learning
Model g ( k, · ) is trained using model distillation from the statically learned domain-weight Q ( k |· ) .
For domain k ∈ [ p ] and input x ∈ X , model g with weights w ∈ R d is deﬁned as:
[Equation]
where g k ( x ) is the linear output for input x and domain k .
The loss function for ensemble output f is deﬁned as (cid:96) ( f ( x ) , y ) :
[Equation]
where γ is a hyperparameter for the linear combination, D KL is the Kullback-Leibler Divergence loss between the softened probability distributions of Q ( k | x ) acting as the teacher model and the g ( k, x ) functioning as a student model with a temperature scaling hyperparameter τ .
2.5 Algorithm
[Equation] [Figure] [Equation] [Figure]
[Equation] [Figure] [Equation] [Equation]
[Figure] The algorithm is described in Figure 2. The problem consists in alternating the optimization of the objective function F in Eq.
4 to ﬁnd the α ’s (line 4) and Eq. 5 (line 14) to optimize w ∗ . Since F is deﬁned over a very large space of base functions, the ﬁrst part of A DA ME consists of applying coordinate descent to Eq. 4.
The W EAK L EARNER function generates a number of candidate models from H k .
For simplicity, the algorithm in Figure 2 generates 2 candidates ( h and h (cid:48) , line 5) for which we are going to select the most optimal given Eq. 4 and the complexity from the theoretical guarantees in Eq.(see line 11).
The algorithm continues adding candidates until the complexity of a new candidate is larger that its improvement on the overall model.
This adaptive process guarantees that the model is only going to be extended with new candidates within the learning bounds of Theorem 1.
This is equivalent to minimizing the following objective function over α and { h, h (cid:48) } :
[Equation]
The candidate generator in W EAK L EARNER produces base predictors trained with data from all domains that are adapted to a particular domain via standard transfer learning (Zhuang et al., 2019).
Experiments
In this Section, we present results for the FM O W- WILDS dataset (Christie et al., 2017; Koh et al., 2020).
The "Baseline" ﬁgures are taken from the WILDS paper. For the A DA N ET and A DA ME(G+Q) ﬁgures, we used ensembles of NFNets-F0 (Brock et al., 2021) that were pretrained on ImageNet, then ﬁnetuned on the FMoW training set, then further ﬁnetuned to adapt to each geographic region to produce 5 per-region ensembles with 5 NFNet-F0s each.
The g model used for A DA ME(G+Q) is also an NFNet-F0 (details in Appendix B).
Table 1 reports average and worstregion accuracies (%) under time shifts in FM O W- WILDS .
Models are trained on data beforeand tested on held-out location coordinates from in-distribution (ID) or out-of-distribution (OOD) test sets.
ID results correspond to the train-to-train setting.
Parentheses show standard deviation acrossreplicates.
In Appendix C we present an extensive set of results for different datasets (e.g., MNIST and sentiment analysis).
[Table]
Conclusions
We presented A DA ME, an algorithm that provides a principled solution for adaptive structural learning of neural network ensembles for multiple source adaptation (MSA). This is an increasingly common and important learning problem due to the possibilities to improve the performance of the worst-case subdomains.
Our solution extends the idea of adaptive ensembles to MSA by introducing a learnable distribution-weighted ensemble function ( g ).
We presented tighter theoretical guarantees based on deep boosting so that the complexity of the ensemble doesn’t depend on the complexity of the worst cross domain model.
Our experimental results on the FM O W- WILDS dataset further demonstrate the effectiveness of our solution.
A Proof of Theorem
Theorem 1.
Fix ρ > 0 .
Then, for any δ > 0 , with probability at least 1 − δ over the draw of a sample
S =
( S 1 , . . .
, S p ) ∼ D m 1 1 ⊗ · · · ⊗ D m p p , the following inequality holds for all ensemble functions f = (cid:80) Tt
{+=
+}
α t g ( k t , · ) h t ∈ F and all λ ∈ ∆ :
[Equation]
where N p = (cid:80) pl =1 N l and C ( ρ, N p , λ k , m k , δ / p )
= (cid:101)
O
[Equation]
Proof.
Assuming a single domain denoted by k and l , being k, l ∈
[ p ] , let F l be a hypothesis set admitting a decomposition F l = (cid:83) N l r =1 G lr for some N l > 1 .
Fix ρ > 0 , then for any δ > 0 , with probability at least 1 − δ over the draw of a sample S k of size m k from D m k , the following inequality holds for all f l = (cid:80) N l r =
α r · g r ∈ F l (Cortes et al., 2014, 2017a):
[Equation]
Fix λ ∈ ∆ , for any k ∈ [ p ] , all ensembles f = (cid:80)
Tt =
1 α t
[Equation]
where N p = (cid:80) pl =1 N l .
By the union bound, the following inequalities hold simultaneously for all k ∈
[ p ] :
[Equation]
Multiplying each by λ k and summing them up yields:
[Equation]
The Rademacher complexity of a product of two (or more) families of functions relates to the sum of the Rademacher complexity of each family (see Lemma 1 and (Cortes et al., 2016)).
The Rademacher complexity of the family of functions J H , G is R m k
( J lr )
= 2( R m k
( H lr )
+ R m k
( G )) which ﬁnalizes the proof.
Lemma 1.
Let F 1 and F 2 be two families of functions mapping X to [ − 1 , +1] .
Let F = { f 1 f 2 : f 1 ∈ F 1 , f 2 ∈ F 2 } .
Then the empirical Rademacher complexities of F for any sample S of size m are bounded:
[Equation]
B FMoW-WILDS Experiments
Here we describe how the FMoW-WILDS models were trained.
First, we loaded an NFNets-F (Brock et al., 2021) pretrained from ImageNet from Github 1 , and zero-initialized the head weights.
Next, we trained the head weights, then ﬁne-tuned all weights on the training set of all domains using 5 random seeds to produce 5 different checkpoints.
During this training process, we used
CutMix, MixUp, random brightness, random contrast, random ﬂip, and 90-degree rotations for image augmentation.
For each checkpoint, we then further ﬁne-tuned on each domain using the same image augmentation but with stochastic-depth dropout rate of 0.5.
The Q-model is also an NFNet, but trained from random initialization to predict the domain id, using the same image augmentation and stochastic-depth dropout using 5 random seeds.
Since we have ﬁve checkpoints, this gives us 5 domain-speciﬁc models per domain , which we ensemble together, giving us 5 domain-speciﬁc ensembles .
Finally, we use the Q-model to ensemble the 5 domain-speciﬁc ensembles to produce the ﬁnal A DA ME(G+Q) ensemble. B.1 Hyperparameters for ﬁnetuning FMoW
(all) • Model architecture: NFNets-F0 (pretrained from ImageNet) • Base learning rate: 1e- • Batch size:• Use batch norm: True • Learning rate schedule: Linear warmup for 600 steps, then decay by 0.995 every 600 steps.
• Stochastic depth dropout rate: 0.
• Num steps: 120,B.2 Hyperparameters for ﬁnetuning FMoW(per-region) • Model architecture: NFNets-F0 (pretrained from FMoW (all))
• Base learning rate: 1e- • Batch size:• Use batch norm: True • Learning rate schedule: Linear warmup for 400 steps, then decay by 0.96 every 300 steps. • Stochastic depth dropout rate: 0. • Num steps: 60, B.3 Hyperparameters for ﬁnetuning FMoW(Q)
• Model architecture: NFNets-F0 (random init) • Base learning rate: 2e- • Batch size:• Use batch norm: False • Learning rate schedule: Linear warmup for 600 steps, then decay by 0.96 every 600 steps.
• Stochastic depth dropout rate: 0.
• Num steps: 60,
[Table]
In this section [Table] we present experimental results for the A DA ME(G+Q) algorithm on several other multiple-source datasets (see Table 2).
Note that the previous study (including AdaBoost, DMSA, and MultiBoost) is restricted to learning an ensemble of decision stump, which A DA ME learns an ensemble of neural networks.
For Sentiment Analysis, A DA ME uses multi-head self-attention for the base models, the Q-model, and the G-model.
For Digits Recognition and Objects Recognition,
A DA ME uses a feed-forward CNN for the base models, the Q-model, and the G-model.