M ULTI -D OMAIN A DVERSARIAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Multi-domain learning (MDL) aims at obtaining a model with minimal average risk across multiple domains.
Our empirical motivation is automated microscopy data, where cultured cells are imaged after being exposed to known and unknown chemical perturbations, and each dataset displays signiﬁcant experimental bias.
This paper presents a multi-domain adversarial learning approach, M U L ANN , to leverage multiple datasets with overlapping but distinct class sets, in a semisupervised setting.
Our contributions include: i) a bound on the average- and worst-domain risk in MDL, obtained using the H -divergence; ii) a new loss to accommodate semi-supervised multi-domain learning and domain adaptation; iii) the experimental validation of the approach, improving on the state-of-the-art on two standard image benchmarks, and a novel bioimage dataset, C ELL .
I NTRODUCTION
Advances in technology have enabled large scale dataset generation by life sciences laboratories.
These datasets contain information about overlapping but non-identical known and unknown experimental conditions.
A challenge is how to best leverage information across multiple datasets on the same subject, and to make discoveries that could not have been obtained from any individual dataset alone.
Transfer learning provides a formal framework for addressing this challenge, particularly crucial in cases where data acquisition is expensive and heavily impacted by experimental settings.
One such ﬁeld is automated microscopy, which can capture thousands of images of cultured cells after exposure to different experimental perturbations (e.g from chemical or genetic sources).
A goal is to classify mechanisms by which perturbations affect cellular processes based on the similarity of cell images.
In principle, it should be possible to tackle microscopy image classiﬁcation as yet another visual object recognition task.
However, two major challenges arise compared to mainstream visual object recognition problems (Russakovsky et al., 2015).
First, biological images are heavily impacted by experimental choices, such as microscope settings and experimental reagents.
Second, there is no standardized set of labeled perturbations, and datasets often contain labeled examples for a subset of possible classes only.
This has limited microscopy image classiﬁcation to single datasets and does not leverage the growing number of datasets collected by the life sciences community.
These challenges make it desirable to learn models across many microscopy datasets, that achieve both good robustness w.r.t.
experimental settings and good class coverage, all the while being robust to the fact that datasets contain samples from overlapping but distinct class sets.
Multi-domain learning (MDL) aims to learn a model of minimal risk from datasets drawn from distinct underlying distributions (Dredze et al., 2010), and is a particular case of transfer learning (Pan & Yang, 2010).
As such, it contrasts with the so-called domain adaptation (DA) problem (Bickel et al., 2007;
Ben-David et al., 2010; Ganin et al., 2016; Pan & Yang, 2010).
DA aims at learning a model with minimal risk on a distribution called "target" by leveraging other distributions called "sources".
Notably, most DA methods assume that target classes are identical to source classes, or a subset thereof in the case of partial DA (Cao et al., 2018; Zhang et al., 2018).
The expected beneﬁts of MDL, compared to training a separate model on each individual dataset, are two-fold.
First, MDL leverages more (labeled and unlabeled) information, allowing better generalization while accommodating the speciﬁcs of each domain (Dredze et al., 2010; Xiao et al., 2016).
Thus, MDL models have a higher chance of ab initio performing well on a new domain
− a problem referred to as domain generalization (Muandet et al., 2013) or zero-shot domain adaptation (Yang & Hospedales, 2015).
Second, MDL enables knowledge transfer between domains: in unsupervised and semi-supervised settings, concepts learned on one domain are applied to another, signiﬁcantly reducing the need for labeled examples from the latter (Pan & Yang, 2010).
Learning a single model from samples drawn from n distributions raises the question of available learning guarantees regarding the model error on each distribution.
Kifer et al.
(2004) introduced the notion of H -divergence to measure the distance between source and target marginal distributions in
DA.
Ben-David et al.
(2006; 2010) have shown that a ﬁnite sample estimate of this divergence can be used to bound the target risk of the learned model.
The contributions of our work are threefold.
First, we extend the DA guarantees to MDL (Sec. 3.1), showing that the risk of the learned model over all considered domains is upper bounded by the oracle risk and the sum of the H -divergences between any two domains.
Furthermore, an upper bound on the classiﬁer imbalance (the difference between the individual domain risk, and the average risk over all domains) is obtained, thus bounding the worst-domain risk.
Second, we propose the approach
Multi-domain Learning Adversarial Neural Network (M U L ANN ), which extends Domain Adversarial
Neural Networks (D ANN s) (Ganin et al., 2016) to semi-supervised DA and MDL.
Relaxing the
DA assumption, M U L ANN handles the so-called class asymmetry issue (when each domain may contain varying numbers of labeled and unlabeled examples of a subset of all possible classes), through designing a new loss (Sec. 3.2).
Finally, M U L ANN is empirically validated in both DA and
MDL settings (Sec. 4), as it signiﬁcantly outperforms the state-of-the-art on two standard image benchmarks (Saenko et al., 2010; Le Cun et al., 1998), and a novel bioimage benchmark, C ELL , where the state of the art involves extensive domain-dependent pre-processing.
Notation.
Let X denote an input space and Y = { 1 , . . .
, L } a set of classes.
For i = 1 , . . . , n , dataset S i is an iid sample drawn from distribution D i on X × Y .
The marginal distribution of D i on X is denoted by D X i .
Let H be a hypothesis space; for each h in H ( h : X (cid:55)→ Y )
we deﬁne the risk under distribution D i as (cid:15) i ( h ) =
P x ,y ∼D i ( h ( x ) (cid:54) = y ) .
h (cid:63)i (respectively h (cid:63) ) denotes the oracle hypothesis according to distribution D i (resp.
with minimal total risk over all domains):
[Equation]
In the semi-supervised setting, the label associated with an instance might be missing.
In the following, "domain" and "distribution" will be used interchangeably, and the "classes of a domain" denote the classes for which labeled or unlabeled examples are available in this domain.
S TATE OF THE ART
Machine learning classically relies on the iid setting: when training and test samples are independently drawn from the same joint distribution P ( X, Y ) (Vapnik, 1998).
Two other settings emerged in the 1990s, "concept drift" and "covariate shift".
They respectively occur when conditional data distributions P ( Y | X ) and marginal data distributions P ( X ) change, either continuously or abruptly, across training data or between train and test data (Shimodaira, 2000).
Since then, transfer learning has come to designate methods to learn across drifting, shifting or distinct distributions, or even distinct tasks (Pratt et al., 1991; Pan & Yang, 2010).
Restricting ourselves to addressing a single task on a common input space, we distinguish two objectives: minimizing the learning risk over all considered distributions (MDL), or over a single target distribution while exploiting samples from richer source(s) (DA).
MDL is thus distinct from multiple source DA by their respective focus on the average risk over all distributions, versus target accuracy only.
Samples from the different domains can be all, partially, or not labeled (supervised, semi-supervised and unsupervised settings).
Finally, different domains can involve the same classes, or some domains can involve classes not included in other domains, referred to as class asymmetry .
In MDL, the different domains can be taken into account by maintaining shared and domain-speciﬁc parameters (Dredze et al., 2010), or through a domain-speciﬁc use of shared parameters.
The domaindependent use of these parameters can be learned, e.g. using domain-guided dropout (Xiao et al., 2016), or based on prior knowledge about domain semantic relationships (Yang & Hospedales, 2015).
Early DA approaches leverage source examples to learn on the target domain in various ways, e.g. through reweighting source datapoints (Mansour, 2009; Huang et al., 2006;
Gong et al., 2013), or deﬁning an extended representation to learn from both source and target (Daumé III & Marcu, 2006).
Other approaches proceed by aligning the source and target representations with PCA-based correlation alignment (Sun et al., 2016), or subspace alignment (Fernando et al., 2015).
In the ﬁeld of computer vision, a somewhat related way of mapping examples in one domain onto the other is image-to-image translation, possibly in combination with a generative adversarial network (see references in Appendix A).
Intuitively, the difﬁculty of DA crucially depends on the distance between source and target distribution.
Accordingly, a large set of DA methods proceed by reducing this distance in the original input space X , e.g. via importance sampling (Bickel et al., 2007) or by modifying the source representation using optimal transport (Courty et al., 2017; Damodaran et al., 2018).
Another option is to map source and target samples on a latent space where they will have minimal distance.
Neural networks have been intensively exploited to build such latent spaces, either through generative adversarial mechanisms (Tzeng et al., 2017; Ghifary et al., 2016), or through combining task objective with an approximation of the distance between source(s) and target.
Examples of used distances include the Maximum Mean Discrepancy due to Gretton et al.
(Tzeng et al., 2014; Bousmalis et al., 2016), some of its variants (Long et al., 2015; 2016), the L 2 contrastive divergence (Motiian et al., 2017), the Frobenius norm of the output feature correlation matrices (Sun & Saenko, 2016), or the H -divergence (Ben-David et al., 2006; 2010; Ganin et al., 2016;
Pei et al., 2018; Long et al., 2017) (more in Sec. 3).
Most DA methods assume that source(s) and target contain examples from the same classes; in particular, in standard benchmarks such as O FFICE (Saenko et al., 2010), all domains contain examples from the same classes.
Notable exceptions are partial DA methods, where target classes are expected to be a subset of source classes e.g. (Zhang et al., 2018; Cao et al., 2018).
DA and partial DA methods share two drawbacks when applied to semi-supervised MDL with non-identical domain class sets.
First, neither generic nor partial DA methods try to mitigate the impact of unlabeled samples from a class without any labeled counterparts.
Second, as they focus on target performance, (partial) DA methods do not discuss the impact of extra labeled source classes on source accuracy.
However, as shown in Sec. 4.3, class asymmetry can heavily impact model performance if not accounted for.
Bioinformatics is increasingly appreciating the need for domain adaptation methods (Borgwardt et al., 2006; Schweikert et al., 2008; Xu & Yang, 2011; Vallania et al., 2017).
Indeed, experimentalists regularly face the issues of concept drift and covariate shift.
Most biological experiments that last more than a few days are subject to technical variations between groups of samples, referred to as batch effects .
Batch effects in image-based screening data are usually tackled with speciﬁc normalization methods (Birmingham et al., 2009).
More recently, work by Ando et al.
(2017) applied
CorAl (Sun et al., 2016) for this purpose, aligning each batch with the entire experiment.
DA has been applied to image-based datasets for improving or accelerating image segmentation tasks (Becker et al., 2015; van Opbroek et al., 2015; Bermúdez-Chacón et al., 2016; Kamnitsas et al., 2017).
However, to our knowledge, MDL has not yet been used in Bioimage Informatics, and this work is the ﬁrst to leverage distinct microscopy screening datasets using MDL.
M ULTI -D OMAIN A DVERSARIAL L EARNING
The H -divergence has been introduced to bound the DA risk (Ben-David et al., 2006; 2010; Ganin et al., 2016).
This section extends the DA theoretical results to the MDL case (Sec. 3.1), supporting the design of the M U L ANN approach (Sec. 3.2).
The reader is referred to Appendix B for formal deﬁnitions and proofs.
3. H - DIVERGENCE FOR MDL
The distance between source and target partly governs the difﬁculty of DA.
The H -divergence has been introduced to deﬁne such a distance which can be empirically estimated with proven guarantees (Batu et al., 2000; Kifer et al., 2004).
This divergence measures how well one can discriminate between samples from two marginals.
It inspired an adversarial approach to DA (Ganin et al., 2016), through the ﬁnding of a feature space in which a binary classiﬁcation loss between source and target projections is maximal, and thus their H -divergence minimal.
Furthermore, the target risk is upper-bounded by the empirical source risk, the empirical H -divergence between source(s) and target marginals, and the oracle DA risk (Ben-David et al., 2006; 2010; Zhang et al., 2012).
Bounding the MDL loss using the H -divergence.
A main difference between DA and MDL is that MDL aims to minimize the average risk over all domains while DA aims to minimize the target risk only.
Considering for simplicity a binary classiﬁcation MDL problem and taking inspiration from (Mansour et al., 2008; Ben-David et al., 2010), the MDL loss can be formulated as an optimal convex combination of domain risks.
A straightforward extension of Ben-David et al.
(Theorem 2 in
Appendix B.2) establishes that the compound empirical risk is upper bounded by the sum of: i) the oracle risk on each domain; ii) a statistical learning term involving the VC dimension of H ; iii) the divergence among any two domains as measured by their H -divergence and summed oracle risk.
This result states that, assuming a representation in which domains are as indistinguishable as possible and on which every 1- and 2-domain classiﬁcation task is well addressed, then there exists a model that performs well on all of them.
In the 2-domain case, the bound is minimized when one minimizes the convex combination of losses in the same proportion as samples.
Bounding the worst risk.
The classiﬁer imbalance w.r.t.
the i -th domain is deﬁned as | (cid:15) i ( h ) − ¯ (cid:15) ( h ) | .
The extent to which marginal D i can best be distinguished by a classiﬁer from H (i.e., the Hdivergence), and the intrinsic difﬁculty (cid:15) (cid:63)i of the i -th classiﬁcation task, yield an upper-bound on the classiﬁer imbalance (proof in Appendix B.3):
Proposition 1.
Given an input space X , n distributions D i over X × { 0 , 1 } and hypothesis class H on X , for any h ∈ H , let (cid:15) i ( h ) (respectively ¯ (cid:15) ( h ) ) denote the classiﬁcation risk of h w.r.t.
distribution
D i (resp.
its average risk over all D i ).
The risk imbalance | (cid:15) i ( h ) − ¯ (cid:15) ( h ) | is upper bounded as: |
[Equation]
with ∆ ij = max ( E D Xj | h (cid:63)i ( x ) − h (cid:63)j ( x ) | , E D Xi |
h
(cid:63)i ( x ) − h (cid:63)j ( x ) | )
Accordingly, every care taken to minimize H -divergences or ∆ ij (e.g. using the class-wise contrastive losses (Motiian et al., 2017)) improves the above upper bound.
An alternative bound of the classiﬁer imbalance can be obtained by using the H ∆ H -divergence (proposition 3, and corollaries 4, 5 for the 2-domain case in Appendix).
3. M U L ANN : M ULTI - DOMAIN ADVERSARIAL LEARNING
As pointed out by e.g. Pei et al.
(2018), when minimizing the H -divergence between two domains, a negative transfer can occur in the case of class asymmetry, when domains involve distinct sets of classes.
For instance, if a domain has unlabeled samples from a class which is not present in the other domains, both global (Ganin et al., 2016) and class-wise (Pei et al., 2018) domain alignments will likely deteriorate at least one of the domain risks by putting the unlabeled samples close to labeled ones from the same domain.
A similar issue arises if a domain has no (labeled or unlabeled) samples in classes which are represented in other domains.
In general, unlabeled samples are only subject to constraints from the domain discriminator, as opposed to labeled samples.
Thus, in the case of class asymmetry, domain alignment will tend to shufﬂe unlabeled samples more than labeled ones.
This limitation is addressed in M U L ANN by deﬁning a new discrimination task referred to as Known
Unknown Discrimination (KUD).
Let us assume that, in each domain, a fraction p (cid:63) of unlabeled samples comes from extra classes, i.e. classes with no labeled samples within the domain.
KUD aims at discriminating, within each domain, labeled samples from unlabeled ones that most likely belong to such extra classes.
More precisely, unlabeled samples of each domain are ranked according to the entropy of their classiﬁcation according to the current classiﬁer, restricted to their domain classes.
Introducing the hyper-parameter p , the top p % examples according to this classiﬁcation entropy are deemed "most likely unknown", and thus discriminated from the labeled ones of the same domain.
The KUD module aims at repulsing the most likely unknown unlabeled samples from the labeled ones within each domain (Fig. 1), thus resisting the contractive effects of global domain alignment.
Overall, M U L ANN involves 3+ n (cid:48) interacting modules, where n (cid:48) is the number of domains with unlabeled data.
The ﬁrst module is the feature extractor with parameters θ f , which maps the input space X to some latent feature space Ω . 2+ n (cid:48) modules are deﬁned on Ω : the classiﬁer module, the domain discriminator module, and the n (cid:48)
KUD modules, with respective parameters θ c , θ d and ( θ k,i ) i .
All modules are simultaneously learned by minimizing loss L ( θ f , θ c , θ d , θ k ) :
[Equation]
1.0 p 0.
T e s t
e rr o r o n
M N I S TM p =0.3 p =0. p =0.7 p = Labeled data Unlabeled data where ζ and λ are hyper-parameters, L ic ( θ f , θ c ) is the empirical classiﬁcation loss on labeled examples in S i ,
L id ( θ f , θ d ) is the domain discrimination loss (multi-class cross-entropy loss of classifying examples from S i in class i ), and L jk ( θ f , θ k,j ) is the KUD loss (binary cross-entropy loss of discriminating labelled samples from S i from the "most likely unknown" unlabelled samples from
S i ).
The loss minimization aims to ﬁnd a saddle point (ˆ θ f , ˆ θ y , ˆ θ d , ˆ θ k ) , achieving an equilibrium between the classiﬁcation performance, the discrimination among domains (to be prevented) and the discrimination among labeled and some unlabeled samples within each domain (to be optimized).
The sensitivity w.r.t.
hyperparameter p will be discussed in Sec. 4.3.
E XPERIMENTAL VALIDATION
This section reports on the experimental validation of M U L ANN in DA and MDL settings on three image datasets (Sec. 4.2), prior to analyzing M U L ANN and investigating the impact of class asymmetry on model performances (Sec. 4.3).
4. I MPLEMENTATION
Datasets The DA setting considers three benchmarks: D IGITS , including the well-known MNIST and MNIST-M (Le Cun et al., 1998; Ganin et al., 2016); Synthetic road signs and German trafﬁc sign benchmark (Chigorin et al., 2012; Stallkamp et al., 2012) and O FFICE (Saenko et al., 2010).
The MDL setting considers the new C ELL benchmark, which is made of ﬂuorescence microscopy images of cells (detailed in Appendix C).
Each image contains tens to hundreds of cells that have been exposed to a given chemical compound, in three domains: California (C), Texas (T) and England (E).
There are 13 classes across the three domains (Appendix, Fig. 2); a drug class is a group of compounds targeting a similar known biological process, e.g. DNA replication.
Four domain shifts are considered: C ↔ T, T ↔ E, E ↔ C and C ↔ T ↔ E.
Baselines and hyperparameters.
In all experiments, M U L ANN is compared to D ANN (Ganin et al., 2016) and its extension M ADA (Pei et al., 2018) (that involves one domain discriminator module per class rather than a single global one).
For D ANN , M ADA and M U L ANN , the same pre-trained VGG-16 architecture (Simonyan & Zisserman, 2014) from Caffe (Jia et al., 2014) is used for O FFICE and C ELL 2 ; the same small convolutional network as Ganin et al.
(2016) is used for
D IGITS (see Appendix D.1 for details).
The models are trained in Torch (Collobert et al., 2011) using stochastic gradient descent with momentum ( ρ = 0 . 9 ).
As in (Ganin et al., 2016), no hyper-parameter grid-search is performed for O FFICE results - double cross-validation is used for all other benchmarks. Hyper-parameter ranges can be found in Appendix D.2.
Semi-supervised setting.
For O FFICE and C ELL , we follow the experimental settings from Saenko et al.
A ﬁxed number of labeled images per class is used for one of the domains in all cases
[Table]
Evaluation goals.
A ﬁrst goal is to assess M U L ANN performance comparatively to the baselines.
A second goal is to assess how the experimental setting impacts model performance.
As domain discriminator and KUD modules can use both labeled and unlabeled images, a major question regards the impact of seeing unlabeled images during training.
Two experiments are conducted to assess this impact: a) the same unlabeled images are used for training and evaluation (referred to as fully transductive setting, noted FT) ; b) some unlabeled images are used for training, and others for evaluation (referred to as non-fully transductive setting, noted NFT).
(The case where no unlabeled images are used during training is discarded due to poor results).
4. E VALUATION
DA on D IGITS and O FFICE .
Table 1 compares M U L ANN with D ANN and M ADA (Sec. 4.1).
Other baselines include: Learning from source and target examples with no transfer loss; Published results from (Motiian et al., 2017)
(legend CCSA), that uses a contrastive loss to penalizes large (resp.
small) distances between same (resp.
different) classes and different domains in the feature space;
Published results from (Tzeng et al., 2015), an extension of D ANN that adds a loss on target softmax values ("soft label loss"; legend Tseng15).
Overall, M U L ANN yields the best results, signiﬁcantly improving upon the former best results on the most difﬁcult cases, i.e., D
→ A, A → D or W → A. As could be expected, the fully transductive results match or signiﬁcantly outperform the non-fully transductive ones.
Notably, M ADA performs similarly to D ANN on D IGITS and RoadSigns, but worse on O FFICE ; a potential explanation is that M ADA is hindered as the number of classes, and thus domain discriminators, increases (respectively 10, 32 and 43 classes).
MDL on C ELL .
The state of the art method for ﬂuorescence microscopy images such as C ELL relies on an ad-hoc approach adapted from (Kang et al., 2016).
Objects (cells) are segmented in each image, and circa 650 shape, intensity and texture features are extracted for each object in each image.
The proﬁle of each image is deﬁned as the vector of its Kolmogorov-Smirnov statistics, computed for each feature by comparing its distribution to that of the same feature from pooled negative controls of the same plate 3 .
Classiﬁcation in proﬁle space is realized using linear discriminant analysis, followed by k-nearest neighbor (LDA+k-NN) ("Baseline P" in Table 2).
As a state-of-the-art shallow approach to MDL to be applied in proﬁle space, CORAL (Sun et al., 2016) was chosen ("P + CORAL" in
Table 2).
A third baseline corresponds to ﬁne-tuning VGG-16 without any transfer loss ("Baseline
NN").
[Table]
Table 2 compares D ANN , M ADA and M U L ANN to the baselines, where columns 4-7 (resp.
8-9) consider raw images (resp.
the proﬁle representations).
The fact that a proﬁle-based baseline generally outperforms an image-based baseline was expected, as proﬁles are designed to reduce the impact of experimental settings (column 4 vs. 8).
The fact that standard deviations tend to be larger here than for O FFICE and D IGITS is explained by a higher intra-class heterogeneity; some classes comprise images from different compounds with similar but not identical biological activity.
Most interestingly, M U L ANN and P+CORAL both improve classiﬁcation accuracy on unlabeled classes at the cost of a slighty worse classiﬁcation accuracy for the labeled classes (in all cases but one).
This is explained as reducing the divergence between domain marginals on the latent feature space prevents the classiﬁer from exploiting dataset-dependent biases.
Overall, M U L ANN and P+CORAL attain comparable results on two-domain cases, with M U L ANN performing signiﬁcantly better in the three-domain case.
Finally, M U L ANN matches or signiﬁcantly outperforms D ANN and M ADA .
4. A NALYSES
Two complementary studies are conducted to investigate the impact of hyperparameter p and that of class asymmetry.
The tSNE (van der Maaten & Hinton, 2008) visualizations of the feature space for
D ANN , M ADA and M U L ANN are displayed in Appendix, Fig. 3.
Sensitivity w.r.t.
the fraction p of "known unknowns".
M U L ANN was designed to counter the negative transfer that is potentially caused by class asymmetry.
This is achieved through the repulsion of labeled examples in each domain from the fraction p of unlabeled examples deemed to belong to extra classes (not represented in the domain).
The sensitivity of M U L ANN performance to the value of p and its difference to the ground truth p (cid:63) is investigated on MNIST ↔
MNIST-M. A ﬁrst remark is that discrepancies between p and p (cid:63) has no inﬂuence on the accuracy on a domain without unlabeled datapoints (Fig. 4 in Appendix).
Fig.
1, right, displays the error depending on p for various values of p (cid:63) .
As could have been expected, it is better to underestimate than to overestimate p (cid:63) ; it is even better to slightly underestimate it than to get it right, as the entropy ranking of unlabeled examples can be perturbed by classiﬁer errors.
Impact of class/domain asymmetry.
Section 4.2 reports on the classiﬁcation accuracy when all classes are represented in all domains of a given shift.
In the general case however, the classes represented by the unlabeled examples are unknown, hence there might exist "orphan" classes, with labeled or unlabeled samples, unique to a single domain.
The impact of such orphan classes, referred to as class asymmetry, is investigated in the 2-domain case.
Four types of samples are considered (Table 3): A class might have labeled examples in both domains ( α ), labeled in one domain and unlabeled in the other domain ( β ), labeled in one domain and absent in the other one (orphan γ ), and ﬁnally unlabeled in one domain and absent in the other one (orphan δ ).
The impact of the class asymmetry is displayed on Fig.
3, reporting the average classiﬁcation accuracy of α, β classes on
Case Dom.
[Table]
Dom. 2 Lab. Lab. Unlab. α,
[Table]
α,
[Table]
[Table]
[Table]
, β
[Table] [Table]
[Table]
, β,
[Table]
β,
[Table]
0.96 Domain 1 test accuracy, lab.
D o m a i n 2 t
e s t a cc u r a c y , un l a b .
No orphans Lab. orphans
Unlab.
orphans Lab. & unlab.
orphans
DANN MADA
MuLANN in the asymmetry experiments domain 1 on the x-axis, and classiﬁcation accuracy of unlabeled β classes on domain 2 on the y-axis, for M U L ANN , D ANN and M ADA on O FFICE (on C ELL in Fig.
5, Appendix).
A clear trend is that adding labeled orphans γ (case "2", Fig. 3) entails a loss of accuracy for all algorithms compared to the no-orphan reference (case "1").
This is explained as follows: on the one hand, the γ samples are subject to the classiﬁer pressure as all labeled samples; on the other hand, they must be shufﬂed with samples from domain 2 due to the domain discriminator(s) pressure.
Thus, the easiest solution is to shufﬂe the unlabeled β samples around, and the loss of accuracy on these β samples is very signiﬁcant (the "2" is lower on the y -axis compared to "1" for all algorithms).
The perturbation is less severe for the labeled ( α, β ) samples in domain 1, which are preserved by the classiﬁer pressure ( x -axis).
The results in case "3" are consistent with the above explanation: since the unlabeled δ samples are only seen by the discriminator(s), their addition has little impact on either the labeled or unlabeled data classiﬁcation accuracy (Figs. 3 and 5).
Finally, there is no clear trend in the impact of both labeled and unlabeled orphans (case "4"): labeled ( α, β ) (resp.
unlabeled β ) are only affected for
M ADA on C ELL (resp.
M U L ANN on O FFICE ).
Overall, these results show that class asymmetry matters for practical applications of transfer learning, and can adversely affect all three adversarial methods (Figs. 3 and 5), with asymmetry in labeled class content ("2") being the most detrimental to model performance.
D ISCUSSION AND FURTHER WORK
This paper extends the use of domain adversarial learning to multi-domain learning, establishing how the H -divergence can be used to bound both the risk across all domains and the worst-domain risk (imbalance on a speciﬁc domain).
The stress is put on the notion of class asymmetry, that is, when some domains contain labeled or unlabeled examples of classes not present in other domains.
Showing the signiﬁcant impact of class asymmetry on the state of the art, this paper also introduces
M U L ANN , where a new loss is meant to resist the contractive effects of the adversarial domain discriminator and repulse (a fraction of) unlabeled examples from labeled ones in each domain.
The merits of the approach are satisfactorily demonstrated by comparison to D ANN and M ADA on D IGITS and O FFICE , and results obtained on the real-world C ELL problem establish a new baseline for the microscopy image community.
A perspective for further study is to bridge the gap between the proposed loss and importance sampling techniques, iteratively exploiting the latent representation to identify orphan samples and adapt the loss while learning.
Further work will also focus on how to identify and preserve relevant domain-speciﬁc behaviours while learning in a domain adversarial setting (e.g., if different cell types have distinct responses to the same class of perturbations).
A CKNOWLEDGMENTS
This work was supported by NCI-NIH RO1 CA184984 (LFW) and the Institute of Computational Health Sciences (ICHS) at UCSF (SJA and LFW).
We would like to thank the Shoichet lab (UCSF) for lending us their GPUs, as well as Theresa Gebert for a preliminary version of Fig.
1 (left) and her incredible energy and support
A E XTENDED STATE - OF - THE - ART : IMAGE TRANSLATION
In the ﬁeld of computer vision, another way of mapping examples in one domain onto the other domain is image-to-image translation.
In the supervised case (the true pairs made of an image and its translation are given), Pic2Pix (Isola et al., 2017) trains a conditional GAN to discriminate true pairs from fake ones.
In the unsupervised case, another loss is designed to enforce cycle consistency (simultaneously learning the mapping φ from domain A to B , ψ from B to A , and requiring φoψ
= Id) (Zhu et al., 2017; Yi et al., 2017).
Note that translation approaches do not per se address domain adaptation as they are agnostic w.r.t.
the classes.
Additional losses are used to overcome this limitation: Domain transfer network (DTN) (Taigman et al., 2016) uses an auto-encoder-like loss in the latent space; GenToAdapt (Sankaranarayanan et al., 2017) uses a classiﬁer loss in the latent space; UNIT (Liu et al., 2017) uses a VAE loss.
StarGAN (Choi et al., 2018) combines image-to-image translation with a GAN, where the discriminator is trained to discriminate true from fake pairs on the one hand, and the domain on the other hand.
ComboGAN (Anoosheh et al., 2017) learns two networks per domain, an encoder and a decoder.
DIRT-T (Shu et al., 2018) uses a conditional GAN and a classiﬁer in the latent space, with two additional losses, respectively enforcing the cluster assumption (the classiﬁer boundary should not cross high density region) and a virtual adversarial training (the hypothesis should be invariant under slight perturbations of the input).
Interestingly, DA and MDL (like deep learning in general) tend to combine quite some losses; two beneﬁts are expected from using a mixture of losses, a smoother optimization landscape and a good stability of the representation (Bousquet & Elisseeff, 2002).
B P ROOFS B. D EFINITION OF THE H - DIVERGENCE
Deﬁnition.
(Kifer et al., 2004; Ben-David et al., 2006; 2010)
Given a domain X , two distributions
D and D (cid:48) over that domain and a binary hypothesis class H on X , the H -divergence between D and
D (cid:48) is deﬁned as:
[Equation]
B. B OUNDING MDL LOSS USING THE H - DIVERGENCE
Theorem 2.
Given an input space X , we consider n distributions D i over X ×{ 0; 1 } and a hypothesis class H on X of VC dimension d .
Let α and γ be in the simplex of dimension n .
If S is a sample of size m which contains γ i m samples from D i , and ˆ h is the empirical minimizer of (cid:80) i
α
i ˆ (cid:15) i on ( S i ) i , then for any δ > 0 , with probability at least 1 − δ , the compound empirical error is upper bounded as:
[Equation]
with
[Equation]
and
[Equation]
A tighter bound can be obtained by replacing d H
( D i , D j ) with 12 d H ∆ H ( D i , D j ) .
The H ∆ H
Proof of theorem 2 For i, j we note β i,j = (cid:15) i ( h (cid:63)i,j ) + (cid:15)
j ( h (cid:63)i,j ) =
min h ∈H ( (cid:15) i ( h ) + (cid:15)
j ( h )) .
For α in the n -dimensional simplex and h ∈ H , we note (cid:15)
α
( h ) =
(cid:80)
i α i (cid:15) i ( h ) .
We have for α in the simplex of dimension n , h ∈ H and j ∈ { 1 , . . .
, m } , using the triangle inequality (similarly to the proof of Theorem 4 in (Ben-David et al., 2010))
[Equation]
The last line follows from the deﬁnitions of β i,j and H -divergence.
Thus using lemma 6 in (Ben-David et al., 2010)
[Equation]
with
[Equation]
m
Hence the result.
(cid:3)
B.
B OUNDING DOMAIN IMBALANCE
Proof of proposition 1 We have for h ∈ H and j ∈ [1 , . . .
, m ] , using the triangle inequality and the deﬁnition of (cid:15) (cid:63)i (similarly to the proof of Theorem 1 in (Ben-David et al., 2006))
[Equation]
We have for i
[Equation]
The second line follows from the triangle inequality and the deﬁnition of the H -divergence.
Thus
[Equation]
By symmetry we obtain
[Equation]
Thus the result.
(cid:3)
Proposition 3.
Given a domain X , m distributions D i over X × { 0; 1 } and a hypothesis class H on
X , we have for h ∈ H and j ∈ [1 , . . .
, m ] | (cid:15) j ( h ) − 1 n (cid:88) i (cid:15) i ( h ) | ≤(cid:32) (cid:15) (cid:63)j + 1 n (cid:88) i (cid:15)
[Equation]
where
[Equation]
Proof For i, j ∈
[ i, . . .
, m ] we have
[Equation]
The second line follows from Lemma 3 from (Ben-David et al., 2010), and the third from the triangle inequality.
From this and proposition 1 we obtain the result.
(cid:3)
Corollaries for the 2-domain case
Corollary 4.
Given a domain X , two distributions D S and D T over X × { 0 , 1 } and a hypothesis class H on X , we have for h ∈ H
[Equation]
8) with ∆ = max ( E D XT | h
(cid:63)S ( x ) − h (cid:63)T ( x ) | , E D XS | h (cid:63)S ( x ) − h (cid:63)T ( x ) | )
Corollary 5.
Given a domain X , two distributions D S and D T over X × { 0; 1 } and a hypothesis class H on X , we have for h ∈ H
[Equation]
where
[Equation]
C C ELL DATASET C. T EXAS DOMAIN
This dataset is extracted from that published in (Kang et al., 2016).
It contains 455 biologically active images, in 11 classes, on four 384-well plates, in three channels: H2B-CFP, XRCC5-YFP and cytoplasmic-mCherry.
Our analysis used 10 classes: ’Actin’, ’Aurora’, ’DNA’, ’ER’, ’HDAC’, ’Hsp90’, ’MT’, ’PLK’, ’Proteasome’, ’mTOR’.
On top of the quality control from the original paper, a visual quality control was implemented to remove images with only apoptotic cells, and XRCC5-YFP channel images were smoothed using a median ﬁlter of size 2 using SciPy (Jones et al., 2001–).
C. C ALIFORNIA DOMAIN
This dataset is designed to be similar to the Texas domain (Kang et al., 2016), generated using the same cell line, but in a different laboratory, by a different biologist, and using different equipment.
It contains 1,077 biologically active images, in 10 classes, on ten 384-well plates, in three channels:
H2B-CFP, XRCC5-YFP and cytoplasmic-mCherry.
The classes are: ’Actin’, ’Aurora’, ’DNA’, ’ER’
’HDAC’
’MT’
’PLK’
’mTOR’
Cell culture, drug screening and image acquisition
Previously (Kang et al., 2016), retroviral transduction of a marker plasmid "pSeg" was used to stably express H2B-CFP and cytoplasmicmCherry tags in A549 human lung adenocarcinoma cells.
A CD-tagging approach (Sigal et al., 2006) was used to add an N-terminal YFP tag to endogenous XRCC5.
Cells were maintained in RPMI1640 media containing 10% FBS, 2 mM glutamine, 50 units/ml penicillin, and 50 µ g/ml streptomycin (all from Life Technologies, Inc.), at 37 ◦ C , 5% CO 2 and 100% humidity.
24h prior to drug addition, cells were seeded onto 384-well plate at a density of 1200 cells/well.
Following compound addition, cells were incubated at 37 ◦ C for 48 hours.
Images were then acquired using a GE InCell Analyzer 2000.
One image was acquired per well using a 10x objective lens with 2x2 binning.
Image processing Uneven illumination was corrected as described in (Stoeger et al., 2015).
Background noise was removed using the ImageJ RollingBall plugin (Schneider et al., 2012).
Images were segmented, object features extracted and biological activity determined as previously described (Kang et al., 2016).
A visual quality control was implemented to remove images with obvious anomalies (e.g. presence of a hair or out-of-focus image) and images with only apoptotic cells.
YFP-XRCC channel images were smoothed using a median ﬁlter of size 2.
C. E NGLAND DOMAIN
This dataset was published by Caie et al.
and retrieved from (Ljosa et al., 2012).
It contains 879 biologically active images of MCF7 breast adenocarcinoma cells, in 15 classes on 55 96-well plates, in 3 channels: Alexa Fluor 488 (Tubulin), Alexa Fluor 568 (Actin) and DAPI (nuclei).
Classes with fewer than 15 images and absent from the other datasets ("Calcium regulation", "Cholesterol",
"Epithelial", "MEK", "mTOR") were not used, which leaves 10 classes: ’Actin’, ’Aurora’, ’DNA’, ’ER’, ’Eg5 inhibitor’, ’HDAC’, ’Kinase’, ’MT’, ’Proteasome’, ’Protein synthesis’.
Image processing As the images were acquired using a 20X objective, they were stitched using
ImageJ plugin (Preibisch et al., 2009) and down-scaled 2 times.
Cells thus appear the same size as in the other domains.
Images were segmented, object features extracted and biological activity obtained as previously described (Kang et al., 2016).
A visual quality control was implemented to remove images with obvious anomalies and images with only apoptotic cells.
Images with too few cells were also removed: an Otsu ﬁlter (Otsu, 1979) was used to estimate the percentage of pixels containing nuclei in each image, and images with less than 1% nuclear pixels were removed.
Tubulin channel images were smoothed using a median ﬁlter of size 2.
C. C OMMON IMAGE PRE - PROCESSING
Images which were not signiﬁcantly distinct from negative controls were identiﬁed as previously (Kang et al., 2016) and excluded from our analysis.
Previous work on the England dataset further focused on images which "clearly [have] one of 12 different primary mechanims of action" (Ljosa et al., 2012).
We chose not to do so, since it results in a simpler problem (90% accuracy easy to reach) with much less room for improvement.
Images from all domains were down-scaled 4 times and ﬂattened to form RGB images.
Images were normalized by subtracting the intensity values from negative controls (DMSO) of the same plate in each channel.
England, Texas and California share images for cell nucleus and cytoplasm, but their third channel differs: Texas and California shows the protein XRCC5, whereas England shows the
Actin protein.
Therefore, the experiments which combine Texas and England, and California and
England used only the ﬁrst two channels, feeding an empty third channel into the network.
Similarly, proﬁles contain 443 features which are related to the ﬁrst two channels, and 202 features which are related to the third channel.
Only the former were used in experiments which involve the England dataset.
C. S EMI - SUPERVISED MDL EXPERIMENTS
[Table]
D E XPERIMENTAL SETTINGS D. A RCHITECTURE
As in (Ganin et al., 2016; Tzeng et al., 2014), a bottleneck fully connected layer is added after the last dense layer of VGG-16.
Learning rates on weights (resp.
biases) from "from scratch" layers is ten (resp, twenty) times that on parameters of ﬁne-tuned layers.
Instance normalization is used on
D IGITS , whereas global normalization is used on O FFICE and C ELL .
O FFICE and C ELL
D IGITS
Feature extractor
VGG-16, layers Conv 1 to FC5x5 conv.
32; ReLU; 2x2 max pool, 2x2 stride
Fully connected5x5 conv.
48; ReLU; 2x2 max pool, 2x2 stride
Classiﬁer
Output of feature extractor
Output of feature extractor
Fully connected 100; ReLU Fully connected 100; ReLU
Fully connected L ; Softmax Fully connected L ; Softmax
Domain discriminator
Output of feature extractor Output of feature extractor
Gradient reversal layer Gradient reversal layer
Fully connected 1,024; ReLU; Dropout (0.5)
Fully connected 100; ReLU
Fully connected 1,024; ReLU; Dropout (0.5)
Fully connected i ; Activation
Fully connected i ; Activation
D. H YPER - PARAMETER SEARCH
[Table]
E A DDITIONAL RESULTS E. 3- DOMAIN RESULTS ON O FFICE
[Table] E. T SNE VISUALIZATION We use tSNE (van der Maaten & Hinton, 2008) to visualize the common feature space in the example of Webcam → Amazon.
Fig.
3 shows that classes are overall better separated with M U L ANN .
In particular, when using M U L ANN , unlabeled examples (blue) are both more grouped and closer to labeled points from the other domain. E.
[Table]
SUPERVISED MDL ON THE B IO
[Table]
[Figure] [Figure]