Original: V ISUAL R EPRESENTATION L EARNING OVER L ATENT D OMAINS
Revised: V ISUAL R EPRESENTATION L EARNING OVER L ATENT D OMAINS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A fundamental shortcoming of deep neural networks is their specialization to a single task and domain.
Revised: A fundamental shortcoming of deep neural networks is their specialization to a single task and domain

Original: While multi-domain learning enables the learning of compact models that span multiple visual domains, these rely on the presence of domain labels, in turn requiring laborious curation of datasets.
Revised: While multi -domain learning enables the learning of compact models that span multiple visual domains, these rely on the presence of domain labels , in turn requiring laborious curation of datasets

Original: This paper proposes a less explored, but highly realistic new setting called latent domain learning : learning over data from different domains, without access to domain annotations.
Revised: This paper pro poses a less explored , but highly realistic new setting called latent domain learning : learning over data from different domains, without access to domain annotations.

Original: Experiments show that this setting isparticularly challenging for standard models and existing multi-domain approaches, calling for new customized solutions: a sparse adaptation strategy is formulated which adaptively accounts for latent domains in data, and signiﬁcantly enhances learning in such settings.
Revised: Experiments show that this setting is challenging for standard models and existing multi-domain approaches, calling for new customized solutions: a sparse adaptation strategy is formulated which enhances performance by accounting for latent domains in data.

Original: Our method can be paired seamlessly with existing models, and boosts performance in conceptually related tasks, e.g. empirical fairness problems and long-tailed recognition.
Revised: Our method can be paired seamlessly with existing models, and beneﬁts conceptually related tasks, e g. empirical fairness problems and long-tailed recognition

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Datasets have been a major driving force behind the rapid progress in computer vision research in the last two decades.
Revised: Datasets have been a major driving force behind the rapid progress in computer vision research in the last two decades.

Original: They provide a testbed for developing new algorithms and comparing them to existing ones.
Revised: They provide a testbed for developing new algorithms and comparing them to existing ones.

Original: However, datasets can also narrow down the focus of research into overspecialized solutions and impede developing a broader understanding of the world.
Revised: However, datasets can also narrow down the focus of research into overspecialized solutions and impede developing a broader understanding of the world.

Original: In recent years this narrow scope of datasets has been widely questioned (Torralba & Efros, 2011;
Revised: In recent years this narrow scope of datasets has been widely questioned (Torralba & Efros, 2011;

Original: Tommasi et al., 2017; Recht et al., 2019) and addressing some of these limitations has become a very active area of research.
Revised: Tommasi et al., 2017; Recht et al., 2019) and addressing some of these limitations has become a very active area of research.

Original: Two actively studied themes to investigate broader learning criteria are multi-domain learning (Nam & Han, 2016;
Revised: Two actively studied themes to investigate broader learning criteria are multi-domain learning (Nam & Han, 2016;

Original: Bulat et al., 2019; Schoenauer-Sebag et al., 2019) and domain adaptation (Ganin et al., 2016;
Revised: Bulat et al., 2019; Schoenauer-Sebag et al., 2019) and domain adaptation (Ganin et al., 2016;

Original: Tzeng et al., 2017; Hoffman et al., 2018; Xu et al., 2018;
Revised: Tzeng et al., 2017; Hoffman et al., 2018; Xu et al., 2018;

Original: Peng et al., 2019a; Sun et al., 2019b).
Revised: Peng et al., 2019a; Sun et al., 2019b).

Original: While multi-domain techniques focus on learning a single model that can generalize over multiple domains, domain adaptation techniques aim to efﬁciently transfer the representations that are learned in one dataset to another.
Revised: While multi-domain techniques focus on learning a single model that can generalize over multiple domains, domain adaptation techniques aim to efﬁciently transfer the representations that are learned in one dataset to another.

Original: Related themes have also been studied in domain generalization (Li et al., 2018; 2019b;a; Gulrajani & Lopez-Paz, 2020) and continual learning (Kirkpatrick et al., 2017; Lopez-Paz & Ranzato, 2017;
Revised: Related themes have also been studied in domain generalization (Li et al., 2018; 2019b;a; Gulrajani & Lopez-Paz, 2020) and continual learning (Kirkpatrick et al., 2017; Lopez-Paz & Ranzato, 2017;

Original: Riemer et al., 2019), where the focus lies on learning representations that can generalize to unseen domains, and to preserve knowledge acquired from previously seen tasks, respectively.
Revised: Riemer et al., 2019), where the focus lies on learning representations that can generalize to unseen domains, and to preserve knowledge acquired from previously seen tasks, respectively.

Original: While there exists no canonical deﬁnition for what exactly a visual domain is, previous works in multi-domain learning assume that different subsets of data exist, with some deﬁning characteristic that allows them to be separated from each another.
Revised: While there exists no canonical deﬁnition for what exactly a visual domain is, previous works in multi-domain learning assume that different subsets of data exist, with some deﬁning characteristic that allows them to be separated from each other.

Original: Each subset, indexed by d = 1 , . . .
Revised: Each subset, indexed by d = 1 , . . .

Original: , D , is assigned to a pre-deﬁned visual domain, and vice-versa multi-domain methods then use such domain associations to parameterize their representations and learn some p θ ( y | x, d ) .
Revised: , D , is assigned to a pre-deﬁned visual domain, and vice-versa multi-domain methods then use such domain associations to parameterize their representations and learn some p θ ( y | x, d ) .

Original: In some cases domains are intuitive and their annotation straightforward.
Revised: In some cases domains are intuitive and their annotation straightforward.

Original: Consider a problem where images have little visual relationship, for example joint learning of Omniglot handwritten symbols (Lake et al., 2015) and CIFAR-10 objects (Krizhevsky & Hinton, 2009).
Revised: Consider a problem where images have little visual relationship, for example joint learning of Omniglot handwritten symbols (Lake et al., 2015) and CIFAR-10 objects (Krizhevsky & Hinton, 2009).

Original: In this case, it is safe to assume that encoding an explicit domain-speciﬁc identiﬁer into p θ is a good idea, and results in the multi-domain literature provide clear evidence that it ishighly beneﬁcial to do so (Rebufﬁ et al., 2018;
Revised: In this case, it is safe to assume that encoding an explicit domain-speciﬁc identiﬁer into p θ is a good idea, and results in the multi-domain literature provide clear evidence that it is beneﬁcial to do so (Rebufﬁ et al., 2018;

Original: Liu et al., 2019a; Guo et al., 2019a; Mancini et al., 2020). 
Revised: Liu et al., 2019a; Guo et al., 2019a; Mancini et al., 2020). [Figure]

Original: The assumption that domain labels are always available has been widely adopted in multi-domain learning; however this assumption is not without difﬁculty.
Revised: The assumption that domain labels are always available has been widely adopted in multi-domain learning; however this assumption is not without difﬁculty.

Original: For one, unless the process of domain[Figure] annotation is automated due to combining existing datasets as in e.g. Rebufﬁ et al.
Revised: For one, unless the process of domain annotation is automated due to combining existing datasets as in e.g. Rebufﬁ et al.

Original: (2017), their manual collection, curation, and domain labeling is very laborious.
Revised: (2017), their manual collection, curation, and domain labeling is very laborious.

Original: And even if adequate resources exist, it is often difﬁcult to decide the optimal criteria for the annotation of d : some datasets contain sketches, paintings and real world images (Li et al., 2017), others images captured during day or night (Sultani et al., 2018).
Revised: And even if adequate resources exist, it is often difﬁcult to decide the optimal criteria for the annotation of d : some datasets contain sketches, paintings and real world images (Li et al., 2017), others images captured during day or night (Sultani et al., 2018).

Original: Automatically collected datasets (Thomee et al., 2016; Sun et al., 2017) contain mixtures of low/high resolution images, taken with different cameras by amateurs/professionals.
Revised: Automatically collected datasets (Thomee et al., 2016; Sun et al., 2017) contain mixtures of low/high resolution images, taken with different cameras by amateurs/professionals.

Original: There is no obvious answer which of these should form their own distinct domain subset.
Revised: There is no obvious answer which of these should form their own distinct domain subset.

Original: Moreover, the work of Bouchacourt et al.
Revised: Moreover, the work of Bouchacourt et al.

Original: (2018) considers semantic groupings of data: they show that when dividing data by subcategories, such as size, shape, etc., and incorporating this information into the model, then this beneﬁts performance.
Revised: (2018) considers semantic groupings of data: they show that when dividing data by subcategories, such as size, shape, etc., and incorporating this information into the model, then this beneﬁts performance.

Original: Should one therefore also encode the number of objects into domains, or their color, shape, and so on?
Revised: Should one therefore also encode the number of objects into domains, or their color, shape, and so on?

Original: Given the relatively loose requirement that domains are supposed to be different while related in some sense (Pan & Yang, 2009), these examples hint at the difﬁculty of deciding whether domains are needed, and – if the answer to that is yes – what the optimal domain criteria are.
Revised: Given the relatively loose requirement that domains are supposed to be different while related in some sense (Pan & Yang, 2009), these examples hint at the difﬁculty of deciding whether domains are needed, and – if the answer to that is yes – what the optimal domain criteria are.

Original: And note that even if such assignments are made very carefully for some problem, nothing guarantees that they will transfer effectively to some other task.
Revised: And note that even if such assignments are made very carefully for some problem, nothing guarantees that they will transfer effectively to some other task.

Original: This paper carefully investigates this ambiguity and studies two central questions:
Revised: This paper carefully investigates this ambiguity and studies two central questions:

Original: Are domain labels always optimal for learning multi-domain representations?
Revised: Are domain labels always optimal for learning multi-domain representations?

Original: Is it possible to learn models  that generalize well over visually diverse domains without domain labels?
Revised: How can models best be learned that generalize well over visually diverse domains, without domain labels

Original: To study this problem, we introduce a new setting (c.f. Fig. 1) in which models are learned over multiple domains without domain annotations —or latent domain learning for short.
Revised: To study this problem, we introduce a new setting (c.f. Fig. 1) in which models are learned over multiple domains without domain annotations — latent domain learning for short.

Original: While latent domain learning is a highly practical research problem in the context of transfer learning, it poses multiple challenges that have not been previously investigated in connection with deep visual representation learning.
Revised: While latent domain learning is a highly practical research problem in the context of transfer learning, it poses multiple challenges that have not been previously investigated in connection with deep visual representation learning.

Original: In particular, we ﬁnd that the removal of domain associations leads tosevere performance losses for standard architectureson most domains, due to imbalances in the underlying distribution and different difﬁculty levels of the associated domain-level tasks.
Revised: In particular, we ﬁnd that the removal of domain associations leads to performance losses for standard architectures due to imbalances in the underlying distribution and different difﬁculty levels of the associated domain-level tasks.

Original: We carry out a rigorous quantitative analysis that includes concepts from multi-domain learning (Rebufﬁ et al., 2018; Chang et al., 2018), and ﬁnd that their performance beneﬁts do not  extend to latent domain learning.
Revised: We carry out a rigorous quantitative analysis that includes concepts from multi-domain learning (Rebufﬁ et al., 2018; Chang et al., 2018), and ﬁnd that their performance beneﬁts do not directly extend to latent domain learning.

Original: To account for this lost performance, we formulate a novel method called sparse latent adaptation (Section 3.2) which enables internal feature representations to dynamically adapt to instances from multiple domains in data, without requiring annotations for this.
Revised: To account for this lost performance, we formulate a novel method called sparse latent adaptation (Section 3.2) which enables internal feature representations to dynamically adapt to instances from multiple domains in data, without requiring annotations for this.

Original: Furthermore, we show that latent domain learning naturally extends to several challenging real world tasks and single domain data, such as fairness problems (Appendix F), and learning over imbalanced distributions (Appendix G).
Revised: Moreover, we show that latent domain methods appear to beneﬁt single domain data and real world tasks, such as fairness problems (Appendix F), and long-tailed recognition (Appendix G).

Original:  L ATENT D OMAIN L EARNING
Revised:  L ATENT D OMAIN L EARNING

Original: This section provides an overview over latent domain learning and contrasts it against other types of related learning problems, in particular multi-domain learning.
Revised: This section provides an overview over latent domain learning and contrasts it against other types of related learning problems, in particular multi-domain learning.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  P ROBLEM S ETTING
Revised: 2. P ROBLEM S ETTING

Original: When learning on multiple domains [Table] the common assumption is that data is sampled i [Table]
Revised: When learning on multiple domains, the common assumption is that data is sampled i.i.d.

Original: i [Table] d. from a mixture of distributions P d with domain indices d = 1 , . . .
Revised: from a mixture of distributions P d with domain indices d = 1 , . . .

Original: , D .
Revised: , D .

Original: Together, they constitute the datagenerating distribution as P =
Revised: Together, they constitute the datagenerating distribution as P =

Original: (cid:80) d π d P d , where each domain is associated with a relative share π d = N d /N , with N the total number of samples [Table] and N d those belonging to the d ’th domain.
Revised: (cid:80) d π d P d , where each domain is associated with a relative share π d = N d /N , with N the total number of samples, and N d those belonging to the d ’th domain.

Original: In multi-domain learning, domain labels are available for all samples (Nam & Han, 2016;
Revised: In multi-domain learning, domain labels are available for all samples (Nam & Han, 2016;

Original: Rebufﬁ et al., 2017; 2018;
Revised: Rebufﬁ et al., 2017; 2018;

Original: Bulat et al., 2019), such that the overall data available for learning consists of
Revised: Bulat et al., 2019), such that the overall data available for learning consists of

Original: D MD
Revised: D MD

Original: = { ( x i , d i , y i ) } with i = 1 , . . .
Revised: = { ( x i , d i , y i ) } with i = 1 , . . .

Original: , N .
Revised: , N .

Original: In latent domain learning the information associating each sample x i with a domain d i is not available.
Revised: In latent domain learning the information associating each sample x i with a domain d i is not available.

Original: As such, domain-speciﬁc labels y i cannot be inferred from sample-domain pairs ( x i , d i ) and one is instead forced to learn a single model f θ over the latent domain dataset D LD
Revised: As such, domain-speciﬁc labels y i cannot be inferred from sample-domain pairs ( x i , d i ) and one is instead forced to learn a single model f θ over the latent domain dataset D LD

Original: = { ( x i , y i ) } .
Revised: = { ( x i , y i ) } .

Original: While latent domain learning can include mutually exclusive classes and disjoint label spaces
Revised: While latent domain learning can include mutually exclusive classes and disjoint label spaces

Original: Y 1 ∪ · · · ∪ Y D (as in long-tailed recognition, see Appendix G), we mainly focus on the setting of shared label spaces, i.e. Y d = Y d (cid:48) .
Revised: Y 1 ∪ · · · ∪ Y D (as in long-tailed recognition, see Appendix G), we mainly focus on the setting of shared label spaces, i.e. Y d = Y d (cid:48) .

Original: For example a dataset may contain images of dogs or elephants that can appear as either photos, paintings, or sketches.
Revised: For example a dataset may contain images of dogs or elephants that can appear as either photos, paintings, or sketches.

Original: Latent domains have previously attracted interest in the context of domain adaptation, where the lack of annotations was recovered through hierarchical Hoffman et al.
Revised: Latent domains have previously attracted interest in the context of domain adaptation, where the lack of annotations was recovered through hierarchical Hoffman et al.

Original: and kernel-based clustering (Gong et al., 2013), via exemplar SVMs (Xu et al., 2014), or by measuring mutual information (Xiong et al., 2014).
Revised: and kernel-based clustering (Gong et al., 2013), via exemplar SVMs (Xu et al., 2014), or by measuring mutual information (Xiong et al., 2014).

Original: More recent work corrects batch statistics of domain adaptation layers using Gaussian mixtures (Mancini et al., 2018), or studies the shift from some source domain to a target distribution that contains multiple latent domains (Peng et al., 2019b; Matsuura & Harada, 2020).
Revised: More recent work corrects batch statistics of domain adaptation layers using Gaussian mixtures (Mancini et al., 2018), or studies the shift from some source domain to a target distribution that contains multiple latent domains (Peng et al., 2019b; Matsuura & Harada, 2020).

Original: Latent domain learning however differs fundamentally from these works: Table 1 contains a comparison to existing transfer learning settings.
Revised: Latent domain learning however differs fundamentally from these works: Table 1 contains a comparison to existing transfer learning settings.

Original: A common baseline in multi-domain learning is to ﬁnetune D models, one for each individual domain (Rebufﬁ et al., 2018;
Revised: A common baseline in multi-domain learning is to ﬁnetune D models, one for each individual domain (Rebufﬁ et al., 2018;

Original: Liu et al., 2019a).
Revised: Liu et al., 2019a).

Original: Doing so requires learning a large number of parameters and shares no parameters across domains, but can serve as a strong baseline to compare against.
Revised: This requires learning a large number of parameters and shares no parameters across domains, but can serve as a strong baseline to compare against.

Original: We show that in many cases, even when domains were carefully annotated, a dynamic latent domain approach can surpass the performance of such domain-supervised baselines, see Section 4.
Revised: We show that in many cases, even when domains were carefully annotated, a dynamic latent domain approach can surpass the performance of such domain-supervised baselines (see Section 4).

Original: 2. O BSERVED VS . U NIFORM A CCURACY
Revised: 2. O BSERVED VS . U NIFORM A CCURACY

Original: Consider a problem in which the data is sampled i.i.d.
Revised: Consider a problem in which the data is sampled i.i.d.

Original: from P = π a P d a + π b P d b , i.e. two hidden domains.
Revised: from P = π a P d a + π b P d b , i.e. two hidden domains.

Original: Since domain labels are not available inlatent domain learning, the best one can do is to treat all samples equally, and measure the observed accuracy:
Revised: When domain labels are not available in the data, a standard strategy is to treat all samples equally, and measure the observed accuracy:

Original: [Equation]
Revised: [Equation]

Original: where y f denotes the class assigned to sample x i by the model f , and y i its corresponding label for training.
Revised: where y f denotes the class assigned to sample x i by the model f , and y i its corresponding label for training.

Original: The OAcc has a problematic property: if P consists of two imbalanced domains such that π a ≥ π b , then the performance on d a dominates it.
Revised: The OAcc has a problematic property: if P consists of two imbalanced domains such that π a ≥ π b , then the performance on d a dominates it.

Original: For example if d a has a 90% overall share, and the model perfectly classiﬁes this domain while obtaining 0% accuracy on d b , then OAcc would still assume 0.9, hiding the underlying damage to domain d b .
Revised: For example if d a has a 90% overall share, and the model perfectly classiﬁes this domain while obtaining 0% accuracy on d b , then OAcc would still assume 0.9, hiding the underlying damage to domain d b .

Original: This motivates alternative formulations for latent domain learning, to anticipate (and account for) imbalanced domains inreal world data.
Revised: This motivates alternative formulations for latent domain learning, to anticipate (and account for) imbalanced domains in data.

Original: If it is possible to identify some semantic domain labeling (as typically included in multi-domain/domain adaptation benchmarks), one can compare performances across individual subgroups.
Revised: If it is possible to identify some semantic domain labeling (as typically included in multi-domain/domain adaptation benchmarks), one can compare performances across individual subgroups.

Original: This allows picking up on  performance lossesthat just occur on some of them, and which traditional metrics (such as OAcc) fail to capture.
Revised: This allows picking up on domain-speciﬁc performance losses which traditional metrics (such as OAcc) fail to capture.

Original: Where this is possible, we therefore propose to also measure latent domain performance in terms of uniform accuracy which decouples accuracies from relative ground-truth domain sizes:
Revised: Where this is possible, we therefore propose to also measure latent domain performance in terms of uniform accuracy which decouples accuracies from relative ground-truth domain sizes:

Original: [Equation]
Revised: [Equation]

Original: Returning to the above example, a uniform measurement reﬂects the model’s lack of performance on d b as UAcc = 0.5.
Revised: Returning to the above example, a uniform measurement reﬂects the model’s lack of performance on d b as UAcc = 0.5.

Original: Once again note while ground-truth domain annotations are required in order to compute uniform accuracy, these are never used to train latent domain models.
Revised: Once again note while ground-truth domain annotations are required in order to compute uniform accuracy, these are never used to train latent domain models.

Original:  M ETHODS
Revised:  M ETHODS

Original: To enable robust learning in the new proposed setting, we formulate a novel module called sparse latent adaptationstrategy which  adaptively accounts for latent domains.
Revised: To enable robust learning in the new proposed setting, we formulate a novel module called sparse latent adaptation which can adaptively account for latent domains.

Original: Section 3.1 reviews adaptation strategies popular in the multi-domain context, which our method extends (and generalizes).
Revised: Section 3.1 reviews adaptation strategies popular in the multi-domain context, which our method extends (and generalizes).

Original: 3. L ATENT A DAPTATION
Revised: 3. L ATENT A DAPTATION

Original: When domain labels d are available (not the case in latent domain learning) one strategy established by Rebufﬁ et al.
Revised: When domain labels d are available (not the case in latent domain learning) one strategy established by Rebufﬁ et al.

Original: (2017) is to modulate networks by constraining the layerwise transformation of residual networks (He et al., 2016)
Revised: (2017) is to modulate networks by constraining the layerwise transformation of residual networks (He et al., 2016)

Original: Φ( x ) = x
Revised: Φ( x ) = x

Original: + f ( x ) to allow at most a linear change V d per each domain from some pretrained mapping Φ 0 (with f 0 in every layer), whereby Φ( x ) − Φ 0 ( x )
Revised: + f ( x ) to allow at most a linear change V d per each domain from some pretrained mapping Φ 0 (with f 0 in every layer), whereby Φ( x ) − Φ 0 ( x )

Original: =
Revised: =

Original: V d x .
Revised: V d x .

Original: Note the slight abuse of notation here in letting x denote a feature map with channels C .
Revised: Note the slight abuse of notation here in letting x denote a feature map with channels C .

Original: Rearranging this yields:
Revised: Rearranging this yields:

Original: [Equation]
Revised: [Equation]

Original: with a domain-supervised switch that assigns corrections to domains, i.e. g d = 1 for d associated with x and 0 otherwise.
Revised: with a domain-supervised switch that assigns corrections to domains, i.e. g d = 1 for d associated with x and 0 otherwise.

Original: Each V d is parametrized through 1x1 convolutions, and f 0 denotes a shared 3x3 convolution obtained e.g. on ImageNet (Deng et al., 2009).
Revised: Each V d is parametrized through 1x1 convolutions, and f 0 denotes a shared 3x3 convolution obtained e.g. on ImageNet (Deng et al., 2009).

Original: This builds on the assumption that models with strong general-purpose representations require minimal changes to adapt to new tasks (Bilen & Vedaldi, 2017), making learning each V d sufﬁcient, while f 0 remains as is.
Revised: This builds on the assumption that models with strong general-purpose representations require minimal changes to adapt to new tasks (Bilen & Vedaldi, 2017), making learning each V d sufﬁcient, while f 0 remains as is.

Original: Such adaptation strategies have been successfully used inNLP (Stickland & Murray, 2019) and few shot  (Li et al., 2021) settings to restrict the number of learnable parameters there.
Revised: Such adaptation strategies have been successfully used in few shot learning (Li et al., 2021) and NLP (Stickland & Murray, 2019) to restrict the number of learnable parameters there.

Original: In latent domain learning access to d is removed, resulting in two new challenges: we have no a priori information about the right number of corrections { V d } , and we cannot use d to decide which one of these to apply.
Revised: In latent domain learning access to d is removed, resulting in two new challenges: we have no a priori information about the right number of corrections { V d } , and we cannot use d to decide which one of these to apply.

Original: To mitigate the lack of domain labels d , ﬁrst we assume that input data is constituted by K latent distributions P k .
Revised: To mitigate the lack of domain labels d , ﬁrst we assume that input data is constituted by K latent distributions P k .

Original: Second we propose to replace the switch g d with a learnable gating mechanism g 1 ( x ) , . .
Revised: Second we propose to replace the switch g d with a learnable gating mechanism g 1 ( x ) , . .

Original: , g K ( x ) that assigns each sample x to latent domains as follows:
Revised: , g K ( x ) that assigns each sample x to latent domains as follows:

Original: [Equation]
Revised: [Equation]

Original: The gates g k control which convolution is applied to which sample x , and correspond to a categorical variable over K categories, i.e. 0 ≤ g k ≤ 1 and (cid:80) k
Revised: The gates g k control which convolution is applied to which sample x , and correspond to a categorical variable over K categories, i.e. 0 ≤ g k ≤ 1 and (cid:80) k

Original: g k
Revised: g k

Original: = 1 .
Revised: = 1 .

Original: Note in particular how parametric dependency of Φ on d is removed.
Revised: Note in particular how parametric dependency of Φ on d is removed.

Original: How to best choose K is discussed in more detail in Section 4.
Revised: How to best choose K is discussed in more detail in Section 4.

Original: While we motivate our latent domain module from learning over multiple domains, the main goal is not to recover the domain labels annotated in some datasets.
Revised: While we motivate our latent domain module from learning over multiple domains, the main goal is not to recover the domain labels annotated in some datasets.

Original: When optimizing some loss (standard cross-entropy in the classiﬁcation case), there is no guarantee that the learned V k will correspond to an annotated visual domain and many additional factors (shape, pose, color, etc.) can enter them as well.
Revised: When optimizing some loss (standard cross-entropy in the classiﬁcation case), there is no guarantee that the learned V k will correspond to an annotated visual domain and many additional factors (shape, pose, color, etc.) can enter them as well.

Original: Latent domain models are simply optimized to produce the lowest training error, and in fact seldom recover ground-truth domains.
Revised: Latent domain models are simply optimized to produce the lowest training error, and in fact seldom recover ground-truth domains (c.f. Fig. 5).

Original: Note the broader concept presented here may in principle also be incorporated with other multi-task concepts (Perez et al., 2018; Guo et al., 2019a), adaptation strategies however stand out due to their methodological simplicity.
Revised: Note the broader concept presented here may in principle also be incorporated with other multi-task concepts (Perez et al., 2018; Guo et al., 2019a), adaptation strategies however stand out due to their methodological simplicity.

Original: Different options exist for parametrizing the gating function g : X → G ⊆ R K .
Revised: Different options exist for parametrizing the gating function g : X → G ⊆ R K .

Original: An ideal gating mechanism for latent domain learning would fulﬁll two seemingly incompatible properties: be able to ﬁlter latent domains in some layers (requiring a discrete gate), but also share parameters between related latent domains in other layers (smooth gates).
Revised: An ideal gating mechanism for latent domain learning would fulﬁll two seemingly incompatible properties: be able to ﬁlter latent domains in some layers (requiring a discrete gate), but also share parameters between related latent domains in other layers (smooth gates).

Original: The next section proposes how this can be resolved without requiring task relationships (Vandenhende et al., 2020) or outer optimization loops (Wu et al., 2018) through the use of sparseness.
Revised: The next section proposes how this can be resolved without requiring task relationships (Vandenhende et al., 2020) or outer optimization loops (Wu et al., 2018) through the use of sparseness.

Original: 3. S PARSE L ATENT A DAPTERS (SLA)
Revised: 3. S PARSE L ATENT A DAPTERS (SLA)

Original: We parameterize the gating function g with a small linear transformation W : C → R K that constitutes the pre-activation q =
Revised: We parameterize the gating function g with a small linear transformation W : C → R K that constitutes the pre-activation q =

Original: Wϕ ( x ) within the gates, where ϕ : X → C denotes an average pooling projection onto the channels.
Revised: Wϕ ( x ) within the gates, where ϕ : X → C denotes an average pooling projection onto the channels.

Original: Different options exist for activating q ∈ R K . A crucial choice is whether the activation  should map to some discrete space G = { 0 , 1 } K or a continuous G =[0 , 1] K in which the V k are shared.
Revised: A crucial choice is whether the activation for q ∈ R K should map to a discrete space G = { 0 , 1 } K or a continuous G = [0 , 1] K in which the V k are shared.

Original: We propose a different strategy that lets gates be smooth when appropriate, but a threshold τ allows for sparse (or discrete) outputs f τ ( q ) =
Revised: We propose a different strategy that lets gates be smooth when appropriate, but a threshold τ allows for sparse (or discrete) outputs f τ ( q ) =

Original: [ q − τ ( q )] + with [ · ] + = max(0 , · ) .
Revised: [ q − τ ( q )] + with [ · ] + = max(0 , · ) .

Original: Crucially f τ can be solved in a differentiable manner (Martins & Astudillo, 2016) by sorting q 1 ≥ · · · ≥ q K , solving k ∗ =
Revised: Crucially f τ can be solved in a differentiable manner (Martins & Astudillo, 2016) by sorting q 1 ≥ · · · ≥ q K , solving k ∗ =

Original: max { k | 1 + kq k > (cid:80) j ≤ k q j } and computing τ =
Revised: max { k | 1 + kq k > (cid:80) j ≤ k q j } and computing τ =

Original: [( (cid:80) j ≤ k ∗ q j ) − 1] /k ∗ .
Revised: [( (cid:80) j ≤ k ∗ q j ) − 1] /k ∗ .

Original: Consider q = [0 . 1 , 1 . 0 , 0 . 5] for which sparse activation results in f τ ( q ) =
Revised: Consider q = [0 . 1 , 1 . 0 , 0 . 5] for which sparse activation results in f τ ( q ) =

Original: while softmax yields [0 . 202 , 0 . 497 , 0 . 301] .
Revised: while softmax yields [0 . 202 , 0 . 497 , 0 . 301] .

Original: Sparse activation ﬁlters out q 1 , while sharing between q 2 and q 3 .
Revised: Sparse activation ﬁlters out q 1 , while sharing between q 2 and q 3 .

Original: We may now deﬁne:
Revised: We may now deﬁne:

Original: [Equation]
Revised: [Equation]

Original: where [ · ] k picks the k ’th element of the gating sequence.
Revised: where [ · ] k picks the k ’th element of the gating sequence.

Original: To the best of our knowledge sparse activation strategies were never previously employed for expert models in computer vision and have so far been restricted to the NLP setting (Deng et al., 2017; Peters et al., 2019).
Revised: To the best of our knowledge sparse activation strategies were never previously employed for expert models in computer vision and have so far been restricted to the NLP setting (Deng et al., 2017; Peters et al., 2019).

Original: Note SLA generalizes residual adaption (Rebufﬁ et al., 2017; 2018), which is recovered by setting K = 1 .
Revised: Note SLA generalizes residual adaption (Rebufﬁ et al., 2017; 2018), which is recovered by setting K = 1 .

Original: While gating is subject to complex interactions such as negative transfer (Rosenbaum et al., 2019), our ablations in Table 5 clearly show that taking a sparse perspective – which allows the model to assume either continuous or discrete forms – outperforms the alternative of a priori ﬁxing either smoothness through self-attention (Lin et al., 2017b) or discrete Gumbel-based sampling (Jang et al., 2016).
Revised: While gating is subject to complex interactions such as negative transfer (Rosenbaum et al., 2019), our ablations in Table 5 clearly show that taking a sparse perspective – which allows the model to assume either continuous or discrete forms – outperforms the alternative of a priori ﬁxing either smoothness through self-attention (Lin et al., 2017b), or discrete Gumbel-based sampling (Jang et al., 2016).

Original: In fact, this choice between discrete (Veit & Belongie, 2018; Guo et al., 2019b) and continuous mechanisms (Shazeer et al., 2017; Sun et al., 2019a; Wang et al., 2019) delineates previous work that employs differentiable gates.
Revised: Note this choice between discrete (Veit & Belongie, 2018; Guo et al., 2019b) and continuous mechanisms (Shazeer et al., 2017; Sun et al., 2019a; Wang et al., 2019) delineates previous work that employs differentiable gates.

Original: It should be noted that a softmax-activated model can in principle also learn to suppress individual preactivation components by letting some q k go to −∞ .
Revised: A softmax-activated model can in principle also learn to suppress individual preactivation components by letting some q k go to −∞ .

Original: This however requires either learning extra calibration parameters at every layer, deﬁning a hard cutoff value (Shazeer et al., 2017) (thereby removing differentiability), or very large row-norms within the linear mapping W — a highly unlikely outcome given the several mechanisms found in state-of-the-art models (in particular weight decay, norm-penalties, or BN (Ioffe & Szegedy, 2015)) which act as direct counterforces to this.
Revised: This however requires either learning extra calibration parameters at every layer, deﬁning a hard cutoff value (Shazeer et al., 2017) (thereby removing differentiability), or very large row-norms within the linear mapping W — a highly unlikely outcome given the several mechanisms found in state-of-the-art models (in particular weight decay, norm-penalties, or BN (Ioffe & Szegedy, 2015)) which act as direct counterforces to this.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our proposed methods on three latent domain benchmarks: Ofﬁce-Home, PACS, and
Revised: We evaluate our proposed methods on three latent domain benchmarks: Ofﬁce-Home, PACS, and

Original: DomainNet (c.f. Fig. 6, which shows example images from these benchmarks).
Revised: DomainNet (c.f. Fig. 6, which shows example images from these benchmarks).

Original: Note the main goal here is not to compare to existing multi-domain or domain adaptation methods that these datasets were initially designed for, but to study our two central research questions: whether domain labels 
Revised: The main goal here is not to compare to existing multi-domain or domain adaptation methods that these datasets were initially designed for, but to study our two central research questions: whether domain labels are useful for effectively learning over multiple domains, and whether one can learn such representations without domain labels.

Original: [Table]
Revised: [Table]

Original: are optimal for effectively learning over multiple domains, and whether we can learn such representations without domain labels.

Original: We also examine a recent fairness benchmark (see Appendix F), and show that SLAsigniﬁcantly improves robustness under single domain long-tailed distributions (Appendix G).
Revised: We also examine a recent fairness benchmark (see Appendix F) [Table] and show that SLA improves robustness under single domain long-tailed distributions (Appendix G).

Original: All experiments were implemented in PyTorch (Paszke et al., 2017), and code will be released alongside the ﬁnal paper version.
Revised: All experiments were implemented in PyTorch (Paszke et al., 2017).

Original: Optimization
Revised: Optimization

Original: In all experiments, we build our method on a ResNet26 model pretrained on  ImageNet (Deng et al., 2009).
Revised: In all experiments, we couple our method with a ResNet26 model pretrained on a downsized version of ImageNet that was used in previous work by Rebufﬁ et al.

Original: In SLA only gates and corrections are learned, the  backbone f 0 remains ﬁxed at its initial parameters which implicitly regularizes the model (Rebufﬁ et al., 2017).
Revised: In SLA only gates and corrections are learned, the residual backbone f 0 remains ﬁxed at its initial parameters, which implicitly regularizes the model (Rebufﬁ et al., 2017).

Original: We train all models in exactly the same way by the same default hyperparameters: for 120 epochs using stochastic gradient descent (momentum parameter of 0.9), batch size of 128, weight decay of 10 − 4 , and an initial learning rate of 0.1 (reduced by 1/10 at epochs 80, 100).
Revised: Training is carried out for 120 epochs using stochastic gradient descent (momentum parameter of 0.9), batch size of 128, weight decay of 10 − 4 , and an initial learning rate of 0.1 (reduced by 1/10 at epochs 80, 100).

Original: We use ofﬁcial splits in each dataset, and report accuracies over ﬁve random initializations (variance is analyzed in Appendix E). All experiments use standard augmentation techniques: random cropping and ﬂipping, as well as normalization.
Revised: All experiments follow the preprocessing of Rebufﬁ et al. (2017; 2018), alongside standard augmentations such as normalization, random cropping, etc. Accuracies are averaged over ﬁve seeds.

Original: Increasing the number of corrections K within SLA consistently increases performance, however K = 2 already represents a strong boost from the baseline of having no adapters.
Revised: Increasing the number of corrections K within SLA results in small, consistent performance gains. As K = 2 already represents a solid boost from the baseline of having no adapters, we focus on this result in the main part, and report results for higher K alongside variances in Appendix C.

Original: Ofﬁce-Home
Revised: Ofﬁce-Home

Original: The underlying data contains a variety of objects classes (alarm clock, backpack, etc.)
Revised: The underlying data contains a variety of objects classes (alarm clock, backpack, etc.)

Original: among four domains: art , clipart , product , and real world (Venkateswara et al., 2017).
Revised: among four domains: art , clipart , product , and real world (Venkateswara et al., 2017).

Original: In Table 2 we show results for d -supervised multi-domain (MD) approaches: RA (Rebufﬁ et al., 2018), domain-adversarial learning (Ganin et al., 2016) and a baseline of 4 × ResNet26, one for each domain.
Revised: In Table 2 we show results for d -supervised multi-domain (MD) approaches: RA (Rebufﬁ et al., 2018), domain-adversarial learning (Ganin et al., 2016) and a baseline of 4 × ResNet26, one for each domain.

Original: For latent domain (LD) baselines, we then learn a single ResNet26, this time as a latent domain model over all domains.
Revised: For latent domain (LD) baselines, we then learn a single ResNet26, this time as a latent domain model over all domains.

Original: Next, we couple SLA with the very same ResNet26.
Revised: Next, we couple SLA with the very same ResNet26.

Original: Learning a single ResNet26 over latent domains with no access to d -labels signiﬁcantly harms performance.
Revised: Learning a single ResNet26 over latent domains with no access to d -labels signiﬁcantly harms performance.

Original: This problem is not addressed by simply increasing the depth of the network: while accuracy improves slightly, a ResNet56 exhibits the same performance losses — in particular on the latent domains product (P) and real world (R).
Revised: This problem is not addressed by simply increasing the depth of the network: while accuracy improves slightly, a ResNet56 exhibits the same performance losses — in particular on the latent domains product (P) and real world (R).

Original: While residual adaptation (RA) (Rebufﬁ et al., 2018) was shown to work extremely well in many multi-domain scenarios, performance here is sub-par, regardless of whether it accesses d (MD: one
Revised: While residual adaptation (RA) (Rebufﬁ et al., 2018) was shown to work extremely well in many multi-domain scenarios, performance here is sub-par, regardless of whether it accesses d (MD: one

Original: V d per-domain) or not (LD).
Revised: V d per-domain) or not (LD).

Original: This likely results from linear modules being reserved for each d when using annotations, enabling no native cross-domain sharing of parameters.
Revised: This likely results from linear modules being reserved for each d when using annotations, enabling no native cross-domain sharing of parameters.

Original: When d is hidden on the other hand, the model is forced to share a single linear adaptation module V between all four hidden domains, without the ﬂexible gating we propose in SLA.
Revised: When d is hidden on the other hand, the model is forced to share a single linear adaptation module V between all four hidden domains, without the ﬂexible gating we propose in SLA.

Original: Learning annotations through latent domain clustering and coupling this with domain-adversarial gradient reversal as in MMLD (Matsuura & Harada, 2020) increases performance relative to its dannotated counterpart (Ganin et al., 2016).
Revised: Learning annotations through latent domain clustering and coupling this with domain-adversarial gradient reversal as in MMLD (Matsuura & Harada, 2020) increases performance relative to its dannotated counterpart (Ganin et al., 2016).

Original: The increase is modest however, likely because enforcing[Table] [Table] domain-invariance on the gradient level negatively impacts the model’s ability to discriminate between classes (Wang et al., 2020).
Revised: The increase is modest however, likely because enforcing domain-invariance on the gradient level negatively impacts the model’s ability to discriminate between classes (Wang et al., 2020).

Original: Another related baseline is MLFN (Chang et al., 2018) which builds on ResNeXt (Xie et al., 2017) to deﬁne a latent-factor architecture that accounts for multimodality in data.
Revised: Another related baseline is MLFN (Chang et al., 2018) which builds on ResNeXt (Xie et al., 2017) to deﬁne a latent-factor architecture that accounts for multi- [Table] [Table] modality in data.

Original: Crucially where our method is ﬁne-grained and shares convolutions at every layer,
Revised: Crucially where our method is ﬁne-grained and shares convolutions at every layer,

Original: MLFN instead enables and disables entire network blocks, allowing us to outperform MLFN
Revised: MLFN instead enables and disables entire network blocks, allowing us to outperform it.

Original: [Table]

Original: SLA outperforms all currently available latent domain models by a consistent margin [Table] and increases
Revised: SLA outperforms the currently available latent domain models by a consistent margin, and increases

Original: UAcc by 12.79% relative to ResNet26.
Revised: UAcc by 12.79% relative to ResNet26.

Original: Best performance is obtained when K happens to equal D , with performance being reducing slightly from overﬁtting of larger domains for K > D .
Revised: Best performance is obtained when K = D , with performance being reducing slightly from overﬁtting of larger domains for K > D (see Appendix C).

Original: PACS
Revised: PACS

Original: The second experiment examines performance on the PACS dataset (Li et al., 2017).
Revised: The second experiment examines performance on the PACS dataset (Li et al., 2017).

Original: Crucially PACS domains ( art, cartoon, photo, sketch ) differ more markedly from one another (c.f. examples in Fig. 6), hence constituting a demanding latent domain problem.
Revised: Crucially PACS domains ( art, cartoon, photo, sketch ) differ more markedly from one another (c.f. examples in Fig. 6), hence constituting an interesting latent domain problem.

Original: Even for more distinct domains as in PACS, results in Table 3 show that SLA improves over existing baselines.
Revised: Even for more distinct domains as in PACS, results in Table 3 show that SLA improves over existing baselines.

Original: The largest gains occur on smaller domains (e.g. art ), where standard models suppress underrepresented parts of the distribution (see additional discussion on imbalanced distributions in
Revised: The largest gains occur on smaller domains (e.g. art ), where standard models suppress underrepresented parts of the distribution (see additional discussion on imbalanced distributions in

Original: Appendix G).
Revised: Appendix G).

Original: Our method again surpasses the accuracy of 4 × ResNet26, while requiring a fraction of the total parameters ( ∼ 9 . 7 mil for K = 5 vs. ∼ 24 . 8 mil ).
Revised: Our method again surpasses the accuracy of 4 × ResNet26, while requiring a fraction of the total parameters ( ∼ 9.7 mil for K = 5 vs. ∼ 24.8 mil).

Original: The performance of SLA again continues to increase with larger K .
Revised: The performance of SLA again continues to increase with larger K (see Appendix C).

Original: The performance increase from using a latent domain-adversarial approach (Matsuura & Harada, 2020) versus using domain-annotations (Ganin et al., 2016) conﬁrms that learning domains alongside the rest of the network can be a better strategy than trusting in annotations.
Revised: The performance increase from using a latent domain-adversarial approach (Matsuura & Harada, 2020) versus using domain-annotations (Ganin et al., 2016) conﬁrms that learning domains alongside the rest of the network can be a better strategy than trusting in annotations.

Original: Our approach again improves over this, without requiring a clustering stage as in MMLD.
Revised: Our approach again improves over this, without requiring a clustering stage as in MMLD.

Original: Results for k-means (we used D = 4 centers and clustered on both image and feature levels without noticeable changes in performance) and subsequent ﬁnetuning show that a two-stage strategy is suboptimal.
Revised: Results for k-means (using D = 4 centers and clustered on the feature level) and subsequent ﬁnetuning show that a two-stage strategy is suboptimal.

Original: This is not surprising since, similar to d -supervision via g d in Φ of eq.
Revised: This is not surprising since, similar to d -supervision via g d in Φ of eq.

Original: (3), clustering learns ﬁxed switches that get used across all layers.
Revised: (3), clustering learns ﬁxed switches that get used across all layers.

Original: In  SLA we ﬂexibly share or separate features individually at every layer (c.f. qualitative results in Fig. 3), synergizing only where appropriate.
Revised: In contrast to this in SLA we ﬂexibly share or separate features individually at every layer (c.f. qualitative results in Fig. 3), synergizing only where appropriate.

Original: DomainNet
Revised: DomainNet

Original: We also evaluate models on a large-scale benchmark called DomainNet (Peng et al., 2019a).
Revised: We also evaluate models on a large-scale benchmark called DomainNet (Peng et al., 2019a).

Original: This dataset contains 518 447 images from six domains (clipart, painting, photos, sketch, 
Revised: This dataset contains 518 447 images from six domains (clipart, painting, photos, sketch, infographics, and quickdraw), with a total of |Y| = 345 object classes. The optimization settings remain unchanged from those in previous sections. Results are shown in Table 4. MLFN performs best on quickdraw, a domain that differs visibly from others (c.f. Fig.

Revised: 6 for examples from each domain), and having entire network blocks dedicated to it seems to beneﬁt performance.

Revised: On all remaining domains, SLA outperforms existing models,

Original: A
Revised: A

Original: C
Revised: C

Original: P
Revised: P

Original: R
Revised: R

Original: A
Revised: A

Original: C
Revised: C

Original: P
Revised: P

Original: R
Revised: R

Original: Office-

Original: [Figure]
Revised: [Figure]

Original: Cosine similarity
Revised: Cosine similarity

Original: A
Revised: A

Original: C
Revised: C

Original: P
Revised: P

Original: S
Revised: S

Original: A
Revised: A

Original: C
Revised: C

Original: P
Revised: P

Original: S
Revised: S

Original: PACS
Revised: PACS

Original: Cosine similarity
Revised: Cosine similarity

Original: [Figure]
Revised: [Figure]

Revised: A

Original: [Figure]
Revised: [Figure]

Original: infographics, and quickdraw), with a total of |Y| = 345 object classes.
Revised: Clipart [Figure]

Original: The optimization settings remain unchanged from those in previous sections.
Revised: C

Original: Results are shown in Table 4.
Revised: [Figure] Product [Figure]

Original: MLFN performs best on quickdraw, a domain that differs visibly from others (c.f. Fig. 6 for examples from each domain), and having entire network blocks dedicated to it seems to beneﬁt performance.
Revised: [Figure]

Original: On all remaining domains, SLA ( K = 2 ) outperforms existing models, regardless of whether they were designed speciﬁcally for multi-domain problems, such as RA, or whether they are much deeper/parameter-intensive (ResNet56).
Revised: regardless of whether they were designed speciﬁcally for multi-domain problems, such as RA, or whether they are much deeper/parameter-intensive (ResNet56).

Original: Qualitative analysis
Revised: Qualitative analysis

Original: We (i) compare global statistics of Ofﬁce-Home and PACS domains as well as (ii) their per-layer treatment within SLA; (iii) analyze sparse gating, (iv) representations learned by SLA, and show that (v) our module shares between geometric properties (shape, pose, etc.).
Revised: We (i) compare global statistics of Ofﬁce-Home and PACS domains as well as (ii) their per-layer treatment within SLA; (iii) analyze sparse gating, (iv) representations learned by SLA, and show that (v) our module shares between geometric properties (shape, pose, etc.).

Original: i) Fig
Revised: i) Fig.

Revised: 2: average cosine similarities of per-domain gating vectors g ∈G L across l =1 , . . .

Revised: , L layers of ResNet26 show that Ofﬁce-Home domains differ less than those in PACS.

Original: Home.
Revised: ii) Fig. 3: layerwise measurements of Corr[ g l ( x ) , g l ( x (cid:48) )] for x, x (cid:48) drawn from differing d (cid:54) = d (cid:48) for Ofﬁce-Home.

Original: If inter-domain correlation is high, then similar corrections V k are responsible for processing samples from two domains.
Revised: If inter-domain correlation is high, then similar corrections V k are responsible for processing samples from two domains.

Original: Across top layers of the network there is little correlation, presumably as low-level information associated with each domain is processed independently.
Revised: Across top layers of the network there is little correlation, presumably as low-level information associated with each domain is processed independently.

Original: In the mid to bottom stages correlation increases: these layers are typically associated with higherorder features (Yosinski et al., 2014; Mahendran & Vedaldi, 2016; Asano et al., 2020), and since 
Revised: In the mid to bottom stages correlation increases: these layers are typically associated with higherorder features (Yosinski et al., 2014; Mahendran & Vedaldi, 2016; Asano et al., 2020), and since label spaces are shared between latent domains, similar object-level features are required to classify objects into their respective categories.

Original: [Figure]
Revised: [Figure]

Original: label spaces are shared between latent domains, similar object-level features are required to classify objects into their respective categories.

Original: iii) Fig.
Revised: iii) Fig.

Original:  sparse  the ﬂexibility to either output singular activations (i.e. become fully discrete) or all non-zero values (a continuous gate).
Revised: 4: sparse gates have the ﬂexibility to either output singular activations (i.e. become fully discrete) or all non-zero values (a continuous gate).

Original: We measure the per-layer sparsity E  ∼  d [ K − (cid:107) g l ( x ) (cid:107) 0 ] / ( K − 1) where (cid:107) · (cid:107) 0 counts values different from zero, ﬁnding sparsity of SLA to vary across model depth.
Revised: We measure the per-layer sparsity E x ∼ P d [ K − (cid:107) g l ( x ) (cid:107) 0 ] / ( K − 1) where (cid:107) · (cid:107) 0 counts values different from zero, ﬁnding sparsity of SLA to vary across model depth.

Original: Interestingly after each downsampling operation SLA tends to be relatively sparse, followed by a dense gate, then again a sparse one, and so forth.
Revised: Interestingly after each downsampling operation SLA tends to be relatively sparse, followed by a dense gate, then again a sparse one, and so forth.

Original: The model thus utilizes the extra ﬂexibility resulting from sparse gates.
Revised: The model thus utilizes the extra ﬂexibility resulting from sparse gates.

Original: Due to PACS domains being relatively distinctive, the dataset is an interesting candidate for additional analysis in (iv) and (v) of how sparse adaptation handles the different ground-truth domains.
Revised: Due to PACS domains being relatively distinctive, the dataset is an interesting candidate for additional analysis in (iv) and (v) of how sparse adaptation handles the different ground-truth domains.

Original: iv) Fig.
Revised: iv) Fig.

Original: 5 (left): gate vectors g ∈ G L for samples from all four domains in PACS visualized by their principal components.
Revised: 5 (left): gate vectors g ∈ G L for samples from all four domains in PACS visualized by their principal components.

Original: SLA exhibits an intuitive clustering of human-annotated PACS domains: visually similar art and photo (•,•) cluster together.
Revised: SLA exhibits an intuitive clustering of human-annotated PACS domains: visually similar art and photo (•,•) cluster together.

Original: The manifold describing sketches (•) is arguably more primitive than those of the other domains, and indeed only maps to a small region.
Revised: The manifold describing sketches (•) is arguably more primitive than those of the other domains, and indeed only maps to a small region.

Original: Cartoon (•) lies somewhere between sketches and real world images.
Revised: Cartoon (•) lies somewhere between sketches and real world images.

Original: This matches intuition: a cartoon is, more or less, just a colored sketch.
Revised: This matches intuition: a cartoon is, more or less, just a colored sketch.

Original: Fig.
Revised: Fig.

Original: 5 also highlights one sample that shows an elephant that SLA places among the cartoon (•) domain, but which has been assigned a ground-truth domain label of photo (•) in the PACS dataset.
Revised: 5 also highlights one sample that shows an elephant that SLA places among the cartoon (•) domain, but which has been assigned a ground-truth domain label of photo (•) in the PACS dataset.

Original: The ground-truth label seems to have been annotated in error, but different from approaches that use d -supervision, our SLA processes latent domains on-the-ﬂy and is therefore not irritated by this.
Revised: The ground-truth label seems to have been annotated in error, but different from approaches that use d -supervision, our SLA processes latent domains on-the-ﬂy and is therefore not irritated by this.

Original: v) Fig.
Revised: v) Fig.

Original: 5 (right): pairs of samples with similar gates.
Revised: 5 (right): pairs of samples with similar gates.

Original: This shows that latent domains are indicative of more than ground-truth domain labels and extend to geometric similarities: pose, color, etc. of the samples are visibly related.
Revised: This shows that latent domains are indicative of more than ground-truth domain labels and extend to geometric similarities: pose, color, etc. of the samples are visibly related.

Original: Compare in particular the poses of elephants (second row).
Revised: Compare in particular the poses of elephants/dogs (second/third row).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we explored two questions: (i) whether domain associations are required for learning effective models over multiple visual domains– as assumed by all existing work – and (ii) if manually curated domain labels are optimal for parametrizing visual representations .
Revised: In this paper we explored two questions: (i) whether domain associations are required for learning effective models over multiple visual domains and (ii) how multi-domain models may best be learned without depending on manually curated domain labels.

Original: As has been shown, the performance of existing models does degrade without domain labels, raising doubts about their suitability for realistic problems that involve diverse data sources.
Revised: As has been shown, the performance of existing models does degrade without domain labels, raising doubts about their suitability for realistic problems that involve diverse data sources.

Original: As a remedy, we proposed a novel adaptation strategy which reclaims (and often exceeds) lost accuracy on latent domains, beneﬁting several problems where some notion of a domain (but no annotation) exists. 
Revised: As a remedy, we proposed a novel adaptation strategy which reclaims (and often exceeds) lost accuracy on latent domains, beneﬁting several problems where some notion of a domain (but no annotation) exists. A CKNOWLEDGEMENT HB is supported by the EPSRC programme grant Visual AI EP/T028572/1.

Revised: TH was supported by EPSRC grant

Revised: EP/R026173/1.

Original: A PPENDIX A D ATASETS
Revised: A PPENDIX A D ATASETS

Original: Fig.
Revised: Fig.

Original: 6 shows examples from the latent domain benchmarks evaluated in Section 4.
Revised: 6 shows examples from the latent domain benchmarks evaluated in Section 4.

Original: The selected images have equivalent classes
Revised: The selected images have equivalent classes

Original: y d = y d (cid:48)
Revised: y d = y d (cid:48)

Original: ∈ Y (for example chair for Ofﬁce-Home), but different domains (e.g. d = { art , clipart , product , real world } ).
Revised: ∈ Y (for example chair for Ofﬁce-Home), but different domains (e.g. d = { art , clipart , product , real world } ).

Original: These examples show that data from different domains often contain very different visual characteristics (compare e g. photo vs. sketch for PACS), even when the object is the same.
Revised: These examples show that data from different domains often contain very different visual characteristics (compare e g. photo vs. sketch for PACS), even when the object is the same.

Original: At the same time, other domains are more alike (e g. art and photo), indicating that different amounts of sharing between per-domain parameters are required, which in SLA is facilitated by its gating mechanism.
Revised: At the same time, other domains are more alike (e.g. art and photo), indicating that different amounts of sharing between per-domain parameters are required, which in SLA is facilitated by its gating mechanism [Figure] Clipart Product

Original: [Figure]
Revised: [Figure]

Original: B R ELATED W ORK
Revised: B R ELATED W ORK

Original: Multi-domain learning relates most closely to our work.
Revised: Multi-domain learning relates most closely to our work.

Original: The state-of-the-art methods introduce small convolutional corrections in residual networks to account for individual domains (Rebufﬁ et al., 2017; 2018), which was recently extended to obtain efﬁcient multi-task models for related language tasks Stickland & Murray (2019).
Revised: The state-of-the-art methods introduce small convolutional corrections in residual networks to account for individual domains (Rebufﬁ et al., 2017; 2018), which was recently extended to obtain efﬁcient multi-task models for related language tasks Stickland & Murray (2019).

Original: Other work makes use of task-speciﬁc attention mechanisms (Liu et al., 2019a), attempts to scale task-speciﬁc losses (Kendall et al., 2018), or addresses tasks at the level of gradients (Chen et al., 2017).
Revised: Other work makes use of task-speciﬁc attention mechanisms (Liu et al., 2019a), attempts to scale task-speciﬁc losses (Kendall et al., 2018), or addresses tasks at the level of gradients (Chen et al., 2017).

Original: Crucially, these approaches all rely ﬁrmly on domain labels.
Revised: Crucially, these approaches all rely ﬁrmly on domain labels.

Original: Our work is loosely related to learning universal representations (Bilen & Vedaldi, 2017), which was used as a guiding principle in designing more transferable models Tamaazousti et al.
Revised: Our work is loosely related to learning universal representations (Bilen & Vedaldi, 2017), which was used as a guiding principle in designing more transferable models (Tamaazousti et al., 2019).

Original: However, these works also assume the presence of domain labels.
Revised: However, these works also assume the presence of domain labels.

Original: One recent line of work does not make this assumption, by normalizing data in separate batches to account for differences in feature distributions Deecke et al. (2019), or by accounting for latent semantic factors to match images Chang et al.
Revised: Multimodal learning does not make this assumption, and was shown to beneﬁt from accounting for latent semantic factors to match images (Chang et al., 2018), or from normalizing data in separate groups (Deecke et al., 2019).

Original: As we show in our experiments (Section 4), latent domain learning however beneﬁts from more customized solutions than these.
Revised: As we show in our experiments (Section 4), latent domain learning however beneﬁts from more customized solutions than these.

Original: The proposed module gives rise to a differentiable dynamic network architecture, studied e.g. for reinforcement learning (Zoph & Le, 2016; Pham et al., 2018), Bayesian optimization (Kandasamy et al., 2018), or when adapting to new tasks (Mallya et al., 2018; Rosenfeld & Tsotsos, 2018).
Revised: The proposed module gives rise to a differentiable dynamic network architecture, studied e.g. for reinforcement learning (Zoph & Le, 2017; Pham et al., 2018), Bayesian optimization (Kandasamy et al., 2018), or when adapting to new tasks (Mallya et al., 2018; Rosenfeld & Tsotsos, 2018).

Original: For such architectures, two components are commonly used: discrete Gumbel-based sampling (Jang et al., 2016), e.g. leveraged in dynamic computer vision architectures (Veit & Belongie, 2018; Sun et al., 2019a), or continuous self-attentive approaches (Lin et al., 2017b), which have been used successfully to scale  models  to large problem spaces (Shazeer et al., 2017; Wang et al., 2019).
Revised: For such architectures, two components are commonly used: discrete Gumbel-based sampling (Jang et al., 2016), e.g. leveraged in dynamic computer vision architectures (Veit & Belongie, 2018; Sun et al., 2019a), or continuous self-attentive approaches (Lin et al., 2017b), which have been used successfully to scale expert models (Jacobs et al., 1991; Jordan & Jacobs, 1994) to large problem spaces (Shazeer et al., 2017; Wang et al., 2019).

Original: From the perspective of algorithmic fairness, a desirable model property is to ensure consistent predictive equality across different identiﬁable subgroups in data (Zemel et al., 2013; Hardt et al., 2016; Fish et al., 2016).
Revised: From the perspective of algorithmic fairness, a desirable model property is to ensure consistent predictive equality across different identiﬁable subgroups in data (Zemel et al., 2013; Hardt et al., 2016; Fish et al., 2016).

Original: This relates to one of the goals in latent domain learning: to limit implicit model bias towards large domains, and improve robustness on small domains.
Revised: This relates to one of the goals in latent domain learning: to limit implicit model bias towards large domains, and improve robustness on small domains.

Original: Recent work explores connections between models and empirical fairness for visual recognition (Bagdasaryan et al., 2019;
Revised: Recent work explores connections between models and empirical fairness for visual recognition (Bagdasaryan et al., 2019;

Original: Hooker et al., 2020; Wang et al., 2020), different from our experiments however (see Appendix F) they focus their analysis on a setting in which annotations for protected attributes are available. 
Revised: Hooker et al., 2020; Wang et al., 2020), different from our experiments however (see Appendix F) they focus their analysis on a setting in which annotations for protected attributes are available. C V ARIATION OF R ESULTS Fig.

Original: C A BLATION
Revised: 7 displays variances of accuracies recorded over ten random initializations on Ofﬁce-Home (left) and PACS (right). We generally found SLA to be robust to different optimization settings, and as a result observed variances are relatively low across experiments. [Figure] [Figure]

Revised: [Figure]

Original: Replacing sparse gating within SLA registers a drop in performance, regardless of whether smooth or discrete mechanisms are used.
Revised: [Figure] [Figure] Larger K brings an improvement of around 0.5-1% in performance at the expense of a linear increase in learnable parameters (c.f. next section). While accuracy is improved by setting K > 2 , gains appear to saturate in line with previous observations around network width (Xie et al., 2017).

Revised: D M EMORY R EQUIREMENTS

Original: Accuracies for soft and straight-through Gumbel-softmax sampling (Jang et al., 2016) were on par; we report straight-through sampling here.
Revised: In SLA every layer contains O ( K |C| + K |C| 2 ) parameters to parametrize gates and corrections V k , respectively. This is however an extremely modest requirement, in particular because f 0 stays ﬁxed: while a ResNet26 contains ∼ 6.2 mil learnable parameters, even when setting K = 5 within SLA it has just 3.5 mil free parameters, and is a fraction of the number of parameters needed to parametrize four ResNet26 (around 24.8 mil parameters). Note also that the complexity of solving sparse gates in SLA scales as O ( K log K ) , a negligible increase given the small K required in our method. E A BLATION

Original: [Table]
Revised: [Table]

Revised: Replacing sparse gating within SLA registers a drop in performance, regardless of whether smooth or discrete mechanisms are used.

Revised: Accuracies for soft and straight-through Gumbel-softmax sampling (Jang et al., 2016) were on par; we report straight-through sampling here.

Original: We also ran experiments where we did not ﬁx the residual backbone f 0 but updated its parameters alongside the learning of SLA.
Revised: We also ran experiments where we did not ﬁx the residual backbone f 0 but updated its parameters alongside the learning of SLA.

Original: In line with what Rebufﬁ et al.
Revised: In line with what Rebufﬁ et al.

Original: (2017) report, this lead to overﬁtting and performance dropped to UAcc = 73.53.
Revised: (2017) report, this lead to overﬁtting and performance dropped to UAcc = 73.53.

Original: D M EMORY R EQUIREMENTS

Original: In SLA every layer contains O ( K |C| + K |C| 2 ) parameters to parametrize gates and corrections

Original: V k , respectively.

Original: This is however an extremely modest requirement, in particular because f 0 stays ﬁxed: while a ResNet26 contains ∼ 6 .

Original: 2 mil learnable parameters, even when setting K = 5 within

Original: SLA it has just ∼ 3 .

Original: 5 mil free parameters, and is a fraction of the number of parameters needed to parametrize four ResNet26 (around 24 .

Original: 8 mil parameters).

Original: Note also that the complexity of solving sparse gates in SLA scales as O ( K log K ) , a negligible increase given the small K required in our method.

Original: E V ARIATION OF R ESULTS

Original: [Figure]

Original: Fig.

Original: 7 displays performance variations for OfﬁceHome (left) and PACS (right).

Original: The variance is observed to be relatively low across experiments, as we generally found SLA to be highly robust to optimize.

Original: Larger K consistently increase performance, at the cost of a slight increase in learnable parameters (see previous section).

Original: Performance gains from higher K eventually saturate, as more and more latent domains are covered by the model.

Original: Note the performance trends hold in terms of both uniform and observed accuracy.

Original: F F AIRNESS
Revised: F F AIRNESS

Original: Recent work elevated the role of small subgroups in data and examined model fairness on CelebA (Bagdasaryan et al., 2019; Wang et al., 2020; Hooker et al., 2020).
Revised: Recent work elevated the role of small subgroups in data and examined model fairness on CelebA (Bagdasaryan et al., 2019; Wang et al., 2020; Hooker et al., 2020).

Original: Because such subgroups may be interpreted as constituting an individual latent domain component P d , they are an interesting candidate to evaluate our purpose-built SLA on.
Revised: Because such subgroups may be interpreted as constituting an individual latent domain component P d , they are an interesting candidate to evaluate our purpose-built SLA on.

Original: The benchmark contains different labeled attributes (e.g. “brown hair”, “glasses”), and is modiﬁed from the original dataset by hiding gender labels.
Revised: The benchmark contains different labeled attributes (e.g. “brown hair”, “glasses”), and is modiﬁed from the original dataset by hiding gender labels.

Original: Models are evaluated on all 39 remaining
Revised: Models are evaluated on all 39 remaining

Original: [Table]
Revised: [Table]

Original: attributes, which subsequently experience varying amounts of gender skew.
Revised: attributes, which subsequently experience varying amounts of gender skew.

Original: Framed as a latent domain problem we have d = { female , male } , but models have no access to thisspurious information.
Revised: Framed as a latent domain problem we have d = { female , male } , but models have no access to this information.

Original: The images used are the entire Aligned&Cropped subset (Liu et al., 2015) over which we ﬁnetune residual models, replacing only the fully-connected layer of the network.
Revised: The images used are the entire Aligned&Cropped subset (Liu et al., 2015) over which we ﬁnetune residual models, replacing only the fully-connected layer of the network.

Original: We use the optimization settings introduced in Section 4 for 70 epochs with reductions at epochs 30, 40, and 50, selecting the best model on the validation split.
Revised: We use the optimization settings introduced in Section 4 for 70 epochs with reductions at epochs 30, 40, and 50, selecting the best model on the validation split.

Original: This experimental setup is identical to previous work on empirical fairness (Wang et al., 2020; Ramaswamy et al., 2020), which however – different from our work – focused on learning models that have access to the gender-attribute d .
Revised: This experimental setup is identical to previous work on empirical fairness (Wang et al., 2020; Ramaswamy et al., 2020), which however – different from our work – focused on learning models that have access to the gender-attribute d .

Original: We evaluate per-attribute accuracy using mean average precision (mAP) and report bias ampliﬁcation (BA) (Zhao et al., 2017).
Revised: We evaluate per-attribute accuracy using mean average precision (mAP) and report bias ampliﬁcation (BA) (Zhao et al., 2017).

Original: This compares the propensity of a model to make positive predictions (i.e. f exceeds some threshold t + ∈ [0 , 1] ) in the gender g ∗ y that appears most frequent within attribute y , compared to the true counted ratio of positive examples
Revised: This compares the propensity of a model to make positive predictions (i.e. f exceeds some threshold t + ∈ [0 , 1] ) in the gender g ∗ y that appears most frequent within attribute y , compared to the true counted ratio of positive examples

Original: y + :
Revised: y + :

Original: [Equation]
Revised: [Equation]

Original: where t + is optimized for on the validation split.
Revised: where t + is optimized for on the validation split.

Original: For example if 60% of male examples are wearing glasses but under the model this is raised to a total of 65%, then bias is ampliﬁed by BA = 0.05.
Revised: For example if 60% of male examples are wearing glasses but under the model this is raised to a total of 65%, then bias is ampliﬁed by BA = 0.05.

Original: We report performance for ResNet18, ResNet34, and ResNet50 in Table 6 and compare this to the same model with SLA inserted ( K = 2 ).
Revised: We report performance for ResNet18, ResNet34, and ResNet50 in Table 6 and compare this to the same model with SLA inserted.

Original: SLA consistently raises both mAP and reduces bias, indicating that it relies less on spurious correlations in data to formulate its predictions.
Revised: SLA consistently raises both mAP and reduces bias, indicating that it relies less on spurious correlations in data to formulate its predictions.

Original: In Fig.
Revised: In Fig.

Original: 8 we compare per-attribute skew toward either female or male (whichever is more frequent) to the gain in performance from ResNet18 to the same model but with SLA inserted.
Revised: 8 we compare per-attribute skew toward either female or male (whichever is more frequent) to the gain in performance from ResNet18 to the same model but with SLA inserted.

Original: We observe a clear trend here, whereby SLA is able to raise performance the most in those attributes that experience the largest amounts of skew.
Revised: We observe a clear trend here, whereby SLA is able to raise performance the most in those attributes that experience the largest amounts of skew.

Original: G L ONG - TAILED R ECOGNITION
Revised: G L ONG - TAILED R ECOGNITION

Original: Standard models often experience difﬁculty when some classes are heavily underrepresented.
Revised: Standard models often experience difﬁculty when some classes are heavily underrepresented.

Original: This problem has recently been studied in long-tailed recognition (Liu et al., 2019b; Cao et al., 2019) with
Revised: This problem has recently been studied in long-tailed recognition (Liu et al., 2019b; Cao et al., 2019) with

Original: [Table]
Revised: [Table]

Original: benchmarks that modify CIFAR-10 and CIFAR-100 to an imbalanced version by dropping some classes (e.g. 6-10 for CIFAR-10)
Revised: benchmarks that modify CIFAR-10 and CIFAR-100 to an imbalanced version by dropping some classes (e.g. 6-10 for CIFAR-10)

Original: (Buda et al., 2018).
Revised: (Buda et al., 2018).

Original: The severity of the imbalance is described via the ratio ρ = n max /n min between the largest and smallest classes.
Revised: The severity of the imbalance is described via the ratio ρ = n max /n min between the largest and smallest classes.

Original: Long-tailed distributions may be viewed as containing an underrepresented latent component with π = 1 / (1 + ρ ) , and previous results (c.f. Section 4) that fortiﬁed small latent domains within P motivate us to evaluate the imbalance setting more closely here.
Revised: Long-tailed distributions may be viewed as containing an underrepresented latent component with π = 1 / (1 + ρ ) , and previous results (c.f. Section 4) that fortiﬁed small latent domains within P motivate us to evaluate the imbalance setting more closely here.

Original: Since our strategy is entirely model-based
Revised: Since our strategy is architecture-based

Original: [Table]
Revised: [Table]

Original: it can be combined with the most recent state-of-theart (loss-based) techniques for long-tailed recognition: a label-distribution-aware margin loss with deferred reweighting (Cao et al., 2019), or reducing contributions from well-classiﬁed examples as in focal losses (Lin et al., 2017a).
Revised: it can be combined with the most recent state-of-the-art (loss-based) techniques for long-tailed recognition: a label-distribution-aware margin loss with deferred reweighting (Cao et al., 2019), or reducing contributions from well-classiﬁed examples as in focal losses (Lin et al., 2017a).

Original: As Table 7 shows, adaptation via sparse gates acts as a regularizer on the underlying ResNet26, and consistently improves performance on long-tail benchmarks.
Revised: As Table 7 shows, adaptation via sparse gates acts as a regularizer on the underlying ResNet26, and consistently improves performance on long-tail benchmarks.