Original: P RINCIPLED H YBRIDS OF G ENERATIVE AND D ISCRIMINATIVE D OMAIN A DAPTATION
Revised: P RINCIPLED H YBRIDS OF G ENERATIVE AND D ISCRIMINATIVE D OMAIN A DAPTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a probabilistic framework for domain adaptation that blends both generative and discriminative modeling in a principled way.
Revised: We propose a probabilistic framework for domain adaptation that blends both generative and discriminative modeling in a principled way.

Original: Under this framework, generative and discriminative models correspond to speciﬁc choices of the prior over parameters.
Revised: Under this framework, generative and discriminative models correspond to speciﬁc choices of the prior over parameters.

Original: This provides us a very general way to interpolate between generative and discriminative extremes through different choices of priors.
Revised: This provides us a very general way to interpolate between generative and discriminative extremes through different choices of priors.

Original: By maximizing both the marginal and the conditional log-likelihoods, models derived from this framework can use both labeled instances from the source domain as well as unlabeled instances from both source and target domains.
Revised: By maximizing both the marginal and the conditional log-likelihoods, models derived from this framework can use both labeled instances from the source domain as well as unlabeled instances from both source and target domains.

Original: Under this framework, we show that the popular reconstruction loss of autoencoder corresponds to an upper bound of the negative marginal log-likelihoods of unlabeled instances, where marginal distributions are given by proper kernel density estimations.
Revised: Under this framework, we show that the popular reconstruction loss of autoencoder corresponds to an upper bound of the negative marginal log-likelihoods of unlabeled instances, where marginal distributions are given by proper kernel density estimations.

Original: This provides a way to interpret the empirical success of autoencoders in domain adaptation and semi-supervised learning.
Revised: This provides a way to interpret the empirical success of autoencoders in domain adaptation and semi-supervised learning.

Original: We instantiate our framework using neural networks, and build a concrete model, DAuto .
Revised: We instantiate our framework using neural networks, and build a concrete model, DAuto .

Original: Empirically, we demonstrate the effectiveness of DAuto on text, image and speech datasets, showing that it outperforms related competitors when domain adaptation is possible.
Revised: Empirically, we demonstrate the effectiveness of DAuto on text, image and speech datasets, showing that it outperforms related competitors when domain adaptation is possible.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Making accurate predictions relies heavily on the existence of labeled data for the desired tasks.
Revised: Making accurate predictions relies heavily on the existence of labeled data for the desired tasks.

Original: However, generating labeled data for new learning tasks is often time-consuming.
Revised: However, generating labeled data for new learning tasks is often time-consuming.

Original: As a result, this poses an obstacle for applying machine learning methods to broader application domains.
Revised: As a result, this poses an obstacle for applying machine learning methods to broader application domains.

Original: Domain adaptation focuses on the situation where we only have access to labeled data from source domain, which is assumed to be different from the target domain we want to apply our model to.
Revised: Domain adaptation focuses on the situation where we only have access to labeled data from source domain, which is assumed to be different from the target domain we want to apply our model to.

Original: The goal of domain adaptation algorithms under this setting is to generalize better in the target domain by exploiting labeled data in the source domain and unlabeled data in the target domain.
Revised: The goal of domain adaptation algorithms under this setting is to generalize better in the target domain by exploiting labeled data in the source domain and unlabeled data in the target domain.

Original: In this paper we propose a probabilistic framework for domain adaptation that combines both generative and discriminative modeling in a principled way.
Revised: In this paper we propose a probabilistic framework for domain adaptation that combines both generative and discriminative modeling in a principled way.

Original: We start from a very simple yet general generative model, and show that a special choice on the prior distribution of model parameters leads to the usual discriminative modeling.
Revised: We start from a very simple yet general generative model, and show that a special choice on the prior distribution of model parameters leads to the usual discriminative modeling.

Original: Due to its generative nature, the framework provides us a principled way to use unlabeled instances from both the source and the target domains.
Revised: Due to its generative nature, the framework provides us a principled way to use unlabeled instances from both the source and the target domains.

Original: Under this framework, if we use non-parametric kernel density estimators for the marginal distribution over instances, we can show that the popular reconstruction loss of autoencoders corresponds to an upper bound of the negative marginal log-likelihoods of unlabeled instances.
Revised: Under this framework, if we use non-parametric kernel density estimators for the marginal distribution over instances, we can show that the popular reconstruction loss of autoencoders corresponds to an upper bound of the negative marginal log-likelihoods of unlabeled instances.

Original: This provides a novel probabilistic interpretation on why unsupervised training with general autoencoders may help with discriminative tasks, though interpretations exist for speciﬁc variants of autoencoders, e.g., denoising autoencoders (Vincent et al., 2008) and contractive autoencoders (Rifai et al., 2011).
Revised: This provides a novel probabilistic interpretation on why unsupervised training with general autoencoders may help with discriminative tasks, though interpretations exist for speciﬁc variants of autoencoders, e.g., denoising autoencoders (Vincent et al., 2008) and contractive autoencoders (Rifai et al., 2011).

Original: Our interpretation may also be used to explain the recent success of autoencoders in semi-supervised learning (Rasmus et al., 2015).
Revised: Our interpretation may also be used to explain the recent success of autoencoders in semi-supervised learning (Rasmus et al., 2015).

Original: We instantiate our framework with ﬂexible neural networks, which are powerful function approximators, leading to a concrete model, DAuto .
Revised: We instantiate our framework with ﬂexible neural networks, which are powerful function approximators, leading to a concrete model, DAuto .

Original: DAuto is designed to achieve the following three objectives simultaneously in a uniﬁed model: 1).
Revised: DAuto is designed to achieve the following three objectives simultaneously in a uniﬁed model: 1).

Original: It learns representations that are informative for the main learning task in the source domain. 2).
Revised: It learns representations that are informative for the main learning task in the source domain. 2).

Original: It learns domain-invariant features that are indistinguishable between the source and the target domains. 3).
Revised: It learns domain-invariant features that are indistinguishable between the source and the target domains. 3).

Original: It learns robust representations under reconstruction loss for instances in both domains.
Revised: It learns robust representations under reconstruction loss for instances in both domains.

Original: To demonstrate the effectiveness of DAuto, we ﬁrst conduct a synthetic experiment using the MNIST dataset, showing its superior performance when adaptation is possible.
Revised: To demonstrate the effectiveness of DAuto, we ﬁrst conduct a synthetic experiment using the MNIST dataset, showing its superior performance when adaptation is possible.

Original: We further compare DAuto with state-of-the-art models on the Amazon benchmark dataset.
Revised: We further compare DAuto with state-of-the-art models on the Amazon benchmark dataset.

Original: As another contribution, we extend DAuto so that it can also be applied in time-series modeling.
Revised: As another contribution, we extend DAuto so that it can also be applied in time-series modeling.

Original: We evaluate it on a speech recognition task, showing its effectiveness in improving recognition accuracies when trained from utterances with different accents.
Revised: We evaluate it on a speech recognition task, showing its effectiveness in improving recognition accuracies when trained from utterances with different accents.

Original: In the end, we also provide qualitative analysis in the case where domain adaptation is hard and all the methods we test fail.
Revised: In the end, we also provide qualitative analysis in the case where domain adaptation is hard and all the methods we test fail.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recently, due to the availability of rich data and powerful computational resources, non-linear representations and hypothesis classes for domain adaptation have been increasingly explored (Glorot et al., 2011; Baktashmotlagh et al., 2013; Chen et al., 2012;
Revised: Recently, due to the availability of rich data and powerful computational resources, non-linear representations and hypothesis classes for domain adaptation have been increasingly explored (Glorot et al., 2011; Baktashmotlagh et al., 2013; Chen et al., 2012;

Original: Ajakan et al., 2014; Ganin et al., 2016).
Revised: Ajakan et al., 2014; Ganin et al., 2016).

Original: This line of works focuses on building common and robust feature representations among multiple domains using either supervised neural networks (Glorot et al., 2011), or unsupervised pretraining using denoising auto-encoders (Vincent et al., 2008; 2010).
Revised: This line of works focuses on building common and robust feature representations among multiple domains using either supervised neural networks (Glorot et al., 2011), or unsupervised pretraining using denoising auto-encoders (Vincent et al., 2008; 2010).

Original: Other works focus on learning feature transformations such that the feature distributions in the source and target domains are close to each other (Ben-David et al., 2007; 2010;
Revised: Other works focus on learning feature transformations such that the feature distributions in the source and target domains are close to each other (Ben-David et al., 2007; 2010;

Original: Ajakan et al., 2014; Ganin et al., 2016).
Revised: Ajakan et al., 2014; Ganin et al., 2016).

Original: In practice it was observed that unsupervised pretraining using stacked denoising auto-encoders (mSDA) (Vincent et al., 2008; Chen et al., 2012) often improves the generalization accuracy (Ganin et al., 2016).
Revised: In practice it was observed that unsupervised pretraining using stacked denoising auto-encoders (mSDA) (Vincent et al., 2008; Chen et al., 2012) often improves the generalization accuracy (Ganin et al., 2016).

Original: One of the limitations of mSDA is that it needs to explicitly form the covariance matrix of input features and then solves a linear system, which can be computationally expensive  in high dimensional settings. On the other hand, it is also not clear how to extend mSDA so that it can also be applied for time-series modeling.
Revised: One of the limitations of mSDA is that it needs to explicitly form the covariance matrix of input features and then solves a linear system, which can be computationally expensive to solve exactly in high dimensional settings, but approximate scheme exists.

Original: Domain adversarial neural networks (DANN) is a discriminative model to learn domain-invariant features (Ganin et al., 2016).
Revised: Domain adversarial neural networks (DANN) is a discriminative model to learn domain-invariant features (Ganin et al., 2016).

Original: It can be formulated as a minimax problem where the feature transformation component tries to learn a representation to confuse a following domain classiﬁcation component.
Revised: It can be formulated as a minimax problem where the feature transformation component tries to learn a representation to confuse a following domain classiﬁcation component.

Original: DANN also enjoys a nice theoretical justiﬁcation to learn a feature map to decrease the A -distance measure (Ben-David et al., 2007) between source and target domains.
Revised: DANN also enjoys a nice theoretical justiﬁcation to learn a feature map to decrease the A -distance measure (Ben-David et al., 2007) between source and target domains.

Original: Other distance measures between distributions can also be applied.
Revised: Other distance measures between distributions can also be applied.

Original: Tzeng et al.
Revised: Tzeng et al.

Original: (2014) and Long et al.
Revised: (2014) and Long et al.

Original: (2015) propose similar models where the maximum mean discrepancy (MMD)
Revised: (2015) propose similar models where the maximum mean discrepancy (MMD)

Original: (Gretton et al., 2012) between two domains are minimized.
Revised: (Gretton et al., 2012) between two domains are minimized.

Revised: Tzeng et al.

Revised: (2017) also proposed a variant of DANN, known as ADDA, where the encoders for source and target domains are not shared.

Revised: Instead of joint training objective classiﬁer and domain discriminator, ADDA runs in two stages.

Revised: At the ﬁrst stage, source encoder and classiﬁer are trained in a supervised way on the source domain.

Revised: At the second stage, both source encoder and source classiﬁer are ﬁxed, and both domain discriminator and target encoder are trained using only unsupervised samples from both domains.

Original: Very recently, Bousmalis et al.
Revised: Very recently, Bousmalis et al.

Original: (2016) propose a model where orthogonal representations that are shared between domains and unique to each domain are learned simultaneously.
Revised: (2016) propose a model where orthogonal representations that are shared between domains and unique to each domain are learned simultaneously.

Original: They achieve this goal by incorporating both similarity and difference penalties for features into the objective function.
Revised: They achieve this goal by incorporating both similarity and difference penalties for features into the objective function.

Original: Finally, domain adaptation can also be viewed as a semi-supervised learning problem by ignoring the domain shift, where source instances are treated as labeled data and target instances are unlabeled data (Dai et al., 2007;
Revised: Finally, domain adaptation can also be viewed as a semi-supervised learning problem by ignoring the domain shift, where source instances are treated as labeled data and target instances are unlabeled data (Dai et al., 2007;

Original: Rasmus et al., 2015).
Revised: Rasmus et al., 2015).

Original: DAuto improves over mSDA (Chen et al., 2012) when the dimension of feature vectors is high and as a result the covariance matrix cannot be explicitly formed.
Revised: DAuto improves over mSDA (Chen et al., 2012) when the dimension of feature vectors is high and as a result the covariance matrix cannot be explicitly formed.

Original: As we will see later, DAuto can also be naturally extended to modeling in time-series domain.
Revised: As we will see later, DAuto can also be naturally extended to modeling in time-series domain.

Original: On the other hand, compared with DANN,
Revised: On the other hand, compared with DANN,

Original: DAuto has a clear probabilistic generative model interpretation and provides us a principled way to use unlabeled data from both the source and target domains during training.
Revised: DAuto has a clear probabilistic generative model interpretation and provides us a principled way to use unlabeled data from both the source and target domains during training.

Original:  A P RINCIPLED H YBRID M ODEL FOR D OMAIN A DAPTATION
Revised:  A P RINCIPLED H YBRID M ODEL FOR D OMAIN A DAPTATION

Original: Generative models provide a principled way to use both labeled data from source domain and unlabeled data from both domains.
Revised: Generative models provide a principled way to use both labeled data from source domain and unlabeled data from both domains.

Original: In this section we start with a general probabilistic framework for domain adaptation using a principled hybrid of both generative and discriminative models.
Revised: In this section we start with a general probabilistic framework for domain adaptation using a principled hybrid of both generative and discriminative models.

Original: We then provide a concrete instantiation of our framework, and show that it leads to popular reconstructionbased domain adaptation models.
Revised: We then provide a concrete instantiation of our framework, and show that it leads to popular reconstructionbased domain adaptation models.

Original: Our derivation can also be used to explain the prevalence and success of autoencoders in both domain adaptation and semi-supervised learning (Rasmus et al., 2015; Bousmalis et al., 2016).
Revised: Our derivation can also be used to explain the prevalence and success of autoencoders in both domain adaptation and semi-supervised learning (Rasmus et al., 2015; Bousmalis et al., 2016).

Original: We end this section by proposing the DAuto model that follows our probabilistic framework and combines the minimax adversarial loss as a regularizer for domain adaptation.
Revised: We end this section by proposing the DAuto model that follows our probabilistic framework and combines the minimax adversarial loss as a regularizer for domain adaptation.

Original: 3. A P ROBABILISTIC F RAMEWORK FOR D OMAIN A DAPTATION
Revised: 3. A P ROBABILISTIC F RAMEWORK FOR D OMAIN A DAPTATION

Original: Let x ∈ R
Revised: Let x ∈ R

Original: d be an input instance and y be its target variable: y ∈ { 0 , 1 } in the classiﬁcation setting or y ∈ R in the regression setting.
Revised: d be an input instance and y be its target variable: y ∈ { 0 , 1 } in the classiﬁcation setting or y ∈ R in the regression setting.

Original: A fully generative model can be speciﬁed as follows:
Revised: A fully generative model can be speciﬁed as follows:

Original: [Equation]
Revised: [Equation]

Original: where φ is the model parameter that governs the generation process of x ; ψ is the model parameter for the conditional distribution y | x , and p ( φ, ψ ) is the prior distribution over both model parameters.
Revised: where φ is the model parameter that governs the generation process of x ; ψ is the model parameter for the conditional distribution y | x , and p ( φ, ψ ) is the prior distribution over both model parameters.

Original: At ﬁrst glance, one might ﬁnd the above generative model unsuitable under the domain adaptation setting since it implicitly assumes that the marginal distribution p ( x ; φ ) is shared among all instances from both domains.
Revised: At ﬁrst glance, one might ﬁnd the above generative model unsuitable under the domain adaptation setting since it implicitly assumes that the marginal distribution p ( x ; φ ) is shared among all instances from both domains.

Original: The key point that makes the above framework still valid under domain adaptation lies in the possible richness of the generation process parametrized by φ : although the marginal distribution may be different in the input space R d , we can still ﬁnd proper transformation f : R d → R D such that the induced marginal distributions (by f ) over both domains are similar in
Revised: The key point that makes the above framework still valid under domain adaptation lies in the possible richness of the generation process parametrized by φ : although the marginal distribution may be different in the input space R d , we can still ﬁnd proper transformation f : R d → R D such that the induced marginal distributions (by f ) over both domains are similar in

Original: R D (Fig. 1).
Revised: R D (Fig. 1).

Original: In fact, this is a necessary and implicit assumption that underlies the recent success of regularization based discriminative models (Tzeng et al., 2014;
Revised: In fact, this is a necessary and implicit assumption that underlies the recent success of regularization based discriminative models (Tzeng et al., 2014;

Original: Long et al., 2015; Ganin et al., 2016).
Revised: Long et al., 2015; Ganin et al., 2016).

Original: To better understand this, we illustrate the generative process for x in Fig.
Revised: To better understand this, we illustrate the generative process for x in Fig.

Original: Using the above joint model, if we assume that the prior distribution p ( φ, ψ ) factorizes as p ( φ, ψ )
Revised: Using the above joint model, if we assume that the prior distribution p ( φ, ψ ) factorizes as p ( φ, ψ )

Original: = p ( φ ) p ( ψ ) , then we will have:
Revised: = p ( φ ) p ( ψ ) , then we will have:

Original: [Equation]
Revised: [Equation]

Original: Note that in this case only the ﬁrst term in R.H.S. in (2) is concerned with the prediction, which means unsupervised learning on p ( x ; φ ) p ( φ ) does not help generalization on prediction.
Revised: Note that in this case only the ﬁrst term in R.H.S. in (2) is concerned with the prediction, which means unsupervised learning on p ( x ; φ ) p ( φ ) does not help generalization on prediction.

Original: In other words, the independence assumption between φ and ψ equivalently reduces our joint model over both x and y into discriminative models that only contain parameters ψ if we only care about prediction accuracy.
Revised: In other words, the independence assumption between φ and ψ equivalently reduces our joint model over both x and y into discriminative models that only contain parameters ψ if we only care about prediction accuracy.

Original: We emphasize that in this reduction, the assumption p ( φ, ψ ) = p ( ψ ) p ( φ ) is crucial, otherwise the optimal ψ ∗ will still depend on φ , i.e., ψ ∗ = ψ ∗ ( φ ) .
Revised: We emphasize that in this reduction, the assumption p ( φ, ψ ) = p ( ψ ) p ( φ ) is crucial, otherwise the optimal ψ ∗ will still depend on φ , i.e., ψ ∗ = ψ ∗ ( φ ) .

Original: On the other extreme, if we have φ = ψ , then this corresponds to having a prior p ( φ, ψ ) =
Revised: On the other extreme, if we have φ = ψ , then this corresponds to having a prior p ( φ, ψ ) =

Original: p 0 ( φ, ψ ) δ ( φ − ψ ) that constrains φ and ψ to be shared in both generative processes:
Revised: p 0 ( φ, ψ ) δ ( φ − ψ ) that constrains φ and ψ to be shared in both generative processes:

Original: [Equation]
Revised: [Equation]

Original: where p 0 is a base distribution and δ ( · ) denotes the Kronecker delta function.
Revised: where p 0 is a base distribution and δ ( · ) denotes the Kronecker delta function.

Original: It can be seen that when φ = ψ , the formulation exactly reduces to the usual MAP inference scheme in a generative model over both x and y , given parameter φ .
Revised: It can be seen that when φ = ψ , the formulation exactly reduces to the usual MAP inference scheme in a generative model over both x and y , given parameter φ .

Original: The above discussion shows that the formulation given in (1) is general enough to incorporate both discriminative and generative modelings as two extreme cases, and depending on the choice of the prior distribution over φ and ψ , we can easily recover both.
Revised: The above discussion shows that the formulation given in (1) is general enough to incorporate both discriminative and generative modelings as two extreme cases, and depending on the choice of the prior distribution over φ and ψ , we can easily recover both.

Original: In a nutshell, when φ and ψ are independent, we recover the discriminative modeling; otherwise if φ and ψ are exactly the same, we recover the generative modeling.
Revised: In a nutshell, when φ and ψ are independent, we recover the discriminative modeling; otherwise if φ and ψ are exactly the same, we recover the generative modeling.

Original: However in practice the sweet spot often lies in a mix of both models (Ng & Jordan, 2002): discriminative training usually wins at predictive accuracy, while generative modeling provides a principled way to use unlabeled data.
Revised: However in practice the sweet spot often lies in a mix of both models (Ng & Jordan, 2002): discriminative training usually wins at predictive accuracy, while generative modeling provides a principled way to use unlabeled data.

Original: To achieve the best of both worlds, now let us consider the case where φ and ψ have a common subspace, i.e., some model parameters are shared in both the generation process of x and y | x .
Revised: To achieve the best of both worlds, now let us consider the case where φ and ψ have a common subspace, i.e., some model parameters are shared in both the generation process of x and y | x .

Original: Clearly under this case the factorization assumption of the prior distribution does not hold anymore, and we cannot hope to recover a discriminative model by simply optimizing ψ .
Revised: Clearly under this case the factorization assumption of the prior distribution does not hold anymore, and we cannot hope to recover a discriminative model by simply optimizing ψ .

Original: To make our discussion concrete, think if we have p ( x ; φ )
Revised: To make our discussion concrete, think if we have p ( x ; φ )

Original: =
Revised: =

Original: g ( f ( x ; ζ ); φ \ ζ ) and p ( y | x ; ψ ) =
Revised: g ( f ( x ; ζ ); φ \ ζ ) and p ( y | x ; ψ ) =

Original: h ( f ( x ; ζ ) , y ; ψ \ ζ ) , where ζ are the shared parameters of both φ and ψ .
Revised: h ( f ( x ; ζ ) , y ; ψ \ ζ ) , where ζ are the shared parameters of both φ and ψ .

Original: Domain adaptation is possible under this setting whenever f ( · ; ζ ) forms a rich class of transformations so that unlabeled instances from both domains have similar induced marginal distribution.
Revised: Domain adaptation is possible under this setting whenever f ( · ; ζ ) forms a rich class of transformations so that unlabeled instances from both domains have similar induced marginal distribution.

Original: This is a necessary condition for domain adaptation to succeed under our framework.
Revised: This is a necessary condition for domain adaptation to succeed under our framework.

Original: As a generative model, it also allows algorithms to use unlabeled instances from both domains to optimize the marginal likelihood function p ( x ; φ ) , which also helps the predictive task p
Revised: As a generative model, it also allows algorithms to use unlabeled instances from both domains to optimize the marginal likelihood function p ( x ; φ ) , which also helps the predictive task p

Original: ( y | x ; ψ ) due to the shared component ζ .
Revised: ( y | x ; ψ ) due to the shared component ζ .

Original: 3. A N I NSTANTIATION USING K ERNEL D ENSITY E STIMATION
Revised: 3. A N I NSTANTIATION USING K ERNEL D ENSITY E STIMATION

Original: Now we use our probabilistic framework and instantiate it with proper choices of both the marginal distributions as well as the conditional distributions.
Revised: Now we use our probabilistic framework and instantiate it with proper choices of both the marginal distributions as well as the conditional distributions.

Original: On one hand, we would like to make as few assumptions as possible about the generation process of x ; on the other hand, the model should be rich enough such that even though instances from source and target domains may have different distributions in R d , the model contains ﬂexible transformation under which the induced distributions are similar enough in both domains.
Revised: On one hand, we would like to make as few assumptions as possible about the generation process of x ; on the other hand, the model should be rich enough such that even though instances from source and target domains may have different distributions in R d , the model contains ﬂexible transformation under which the induced distributions are similar enough in both domains.

Original: Taking both considerations into account, we propose to use nonparametric kernel density estimator (KDE) to model p ( x ; φ ) .
Revised: Taking both considerations into account, we propose to use nonparametric kernel density estimator (KDE) to model p ( x ; φ ) .

Original: Speciﬁcally, let K ( · ) be the chosen kernel and { x i } ni =1 be a set of unlabeled instances.
Revised: Speciﬁcally, let K ( · ) be the chosen kernel and { x i } ni =1 be a set of unlabeled instances.

Original: Our KDE for p ( x ; φ ) is given by:
Revised: Our KDE for p ( x ; φ ) is given by:

Original: [Equation]
Revised: [Equation]

Original: where w > 0 is the bandwidth and f : R d → R D and g : R D
Revised: where w > 0 is the bandwidth and f : R d → R D and g : R D

Original: → R d are two feature transformations.
Revised: → R d are two feature transformations.

Original: Our deﬁnition of KDE differs from the original one (Wassermann, 2006) by the additional parametric transformations g ◦ f applied to x , and when g ◦ f =
Revised: Our deﬁnition of KDE differs from the original one (Wassermann, 2006) by the additional parametric transformations g ◦ f applied to x , and when g ◦ f =

Original: I , the identity map, our deﬁnition reduces to the original one.
Revised: I , the identity map, our deﬁnition reduces to the original one.

Original: Note that when applied to the source and target domains separately, the original
Revised: Note that when applied to the source and target domains separately, the original

Original: KDE does not give similar density estimations if their empirical distributions are far from each other, which is exactly the case under domain adaptation.
Revised: KDE does not give similar density estimations if their empirical distributions are far from each other, which is exactly the case under domain adaptation.

Original: [Equation]
Revised: [Equation]

Original: For the conditional distribution y | x , depending on whether y ∈ R or y ∈ { 0 , 1 } , typical choices include linear regression or logistic regression.
Revised: For the conditional distribution y | x , depending on whether y ∈ R or y ∈ { 0 , 1 } , typical choices include linear regression or logistic regression.

Original: While both these two models are linear and limited, we can ﬁrst augment them with rich nonlinear transformation f applied to the input instance.
Revised: While both these two models are linear and limited, we can ﬁrst augment them with rich nonlinear transformation f applied to the input instance.

Original: Note that the transformation f , along with its parameters, are shared between both p ( x ; φ ) and p ( y | x ; ψ ) .
Revised: Note that the transformation f , along with its parameters, are shared between both p ( x ; φ ) and p ( y | x ; ψ ) .

Original: Finally, our model is completed by specifying the prior distribution as follows:
Revised: Finally, our model is completed by specifying the prior distribution as follows:

Original: [Equation]
Revised: [Equation]

Original: The δ ( · ) constrains the common parameter ζ to be shared by both p ( x ; φ ) and p ( y | x ; ψ ) .
Revised: The δ ( · ) constrains the common parameter ζ to be shared by both p ( x ; φ ) and p ( y | x ; ψ ) .

Original: The base distribution p 0 ( φ, ψ ) can be chosen as a ﬂat (possibly improper) prior, which corresponds to the usual
Revised: The base distribution p 0 ( φ, ψ ) can be chosen as a ﬂat (possibly improper) prior, which corresponds to the usual

Original: MLE criterion; or other forms of distributions that effectively introduce regularizations on both φ and ψ .
Revised: MLE criterion; or other forms of distributions that effectively introduce regularizations on both φ and ψ .

Original: 3. L EARNING BY M AXIMIZING J OINT L IKELIHOOD
Revised: 3. L EARNING BY M AXIMIZING J OINT L IKELIHOOD

Original: One of the advantages a generative model lends us is a principled way of using unlabeled data from both domains.
Revised: One of the advantages a generative model lends us is a principled way of using unlabeled data from both domains.

Original: Let { ( x i , y i ) } mi =1 be labeled data from source domain and { x j } nj =1 be unlabeled data from both domains.
Revised: Let { ( x i , y i ) } mi =1 be labeled data from source domain and { x j } nj =1 be unlabeled data from both domains.

Original: Instead of just maximizing the conditional likelihood using labeled data, we can jointly maximize both the conditional likelihood and the marginal likelihood:
Revised: Instead of just maximizing the conditional likelihood using labeled data, we can jointly maximize both the conditional likelihood and the marginal likelihood:

Original: [Equation]
Revised: [Equation]

Original: It is clear that the negative conditional log-likelihood function is exactly the cross-entropy loss between the true label
Revised: It is clear that the negative conditional log-likelihood function is exactly the cross-entropy loss between the true label

Original: y i and its predicted counterpart.
Revised: y i and its predicted counterpart.

Original: For the negative marginal likelihood, if we choose a Gaussian kernel and plug in our KDE estimator given in (4), we have:
Revised: For the negative marginal likelihood, if we choose a Gaussian kernel and plug in our KDE estimator given in (4), we have:

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 only depends on the bandwidth and c is a constant that does not depend on x j .
Revised: where λ > 0 only depends on the bandwidth and c is a constant that does not depend on x j .

Original: The upper bound becomes tight as w → 0 .
Revised: The upper bound becomes tight as w → 0 .

Original: Note that in the above derivation we omit the prior distribution p ( φ, ψ ) =
Revised: Note that in the above derivation we omit the prior distribution p ( φ, ψ ) =

Original: p 0 ( φ, ψ ) δ ( φ ( ζ ) − ψ ( ζ )) by assuming that p 0 is a ﬂat constant distribution.
Revised: p 0 ( φ, ψ ) δ ( φ ( ζ ) − ψ ( ζ )) by assuming that p 0 is a ﬂat constant distribution.

Original: If other design choice is made, e.g., a Gaussian over φ ( ψ ) , then there will be a corresponding (cid:96) 2 regularization term for the model parameters φ ( ψ ) .
Revised: If other design choice is made, e.g., a Gaussian over φ ( ψ ) , then there will be a corresponding (cid:96) 2 regularization term for the model parameters φ ( ψ ) .

Original: Putting all together, maximizing a combination of conditional and marginal likelihoods correspond to the following unconstrained minimization problem:
Revised: Putting all together, maximizing a combination of conditional and marginal likelihoods correspond to the following unconstrained minimization problem:

Original: [Equation]
Revised: [Equation]

Original: Remark .
Revised: Remark .

Original: The maximum of joint likelihood estimator is also the maximum-a-posteriori estimator, leading to a fully probabilistic interpretation of (7).
Revised: The maximum of joint likelihood estimator is also the maximum-a-posteriori estimator, leading to a fully probabilistic interpretation of (7).

Original: The second term in the objective function has an interesting interpretation: it essentially measures the reconstruction error after a transformation by g ◦ f .
Revised: The second term in the objective function has an interesting interpretation: it essentially measures the reconstruction error after a transformation by g ◦ f .

Original: Based on (6), if we interpret f as an encoder and g as the corresponding decoder, then minimizing the reconstruction loss from an autoencoder exactly corresponds to maximizing a lower bound of the marginal probability distribution p ( x ) , where p ( x ) is given by our kernel density estimation.
Revised: Based on (6), if we interpret f as an encoder and g as the corresponding decoder, then minimizing the reconstruction loss from an autoencoder exactly corresponds to maximizing a lower bound of the marginal probability distribution p ( x ) , where p ( x ) is given by our kernel density estimation.

Original: Furthermore, the squared (cid:96) 2 measure in the reconstruction error is not restricted: for example, if instead of a Gaussian kernel we chose the Laplacian kernel, then we would have an (cid:96)measure of the reconstruction loss in (7).
Revised: Furthermore, the squared (cid:96) 2 measure in the reconstruction error is not restricted: for example, if instead of a Gaussian kernel we chose the Laplacian kernel, then we would have an (cid:96)measure of the reconstruction loss in (7).

Original: This interpretation may also be used to explain the practical success of autoencoders in semi-supervised learning (Rasmus et al., 2015).
Revised: This interpretation may also be used to explain the practical success of autoencoders in semi-supervised learning (Rasmus et al., 2015).

Original: [Equation]
Revised: [Equation]

Original: The model contains three different objectives: the domain classiﬁcation loss, reconstruction loss as well as the label prediction loss.
Revised: The model contains three different objectives: the domain classiﬁcation loss, reconstruction loss as well as the label prediction loss.

Original: In the inference phase, only f and h will be used.
Revised: In the inference phase, only f and h will be used.

Original: 3. A C ONCRETE M ODEL (DA UTO )
Revised: 3. A C ONCRETE M ODEL (DA UTO )

Original: We use neural networks as ﬂexible function approximators for our desired transformations f and g .
Revised: We use neural networks as ﬂexible function approximators for our desired transformations f and g .

Original: Speciﬁcally, we use fully-connected neural networks to parametrize f and g and softmax function to parametrize h .
Revised: Speciﬁcally, we use fully-connected neural networks to parametrize f and g and softmax function to parametrize h .

Original: If y ∈ R , we can simply change the softmax function to be an afﬁne function as the output.
Revised: If y ∈ R , we can simply change the softmax function to be an afﬁne function as the output.

Original: For the simplicity of discussion, assume we only use a one-layer fully connected network to represent f and g : f ( x ) =
Revised: For the simplicity of discussion, assume we only use a one-layer fully connected network to represent f and g : f ( x ) =

Original: σ ( W 1 x ) and g ( z )
Revised: σ ( W 1 x ) and g ( z )

Original: =
Revised: =

Original: σ ( W 2 z ) , where W 1 ∈ R D
Revised: σ ( W 2 z ) , where W 1 ∈ R D

Original: ×
Revised: ×

Original: d , W 2 ∈ R d × D and σ ( · ) is an element-wise nonlinear activation function, e.g., the rectify linear unit.
Revised: d , W 2 ∈ R d × D and σ ( · ) is an element-wise nonlinear activation function, e.g., the rectify linear unit.

Original: For ease of notation, we also omit the translation terms from above afﬁne transformations.
Revised: For ease of notation, we also omit the translation terms from above afﬁne transformations.

Original: Let h ( z ) =
Revised: Let h ( z ) =

Original: softmax ( W h z ) be the softmax layer to compute the conditional probability of class assignment.
Revised: softmax ( W h z ) be the softmax layer to compute the conditional probability of class assignment.

Original: Although our model has the capacity to learn the shared transformation f under which unlabeled data from both domains have similar marginal distributions, the objective function discussed so far does not necessarily induce such a transformation.
Revised: Although our model has the capacity to learn the shared transformation f under which unlabeled data from both domains have similar marginal distributions, the objective function discussed so far does not necessarily induce such a transformation.

Original: For the purpose of domain adaptation, it is necessary to add a regularizer that enforces this constraint.
Revised: For the purpose of domain adaptation, it is necessary to add a regularizer that enforces this constraint.

Original: One popular and effective choice is the A -distance introduced by (Kifer et al., 2004; Ben-David et al., 2007; 2010).
Revised: One popular and effective choice is the A -distance introduced by (Kifer et al., 2004; Ben-David et al., 2007; 2010).

Original: It can be shown that the A -distance can be approximated by the binary classiﬁcation error of the domain classiﬁer that discriminates instances from the source or the target domain (Ben-David et al., 2007).
Revised: It can be shown that the A -distance can be approximated by the binary classiﬁcation error of the domain classiﬁer that discriminates instances from the source or the target domain (Ben-David et al., 2007).

Original: The intuition here is: given a ﬁxed class of binary labeling functions, if there exists a function that is easy to tell instances in the source domain from those in the target domain, then the distance between these two domains is large.
Revised: The intuition here is: given a ﬁxed class of binary labeling functions, if there exists a function that is easy to tell instances in the source domain from those in the target domain, then the distance between these two domains is large.

Original: On the other hand, if the set of labeling functions are confused by such task, then we can think the distance between these two domains is small.
Revised: On the other hand, if the set of labeling functions are confused by such task, then we can think the distance between these two domains is small.

Original: Following DANN, here we take the same approach: let ˜ h
Revised: Following DANN, here we take the same approach: let ˜ h

Original: = softmax ( W d z ) be the domain classiﬁer where z
Revised: = softmax ( W d z ) be the domain classiﬁer where z

Original: = σ ( W 1 x ) is the shared representation constructed by encoder f .
Revised: = σ ( W 1 x ) is the shared representation constructed by encoder f .

Original: The regularizer takes the form as a convex surrogate loss for the binary 0error.
Revised: The regularizer takes the form as a convex surrogate loss for the binary 0error.

Original: A common choice is the cross-entropy loss.
Revised: A common choice is the cross-entropy loss.

Original: Putting all together, the optimization problem of our joint model is given by:
Revised: Putting all together, the optimization problem of our joint model is given by:

Original: [Equation]
Revised: [Equation]

Original: where L y ( · , · ) is the prediction loss, L r ( · ) is the reconstruction loss and L d ( · ) is the domain classiﬁ- cation loss.
Revised: where L y ( · , · ) is the prediction loss, L r ( · ) is the reconstruction loss and L d ( · ) is the domain classiﬁ- cation loss.

Original: We illustrate the model architecture in Fig.
Revised: We illustrate the model architecture in Fig.

Original: As we show above, the cross-entropy loss for the learning task and the reconstruction loss are essentially the negative joint log-likelihoods of both labeled instances and unlabeled instances.
Revised: As we show above, the cross-entropy loss for the learning task and the reconstruction loss are essentially the negative joint log-likelihoods of both labeled instances and unlabeled instances.

Original: The domain classiﬁcation loss works as a regularizer to incorporate our prior knowledge that the encoder f should form an invariant transformation.
Revised: The domain classiﬁcation loss works as a regularizer to incorporate our prior knowledge that the encoder f should form an invariant transformation.

Original: As a result, DAuto is designed to achieve the following three objectives simultaneously in a uniﬁed framework: 1).
Revised: As a result, DAuto is designed to achieve the following three objectives simultaneously in a uniﬁed framework: 1).

Original: It learns representations that are informative for the main learning task in the source domain. 2).
Revised: It learns representations that are informative for the main learning task in the source domain. 2).

Original: It learns robust representations under reconstruction loss. 3).
Revised: It learns robust representations under reconstruction loss. 3).

Original: It learns domain-invariant features that are indistinguishable between the source and the target domains.
Revised: It learns domain-invariant features that are indistinguishable between the source and the target domains.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We ﬁrst evaluate DAuto on synthetic experiments with MNIST, and then compare it with state-of-theart models, including mSDA, the Ladder network and DANN.
Revised: We ﬁrst evaluate DAuto on synthetic experiments with MNIST, and then compare it with state-of-theart models, including mSDA, the Ladder network, DANN and ADDA.

Original: We report experimental results on the Amazon benchmark dataset and a large-scale time-series dataset for speech recognition.
Revised: We report experimental results on the Amazon benchmark dataset, three digit datasets (MNIST, SVHN and USPS) and a large-scale time-series dataset for speech recognition.

Original: 4. D ATASETS AND E XPERIMENTAL S ETUP
Revised: 4. D ATASETS AND E XPERIMENTAL S ETUP

Original: Synthetic experiment with MNIST .
Revised: Synthetic experiment with MNIST .

Original: The experiment contains 4 tasks: each task is a binary classiﬁcation problem on judging whether the given image is a speciﬁc number or not.
Revised: The experiment contains 4 tasks: each task is a binary classiﬁcation problem on judging whether the given image is a speciﬁc number or not.

Original: We choose 4 digits for these tasks: 3, 7, 8 and 9, because 9 and 7 are similar in many handwritten images while 3 and 9 (7 and 8) are quite different.
Revised: We choose 4 digits for these tasks: 3, 7, 8 and 9, because 9 and 7 are similar in many handwritten images while 3 and 9 (7 and 8) are quite different.

Original: Clearly domain adaptation is not always possible.
Revised: Clearly domain adaptation is not always possible.

Original: We would like to verify that DAuto succeeds when two domains are close to each other, and also show a failure case when domains are sufﬁciently different.
Revised: We would like to verify that DAuto succeeds when two domains are close to each other, and also show a failure case when domains are sufﬁciently different.

Original: To do so, for each task on recognizing digit i ∈ { 3 , 7 , 8 , 9 } , we sample 1000 images from the training set, of which 500 are digit i and the others are digits not in { 3 , 7 , 8 , 9 } ; we sample ∼ 1,500 images from the original test set, of which ∼ 750 images are digit i and the others are digits not in { 3 , 7 , 8 , 9 } .
Revised: To do so, for each task on recognizing digit i ∈ { 3 , 7 , 8 , 9 } , we sample 1000 images from the training set, of which 500 are digit i and the others are digits not in { 3 , 7 , 8 , 9 } ; we sample ∼ 1,500 images from the original test set, of which ∼ 750 images are digit i and the others are digits not in { 3 , 7 , 8 , 9 } .

Original: There are 16 pairs of experiments altogether for each possible pair of i, j ∈ { 3 , 7 , 8 , 9 } as source and target domains.
Revised: There are 16 pairs of experiments altogether for each possible pair of i, j ∈ { 3 , 7 , 8 , 9 } as source and target domains.

Original: We design a well-controlled experiment to compare DAuto with a standard MLP and DANN: all algorithms share the same network structure.
Revised: We design a well-controlled experiment to compare DAuto with a standard multilayer perceptron (MLP) and DANN: all algorithms share the same network structure.

Original: Also, we apply the same training procedure to all algorithms so that the difference in performance can only be explained by the additional domain regularizer as well as the reconstruction loss in DAuto.
Revised: Also, we apply the same training procedure to all algorithms so that the difference in performance can only be explained by the additional domain regularizer as well as the reconstruction loss in DAuto.

Original: Multi-class Classiﬁcation .
Revised: Multi-class Classiﬁcation .

Original: DAuto can be easily applied in multi-class classiﬁcation as well.
Revised: DAuto can be easily applied in multi-class classiﬁcation as well.

Original: To see this, we test our model on MNIST, USPS and SVHN, all of which contain images of 10 digits.
Revised: To see this, we test our model on MNIST, USPS and SVHN, all of which contain images of 10 digits.

Original: MNIST contains 60,000/10,000 train/test instances; USPS contains 7,291/2,007 train/test instances, and SVHN contains 73,257/26,032 train/test instances.
Revised: MNIST contains 60,000/10,000 train/test instances; USPS contains 7,291/2,007 train/test instances, and SVHN contains 73,257/26,032 train/test instances.

Original: Before training, we preprocess all the instances into gray scale single-channel images of size 16x16, so that they can be used by the same network.
Revised: Before training, we preprocess all the instances into gray scale single-channel images of size 16x16, so that they can be used by the same network.

Original: Again, we perform a controlled experiment to ensure a fair comparison between the evaluated models.
Revised: Again, we perform a controlled experiment to ensure a fair comparison between the evaluated models.

Original: The network structures are exactly the same for all the approaches: 2 hidden layers with 1024, 512 units, following by a softmax output layer with 10 units.
Revised: The network structures are exactly the same for all the approaches: 2 hidden layers with 1024, 512 units, following by a softmax output layer with 10 units.

Revised: The domain discriminators in all the models are chosen to be logistic regression.

Original: During training, both batch size (400) and dropout rate (0.3) are also ﬁxed to be same among all the experiments.
Revised: During training, both batch size (400) and dropout rate (0.3) are also ﬁxed to be same among all the experiments.

Original: Hence again, the difference in performance can only be explained by the different objective functions of different models.
Revised: Hence again, the difference in performance can only be explained by the different objective functions of different models.

Original: Sentiment Analysis .
Revised: Sentiment Analysis .

Original: The Amazon dataset consists of reviews of products on Amazon (Blitzer et al., 2007).
Revised: The Amazon dataset consists of reviews of products on Amazon (Blitzer et al., 2007).

Original: The task is to predict the polarity of a text review, i.e., whether the review for a speciﬁc product is positive or negative.
Revised: The task is to predict the polarity of a text review, i.e., whether the review for a speciﬁc product is positive or negative.

Original: The dataset contains text reviews for the following four categories: books (B), DVDs (D), electronics (E), and kitchen appliances (K).
Revised: The dataset contains text reviews for the following four categories: books (B), DVDs (D), electronics (E), and kitchen appliances (K).

Original: Each of the product containstext reviews as training data, and 4465 (B), 3586 (D), 5681 (E), 5945 (K) reviews as test data.
Revised: Each of the product containstext reviews as training data, and 4465 (B), 3586 (D), 5681 (E), 5945 (K) reviews as test data.

Original: Each text review is described by a feature vector of 5000 dimensions, where each dimension correspond to a word in the dictionary.
Revised: Each text review is described by a feature vector of 5000 dimensions, where each dimension correspond to a word in the dictionary.

Original: The dataset is a benchmark data that has been frequently used for the purpose of sentiment analysis (Blitzer et al., 2006; 2007;
Revised: The dataset is a benchmark data that has been frequently used for the purpose of sentiment analysis (Blitzer et al., 2006; 2007;

Original: Chen et al., 2012; Ajakan et al., 2014;
Revised: Chen et al., 2012; Ajakan et al., 2014;

Original: Ganin & Lempitsky, 2015; Long et al., 2016).
Revised: Ganin & Lempitsky, 2015; Long et al., 2016).

Original: For each source-target pair, we train the corresponding models completely on labeled source instances with access to unlabeled target instances.
Revised: For each source-target pair, we train the corresponding models completely on labeled source instances with access to unlabeled target instances.

Original: We use the classiﬁcation accuracy in the target domain as our main metric.
Revised: We use the classiﬁcation accuracy in the target domain as our main metric.

Original: Speech recognition .
Revised: Speech recognition .

Original: DAuto can be naturally extended to time-series modeling.
Revised: DAuto can be naturally extended to time-series modeling.

Original: In this experiment we apply DAuto to speech recognition, where a recurrent neural network is trained to ingest speech spectrograms and generate text transcripts.
Revised: In this experiment we apply DAuto to speech recognition, where a recurrent neural network is trained to ingest speech spectrograms and generate text transcripts.

Original: The model we use is a variant of DeepSpeech 2 (Amodei et al., 2015), which is composed of 1 convolution layer followed by 3 stacked bidirectional LSTM layers, and one fully connected layer before a softmax layer.
Revised: The model we use is a variant of DeepSpeech 2 (Amodei et al., 2015), which is composed of 1 convolution layer followed by 3 stacked bidirectional LSTM layers, and one fully connected layer before a softmax layer.

Original: At each time step, the input to the network is a log spectrogram feature, and the output is a character or blank.
Revised: At each time step, the input to the network is a log spectrogram feature, and the output is a character or blank.

Original: The network is trained end-to-end using the connectionist temporal classiﬁcation (CTC) loss (Graves et al., 2006), which is the negative log-likelihood of training utterances.
Revised: The network is trained end-to-end using the connectionist temporal classiﬁcation (CTC) loss (Graves et al., 2006), which is the negative log-likelihood of training utterances.

Original: To extend DAuto in sequential models, besides the global CTC loss, we regularize it at each time step with both the reconstruction loss from the autoencoder as well as the adversarial loss from the domain classiﬁer.
Revised: To extend DAuto in sequential models, besides the global CTC loss, we regularize it at each time step with both the reconstruction loss from the autoencoder as well as the adversarial loss from the domain classiﬁer.

Original: We evaluate DAuto and compare it with other algorithms for an adaptation task across three different accented datasets, of which one is recorded from native English speakers, and the other two are recorded from speakers with Mandarin and Indian accents, respectively.
Revised: We evaluate DAuto and compare it with other algorithms for an adaptation task across three different accented datasets, of which one is recorded from native English speakers, and the other two are recorded from speakers with Mandarin and Indian accents, respectively.

Original: Each dataset contains ∼ 33 hours of labeled audio data from ∼ 25,000 user utterances, and we randomly sample 80 percent of them as training set and the rest are used as test set.
Revised: Each dataset contains ∼ 33 hours of labeled audio data from ∼ 25,000 user utterances, and we randomly sample 80 percent of them as training set and the rest are used as test set.

Original: We compare DAuto with the following methods: 1.
Revised: We compare DAuto with the following methods: 1.

Original: No-Adapt .
Revised: No-Adapt .

Original: This is the baseline model which ignores the possible shifts between domains.
Revised: This is the baseline model which ignores the possible shifts between domains.

Original: In order to make the comparison as fair as possible, in all our experiments DAuto shares the same prediction model with No-Adapt. 2.
Revised: In order to make the comparison as fair as possible, in all our experiments DAuto shares the same prediction model with No-Adapt. 2.

Original: mSDA .
Revised: mSDA .

Original: mSDA pretrains all the unlabeled instances from both the source and the target domains to build a feature map for the input space.
Revised: mSDA pretrains all the unlabeled instances from both the source and the target domains to build a feature map for the input space.

Original: The constructed representations from mSDA are used to train a SVM classiﬁer as suggested in the original paper (Chen et al., 2012).
Revised: The constructed representations from mSDA are used to train a SVM classiﬁer as suggested in the original paper (Chen et al., 2012).

Original: In all the experiments, we set the corruption level to be 0.5 in training mSDA, and stack the same number of layers of autoencoders as in DAuto. 3.
Revised: In all the experiments, we set the corruption level to be 0.5 in training mSDA, and stack the same number of layers of autoencoders as in DAuto. 3.

Original: Ladder Network (Ladder) .
Revised: Ladder Network (Ladder) .

Original: The Ladder network (Rasmus et al., 2015) is a novel structure aiming for semi-supervised learning.
Revised: The Ladder network (Rasmus et al., 2015) is a novel structure aiming for semi-supervised learning.

Original: It is a hierarchical denoising autoencoder where reconstruction errors between each pair of hidden layers are incorporated into the objective function.
Revised: It is a hierarchical denoising autoencoder where reconstruction errors between each pair of hidden layers are incorporated into the objective function.

Revised: DANN and 5.

Original: DANN .
Revised: ADDA .

Original: Again, we use exactly the same inference structure for DANN  as in No-Adapt, Ladder, and DAuto.
Revised: Again, we use exactly the same inference structure for DANN and ADDA as in No-Adapt, Ladder, and DAuto.

Original: For all the experiments, we use early-stop to avoid overﬁtting.
Revised: For all the experiments, we use early-stop to avoid overﬁtting.

Original: We implement all the models and ensure that all the preprocessing for data are the same for all the algorithms, so that the differences in experimental results can only be explained by the differences in models themselves.
Revised: We implement all the models and ensure that all the preprocessing for data are the same for all the algorithms, so that the differences in experimental results can only be explained by the differences in models themselves.

Original: We defer detailed description about models used in each experiment to the supplementary material.
Revised: We defer detailed description about models used in each experiment to the supplementary material.

Original: 4. R ESULTS AND A NALYSIS
Revised: 4. R ESULTS AND A NALYSIS

Original: MNIST .
Revised: MNIST .

Original: We list the classiﬁcation accuracies on 16 pairs of tasks using No-Adapt, DANN and
Revised: We list the classiﬁcation accuracies on 16 pairs of tasks using No-Adapt, DANN and

Original: DAuto in Table 1.
Revised: DAuto in Table 1.

Original: Besides the 12 pairs of tasks under the domain adaptation setting, we also showadditional tasks where both the training and test sets are from the same domain.
Revised: Besides the 12 pairs of tasks under the domain adaptation setting, we also showadditional tasks where both the training and test sets are from the same domain.

Original: The scores from these 4 tasks can be used as empirical upper bounds to compare with the performance of domainadaptation algorithms.
Revised: The scores from these 4 tasks can be used as empirical upper bounds to compare with the performance of domain

Original: DAuto signiﬁcantly improves over the No-Adapt baseline in 10 out of the total 12 possible pairs, showing that it indeeds has the desired capability for domain adaptation.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: adaptation algorithms.

Revised: DAuto signiﬁcantly improves over the No-Adapt baseline in 10 out of the total 12 possible pairs, showing that it indeeds has the desired capability for domain adaptation.

Original: On the other hand, we would also like to highlight that not all domain adaptation tasks are successful: the prediction accuracies of all the three algorithms on task 7 → 8 and 7 → 3 are only marginally above random guesses (0.5).
Revised: On the other hand, we would also like to highlight that not all domain adaptation tasks are successful: the prediction accuracies of all the three algorithms on task 7 → 8 and 7 → 3 are only marginally above random guesses (0.5).

Original: When the digits are similar, we observe very successful domain transfer using DAuto even the algorithm does not see any instances from the target category during training.
Revised: When the digits are similar, we observe very successful domain transfer using DAuto even the algorithm does not see any instances from the target category during training.

Original: To qualitatively study both the successful and failure cases, we project both representations with and without DAuto adaptation onto 2 dimensional space using PCA, shown in Fig.
Revised: To qualitatively study both the successful and failure cases, we project both representations with and without DAuto adaptation onto 2 dimensional space using PCA, shown in Fig.

Original: Several interesting observations can be made from Fig.
Revised: Several interesting observations can be made from Fig.

Original: 3: when domain adaptation is successful (Fig. 3d, 3h), the principal directions of learned representations from both domains are well-aligned with each other; on the other hand, when adaptation fails (Fig. 3b), representations do not share the same principal directions.
Revised: 3: when domain adaptation is successful (Fig. 3g, 3f), the principal directions of learned representations from both domains are well-aligned with each other; on the other hand, when adaptation fails (Fig. 3e), representations do not share the same principal directions.

Original: As a special case, when the source and target domains share the same distribution (Fig. 3f),
Revised: As a special case, when the source and target domains share the same distribution (Fig. 3h),

Original: DAuto still works and will not degrade.
Revised: DAuto still works and will not degrade.

Original: Multi-class Classiﬁcation .
Revised: Multi-class Classiﬁcation .

Original: The results on multi-class classiﬁcation of digits are shown in Table 2, where we highlight the successful domain adaptations using green colors and failure cases using red colors.
Revised: The results on multi-class classiﬁcation of digits are shown in Table 2, where we highlight the successful domain adaptations using green colors and failure cases using red colors.

Original: The datasets in row correspond to the source domains and those in columns correspond to the target domains.
Revised: The datasets in row correspond to the source domains and those in columns correspond to the target domains.

Original: Both DANN and DAuto  contain one failure case (USPS → SVHN and
Revised: DANN, ADDA and DAuto all contain one failure case (USPS → SVHN,

Original:  MNIST → USPS), which may be explained by the intrinsic difference between the SVHN datasets and the other two.
Revised: USPS → SVHN and MNIST → USPS), which may be explained by the intrinsic difference between the SVHN datasets and the other two.

Original: However, on those three datasets, whenever both DANN and DAuto succeed in domain adaptation, DAuto usually outperforms DANN by around 2 percent accuracy.
Revised: However, on those three datasets, whenever both DANN and DAuto succeed in domain adaptation, DAuto usually outperforms DANN by around 2 percent accuracy.

Revised: On the other hand, ADDA achieves far better result on MNIST → USPS than both DANN and DAuto, while it also performs worse than DAuto on SVHN → MNIST and USPS → MNIST.

Revised: Note that in this experiment all the methods share exactly the same experimental protocol, hence the

Revised: [Table]

Revised: [Figure]

Original: Note that in this experiment both DANN and DAuto share exactly the same experimental protocol, hence the difference can only be explained by their different objective functions.
Revised: c y [Figure] [Table] difference can only be explained by their different objective functions and model designs.

Original: In other words, the adaptation can beneﬁt from the reconstruction error from autoencoders, which works as an unsupervised regularizer.
Revised: In other words, the adaptation can beneﬁt from the reconstruction error from autoencoders, which works as an unsupervised regularizer.

Original: Amazon .
Revised: Amazon .

Original: To show the effectiveness of different domain adaptation algorithms when labeled instances are scarce, we evaluate the ﬁve algorithms on the 16 tasks by gradually increasing the size of the labeled training instances, but still use the whole test dataset to measure the performance.
Revised: To show the effectiveness of different domain adaptation algorithms when labeled instances are scarce, we evaluate the ﬁve algorithms on the 16 tasks by gradually increasing the size of the labeled training instances, but still use the whole test dataset to measure the performance.

Original: More speciﬁcally, we use 0.2, 0.5, 0.8 and 1.0 fraction of the available labeled instances from source domain during training.
Revised: More speciﬁcally, we use 0.2, 0.5, 0.8 and 1.0 fraction of the available labeled instances from source domain during training.

Original: A successful domain adaptation algorithm should be able to take advantage of the unlabeled instances from the target domain to help generalization even when the amount of labeled instances available is small.
Revised: A successful domain adaptation algorithm should be able to take advantage of the unlabeled instances from the target domain to help generalization even when the amount of labeled instances available is small.

Original: We plot the results in Fig.
Revised: We plot the results in Fig.

Original: 4: all the domain adaptation algorithms  are able to use the unlabeled instances from the target domain to help generalization.
Revised: 4: all the domain adaptation algorithms [Table] are able to use the unlabeled instances from the target domain to help generalization.

Original: However, the differences between the baseline MLP and DAuto are getting smaller with the increase of the size of training data.
Revised: However, the differences between the baseline MLP and DAuto are getting smaller with the increase of the size of training data.

Original: This phenomenon supports our probabilistic framework showing that DAuto can effectively use the unlabeled data.
Revised: This phenomenon supports our probabilistic framework showing that DAuto can effectively use the unlabeled data.

Original: We report the detailed numeric scores achieved by each algorithm and their statistical signiﬁcance test in the supplementary material.
Revised: We report the detailed numeric scores achieved by each algorithm and their statistical signiﬁcance test in the supplementary material.

Original: In summary, DAuto outperforms all the other competitors on 12 out of the 16 tasks, which is consistently better than its competitors.
Revised: In summary, DAuto outperforms all the other competitors on 12 out of the 16 tasks, which is consistently better than its competitors.

Original: Speech Recognition .
Revised: Speech Recognition .

Original: To avoid possible external error during the decoding process, which is usually not stable on noisy data, we directly report the CTC loss obtained from different algorithms.
Revised: To avoid possible external error during the decoding process, which is usually not stable on noisy data, we directly report the CTC loss obtained from different algorithms.

Original: We have 3 × 3 pairs of domain adaptation tasks, where each source/target domain ranges from speech with { IN, CN, US } accents.
Revised: We have 3 × 3 pairs of domain adaptation tasks, where each source/target domain ranges from speech with { IN, CN, US } accents.

Original: We show the results in Table 3.
Revised: We show the results in Table 3.

Original: Again, we observe decreases in performance when applying domain adaptation algorithms: the transfer between speech with Indian accents and the other two fails.
Revised: Again, we observe decreases in performance when applying domain adaptation algorithms: the transfer between speech with Indian accents and the other two fails.

Original: However, when two domains are similar (US vs CN) where adaptation is possible,
Revised: However, when two domains are similar (US vs CN) where adaptation is possible,

Original: DAuto consistently outperforms DANN.
Revised: DAuto consistently outperforms DANN.

Original: We want to bring readers’ attention that both DANN and
Revised: We want to bring readers’ attention that both DANN and

Original: DAuto improve over baseline method when training with unlabeled instances from the same domain (diagonal of the table).
Revised: DAuto improve over baseline method when training with unlabeled instances from the same domain (diagonal of the table).

Original: This means both DANN and DAuto have the effect of semi-supervised learning algorithms: when the source and the target domains are indeed same, both DANN and
Revised: This means both DANN and DAuto have the effect of semi-supervised learning algorithms: when the source and the target domains are indeed same, both DANN and

Original: DAuto beneﬁt from using unlabeled instances.
Revised: DAuto beneﬁt from using unlabeled instances.

Original: From the generative interpretation of our probabilistic framework, DAuto achieves this goal by maximizing the marginal probability of unlabeled instances, and due to the shared component, this further helps training the discriminative model [Table] l [Table]
Revised: From the generative interpretation of our probabilistic framework, DAuto achieves this goal by maximizing the marginal probability of unlabeled instances, and due to the shared component, this further helps training the discriminative model.

Original: i c a t

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a probabilistic framework that incorporates both generative and discriminative modeling in a principled way, which also helps us to interpolate between generative and discriminative extremes through a speciﬁc choice of prior distribution where a subset of model parameters is shared.
Revised: We propose a probabilistic framework that incorporates both generative and discriminative modeling in a principled way, which also helps us to interpolate between generative and discriminative extremes through a speciﬁc choice of prior distribution where a subset of model parameters is shared.

Original: The instantiated model, DAuto, allows us to use unlabeled instances from both domains in a principled way.
Revised: The instantiated model, DAuto, allows us to use unlabeled instances from both domains in a principled way.

Original: This instantiation also shows that the empirical success of autoencoders in semi-supervised learning and domain adaptation can be explained as maximizing the marginal log-likelihoods of unlabeled data, where kernel density estimators are used to model the marginal distributions.
Revised: This instantiation also shows that the empirical success of autoencoders in semi-supervised learning and domain adaptation can be explained as maximizing the marginal log-likelihoods of unlabeled data, where kernel density estimators are used to model the marginal distributions.

Original: This provides the ﬁrst probabilistic justiﬁcation for joint training with autoencoders in practice.
Revised: This provides the ﬁrst probabilistic justiﬁcation for joint training with autoencoders in practice.

Original: Experimentally we show that DAuto can be successfully applied to domain adaptation problems, and has a natural extension to time series as well.
Revised: Experimentally we show that DAuto can be successfully applied to domain adaptation problems, and has a natural extension to time series as well.

Original: A M ORE DETAILS ABOUT EXPERIMENTS A. MNIST S YNTHETIC E XPERIMENTS
Revised: A M ORE DETAILS ABOUT EXPERIMENTS A. MNIST S YNTHETIC E XPERIMENTS

Original: The networks we used in this experiment contains 3 hidden layers, each contain 500, 200 andhidden units, respectively.
Revised: The networks we used in this experiment contains 3 hidden layers, each contain 500, 200 andhidden units, respectively.

Original: The input layer contains 28 × 28 = 784 units, and the output layer is a single unit parametrized by a logistic regression model.
Revised: The input layer contains 28 × 28 = 784 units, and the output layer is a single unit parametrized by a logistic regression model.

Original: Except the output layer, all the hidden layers use ReLU as the nonlinear activation function.
Revised: Except the output layer, all the hidden layers use ReLU as the nonlinear activation function.

Original: The same network structure is used for both DANN and
Revised: The same network structure is used for both DANN and

Original: DAuto.
Revised: DAuto.

Original: During training, we use AdaDelta as the optimization algorithm for all the models.
Revised: During training, we use AdaDelta as the optimization algorithm for all the models.

Original: For DAuto, both hyperparameters λ and µ are chosen from the following range: λ, µ ∈ { 10 − 8 , 10 − 7 , . . .
Revised: For DAuto, both hyperparameters λ and µ are chosen from the following range: λ, µ ∈ { 10 − 8 , 10 − 7 , . . .

Original: , 10 2 } , where we further partition the target domain set as development set and the held-out test set, combined with early stopping for model selection.
Revised: , 10 2 } , where we further partition the target domain set as development set and the held-out test set, combined with early stopping for model selection.

Original: For all the experiments, we ﬁx the learning rate to be 1.0.
Revised: For all the experiments, we ﬁx the learning rate to be 1.0.

Original: A. A MAZON B ENCHMARK D ATASET
Revised: A. A MAZON B ENCHMARK D ATASET

Original: MLP, Ladder, DANN and DAuto all share the same network structure: the input layer contains 5, units, followed by a fully-connected hidden layer with 500 units and the output layer, which is a logistic regression model.
Revised: MLP, Ladder, DANN and DAuto all share the same network structure: the input layer contains 5, units, followed by a fully-connected hidden layer with 500 units and the output layer, which is a logistic regression model.

Original: Correspondingly, mSDA pretrains a two-layer stacked auto-encoder, hence the feature representation pretrained by mSDA in this experiment has 10,000 dimensions.
Revised: Correspondingly, mSDA pretrains a two-layer stacked auto-encoder, hence the feature representation pretrained by mSDA in this experiment has 10,000 dimensions.

Original: Ladder and DAuto also use unlabeled instances to pretrain model parameters in an purely unsupervised way.
Revised: Ladder and DAuto also use unlabeled instances to pretrain model parameters in an purely unsupervised way.

Original: Again, for all the neural network based models, we use ReLU as the nonlinear activation function and use AdaDelta as the optimization algorithm during training.
Revised: Again, for all the neural network based models, we use ReLU as the nonlinear activation function and use AdaDelta as the optimization algorithm during training.

Original: For DAuto, both hyperparameters λ, µ ∈ { 10 − 4 , . . .
Revised: For DAuto, both hyperparameters λ, µ ∈ { 10 − 4 , . . .

Original: Again, we ﬁx the learning rate to be 1.0.
Revised: Again, we ﬁx the learning rate to be 1.0.

Original: We report the classiﬁcation accuracies on the test data of the 16 pairs of tasks to have a thorough comparisons among the 5 models in Table 4.
Revised: We report the classiﬁcation accuracies on the test data of the 16 pairs of tasks to have a thorough comparisons among the 5 models in Table 4.

Original: DAuto outperforms all the other competitors on 12 out of 16 tasks, whereas DANN achieves the best test set accuracy on D → D ; Ladder scores highest on
Revised: DAuto outperforms all the other competitors on 12 out of 16 tasks, whereas DANN achieves the best test set accuracy on D → D ; Ladder scores highest on

Original: D → E and D → K ; mSDA performs the best on E → B and E → D .
Revised: D → E and D → K ; mSDA performs the best on E → B and E → D .

Original: We emphasize that DAuto performs consistently better than all the other competitors.
Revised: We emphasize that DAuto performs consistently better than all the other competitors.

Original: Note that the only difference between
Revised: Note that the only difference between

Original: DANN and DAuto is the autoencoder regularizer that forces the feature learning part to learn robust features, we conclude that DAuto successfully helps to build robust representations.
Revised: DANN and DAuto is the autoencoder regularizer that forces the feature learning part to learn robust features, we conclude that DAuto successfully helps to build robust representations.

Original: [Table]
Revised: [Table]

Original: To check whether the accuracy differences between those ﬁve methods are signiﬁcant or not, we perform a paired t -test and report the two-sided p -value under the null hypothesis that two related paired samples have identical means.
Revised: To check whether the accuracy differences between those ﬁve methods are signiﬁcant or not, we perform a paired t -test and report the two-sided p -value under the null hypothesis that two related paired samples have identical means.

Original: We show the p -value matrix in Table 5 where for each pair of methods, we report the mean p -value among the 16 tasks.
Revised: We show the p -value matrix in Table 5 where for each pair of methods, we report the mean p -value among the 16 tasks.

Original: [Table]
Revised: [Table]

Original: From Fig.
Revised: From Fig.

Original: 4, we can also observe that given a small fraction of data, there are large gaps between
Revised: 4, we can also observe that given a small fraction of data, there are large gaps between

Original: DAuto and the MLP baseline on most of the tasks.
Revised: DAuto and the MLP baseline on most of the tasks.

Original: Interestingly, these gaps become smaller on certain tasks, e.g., kitchen->electronics, books->kitchen.
Revised: Interestingly, these gaps become smaller on certain tasks, e.g., kitchen->electronics, books->kitchen.

Original: Besides, we observe a gap consistently as the fraction of training data increases.
Revised: Besides, we observe a gap consistently as the fraction of training data increases.

Original: This convinces our conjecture that DAuto is more effective than the MLP baseline in learning informative representations from low-resource data, and has the semi-supervised learning effect due to its generative nature.
Revised: This convinces our conjecture that DAuto is more effective than the MLP baseline in learning informative representations from low-resource data, and has the semi-supervised learning effect due to its generative nature.

Original: A. S PEECH R ECOGNITION WITH A CCENTS
Revised: A. S PEECH R ECOGNITION WITH A CCENTS

Original: The model structure of the recurrent network we used for this experiment is as follows: at each time step, the input feature is a log spectrogram feature of 161 dimensions.
Revised: The model structure of the recurrent network we used for this experiment is as follows: at each time step, the input feature is a log spectrogram feature of 161 dimensions.

Original: This is followed by a 1D convolution layer with 1024 kernels, and 3 stacked bidirectional LSTM layers, each of which contains 1024 hidden units.
Revised: This is followed by a 1D convolution layer with 1024 kernels, and 3 stacked bidirectional LSTM layers, each of which contains 1024 hidden units.

Original: The output of the last bidirectional LSTM layer is connected to a fullyconnected softmax layer with 29 outputs, which represents 26 characters and three special characters: space, apostrophe and blank.
Revised: The output of the last bidirectional LSTM layer is connected to a fullyconnected softmax layer with 29 outputs, which represents 26 characters and three special characters: space, apostrophe and blank.

Original: We use the publicly available CTC loss implementation https: //github.com/baidu-research/warp-ctc and our experiments are performed under the
Revised: We use the publicly available CTC loss implementation https: //github.com/baidu-research/warp-ctc and our experiments are performed under the