Original: J OINT AUTOENCODERS : A FLEXIBLE META - LEARNING FRAMEWORK
Revised: J OINT AUTOENCODERS : A FLEXIBLE META - LEARNING FRAMEWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The incorporation of prior knowledge into learning is essential in achieving good performance based on small noisy samples.
Revised: The incorporation of prior knowledge into learning is essential in achieving good performance based on small noisy samples.

Original: Such knowledge is often incorporated through the availability of related data arising from domains and tasks similar to the one of current interest.
Revised: Such knowledge is often incorporated through the availability of related data arising from domains and tasks similar to the one of current interest.

Original: Ideally one would like to allow both the data for the current task and for previous related tasks to self-organize the learning system in such a way that commonalities and differences between the tasks are learned in a datadriven fashion.
Revised: Ideally one would like to allow both the data for the current task and for previous related tasks to self-organize the learning system in such a way that commonalities and differences between the tasks are learned in a datadriven fashion.

Original: We develop a framework for learning multiple tasks simultaneously, based on sharing features that are common to all tasks, achieved through the use of a modular deep feedforward neural network consisting of shared branches, dealing with the common features of all tasks, and private branches, learning the speciﬁc unique aspects of each task.
Revised: We develop a framework for learning multiple tasks simultaneously, based on sharing features that are common to all tasks, achieved through the use of a modular deep feedforward neural network consisting of shared branches, dealing with the common features of all tasks, and private branches, learning the speciﬁc unique aspects of each task.

Original: Once an appropriate weight sharing architecture has been established, learning takes place through standard algorithms for feedforward networks, e.g., stochastic gradient descent and its variations.
Revised: Once an appropriate weight sharing architecture has been established, learning takes place through standard algorithms for feedforward networks, e.g., stochastic gradient descent and its variations.

Original: The method deals with meta-learning (such as domain adaptation, transfer and multi-task learning) in a uniﬁed fashion, and can easily deal with data arising from different types of sources.
Revised: The method deals with meta-learning (such as domain adaptation, transfer and multi-task learning) in a uniﬁed fashion, and can easily deal with data arising from different types of sources.

Original: Numerical experiments demonstrate the effectiveness of learning in domain adaptation and transfer learning setups, and provide evidence for the ﬂexible and task-oriented representations arising in the network.
Revised: Numerical experiments demonstrate the effectiveness of learning in domain adaptation and transfer learning setups, and provide evidence for the ﬂexible and task-oriented representations arising in the network.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A major goal of inductive learning is the selection of a rule that generalizes well based on a ﬁnite set of examples.
Revised: A major goal of inductive learning is the selection of a rule that generalizes well based on a ﬁnite set of examples.

Original: It is well-known ((Hume, 1748)) that inductive learning is impossible unless some regularity assumptions are made about the world.
Revised: It is well-known ((Hume, 1748)) that inductive learning is impossible unless some regularity assumptions are made about the world.

Original: Such assumptions, by their nature, go beyond the data, and are based on prior knowledge achieved through previous interactions with ’similar’ problems.
Revised: Such assumptions, by their nature, go beyond the data, and are based on prior knowledge achieved through previous interactions with ’similar’ problems.

Original: Following its early origins ((Baxter, 2000; Thrun and Pratt, 1998)), the incorporation of prior knowledge into learning has become a major effort recently, and is gaining increasing success by relying on the rich representational ﬂexibility available through current deep learning schemes (Bengio et al., 2013).
Revised: Following its early origins ((Baxter, 2000; Thrun and Pratt, 1998)), the incorporation of prior knowledge into learning has become a major effort recently, and is gaining increasing success by relying on the rich representational ﬂexibility available through current deep learning schemes (Bengio et al., 2013).

Original: Various aspects of prior knowledge are captured in different settings of meta-learning, such as learning-to-learn, domain adaptation, transfer learning, multi-task learning, etc.
Revised: Various aspects of prior knowledge are captured in different settings of meta-learning, such as learning-to-learn, domain adaptation, transfer learning, multi-task learning, etc.

Original: (e.g., (Goodfellow et al., 2016)).
Revised: (e.g., (Goodfellow et al., 2016)).

Original: In this work, we consider the setup of multi-task learning, ﬁrst formalized in (Baxter, 2000), where a set of tasks is available for learning, and the objective is to extract knowledge from a subset of tasks in order to facilitate learning of other, related, tasks.
Revised: In this work, we consider the setup of multi-task learning, ﬁrst formalized in (Baxter, 2000), where a set of tasks is available for learning, and the objective is to extract knowledge from a subset of tasks in order to facilitate learning of other, related, tasks.

Original: Within the framework of representation learning, the core idea is that of shared representations, allowing a given task to beneﬁt from what has been learned from other tasks, since the shared aspects of the representation are based on more information (Zhang et al., 2008).
Revised: Within the framework of representation learning, the core idea is that of shared representations, allowing a given task to beneﬁt from what has been learned from other tasks, since the shared aspects of the representation are based on more information (Zhang et al., 2008).

Original: We consider both unsupervised and semi-supervised learning setups.
Revised: We consider both unsupervised and semi-supervised learning setups.

Original: In the former setting we have several related datasets, arising from possibly different domains, and aim to compress each dataset based on features that are shared between the datasets, and on features that are unique to each problem.
Revised: In the former setting we have several related datasets, arising from possibly different domains, and aim to compress each dataset based on features that are shared between the datasets, and on features that are unique to each problem.

Original: Neither the shared nor the individual features are given apriori, but are learned using a deep neural network architecture within an autoencoding scheme.
Revised: Neither the shared nor the individual features are given apriori, but are learned using a deep neural network architecture within an autoencoding scheme.

Original: While such a joint representation could, in principle, serve as a basis for supervised learning, it has become increasingly evident that representations should contain some information about the output (label) identity in order to perform well, and that using pre-training based on unlabeled data is not always advantageous (e.g., chap.in (Goodfellow et al., 2016)).
Revised: While such a joint representation could, in principle, serve as a basis for supervised learning, it has become increasingly evident that representations should contain some information about the output (label) identity in order to perform well, and that using pre-training based on unlabeled data is not always advantageous (e.g., chap.in (Goodfellow et al., 2016)).

Original: However, since unlabeled data is far more abundant than labeled data, much useful information can be gained from it.
Revised: However, since unlabeled data is far more abundant than labeled data, much useful information can be gained from it.

Original: We therefore propose a joint encoding-classiﬁcation scheme where both labeled and unlabeled data are used for the multiple tasks, so that internal representations found reﬂect both types of data, but are learned simultaneously.
Revised: We therefore propose a joint encoding-classiﬁcation scheme where both labeled and unlabeled data are used for the multiple tasks, so that internal representations found reﬂect both types of data, but are learned simultaneously.

Original: The main contributions of this work are: (i) A generic and ﬂexible modular setup for combining unsupervised, supervised and transfer learning.
Revised: The main contributions of this work are: (i) A generic and ﬂexible modular setup for combining unsupervised, supervised and transfer learning.

Original: (ii) Efﬁcient end-to-end transfer learning using mostly unsupervised data (i.e., very few labeled examples are required for successful transfer learning).
Revised: (ii) Efﬁcient end-to-end transfer learning using mostly unsupervised data (i.e., very few labeled examples are required for successful transfer learning).

Original: (iii)
Revised: (iii)

Original: Explicit extraction of task-speciﬁc and shared representations.
Revised: Explicit extraction of task-speciﬁc and shared representations.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Previous related work can be broadly separated into two classes of models: (i) Generative models attempting to learn the input representations.
Revised: Previous related work can be broadly separated into two classes of models: (i) Generative models attempting to learn the input representations.

Original: (ii) Non-generative methods that construct separate or shared representations in a bottom-up fashion driven by the inputs.
Revised: (ii) Non-generative methods that construct separate or shared representations in a bottom-up fashion driven by the inputs.

Original: We ﬁrst discuss several works within the non-generative setting.
Revised: We ﬁrst discuss several works within the non-generative setting.

Original: The Deep Domain Confusion (DDC) algorithm in (Tzeng et al., 2014) studies the problems of unsupervised domain adaptation based on sets of unlabeled samples from the source and target domains, and supervised domain adaptation where a (usually small) subset of the target domain is labeled .
Revised: The Deep Domain Confusion (DDC) algorithm in (Tzeng et al., 2014) studies the problems of unsupervised domain adaptation based on sets of unlabeled samples from the source and target domains, and supervised domain adaptation where a (usually small) subset of the target domain is labeled .

Original: By incorporating an adaptation layer and a domain confusion loss they learn a representation that optimizes both classiﬁcation accuracy and domain invariance, where the latter is achieved by minimizing an appropriate discrepancy measure.
Revised: By incorporating an adaptation layer and a domain confusion loss they learn a representation that optimizes both classiﬁcation accuracy and domain invariance, where the latter is achieved by minimizing an appropriate discrepancy measure.

Original: By maintaining a small distance between the source and target representations, the classiﬁer makes good use of the relevant prior knowledge.
Revised: By maintaining a small distance between the source and target representations, the classiﬁer makes good use of the relevant prior knowledge.

Original: The algorithm suggested in (Ganin and Lempitsky, 2015) augments standard deep learning with a domain classiﬁer that is connected to the feature extractor, and acts to modify the gradient during backpropagation.
Revised: The algorithm suggested in (Ganin and Lempitsky, 2015) augments standard deep learning with a domain classiﬁer that is connected to the feature extractor, and acts to modify the gradient during backpropagation.

Original: This adaptation promotes the similarity between the feature distributions in a domain adaptation task.
Revised: This adaptation promotes the similarity between the feature distributions in a domain adaptation task.

Original: The Deep Reconstruction Classiﬁcation
Revised: The Deep Reconstruction Classiﬁcation

Original: Network (DRCN) in (Ghifary et al., 2016) tackles the unsupervised domain adaptation task by jointly learning a shared encoding representation of the source and target domains based on minimizing a loss function that balances between the classiﬁcation loss of the (labeled) source data and the reconstruction cost of the target data.
Revised: Network (DRCN) in (Ghifary et al., 2016) tackles the unsupervised domain adaptation task by jointly learning a shared encoding representation of the source and target domains based on minimizing a loss function that balances between the classiﬁcation loss of the (labeled) source data and the reconstruction cost of the target data.

Original: The shared encoding parameters allow the target representation to beneﬁt from the ample source supervised data.
Revised: The shared encoding parameters allow the target representation to beneﬁt from the ample source supervised data.

Original: In addition to these mostly algorithmic approaches, a number of theoretical papers have attempted to provide a deeper understanding of the beneﬁts available within this setting (Ben-David et al., 2009; Maurer et al., 2016).
Revised: In addition to these mostly algorithmic approaches, a number of theoretical papers have attempted to provide a deeper understanding of the beneﬁts available within this setting (Ben-David et al., 2009; Maurer et al., 2016).

Original: Next, we mention some recent work within the generative approach, brieﬂy.
Revised: Next, we mention some recent work within the generative approach, brieﬂy.

Original: Recent work has suggested several extensions of the increasingly popular Generative Adversarial Networks (GAN) framework (Goodfellow et al., 2014).
Revised: Recent work has suggested several extensions of the increasingly popular Generative Adversarial Networks (GAN) framework (Goodfellow et al., 2014).

Original: The Coupled Generative Adversarial Network (CoGAN) framework in (Liu and Tuzel, 2016) aims to generate pairs of corresponding representations from inputs arising from different domains.
Revised: The Coupled Generative Adversarial Network (CoGAN) framework in (Liu and Tuzel, 2016) aims to generate pairs of corresponding representations from inputs arising from different domains.

Original: They propose learning joint distributions over two domains based only on samples from the marginals.
Revised: They propose learning joint distributions over two domains based only on samples from the marginals.

Original: This yields good results for small datasets, but is unfortunately challenging to achieve for large adaptation tasks, and is computationally cumbersome.
Revised: This yields good results for small datasets, but is unfortunately challenging to achieve for large adaptation tasks, and is computationally cumbersome.

Original: The Adversarial Discriminative Domain Adaptation (ADDA) approach (Tzeng et al., 2017) subsumes some previous results within the GAN framework of domain adaptation.
Revised: The Adversarial Discriminative Domain Adaptation (ADDA) approach (Tzeng et al., 2017) subsumes some previous results within the GAN framework of domain adaptation.

Original: The approach learns a discriminative representation using the data in the labeled source domain, and then learns to adapt the model for use in the (unlabeled) target domain through a domain adversarial loss function.
Revised: The approach learns a discriminative representation using the data in the labeled source domain, and then learns to adapt the model for use in the (unlabeled) target domain through a domain adversarial loss function.

Original: The idea is implemented through a minimax formulation similar to the original GAN setup.
Revised: The idea is implemented through a minimax formulation similar to the original GAN setup.

Original: The extraction of shared and task-speciﬁc representations is the subject of a number of works, such as (Evgeniou and Pontil, 2004) and (Parameswaran and Weinberger, 2010).
Revised: The extraction of shared and task-speciﬁc representations is the subject of a number of works, such as (Evgeniou and Pontil, 2004) and (Parameswaran and Weinberger, 2010).

Original: However, works in this direction typically require inputs of the same dimension and for the sizes of their shared and task-speciﬁc features to be the same.
Revised: However, works in this direction typically require inputs of the same dimension and for the sizes of their shared and task-speciﬁc features to be the same.

Original: A great deal of work has been devoted to multi-modal learning where the inputs arise from different modalities.
Revised: A great deal of work has been devoted to multi-modal learning where the inputs arise from different modalities.

Original: Exploiting data from multiple sources (or views) to extract meaningful features, is often done by seeking representations that are sensitive only to the common variability in the views and are indifferent to view-speciﬁc variations.
Revised: Exploiting data from multiple sources (or views) to extract meaningful features, is often done by seeking representations that are sensitive only to the common variability in the views and are indifferent to view-speciﬁc variations.

Original: Many methods in this category attempt to maximize the correlation between the learned representations, as in the linear canonical correlation analysis (CCA) technique and its various nonlinear extensions (Andrew et al., 2013; Michaeli et al., 2016).
Revised: Many methods in this category attempt to maximize the correlation between the learned representations, as in the linear canonical correlation analysis (CCA) technique and its various nonlinear extensions (Andrew et al., 2013; Michaeli et al., 2016).

Original: Other methods use losses based on both correlation and reconstruction error (in an auto-encoding like scheme) (Wang et al., 2015), or employ diffusion processes to reveal the common underlying manifold (Lederman and Talmon, 2015).
Revised: Other methods use losses based on both correlation and reconstruction error (in an auto-encoding like scheme) (Wang et al., 2015), or employ diffusion processes to reveal the common underlying manifold (Lederman and Talmon, 2015).

Original: However, all multi-view representation learning algorithms rely on paired examples from the two views.
Revised: However, all multi-view representation learning algorithms rely on paired examples from the two views.

Original: This setting is thus very different from transfer learning, multi-task learning, or domain adaptation, where one has access only to unpaired samples from each of the domains.
Revised: This setting is thus very different from transfer learning, multi-task learning, or domain adaptation, where one has access only to unpaired samples from each of the domains.

Original: While GANs provide a powerful approach to multi-task learning and domain adaptation, they are often hard to train and ﬁne tune ((Goodfellow, 2016)).
Revised: While GANs provide a powerful approach to multi-task learning and domain adaptation, they are often hard to train and ﬁne tune ((Goodfellow, 2016)).

Original: Our approach offers a complementary nongenerative perspective, and operates in an end-to-end fashion allowing the parallel training of multiple tasks, incorporating both unsupervised, supervised and transfer settings within a single architecture.
Revised: Our approach offers a complementary nongenerative perspective, and operates in an end-to-end fashion allowing the parallel training of multiple tasks, incorporating both unsupervised, supervised and transfer settings within a single architecture.

Original: This simplicity allows the utilization of standard optimization techniques for regular deep feedforward networks, so that any advances in that domain translate directly into improvements in our results.
Revised: This simplicity allows the utilization of standard optimization techniques for regular deep feedforward networks, so that any advances in that domain translate directly into improvements in our results.

Original: The approach does not require paired inputs and can operate with inputs arising from entirely different domains, such as speech and audio (although this has not been demonstrated empirically here).
Revised: The approach does not require paired inputs and can operate with inputs arising from entirely different domains, such as speech and audio (although this has not been demonstrated empirically here).

Original: Our work is closest to (Bousmalis et al., 2016)which shares with us the separation into common and private branches.
Revised: Our work is closest to (Bousmalis et al., 2016)which shares with us the separation into common and private branches.

Original: They base their optimization on several loss functions beyond the reconstruction and classiﬁcation losses, enforcing constraints on intermediate representations.
Revised: They base their optimization on several loss functions beyond the reconstruction and classiﬁcation losses, enforcing constraints on intermediate representations.

Original: Speciﬁcally, they penalize differences between the common and private branches of the same task, and encourage similarity between the different representations of the source and target in the common branch.
Revised: Speciﬁcally, they penalize differences between the common and private branches of the same task, and encourage similarity between the different representations of the source and target in the common branch.

Original: This multiplicity of loss functions adds several free parameters to the problem that require further ﬁne-tuning.
Revised: This multiplicity of loss functions adds several free parameters to the problem that require further ﬁne-tuning.

Original: Our framework uses only losses penalizing reconstruction and classiﬁcation errors, thereby directly focusing on the task without adding internal constrains.
Revised: Our framework uses only losses penalizing reconstruction and classiﬁcation errors, thereby directly focusing on the task without adding internal constrains.

Original: Moreover, since DSN does not use a classiﬁcation error for the target it cannot use labeled targets, and thus can only perform unsupervised transfer learning.
Revised: Moreover, since DSN does not use a classiﬁcation error for the target it cannot use labeled targets, and thus can only perform unsupervised transfer learning.

Original: Also, due to the internal loss functions, it is not clear how to extend
Revised: Also, due to the internal loss functions, it is not clear how to extend

Original: DSN to multi-task learning, which is immediate in our formalism.
Revised: DSN to multi-task learning, which is immediate in our formalism.

Original: Practically, the proposed DSN architecture is costly; it is larger by more than on order of magnitude than either the models we have studied or ADDA.
Revised: Practically, the proposed DSN architecture is costly; it is larger by more than on order of magnitude than either the models we have studied or ADDA.

Original: Thus it is computationally challenging as well as relatively struggling to deal with small datasets.
Revised: Thus it is computationally challenging as well as relatively struggling to deal with small datasets.

Original:  J OINT AUTOENCODERS
Revised:  J OINT AUTOENCODERS

Original: In this section, we introduce joint autoencoders (JAE), a general method for multi-task learning by unsupervised extraction of features shared by the tasks as well as features speciﬁc to each task.
Revised: In this section, we introduce joint autoencoders (JAE), a general method for multi-task learning by unsupervised extraction of features shared by the tasks as well as features speciﬁc to each task.

Original: We begin by presenting a simple case, point out the various possible generalizations, and ﬁnally describe two transfer and multi-task learning procedures utilizing joint autoencoders.
Revised: We begin by presenting a simple case, point out the various possible generalizations, and ﬁnally describe two transfer and multi-task learning procedures utilizing joint autoencoders.

Original: 3. J OINT AUTOENCODERS FOR RECONSTRUCTION
Revised: 3. J OINT AUTOENCODERS FOR RECONSTRUCTION

Original: Consider a multi-task learning scenario with T tasks t 1 , ..., t T deﬁned by domains (cid:8)(cid:0)
Revised: Consider a multi-task learning scenario with T tasks t 1 , ..., t T deﬁned by domains (cid:8)(cid:0)

Original: X i (cid:1)(cid:9) T i =1 .
Revised: X i (cid:1)(cid:9) T i =1 .

Original: Each task t i is equipped with a set of unlabeled samples (cid:8) x in ∈ X
Revised: Each task t i is equipped with a set of unlabeled samples (cid:8) x in ∈ X

Original: i (cid:9) N i,u n =1 ,where N i,u denotes the size of the unlabeled data set, and with a reconstruction loss function (cid:96) i r (cid:0) x i n , ˜ x i n (cid:1) , where ˜ x i n is the reconstruction of the sample x in .
Revised: i (cid:9) N i,u n =1 ,where N i,u denotes the size of the unlabeled data set, and with a reconstruction loss function (cid:96) i r (cid:0) x i n , ˜ x i n (cid:1) , where ˜ x i n is the reconstruction of the sample x in .

Original: Throughout the paper, we will interpret (cid:96) ir as the L 2 distance between x in and ˜ x in , but in principle (cid:96) ir can represent any unsupervised learning goal.
Revised: Throughout the paper, we will interpret (cid:96) ir as the L 2 distance between x in and ˜ x in , but in principle (cid:96) ir can represent any unsupervised learning goal.

Original: The tasks are assumed to be related, and we are interested in exploiting this similarity to improve the reconstruction.
Revised: The tasks are assumed to be related, and we are interested in exploiting this similarity to improve the reconstruction.

Original: To do this, we make the following two observations:
Revised: To do this, we make the following two observations:

Original: (i)
Revised: (i)

Original: Certain aspects of the unsupervised tasks we are facing may be similar, but other aspects may be quite different (e
Revised: Certain aspects of the unsupervised tasks we are facing may be similar, but other aspects may be quite different (e

Original: g., when two domains contain color and grayscale images, respectively).
Revised: g., when two domains contain color and grayscale images, respectively).

Original: (ii)
Revised: (ii)

Original: The similarity between the tasks can be rather “deep”.
Revised: The similarity between the tasks can be rather “deep”.

Original: For example, cartoon images and natural images may beneﬁt from different low-level features, but may certainly share high-level structures.
Revised: For example, cartoon images and natural images may beneﬁt from different low-level features, but may certainly share high-level structures.

Original: To accommodate these two observations, we associate with each task t i a pair of functions: f ip (cid:0) x ; θ ip (cid:1) , the “private branch”, and f is (cid:16) x ; θ is , ˜ θ s (cid:17) , the “shared branch”
Revised: To accommodate these two observations, we associate with each task t i a pair of functions: f ip (cid:0) x ; θ ip (cid:1) , the “private branch”, and f is (cid:16) x ; θ is , ˜ θ s (cid:17) , the “shared branch”

Original: The functions f ip are responsible for the task-speciﬁc representations of t i and are parametrized by parameters θ ip .
Revised: The functions f ip are responsible for the task-speciﬁc representations of t i and are parametrized by parameters θ ip .

Original: The functions f is are responsible for the shared representations, and are parametrized, in addition to parameters θ is , by ˜ θ s shared by all tasks.
Revised: The functions f is are responsible for the shared representations, and are parametrized, in addition to parameters θ is , by ˜ θ s shared by all tasks.

Original: The key idea is that the weight sharing forces the common branches to learn to represent the common features of the two sources.
Revised: The key idea is that the weight sharing forces the common branches to learn to represent the common features of the two sources.

Original: Consequently, the private branches are implicitly forced to capture only the features that are not common to the other task.
Revised: Consequently, the private branches are implicitly forced to capture only the features that are not common to the other task.

Original: We aim at minimizing the
Revised: We aim at minimizing the

Original: [Figure]
Revised: [Figure]

Original: ) 𝑓 𝑝1 𝜃 𝑝 𝑓 𝑝2 𝜃 𝑝 𝑓 𝑠2 𝜃 𝑠2 ; 𝜃 𝑠 𝑓 𝑠 1 𝜃 𝑠 1 ; 𝜃 𝑠 𝜃 𝑝 𝜃 𝑝 𝜃 𝑠 𝜃 𝑠 𝜃 𝑠 𝜃 𝑠 𝜃 2 𝑝 𝜃 2𝑝 𝜃 𝑠 ℒ 𝑟 ℒ 𝑟
Revised: ) 𝑓 𝑝1 𝜃 𝑝 𝑓 𝑝2 𝜃 𝑝 𝑓 𝑠2 𝜃 𝑠2 ; 𝜃 𝑠 𝑓 𝑠 1 𝜃 𝑠 1 ; 𝜃 𝑠 𝜃 𝑝 𝜃 𝑝 𝜃 𝑠 𝜃 𝑠 𝜃 𝑠 𝜃 𝑠 𝜃 2 𝑝 𝜃 2𝑝 𝜃 𝑠 ℒ 𝑟 ℒ 𝑟

Original: [Figure]
Revised: [Figure]

Original: b)
Revised: b)

Original: [Figure]
Revised: [Figure]

Original: cumulative weighted loss
Revised: cumulative weighted loss

Original: [Equation]
Revised: [Equation]

Original: In practice, we implement all functions as autoencoders and the shared parameters ˜ θ s as the bottleneck of the shared branch of each task, with identical weights across the tasks.
Revised: In practice, we implement all functions as autoencoders and the shared parameters ˜ θ s as the bottleneck of the shared branch of each task, with identical weights across the tasks.

Original: Our framework, however, supports more ﬂexible sharing as well, such as sharing more than a single layer, or even partially shared layers.
Revised: Our framework, however, supports more ﬂexible sharing as well, such as sharing more than a single layer, or even partially shared layers.

Original: The resulting network can be trained with standard backpropagation on all reconstruction losses simultaneously.
Revised: The resulting network can be trained with standard backpropagation on all reconstruction losses simultaneously.

Original: Figure 1(a) illustrates a typical autoencoder for the MNIST dataset, and Figure 1(b) illustrates the architecture obtained from implementing all branches in the formal description above with such autoencoders (AE).
Revised: Figure 1(a) illustrates a typical autoencoder for the MNIST dataset, and Figure 1(b) illustrates the architecture obtained from implementing all branches in the formal description above with such autoencoders (AE).

Original: We call this architecture a joint autoencoder (JAE).
Revised: We call this architecture a joint autoencoder (JAE).

Original: As mentioned before, in this simple example, both inputs are MNIST digits, all branches have the same architecture, and the bottlenecks are single layers of the same dimension.
Revised: As mentioned before, in this simple example, both inputs are MNIST digits, all branches have the same architecture, and the bottlenecks are single layers of the same dimension.

Original: However, this need not be the case.
Revised: However, this need not be the case.

Original: The inputs can be entirely different (e.g., image and text), all branches may have different architectures, the bottleneck sizes can vary, and more than a single layer can be shared.
Revised: The inputs can be entirely different (e.g., image and text), all branches may have different architectures, the bottleneck sizes can vary, and more than a single layer can be shared.

Original: Furthermore, the shared layers need not be the bottlenecks, in general.
Revised: Furthermore, the shared layers need not be the bottlenecks, in general.

Original: Finally, the generalization to more than two tasks is straightforward - we simply add a pair of autoencoders for each task, and share some of the layers of the common-feature autoencoders.
Revised: Finally, the generalization to more than two tasks is straightforward - we simply add a pair of autoencoders for each task, and share some of the layers of the common-feature autoencoders.

Original: Weight sharing can take place between subsets of tasks, and can occur at different levels for the different tasks.
Revised: Weight sharing can take place between subsets of tasks, and can occur at different levels for the different tasks.

Original: 3. J OINT AUTOENCODERS FOR MULTI - TASK , SEMI - SUPERVISED AND TRANSFER LEARNING
Revised: 3. J OINT AUTOENCODERS FOR MULTI - TASK , SEMI - SUPERVISED AND TRANSFER LEARNING

Original: Consider now a situation in which, in addition to the unlabeled samples from all domains X i , we also have datasets of labeled pairs (cid:8)(cid:0) x
Revised: Consider now a situation in which, in addition to the unlabeled samples from all domains X i , we also have datasets of labeled pairs (cid:8)(cid:0) x

Original: ik , y ik (cid:1)(cid:9) N i,l k =1 where N i,l is the size of the labeled set for task t i and is assumed to be much smaller than N i,u .
Revised: ik , y ik (cid:1)(cid:9) N i,l k =1 where N i,l is the size of the labeled set for task t i and is assumed to be much smaller than N i,u .

Original: The supervised component of each task t i is reﬂected in the supervised loss (cid:96) ic (cid:0) x in ,  y in (cid:1) , typically multi-class classiﬁcation.
Revised: The supervised component of each task t i is reﬂected in the supervised loss (cid:96) ic (cid:0) y in , ˜ y in (cid:1) , typically multi-class classiﬁcation.

Original: We extend our loss deﬁnition in Equation 1 to be
Revised: We extend our loss deﬁnition in Equation 1 to be

Original: [Equation]
Revised: [Equation]

Original: where we now interpret the functions f is , f ip to also output a classiﬁcation.
Revised: where we now interpret the functions f is , f ip to also output a classiﬁcation.

Original: Figure 1(c) illustrates the schematic structure of a JAE extended to include supervised losses.
Revised: Figure 1(c) illustrates the schematic structure of a JAE extended to include supervised losses.

Original: Note that this framework supports various learning scenarios.
Revised: Note that this framework supports various learning scenarios.

Original: Indeed, if a subset of the tasks has N i,l = 0 , the problem becomes one of unsupervised domain adaptation.
Revised: Indeed, if a subset of the tasks has N i,l = 0 , the problem becomes one of unsupervised domain adaptation.

Original: The case where N i,l are all or mostly small describes semi-supervised learning.
Revised: The case where N i,l are all or mostly small describes semi-supervised learning.

Original: If some of the labeled sets are large while the others are either small or empty, we ﬁnd ourselves facing a transfer learning challenge.
Revised: If some of the labeled sets are large while the others are either small or empty, we ﬁnd ourselves facing a transfer learning challenge.

Original: Finally, when all labeled sets are of comparable sizes, this is multi-task learning, either supervised (when N i,l are all positive) or unsupervised (when N i,l = 0 ).
Revised: Finally, when all labeled sets are of comparable sizes, this is multi-task learning, either supervised (when N i,l are all positive) or unsupervised (when N i,l = 0 ).

Original: We describe two strategies to improve supervised learning by exploiting shared features.
Revised: We describe two strategies to improve supervised learning by exploiting shared features.

Original: Common-branch transfer In this approach, we ﬁrst train joint autoencoders on both source and target tasks simultaneously, using all available unlabeled data.
Revised: Common-branch transfer In this approach, we ﬁrst train joint autoencoders on both source and target tasks simultaneously, using all available unlabeled data.

Original: Then, for the source tasks (the ones with more labeled examples), we ﬁne-tune the branches up to the shared layer using the sets of labeled samples, and freeze the learned shared layers.
Revised: Then, for the source tasks (the ones with more labeled examples), we ﬁne-tune the branches up to the shared layer using the sets of labeled samples, and freeze the learned shared layers.

Original: Finally, for the target tasks, we use the available labeled data to train only its private branches while ﬁxing the shared layers ﬁne-tuned on the source data.
Revised: Finally, for the target tasks, we use the available labeled data to train only its private branches while ﬁxing the shared layers ﬁne-tuned on the source data.

Original: End-to-end learning The second, end-to-end approach, combines supervised and unsupervised training.
Revised: End-to-end learning The second, end-to-end approach, combines supervised and unsupervised training.

Original: Here we extend the JAE architecture by adding new layers, with supervised loss functions for each task; see Figure 1(c).
Revised: Here we extend the JAE architecture by adding new layers, with supervised loss functions for each task; see Figure 1(c).

Original: We train the new network using all losses from all tasks simultaneously - reconstruction losses using unlabeled data, and supervised losses using labeled data.
Revised: We train the new network using all losses from all tasks simultaneously - reconstruction losses using unlabeled data, and supervised losses using labeled data.

Original: When the size of the labeled sets is highly non-uniform, the network is naturally suitable for transfer learning.
Revised: When the size of the labeled sets is highly non-uniform, the network is naturally suitable for transfer learning.

Original: When the labeled sample sizes are roughly of the same order of magnitude, the setup is suitable for semi-supervised learning.
Revised: When the labeled sample sizes are roughly of the same order of magnitude, the setup is suitable for semi-supervised learning.

Original: 3. O N THE DEPTH OF SHARING
Revised: 3. O N THE DEPTH OF SHARING

Original: It is common knowledge that similar low-level features are often helpful for similar tasks.
Revised: It is common knowledge that similar low-level features are often helpful for similar tasks.

Original: For example, in many vision applications, CNNs exhibit the same Gabor-type ﬁlters in their ﬁrst layer, regardless of the objects they are trained to classify.
Revised: For example, in many vision applications, CNNs exhibit the same Gabor-type ﬁlters in their ﬁrst layer, regardless of the objects they are trained to classify.

Original: This observation makes low-level features immediate candidates for sharing in multi-task learning settings.
Revised: This observation makes low-level features immediate candidates for sharing in multi-task learning settings.

Original: However, unsurprisingly, sharing low-level features is not as beneﬁcial when working with domains of different nature (e.g., handwritten digits vs. street signs).
Revised: However, unsurprisingly, sharing low-level features is not as beneﬁcial when working with domains of different nature (e.g., handwritten digits vs. street signs).

Original: Our approach allows to share weights in deeper layers of a neural net, while leaving the shallow layers un-linked.
Revised: Our approach allows to share weights in deeper layers of a neural net, while leaving the shallow layers un-linked.

Original: The key idea is that by forcing all shared-branch nets to share deep weights, their preceding shallow layers must learn to transform the data from the different domains into a common form.
Revised: The key idea is that by forcing all shared-branch nets to share deep weights, their preceding shallow layers must learn to transform the data from the different domains into a common form.

Original: We support this intuition through several experiments.
Revised: We support this intuition through several experiments.

Original: As our preliminary results in Section 4.2.1 show, for similar domains, sharing deep layers provides the same performance boost as sharing shallow layers.
Revised: As our preliminary results in Section 4.2.1 show, for similar domains, sharing deep layers provides the same performance boost as sharing shallow layers.

Original: Thus, we pay no price for relying only on “deep similarities”.
Revised: Thus, we pay no price for relying only on “deep similarities”.

Original: But for domains of a different nature, sharing deep layers has a clear advantage.
Revised: But for domains of a different nature, sharing deep layers has a clear advantage.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: All experiments were implemented in Keras over Tensorﬂow.
Revised: All experiments were implemented in Keras over Tensorﬂow.

Original: The code will be made available soon, and the network architectures used are given in detail in the appendix.
Revised: The code will be made available soon, and the network architectures used are given in detail in the appendix.

Original: 4. U NSUPERVISED LEARNING
Revised: 4. U NSUPERVISED LEARNING

Original: We present experimental results demonstrating the improvement in unsupervised learning of multiple tasks on the MNIST and CIFAR-10 datasets.
Revised: We present experimental results demonstrating the improvement in unsupervised learning of multiple tasks on the MNIST and CIFAR-10 datasets.

Original: For the MNIST experiment, we have separated the training images into two subsets: X 1 , containing the digits { 0 − 4 } and X 2 , containing the digits { 5 − 9 } .
Revised: For the MNIST experiment, we have separated the training images into two subsets: X 1 , containing the digits { 0 − 4 } and X 2 , containing the digits { 5 − 9 } .

Original: We compared the L 2 reconstruction error achieved by the JAE to a baseline of a pair of
Revised: We compared the L 2 reconstruction error achieved by the JAE to a baseline of a pair of

Original: AEs trained on each dataset with architecture identical to a single branch of the JAE.
Revised: AEs trained on each dataset with architecture identical to a single branch of the JAE.

Original: The joint autoencoder ( MSE =5 . 4 ) out-performed the baseline ( MSE = 5 . 6 ) by 4% .
Revised: The joint autoencoder ( MSE =5 . 4 ) out-performed the baseline ( MSE = 5 . 6 ) by 4% .

Original: The autoencoders had the same cumulative bottleneck size as the JAE, to ensure the same hidden representation size.
Revised: The autoencoders had the same cumulative bottleneck size as the JAE, to ensure the same hidden representation size.

Original: To ensure we did not beneﬁt solely from increased capacity, we also compared the AEs to a JAE with the same total number of parameters as the baseline, obtained by reducing the size of each layer by √ 2 .
Revised: To ensure we did not beneﬁt solely from increased capacity, we also compared the AEs to a JAE with the same total number of parameters as the baseline, obtained by reducing the size of each layer by √ 2 .

Original: This model achieved an MSE of 5 . 52 , a 1 . 4% improvement over the baseline.
Revised: This model achieved an MSE of 5 . 52 , a 1 . 4% improvement over the baseline.

Original: To further understand the features learned by the shared and private bottlenecks, we visualize the activations of the bottlenecks on 1000 samples from each dataset, using 2 D t-SNE embeddings (van der Maaten and Hinton, 2008).
Revised: To further understand the features learned by the shared and private bottlenecks, we visualize the activations of the bottlenecks on 1000 samples from each dataset, using 2 D t-SNE embeddings (van der Maaten and Hinton, 2008).

Original: Figure 2(a) demonstrates that the common branches containing the shared layer (green and magenta) are much more mixed between themselves than the private branches (red and black), indicating that they indeed extract shared features.
Revised: Figure 2(a) demonstrates that the common branches containing the shared layer (green and magenta) are much more mixed between themselves than the private branches (red and black), indicating that they indeed extract shared features.

Original: Figure 2(b) displays examples of digits reconstructions.
Revised: Figure 2(b) displays examples of digits reconstructions.

Original: The columns show (from left to right)
Revised: The columns show (from left to right)

Original: the original digit, the image reconstructed by the full JAE, the output of the private branches and the shared branches.
Revised: the original digit, the image reconstructed by the full JAE, the output of the private branches and the shared branches.

Original: We see that the common branches capture the general shape of the digit, while the private branches capture the ﬁne details which are speciﬁc to each subset.
Revised: We see that the common branches capture the general shape of the digit, while the private branches capture the ﬁne details which are speciﬁc to each subset.

Original: [Figure]
Revised: [Figure]

Original: We verify quantitatively the claim about the differences in separation between the private and shared branches.
Revised: We verify quantitatively the claim about the differences in separation between the private and shared branches.

Original: The Fisher criterion for the separation between the t-SNE embeddings of the private branches is 7 . 22 · 10 − 4 , whereas its counterpart for the shared branches is 2 .
Revised: The Fisher criterion for the separation between the t-SNE embeddings of the private branches is 7 . 22 · 10 − 4 , whereas its counterpart for the shared branches is 2 .

Original: 77 · 10 − 4 , 2 .
Revised: 77 · 10 − 4 , 2 .

Original: 6 times less.
Revised: 6 times less.

Original: Moreover, the shared branch embedding variance for both datasets is approximately identical, whereas the private branches map the dataset they were trained on to locations with variance greater by 1 .
Revised: Moreover, the shared branch embedding variance for both datasets is approximately identical, whereas the private branches map the dataset they were trained on to locations with variance greater by 1 .

Original: 35 than the dataset they had no access to.
Revised: 35 than the dataset they had no access to.

Original: This illustrates the extent to which the shared branches learn to separate both datasets better than the private ones.
Revised: This illustrates the extent to which the shared branches learn to separate both datasets better than the private ones.

Original: For CIFAR-10 we trained the baseline autoencoder on single-class subsets of the database (e.g., all airplane images) and trained the JAE on pairs of such subsets.
Revised: For CIFAR-10 we trained the baseline autoencoder on single-class subsets of the database (e.g., all airplane images) and trained the JAE on pairs of such subsets.

Original: Table 1 shows a few typical results, demonstrating a consistent advantage for JAEs.
Revised: Table 1 shows a few typical results, demonstrating a consistent advantage for JAEs.

Original: Besides the lower reconstruction error, we can see that visually similar image classes, enjoy a greater boost in performance.
Revised: Besides the lower reconstruction error, we can see that visually similar image classes, enjoy a greater boost in performance.

Original: For instance, the pair deer-horses enjoyed a performance boost of 37% , greater than the typical boost of 33 − 35% .
Revised: For instance, the pair deer-horses enjoyed a performance boost of 37% , greater than the typical boost of 33 − 35% .

Original: As with MNIST, we also compared the pair of autoencoders to a JAE with the same total number of parameters (obtained by √ 2 size reduction of each layer), achieving 22 − 24% boost.
Revised: As with MNIST, we also compared the pair of autoencoders to a JAE with the same total number of parameters (obtained by √ 2 size reduction of each layer), achieving 22 − 24% boost.

Original: Thus, the observed improvement is clearly not a result of mere increased network capacity.
Revised: Thus, the observed improvement is clearly not a result of mere increased network capacity.

Original: [Table]
Revised: [Table]

Original: Performance of JAEs and JAEs reduced by a √ 2 factor vs standard AEs in terms of reconstruction
Revised: Performance of JAEs and JAEs reduced by a √ 2 factor vs standard AEs in terms of reconstruction

Original: MSE on pairs of objects in CIFAR-10: airplanes (A), deer (D), horses (H), ships(S).
Revised: MSE on pairs of objects in CIFAR-10: airplanes (A), deer (D), horses (H), ships(S).

Original: For each pair of objects, we give the standard AE error, JAE and JAE-reduced error and the improvement percentage.
Revised: For each pair of objects, we give the standard AE error, JAE and JAE-reduced error and the improvement percentage.

Original: We remark that we experimented with an extension of unsupervised JAEs to variational autoencoders ((Kingma and Welling, 2014)).
Revised: We remark that we experimented with an extension of unsupervised JAEs to variational autoencoders ((Kingma and Welling, 2014)).

Original: Unlike standard VAEs, we trained three hidden code layers, requiring each to have a small Kullback-Leibler divergence from a given normal distribution.
Revised: Unlike standard VAEs, we trained three hidden code layers, requiring each to have a small Kullback-Leibler divergence from a given normal distribution.

Original: One of these layers was used to reconstruct both datasets (analogous to the shared bottleneck in a JAE), while the other two were dedicated each to one of the datasets (analogous to the private branches).
Revised: One of these layers was used to reconstruct both datasets (analogous to the shared bottleneck in a JAE), while the other two were dedicated each to one of the datasets (analogous to the private branches).

Original: The reconstruction results on the halves of the MNIST dataset were promising, yielding an improvement of 12% over a pair of VAEs of the same cumulative size.
Revised: The reconstruction results on the halves of the MNIST dataset were promising, yielding an improvement of 12% over a pair of VAEs of the same cumulative size.

Original: Unfortunately, we were not able to achieve similar results on the CIFAR-10 dataset, nor to perform efﬁcient multi-task\ transfer learning with joint VAEs.
Revised: Unfortunately, we were not able to achieve similar results on the CIFAR-10 dataset, nor to perform efﬁcient multi-task\ transfer learning with joint VAEs.

Original: This remains an intriguing project for the future,
Revised: This remains an intriguing project for the future,

Original: 4. T RANSFER LEARNING
Revised: 4. T RANSFER LEARNING

Original: Next, we compare the performance on MNIST of the two JAE-based transfer learning methods detailed in Section 3.2.
Revised: Next, we compare the performance on MNIST of the two JAE-based transfer learning methods detailed in Section 3.2.

Original: For both methods, X 1 contains digits from { 0 − 4 } and X 2 contains the digits { 5 − 9 } .
Revised: For both methods, X 1 contains digits from { 0 − 4 } and X 2 contains the digits { 5 − 9 } .

Original: The source and target datasets comprise 2000 and 500 samples, respectively.
Revised: The source and target datasets comprise 2000 and 500 samples, respectively.

Original: All results are measured on the full MNIST test set.
Revised: All results are measured on the full MNIST test set.

Original: The common-branch transfer method yields 92 .
Revised: The common-branch transfer method yields 92 .

Original: 3% and 96 .
Revised: 3% and 96 .

Original: 1% classiﬁcation precision for the X 1 → X 2 and X 2 → X 1 transfer tasks, respectively.
Revised: 1% classiﬁcation precision for the X 1 → X 2 and X 2 → X 1 transfer tasks, respectively.

Original: The end-to-end approach results in 96 .
Revised: The end-to-end approach results in 96 .

Original: 6% and 98 .
Revised: 6% and 98 .

Original: 3% scores on the same tasks, which demonstrates the superiority of the end-to-end approach.
Revised: 3% scores on the same tasks, which demonstrates the superiority of the end-to-end approach.

Original: 4.2. S HARED LAYER DEPTH
Revised: 4.2. S HARED LAYER DEPTH

Original: We investigate the inﬂuence of shared layer depth on the transfer performance.
Revised: We investigate the inﬂuence of shared layer depth on the transfer performance.

Original: We see in Tablethat for highly similar pairs of tasks such as the two halves of the MNIST dataset, the depth has little signiﬁcance, while for dissimilar pairs such as MNIST-USPS, “deeper is better” - the performance improves with the shared layer depth.
Revised: We see in Tablethat for highly similar pairs of tasks such as the two halves of the MNIST dataset, the depth has little signiﬁcance, while for dissimilar pairs such as MNIST-USPS, “deeper is better” - the performance improves with the shared layer depth.

Original: Moreover, when the input dimensions differ, early sharing is impossible - the data must ﬁrst be transformed to have the same dimensions.
Revised: Moreover, when the input dimensions differ, early sharing is impossible - the data must ﬁrst be transformed to have the same dimensions.

Original: 4.2. MNIST, USPS AND SVHN DIGITS DATASETS
Revised: 4.2. MNIST, USPS AND SVHN DIGITS DATASETS

Original: We have seen that the end-to-end JAE-with-transfer algorithm outperforms the alternative approach.
Revised: We have seen that the end-to-end JAE-with-transfer algorithm outperforms the alternative approach.

Original: We now compare it to other domain adaptation methods that use little to no target samples for
Revised: We now compare it to other domain adaptation methods that use little to no target samples for

Original: [Table]
Revised: [Table]

Original: Inﬂuence of the shared layer depth on the transfer learning performance.
Revised: Inﬂuence of the shared layer depth on the transfer learning performance.

Original: For the MNIST-USPS pair, only partial data are available for dimensional reasons.
Revised: For the MNIST-USPS pair, only partial data are available for dimensional reasons.

Original: supervised learning, applied to the MNIST, USPS and SVHN digits datasets.
Revised: supervised learning, applied to the MNIST, USPS and SVHN digits datasets.

Original: The transfer tasks we consider are MNIST → USPS , USPS → MNIST and SVHN → MNIST.
Revised: The transfer tasks we consider are MNIST → USPS , USPS → MNIST and SVHN → MNIST.

Original: Following (Tzeng et al., 2017) and (Long et al., 2013), we use 2000 samples for MNIST and 1800 samples from USPS.
Revised: Following (Tzeng et al., 2017) and (Long et al., 2013), we use 2000 samples for MNIST and 1800 samples from USPS.

Original: For
Revised: For

Original: SVHN → MNIST, we use the complete training sets.
Revised: SVHN → MNIST, we use the complete training sets.

Original: In all three tasks, both the source and the target samples are used for the unsupervised JAE training.
Revised: In all three tasks, both the source and the target samples are used for the unsupervised JAE training.

Original: In addition, the source samples are used for the source supervised element of the network.
Revised: In addition, the source samples are used for the source supervised element of the network.

Original: We study the weakly-supervised performance of JAE and ADDA allowing access to a small number of target samples, ranging from 5 to 50 per digit.
Revised: We study the weakly-supervised performance of JAE and ADDA allowing access to a small number of target samples, ranging from 5 to 50 per digit.

Original: For the supervised version of ADDA, we ﬁne-tune the classiﬁers using the small labeled target sets after the domain adaptation.
Revised: For the supervised version of ADDA, we ﬁne-tune the classiﬁers using the small labeled target sets after the domain adaptation.

Original: Figure 3 ( a ) − ( c ) provides the results of our experiments.
Revised: Figure 3 ( a ) − ( c ) provides the results of our experiments.

Original: For recent methods such as CoGAN, gradient reversal, domain confusion and DSN, we display results with zero supervision, as they do not support weakly-supervised training.
Revised: For recent methods such as CoGAN, gradient reversal, domain confusion and DSN, we display results with zero supervision, as they do not support weakly-supervised training.

Original: For DSN, we provide preliminary results on MNIST ↔ USPS, without model optimization that is likely to prevent over-ﬁtting.
Revised: For DSN, we provide preliminary results on MNIST ↔ USPS, without model optimization that is likely to prevent over-ﬁtting.

Original: On all tasks, we achieve results comparable or superior to existing methods using very limited supervision, despite JAE being both conceptually and computationally simpler than competing approaches.
Revised: On all tasks, we achieve results comparable or superior to existing methods using very limited supervision, despite JAE being both conceptually and computationally simpler than competing approaches.

Original: In particular, we do not train a GAN as in CoGAN, and require a single end-to-end training period, unlike ADDA that trains three separate networks in three steps.
Revised: In particular, we do not train a GAN as in CoGAN, and require a single end-to-end training period, unlike ADDA that trains three separate networks in three steps.

Original: Computationally, the models used for MNIST → USPS and USPS → MNIST have 1 .
Revised: Computationally, the models used for MNIST → USPS and USPS → MNIST have 1 .

Original: 36 M parameters, whereas ADDA uses over 1 .
Revised: 36 M parameters, whereas ADDA uses over 1 .

Original: 5 M weights.
Revised: 5 M weights.

Original: For SVHN → MNIST, we use a model with 3 M weights, comparable to the 1 .
Revised: For SVHN → MNIST, we use a model with 3 M weights, comparable to the 1 .

Original: 5 M parameters in ADDA and smaller by an order of magnitude than DSN.
Revised: 5 M parameters in ADDA and smaller by an order of magnitude than DSN.

Original: The SVHN → MNIST task is considered the hardest (for instance, GAN-based approaches fail to address it) yet the abundance of unsupervised training data allows us to achieve good results, relative to previous methods.
Revised: The SVHN → MNIST task is considered the hardest (for instance, GAN-based approaches fail to address it) yet the abundance of unsupervised training data allows us to achieve good results, relative to previous methods.

Original: We provide further demonstration that knowledge is indeed transferred from the source to the target in the MNIST → USPS transfer task with 50 samples per digit.
Revised: We provide further demonstration that knowledge is indeed transferred from the source to the target in the MNIST → USPS transfer task with 50 samples per digit.

Original: Source supervised learning, target unsupervised learning and target classiﬁer training are frozen after the source classiﬁer saturates (epoch 4 ).
Revised: Source supervised learning, target unsupervised learning and target classiﬁer training are frozen after the source classiﬁer saturates (epoch 4 ).

Original: The subsequent target test improvement by 2% is due solely to the source dataset reconstruction training, passed to the target via the shared bottleneck layer (Figure 3(d)).
Revised: The subsequent target test improvement by 2% is due solely to the source dataset reconstruction training, passed to the target via the shared bottleneck layer (Figure 3(d)).

Original: 4.2. T HREE - WAY T RANSFER L EARNING
Revised: 4.2. T HREE - WAY T RANSFER L EARNING

Original: We demonstrate the ability to extend our approach to multiple tasks with ease by transferring knowledge from SVHN to MNIST and USPS simultaneously.
Revised: We demonstrate the ability to extend our approach to multiple tasks with ease by transferring knowledge from SVHN to MNIST and USPS simultaneously.

Original: That is, we train a triple-task JAE reconstructing all three datasets, with additional supervised training on SVHN and weakly-supervised training on the target sets.
Revised: That is, we train a triple-task JAE reconstructing all three datasets, with additional supervised training on SVHN and weakly-supervised training on the target sets.

Original: All labeled samples are used for the source, while the targets usesamples per digit.
Revised: All labeled samples are used for the source, while the targets usesamples per digit.

Original: The results illustrate the beneﬁts of multi-task learning: 94 . 5% classiﬁcation accuracy for MNIST, a 0 .
Revised: The results illustrate the beneﬁts of multi-task learning: 94 . 5% classiﬁcation accuracy for MNIST, a 0 .

Original: 8% improvement over the SVHN → MNIST task, and 88 .
Revised: 8% improvement over the SVHN → MNIST task, and 88 .

Original: 9 % accuracy in
Revised: 9 % accuracy in

Original: UPS, a 1 . 2% improvement over SVHN → USPS.
Revised: UPS, a 1 . 2% improvement over SVHN → USPS.

Original: This is consistent with unsupervised learning being useful for the classiﬁcation.
Revised: This is consistent with unsupervised learning being useful for the classiﬁcation.

Original: USPS is much smaller, thus it has a lower score, but it beneﬁts relatively more from the presence of the other, larger, task.
Revised: USPS is much smaller, thus it has a lower score, but it beneﬁts relatively more from the presence of the other, larger, task.

Original: We stress that the extension to multiple tasks was straightforward, and indeed we did not tweak the various ’ models, opting instead for previously used
Revised: We stress that the extension to multiple tasks was straightforward, and indeed we did not tweak the various ’ models, opting instead for previously used

Original: JAEs, with a single shared bottleneck.
Revised: JAEs, with a single shared bottleneck.

Original: Most state-of-the-art transfer methods do not allow for an obvious, immediate adaptation for transfer learning between multiple tasks.
Revised: Most state-of-the-art transfer methods do not allow for an obvious, immediate adaptation for transfer learning between multiple tasks.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a general scheme for incorporating prior knowledge within deep feedforward neural networks for domain adaptation, multi-task and transfer learning problems.
Revised: We presented a general scheme for incorporating prior knowledge within deep feedforward neural networks for domain adaptation, multi-task and transfer learning problems.

Original: The approach is general and ﬂexible, operates in an end-to-end setting, and enables the system to self-organize to solve tasks based on prior or concomitant exposure to similar tasks, requiring standard gradient based optimization for learning.
Revised: The approach is general and ﬂexible, operates in an end-to-end setting, and enables the system to self-organize to solve tasks based on prior or concomitant exposure to similar tasks, requiring standard gradient based optimization for learning.

Original: The basic idea of the approach is the sharing of representations for aspects which are common to all domains/tasks while maintaining private branches for task-speciﬁc features.
Revised: The basic idea of the approach is the sharing of representations for aspects which are common to all domains/tasks while maintaining private branches for task-speciﬁc features.

Original: The method is applicable to data from multiple sources and types, and has the advantage of being able to share weights at arbitrary network levels, enabling abstract levels of sharing.
Revised: The method is applicable to data from multiple sources and types, and has the advantage of being able to share weights at arbitrary network levels, enabling abstract levels of sharing.

Original: We demonstrated the efﬁcacy of our approach on several domain adaptation and transfer learning problems, and provided intuition about the meaning of the representations in various branches.
Revised: We demonstrated the efﬁcacy of our approach on several domain adaptation and transfer learning problems, and provided intuition about the meaning of the representations in various branches.

Original: In a broader context, it is well known that the imposition of structural constraints on neural networks, usually based on prior domain knowledge, can signiﬁcantly enhance their performance.
Revised: In a broader context, it is well known that the imposition of structural constraints on neural networks, usually based on prior domain knowledge, can signiﬁcantly enhance their performance.

Original: The prime example of this is, of course, the convolutional neural network.
Revised: The prime example of this is, of course, the convolutional neural network.

Original: Our work can be viewed within that general philosophy, showing that improved functionality can be attained by the modular prior structures imposed on the system, while maintaining simple learning rules.
Revised: Our work can be viewed within that general philosophy, showing that improved functionality can be attained by the modular prior structures imposed on the system, while maintaining simple learning rules.

Original: Appendix A. I MPLEMENTATION DETAILS
Revised: Appendix A. I MPLEMENTATION DETAILS

Original: All the images are scaled to [0 , 1] .
Revised: All the images are scaled to [0 , 1] .

Original: In all cases, the training is done using the ADAM optimizer with learning rate 10 − 3 , β 1 = 0 .
Revised: In all cases, the training is done using the ADAM optimizer with learning rate 10 − 3 , β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: The Keras default Xavier initialization is used.
Revised: The Keras default Xavier initialization is used.

Original: Shared layers are denoted in red and connected by a bidirectional arrow: ↔ .
Revised: Shared layers are denoted in red and connected by a bidirectional arrow: ↔ .

Original: Conv n × ( k × k ) stands for a convolution layer with n ﬁlters of size k × k .
Revised: Conv n × ( k × k ) stands for a convolution layer with n ﬁlters of size k × k .

Original: ReLU stands for a rectiﬁed linear unit, i.e. the function max ( x, 0) .
Revised: ReLU stands for a rectiﬁed linear unit, i.e. the function max ( x, 0) .

Original: MP k × k stride l stands for max-pooling of size k × k with stride l .
Revised: MP k × k stride l stands for max-pooling of size k × k with stride l .

Original: FC k stands for a fully-connected layer of size k .
Revised: FC k stands for a fully-connected layer of size k .

Original: The symbol ⊕ stands for the merge operation.
Revised: The symbol ⊕ stands for the merge operation.

Original: For instance, if it appears after fully-connected layers of size 500 each, it denoted the resulting merged layer of size 1000 .
Revised: For instance, if it appears after fully-connected layers of size 500 each, it denoted the resulting merged layer of size 1000 .

Original: Outputs are processed by a SoftMax.
Revised: Outputs are processed by a SoftMax.

Original: Unsupervised learning - MNIST
Revised: Unsupervised learning - MNIST

Original: For the MNIST reconstruction experiments, we utilize a CNN-based version of the autoencoder and
Revised: For the MNIST reconstruction experiments, we utilize a CNN-based version of the autoencoder and

Original: JAE presented in Figure 1.
Revised: JAE presented in Figure 1.

Original: Mini-batch size is set to 256 , with 10 epochs.
Revised: Mini-batch size is set to 256 , with 10 epochs.

Original: The JAE losses are weighed equally.
Revised: The JAE losses are weighed equally.

Original: Unsupervised learning - CIFARMini-batch size is set to 128 , with 10 epochs.
Revised: Unsupervised learning - CIFARMini-batch size is set to 128 , with 10 epochs.

Original: The JAE losses are weighed equally.
Revised: The JAE losses are weighed equally.

Original: “Deconv n × k × k ” stands for a deconvolution layer with n ﬁlters of size k × k with 2 × 2 upsampling.
Revised: “Deconv n × k × k ” stands for a deconvolution layer with n ﬁlters of size k × k with 2 × 2 upsampling.

Original: [Figure]
Revised: [Figure]

Original: Mini-batch size is set to 64 , with 10 epochs.
Revised: Mini-batch size is set to 64 , with 10 epochs.

Original: The reconstruction losses are weighed 4 times higher than the classiﬁcation losses.
Revised: The reconstruction losses are weighed 4 times higher than the classiﬁcation losses.

Original: [Figure]
Revised: [Figure]

Original: Mini-batch size is set to 64 , with 10 epochs.
Revised: Mini-batch size is set to 64 , with 10 epochs.

Original: The reconstruction losses are weighed 4 times lower than the classiﬁcation losses.
Revised: The reconstruction losses are weighed 4 times lower than the classiﬁcation losses.

Original: In this case, as opposed to the previous one, the classiﬁcation task is challenging enough to avoid early overﬁtting.
Revised: In this case, as opposed to the previous one, the classiﬁcation task is challenging enough to avoid early overﬁtting.

Original: [Figure]
Revised: [Figure]