Original: E XPLICIT I NFORMATION P LACEMENT ON L ATENT V ARIABLES USING A UXILIARY G ENERATIVE M 
Revised: E XPLICIT I NFORMATION P LACEMENT ON L ATENT V ARIABLES USING A UXILIARY G ENERATIVE M ODELLING T

Original: OD ELLING T ASK
Revised: ASK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep latent variable models, such as variational autoencoders, have been successfully used to disentangle factors of variation in image datasets.
Revised: Deep latent variable models, such as variational autoencoders, have been successfully used to disentangle factors of variation in image datasets.

Original: The structure of the representations learned by such models is usually observed after training and iteratively reﬁned by tuning the network architecture and loss function.
Revised: The structure of the representations learned by such models is usually observed after training and iteratively reﬁned by tuning the network architecture and loss function.

Original: Here we propose a method that can explicitly place information into a speciﬁc subset of the latent variables.
Revised: Here we propose a method that can explicitly place information into a speciﬁc subset of the latent variables.

Original: We demonstrate the use of the method in a task of disentangling global structure from local features in images.
Revised: We demonstrate the use of the method in a task of disentangling global structure from local features in images.

Original: One subset of the latent variables is encouraged to represent local features through an auxiliary modelling task.
Revised: One subset of the latent variables is encouraged to represent local features through an auxiliary modelling task.

Original: In this auxiliary task, the global structure of an image is destroyed by dividing it into pixel patches which are then randomly shufﬂed.
Revised: In this auxiliary task, the global structure of an image is destroyed by dividing it into pixel patches which are then randomly shufﬂed.

Original: The full set of latent variables is trained to model the original data, obliging the remainder of the latent representation to model the global structure.
Revised: The full set of latent variables is trained to model the original data, obliging the remainder of the latent representation to model the global structure.

Original: We demonstrate that this approach successfully disentangles the latent variables for global structure from local structure by observing the generative samples of SVHN and CIFAR10.
Revised: We demonstrate that this approach successfully disentangles the latent variables for global structure from local structure by observing the generative samples of SVHN and CIFAR10.

Original: We also clustering the disentangled global structure of SVHN and found that the emerging clusters represent meaningful groups of global structures – including digit identities and the number of digits presence.
Revised: We also clustering the disentangled global structure of SVHN and found that the emerging clusters represent meaningful groups of global structures – including digit identities and the number of digits presence.

Original: Finally, we discuss the problem of evaluating the clustering accuracy when ground truth categories are not expressive enough.
Revised: Finally, we discuss the problem of evaluating the clustering accuracy when ground truth categories are not expressive enough.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Prior knowledge can be imposed on a machine learning algorithm in many ways.
Revised: Prior knowledge can be imposed on a machine learning algorithm in many ways.

Original: One of the most important methods is through the construction of the neural network.
Revised: One of the most important methods is through the construction of the neural network.

Original: A notable example is the use of convolutional neural networks (CNN), where CNNs have shown to improve performance of deep neural networks in many image datasets due to its ability to impose priors about spatiality of the data.
Revised: A notable example is the use of convolutional neural networks (CNN), where CNNs have shown to improve performance of deep neural networks in many image datasets due to its ability to impose priors about spatiality of the data.

Original: The choice of loss function can also be seen as an imposition of prior knowledge, e.g. the square distance error is more suited for a continuous dataset than a discrete dataset.
Revised: The choice of loss function can also be seen as an imposition of prior knowledge, e.g. the square distance error is more suited for a continuous dataset than a discrete dataset.

Original: In this work, we propose a method that can impose prior knowledge onto the representation of latent variable models.
Revised: In this work, we propose a method that can impose prior knowledge onto the representation of latent variable models.

Original: In contrast to previous methods, the imposition of prior knowledge is done through the training of a model rather than the construction of a network or its loss function.
Revised: In contrast to previous methods, the imposition of prior knowledge is done through the training of a model rather than the construction of a network or its loss function.

Original: This method aims to improve the ability of representation learning algorithms with more explicit control over which information is represented at each latent variable.
Revised: This method aims to improve the ability of representation learning algorithms with more explicit control over which information is represented at each latent variable.

Original: Many disentangled representation learning methods, such as Variational Auto-Encoder (VAE), implicitly assign the meaning of each latent dimension after training.
Revised: Many disentangled representation learning methods, such as Variational Auto-Encoder (VAE), implicitly assign the meaning of each latent dimension after training.

Original: Since, the representation learned by VAEs is only enforced through the objective function encouraging factorised latent variables.
Revised: Since, the representation learned by VAEs is only enforced through the objective function encouraging factorised latent variables.

Original: Each latent dimension is only created through the optimisation dynamics which is unknown to the model designer.
Revised: Each latent dimension is only created through the optimisation dynamics which is unknown to the model designer.

Original: For example, in the case of the MNIST digit dataset, we have a prior knowledge that the dataset composes of digits with different orientations and thicknesses of strokes.
Revised: For example, in the case of the MNIST digit dataset, we have a prior knowledge that the dataset composes of digits with different orientations and thicknesses of strokes.

Original: However, it is not possible to explicitly assign the ﬁrst dimension of the latent variable to represent the orientation while forcing the second dimension to represent the thickness.
Revised: However, it is not possible to explicitly assign the ﬁrst dimension of the latent variable to represent the orientation while forcing the second dimension to represent the thickness.

Original: We can only observe the information placement after a training.
Revised: We can only observe the information placement after a training.

Original: If the information can be placed explicitly in the model then different types of prior knowledge can be imposed and, thus improving the model designing process.
Revised: If the information can be placed explicitly in the model then different types of prior knowledge can be imposed and, thus improving the model designing process.

Original: To this end, we show that explicit information placement can be made possible through the training setup which we call an auxiliary generative modelling task.
Revised: To this end, we show that explicit information placement can be made possible through the training setup which we call an auxiliary generative modelling task.

Original: We demonstrate the ability of this method through the task of explicitly placing global and local information onto different subsets of latent variables.
Revised: We demonstrate the ability of this method through the task of explicitly placing global and local information onto different subsets of latent variables.

Original: By doing so, global and local information can be disentangled.
Revised: By doing so, global and local information can be disentangled.

Original: We contend that disentangling this type of information is appropriate because it is important prior knowledge which is universally applicable to natural images.
Revised: We contend that disentangling this type of information is appropriate because it is important prior knowledge which is universally applicable to natural images.

Original: For example, a perceptual module of a robot would beneﬁt from learning a separation between the texture and identity of an object.
Revised: For example, a perceptual module of a robot would beneﬁt from learning a separation between the texture and identity of an object.

Original: We also propose a suite of experiments that aim to demonstrate that global and local information can be explicitly placed on subsets of the latent variables with our method.
Revised: We also propose a suite of experiments that aim to demonstrate that global and local information can be explicitly placed on subsets of the latent variables with our method.

Original: 1. D ISENTANGLING LOCAL AND GLOBAL STRUCTURE USING VAE
Revised: 1. D ISENTANGLING LOCAL AND GLOBAL STRUCTURE USING VAE

Original: It is well known that a deep convolutional neural network trained on an image recognition task will learn a hierarchy of features, where the lower convolutional layers correspond to local structure and the deeper layers near the classiﬁer neurons respond to higher level features (Gatys et al., 2015;
Revised: It is well known that a deep convolutional neural network trained on an image recognition task will learn a hierarchy of features, where the lower convolutional layers correspond to local structure and the deeper layers near the classiﬁer neurons respond to higher level features (Gatys et al., 2015;

Original: Olah et al., 2018).
Revised: Olah et al., 2018).

Original: This disentanglement emerges from the network architecture and the training objective.
Revised: This disentanglement emerges from the network architecture and the training objective.

Original: As the information is pushed through multiple layers of convolutional neurons, it is lossy compressed down to only a class identity in the ﬁnal output layer.
Revised: As the information is pushed through multiple layers of convolutional neurons, it is lossy compressed down to only a class identity in the ﬁnal output layer.

Original: In a latent variable model such as an autoencoder, however, information is processed differently.
Revised: In a latent variable model such as an autoencoder, however, information is processed differently.

Original: In contrast to a recognition model, a latent variable model cannot throw away most of the information in its input layer.
Revised: In contrast to a recognition model, a latent variable model cannot throw away most of the information in its input layer.

Original: Rather, it must reorganise this information into a compact representation which can be used to reconstruct the original data.
Revised: Rather, it must reorganise this information into a compact representation which can be used to reconstruct the original data.

Original: One way to control the structure of the representation, for example to promote disentanglement, is to adjust the objective functions which control the level of compression and redundancy in the latent variables (Higgins et al., 2017;
Revised: One way to control the structure of the representation, for example to promote disentanglement, is to adjust the objective functions which control the level of compression and redundancy in the latent variables (Higgins et al., 2017;

Original: Chen et al., 2018; Kim & Mnih, 2018;
Revised: Chen et al., 2018; Kim & Mnih, 2018;

Original: Esmaeili et al., 2018; Dilokthanakul et al., 2016).
Revised: Esmaeili et al., 2018; Dilokthanakul et al., 2016).

Original: Latent variable models (LVMs) assume that the data, x , is generated from a latent variable, z , and a generative process, p ( x | z ) .
Revised: Latent variable models (LVMs) assume that the data, x , is generated from a latent variable, z , and a generative process, p ( x | z ) .

Original: One way to ﬁt the LVM to the data is to restrict the generative process to a set of parameterised distributions p ✓ ( x | z ) and search for parameters ✓ that maximise the loglikelihood of the data, E p ( x ) [log p ( x | ✓ )] .
Revised: One way to ﬁt the LVM to the data is to restrict the generative process to a set of parameterised distributions p θ ( x | z ) and search for parameters θ that maximise the loglikelihood of the data, E p ( x ) [log p ( x | θ )] .

Original: A Variational Autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) is an LVM that parameterises its generative process p ✓ ( x | z ) and its variational posterior q  ( z | x ) with deep neural networks, ✓ ,  .
Revised: A Variational Autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) is an LVM that parameterises its generative process p θ ( x | z ) and its variational posterior q φ ( z | x ) with deep neural networks, θ, φ .

Original: The parameters of the neural networks are trained with stochastic gradient descent to maximise the variational free-energy objective.
Revised: The parameters of the neural networks are trained with stochastic gradient descent to maximise the variational free-energy objective.

Original: The variational free-energy objective is a lower bound to the log-likelihood of the data which can be written as
Revised: The variational free-energy objective is a lower bound to the log-likelihood of the data which can be written as

Original: [Equation]
Revised: [Equation]

Original: The VAE is one of the most successful latent variable models, and it has been shown to be effective at modelling image datasets.
Revised: The VAE is one of the most successful latent variable models, and it has been shown to be effective at modelling image datasets.

Original: It has also been shown to be capable of creating latent structure that is compositional and interpretable (Higgins et al., 2017; Kim & Mnih, 2018).
Revised: It has also been shown to be capable of creating latent structure that is compositional and interpretable (Higgins et al., 2017; Kim & Mnih, 2018).

Original: These desirable properties are thought to be the result of the free-energy objective, Eq.1, which also has the effect of encouraging the latent variables to be factorised.
Revised: These desirable properties are thought to be the result of the free-energy objective, Eq.1, which also has the effect of encouraging the latent variables to be factorised.

Original: Eq.
Revised: Eq.

Original: 1 consists of two terms known as the reconstruction cost and the KL cost.
Revised: 1 consists of two terms known as the reconstruction cost and the KL cost.

Original: Following Hoffman and Johnson (Hoffman & Johnson, 2016) and
Revised: Following Hoffman and Johnson (Hoffman & Johnson, 2016) and

Original: Chen et al.
Revised: Chen et al.

Original: (Chen et al., 2018), the KL term can be rewritten as
Revised: (Chen et al., 2018), the KL term can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: We can see from the second term in Eq.2 that the KL cost tries to reduce the total correlation of the latent variables, thus promoting factorised representation.
Revised: We can see from the second term in Eq.2 that the KL cost tries to reduce the total correlation of the latent variables, thus promoting factorised representation.

Original: However, in a complex dataset where images have a variety of textures as well as different global contexts, VAEs struggle to disentangle local structure from global structure and often focus their representations on the local structure which is easier to model.
Revised: However, in a complex dataset where images have a variety of textures as well as different global contexts, VAEs struggle to disentangle local structure from global structure and often focus their representations on the local structure which is easier to model.

Original: For example, Dilokthanakul et al.
Revised: For example, Dilokthanakul et al.

Original: (2016) had shown in an SVHN experiment that multi-modal versions of VAEs tend to cluster local structures into the same mode while ignoring the global context, i.e. disentanglement based on the
Revised: (2016) had shown in an SVHN experiment that multi-modal versions of VAEs tend to cluster local structures into the same mode while ignoring the global context, i.e. disentanglement based on the

Original: [Figure]
Revised: [Figure]

Original: background colours of the image.
Revised: background colours of the image.

Original: Importantly, it is not clear whether the factorised representation objective alone would be enough to disentangle this kind of multi-level structure.
Revised: Importantly, it is not clear whether the factorised representation objective alone would be enough to disentangle this kind of multi-level structure.

Original: Imposing hierarchical structure onto the VAE latent variables, where lower-level latent variables are generated from higher-level latent variables, might be able to disentangle multi-level structure.
Revised: Imposing hierarchical structure onto the VAE latent variables, where lower-level latent variables are generated from higher-level latent variables, might be able to disentangle multi-level structure.

Original: However, Zhao et al.
Revised: However, Zhao et al.

Original: (2017) argued that hierarchical VAE failed to use the higher latent variables and avoided this problem by only using one level of latent structure (ﬂat latent structure) and carefully creating a neural network architecture that has different levels of computation for each latent dimension.
Revised: (2017) argued that hierarchical VAE failed to use the higher latent variables and avoided this problem by only using one level of latent structure (ﬂat latent structure) and carefully creating a neural network architecture that has different levels of computation for each latent dimension.

Original: This method successfully disentangles local structure from global structure, but at the cost of needing to carefully tune the architecture, which is the process that is poorly understood.
Revised: This method successfully disentangles local structure from global structure, but at the cost of needing to carefully tune the architecture, which is the process that is poorly understood.

Original: Similarly, Chen et al.
Revised: Similarly, Chen et al.

Original: (2016) and Gulrajani et al.
Revised: (2016) and Gulrajani et al.

Original: (2016) showed that information placement can be controlled through the capacity of the decoder.
Revised: (2016) showed that information placement can be controlled through the capacity of the decoder.

Original: They used the pixelCNN decoder with varying sizes of receptive ﬁelds to control how much information the latent variables need to model.
Revised: They used the pixelCNN decoder with varying sizes of receptive ﬁelds to control how much information the latent variables need to model.

Original: However, their methods cannot represent local structures with latent variables, as the local structures will be captured via the pixelCNN, which is an autoregressive model.
Revised: However, their methods cannot represent local structures with latent variables, as the local structures will be captured via the pixelCNN, which is an autoregressive model.

Original:  M ETHODS 2. E XPLICIT PLACEMENT OF INFORMATION THROUGH AUXILIARY GENERATIVE MODELLING
Revised:  M ETHODS 2. E XPLICIT PLACEMENT OF INFORMATION THROUGH AUXILIARY GENERATIVE MODELLING

Original: In this section, we outline our method, which allows the explicit placement of information in each latent variable.
Revised: In this section, we outline our method, which allows the explicit placement of information in each latent variable.

Original: Let’s denote the original data as a stochastic variable x and a modiﬁed version of the data ˆ x .
Revised: Let’s denote the original data as a stochastic variable x and a modiﬁed version of the data ˆ x .

Original: ˆ x is created from a transformation of x in such a way that the information of interest is destroyed.
Revised: ˆ x is created from a transformation of x in such a way that the information of interest is destroyed.

Original: This transformation has to be crafted for different types of information, e.g. global information can be destroyed by shufﬂing patches of the images, or colour information is destroyed by conversion to grey scale.
Revised: This transformation has to be crafted for different types of information, e.g. global information can be destroyed by shufﬂing patches of the images, or colour information is destroyed by conversion to grey scale.

Original: Next, we create two sets of latent variables, z i and z r .
Revised: Next, we create two sets of latent variables, z i and z r .

Original: We aim to model the information of interest with z
Revised: We aim to model the information of interest with z

Original: i and the remainder with z r .
Revised: i and the remainder with z r .

Original: We assume a generative processes as follows.
Revised: We assume a generative processes as follows.

Original: [Equation]
Revised: [Equation]

Original: The process of generating x is the usual generative assumption in VAE.
Revised: The process of generating x is the usual generative assumption in VAE.

Original: In addition to x , ˆ x is assumed to be generated from z r which is shared with x .
Revised: In addition to x , ˆ x is assumed to be generated from z r which is shared with x .

Original: By also learning the generative model p ˆ ✓ (ˆ x ) , we can control the representation of z r by controlling the information contained in ˆ x .
Revised: By also learning the generative model p ˆ θ (ˆ x ) , we can control the representation of z r by controlling the information contained in ˆ x .

Original: In this case, ˆ x does not hold details about the information of interest and, therefore, forces z r to model the leftover information after transformation of x .
Revised: In this case, ˆ x does not hold details about the information of interest and, therefore, forces z r to model the leftover information after transformation of x .

Original: Due to the compression objective of VAE, the remaining variable z i is pressured to represent the information of interest in order to reduce redundancy in the latent variables.
Revised: Due to the compression objective of VAE, the remaining variable z i is pressured to represent the information of interest in order to reduce redundancy in the latent variables.

Original: This is what we refer to as an auxiliary modelling task, which explicitly places information of interest in a subset of the latent variables.
Revised: This is what we refer to as an auxiliary modelling task, which explicitly places information of interest in a subset of the latent variables.

Original: 2. D EFINITION OF GLOBAL AND LOCAL STRUCTURE OF IMAGES
Revised: 2. D EFINITION OF GLOBAL AND LOCAL STRUCTURE OF IMAGES

Original: In this work, we deﬁne notions of global and local information as follow:
Revised: In this work, we deﬁne notions of global and local information as follow:

Original: • Local information encapsulates the correlations between nearby pixels in an image.
Revised: • Local information encapsulates the correlations between nearby pixels in an image.

Original: • Global information encapsulates the correlations between pixels that are further away from each other.
Revised: • Global information encapsulates the correlations between pixels that are further away from each other.

Original: With these deﬁnitions, we construct a transformation procedure ˆ x = g ( x ) that would destroy the information of interest, in this case the global information, from x .
Revised: With these deﬁnitions, we construct a transformation procedure ˆ x = g ( x ) that would destroy the information of interest, in this case the global information, from x .

Original: Therefore, z i and z r are expected to represent the global information and the local information respectively.
Revised: Therefore, z i and z r are expected to represent the global information and the local information respectively.

Original: From now on, we will write z g for the latent variable modelling the global information and z l for the latent variable modelling local information.
Revised: From now on, we will write z g for the latent variable modelling the global information and z l for the latent variable modelling local information.

Original: We propose to destroy the global information by randomly shufﬂing patches of pixels in the image.
Revised: We propose to destroy the global information by randomly shufﬂing patches of pixels in the image.

Original: The shufﬂing transformation g ( · ) is done by ﬁrst dividing an image into m patches of n ⇥ n pixels.
Revised: The shufﬂing transformation g ( · ) is done by ﬁrst dividing an image into m patches of n × n pixels.

Original: Each patch is assigned an index number associated to it.
Revised: Each patch is assigned an index number associated to it.

Original: The indexes are randomly shufﬂed and then the patches are rearranged.
Revised: The indexes are randomly shufﬂed and then the patches are rearranged.

Original: This procedure has two effects: (i) the local correlations between pixels inside a patch are preserved and (ii) the long range correlations between pixels are destroyed through random rearrangement.
Revised: This procedure has two effects: (i) the local correlations between pixels inside a patch are preserved and (ii) the long range correlations between pixels are destroyed through random rearrangement.

Original: For example, in the SVHN dataset, we expect z l to model the colour scheme of the image because the colour correlations will be preserved regardless of the shufﬂing process.
Revised: For example, in the SVHN dataset, we expect z l to model the colour scheme of the image because the colour correlations will be preserved regardless of the shufﬂing process.

Original: If one pixel is blue it is more likely that nearby pixels are also blue.
Revised: If one pixel is blue it is more likely that nearby pixels are also blue.

Original: We also expect z g to represent the remaining correlations which should include the identity of the digits and their global styles.
Revised: We also expect z g to represent the remaining correlations which should include the identity of the digits and their global styles.

Original: In the next section, experiments are designed to measure the extent to which these expectations are met and, therefore how much the information can be explicitly placed on each latent variable using our method.
Revised: In the next section, experiments are designed to measure the extent to which these expectations are met and, therefore how much the information can be explicitly placed on each latent variable using our method.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: For our experiments, we used the following datasets:
Revised: For our experiments, we used the following datasets:

Original: SVHN (Netzer et al., 2011) consists of 604388 training and 26032 test 32x32x3 RGB images of street numbers obtained from Google street view.
Revised: SVHN (Netzer et al., 2011) consists of 604388 training and 26032 test 32x32x3 RGB images of street numbers obtained from Google street view.

Original: CIFAR

Revised: CIFAR

Original: (Krizhevsky, 2009) consists of 60000 32x32 colour images in 10 classes, with 6000 images 
Revised: (Krizhevsky, 2009) consists of 60000 32x32 colour images in 10 classes, with 6000 images per class including images of trucks, cats, horses

Original: per class including images of trucks, cats , horses, etc.
Revised: etc

Original: The aim of the experiments are to test whether the local and global information can be successfully disentangled which would provide an evidence that the proposed method can perform explicit information placement.
Revised: The aim of the experiments are to test whether the local and global information can be successfully disentangled which would provide an evidence that the proposed method can perform explicit information placement.

Original: 3. E XPERIMENT 1: D ISENTANGLEMENT IN A SIMPLE VAE
Revised: 3. E XPERIMENT 1: D ISENTANGLEMENT IN A SIMPLE VAE

Original: In Experiment 1, we trained a simple VAE with our auxiliary generative modelling task.
Revised: In Experiment 1, we trained a simple VAE with our auxiliary generative modelling task.

Original: This VAE models the data distribution of x and ˆ x with two sets of latent variables z g and z l ; p ( x, ˆ x ) =
Revised: This VAE models the data distribution of x and ˆ x with two sets of latent variables z g and z l ; p ( x, ˆ x ) =

Original: R z g ,z l p ( x, ˆ x | z g , z l ) p ( z g , z l ) dz g dz l .
Revised: (cid:82) z g ,z l p ( x, ˆ x | z g , z l ) p ( z g , z l ) dz g dz l .

Original: The variational posterior q  ( z g , z l | x, ˆ x ) is modelled as a deep convolutional neural network  .
Revised: The variational posterior q φ ( z g , z l | x, ˆ x ) is modelled as a deep convolutional neural network φ .

Original: The encoding process takes a pair of x and ˆ x as an input through the variational posterior (encoder)
Revised: The encoding process takes a pair of x and ˆ x as an input through the variational posterior (encoder)

Original: q  ( · ) and output diagonal Gaussian parameters ( µ z g ,  z g ) and ( µ z l ,  z l ) for q  .
Revised: q φ ( · ) and output diagonal Gaussian parameters ( µ z g , σ z g ) and ( µ z l , σ z l ) for q φ .

Original: Next, the samples z g , z l ⇠ q  ( z g , z l | x, ˆ x ) are pushed through the decoder networks p ✓ ( x | z g , z l ) and p ˆ ✓ (ˆ x | z l ) which are modelled as discretized logistic distributions (Kingma et al., 2016).
Revised: Next, the samples z g , z l ∼ q φ ( z g , z l | x, ˆ x ) are pushed through the decoder networks p θ ( x | z g , z l ) and p ˆ θ (ˆ x | z l ) which are modelled as discretized logistic distributions (Kingma et al., 2016).

Original: We then optimise  , ✓ and ˆ ✓ with the standard Monte-Carlo estimate of the free-energy objective,
Revised: We then optimise φ , θ and ˆ θ with the standard Monte-Carlo estimate of the free-energy objective,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where  is a hyperparameter adjusting the compression terms which had been shown to help improve disentanglement (Higgins et al., 2017).
Revised: where β is a hyperparameter adjusting the compression terms which had been shown to help improve disentanglement (Higgins et al., 2017).

Original: The prior p ( z l , z g ) is a unit diagonal Gaussian.
Revised: The prior p ( z l , z g ) is a unit diagonal Gaussian.

Original: The variational posterior q  ( z g , z l | x, ˆ x ) is assumed to be factorised such that q  ( z g , z l | x, ˆ x ) =
Revised: The variational posterior q φ ( z g , z l | x, ˆ x ) is assumed to be factorised such that q φ ( z g , z l | x, ˆ x ) =

Original: q  g ( z g | x ) q  l ( z l | ˆ x ) .
Revised: q φ g ( z g | x ) q φ l ( z l | ˆ x ) .

Original: This means we can use two separate neural network encoders  g and  l to encode the latent variables as shown in Fig 1.
Revised: This means we can use two separate neural network encoders φ g and φ l to encode the latent variables as shown in Fig 1.

Original: This choice of factorisation is chosen to prevent global information going through z l which means z l can only represent the information in ˆ x .
Revised: This choice of factorisation is chosen to prevent global information going through z l which means z l can only represent the information in ˆ x .

Original: Although this factorisation is useful, it is conceivable that z l could beneﬁt from amortising information from x by sharing an encoder.
Revised: Although this factorisation is useful, it is conceivable that z l could beneﬁt from amortising information from x by sharing an encoder.

Original: 3.1. E XPERIMENT 1.1: V ISUAL INSPECTION OF GENERATIVE SAMPLES
Revised: 3.1. E XPERIMENT 1.1: V ISUAL INSPECTION OF GENERATIVE SAMPLES

Original: One way to judge the quality of the learnt latent variables is to inspect samples generated from them at different values.
Revised: One way to judge the quality of the learnt latent variables is to inspect samples generated from them at different values.

Original: We examine the local latent variables z l by randomly sampling a point of z g ⇠ p ( z g ) and 100 points of z l ⇠ p ( z l ) .
Revised: We examine the local latent variables z l by randomly sampling a point of z g ∼ p ( z g ) and 100 points of z l ∼ p ( z l ) .

Original: We then generate 100 generative samples by passing these codes through the decoder.
Revised: We then generate 100 generative samples by passing these codes through the decoder.

Original: We repeat the same process for the global variables but with one point of z l and 100 points of z g .
Revised: We repeat the same process for the global variables but with one point of z l and 100 points of z g .

Original: As shown in Fig 2, varying z l results in variation in colours and brightness while z g dictates the object identity and global style.
Revised: As shown in Fig 2, varying z l results in variation in colours and brightness while z g dictates the object identity and global style.

Original: This shows that the method works as intended.
Revised: This shows that the method works as intended.

Original: Similar to the normal VAE, our generative samples of CIFAR10 images look relatively blurry and their object identity is hard to identify.
Revised: Similar to the normal VAE, our generative samples of CIFAR10 images look relatively blurry and their object identity is hard to identify.

Original: However, the disentanglement between colour and global structure can be observed.
Revised: However, the disentanglement between colour and global structure can be observed.

Revised: Several β values had been used and it had been observed that the disentanglement between global and local information is robust to different values of β .

Original: [Figure]
Revised: [Figure]

Original: 3.1. E XPERIMENT 1.2: Q UANTITATIVE INSPECTION OF DISENTANGLEMENT
Revised: 3.1. E XPERIMENT 1.2: Q UANTITATIVE INSPECTION OF DISENTANGLEMENT

Original: In order to show more quantitatively that the method can explicitly place global information in a subset of the latent variables, an experiment is carried out where an SVHN classiﬁer is trained to inspect the generative samples of the model.
Revised: In order to show more quantitatively that the method can explicitly place global information in a subset of the latent variables, an experiment is carried out where an SVHN classiﬁer is trained to inspect the generative samples of the model.

Original: The classiﬁer has an accuracy of 95% on the SVHN test set.
Revised: The classiﬁer has an accuracy of 95% on the SVHN test set.

Original: The SVHN test data is encoded into the latent space with the encoder of the model used in previous experiment with  = 1 . 0 .
Revised: The SVHN test data is encoded into the latent space with the encoder of the model used in previous experiment with β = 1 . 0 .

Original: Then, three types of samples are generated from the encodings: (i) images generated directly from the encoded latents, (ii) images in which z l is replaced with a random sample from N (0 , 1) while preserving z g , and (iii) images in which z g is replaced with a random sample from N (0 , 1) while preserving z l .
Revised: Then, three types of samples are generated from the encodings: (i) images generated directly from the encoded latents, (ii) images in which z l is replaced with a random sample from N (0 , 1) while preserving z g , and (iii) images in which z g is replaced with a random sample from N (0 , 1) while preserving z l .

Original: The result in Fig.
Revised: The result in Fig.

Original: 3 shows that: (i) the direct reconstruction slightly perturbs digit identity, yielding an accuracy of 86%, (ii) changing z l also slightly perturbs the digit identity, yielding 80% accuracy, (iii) while changing in z g completely changes the identity of the digits, reducing the accuracy to 11%.
Revised: 3 shows that: (i) the direct reconstruction slightly perturbs digit identity, yielding an accuracy of 86%, (ii) changing z l also slightly perturbs the digit identity, yielding 80% accuracy, (iii) while changing in z g completely changes the identity of the digits, reducing the accuracy to 11%.

Original: The difference between 80% and 11% (ie: chance) in (ii) and (iii) demonstrates quantitatively the disentangling we were aiming for.
Revised: The difference between 80% and 11% (ie: chance) in (ii) and (iii) demonstrates quantitatively the disentangling we were aiming for.

Original: Increasing  results in blurred reconstruction images and results in lower classiﬁcation accuracy.
Revised: Increasing β results in blurred reconstruction images and results in lower classiﬁcation accuracy.

Original: 3. E XPERIMENT 2: D ISENTANGLEMENT IN MULTI - MODAL VAE
Revised: 3. E XPERIMENT 2: D ISENTANGLEMENT IN MULTI - MODAL VAE

Original: In previous experiments, we have shown that changing z l only slightly effects the identity of digit while changing z g signiﬁcantly altered the digit identity.
Revised: In previous experiments, we have shown that changing z l only slightly effects the identity of digit while changing z g signiﬁcantly altered the digit identity.

Original: In this experiment, we would like to inspect  the content of z g in more details, as well as demonstrating the use of our method in an unsupervised clustering task.
Revised: In this experiment, we would like to inspect [Figure] the content of z g in more details, as well as demonstrating the use of our method in an unsupervised clustering task.

Original: The model in experiment 1 is modiﬁed with an additional discrete latent variable, y , resulting in the following generative process:
Revised: The model in experiment 1 is modiﬁed with an additional discrete latent variable, y , resulting in the following generative process:

Original: [Equation]
Revised: [Equation]

Original: The variational posterior is assumed to be factorised as q 
Revised: The variational posterior is assumed to be factorised as q φ ( y, z g , z l | x, ˆ x ) = q φ g ( y, z g | x ) q φ

Original: [Equation]
Revised: [Equation]

Original:  with diagonal Gaussians as the posteriors of continuous variables z g , z l and a Gumbel-Softmax (Concrete) distribution (Maddison et al., 2017; Jang et al., 2017) for the class variable y with constant temperature ⌧ .
Revised: z [Equation] | ˆ x ) with diagonal Gaussians as the posteriors of continuous variables z g , z l and a Gumbel-Softmax (Concrete) distribution (Maddison et al., 2017; Jang et al., 2017) for the class variable y with constant temperature τ .

Original: Finally, we optimise the free-energy objective:
Revised: Finally, we optimise the free-energy objective:

Original: [Equation]
Revised: [Equation]

Original:  where  is the compression pressure on latent variable z l and z g .
Revised: , (11) where β is the compression pressure on latent variable z l and z g .

Original: 3.2. E XPERIMENT 2.1: D IGIT C LUSTERING ON THE GLOBAL VARIABLE
Revised: 3.2. E XPERIMENT 2.1: D IGIT C LUSTERING ON THE GLOBAL VARIABLE

Original: We evaluated the model in the digit’s identity clustering task.
Revised: We evaluated the model in the digit’s identity clustering task.

Original: This is a standard evaluation metric for clustering methods.
Revised: This is a standard evaluation metric for clustering methods.

Original: The model is evaluated with the clustering accuracy (ACC) on the test set after training and hyper-parameter tuning on the training set.
Revised: The model is evaluated with the clustering accuracy (ACC) on the test set after training and hyper-parameter tuning on the training set.

Original: The training and evaluation are repeated with the same hyper-parameter setting for four runs.
Revised: The training and evaluation are repeated with the same hyper-parameter setting for four runs.

Original: ACC calculates the accuracy by assigning the best ground truth label for each cluster and, thus, allows for a larger number of predicted cluster assignments than ground truth labels
Revised: ACC calculates the accuracy by assigning the best ground truth label for each cluster and, thus, allows for a larger number of predicted cluster assignments than ground truth labels.

Original: [Table]
Revised: β

Original: [Table] Interestingly, digit ”1” uses 7 clusters which is much more than other digit classes. This is likely to be the results of distractors in images with digit ”1” present. There is a clear confusion between digit ”5” and ”6” which are sharing some of the clusters.
Revised: = 40 was searched from β ∈ { 1 , 10 , 20 , 30 , 40 , 50 , 60 } for the highest digit’s identity clustering result.

Original: As shown in Table 1, the method achieves a competitive score to dedicated clustering algorithms when clustering SVHN identities.
Revised: As shown in Table 1, the method achieves a competitive score to dedicated clustering algorithms when clustering SVHN identities.

Original: This supports earlier results that the local structure can be disentangled away with an auxiliary task.
Revised: This supports earlier results that the local structure can be disentangled away with an auxiliary task.

Original: However, there are many ways to cluster SVHN’s global image structure which can be much richer than the identity of the middle digits and, as such, this evaluation benchmark cannot be used to fairly compare the ability of clustering algorithms. 
Revised: However, there are many ways to cluster SVHN’s global image structure which can be much richer than the identity of the middle digits and, as such, this evaluation benchmark cannot be used to fairly compare the ability of clustering algorithms. [Table]

Original: Although z g is encouraged to contain global structure, it is not given that the latent y will hold the information about the digit class rather than encoding it in z g .
Revised: Although z g is encouraged to contain global structure, it is not given that the latent y will hold the information about the digit class rather than encoding it in z g .

Original: Some SVHN data points have ‘distracting’ digits on the sides of the digit of interest (middle digit).
Revised: Some SVHN data points have ‘distracting’ digits on the sides of the digit of interest (middle digit).

Original: It makes sense that some clusters y could be based on whether or not an image has distracting digits.
Revised: It makes sense that some clusters y could be based on whether or not an image has distracting digits.

Original: In fact, these kinds of clusters were frequently observed.
Revised: In fact, these kinds of clusters were frequently observed.

Original: Fig 5 shows the clusters that are based on the presence of distracting digits.
Revised: Fig 5 shows the clusters that are based on the presence of distracting digits.

Original: This result shows that (i) z g mainly contains the information about digit information and the number of digits in an image, which are global information, and (ii) it suggests that the way the unsupervised clustering accuracy metrics are usually reported has a ﬂaw which is not discussed in earlier deep clustering works.
Revised: This result shows that (i) z g mainly contains the information about digit information and the number of digits in an image, which are global information, and (ii) it suggests that the way the unsupervised clustering accuracy metrics are usually reported has a ﬂaw which is not discussed in earlier deep clustering works.

Original: The architecture is normally tuned to achieve the wanted clustering preference rather than relying on robust information placement.
Revised: The architecture is normally tuned to achieve the wanted clustering preference rather than relying on robust information placement.

Original: This problem presents itself when the ground truth labels are not expressive enough.
Revised: This problem presents itself when the ground truth labels are not expressive enough.

Original: 3.2. E XPERIMENT 2.2: V ISUALISING THE GENERATIVE SAMPLES
Revised: 3.2. E XPERIMENT 2.2: V ISUALISING THE GENERATIVE SAMPLES

Original: Next, we inspect the latent variables z g and y of a chosen architecture.
Revised: Next, we inspect the latent variables z g and y of a chosen architecture.

Original: As shown in Fig 6, we found that z g represents the digit’s global style, e.g. orientation and stroke width, while y represents the digit identity and whether or not the digit is darker than the background colour.
Revised: As shown in Fig 6, we found that z g represents the digit’s global style, e.g. orientation and stroke width, while y represents the digit identity and whether or not the digit is darker than the background colour.

Original: These clusters are interesting because they represent the global colour style, in contrast to the local colour style modelled with z l .
Revised: These clusters are interesting because they represent the global colour style, in contrast to the local colour style modelled with z l .

Revised: This visualisation further conﬁrm that z g and y tend to model global information which supports our hypothesis that the method can be used to explicitly place information onto a subset of the latent variables.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Several specialised clustering systems have attempted to cluster the SVHN dataset according to digit identity (Hu et al., 2017; Kilinc & Uysal, 2018).
Revised: Several specialised clustering systems have attempted to cluster the SVHN dataset according to digit identity (Hu et al., 2017; Kilinc & Uysal, 2018).

Original: However, there is no clear justiﬁcation for why these algorithms should cluster digit identity rather than the background colour or texture.
Revised: However, there is no clear justiﬁcation for why these algorithms should cluster digit identity rather than the background colour or texture.

Original: It is also not clear how much architectural tuning is needed to bias the clustering preference of these algorithms.
Revised: It is also not clear how much architectural tuning is needed to bias the clustering preference of these algorithms.

Original: Notably, an architecture called TAGGER (Greff et al., 2016) can learn a disentangled representation of background and objects by explicitly imposing an assumption on the network architecture that a representation of an object consists of a linear combination of location masks and texture patterns.
Revised: Notably, an architecture called TAGGER (Greff et al., 2016) can learn a disentangled representation of background and objects by explicitly imposing an assumption on the network architecture that a representation of an object consists of a linear combination of location masks and texture patterns.

Original: However, TAGGER separates objects from background at the pixel level and has not been shown to scale-up to a bigger and more natural dataset which might require separation at the level of the latent space.
Revised: However, TAGGER separates objects from background at the pixel level and has not been shown to scale-up to a bigger and more natural dataset which might require separation at the level of the latent space.

Original: The topic of disentangled representations has gained plenty of recent attention from researchers, with much of the effort directed at studying latent variable models such as VAEs.
Revised: The topic of disentangled representations has gained plenty of recent attention from researchers, with much of the effort directed at studying latent variable models such as VAEs.

Original: Much of this work tries to dissect the objective of the VAE (Hoffman & Johnson, 2016; Chen et al., 2018; Esmaeili et al., 2018) and proposes modiﬁed objectives that emphasise different aspects of the latent structure.
Revised: Much of this work tries to dissect the objective of the VAE (Hoffman & Johnson, 2016; Chen et al., 2018; Esmaeili et al., 2018) and proposes modiﬁed objectives that emphasise different aspects of the latent structure.

Original: However, using objective modiﬁcations alone, it is difﬁcult to scale disentanglement to more challenging datasets.
Revised: However, using objective modiﬁcations alone, it is difﬁcult to scale disentanglement to more challenging datasets.

Original: The closely related work by Zhao et al.
Revised: The closely related work by Zhao et al.

Original: (2017) tackled the same problem of multi-level structure disentanglement with VAEs.
Revised: (2017) tackled the same problem of multi-level structure disentanglement with VAEs.

Original: In their work, the disentanglement is achieved through careful architectural design, where more abstract information goes through more computational layers.
Revised: In their work, the disentanglement is achieved through careful architectural design, where more abstract information goes through more computational layers.

Original: The information placement is very sensitive to the architectural design and needs an iterative design process.
Revised: The information placement is very sensitive to the architectural design and needs an iterative design pro- cess.

Original: Our method achieves similar disentanglement through an alternative route, with more explicit control on what information is to be placed on speciﬁc latent variables.
Revised: Our method achieves similar disentanglement through an alternative route, with more explicit control on what information is to be placed on speciﬁc latent variables.

Original: The Attend Infer Repeat (AIR) model (Eslami et al., 2016) is another example of information placement using architectural design.
Revised: The Attend Infer Repeat (AIR) model (Eslami et al., 2016) is another example of information placement using architectural design.

Original: AIR has achieved the disentanglement of object’s identity, location and quantity through the use of specialised encoders and decoders.
Revised: AIR has achieved the disentanglement of object’s identity, location and quantity through the use of specialised encoders and decoders.

Original: While AIR has not been able to be scaled up to more complex datasets such as multi-digit SVHN, it would be interesting to combined the ability to disentangle global structure of our method with AIR’s ability to count and identify objects.
Revised: While AIR has not been able to be scaled up to more complex datasets such as multi-digit SVHN, it would be interesting to combined the ability to disentangle global structure of our method with AIR’s ability to count and identify objects.

Original: This is left for future work.
Revised: This is left for future work.

Revised: DC-IGN (Kulkarni et al., 2015) can perform explicit information placement through the training process.

Revised: However, the method need the knowledge of the feature variations in batches of data (similar to having access to the labels of the data).

Revised: Similar to our method, Tranforming autoencoders (Hinton et al., 2011) uses transformation of original images as self-supervision signals.

Original:   C ONCLUSIONS
Revised: However, the goal of the method is to learn the factor corresponding to the transformation. Implicit Autoencoder (Makhzani, 2018) shows that adversarial regularisation can force a latent variable to model only global information as the information has to be as compact as possible. This work shows a possibility of implicitly imposing inductive bias in the algorithmic design.  C ONCLUSIONS

Original: In this paper, we proposed a training method which explicitly controls where the information is to be placed in the latent variables of a VAE.
Revised: In this paper, we proposed a training method which explicitly controls where the information is to be placed in the latent variables of a VAE.

Original: We demonstrated the ability of the method through the task of disentangling local and global information.
Revised: We demonstrated the ability of the method through the task of disentangling local and global information.

Original: We showed that the latent variables are successfully disentangled by observing generative samples of the SVHN and CIFAR10 datasets.
Revised: We showed that the latent variables are successfully disentangled by observing generative samples of the SVHN and CIFAR10 datasets.

Original: This experiment also shows that the local structure and global structure of generated images can be controlled by varying the local and global latent codes.
Revised: This experiment also shows that the local structure and global structure of generated images can be controlled by varying the local and global latent codes.

Original: Next, we attempted to quantify the quality of the global latent variable through SVHN clustering.
Revised: Next, we attempted to quantify the quality of the global latent variable through SVHN clustering.

Original: We inspected the clusters of the global variable and observed that it represents global information such as digit identity, number of digits and the global colour style.
Revised: We inspected the clusters of the global variable and observed that it represents global information such as digit identity, number of digits and the global colour style.

Original: In addition, we observed that the local colour style is modelled with the local variables as expected.
Revised: In addition, we observed that the local colour style is modelled with the local variables as expected.

Original: These experiments give substantial evidence that our method can be used to explicitly place global and local information onto the subset of the latent variables.
Revised: These experiments give substantial evidence that our method can be used to explicitly place global and local information onto the subset of the latent variables.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Acknowledgement will only be visible in unanonymised version.
Revised: Acknowledgement will only be visible in unanonymised version