Original: G RADIENT O RIGIN N ETWORKS
Revised: G RADIENT O RIGIN N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper proposes a new type of generative model that is able to quickly learn a latent representation without an encoder.
Revised: This paper proposes a new type of generative model that is able to quickly learn a latent representation without an encoder.

Original: This is achieved using empirical Bayes to calculate the expectation of the posterior, which is implemented by initialising a latent vector with zeros, then using the gradient of the log-likelihood of the data with respect to this zero vector as new latent points.
Revised: This is achieved using empirical Bayes to calculate the expectation of the posterior, which is implemented by initialising a latent vector with zeros, then using the gradient of the log-likelihood of the data with respect to this zero vector as new latent points.

Original: The approach has similar characteristics to autoencoders, but with a simpler architecture, and is demonstrated in a variational autoencoder equivalent that permits sampling.
Revised: The approach has similar characteristics to autoencoders, but with a simpler architecture, and is demonstrated in a variational autoencoder equivalent that permits sampling.

Original: This also allows implicit representation networks to learn a space of implicit functions without requiring a hypernetwork, retaining their representation advantages across datasets.
Revised: This also allows implicit representation networks to learn a space of implicit functions without requiring a hypernetwork, retaining their representation advantages across datasets.

Original: The experiments show that the proposed method converges faster, with signiﬁcantly lower reconstruction error than autoencoders, while requiring half the parameters
Revised: The experiments show that the proposed method converges faster, with signiﬁcantly lower reconstruction error than autoencoders, while requiring half the parameters

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Observable data in nature has some parameters which are known, such as local coordinates, but also some unknown parameters such as how the data is related to other examples.
Revised: Observable data in nature has some parameters which are known, such as local coordinates, but also some unknown parameters such as how the data is related to other examples.

Original: Generative models, which learn a distribution over observables, are central to our understanding of patterns in nature and allow for efﬁcient query of new unseen examples.
Revised: Generative models, which learn a distribution over observables, are central to our understanding of patterns in nature and allow for efﬁcient query of new unseen examples.

Original: Recently, deep generative models have received interest due to their ability to capture a broad set of features when modelling data distributions.
Revised: Recently, deep generative models have received interest due to their ability to capture a broad set of features when modelling data distributions.

Original: As such, they offer direct applications such as synthesising high ﬁdelity images (Karras et al., 2020), super-resolution (Dai et al., 2019), speech synthesis (Li et al., 2019), and drug discovery (Segler et al., 2018), as well as beneﬁts for downstream tasks like semi-supervised learning (Chen et al., 2020).
Revised: As such, they offer direct applications such as synthesising high ﬁdelity images (Karras et al., 2020), super-resolution (Dai et al., 2019), speech synthesis (Li et al., 2019), and drug discovery (Segler et al., 2018), as well as beneﬁts for downstream tasks like semi-supervised learning (Chen et al., 2020).

Original: A number of methods have been proposed such as Variational Autoencoders (VAEs, Figure 1a), which learn to encode the data to a latent space that follows a normal distribution permitting sampling (Kingma & Welling, 2014).
Revised: A number of methods have been proposed such as Variational Autoencoders (VAEs, Figure 1a), which learn to encode the data to a latent space that follows a normal distribution permitting sampling (Kingma & Welling, 2014).

Original: Generative Adversarial Networks (GANs) have two competing networks, one which generates data and another which discriminates from implausible results (Goodfellow et al., 2014).
Revised: Generative Adversarial Networks (GANs) have two competing networks, one which generates data and another which discriminates from implausible results (Goodfellow et al., 2014).

Original: Variational approaches that approximate the posterior using gradient descent (Lipton & Tripathi, 2017) and short run MCMC (Nijkamp et al., 2020) respectively have been proposed, but to obtain a latent vector for a sample, they require iterative gradient updates.
Revised: Variational approaches that approximate the posterior using gradient descent (Lipton & Tripathi, 2017) and short run MCMC (Nijkamp et al., 2020) respectively have been proposed, but to obtain a latent vector for a sample, they require iterative gradient updates.

Original: Autoregressive
Revised: Autoregressive

Original: Models (Van Den Oord et al., 2016) decompose the data distribution as the product of conditional distributions and Normalizing Flows (Rezende & Mohamed, 2015) chain together invertible functions; both methods allow exact likelihood inference.
Revised: Models (Van Den Oord et al., 2016) decompose the data distribution as the product of conditional distributions and Normalizing Flows (Rezende & Mohamed, 2015) chain together invertible functions; both methods allow exact likelihood inference.

Original: Energy-Based Models (EBMs) map data points to energy values proportional to likelihood thereby permitting sampling through the use of Monte Carlo
Revised: Energy-Based Models (EBMs) map data points to energy values proportional to likelihood thereby permitting sampling through the use of Monte Carlo

Original: Markov Chains (Du & Mordatch, 2019).
Revised: Markov Chains (Du & Mordatch, 2019).

Original: In general to support encoding, these approaches require separate encoding networks, are limited to invertible functions, or require multiple sampling steps.
Revised: In general to support encoding, these approaches require separate encoding networks, are limited to invertible functions, or require multiple sampling steps.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Implicit representation learning (Park et al., 2019; Tancik et al., 2020), where a network is trained on data parameterised continuously rather than in discrete grid form, has seen a surge of interest due to the small number of parameters, speed of convergence, and ability to model ﬁne details.
Revised: Implicit representation learning (Park et al., 2019; Tancik et al., 2020), where a network is trained on data parameterised continuously rather than in discrete grid form, has seen a surge of interest due to the small number of parameters, speed of convergence, and ability to model ﬁne details.

Original: In particular, sinusoidal representation networks (SIRENs) (Sitzmann et al., 2020b) achieve impressive results, modelling complex signals with high precision, thanks to their use of periodic activations paired with carefully initialised MLPs.
Revised: In particular, sinusoidal representation networks (SIRENs) (Sitzmann et al., 2020b) achieve impressive results, modelling complex signals with high precision, thanks to their use of periodic activations paired with carefully initialised MLPs.

Original: So far, however, these models have been limited to modelling single data samples, or use an additional hypernetwork or meta learning (Sitzmann et al., 2020a) to estimate the weights of a simple implicit model, which adds signiﬁcant complexity.
Revised: So far, however, these models have been limited to modelling single data samples, or use an additional hypernetwork or meta learning (Sitzmann et al., 2020a) to estimate the weights of a simple implicit model, which adds signiﬁcant complexity.

Original: This paper proposes Gradient Origin Networks (GONs), which are a new type of generative model (Figure 1b) that do not require encoders or additional hypernetworks.
Revised: This paper proposes Gradient Origin Networks (GONs), which are a new type of generative model (Figure 1b) that do not require encoders or additional hypernetworks.

Original: This is achieved by initialising latent points at the origin, then using the gradient of the log-likelihood of the data with respect to these points as the latent space.
Revised: This is achieved by initialising latent points at the origin, then using the gradient of the log-likelihood of the data with respect to these points as the latent space.

Original: At inference, the latent vector can be obtained in a single step without requiring iteration; GONs are shown to have similar characteristics to convolutional autoencoders and variational autoencoders while requiring approximately half the parameters.
Revised: At inference, the latent vector can be obtained in a single step without requiring iteration; GONs are shown to have similar characteristics to convolutional autoencoders and variational autoencoders while requiring approximately half the parameters.

Original: This is also shown to work well at learning a space of implicit functions using implicit representation networks (such as
Revised: This is also shown to work well at learning a space of implicit functions using implicit representation networks (such as

Original: SIRENs), resulting in a simpler overall architecture.
Revised: SIRENs), resulting in a simpler overall architecture.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We ﬁrst introduce some preliminary concepts that will be used to derive our proposed approach.
Revised: We ﬁrst introduce some background context that will be used to derive our proposed approach.

Original: 2. E MPIRICAL B AYES
Revised: 2. E MPIRICAL B AYES

Original: The concept of empirical Bayes (Robbins, 1956; Saremi & Hyvarinen, 2019), for some random variable z ∼ p( z) and particular observation z 0 ∼ p( z 0) , provides an estimator of z expressed purely in terms of p ( z 0 ) that minimises the expected squared error.
Revised: The concept of empirical Bayes (Robbins, 1956; Saremi & Hyvarinen, 2019), for some random variable z ∼ p z and particular observation z 0 ∼ p z 0 , provides an estimator of z expressed purely in terms of p ( z 0 ) that minimises the expected squared error.

Original: This Bayes least squares estimator can be written as a conditional mean:
Revised: This Bayes least squares estimator can be written as a conditional mean:

Original: [Equation]
Revised: [Equation]

Original: Of particular relevance is the case where z 0 is a noisy observation of z with covariance Σ .
Revised: Of particular relevance is the case where z 0 is a noisy observation of z with covariance Σ .

Original: In this case p ( z 0 ) can be represented by marginalising out z :
Revised: In this case p ( z 0 ) can be represented by marginalising out z :

Original: [Equation]
Revised: [Equation]

Original: Differentiating this with respect to z 0 and multiplying both sides by Σ gives:
Revised: Differentiating this with respect to z 0 and multiplying both sides by Σ gives:

Original: [Equation]
Revised: [Equation]

Original: After dividing through by p ( z 0 ) and combining with Equation 

Revised: After dividing through by p ( z 0 ) and combining with Equation

Original: we obtain a closed form estimator of z (Miyasawa, 1961) written in terms of the score function ∇ log p ( z 0 ) (Hyvärinen, 2005):
Revised: we obtain a closed form estimator of z (Miyasawa, 1961) written in terms of the score function ∇ log p ( z 0 ) (Hyvärinen, 2005):

Original: [Equation]
Revised: [Equation]

Original: This optimal procedure is achieved in what can be interpreted as a single gradient descent step, with no knowledge of the prior p ( z ) .
Revised: This optimal procedure is achieved in what can be interpreted as a single gradient descent step, with no knowledge of the prior p ( z ) .

Original: By rearranging Equation 4, a deﬁnition of ∇ log p ( z 0 ) can be derived; this can be used to train models that approximate the score function (Song & Ermon, 2019).
Revised: By rearranging Equation 4, a deﬁnition of ∇ log p ( z 0 ) can be derived; this can be used to train models that approximate the score function (Song & Ermon, 2019).

Original: 2. V ARIATIONAL A UTOENCODERS
Revised: V ARIATIONAL A UTOENCODERS

Original: Variational Autoencoders (VAEs; Kingma & Welling 2014) are a probabilistic take on standard autoencoders that permit sampling.
Revised: Variational Autoencoders (VAEs; Kingma & Welling 2014) are a probabilistic take on standard autoencoders that permit sampling.

Original: A latent-based generative model p θ ( x | z ) is deﬁned with a normally distributed prior over the latent variables, p θ ( z ) =
Revised: A latent-based generative model p θ ( x | z ) is deﬁned with a normally distributed prior over the latent variables, p θ ( z ) =

Original: N ( z ; 0 , I d ) .
Revised: N ( z ; 0 , I d ) .

Original: p θ ( x | z ) is typically parameterised as a Bernoulli, Gaussian, multinomial distribution, or mixture of logits.
Revised: p θ ( x | z ) is typically parameterised as a Bernoulli, Gaussian, multinomial distribution, or mixture of logits.

Original: In this case, the true posterior p θ ( z | x ) is intractable, so a secondary encoding network q φ ( z | x ) is used to approximate the true posterior; the pair of networks thus resembles a traditional autoencoder.
Revised: In this case, the true posterior p θ ( z | x ) is intractable, so a secondary encoding network q φ ( z | x ) is used to approximate the true posterior; the pair of networks thus resembles a traditional autoencoder.

Original: This allows VAEs to approximate p θ ( x ) by maximising the evidence lower bound (ELBO), deﬁned as:
Revised: This allows VAEs to approximate p θ ( x ) by maximising the evidence lower bound (ELBO), deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: To optimise this lower bound with respect to θ and φ , gradients must be backpropagated through the stochastic process of generating samples from z (cid:48) ∼ q φ ( z | x ) .
Revised: To optimise this lower bound with respect to θ and φ , gradients must be backpropagated through the stochastic process of generating samples from z (cid:48) ∼ q φ ( z | x ) .

Original: This is permitted by reparameterising z using the differentiable function z (cid:48)
Revised: This is permitted by reparameterising z using the differentiable function z (cid:48)

Original: = µ ( z ) + σ ( z ) (cid:12) (cid:15) , where (cid:15) ∼ N ( 0 , I d ) and µ ( z ) and σ ( z ) 2 are the mean and variance respectively of a multivariate Gaussian distribution with diagonal covariance.
Revised: = µ ( z ) + σ ( z ) (cid:12) (cid:15) , where (cid:15) ∼ N ( 0 , I d ) and µ ( z ) and σ ( z ) 2 are the mean and variance respectively of a multivariate Gaussian distribution with diagonal covariance.

Original:  M ETHOD
Revised:  M ETHOD

Original: Consider some dataset x ∼ p d of continuous or discrete signals x ∈ R m , it is typical to assume that the data can be represented by low dimensional latent variables z ∈ R k , which can be used by a generative neural network to reconstruct the data.
Revised: Consider some dataset x ∼ p d of continuous or discrete signals x ∈ R m , it is typical to assume that the data can be represented by low dimensional latent variables z ∈ R k , which can be used by a generative neural network to reconstruct the data.

Original: These variables are often estimated through the use of a secondary encoding network that is trained concurrently with the generative network.
Revised: These variables are often estimated through the use of a secondary encoding network that is trained concurrently with the generative network.

Original: An encoding network adds additional complexity (and parameters) to the model, it can be difﬁcult to balance capacities of the two networks, and for complex hierarchical generative models designing a suitable architecture can be difﬁcult.
Revised: An encoding network adds additional complexity (and parameters) to the model, it can be difﬁcult to balance capacities of the two networks, and for complex hierarchical generative models designing a suitable architecture can be difﬁcult.

Original: This has led some to instead approximate latent variables by performing gradient descent on the generative network (Bojanowski et al., 2018; Nijkamp et al., 2020).
Revised: This has led some to instead approximate latent variables by performing gradient descent on the generative network (Bojanowski et al., 2018; Nijkamp et al., 2020).

Original: While this addresses the aforementioned problems, it signiﬁcantly increases the run time of the inference process, introduces additional hyperparameters to tune, and convergence is not guaranteed.
Revised: While this addresses the aforementioned problems, it signiﬁcantly increases the run time of the inference process, introduces additional hyperparameters to tune, and convergence is not guaranteed.

Original: 3. G RADIENT O RIGIN N ETWORKS
Revised: 3. G RADIENT O RIGIN N ETWORKS

Original: We propose a generative model that consists only of a decoding network, using empirical Bayes to approximate the posterior in a single step.
Revised: We propose a generative model that consists only of a decoding network, using empirical Bayes to approximate the posterior in a single step.

Original: That is, for some data point x and latent variable z ∼ p
Revised: That is, for some data point x and latent variable z ∼ p

Original: z , we wish to ﬁnd an approximation of p ( z | x ) .
Revised: z , we wish to ﬁnd an approximation of p ( z | x ) .

Original: Given some noisy observation z 0 =
Revised: Given some noisy observation z 0 =

Original: z + N ( 0 , I d ) of z then empirical Bayes can be applied to approximate z .
Revised: z + N ( 0 , I d ) of z then empirical Bayes can be applied to approximate z .

Original: Speciﬁcally, since we wish to approximate z conditioned on x , we instead calculate ˆ z x , the least squares estimate of p ( z | x ) (proof in Appendix A):
Revised: Speciﬁcally, since we wish to approximate z conditioned on x , we instead calculate ˆ z x , the least squares estimate of p ( z | x ) (proof in Appendix A):

Original: [Equation]
Revised: [Equation]

Original: Using Bayes’ rule, log p ( z 0 | x ) can be written as log p ( z 0 | x ) = log p ( x | z 0 ) + log p ( z 0 ) − log p ( x ) .
Revised: Using Bayes’ rule, log p ( z 0 | x ) can be written as log p ( z 0 | x ) = log p ( x | z 0 ) + log p ( z 0 ) − log p ( x ) .

Original: Since log p ( x ) is a normalising constant that does not affect the gradient, we can rewrite Equationin terms only of the decoding network and p ( z 0 ) :
Revised: Since log p ( x ) is a normalising constant that does not affect the gradient, we can rewrite Equationin terms only of the decoding network and p ( z 0 ) :

Original: [Equation]
Revised: [Equation]

Original: It still remains, however, how to construct a noisy estimate of z 0 with no knowledge of z .
Revised: It still remains, however, how to construct a noisy estimate of z 0 with no knowledge of z .

Original: If we assume z follows a known distribution, then it is possible to develop reasonable estimates.
Revised: If we assume z follows a known distribution, then it is possible to develop reasonable estimates.

Original: For instance, if we assume p ( z ) =
Revised: For instance, if we assume p ( z ) =

Original: N ( z ; 0 , I d ) then we could sample from p ( z 0 ) =
Revised: N ( z ; 0 , I d ) then we could sample from p ( z 0 ) =

Original: N ( z 0 ; 0 , 2 I d ) however this could be far from the true distribution of p ( z 0 | z )
Revised: N ( z 0 ; 0 , 2 I d ) however this could be far from the true distribution of p ( z 0 | z )

Original: =
Revised: =

Original: N ( z 0 ; z , I d ) .
Revised: N ( z 0 ; z , I d ) .

Original: Instead we propose initialising z 0 at the origin since this is the distribution
Revised: Instead we propose initialising z 0 at the origin since this is the distribution

Original: ’s mean.
Revised: ’s mean.

Original: Initialising at a constant position also decreases the input variation and thus simpliﬁes the optimisation procedure.
Revised: Initialising at a constant position also decreases the input variation and thus simpliﬁes the optimisation procedure.

Original: 3. A UTOENCODING WITH GON S
Revised: 3. A UTOENCODING WITH GON S

Original: Before exploring GONs as generative models, we discuss the case where the prior p ( z ) is unknown; such a model is referred to as an autoencoder.
Revised: Before exploring GONs as generative models, we discuss the case where the prior p ( z ) is unknown; such a model is referred to as an autoencoder.

Original: As such, the distribution p ( z 0 | z ) is also unknown thus it is again unclear how we can construct a noisy estimate of z .
Revised: As such, the distribution p ( z 0 | z ) is also unknown thus it is again unclear how we can construct a noisy estimate of z .

Original: By training a model end-to-end where z 0 is chosen as the origin, however, a prior is implicitly learned over z such that it is reachable from z 0 .
Revised: By training a model end-to-end where z 0 is chosen as the origin, however, a prior is implicitly learned over z such that it is reachable from z 0 .

Original: Although p ( z ) unknown, we do not wish to impose a prior on z ; the term which enforces this is in
Revised: Although p ( z ) unknown, we do not wish to impose a prior on z ; the term which enforces this is in

Original: Equation 7 is log p ( z 0 ) , so we can safely ignore this term and simply maximise the likelihood of the data given z 0 .
Revised: Equation 7 is log p ( z 0 ) , so we can safely ignore this term and simply maximise the likelihood of the data given z 0 .

Original: Our estimator of z can therefore be deﬁned simply as ˆ z x ( z 0 ) =
Revised: Our estimator of z can therefore be deﬁned simply as ˆ z x ( z 0 ) =

Original: z 0 + ∇ z 0 log p ( z 0 | x ) , which can otherwise be interpreted as a single gradient descent step on the conditional log-likelihood of the data.
Revised: z 0 + ∇ z 0 log p ( z 0 | x ) , which can otherwise be interpreted as a single gradient descent step on the conditional log-likelihood of the data.

Original: From this estimate, the data can be reconstructed by passing ˆ z through the decoder to parameterise p ( x | ˆ z x ) .
Revised: From this estimate, the data can be reconstructed by passing ˆ z through the decoder to parameterise p ( x | ˆ z x ) .

Original: This procedure can be viewed more explicitly when using a neural network
Revised: This procedure can be viewed more explicitly when using a neural network

Original: F : R k
Revised: F : R k

Original: → R m to output the mean of p ( x | ˆ z x ) parameterised by a normal distribution; in this case the loss function is deﬁned in terms of mean squared error loss L MSE :
Revised: → R m to output the mean of p ( x | ˆ z x ) parameterised by a normal distribution; in this case the loss function is deﬁned in terms of mean squared error loss L MSE :

Original: [Equation]
Revised: [Equation]

Original: The gradient computation thereby plays a similar role to an encoder, while F can be viewed as a decoder, with the outer loss term determining the overall reconstruction quality.
Revised: The gradient computation thereby plays a similar role to an encoder, while F can be viewed as a decoder, with the outer loss term determining the overall reconstruction quality.

Original: Using a single network to perform both roles has the advantage of simplifying the overall architecture, avoiding the need to balance networks, and avoiding bottlenecks; this is demonstrated in Figure 1b which provides a visualisation of the GON process.
Revised: Using a single network to perform both roles has the advantage of simplifying the overall architecture, avoiding the need to balance networks, and avoiding bottlenecks; this is demonstrated in Figure 1b which provides a visualisation of the GON process.

Original: 3. V ARIATIONAL GON S
Revised: 3. V ARIATIONAL GON S

Original: The variational approach can be naturally applied to GONs, allowing sampling in a single step while only requiring the generative network, reducing the parameters necessary.
Revised: The variational approach can be naturally applied to GONs, allowing sampling in a single step while only requiring the generative network, reducing the parameters necessary.

Original: Similar to before, a feedforward neural network F parameterises p ( x | z ) , while the expectation of p ( z | x ) is calculated with empirical Bayes.
Revised: Similar to before, a feedforward neural network F parameterises p ( x | z ) , while the expectation of p ( z | x ) is calculated with empirical Bayes.

Original: A normal prior is assumed over p ( z ) thus Equation 7 can be written as:
Revised: A normal prior is assumed over p ( z ) thus Equation 7 can be written as:

Original: [Equation]
Revised: [Equation]

Original: where z 0 = 0 as discussed in Section 3.1.
Revised: where z 0 = 0 as discussed in Section 3.1.

Original: While it would be possible to use this estimate directly within a constant-variance VAE (Ghosh et al., 2019), we opt to incorporate the reparameterisation trick into the generative network as a stochastic layer, to represent the distribution over which x could be encoded to, using empirical Bayes to estimate z .
Revised: While it would be possible to use this estimate directly within a constant-variance VAE (Ghosh et al., 2019), we opt to incorporate the reparameterisation trick into the generative network as a stochastic layer, to represent the distribution over which x could be encoded to, using empirical Bayes to estimate z .

Original: Similar to the autoencoding approach, we could ignore log p ( z 0 ) , however we ﬁnd assuming a normally distributed z 0 implicitly constrains z , aiding the optimisation procedure.
Revised: Similar to the autoencoding approach, we could ignore log p ( z 0 ) , however we ﬁnd assuming a normally distributed z 0 implicitly constrains z , aiding the optimisation procedure.

Original: Speciﬁcally, the forward pass of F is implemented as follows:
Revised: Speciﬁcally, the forward pass of F is implemented as follows:

Original: ˆ z x (or z 0 ) is mapped by linear transformations to µ (ˆ z x ) and σ (ˆ z x ) and the reparameterisation trick is applied, subsequently the further transformations formerly deﬁned as F in the GON formulation are applied, providing parameters for for p ( x | ˆ z x ) .
Revised: ˆ z x (or z 0 ) is mapped by linear transformations to µ (ˆ z x ) and σ (ˆ z x ) and the reparameterisation trick is applied, subsequently the further transformations formerly deﬁned as F in the GON formulation are applied, providing parameters for for p ( x | ˆ z x ) .

Original: Training is performed end-to-end, minimising the ELBO:
Revised: Training is performed end-to-end, minimising the ELBO:

Original: [Equation]
Revised: [Equation]

Original: These steps are shown in Figure 1d, which in practice has a simple implementation.
Revised: These steps are shown in Figure 1d, which in practice has a simple implementation.

Original: 3. I MPLICIT GON S
Revised: 3. I MPLICIT GON S

Original: In the ﬁeld of implicit representation networks, the aim is to learn a neural approximation of a function Φ that satisﬁes an implicit equation of the form:
Revised: In the ﬁeld of implicit representation networks, the aim is to learn a neural approximation of a function Φ that satisﬁes an implicit equation of the form:

Original: [Equation]
Revised: [Equation]

Original: where R ’s deﬁnition is problem dependent but often corresponds to a loss function.
Revised: where R ’s deﬁnition is problem dependent but often corresponds to a loss function.

Original: Equations with this structure arise in a myriad of ﬁelds, namely 3D modelling, image, video, and audio representation (Sitzmann et al., 2020b).
Revised: Equations with this structure arise in a myriad of ﬁelds, namely 3D modelling, image, video, and audio representation (Sitzmann et al., 2020b).

Original: In these cases, data samples x = { ( c , Φ x ( c )) } can be represented in this form in terms of coordinates c ∈ R n and the corresponding data at those points Φ: R n → R m .
Revised: In these cases, data samples x = { ( c , Φ x ( c )) } can be represented in this form in terms of coordinates c ∈ R n and the corresponding data at those points Φ: R n → R m .

Original: Due to the continuous nature of Φ , data with large spatial dimensions can be represented much more efﬁciently than approaches using convolutions, for instance.
Revised: Due to the continuous nature of Φ , data with large spatial dimensions can be represented much more efﬁciently than approaches using convolutions, for instance.

Original: Despite these beneﬁts, representing a distribution of data points using implicit methods is more challenging, leading to the use of hypernetworks which estimate the weights of an implicit network for each data point (Sitzmann et al., 2020a); this increases the number of parameters and adds signiﬁcant complexity.
Revised: Despite these beneﬁts, representing a distribution of data points using implicit methods is more challenging, leading to the use of hypernetworks which estimate the weights of an implicit network for each data point (Sitzmann et al., 2020a); this increases the number of parameters and adds signiﬁcant complexity.

Original: By applying the GON procedure to implicit representation networks, it is possible learn a space of implicit functions without the need for additional networks.
Revised: By applying the GON procedure to implicit representation networks, it is possible learn a space of implicit functions without the need for additional networks.

Original: We assume there exist latent vectors z ∈ R k corresponding to data samples; the concatentation of these latent variables with the data coordinates can therefore geometrically be seen as points on a manifold that describes the full dataset in keeping with the manifold hypothesis (Fefferman et al., 2016).
Revised: We assume there exist latent vectors z ∈ R k corresponding to data samples; the concatentation of these latent variables with the data coordinates can therefore geometrically be seen as points on a manifold that describes the full dataset in keeping with the manifold hypothesis (Fefferman et al., 2016).

Original: An implicit Gradient Origin
Revised: An implicit Gradient Origin

Original: Network can be trained on this space to mimic Φ using a neural network F : R n + k
Revised: Network can be trained on this space to mimic Φ using a neural network F : R n + k

Original: → R m , thereby learning a space of implicit functions by modifying Equation 8:
Revised: → R m , thereby learning a space of implicit functions by modifying Equation 8:

Original: [Equation]
Revised: [Equation]

Original: where both integrations are performed over the space of coordinates and c ⊕ z represents a concatenation (Figure 1c).
Revised: where both integrations are performed over the space of coordinates and c ⊕ z represents a concatenation (Figure 1c).

Original: Similar to the non-implicit approach, the computation of latent vectors can be expressed as z = −∇
Revised: Similar to the non-implicit approach, the computation of latent vectors can be expressed as z = −∇

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: Φ
Revised: Φ

Original: [Equation]
Revised: [Equation]

Original: c ) , F ( c ⊕ z 0 ) (cid:1) d c .
Revised: c ) , F ( c ⊕ z 0 ) (cid:1) d c .

Original: In particular, we parameterise F with a
Revised: In particular, we parameterise F with a

Original: SIREN (Sitzmann et al., 2020b), ﬁnding that it is capable of modelling both high and low frequency components in high dimensional spaces.
Revised: SIREN (Sitzmann et al., 2020b), ﬁnding that it is capable of modelling both high and low frequency components in high dimensional spaces.

Original: 3. GON G ENERALISATIONS
Revised: 3. GON G ENERALISATIONS

Original: There are a number of interesting generalisations that make this approach applicable to other tasks.
Revised: There are a number of interesting generalisations that make this approach applicable to other tasks.

Original: In Equations 8 and 12 we use the same L in both the inner term and outer term, however, as with variational GONs, it is possible to use different functions; through this, training a GON concurrently as a generative model and classiﬁer is possible, or through some minor modiﬁcations to the loss involving the addition of the categorical cross entropy loss function L CCE to maximise the likelihood of classiﬁcation, solely as a classiﬁer:
Revised: In Equations 8 and 12 we use the same L in both the inner term and outer term, however, as with variational GONs, it is possible to use different functions; through this, training a GON concurrently as a generative model and classiﬁer is possible, or through some minor modiﬁcations to the loss involving the addition of the categorical cross entropy loss function L CCE to maximise the likelihood of classiﬁcation, solely as a classiﬁer:

Original: [Equation]
Revised: [Equation]

Original: 13) where y are the target labels and f is a single linear layer followed by softmax.
Revised: (13) where y are the target labels and f is a single linear layer followed by softmax.

Original: Another possibility is modality conversion for translation tasks; in this case the inner reconstruction loss is performed on the source signal and the outer loss on the target signal.
Revised: Another possibility is modality conversion for translation tasks; in this case the inner reconstruction loss is performed on the source signal and the outer loss on the target signal.

Original: 3. J USTIFICATION
Revised: 3. J USTIFICATION

Original: Beyond empirical Bayes, we provide some additional analysis on why a single gradient step is sufﬁcient as an encoder.
Revised: Beyond empirical Bayes, we provide some additional analysis on why a single gradient step is sufﬁcient as an encoder.

Original: Firstly, the network gradient is itself a non-linear function that can approximate complex functions; the derivative of a feedforward neural network with depth L corresponds to a product of L − 1 networks (Saremi, 2019), raising interesting connections with products of experts (Hinton, 1999).
Revised: Firstly, the network gradient is itself a non-linear function that can approximate complex functions; the derivative of a feedforward neural network with depth L corresponds to a product of L − 1 networks (Saremi, 2019), raising interesting connections with products of experts (Hinton, 1999).

Original: There are therefore beneﬁts to using an activation function whose derivative is also an effective non-linearity function.
Revised: There are therefore beneﬁts to using an activation function whose derivative is also an effective non-linear function.

Original: Secondly, the gradient of the loss inherently offers substantial information about data making it a good encoder.
Revised: Secondly, the gradient of the loss inherently offers substantial information about data making it a good encoder.

Original: Finally, a good latent space should satisfy local consistency (Zhou et al., 2003; Kamnitsas et al., 2018).
Revised: Finally, a good latent space should satisfy local consistency (Zhou et al., 2003; Kamnitsas et al., 2018).

Original: GONs satisfy this since similar data points will have similar gradients due to the constant latent initialisation.
Revised: GONs satisfy this since similar data points will have similar gradients due to the constant latent initialisation.

Original: As such, the network needs only to ﬁnd an equilibrium where its prior is the gradient operation, allowing for signiﬁcant freedom.
Revised: As such, the network needs only to ﬁnd an equilibrium where its prior is the gradient operation, allowing for signiﬁcant freedom.

Original:  R ESULTS
Revised:  R ESULTS

Original: We evaluate Gradient Origin Networks on a variety of image datasets: MNIST (LeCun et al., 1998),
Revised: We evaluate Gradient Origin Networks on a variety of image datasets: MNIST (LeCun et al., 1998),

Original: Fashion-MNIST (Xiao et al., 2017), Small NORB (LeCun et al., 2004), COIL-

Revised: Fashion-MNIST (Xiao et al., 2017), Small NORB (LeCun et al., 2004), COIL-

Original: (Nane et al., 1996),
Revised: (Nane et al., 1996),

Original: CIFAR-10 (Krizhevsky et al., 2009), CelebA (Liu et al., 2015), and LSUN Bedroom (Yu et al., 2015).
Revised: CIFAR-10 (Krizhevsky et al., 2009), CelebA (Liu et al., 2015) [Table] and LSUN Bedroom (Yu et al., 2015).

Original: Simple models are used: for small images, implicit GONs consist of approximately 4 hidden layers of 256 units and convolutional GONs consist of 4 convolution layers with Batch Normalization (Ioffe & Szegedy, 2015) and the ELU non-linearity (Clevert et al., 2016), for larger images the same general architecture is used, scaled up with additional layers; all training is performed with the Adam optimiser (Kingma & Ba, 2015).
Revised: Simple models are used: for small images, implicit GONs consist of approximately 4 hidden layers of 256 units and convolutional GONs consist of 4 convolution layers with Batch Normalization (Ioffe & Szegedy, 2015) and the ELU non-linearity (Clevert et al., 2016), for larger images the same general architecture is used, scaled up with additional layers; all training is performed with the Adam optimiser (Kingma & Ba, 2015).

Original: All graphs shown in this paper are the average of 3 runs.
Revised: Error bars in graphs represent standard deviations over three runs.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. Q UANTITATIVE E VALUATION
Revised: 4. Q UANTITATIVE E VALUATION

Original: A quantitative evaluation of the representation ability of GONs is performed in Table 1 against a number of baseline approaches.
Revised: A quantitative evaluation of the representation ability of GONs is performed in Table 1 against a number of baseline approaches.

Original: We compare against the single step methods: standard autoencoder, an autoencoder with tied weights (Gedeon, 1998), and a GON with gradients detached from z , as well as the multi-step methods: 10 gradient descent steps per data point, with and without detaching gradients, and a GLO (Bojanowski et al., 2018) which assigns a persistent latent vector to each data point and optimises them with gradient descent, therefore taking orders of magnitude longer to reconstruct validation data than other approaches.
Revised: We compare against the single step methods: standard autoencoder, an autoencoder with tied weights (Gedeon, 1998), and a GON with gradients detached from z , as well as the multi-step methods: 10 gradient descent steps per data point, with and without detaching gradients, and a GLO (Bojanowski et al., 2018) which assigns a persistent latent vector to each data point and optimises them with gradient descent, therefore taking orders of magnitude longer to reconstruct validation data than other approaches.

Original: For the 10-step methods, a learning rate of 0.1 is applied as used in other literature (Nijkamp et al., 2019); the GLO is trained with MSE for consistency with the other approaches and we do not project latents onto a hypersphere as proposed by the authors since in this experiment sampling is unimportant and this would handicap the approach.
Revised: For the 10-step methods, a learning rate of 0.1 is applied as used in other literature (Nijkamp et al., 2019); the GLO is trained with MSE for consistency with the other approaches and we do not project latents onto a hypersphere as proposed by the authors since in this experiment sampling is unimportant and this would handicap the approach.

Original: GONs achieve much lower validation loss than other single step methods and are competitive with the multi-step approaches; in fact, GONs achieve the lowest loss on three out of the ﬁve datasets.
Revised: GONs achieve much lower validation loss than other single step methods and are competitive with the multi-step approaches; in fact, GONs achieve the lowest loss on four out of the ﬁve datasets.

Original: Our variational GON is compared with a VAE, whose decoder is the same as the GON, quantitatively in terms of ELBO on the test set in Table 2.
Revised: Our variational GON is compared with a VAE, whose decoder is the same as the GON, quantitatively in terms of ELBO on the test set in Table 2.

Original: We ﬁnd that the representation ability of GONs is pertinent here also, allowing the variational GON to achieve lower ELBO on ﬁve out of the six datasets tested.
Revised: We ﬁnd that the representation ability of GONs is pertinent here also, allowing the variational GON to achieve lower ELBO on ﬁve out of the six datasets tested.

Original: Both models were trained with the original VAE formulation for fairness, however, we note that variations aimed to improve VAE sample quality such as β -VAEs (Higgins et al., 2017) are also applicable to variational GONs to the same effect.
Revised: Both models were trained with the original VAE formulation for fairness, however, we note that variations aimed to improve VAE sample quality such as β -VAEs (Higgins et al., 2017) are also applicable to variational GONs to the same effect.

Original: An ablation study is performed, comparing convolutional GONs with autoencoders whose decoders have exactly the same architecture as the GONs (Figure 2a) and where the autoencoder decoders mirror their respective encoders.
Revised: An ablation study is performed, comparing convolutional GONs with autoencoders whose decoders have exactly the same architecture as the GONs (Figure 2a) and where the autoencoder decoders mirror their respective encoders.

Original: The mean reconstruction loss over the test set is measured after each epoch for a variety of latent sizes.
Revised: The mean reconstruction loss over the test set is measured after each epoch for a variety of latent sizes.

Original: Despite having half the number of parameters, GONs
Revised: Despite having half the number of parameters, GONs

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: signiﬁcantly lower reconstruction loss over a wide range of architectures.
Revised: signiﬁcantly lower reconstruction loss over a wide range of architectures.

Original: Further experiments evaluating the impact of latent size and model capacity are performed in Appendix B.
Revised: Further experiments evaluating the impact of latent size and model capacity are performed in Appendix B.

Original: Our hypothesis that a single gradient step is sufﬁcient when jointly optimised with the forwards pass is tested on the CIFAR-10 dataset in Figure 2b.
Revised: Our hypothesis that a single gradient step is sufﬁcient when jointly optimised with the forwards pass is tested on the CIFAR-10 dataset in Figure 2b.

Original: We observe negligible difference between a single step and multiple jointly optimised steps, in support of our hypothesis.
Revised: We observe negligible difference between a single step and multiple jointly optimised steps, in support of our hypothesis.

Original: Performing multiple steps greatly increases run-time so there is seemingly no beneﬁt to this.
Revised: Performing multiple steps greatly increases run-time so there is seemingly no beneﬁt to this.

Original: Additionally, the importance of the joint optimisation procedure is determined by detaching the gradients from z before reconstructing (Figure 2b); this results in markedly worse performance, even when in conjunction with multiple steps.
Revised: Additionally, the importance of the joint optimisation procedure is determined by detaching the gradients from z before reconstructing (Figure 2b); this results in markedly worse performance, even when in conjunction with multiple steps.

Original: In Figure 2c we assess whether the greater performance of GONs relative to autoencoders comes at the expense of generalisation; we ﬁnd that the opposite is true, that the discrepancy between reconstruction loss on the training and test sets is greater with autoencoders.
Revised: In Figure 2c we assess whether the greater performance of GONs relative to autoencoders comes at the expense of generalisation; we ﬁnd that the opposite is true, that the discrepancy between reconstruction loss on the training and test sets is greater with autoencoders.

Original: This result extends to other datasets as can be seen in Appendix E.
Revised: This result extends to other datasets as can be seen in Appendix E.

Original: Figure 3 demonstrates the effect activation functions have on convolutional GON performance for different numbers of latent variables.
Revised: Figure 3 demonstrates the effect activation functions have on convolutional GON performance for different numbers of latent variables.

Original: Since optimising GONs requires computing second order derivatives, the choice of nonlinearity requires different characteristics to standard models.
Revised: Since optimising GONs requires computing second order derivatives, the choice of nonlinearity requires different characteristics to standard models.

Original: In particular, GONs prefer functions that are not only effective activation functions, but also whose ﬁrst and second derivatives are nonlinear functions, unlike ReLUs where ReLU (cid:48)(cid:48)
Revised: In particular, GONs prefer functions that are not only effective activation functions, but also whose ﬁrst and second derivatives are nonlinear functions, unlike ReLUs where ReLU (cid:48)(cid:48)

Original: ( x ) = 0 .
Revised: ( x ) = 0 .

Original: The ELU non-linearity is effective with all tested architectures.
Revised: The ELU non-linearity is effective with all tested architectures.

Original: 4. Q UALITATIVE E VALUATION
Revised: 4. Q UALITATIVE E VALUATION

Original: The representation ability of implicit GONs is shown in Figure 4 where we train on large image datasets using a relatively small number of parameters.
Revised: The representation ability of implicit GONs is shown in Figure 4 where we train on large image datasets using a relatively small number of parameters.

Original: In particular, Figure 4a shows MNIST can be well-represented with just 4,385 parameters (a SIREN with 3 hidden layers each withhidden units, and 32 latent dimensions).
Revised: In particular, Figure 4a shows MNIST can be well-represented with just 4,385 parameters (a SIREN with 3 hidden layers each withhidden units, and 32 latent dimensions).

Original: An advantage of modelling data with implicit networks is that coordinates can be at arbitrarily high resolutions.
Revised: An advantage of modelling data with implicit networks is that coordinates can be at arbitrarily high resolutions.

Original: In Figure 5 we train on 32x32 images then reconstruct at 256x256.
Revised: In Figure 5 we train on 32x32 images then reconstruct at 256x256.

Original: A signiﬁcant amount of high frequency detail is modelled despite only seeing low resolution images.
Revised: A signiﬁcant amount of high frequency detail is modelled despite only seeing low resolution images.

Original: The structure of the implicit GON latent space is shown by sampling latent codes from pairs of images, and then spherically interpolating between them to synthesise new samples (Figure 6).
Revised: The structure of the implicit GON latent space is shown by sampling latent codes from pairs of images, and then spherically interpolating between them to synthesise new samples (Figure 6).

Original: These samples are shown to capture variations in shape (the shoes in Figure 6b), size, and rotation (Figure 6d).
Revised: These samples are shown to capture variations in shape (the shoes in Figure 6b), size, and rotation (Figure 6d).

Original: We assess the quality of samples from variational GONs using convolutional models trained to convergence in Figure 7.
Revised: We assess the quality of samples from variational GONs using convolutional models trained to convergence in Figure 7.

Original: These are diverse and often contain ﬁne details.
Revised: These are diverse and often contain ﬁne details.

Original: Samples from an implicit
Revised: Samples from an implicit

Original: GON trained with early stopping, as a simple alternative, can be found in Appendix D however this approach results in ﬁne details being lost.
Revised: GON trained with early stopping, as a simple alternative, can be found in Appendix D however this approach results in ﬁne details being lost.

Original: GONs are also found to converge quickly; we plot
Revised: GONs are also found to converge quickly; we plot

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: reconstructions at multiple time points during the ﬁrst minute of training (Figure 8).
Revised: reconstructions at multiple time points during the ﬁrst minute of training (Figure 8).

Original: After onlyseconds of training on a single GPU, a large amount of signal information from MNIST is modelled.
Revised: After onlyseconds of training on a single GPU, a large amount of signal information from MNIST is modelled.

Original: In order to evaluate how well GONs can represent high resolution complex data, we train a convolutional GON on the LSUN Bedroom dataset scaled to 128x128 (Figure 9a).
Revised: In order to evaluate how well GONs can represent high resolution complex data, we train a convolutional GON on the LSUN Bedroom dataset scaled to 128x128 (Figure 9a).

Original: As with smaller, more simple data, we ﬁnd training to always be extremely stable and consistent over a wide range of hyperparameter settings.
Revised: As with smaller, more simple data, we ﬁnd training to always be extremely stable and consistent over a wide range of hyperparameter settings.

Original: Reconstructions are of excellent quality given the simple network architecture.
Revised: Reconstructions are of excellent quality given the simple network architecture.

Original: A convolutional variational GON is also trained on the CelebA dataset scaled to 64x64 (Figure 9b).
Revised: A convolutional variational GON is also trained on the CelebA dataset scaled to 64x64 (Figure 9b).

Original: Unconditional samples are somewhat blurry as commonly associated with traditional VAE models on complex natural images (Zhao et al., 2017) but otherwise show wide variety.
Revised: Unconditional samples are somewhat blurry as commonly associated with traditional VAE models on complex natural images (Zhao et al., 2017) but otherwise show wide variety.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Despite similarities with autoencoder approaches, the absence of an encoding network offers several advantages.
Revised: Despite similarities with autoencoder approaches, the absence of an encoding network offers several advantages.

Original: VAEs with overpowered decoders are known to ignore the latent variables (Chen et al., 2017) whereas GONs only have one network that equally serves both encoding and decoding functionality.
Revised: VAEs with overpowered decoders are known to ignore the latent variables (Chen et al., 2017) whereas GONs only have one network that equally serves both encoding and decoding functionality.

Original: Designing inference networks for complex decoders is not a trivial task (Vahdat &
Revised: Designing inference networks for complex decoders is not a trivial task (Vahdat &

Original: Kautz, 2020), however, inferring latent variables using a GON simpliﬁes this procedure.
Revised: Kautz, 2020), however, inferring latent variables using a GON simpliﬁes this procedure.

Original: Similar to GONs, Normalizing Flow methods are also capable of encoding and decoding with a single
Revised: Similar to GONs, Normalizing Flow methods are also capable of encoding and decoding with a single

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: set of weights, however, they achieve this by restricting the network to be invertible.
Revised: set of weights, however, they achieve this by restricting the network to be invertible.

Original: This requires considerable architectural restrictions that affect performance, make them less parameter efﬁcient, and are unable to reduce the dimension of the data (Kingma & Dhariwal, 2018).
Revised: This requires considerable architectural restrictions that affect performance, make them less parameter efﬁcient, and are unable to reduce the dimension of the data (Kingma & Dhariwal, 2018).

Original: Similarly, autoencoders with tied weights also encode and decode with a single set of weights by using the transpose of the encoder’s weight matrices in the decoder; this, however, is only applicable to simple architectures.
Revised: Similarly, autoencoders with tied weights also encode and decode with a single set of weights by using the transpose of the encoder’s weight matrices in the decoder; this, however, is only applicable to simple architectures.

Original: GONs on the other hand use gradients as encodings which allow arbitrary functions to be used.
Revised: GONs on the other hand use gradients as encodings which allow arbitrary functions to be used.

Original: A number of previous works have used gradient-based computations to learn latent vectors however as far as we are aware, we are the ﬁrst to use a single gradient step jointly optimised with the feedforward pass, making it fundamentally different to these approaches.
Revised: A number of previous works have used gradient-based computations to learn latent vectors however as far as we are aware, we are the ﬁrst to use a single gradient step jointly optimised with the feedforward pass, making it fundamentally different to these approaches.

Original: Latent encodings have been estimated for pre-trained generative models without encoders, namely Generative Adversarial Networks, using approaches such as standard gradient descent (Lipton & Tripathi, 2017; Zhu et al., 2016).
Revised: Latent encodings have been estimated for pre-trained generative models without encoders, namely Generative Adversarial Networks, using approaches such as standard gradient descent (Lipton & Tripathi, 2017; Zhu et al., 2016).

Original: A number of approaches have trained generative models directly with gradient-based inference (Han et al., 2017;
Revised: A number of approaches have trained generative models directly with gradient-based inference (Han et al., 2017;

Original: Bojanowski et al., 2018; Zadeh et al., 2019); these assign latent vectors to data points and jointly learn them with the network parameters through standard gradient descent or Langevin dynamics.
Revised: Bojanowski et al., 2018; Zadeh et al., 2019); these assign latent vectors to data points and jointly learn them with the network parameters through standard gradient descent or Langevin dynamics.

Original: This is very slow, however, and convergence for unseen data samples is not guaranteed.
Revised: This is very slow, however, and convergence for unseen data samples is not guaranteed.

Original: Short run MCMC has also been applied (Nijkamp et al., 2020) however this still requires approximately 25 update steps.
Revised: Short run MCMC has also been applied (Nijkamp et al., 2020) however this still requires approximately 25 update steps.

Original: Since GONs train end-to-end, the optimiser can make use of the second order derivatives to allow for inference in a single step.
Revised: Since GONs train end-to-end, the optimiser can make use of the second order derivatives to allow for inference in a single step.

Original: Also of relevance is model-agnostic meta-learning (Finn et al., 2017), which trains an architecture so that a few gradient descent steps are all that are necessary to new tasks.
Revised: Also of relevance is model-agnostic meta-learning (Finn et al., 2017), which trains an architecture so that a few gradient descent steps are all that are necessary to new tasks.

Original: This is achieved by backpropagating through these gradients, similar to GONs.
Revised: This is achieved by backpropagating through these gradients, similar to GONs.

Original: In the case of implicit GONs, the integration terms in Equation 12 result in computation time that scales in proportion with the data dimension.
Revised: In the case of implicit GONs, the integration terms in Equation 12 result in computation time that scales in proportion with the data dimension.

Original: This makes training slower for very high dimensional data, although we have not yet investigated Monte Carlo integration.
Revised: This makes training slower for very high dimensional data, although we have not yet investigated Monte Carlo integration.

Original: In general GONs are stable and consistent, capable of generating quality samples with an exceptionally small number of parameters, and converge to diverse results with few iterations.
Revised: In general GONs are stable and consistent, capable of generating quality samples with an exceptionally small number of parameters, and converge to diverse results with few iterations.

Original: Nevertheless, there are avenues to explore so as to improve the quality of samples and scale to larger datasets.
Revised: Nevertheless, there are avenues to explore so as to improve the quality of samples and scale to larger datasets.

Original: In particular, it would be beneﬁcial to focus on how to better sample these models, perform formal analysis on the gradients, and investigate whether the distance function could be improved to better capture ﬁne details.
Revised: In particular, it would be beneﬁcial to focus on how to better sample these models, perform formal analysis on the gradients, and investigate whether the distance function could be improved to better capture ﬁne details.

Original: C ONCLUSION
Revised: C ONCLUSION

Original: In conclusion, we have introduced a method based on empirical Bayes which computes the gradient of the data ﬁtting loss with respect to the origin, and then jointly ﬁts the data while learning this new point of reference in the latent space.
Revised: In conclusion, we have introduced a method based on empirical Bayes which computes the gradient of the data ﬁtting loss with respect to the origin, and then jointly ﬁts the data while learning this new point of reference in the latent space.

Original: The results show that this approach is able to represent datasets using a small number of parameters with a simple overall architecture, which has advantages in applications such as implicit representation networks.
Revised: The results show that this approach is able to represent datasets using a small number of parameters with a simple overall architecture, which has advantages in applications such as implicit representation networks.

Original: GONs are shown to converge faster with lower overall reconstruction loss than autoencoders, using the exact same architecture but without the encoder.
Revised: GONs are shown to converge faster with lower overall reconstruction loss than autoencoders, using the exact same architecture but without the encoder.

Original: Experiments show that the choice of non-linearity is important, as the network derivative jointly acts as the encoding function.
Revised: Experiments show that the choice of non-linearity is important, as the network derivative jointly acts as the encoding function.

Original: A P ROOF OF C ONDITIONAL E MPIRICAL B AYES
Revised: A P ROOF OF C ONDITIONAL E MPIRICAL B AYES

Original: For latent variables z ∼ p
Revised: For latent variables z ∼ p

Original: z , a noisy observation of z , z 

Revised: z , a noisy observation of z , z

Original: =
Revised: =

Original: z
Revised: z

Original: + N ( 0 , I ) and data point x ∼ p d , we wish to ﬁnd an estimator of p ( z | x ) .
Revised: + N ( 0 , I ) and data point x ∼ p d , we wish to ﬁnd an estimator of p ( z | x ) .

Original: To achieve this we condition the Bayes least squares estimator on x :
Revised: To achieve this we condition the Bayes least squares estimator on x :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Through the deﬁnition of the probabilistic chain rule and by marginalising out z we can deﬁne p ( z 0 | x ) = (cid:82) p ( z 0 | z , x ) p ( z | x ) d z which can be simpliﬁed to (cid:82) p ( z 0 | z ) p ( z | x ) d z since z 0 is dependent only on z .
Revised: Through the deﬁnition of the probabilistic chain rule and by marginalising out z we can deﬁne p ( z 0 | x ) = (cid:82) p ( z 0 | z , x ) p ( z | x ) d z which can be simpliﬁed to (cid:82) p ( z 0 | z ) p ( z | x ) d z since z 0 is dependent only on z .

Original: Writing this out fully we obtain:
Revised: Writing this out fully we obtain:

Original: [Equation]
Revised: [Equation]

Original: Differentiating with respect to z 0 and multiplying both sides by Σ gives:
Revised: Differentiating with respect to z 0 and multiplying both sides by Σ gives:

Original: [Equation]
Revised: [Equation]

Original: After dividing both sides by p ( z 0 | x ) and combining with Equation 14 we get:
Revised: After dividing both sides by p ( z 0 | x ) and combining with Equation 14 we get:

Original: [Equation]
Revised: [Equation]

Original: Finally, this can be rearranged to give the single step estimator of z :
Revised: Finally, this can be rearranged to give the single step estimator of z :

Original: [Equation]
Revised: [Equation]

Original: B C OMPARISON WITH A UTOENCODERS
Revised: B C OMPARISON WITH A UTOENCODERS

Original: This section compares Gradient Origin Networks with autoencoders that use exactly the same architecture, as described in the paper.
Revised: This section compares Gradient Origin Networks with autoencoders that use exactly the same architecture, as described in the paper.

Original: All experiments are performed with the CIFAR-10 dataset.
Revised: All experiments are performed with the CIFAR-10 dataset.

Original: After each epoch the model is ﬁxed and the mean reconstruction loss is measured over the training set.
Revised: After each epoch the model is ﬁxed and the mean reconstruction loss is measured over the training set.

Original: In Figure 10a the depth of networks is altered by removing blocks thereby downscaling to various resolutions.
Revised: In Figure 10a the depth of networks is altered by removing blocks thereby downscaling to various resolutions.

Original: We ﬁnd that GONs outperform autoencoders until latents have a spatial size of 8x (where the equivalent GON now only has only 2 convolution layers).
Revised: We ﬁnd that GONs outperform autoencoders until latents have a spatial size of 8x (where the equivalent GON now only has only 2 convolution layers).

Original: Considering the limit where neither model has any parameters, the latent space is the input data i.e. z = x .
Revised: Considering the limit where neither model has any parameters, the latent space is the input data i.e. z = x .

Original: Substituting the deﬁnition of a GON (Equation 8) this becomes −∇ z 0 L = x which simpliﬁes to 0 = x which is a contradiction.
Revised: Substituting the deﬁnition of a GON (Equation 8) this becomes −∇ z 0 L = x which simpliﬁes to 0 = x which is a contradiction.

Original: This is not a concern in normal practice, as evidenced by the results presented here.
Revised: This is not a concern in normal practice, as evidenced by the results presented here.

Original: Figure 10b explores the relationship between autoencoders and GONs when changing the number of convolution ﬁlters; GONs are found to outperform autoencoders in all cases.
Revised: Figure 10b explores the relationship between autoencoders and GONs when changing the number of convolution ﬁlters; GONs are found to outperform autoencoders in all cases.

Original: The discrepancy between loss curves decreases as the number of ﬁlters increases likely due to the diminishing returns of providing more capacity to the GON when its loss is signiﬁcantly closer to 0.
Revised: The discrepancy between loss curves decreases as the number of ﬁlters increases likely due to the diminishing returns of providing more capacity to the GON when its loss is signiﬁcantly closer to 0.

Original: A similar pattern is found when decreasing the latent space (Figure 10c); in this case the latent space likely becomes the limiting factor.
Revised: A similar pattern is found when decreasing the latent space (Figure 10c); in this case the latent space likely becomes the limiting factor.

Original: With larger latent spaces GONs signiﬁcantly outperform autoencoders, however, when the latent bottleneck becomes more pronounced this lead lessens.
Revised: With larger latent spaces GONs signiﬁcantly outperform autoencoders, however, when the latent bottleneck becomes more pronounced this lead lessens.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: the number of network parameters increases (b) and the latent size decreases (c), the performance lead of
Revised: the number of network parameters increases (b) and the latent size decreases (c), the performance lead of

Original: GONs over AEs decreases due to diminishing returns/bottlenecking.
Revised: GONs over AEs decreases due to diminishing returns/bottlenecking.

Original: C I NITIALISING z 0 AT THE O RIGIN
Revised: C I NITIALISING z 0 AT THE O RIGIN

Original: We evaluate different initialisations of z 0 in Figure 11 by sampling z 0 ∼ N ( 0 , σ 2 I ) for a variety of standard deviations σ .
Revised: We evaluate different initialisations of z 0 in Figure 11 by sampling z 0 ∼ N ( 0 , σ 2 I ) for a variety of standard deviations σ .

Original: The proposed approach ( σ = 0 ) achieves the lowest reconstruction loss (Figure 11a); results for σ > 0 are similar, suggesting that the latent space is adjusted so z 0 simulates the origin.
Revised: The proposed approach ( σ = 0 ) achieves the lowest reconstruction loss (Figure 11a); results for σ > 0 are similar, suggesting that the latent space is adjusted so z 0 simulates the origin.

Original: An alternative parameterisation of z is to use a use a single gradient descent style step z
Revised: An alternative parameterisation of z is to use a use a single gradient descent style step z

Original: = z 0 − ∇ z 0 L (Figure 11b), however, losses are higher than the proposed GON initialisation.
Revised: = z 0 − ∇ z 0 L (Figure 11b), however, losses are higher than the proposed GON initialisation.

Original: [Figure]
Revised: [Figure]

Original: D L ATENT S PACE AND E ARLY S TOPPING
Revised: D L ATENT S PACE AND E ARLY S TOPPING

Original: Figure 12 shows a GON trained as a classiﬁer where the latent space is squeezed into 2D for visualisation (Equation 13) and Figure 13 shows that the negative gradients of a GON aftersteps of training are approximately normally distributed; Figure 14 shows that the latents of a typical convolutional VAE after 800 steps are signiﬁcantly less normally distributed than GONs.
Revised: Figure 12 shows a GON trained as a classiﬁer where the latent space is squeezed into 2D for visualisation (Equation 13) and Figure 13 shows that the negative gradients of a GON aftersteps of training are approximately normally distributed; Figure 14 shows that the latents of a typical convolutional VAE after 800 steps are signiﬁcantly less normally distributed than GONs.

Original: To obtain new samples with implicit GONs, we can use early stopping as a simple alternative to variational approaches.
Revised: To obtain new samples with implicit GONs, we can use early stopping as a simple alternative to variational approaches.

Original: These samples (Figure 15) are diverse and capture the object shapes.
Revised: These samples (Figure 15) are diverse and capture the object shapes.

Original: ples of an implicit GON classiﬁer trained on MNIST (class colours).
Revised: ples of an implicit GON classiﬁer trained on MNIST (class colours).

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E GON G ENERALISATION
Revised: E GON G ENERALISATION

Original: In Figure 16 the training and test losses for GONs and autoencoders are plotted for a variety of datasets.
Revised: In Figure 16 the training and test losses for GONs and autoencoders are plotted for a variety of datasets.

Original: In all cases, GONs generalise better than their equivalent autoencoders while achieving lower losses with fewer parameters.
Revised: In all cases, GONs generalise better than their equivalent autoencoders while achieving lower losses with fewer parameters.

Original: [Figure]
Revised: [Figure]

Original: F S UPER - RESOLUTION
Revised: F S UPER - RESOLUTION

Original: Additional super-sampled images obtained by training an implicit GON on 28x28 MNIST data then reconstructing test data at a resolution of 256x256 are shown in Figure 17.
Revised: Additional super-sampled images obtained by training an implicit GON on 28x28 MNIST data then reconstructing test data at a resolution of 256x256 are shown in Figure 17.

Original: [Figure]
Revised: [Figure]