Original: Natural Reweighted Wake-Sleep Algorithm
Revised: Natural Reweighted Wake-Sleep Algorithm

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Natural gradient has been successfully employed in a wide range of optimizationproblems.
Revised: Natural gradient has been successfully employed in a wide range of optimization problems.

Original: However, for the training of neural networks the resulting increase incomputational complexity sets a limitation to its practical application.
Revised: However, for the training of neural networks the resulting increase in computational complexity sets a limitation to its practical application.

Original: Helmholtz
Revised: Helmholtz

Original: Machines are a particular type of generative models, composed of two Sigmoid
Revised: Machines are a particular type of generative models, composed of two Sigmoid

Original: Belief Networks, commonly trained using the Wake-Sleep algorithm.
Revised: Belief Networks, commonly trained using the Wake-Sleep algorithm.

Original: The locality of the connections in this type of networks induces sparsity and a particularstructure for the Fisher information matrix that can be exploited for the evaluationof its inverse, allowing the efﬁcient computation of the natural gradient also forlarge networks.
Revised: The locality of the connections in this type of networks induces sparsity and a particular structure for the Fisher information matrix that can be exploited for the evaluation of its inverse, allowing the efﬁcient computation of the natural gradient also for large networks.

Original: We introduce a novel algorithm called Natural Reweighted WakeSleep, a geometric adaptation of Reweighted Wake-Sleep, based on the computation of the natural gradient.
Revised: We introduce a novel algorithm called Natural Reweighted Wake-Sleep, a geometric adaptation of Reweighted Wake-Sleep, based on the computation of the natural gradient.

Original: We present an experimental analysis of the algorithmin terms of speed of convergence and the value of the log-likelihood, both with respect to number of iterations and training time, demonstrating improvements overnon-geometric baselines.
Revised: We present an experimental analysis of the algorithm in terms of speed of convergence and the value of the log-likelihood, both with respect to number of iterations and training time, demonstrating improvements over non-geometric baselines.

Original:  Introduction
Revised:  Introduction

Original: Deep generative models have been successfully employed in unsupervised learning to model complex and high dimensional distributions thanks to their ability to extract higher-order representationsof the data and thus generalize better [16, 8].
Revised: Deep generative models have been successfully employed in unsupervised learning to model complex and high dimensional distributions thanks to their ability to extract higher-order representations of the data and thus generalize better [16, 8].

Original: An approach which proved to be successful and thuscommon to several models is based on the use of two separate networks: the recognition network,i.e., the encoder, which provides a compressed latent representation for the input data, and the generative network, i.e., the decoder, able to reconstruct the observation in output.
Revised: An approach which proved to be successful and thus common to several models is based on the use of two separate networks: the recognition network, i e., the encoder which provides a compressed latent representation for the input data, and the generative network, i e., the decoder, able to reconstruct the observation in output.

Original: Helmholtz Machines(HM)
Revised: Helmholtz Machines (HM)

Original: [11] consist of a recognition and a generative network both modelled as a Sigmoid Belief Network (SBN) [22], and differently from standard VAEs [19, 23], are characterized by discrete hiddenvariables.
Revised: [11] consist of a recognition and a generative network both modelled as a Sigmoid Belief Network (SBN) [22] and differently from standard VAEs [19, 23], are characterized by discrete hidden variables.

Original: The training of stochastic networks is a challenging task in deep learning [12], in particular forgenerative models based on recognition and generative networks.
Revised: The training of stochastic networks is a challenging task in deep learning [12], in particular for generative models based on recognition and generative networks.

Original: A solution to this problem consists in the introduction of a family of tractable approximate posterior distributions parameterizedby the encoder network.
Revised: A solution to this problem consists in the introduction of a family of tractable approximate posterior distributions parameterized by the encoder network.

Original: However, in the presence of discrete hidden variables, as for HMs, thisapproach cannot be directly employed, and thus standard training procedures relies on the wellknown Wake-Sleep [15] algorithm, in which two optimization steps for the parameters of the recognition and generative networks are alternated.
Revised: However, in the presence of discrete hidden variables, as for HMs, this approach cannot be directly employed, and thus standard training procedures relies on the wellknown Wake-Sleep [15] algorithm, in which two optimization steps for the parameters of the recognition and generative networks are alternated.

Original: The Wake-Sleep algorithm, as well as more recentadvances [10, 9, 24, 14], relies on the conditional independence assumption between the hiddenvariables of each layer, as a consequence of the directed graphical structure of a SBN.
Revised: The Wake-Sleep algorithm, as well as more recent advances [10, 9, 24, 14], relies on the conditional independence assumption between the hidden variables of each layer, as a consequence of the directed graphical structure of a SBN.

Original: This leadsto a computationally efﬁcient formula for the weights update which does not require the gradientsto be back-propagated through the full network.
Revised: This leads to a computationally efﬁcient formula for the weights update which does not require the gradients to be back-propagated through the full network.

Original: Besides the choice of the speciﬁc loss function tobe optimized, depending on the nature of the generative model, in the literature several approachesto speed up the convergence during training have been proposed, through the deﬁnition of differenttraining algorithms.
Revised: Besides the choice of the speciﬁc loss function to be optimized, depending on the nature of the generative model, in the literature several approaches to speed up the convergence during training have been proposed, through the deﬁnition of different training algorithms.

Original: One line of research, initiated by Amari and co-workers [3, 2], takes advantage of a geometric framework based on notions of Information Geometry [5], which allows the deﬁnition of the natural gradient, i.e., the Riemannian gradient of a function computed with respect to the
Revised: One line of research, initiated by Amari and co-workers [3, 2], takes advantage of a geometric framework based on notions of Information Geometry [5] which allows the deﬁnition of the natural gradient, i.e., the Riemannian gradient of a function computed with respect to the

Original: Fisher-Rao metric.
Revised: Fisher-Rao metric.

Original: In general the computation of the natural gradient requires the inversion of anestimation of the Fisher information matrix, whose dimension depends on the number of weights,and for this reason it cannot be directly applied for the training of large neural network due to itscomputation costs.
Revised: In general the computation of the natural gradient requires the inversion of an estimation of the Fisher information matrix, whose dimension depends on the number of weights, and for this reason it cannot be directly applied for the training of large neural network due to its computation costs.

Original: Motivated by preliminary results from [6], we observe that the Fisher information matrix associatedto a HM takes a block diagonal structure, where the block sizes depend linearly on the size ofindividual hidden layers.
Revised: Motivated by preliminary results from [6], we observe that the Fisher information matrix associated to a HM takes a block diagonal structure, where the block sizes depend linearly on the size of individual hidden layers.

Original: Starting from this observation, we propose a geometric version of the 
Revised: Starting from this observation, we propose a geometric version of the Reweighted

Original: Reweighted Wake-Sleep algorithm for the training of HMs, where the gradient is replaced by thecorresponding natural gradient.
Revised: Wake-Sleep algorithm for the training of HMs, where the gradient is replaced by the corresponding natural gradient.

Original:  The Natural Reweighted Wake-Sleep Algorithm
Revised:  The Natural Reweighted Wake-Sleep Algorithm

Original: A Helmholtz Machine with L layers is composed by two Sigmoid Belief Networks, a generativenetwork p , parameterized by θ (a set of weights and biases W i for each layer i , with i = 0 , .., L ),and a recognition (or inference) network q , parameterized by φ (a set of weights and biases V i foreach layer i , with i = 0 , .., L − 1 ).
Revised: A Helmholtz Machine with L layers is composed by two Sigmoid Belief Networks, a generative network p , parameterized by θ (a set of weights and biases W i for each layer i , with i = 0 , .., L ), and a recognition (or inference) network q , parameterized by φ (a set of weights and biases V i for each layer i , with i = 0 , .., L − 1 ).

Original: Following [10], the training of Helmholtz Machines can be recast in terms of a variational objective [13, 19, 23].
Revised: Following [10], the training of Helmholtz Machines can be recast in terms of a variational objective [13, 19, 23].

Original: This is analogous to the learning in a Variational Autoencoder which requires themaximization of a lower bound for the likelihood.
Revised: This is analogous to the learning in a Variational Autoencoder which requires the maximization of a lower bound for the likelihood.

Original: In the Reweighthed Wake-Sleep (RWS) [10, 20]algorithm, the derivative of the log-likelihood of x are approximated with a Monte Carlo samplingin which the samples are reweighted [10, 20] with the importance weigths˜ ω k , i.e., [Equation] h ) ∂θ (cid:21) (cid:39)
Revised: In the Reweighthed Wake-Sleep (RWS) [10, 20] algorithm, the derivative of the log-likelihood of x are approximated with a Monte Carlo sampling in which the samples are reweighted [10, 20] with the importance weigths

Original: K (cid:88) k

Original: = ˜ ω k  ∂ ln p ( x, h ( k ) )
Revised: ˜ ω k , i.e., ∂ ln p ( x )

Original: ∂θ with h ( k ) ∼ q ( h | x ) .

Original: [Equation]

Original: =

Original: [Equation]

Original: k (cid:48)

Original: [Equation]
Revised: [Equation]

Original: k (cid:48) , with ω k = p ( x, h ( k ) )

Original: q ( h ( k ) | x ) . The RWS algorithm also introduces another training step called the q-wake update, additionally tothe wake and sleep phases to outperform the regular WS.
Revised: The RWS algorithm also introduces another training step called the q-wake update, additionally to the wake and sleep phases to outperform the regular WS.

Original: Our method builds on the RWS to calculatethe gradients for each step.
Revised: Our method builds on the RWS to calculate the gradients for each step.

Original: Information Geometry [1, 5, 4, 7] studies the geometry of statistical models using the language of
Revised: Information Geometry [1, 5, 4, 7] studies the geometry of statistical models using the language of

Original: Riemannian and afﬁne geometry.
Revised: Riemannian and afﬁne geometry.

Original: In this framework, the steepest direction of a function is given bythe natural gradient update step, computed by
Revised: In this framework, the steepest direction of a function is given by the natural gradient update step, computed by

Original: [Equation]
Revised: [Equation]

Original: where L is the loss function and ∇L its vector of partial derivatives.
Revised: where L is the loss function and ∇L its vector of partial derivatives.

Original: F ( θ ) is the Fisher informationmatrix associated to the Riemannian Fisher-Rao metric on the manifold of probability distributionsparameterized by θ , i.e., the weights of the network.
Revised: F ( θ ) is the Fisher information matrix associated to the Riemannian Fisher-Rao metric on the manifold of probability distributions parameterized by θ , i.e., the weights of the network.

Original: Finaly, η > 0 is the learning rate.
Revised: Finally, η > 0 is the learning rate.

Original: The Fisher information matrix F , needed for the evaluation of the natural gradient of a given function, has a structure which strongly depends on the nature of the statistical model.
Revised: The Fisher information matrix F , needed for the evaluation of the natural gradient of a given function, has a structure which strongly depends on the nature of the statistical model.

Original: For both Sigmoid
Revised: For both Sigmoid

Original: Belief Networks (SBNs) constituting a Helmholtz Machine, the Fisher information matrix can becomputed as follows.
Revised: Belief Networks (SBNs) constituting a Helmholtz Machine, the Fisher information matrix can be computed as follows.

Original: Let i denote a layer, and j one of its hidden nodes, we denote with W ij the j -thcolumn of the matrix W i .
Revised: Let i denote a layer, and j one of its hidden nodes, we denote with W ij the j -th column of the matrix W i .

Original: The blocks associated to the i -th layer and its j -th hidden unit for boththe distributions p and q associated to the generative and the recognition Sigmoid Belief Networks,are denoted with F ip,j and F iq,j , respectively.
Revised: The blocks associated to the i -th layer and its j -th hidden unit for both the distributions p and q associated to the generative and the recognition networks, are denoted with F ip,j and F iq,j , respectively.

Original: These matrices can be estimated using Monte Carlomethods, based on 
Revised: These matrices can be estimated using Monte Carlo methods, based on the samples in the batch.

Original: [Equation]
Revised: [Equation]

Original: The Fisher matrix is block-diagonal, as it can be seen in Fig.
Revised: The Fisher matrix is block-diagonal, as it can be seen in Fig.

Original: 1 and all other blocks besides F ip,jand F iq,j are zero by deﬁnition.
Revised: 1 and all other blocks besides F ip,j and F iq,j are zero by deﬁnition.

Original: Such block structure is very convenient and represents the mainargument for the efﬁciency of the algorithm.
Revised: Such block structure is very convenient and represents the main argument for the efﬁciency of the algorithm.

Original: Notice that the estimations in Eqs.
Revised: Notice that the estimations in Eqs.

Original: (4) and (5) arenot to be confused with the approximations typically introduced for the simpliﬁcation of the Fishermatrix, needed to make it computationally tractable in feed-forward neural networks.
Revised: (4) and (5) are not to be confused with the approximations typically introduced for the simpliﬁcation of the Fisher matrix needed to make it computationally tractable in feed-forward neural networks.

Original: Each blockof the Fisher matrix is estimated by Monte Carlo sampling based on n samples.
Revised: Each block of the Fisher matrix is estimated by Monte Carlo sampling based on n samples.

Original: To obtain a lowervariance estimator for the expected value in Eq.
Revised: To obtain a lower variance estimator for the expected value in Eq.

Original: (4) we use samples h ( k ) ∼ q ( h | x ) p D ( x ) from theapproximate distribution, and reweight them by the importance sampling weights ˜ ω
Revised: (4) we use samples h ( k ) ∼ q ( h | x ) p D ( x ) from the approximate distribution, and reweight them by the importance sampling weights ˜ ω

Original: k as in Eq.
Revised: k as in Eq.

Original: The Fisher matrices in Eqs.
Revised: Since the Fisher matrices in Eqs.

Original: (4) and (5) only depend on the statistical models associated to thejoint distributions p ( x, h ) and q ( x, h ) , thussince the model of the Helmholtz Machine remainsunchanged, the same Fisher matrix can be used for different training algorithms (WS, RWS, etc.).
Revised: (4) and (5) only depend on the statistical models associated to the joint distributions p ( x, h ) and q ( x, h ) , thus the model of the Helmholtz Machine remains unchanged, the same Fisher matrix can be used for different training algorithms (WS, RWS, etc.).

Original: We introduced a matrix representation, where the H i matrices are obtained by concatenating foreach sample the h i vector as a row vector, while the diagonal matrices Q ip and Q iq depends onthe evaluation of the activation function.
Revised: We introduced a matrix representation, where the H i matrices are obtained by concatenating for each sample the h i vector as a row vector, while the diagonal matrices Q ip and Q iq depends on the evaluation of the activation function.

Original: The matrices H T QH associated with the estimation ofthe Fisher blocks may be singular depending on the number of samples in the batch used in theestimation compared its size, which depends on the size l i of each layer i .
Revised: The matrices H T QH associated with the estimation of the Fisher blocks may be singular depending on the number of samples in the batch used in the estimation and the size l i of each layer i .

Original: During training n is thesize of the mini-batch b multiplied by the number of samples s from the network (respectively p orq , depending on the Fisher matrix under consideration).
Revised: During training n is the size of the mini-batch b multiplied by the number of samples s from the network (respectively p or q , depending on the Fisher matrix under consideration).

Original: Notice that during training typically n < l i ,thus to guarantee the invertibility we add a damping factor α > 0 multiplying the identity matrix inaddition to H T QH .
Revised: Notice that during training typically n < l i , thus to guarantee the invertibility we add a damping factor α > 0 multiplying the identity matrix in addition to H T QH .

Original: Now we can use the Shermann-Morisson formula to calculate the inverse of arank- k update matrix by
Revised: Now we can use the Shermann-Morisson formula to calculate the inverse of a rank- k update matrix by

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: so that ˜ F − 1 −→ 1 l for α −→ ∞ and ˜ F − 1 −→ F − 1 for α −→ 0 , if F is full rank.
Revised: so that ˜ F − 1 −→ 1 l for α −→ ∞ and ˜ F − 1 −→ F − 1 for α −→ 0 , if F is full rank.

Original: By keeping in memorythe rank- k representation of the matrices, the overall complexity is O ( l 0 l 1 n 2 ) where l 0 and l 1 arethe bottom two layers of the Helmholtz Machine, which are usually the largest.
Revised: By keeping in memory the rank- k representation of the matrices, the overall complexity is O ( l 0 l 1 n 2 ) where l 0 and l 1 are the bottom two layers of the Helmholtz Machine, which are usually the largest.

Original: Assuming the locality of the gradient descent learning step, we make the approximation of slowlychanging metric during few training steps.
Revised: Assuming the locality of the gradient descent learning step, we make the approximation of slowly changing metric during few training steps.

Original: Under these assumptions we can reuse the Fisher matrixfor a certain amount of steps K , without recalculating it.
Revised: Under these assumptions we can reuse the Fisher matrix for a certain amount of steps K , without recalculating it.

Original: We call this technique, K-step update.
Revised: We call this technique, K-step update.

Original: Belief Networks.

Original: alg s lr
Revised: alg s lr

Original: LL
Revised: LL

Original: T/E
Revised: T/E

Original: WS [Table] [Table]
Revised: WS 0.

Original: 37s
Revised: 37s

Original: [Table]
Revised: RWS 0.

Original: 250s
Revised: 50s

Revised: NRWS 0.

Original: [Table]
Revised: [Table]

Original:  Experiments
Revised:  Experiments

Original: We use the binarized version of the MNIST database of handwritten digits [21].
Revised: We use the binarized version of the MNIST database of handwritten digits [21].

Original: The model architecture is a binary Helmholtz Machine with layers of sizes 300, 200, 100, 75, 50, 35, 30, 25, 20,15, 10, 10, as in Bornschein et al.
Revised: The model architecture is a binary Helmholtz Machine with layers of sizes 300, 200, 100, 75, 50, 35, 30, 25, 20, 15, 10, 10, as in Bornschein et al.

Original: The training is performed without data augmentation, withbinary variables in {− 1 , 1 } .
Revised: The training is performed without data augmentation, with binary variables in {− 1 , 1 } .

Original: We used a mini-batch size of 32 for all experiments and no regularizersnor decaying learning-rate.
Revised: We used a mini-batch size of 32 for all experiments and no regularizers nor decaying learning-rate.

Original: All experiments were run with CUDA optimized Tensorﬂow on Nvidia
Revised: All experiments were run with CUDA optimized Tensorﬂow on Nvidia

Original: GTX1080 Ti GPUs.
Revised: GTX1080 Ti GPUs.

Revised: [Figure]

Original: In Fig.
Revised: In Fig.

Original: 2a we present loss curves during training, for training and validation.
Revised: 2a we present loss curves during training, for training and validation.

Original: The advantage of the
Revised: The advantage of the

Original: NRWS in this case comes in the form of convergence to a better minimum.
Revised: NRWS in this case comes in the form of convergence to a better minimum.

Original: NRWS converges faster[Figure] than the their non-geometric counterparts in epochs.
Revised: NRWS converges faster than the their non-geometric counterparts in epochs.

Original: Even increasing the learning rate of RWS it isnot possible to achieve the same convergence speed (left panel of Fig. 2a).
Revised: Even increasing the learning rate of RWS it is not possible to achieve the same convergence speed (left panel of Fig. 2a).

Original: Even in time (right panel), NRWS is fast enough to compete with vanilla RWS.
Revised: Even in time (right panel), NRWS is fast enough to compete with vanilla RWS.

Original: While NRWS outperformes the other methods,a possible drawback is that the in real-time it takes roughly 5 times as long to reach the same amountof epochs as its non-natural counterpart, still reaching a better optimum.
Revised: While NRWS outperformes the other methods, a possible drawback is that the in real-time it takes roughly 5 times as long to reach the same amount of epochs as its non-natural counterpart, still reaching a better optimum.

Original: Reducing the number ofsamples for the importance weighting results in faster convergence but it could also impact theperformances.
Revised: Reducing the number of samples for the importance weighting results in faster convergence but it could also impact the performances.

Original: We compare our SGDs implementations of WS, RWS and NRWS with state-of-theart [10, 9] in Table 1.
Revised: We compare our SGDs implementations of WS, RWS and NRWS with state-of-theart [10, 9] in Table 1.

Original: Implementations from the literature also take advantage of accelerated gradientmethods (ADAM[18]), learning-rate decay (from 10 − 3 to 3 × 10 − 4 ), L 1 and L 2 regularizers and anincreased number of samples towards the end of the training (from 10 to 100 ), in order to achievebetter results.
Revised: Implementations from the literature also take advantage of accelerated gradient methods (ADAM[18]), learning-rate decay (from 10 − 3 to 3 × 10 − 4 ), L 1 and L 2 regularizers and an increased number of samples towards the end of the training (from 10 to 100 ), in order to achieve better results.

Original: While further hyperparameter tuning could be successfully employed to improve ourreported results, as well as variable learning rates, regularizers and number of samples [9], this isout of the scope of the present paper.
Revised: While further hyperparameter tuning could be successfully employed to improve our reported results, as well as variable learning rates, regularizers and number of samples [9], this is out of the scope of the present paper.

Original: Even with a simple training procedure (ﬁxed learning rate, noregularization and ﬁxed number of samples 10) we notice how the IS Likelihood on 10k samples iscomparable and evenslightly better than RWS  as reported from the literature [10].
Revised: Even with a simple training procedure (ﬁxed learning rate, no regularization and ﬁxed number of samples 10) we notice how the IS Likelihood on 10k samples is comparable and even better than RWS and BiHM as reported from the literature [10, 9].

Original: Additionally, we tested the algorithms using the NADAM (Nesterov-Adam) optimizer [17, 25, 18].
Revised: Additionally, we tested the algorithms using the NADAM (Nesterov-Adam) optimizer [17, 25, 18].

Original: In Fig.
Revised: In Fig.

Original: 2 we see that the NRWS also beneﬁts from the accelerated gradient method outperformingin epochs as well as in real-world time the RWS.
Revised: 2 we see that the NRWS also beneﬁts from the accelerated gradient method outperforming in epochs as well as in real-world time the RWS.

Original: Similarily to SGD, the speed up of RWS by usinga larger learning rate, does not help it catch up with NRWS.
Revised: Similarily to SGD, the speed up of RWS by using a larger learning rate, does not help it catch up with NRWS.

Original: However, the adaptive steps and theaccumulated momentum of NADAM are implicitly assuming an Euclidean metric in the tangentspace, which is not the geometrically correct approach.
Revised: However, the adaptive steps and the accumulated momentum of NADAM are implicitly assuming an Euclidean metric in the tangent space, which is not the geometrically correct approach.

Original: This motivates the exploration of adaptiveriemannian gradient methods for the NRWS algorithm, as a future work.
Revised: This motivates the exploration of adaptive riemannian gradient methods for the NRWS algorithm, as a future work.

Original:  Conclusions
Revised:  Conclusions

Original: We showed how the graphical structure of Helmholtz Machines allows for the efﬁcient computationof the Fisher matrix and thus the natural gradient during training, by exploiting the locality of theconnection matrix.
Revised: We showed how the graphical structure of Helmholtz Machines allows for the efﬁcient computation of the Fisher matrix and thus the natural gradient during training, by exploiting the locality of the connection matrix.

Original: We introduced the Natural Reweighted Wake-Sleep (NRWS) algorithm and wedemonstrated an improvement of the convergence during training for standard gradient descent overthe state-of-the-art baselines WS and RWS.
Revised: We introduced the Natural Reweighted Wake-Sleep (NRWS) algorithm and we demonstrated an improvement of the convergence during training for standard gradient descent over the state-of-the-art baselines WS and RWS.

Original: NRWS was not only faster to converge, but the obtainedoptimum resulted in better values for the IS likelihood estimation compared to the values reportedwith RWS [10].
Revised: NRWS was not only faster to converge, but the obtained optimum resulted in better values for the IS likelihood estimation compared to the values reported with RWS and with BiHM.

Revised: These results hold with a ﬁxed number of samples and learning rate, without taking advantage of decaying learning rate and increasing number of samples during training as its literature counterparts.

Original: When using the NADAM optimizer, NRWS maintain the speed advantage and theconvergence to a better optimum, compared with its non-geometric counterpart.
Revised: When using the NADAM optimizer, NRWS maintain the speed advantage and the convergence to a better optimum, compared with its non-geometric counterpart.

Original: This encouragesthe exploration of adaptive gradient methods for the Natural Reweighted Wake-Sleep algorithm inwhich the Fisher-Rao matrix is explicit considered for the momentum accumulation and the adaptivestep.
Revised: This encourages the exploration of adaptive gradient methods for the Natural Reweighted Wake-Sleep algorithm in which the Fisher-Rao matrix is explicit considered for the momentum accumulation and the adaptive step.

Original: Moreover, we plan to further study ways to obtain a better estimation of the Fisher matrixwhile training.
Revised: Moreover, we plan to further study ways to obtain a better estimation of the Fisher matrix while training.

Original: As a ﬁnal remark, we highlight that since the computation of the Fisher matrixis only dependent on the underlying statistical model, as a consequence other algorithms for thetraining of HMs could beneﬁt from the use of the natural gradient, such as Bidirectional Helmholtz 
Revised: As a ﬁnal remark, we highlight that since the computation of the Fisher matrix is only dependent on the underlying statistical model, also other algorithms for the training of HMs could beneﬁt from the use of the natural gradient, such as Bidirectional Helmholtz Machines.

Original: Machines [9].
Revised: This work was supported by the DeepRiemann project, co-funded by the European Regional Development Fund and the Romanian Government through the Competitiveness Operational Program 2014-2020, Action 1.1.4, project ID P_37_714, contract no. 136/27.09.2016.