Original: O N THE MAPPING BETWEEN H OPFIELD NETWORKS AND R ESTRICTED B OLTZMANN M ACHINES
Revised: O N THE MAPPING BETWEEN H OPFIELD NETWORKS AND R ESTRICTED B OLTZMANN M ACHINES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Hopﬁeld networks (HNs) and Restricted Boltzmann Machines (RBMs) are two important models at the interface of statistical physics, machine learning, and neuroscience.
Revised: Hopﬁeld networks (HNs) and Restricted Boltzmann Machines (RBMs) are two important models at the interface of statistical physics, machine learning, and neuroscience.

Original: Recently, there has been interest in the relationship between HNs and
Revised: Recently, there has been interest in the relationship between HNs and

Original: RBMs, due to their similarity under the statistical mechanics formalism.
Revised: RBMs, due to their similarity under the statistical mechanics formalism.

Original: An exact mapping between HNs and RBMs has been previously noted for the special case of orthogonal (“uncorrelated”)
Revised: An exact mapping between HNs and RBMs has been previously noted for the special case of orthogonal (“uncorrelated”)

Original: encoded patterns.
Revised: encoded patterns.

Original: We present here an exact mapping in the general case of correlated pattern HNs, which are more broadly applicable to existing datasets.
Revised: We present here an exact mapping in the general case of correlated pattern HNs, which are more broadly applicable to existing datasets.

Original: Speciﬁcally, we show that any HN with N binary variables and p < N arbitrary binary patterns can be transformed into an RBM with N binary visible variables and p gaussian hidden variables.
Revised: Speciﬁcally, we show that any HN with N binary variables and p < N arbitrary binary patterns can be transformed into an RBM with N binary visible variables and p gaussian hidden variables.

Original: We outline the conditions under which the reverse mapping exists, and conduct experiments on the MNIST dataset which suggest the mapping provides a useful initialization to the RBM weights.
Revised: We outline the conditions under which the reverse mapping exists, and conduct experiments on the MNIST dataset which suggest the mapping provides a useful initialization to the RBM weights.

Original: We discuss extensions, the potential importance of this correspondence for the training of RBMs, and for understanding the performance of deep architectures which utilize RBMs.
Revised: We discuss extensions, the potential importance of this correspondence for the training of RBMs, and for understanding the performance of deep architectures which utilize RBMs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Hopﬁeld networks (HNs) (Hopﬁeld, 1982; Amit, 1989) are a classical neural network architecture that can store prescribed patterns as ﬁxed-point attractors of a dynamical system.
Revised: Hopﬁeld networks (HNs) (Hopﬁeld, 1982; Amit, 1989) are a classical neural network architecture that can store prescribed patterns as ﬁxed-point attractors of a dynamical system.

Original: In their standard formulation with binary valued units, HNs can be regarded as spin glasses with pairwise interactions
Revised: In their standard formulation with binary valued units, HNs can be regarded as spin glasses with pairwise interactions

Original: J ij that are fully determined by the patterns to be encoded.
Revised: J ij that are fully determined by the patterns to be encoded.

Original: HNs have been extensively studied in the statistical mechanics literature (e.g. (Kanter & Sompolinsky, 1987;
Revised: HNs have been extensively studied in the statistical mechanics literature (e.g. (Kanter & Sompolinsky, 1987;

Original: Amit et al., 1985)), where they can be seen as an interpolation between the ferromagnetic Ising model ( p = 1 pattern) and the Sherrington-Kirkpatrick spin glass model (many random patterns) (Kirkpatrick & Sherrington, 1978; Barra & Guerra, 2008).
Revised: Amit et al., 1985)), where they can be seen as an interpolation between the ferromagnetic Ising model ( p = 1 pattern) and the Sherrington-Kirkpatrick spin glass model (many random patterns) (Kirkpatrick & Sherrington, 1978; Barra & Guerra, 2008).

Original: By encoding patterns as dynamical attractors which are robust to perturbations, HNs provide an elegant solution to pattern recognition and classiﬁcation tasks.
Revised: By encoding patterns as dynamical attractors which are robust to perturbations, HNs provide an elegant solution to pattern recognition and classiﬁcation tasks.

Original: They are considered the prototypical attractor neural network, and are the historical precursor to modern recurrent neural networks.
Revised: They are considered the prototypical attractor neural network, and are the historical precursor to modern recurrent neural networks.

Original: Concurrently, spin glasses have been used extensively in the historical machine learning literature where they comprise a sub-class of “Boltzmann machines” (BMs) (Ackley et al., 1985).
Revised: Concurrently, spin glasses have been used extensively in the historical machine learning literature where they comprise a sub-class of “Boltzmann machines” (BMs) (Ackley et al., 1985).

Original: Given a collection of data samples drawn from a data distribution, one is generally interested in “training” a BM by tuning its weights J ij such that its equilibrium distribution can reproduce the data distribution as closely as possible (Hinton, 2012).
Revised: Given a collection of data samples drawn from a data distribution, one is generally interested in “training” a BM by tuning its weights J ij such that its equilibrium distribution can reproduce the data distribution as closely as possible (Hinton, 2012).

Original: The resulting optimization problem is dramatically simpliﬁed when the network has a two-layer structure where each layer has no self-interactions, so that there are only inter-layer connections (Hinton, 2012)
Revised: The resulting optimization problem is dramatically simpliﬁed when the network has a two-layer structure where each layer has no self-interactions, so that there are only inter-layer connections (Hinton, 2012)

Original: (see Fig. 1).
Revised: (see Fig. 1).

Original: This architecture is known as a Restricted Boltzmann Machine (RBM), and the two layers are sometimes called the visible layer and the hidden layer.
Revised: This architecture is known as a Restricted Boltzmann Machine (RBM), and the two layers are sometimes called the visible layer and the hidden layer.

Original: The visible layer characteristics (dimension, type of units) are determined by the training data, whereas the hidden layer can have binary or continuous units and the dimension is chosen somewhat arbitrarily.
Revised: The visible layer characteristics (dimension, type of units) are determined by the training data, whereas the hidden layer can have binary or continuous units and the dimension is chosen somewhat arbitrarily.

Original: In addition to generative modelling, RBMs  have been used for a variety of learning tasks, such as classiﬁcation and feature extraction.
Revised: In addition to generative modelling, RBMs and their multi-layer extensions have been used for a variety of learning tasks, such as classiﬁcation, feature extraction, and dimension reduction (e.g. Salakhutdinov et al.

Revised: ; Hinton & Salakhutdinov (2006)).

Original: Recently, there has been interest in the relationship between HNs and RBMs, as both are built on the Ising model formalism and fulﬁll similar roles, with the aim of better understanding RBM behaviour and potentially improving performance.
Revised: Recently, there has been interest in the relationship between HNs and RBMs, as both are built on the Ising model formalism and fulﬁll similar roles, with the aim of better understanding RBM behaviour and potentially improving performance.

Original: An exact mapping between HNs and RBMs has been previously noted for the special case of uncorrelated (orthogonal) patterns (Barra et al., 2012).
Revised: An exact mapping between HNs and RBMs has been previously noted for the special case of uncorrelated (orthogonal) patterns (Barra et al., 2012).

Original: Several related models have since been studied (Agliari et al., 2013; M´ezard, 2017), which partially relax the uncorrelated pattern constraint.
Revised: Several related models have since been studied (Agliari et al., 2013; M´ezard, 2017), which partially relax the uncorrelated pattern constraint.

Original: However, the patterns observed in most real datasets exhibit signiﬁcant correlations, precluding the use of these approaches.
Revised: However, the patterns observed in most real datasets exhibit signiﬁcant correlations, precluding the use of these approaches.

Original: In this paper, we demonstrate exact correspondence between HNs and RBMs in the fully general case of correlated pattern HNs.
Revised: In this paper, we demonstrate exact correspondence between HNs and RBMs in the fully general case of correlated pattern HNs.

Original: Speciﬁcally, we show that any HN with N binary units and p < N arbitrary binary patterns encoded via the projection rule (Kanter & Sompolinsky, 1987; Personnaz et al., 1986), can be transformed into an RBM with N binary and p gaussian variables.
Revised: Speciﬁcally, we show that any HN with N binary units and p < N arbitrary binary patterns encoded via the projection rule (Kanter & Sompolinsky, 1987; Personnaz et al., 1986), can be transformed into an RBM with N binary and p gaussian variables.

Original: We then characterize when the reverse map from RBMs to HNs can be made.
Revised: We then characterize when the reverse map from RBMs to HNs can be made.

Original: We consider a practical example using the mapping, and discuss the potential importance of this correspondence for the training and interpretability of RBMs.
Revised: We consider a practical example using the mapping, and discuss the potential importance of this correspondence for the training and interpretability of RBMs.

Original:  R ESULTS
Revised:  R ESULTS

Original: We ﬁrst introduce the classical solution to the problem of encoding N -dimensional binary {− 1 , +1 } vectors { ξ µ } pµ =1 , termed “patterns”, as global minima of a pairwise spin glass H ( s ) =
Revised: We ﬁrst introduce the classical solution to the problem of encoding N -dimensional binary {− 1 , +1 } vectors { ξ µ } p µ =1 , termed “patterns”, as global minima of a pairwise spin glass H ( s ) =

Original: − 12 s T Js .
Revised: − 12 s T Js .

Original: This is often framed as a pattern retrieval problem, where the goal is to specify or learn J ij such that an energy-decreasing update rule for H ( s ) converges to the patterns (i.e. they are stable ﬁxed points).
Revised: This is often framed as a pattern retrieval problem, where the goal is to specify or learn J ij such that an energy-decreasing update rule for H ( s ) converges to the patterns (i.e. they are stable ﬁxed points).

Original: Consider the N × p matrix ξ with the p patterns as its columns.
Revised: Consider the N × p matrix ξ with the p patterns as its columns.

Original: Then the classical prescription known as the projection rule (or pseudo-inverse rule) (Kanter & Sompolinsky, 1987; Personnaz et al., 1986), J = ξ ( ξ T ξ ) − 1 ξ T , guarantees that the p patterns will be global minima of H ( s ) .
Revised: Then the classical prescription known as the projection rule (or pseudo-inverse rule) (Kanter & Sompolinsky, 1987; Personnaz et al., 1986), J = ξ ( ξ T ξ ) − 1 ξ T , guarantees that the p patterns will be global minima of H ( s ) .

Original: This resulting spin model is commonly called a (projection) Hopﬁeld network, and has the Hamiltonian
Revised: This resulting spin model is commonly called a (projection) Hopﬁeld network, and has the Hamiltonian

Original: [Equation]
Revised: [Equation]

Original: Note that ξ T ξ invertibility is guaranteed as long as the patterns are linearly independent (we therefore require p ≤ N ).
Revised: Note that ξ T ξ invertibility is guaranteed as long as the patterns are linearly independent (we therefore require p ≤ N ).

Original: Also note that in the special (rare) case of orthogonal patterns ξ µ · ξ ν = Nδ µν (also called “uncorrelated”), studied in the previous work (Barra et al., 2012), one has ξ T ξ = N I and so the pseudo-inverse interactions reduce to the well-known Hebbian form J = 1 N ξ T ξ T (the properties of which are studied extensively in Amit et al.
Revised: Also note that in the special (rare) case of orthogonal patterns ξ µ · ξ ν = Nδ µν (also called “uncorrelated”), studied in the previous work (Barra et al., 2012), one has ξ T ξ = N I and so the pseudo-inverse interactions reduce to the well-known Hebbian form J = 1 N ξξ T (the properties of which are studied extensively in Amit et al.

Original: Additional details on the projection HN Eq.
Revised: Additional details on the projection HN Eq.

Original: (1) are provided in Appendix A. To make progress in analyzing Eq.
Revised: (1) are provided in Appendix A. To make progress in analyzing Eq.

Original: (1), we ﬁrst consider a transformation of ξ which eliminates the inverse factor.
Revised: (1), we ﬁrst consider a transformation of ξ which eliminates the inverse factor.

Original: 2. M APPING A H OPFIELD N ETWORK TO A R ESTRICTED B OLTZMANN M ACHINE
Revised: 2. M APPING A H OPFIELD N ETWORK TO A R ESTRICTED B OLTZMANN M ACHINE

Original: In order to obtain a more useful representation of the quadratic form Eq.
Revised: In order to obtain a more useful representation of the quadratic form Eq.

Original: (1) (for our purposes), we utilize the QR-decomposition (Schott & Stewart, 1999) of ξ to “orthogonalize” the patterns,
Revised: (1) (for our purposes), we utilize the QR-decomposition (Schott & Stewart, 1999) of ξ to “orthogonalize” the patterns,

Original: [Equation]
Revised: [Equation]

Original: with Q ∈ R N × p , R ∈ R p × p .
Revised: with Q ∈ R N × p , R ∈ R p × p .

Original: The columns of Q are the orthogonalized patterns, and form an orthonormal basis (of non-binary vectors) for the p -dimensional subspace spanned by the binary patterns.
Revised: The columns of Q are the orthogonalized patterns, and form an orthonormal basis (of non-binary vectors) for the p -dimensional subspace spanned by the binary patterns.

Original: R is upper triangular, and if its diagonals are held positive then Q and R are both unique (Schott & Stewart, 1999).
Revised: R is upper triangular, and if its diagonals are held positive then Q and R are both unique (Schott & Stewart, 1999).

Original: Note both the order and sign of the columns of ξ are irrelevant for HN pattern recall, so there are n = 2 p · p !
Revised: Note both the order and sign of the columns of ξ are irrelevant for HN pattern recall, so there are n = 2 p · p !

Original: possible Q , R pairs.
Revised: possible Q , R pairs.

Original: Fixing a pattern ordering, we can use the orthogonality of Q to re-write the interaction matrix as
Revised: Fixing a pattern ordering, we can use the orthogonality of Q to re-write the interaction matrix as

Original: [Equation]
Revised: [Equation]

Revised: (the last equality follows from ( R T R ) − 1 =

Revised: R − 1 ( R T ) − 1 ).

Original: resembling the simple Hebbian rule but with non-binary orthogonal patterns.
Revised: Eq. (3) resembles the simple Hebbian rule but with non-binary orthogonal patterns.

Original: Deﬁning q ≡ Q T s in analogy to the classical pattern overlap parameter m ≡ 1 N ξ T s (Amit et al., 1985), we have 
Revised: Deﬁning q ≡ Q T s in analogy to the classical pattern overlap parameter m ≡ 1 N ξ T s (Amit et al., 1985), we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using a Gaussian integral as in Amit et al.
Revised: Using a Gaussian integral as in Amit et al.

Original: Barra et al.
Revised: Barra et al.

Original: (2012); M´ezard (2017) to transform (exactly) the partition function Z ≡ (cid:80) { s } e − βH ( s ) of Eq.
Revised: (2012); M´ezard (2017) to transform (exactly) the partition function Z ≡ (cid:80) { s } e − βH ( s ) of Eq.

Original: (1), we get
Revised: (1), we get

Revised: Z

Original: [Equation]
Revised: [Equation]

Revised: The second line can be seen as the partition function of an expanded Hamiltonian for the N (binary) original variables { s i } and the p (continuous) auxiliary variables { λ µ } , i.e.

Original: [Equation]
Revised: [Equation]

Original: Note that this is the Hamiltonian of a binary-continuous RBM with inter-layer weights Q iµ .
Revised: Note that this is the Hamiltonian of a binary-continuous RBM with inter-layer weights Q iµ .

Original: The original HN is therefore equivalent to an RBM described by Eq.
Revised: The original HN is therefore equivalent to an RBM described by Eq.

Original: (6) (depicted in Fig. 1).
Revised: (6) (depicted in Fig. 1).

Original: As mentioned above, there are many RBMs which correspond to the same HN due to the combinatorics of choosing Q .
Revised: As mentioned above, there are many RBMs which correspond to the same HN due to the combinatorics of choosing Q .

Original: In fact, instead of QR factorization one can use any decomposition which satisﬁes
Revised: In fact, instead of QR factorization one can use any decomposition which satisﬁes

Original: J = UU T , with orthogonal U ∈ R N × p (see Appendix B), in which case U acts as the RBM weights.
Revised: J = UU T , with orthogonal U ∈ R N × p (see Appendix B), in which case U acts as the RBM weights.

Original: Also note the inclusion of an applied ﬁeld term − (cid:80) i b i s i in Eq.
Revised: Also note the inclusion of an applied ﬁeld term − (cid:80) i b i s i in Eq.

Original: (1) trivially carries through the procedure, i.e. 
Revised: (1) trivially carries through the procedure, i.e. ˜ H

Original: [Equation]
Revised: [Equation]

Original: (RBMs).
Revised: ( { s [Equation] [Equation]

Original: The HN has Nbinary units and pairwiseinteractions J deﬁned by p < N (possibly correlated) patterns { ξ µ } pµ=1 .
Revised: chines ( has N and pairwise J deﬁned by p < N (possibly correlated) patterns { ξ µ } pµ

Original: The patterns are encoded as minima of Eq.
Revised: The patterns are encoded as minima of Eq.

Original: (1) through the projection rule J = ξ ( ξ T ξ ) − 1 ξ T , where ξ µ form the columns of ξ .
Revised: (1) through the projection rule J = ξ ( ξ T ξ ) − 1 ξ T , where ξ µ form the columns of ξ .

Original: We orthogonalize the patterns through a QR decomposition ξ = QR .
Revised: We orthogonalize the patterns through a QR decomposition ξ = QR .

Original: The HN is equivalent to an RBM with N binary visible units and p gaussian hidden units with inter
Revised: The HN is equivalent to an RBM with N binary visible units and p gaussian hidden units with inter-layer weights deﬁned as the patterns Q iµ

Original:  Instead of working with the joint form Eq.
Revised: Eq Instead of working with the joint form Eq.

Original: (6), one could take a different direction from Eq.
Revised: (6), one could take a different direction from Eq.

Original: (5) and sum out the original variables { s i } , i.e.
Revised: (5) and sum out the original variables { s i } , i.e.

Original: [Equation]
Revised: [Equation]

Original: In the saddle point approximation for the free energy F =  − β − 1 ln Z , we ﬁnd
Revised: This continuous, p -dimensional representation is useful for numerical estimation of Z (Section 3.1). We may write Eq. (7) as Z = (cid:82) e − F 0 ( λ ) dλ µ , where [Equation]

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (8) is an approximate Lyapunov function for the mean dynamics of { λ µ } ; ∇ λ F 0 describes the effective behaviour of the stochastic dynamics of the N binary variables { s i } at temperature β − 1 .
Revised: (8) is an approximate Lyapunov function for the mean dynamics of { λ µ } ; ∇ λ F 0 describes the effective behaviour of the stochastic dynamics of the N binary variables { s i } at temperature β − 1 .

Original: 2. C OMMENTS ON THE REVERSE MAPPING
Revised: 2. C OMMENTS ON THE REVERSE MAPPING

Original: With the mapping from HNs (with correlated patterns) to RBMs established, we now consider the reverse direction.
Revised: With the mapping from HNs (with correlated patterns) to RBMs established, we now consider the reverse direction.

Original: Consider a binary-continuous RBM with inter-layer weights W iµ which couple a visible layer of N binary variables { s i } to a hidden layer of p continuous variables { λ µ } ,
Revised: Consider a binary-continuous RBM with inter-layer weights W iµ which couple a visible layer of N binary variables { s i } to a hidden layer of p continuous variables { λ µ } ,

Original: [Equation]
Revised: [Equation]

Original: Here we use W instead of Q for the RBM weights to emphasize that the RBM is not necessarily an HN.
Revised: Here we use W instead of Q for the RBM weights to emphasize that the RBM is not necessarily an HN.

Original: First, following Mehta et al.
Revised: First, following Mehta et al.

Original: (2019), we transform the RBM to a BM with binary states by integrating out the hidden variables.
Revised: (2019), we transform the RBM to a BM with binary states by integrating out the hidden variables.

Original: The corresponding Hamiltonian for the visible units alone is (see Appendix D for details),
Revised: The corresponding Hamiltonian for the visible units alone is (see Appendix D for details),

Original: [Equation]
Revised: [Equation]

Original: a pairwise Ising model with a particular coupling structure J ij = (cid:80) µ W iµ W jµ , which in vector form is
Revised: a pairwise Ising model with a particular coupling structure J ij = (cid:80) µ [Equation] iµ W jµ , which in vector form is

Original: [Equation]
Revised: [Equation]

Original: where { w µ } are the pthe columns of W .
Revised: where { w µ } are the p columns of W .

Original: In general, this Ising model Eq.
Revised: In general, this Ising model Eq.

Original: (10) produced by integrating out the hidden variables need not have Hopﬁeld structure (discussed below).
Revised: (10) produced by integrating out the hidden variables need not have Hopﬁeld structure (discussed below).

Original: However, it automatically does (as noted in Barra et al.
Revised: However, it automatically does (as noted in Barra et al.

Original: , in the very special case where W iµ ∈ {− 1 , +1 } .
Revised: , in the very special case where W iµ ∈ {− 1 , +1 } .

Original: In that case, the binary patterns are simply { w µ } , so that Eq.
Revised: In that case, the binary patterns are simply { w µ } , so that Eq.

Original: (11) represents a Hopﬁeld network with the Hebbian prescription.
Revised: (11) represents a Hopﬁeld network with the Hebbian prescription.

Original: This situation is likely rare and may only arise as a by-product of constrained training; for a generically trained
Revised: This situation is likely rare and may only arise as a by-product of constrained training; for a generically trained

Original: RBM the weights will not be binary.
Revised: RBM the weights will not be binary.

Original: It is therefore interesting to clarify when and how real-valued
Revised: It is therefore interesting to clarify when and how real-valued

Original: RBM interactions W can be associated with HNs.
Revised: RBM interactions W can be associated with HNs.

Original: Approximate binary representation of W : In Section 2.1, we orthogonalized the binary matrix ξ via the QR decomposition ξ = QR , where Q is an orthogonal (but non-binary) matrix, which allowed us to map a projection HN (deﬁned by its patterns ξ , Eq.
Revised: Approximate binary representation of W : In Section 2.1, we orthogonalized the binary matrix ξ via the QR decomposition ξ = QR , where Q is an orthogonal (but non-binary) matrix, which allowed us to map a projection HN (deﬁned by its patterns ξ , Eq.

Original: (1)) to an RBM (deﬁned by its inter-layer weights Q , Eq.
Revised: (1)) to an RBM (deﬁned by its inter-layer weights Q , Eq.

Original: Here we consider the reverse map.
Revised: Here we consider the reverse map.

Original: Given a trained RBM with weights W ∈ R N × p , we look for an invertible transformation X ∈ R p × p which binarizes W .
Revised: Given a trained RBM with weights W ∈ R N × p , we look for an invertible transformation X ∈ R p × p which binarizes W .

Original: We make the mild assumption that W is rank p .
Revised: We make the mild assumption that W is rank p .

Original: If we ﬁnd such an X , then B = W X will be the Hopﬁeld pattern matrix (analogous to ξ ), with B iµ ∈ {− 1 , +1 } .
Revised: If we ﬁnd such an X , then B = W X will be the Hopﬁeld pattern matrix (analogous to ξ ), with B iµ ∈ {− 1 , +1 } .

Original: This is a non-trivial problem, and an exact solution is not guaranteed.
Revised: This is a non-trivial problem, and an exact solution is not guaranteed.

Original: As a ﬁrst step to study the problem, we relax it to that of ﬁnding an invertible matrix X  which minimizes the binarization error
Revised: As a ﬁrst step to study the problem, we relax it to that of ﬁnding a matrix X ∈ GL p ( R ) (i.e. invertible, p × p , real) which minimizes the binarization error

Original: [Equation]
Revised: [Equation]

Original: We denote the approximately binary transformation of W via a particular solution X by
Revised: We denote the approximately binary transformation of W via a particular solution X by

Original: [Equation]
Revised: [Equation]

Original: We also deﬁne the associated error matrix E ≡ B p − sgn ( B p ) .
Revised: We also deﬁne the associated error matrix E ≡ B p − sgn ( B p ) .

Original: We stress that B p is non-binary and approximates B ≡ sgn ( B p ) , the columns of which will be HN patterns under certain conditions on
Revised: We stress that B p is non-binary and approximates B ≡ sgn ( B p ) , the columns of which will be HN patterns under certain conditions on

Original: E .
Revised: E .

Original: We provide an initial characterization in Appendix D.
Revised: We provide an initial characterization in Appendix D.

Original:  E XPERIMENTS ON MNIST D ATASET
Revised:  E XPERIMENTS ON MNIST D ATASET

Original: Next we investigate whether the Hopﬁeld-RBM correspondence can provide an advantage for training binary-gaussian RBMs.
Revised: Next we investigate whether the Hopﬁeld-RBM correspondence can provide an advantage for training binary-gaussian RBMs.

Original: We consider the popular MNIST dataset of handwritten digits (LeCun et al., 1998) which consists of 28 × 28 images of handwritten images, with greyscale pixel valuesto 255 .
Revised: We consider the popular MNIST dataset of handwritten digits (LeCun et al., 1998) which consists of 28 × 28 images of handwritten images, with greyscale pixel valuesto 255 .

Original: We treat the sample images as N ≡ 784 dimensional binary vectors of {− 1 , +1 } by setting all non-zero values to +1 .
Revised: We treat the sample images as N ≡ 784 dimensional binary vectors of {− 1 , +1 } by setting all non-zero values to +1 .

Original: The dataset includes M ≡ 60 , 000 training images and 10 , 000 testing images, as well as their class labels µ ∈ { 0 , . .
Revised: The dataset includes M ≡ 60 , 000 training images and 10 , 000 testing images, as well as their class labels µ ∈ { 0 , . .

Original: 3. G ENERATIVE OBJECTIVE
Revised: 3. G ENERATIVE OBJECTIVE

Original: The primary task for generative models such as RBMs is to reproduce a data distribution.
Revised: The primary task for generative models such as RBMs is to reproduce a data distribution.

Original: Given a data distribution p data , the generative objective is to train a model (here an RBM deﬁned by its parameters θ ), such that the model distribution p θ is as close to p data as possible.
Revised: Given a data distribution p data , the generative objective is to train a model (here an RBM deﬁned by its parameters θ ), such that the model distribution p θ is as close to p data as possible.

Original: This is often quantiﬁed by the Kullback-Leibler (KL) divergence D KL ( p data (cid:107) p θ )
Revised: This is often quantiﬁed by the Kullback-Leibler (KL) divergence D KL ( p data (cid:107) [Equation]

Original: = (cid:80) s p data ( s ) ln (cid:16) p data ( s )p
Revised: = (cid:80) s p data ( s ) ln (cid:16) p data ( s )

Original: θ ( s ) (cid:17) .
Revised: [Equation] s ) (cid:17) .

Original: One generally does not have access to the actual data distribution, instead there is usually a representative training set S =
Revised: One generally does not have access to the actual data distribution, instead there is usually a representative training set S =

Original: { s a } M a =1 sampled from it.
Revised: { s a } Ma =1 sampled from it.

Original: As the data distribution is constant with respect to θ , the generative objective is equivalent to maximizing L ( θ ) = 1 M (cid:80) a ln p θ ( s a ) .
Revised: As the data distribution is constant with respect to θ , the generative objective is equivalent to maximizing L ( θ ) = 1 M (cid:80) a ln p θ ( s a ) .

Original: 3.1. H OPFIELD 
Revised: 3.1. H OPFIELD RBM SPECIFICATION

Original: RBM SPECIFICATION

Original: With labelled classes of training data, speciﬁcation of an RBM via a one-shot Hopﬁeld rule (“Hopﬁeld RBM”) is straightforward.
Revised: With labelled classes of training data, speciﬁcation of an RBM via a one-shot Hopﬁeld rule (“Hopﬁeld RBM”) is straightforward.

Original: In the simplest approach, we deﬁne p = 10 representative patterns via the  class means
Revised: In the simplest approach, we deﬁne p = 10 representative patterns via the (binarized) class means

Original: [Equation]
Revised: [Equation]

Original:  where µ ∈ { 0 , . . .
Revised: (14) where µ ∈ { 0 , . . .

Original: , 9 } and S µ is the set of sample images for class µ .
Revised: , 9 } and S µ is the set of sample images for class µ .

Original: These patterns comprise the columns of the N × p pattern matrix ξ , which is then orthogonalized as in Eq.
Revised: These patterns comprise the columns of the N × p pattern matrix ξ , which is then orthogonalized as in Eq.

Original: (2) to obtain the RBM weights W which couple N binary visible units to p gaussian hidden units.
Revised: (2) to obtain the RBM weights W which couple N binary visible units to p gaussian hidden units.

Original: We also consider reﬁning this approach by considering sub-classes within each class, representing, for example, the different ways one might draw a “7”.
Revised: We also consider reﬁning this approach by considering sub-classes within each class, representing, for example, the different ways one might draw a “7”.

Original: As a proof of principle, we split each digit class into k sub-patterns using hierarchical clustering.
Revised: As a proof of principle, we split each digit class into k sub-patterns using hierarchical clustering.

Original: We found good results with Agglomerative clustering using Ward linkage and Euclidean distance (see Murtagh & Contreras (2012) for an overview of this and related methods).
Revised: We found good results with Agglomerative clustering using Ward linkage and Euclidean distance (see Murtagh & Contreras (2012) for an overview of this and related methods).

Original: In this way, we can deﬁne a hierarchy of Hopﬁeld RBMs.
Revised: In this way, we can deﬁne a hierarchy of Hopﬁeld RBMs.

Original: At one end, k = 1 , we have our simplest RBM which has p = 10 hidden units and encodespatterns (using Eq.
Revised: At one end, k = 1 , we have our simplest RBM which has p = 10 hidden units and encodes 10 patterns (using Eq.

Original: (14)), one for each digit class.
Revised: (14)), one for each digit class.

Original: At the other end,0 (cid:28) 10 k/N < 1 , we can specify increasingly reﬁned RBMs that encode k sub-patterns for each of the 10 digit classes, for a total of p = 10 k patterns and hidden units.
Revised: At the other end, 10 k/N → 1 , we can specify increasingly reﬁned RBMs that encode k sub-patterns for each of the 10 digit classes, for a total of p = 10 k patterns and hidden units.

Original: This approach has an additional cost of identifying the sub-classes, but is still typically faster than training the RBM weights directly (discussed below).
Revised: This approach has an additional cost of identifying the sub-classes, but is still typically faster than training the RBM weights directly (discussed below).

Original: The generative performance as a function of k and β is shown in Fig. 2, and increases monotonically with k in the range plotted.
Revised: The generative performance as a function of k and β is shown in Fig. 2, and increases monotonically with k in the range plotted.

Original: If β is too high (very low temperature) the free energy basins will be very deep directly at the patterns, and so the model distribution will not capture the diversity of images from the data.
Revised: If β is too high (very low temperature) the free energy basins will be very deep directly at the patterns, and so the model distribution will not capture the diversity of images from the data.

Original: If β is too low (high temperature), there is a “melting transition” where the original pattern basins disappear entirely, and the data will therefore be poorly modelled.
Revised: If β is too low (high temperature), there is a “melting transition” where the original pattern basins disappear entirely, and the data will therefore be poorly modelled.

Original: Taking α = p/N ∼ 0 .
Revised: Taking α = p/N ∼ 0 .

Original: 1 , Fig.
Revised: 1 , Fig.

Original: 1 of Kanter & Sompolinsky (1987) predicts β c ≈ 1 .
Revised: 1 of Kanter & Sompolinsky (1987) predicts β c ≈ 1 .

Original: 5 for the theoretical melting transition for the pattern basins.
Revised: 5 for the theoretical melting transition for the pattern basins.

Original: Interestingly, this is quite close to our observed peak near β = 2 .
Revised: Interestingly, this is quite close to our observed peak near β = 2 .

Original: Note also as k is increased, the generative performance is sustained at lower temperatures
Revised: Note also as k is increased, the generative performance is sustained at lower temperatures.

Original: In situations where one already has access to the class labels, this approach to obtain RBM weights is very fast.
Revised: In situations where one already has access to the class labels, this approach to obtain RBM weights is very fast.

Original: The class averaging has negligible computational cost O ( MN ) for the whole training set ( M samples), and the QR decomposition has a modest complexity of O
Revised: The class averaging has negligible computational cost O ( MN ) for the whole training set ( M samples), and the QR decomposition has a modest complexity of O

Original: ( Np 2 ) (Schott & Stewart, 1999).
Revised: ( Np 2 ) (Schott & Stewart, 1999).

Original: Conventional RBM training, discussed below, requires signiﬁcantly more computation.
Revised: Conventional RBM training, discussed below, requires signiﬁcantly more computation.

Original: 3.1. C ONVENTIONAL RBM TRAINING
Revised: 3.1. C ONVENTIONAL RBM TRAINING

Original: RBM training is performed through gradient ascent on the log-likelihood of the data, L ( θ ) =
Revised: RBM training is performed through gradient ascent on the log-likelihood of the data, L ( θ ) =

Original: M (cid:80) a ln p θ ( s a ) (equivalent here to minimizing KL divergence, as mentioned above).
Revised: M (cid:80) a ln p θ ( s a ) (equivalent here to minimizing KL divergence, as mentioned above).

Original: We are focused here on the weights W in order to compare to the Hopﬁeld RBM weights, and so we neglect the biases on both layers.
Revised: We are focused here on the weights W in order to compare to the Hopﬁeld RBM weights, and so we neglect the biases on both layers.

Original: As is common (Hinton, 2012), we approximate the total gradient by splitting the training dataset into “mini-batches”, denoted B .
Revised: As is common (Hinton, 2012), we approximate the total gradient by splitting the training dataset into “mini-batches”, denoted B .

Original: The resulting gradient ascent rule for the weights is (see Appendix E)
Revised: The resulting gradient ascent rule for the weights is (see Appendix E)

Original: [Equation]
Revised: [Equation]

Original: where (cid:104) s i λ 
Revised: where (cid:104) s i λ µ (cid:105) model ≡ Z − 1 (cid:80) { s } (cid:82) s i λ µ e − βH

Original: [Table] is an average over the model distribution.
Revised: ( s , λ ) (cid:81) µ dλ µ is an average over the model distribution.

Original: Below, we use a moderate mini-batch size of | B | = 100 and η = 10 − 4 .

Original: The ﬁrst bracketed term of Eq.
Revised: The ﬁrst bracketed term of Eq.

Original: (15) is simple to calculate at each iteration of the weights.
Revised: (15) is simple to calculate at each iteration of the weights.

Original: The second term, however, is intractable as it requires one to calculate the partition function Z .
Revised: The second term, however, is intractable as it requires one to calculate the partition function Z .

Original: We instead approximate it using contrastive divergence (CD) (Hinton, 2012).
Revised: We instead approximate it using contrastive divergence (CD) (Carreira-Perpinan & Hinton, 2005; Hinton, 2012).

Original: See Appendix E for details.
Revised: See Appendix E for details.

Original: Each full step of RBM weight updates involves O ( KBNp ) operations (Melchior et al., 2017).
Revised: Each full step of RBM weight updates involves O ( KBNp ) operations (Melchior et al., 2017).

Original: Training generally involves many mini-batch iterations, such that the entire dataset is iterated over (“one epoch”)
Revised: Training generally involves many mini-batch iterations, such that the entire dataset is iterated over (“one epoch”)

Original: many times.
Revised: many times.

Original: In our experiments we train for 50 epochs with mini-batches of size( 3 · 10 5 weight updates), so the overall training time can be extensive compared to the one-shot Hopﬁeld approach presented above.
Revised: In our experiments we train for 50 epochs with mini-batches of size 100 ( 3 · 10 5 weight updates), so the overall training time can be extensive compared to the one-shot Hopﬁeld approach presented above.

Original: For further details on RBM training see e.g. Hinton (2012); Melchior et al.
Revised: For further details on RBM training see e.g. Hinton (2012); Melchior et al.

Original: In Fig.
Revised: In Fig.

Original: 3, we give an example of the Hopﬁeld RBM weights (for k = 1 ), as well as how they evolve during conventional RBM training.
Revised: 3, we give an example of the Hopﬁeld RBM weights (for k = 1 ), as well as how they evolve during conventional RBM training.

Original: Note Fig. 3(a), (b) appear qualitatively similar, suggesting  the generative training simply “ﬁne tunes” the deterministic initialization Q from Eqs. 
Revised: Note Fig. 3(a), (b) appear qualitatively similar, suggesting that the proposed initialization Q from Eqs. (2), (14) may be near a local optimum of the objective.

Original: [Figure]
Revised: [Figure]

Original: In Fig. 4, we compare conventional RBM training on two different weight initializations:  W iµ ∼ N (0 , 0 . 01) ; and our proposed  Hopﬁeld weights.
Revised: In Fig. 4(a), (b), we compare conventional RBM training on four different weight initializations: (i) random W iµ ∼ N (0 , 0 . 01) (purple), commonly used in the literature; (ii) our proposed weights from the projection rule Hopﬁeld mapping for correlated patterns (blue); (iii) the “Hebbian” Hopﬁeld mapping described in previous work for uncorrelated patterns, W = N − 1 / 2 ξ (Barra et al., 2012) (green); and (iv) the top p PCA components of the training data (pink).

Original: In Fig. 4(c), (d) we compare generated sample images from two RBMs, each with p = 50 hidden units but different initial weights.
Revised: In Fig. 4(c), (d) we compare generated sample images from two RBMs, each with p = 50 hidden units but different initial weights (random in (c) and the HN mapping in (d)).

Original: The quality of samples in Fig. 4(d) reﬂect the rapid training of the Hopﬁeld initialized RBM.
Revised: The quality of samples in Fig. 4(d) reﬂect the efﬁcient training of the Hopﬁeld initialized RBM.

Original: [Figure]

Original: Fig. 4(a), (b) shows two key advantages of the Hopﬁeld RBM approach for generative performance.
Revised: Fig. 4(a), (b) show that the Hopﬁeld initialized weights provide an advantage over other approaches during the early stages of training.

Original: First, there is a clear beneﬁt even without training (the conventional RBM takes > 25 epochs to reach a similar level).
Revised: The PCA and Hebbian initializations start at much lower values of the objective and require one or more epochs of training to perform similarly (Fig. 4(a) inset), while the randomly initialized RBM takes > 25 epochs to reach a similar level. All initializations ultimately reach the same value.

Original: This is noteworthy because our weight initialization is fast compared to  RBM training.
Revised: This is noteworthy because the proposed weight initialization is fast compared to conventional RBM training.

Revised: PCA performs best for intermediate training times.

Original: Additionally, the randomly initialized RBM train slowly, taking roughly 40 epochs in Fig. 4(a), and in Fig. 4(b) we increased the basal learning rate η 0 = 10 − 4 by a factor of 5 for the ﬁrst 25 epochs due to slow training.
Revised: [Figure] Despite being a common choice the random initialization trains surprisingly slowly, taking roughly 40 epochs in Fig. 4(a) and in Fig. 4(b) we had to increase the basal learning rate η 0 = 10 −by a factor of 5 for the ﬁrst 25 epochs due to slow training.

Original: The Hopﬁeld initialized RBM, by comparison, appears to train much faster.
Revised: The non-random initializations, by comparison, arrive at the same maximum value much sooner.

Original: This supports the idea that  conventional training  simply ﬁne tunes the orthogonalized
Revised: The relatively small change over training for the Hopﬁeld initialized weights supports the idea that they may be near a local optimum of the objective and that conventional training may simply be mildly tuning them (Fig. 3). That the HN initialization performs well at 0 epochs suggests that the p Hopﬁeld patterns concisely summarize the dataset.

Original: Hopﬁeld weights (Fig. 3).
Revised: This is intuitive, as the projection rule encodes the patterns (and nearby states) as high probability basins in the free energy landscape of Eq. As the data itself is clustered near the patterns, these basins should model the true data distribution well.

Original: Overall, our results suggest that the HN correspondence provides an excellent initialization for generative modelling with binary-gaussian RBMs.
Revised: Overall, our results suggest that the HN correspondence provides a useful initialization for generative modelling with binary-gaussian RBMs, displaying excellent performance with minimal training.

Original: 3. C LASSIFICATION OBJECTIVE
Revised: 3. C LASSIFICATION OBJECTIVE

Original: As with the generative objective, we ﬁnd that the Hopﬁeld initialization provides an advantage for classiﬁcation tasks.
Revised: As with the generative objective, we ﬁnd that the Hopﬁeld initialization provides an advantage for classiﬁcation tasks.

Original: Here we consider the closely related MNIST classiﬁcation problem.
Revised: Here we consider the closely related MNIST classiﬁcation problem.

Original: The goal is to train a model on the MNIST Training dataset which accurately predicts the class of presented images.
Revised: The goal is to train a model on the MNIST Training dataset which accurately predicts the class of presented images.

Original: The key statistic is the number of misclassiﬁed images on the MNIST Testing dataset.
Revised: The key statistic is the number of misclassiﬁed images on the MNIST Testing dataset.

Original: We found relatively poor classiﬁcation results with the single (large) RBM architecture from the preceding Section 3.1.
Revised: We found relatively poor classiﬁcation results with the single (large) RBM architecture from the preceding Section 3.1.

Original: Instead, we use a minimal product-of-experts (PoE) architecture as described in Hinton (2002): the input data is ﬁrst passed to 10 RBMs, one for each class µ .
Revised: Instead, we use a minimal product-of-experts (PoE) architecture as described in Hinton (2002): the input data is ﬁrst passed to 10 RBMs, one for each class µ .

Original: This “layer of
Revised: This “layer of

Original: RBMs” functions as a pre-processing layer which maps the high dimensional sample s to a feature vector f ( s ) ∈ R 10 .
Revised: RBMs” functions as a pre-processing layer which maps the high dimensional sample s to a feature vector f ( s ) ∈ R 10 .

Original: This feature vector is then passed to a logistic regression layer in order to predict the class of s .
Revised: This feature vector is then passed to a logistic regression layer in order to predict the class of s .

Original: The RBM layer and the classiﬁcation layer are trained separately.
Revised: The RBM layer and the classiﬁcation layer are trained separately.

Original: The ﬁrst step is to train the RBM layer to produce useful features for classiﬁcation.
Revised: The ﬁrst step is to train the RBM layer to produce useful features for classiﬁcation.

Original: As in Hinton (2002), each small RBM is trained to model the distribution of samples from a speciﬁc digit class µ .
Revised: As in Hinton (2002), each small RBM is trained to model the distribution of samples from a speciﬁc digit class µ .

Original: We use CD-20 generative training as in Section 3.1, with the caveat that each expert is trained solely on examples from their respective class.
Revised: We use CD-20 generative training as in Section 3.1, with the caveat that each expert is trained solely on examples from their respective class.

Original: Each RBM connects N binary visible units to k gaussian hidden units, and becomes an “expert” at generating samples from one class.
Revised: Each RBM connects N binary visible units to k gaussian hidden units, and becomes an “expert” at generating samples from one class.

Original: To focus on the effect of interlayer weight initialization, we set the layer biases to 0.
Revised: To focus on the effect of interlayer weight initialization, we set the layer biases to 0.

Original: After generative training, each expert should have relatively high probability p ( µ ) θ ( s a ) for sample digits s a of the corresponding class µ , and lower probability for digits from other classes.
Revised: After generative training, each expert should have relatively high probability p ( µ ) θ ( s a ) for sample digits s a of the corresponding class µ , and lower probability for digits from other classes.

Original: This idea is used to deﬁne 10 features, one from each expert, based on the log-probability of a given sample under each expert, ln p ( µ ) θ ( s a ) = − βH (  s a ) − ln Z  .
Revised: This idea is used to deﬁne 10 features, one from each expert, based on the log-probability of a given sample under each expert, ln p ( µ ) θ ( s a ) = − βH ( µ ) ( s a ) − ln Z ( µ ) .

Original: Note that β and ln Z  are constants with respect to the data and thus irrelevant for classiﬁcation.
Revised: Note that β and ln Z ( µ ) are constants with respect to the data and thus irrelevant for classiﬁcation.

Original: For a binary-gaussian RBM, H (  s a ) has the simple form Eq.
Revised: For a binary-gaussian RBM, H ( µ ) ( s a ) has the simple form Eq.

Original: (10), so the features we use are
Revised: (10), so the features we use are

Original: [Equation]
Revised: [Equation]

Original: With the feature map deﬁned, we then train a standard logistic regression classiﬁer (using scikitlearn (Pedregosa et al., 2011)) on these features.
Revised: With the feature map deﬁned, we then train a standard logistic regression classiﬁer (using scikitlearn (Pedregosa et al., 2011)) on these features.

Original: In Fig.
Revised: In Fig.

Original: 5, we report the classiﬁcation error on the
Revised: 5, we report the classiﬁcation error on the

Original: MNIST Testing set of 10,000 images (held out during both generative and classiﬁcation training).
Revised: MNIST Testing set of 10,000 images (held out during both generative and classiﬁcation training).

Original: Note the size p = 10 of the feature vector is independent of the hidden dimension k of each RBM, so the classiﬁer is very efﬁcient.
Revised: Note the size p = 10 of the feature vector is independent of the hidden dimension k of each RBM, so the classiﬁer is very efﬁcient.

Revised: performance for the various weight initializations.

Original: For each digit model (expert), we perform CD20 training according to Eq.
Revised: For each digit model (expert), we perform CD-20 training according to Eq.

Original: (15) (as in Fig. 4) for a ﬁxed number of epochs.
Revised: (15) (as in Fig. 4) for a ﬁxed number of epochs.

Original: A given sample image is mapped to the 10-dimensional feature vector with elements Eq.
Revised: A given sample image is mapped to the 10dimensional feature vector with elements Eq.

Original: These features are used to train a logistic regression classiﬁer, and the average MNIST Test set errors are reported.
Revised: These features are used to train a logistic regression classiﬁer, and the average MNIST Test set errors are reported.

Original: The initializations considered for each expert’s weights W ( µ ) 
Revised: The initializations considered for each expert’s weights W ( µ ) init ∈

Original: init ∈ R N × k are W iµ ∼ N (0 , 0 . 01)
Revised: R N × k are W iµ ∼ N (0 , 0 . 01)

Original: (purple, dash-dot lines), and the orthogonalized Hopﬁeld sub-patterns for digit class µ (blue, dashed lines).
Revised: (purple, dashdot), PCA (pink, dashed), and the orthogonalized Hopﬁeld sub-patterns for digit class µ (blue, solid).

Original: The error bars show the min/max of three runs.
Revised: The error bars show the min/max of three runs.

Original: Despite this relatively simple approach, the PoE initialized using the  Hopﬁeld patterns (“Hopﬁeld PoE”) performs fairly well (Fig. 5, blue curves).
Revised: Despite this relatively simple approach, the PoE initialized using the orthogonalized Hopﬁeld patterns (“Hopﬁeld PoE”) performs fairly well (Fig. 5, blue curves), especially as the number of subpatterns is increased.

Original: We found that generative training beyond  epochs did not  improve performance in general.
Revised: We found that generative training beyond 50 epochs did not signiﬁcantly improve performance for the projection HN or PCA. (in Fig.

Revised: E.1, we train to 100 epochs and also display the aforementioned “Hebbian” initial condition, which performs much worse for classiﬁcation).

Original: Intuitively, increasing the number of hidden nodes k increases classiﬁcation performance independent of weight initialization.
Revised: Intuitively, increasing the number of hidden units k increases classiﬁcation performance independent of weight initialization (with sufﬁcient training).

Original: For k ﬁxed, the Hopﬁeld initialization provides a signiﬁcant beneﬁt to classiﬁcation performance compared to the randomly initialized weights (purple curves).
Revised: For k ﬁxed, the Hopﬁeld initialization provides a signiﬁcant beneﬁt to classiﬁcation performance compared to the randomly initialized weights (purple curves).

Revised: For few sub-patterns (circles k =and squares k = 20 ), the Hopﬁeld initialized models perform best without additional training and until 1 epoch, after which PCA (pink) performs better.

Original: When each RBM has k = 100 hidden features (triangles), the Hopﬁeld  PoE reaches 3 . 1% training error, whereas the randomly initialized PoE reaches 4 . 5% .
Revised: When each RBM has k = 100 hidden features (triangles), the Hopﬁeld and PCA PoE reach 3 . 0% training error, whereas the randomly initialized PoE reaches 4 . 5% .

Original: The performance advantage is noteworthy in two ways. First, the Hopﬁeld  PoE with just k = 10 encoded patterns(blue circles) performs better than or equal to the k = 100 randomly initialized PoE at each epoch despite having an order of magnitude fewer trainable parameters.
Revised: However, the Hopﬁeld PoE performs much better than PCA with minimal training, and maintains its advantage until 10 epochs, after which they are similar. Interestingly, both the Hopﬁeld and PCA initialized PoE with just k = 10 encoded patterns performs better than or equal to the k = 100 randomly initialized PoE at each epoch despite having an order of magnitude fewer trainable parameters.

Original: Secondly, without any generative training (0 epochs), the k = 100 Hopﬁeld PoE performs slightly better ( 4 . 4% ) than the k = 100 randomly initialized PoE with 50 epochs of training.
Revised: Finally, without any generative training (0 epochs), the k = 100 Hopﬁeld PoE performs slightly better ( 4 . 4% ) than the k = 100 randomly initialized PoE with 50 epochs of training.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have presented an explicit, exact mapping from projection rule Hopﬁeld networks to Restricted
Revised: We have presented an explicit, exact mapping from projection rule Hopﬁeld networks to Restricted

Original: Boltzmann Machines with binary visible units and gaussian hidden units.
Revised: Boltzmann Machines with binary visible units and gaussian hidden units.

Original: This provides a generalization of previous results which considered uncorrelated patterns (Barra et al., 2012), or special cases of correlated patterns (Agliari et al., 2013; M´ezard, 2017).
Revised: This provides a generalization of previous results which considered uncorrelated patterns (Barra et al., 2012), or special cases of correlated patterns (Agliari et al., 2013; M´ezard, 2017).

Original: We provide an initial characterization of the reverse map from RBMs to HNs, along with a matrix-factorization approach to construct approximate associated HNs when the exact reverse map is not possible.
Revised: We provide an initial characterization of the reverse map from RBMs to HNs, along with a matrix-factorization approach to construct approximate associated HNs when the exact reverse map is not possible.

Original: Importantly, our HN to
Revised: Importantly, our HN to

Original: RBM mapping can be applied to correlated patterns such as those found in real world datasets.
Revised: RBM mapping can be applied to correlated patterns such as those found in real world datasets.

Original: As a result, we are able to conduct experiments (Section 3) on the MNIST dataset which suggest the mapping provides several advantages.
Revised: As a result, we are able to conduct experiments (Section 3) on the MNIST dataset which suggest the mapping provides several advantages.

Original: The conversion of an HN to an equivalent RBM has practical utility: it trades simplicity of presentation for faster processing.
Revised: The conversion of an HN to an equivalent RBM has practical utility: it trades simplicity of presentation for faster processing.

Original: The weight matrix of the RBM is potentially much smaller than the HN ( Np elements instead of N ( N − 1) / 2 ).
Revised: The weight matrix of the RBM is potentially much smaller than the HN ( Np elements instead of N ( N − 1) / 2 ).

Original: More importantly, proper sampling of stochastic trajectories in HNs requires asynchronous updates of the units, whereas RBM dynamics can be simulated in a parallelizable, layer-wise fashion.
Revised: More importantly, proper sampling of stochastic trajectories in HNs requires asynchronous updates of the units, whereas RBM dynamics can be simulated in a parallelizable, layer-wise fashion.

Original: We also utilized the mapping to efﬁciently estimate the partition function of the Hopﬁeld network (Fig. 2) by summing out the spins after representing it as an RBM. 
Revised: We also utilized the mapping to efﬁciently estimate the partition function of the Hopﬁeld network (Fig. 2) by summing out the spins after representing it as an RBM. This mapping also has another practical utility.

Original: When used as a deterministic RBM weight initialization, the HN correspondence enables rapid training of generative models (Section 3.1, Fig. 4).
Revised: When used as an RBM weight initialization, the HN correspondence enables efﬁcient training of generative models (Section 3.1, Fig. 4).

Original: RBMs initialized with random weights and trained for a moderate amount of time perform worse than RBMs initialized to orthogonalized Hopﬁeld patterns and not trained at all.
Revised: RBMs initialized with random weights and trained for a moderate amount of time perform worse than RBMs initialized to orthogonalized Hopﬁeld patterns and not trained at all.

Original: Further, with mild training of just a few epochs, Hopﬁeld RBMs outperform conventionally initialized RBMs trained several times longer.
Revised: Further, with mild training of just a few epochs, Hopﬁeld RBMs outperform conventionally initialized RBMs trained several times longer.

Revised: The revealed initialization also shows advantages over alternative non-random initializations (PCA and the “Hebbian” Hopﬁeld mapping) during early training.

Original: By leveraging this advantage for generative tasks, wealso show that the correspondence can also be used to improve classiﬁcation performance (Section 3.2, Fig. 5).
Revised: By leveraging this advantage for generative tasks, we show that the correspondence can also be used to improve classiﬁcation performance (Section 3.2, Fig. 5, Appendix E.3).

Original: Overall, the RBM initialization revealed by the mapping allows for smaller models which perform better despite shorter training time.
Revised: Overall, the RBM initialization revealed by the mapping allows for smaller models which perform better despite shorter training time (for instance, using fewer hidden units to achieve similar classiﬁcation performance).

Original: Reducing the size and training time of models is critical, as more realistic datasets (e.g. gene expression data from single-cell RNA sequencing) may require orders of magnitude more visible units.
Revised: Reducing the size and training time of models is critical, as more realistic datasets (e.g. gene expression data from single-cell RNA sequencing) may require orders of magnitude more visible units.

Original: For generative modelling of such high dimensional data, our proposed weight initialization based on orthogonalized Hopﬁeld patterns could make RBMs practicalto use.
Revised: For generative modelling of such high dimensional data, our proposed weight initialization based on orthogonalized Hopﬁeld patterns could be of practical use.

Original: Our  experiments are a proof-of-principle; it is likely that the beneﬁts can be extended to the large family of deep architectures which are built upon RBMs, such as deep belief networks (Hinton et al., 2006) and deep boltzmann machines (Salakhutdinov & Hinton, 2009).
Revised: Our theory and experiments are a proof-of-principle; if they can be extended to the large family of deep architectures which are built upon RBMs, such as deep belief networks (Hinton et al., 2006) and deep Boltzmann machines (Salakhutdinov & Hinton, 2009), it would be of great beneﬁt. This will be explored in future work.

Original: More broadly, exposing the relationship between RBMs and their representative HNs helps to address the infamous interpretability problem of machine learning which criticizes trained models as “black boxes”.
Revised: More broadly, exposing the relationship between RBMs and their representative HNs helps to address the infamous interpretability problem of machine learning which criticizes trained models as “black boxes”.

Original: HNs are relatively transparent models, where the role of the patterns as robust dynamical attractors is theoretically well-understood.
Revised: HNs are relatively transparent models, where the role of the patterns as robust dynamical attractors is theoretically well-understood.

Original: We believe this correspondence, along with future work to further characterize the reverse map, will be especially fruitful for explaining the performance of deep architectures constructed from RBMs.
Revised: We believe this correspondence, along with future work to further characterize the reverse map, will be especially fruitful for explaining the performance of deep architectures constructed from RBMs.

Original: A H OPFIELD NETWORK DETAILS
Revised: A H OPFIELD NETWORK DETAILS

Original: Consider p < N N -dimensional binary patterns { ξ µ } pµ =1 that are to be “stored”.
Revised: Consider p < N N -dimensional binary patterns { ξ µ } pµ =1 that are to be “stored”.

Original: From them, construct the N × p matrix ξ whose columns are the p patterns.
Revised: From them, construct the N × p matrix ξ whose columns are the p patterns.

Original: If they are mutually orthogonal (e g. randomly sampled patterns in the large N → ∞ limit), then choosing interactions according to the
Revised: If they are mutually orthogonal (e.g. randomly sampled patterns in the large N → ∞ limit), then choosing interactions according to the

Original: Hebbian rule, J Hebb =
Revised: Hebbian rule, J Hebb =

Original: 1 N ξξ T , guarantees that they will all be stable minima of H ( s )
Revised: 1 N ξξ T , guarantees that they will all be stable minima of H ( s )

Original: = − 12 s T Js , provided α ≡ p/N < α c , where α c ≈ 0 .
Revised: = − 12 s T Js , provided α ≡ p/N < α c , where α c ≈ 0 .

Original: 14 (Amit et al., 1985).
Revised: 14 (Amit et al., 1985).

Original: If they are not mutually orthogonal (referred to as correlated ), then using the “projection rule” J Proj = ξ ( ξ T ξ ) − 1 ξ T guarantees that they will all be stable minima of H ( s ) , provided p < N (Kanter & Sompolinsky, 1987; Personnaz et al., 1986).
Revised: If they are not mutually orthogonal (referred to as correlated ), then using the “projection rule” J Proj = ξ ( ξ T ξ ) − 1 ξ T guarantees that they will all be stable minima of H ( s ) , provided p < N (Kanter & Sompolinsky, 1987; Personnaz et al., 1986).

Original: Note J Proj
Revised: Note J Proj

Original: → J Hebb in the limit of orthogonal patterns.
Revised: → J Hebb in the limit of orthogonal patterns.

Original: In the main text, we use J as shorthand for J Proj .
Revised: In the main text, we use J as shorthand for J Proj .

Original: We provide some relevant notation from Kanter & Sompolinsky (1987).
Revised: We provide some relevant notation from Kanter & Sompolinsky (1987).

Original: Deﬁne the overlap of a state s with the p patterns as m ( s ) ≡ 1 N ξ T s , and deﬁne the projection of a state s onto the p patterns as a ( s ) ≡ ( ξ T ξ ) −
Revised: Deﬁne the overlap of a state s with the p patterns as m ( s ) ≡ 1 N ξ T s , and deﬁne the projection of a state s onto the p patterns as a ( s ) ≡ ( ξ T ξ ) −

Original: 1 ξ T s ≡ A − 1 m ( s ) .
Revised: 1 ξ T s ≡ A − 1 m ( s ) .

Original: Note A ≡ ξ T ξ is the overlap matrix, and m µ , a µ ∈ [ − 1 , 1] .
Revised: Note A ≡ ξ T ξ is the overlap matrix, and m µ , a µ ∈ [ − 1 , 1] .

Original: We can re-write the projection rule Hamiltonian Eq.
Revised: We can re-write the projection rule Hamiltonian Eq.

Original: (1) as
Revised: (1) as

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we include the self-interactions rather than keeping track of their omission; the results are the same as N → ∞ .
Revised: For simplicity, we include the self-interactions rather than keeping track of their omission; the results are the same as N → ∞ .

Original: From Eq.
Revised: From Eq.

Original: (A.1), several quadratic forms can be written depending on which variables one wants to work with:
Revised: (A.1), several quadratic forms can be written depending on which variables one wants to work with:

Original: I. H ( s ) = − N 2 2 m T ( ξ T ξ ) − 1 m II.
Revised: I. H ( s ) = − N 2 2 m T ( ξ T ξ ) − 1 m II.

Original: H ( s ) = − 12 a T ( ξ T ξ ) a
Revised: H ( s ) = − 12 a T ( ξ T ξ ) a

Original: These are the starting points for the alternative Boltzmann Machines (i.e. not RBMs) presented in
Revised: These are the starting points for the alternative Boltzmann Machines (i.e. not RBMs) presented in

Original: Appendix C.
Revised: Appendix C.

Original: B A DDITIONAL HN TO RBM MAPPINGS
Revised: B A DDITIONAL HN TO RBM MAPPINGS

Original: We used QR factorization in the main text to establish the HN to RBM mapping.
Revised: We used QR factorization in the main text to establish the HN to RBM mapping.

Original: However, one can use any decomposition which satisﬁes
Revised: However, one can use any decomposition which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: such that U ∈ R N × p is orthogonal ( orthogonal for tall matrices means U T U
Revised: such that U ∈ R N × p is orthogonal ( orthogonal for tall matrices means U T U

Original: = I ).
Revised: = I ).

Original: In that case, U becomes the RBM weights.
Revised: In that case, U becomes the RBM weights.

Original: We provide two simple alternatives below, and show they are all part of the same family of orthogonal decompositions.
Revised: We provide two simple alternatives below, and show they are all part of the same family of orthogonal decompositions.

Original: “Square root” decomposition : Deﬁne the matrix K ≡ ξ ( ξ T ξ ) − 1 / 2 .
Revised: “Square root” decomposition : Deﬁne the matrix K ≡ ξ ( ξ T ξ ) − 1 / 2 .

Original: Note that K is orthogonal, and that J Proj = KK T .
Revised: Note that K is orthogonal, and that J Proj = KK T .

Original: Singular value decomposition: More generally, consider the SVD of the pattern matrix ξ :
Revised: Singular value decomposition: More generally, consider the SVD of the pattern matrix ξ :

Original: [Equation]
Revised: [Equation]

Original: where U ∈ R N × p , V ∈ R p × p store the left and right singular vectors (respectively) of ξ as orthogonal columns, and Σ ∈ R p × p is diagonal and contains the singular values of ξ .
Revised: where U ∈ R N × p , V ∈ R p × p store the left and right singular vectors (respectively) of ξ as orthogonal columns, and Σ ∈ R p × p is diagonal and contains the singular values of ξ .

Original: Note in the limit of orthogonal patterns, we have Σ = √ N I .
Revised: Note in the limit of orthogonal patterns, we have Σ = √ N I .

Original: This decomposition gives several relations for quantities of interest:
Revised: This decomposition gives several relations for quantities of interest:

Original: [Equation]
Revised: [Equation]

Original: The last line is simply the diagonalization of J Proj , and shows that our RBM mapping is preserved if we swap the Q from QR with U from SVD.
Revised: The last line is simply the diagonalization of J Proj , and shows that our RBM mapping is preserved if we swap the Q from QR with U from SVD.

Original: However, since there are p degenerate eigenvalues σ 2 = 1 , U is not unique - any orthogonal basis for the 1 -eigenspace can be chosen.
Revised: However, since there are p degenerate eigenvalues σ 2 = 1 , U is not unique - any orthogonal basis for the 1 -eigenspace can be chosen.

Original: Thus U (cid:48)
Revised: Thus U (cid:48)

Original: =
Revised: =

Original: UO where O is orthogonal is also valid, and the QR decomposition and “square root” decomposition correspond to particular choices of O .
Revised: UO where O is orthogonal is also valid, and the QR decomposition and “square root” decomposition correspond to particular choices of O .

Original: C HN TO BM MAPPINGS USING ALTERNATIVE REPRESENTATIONS OF THE H OPFIELD H AMILTONIAN
Revised: C HN TO BM MAPPINGS USING ALTERNATIVE REPRESENTATIONS OF THE H OPFIELD H AMILTONIAN

Original: In addition to the orthogonalized representation in the main text, there are two natural representations to consider based on the pattern overlaps and projections introduced in Appendix A.
Revised: In addition to the orthogonalized representation in the main text, there are two natural representations to consider based on the pattern overlaps and projections introduced in Appendix A.

Original: These lead to generalized Boltzmann Machines (BMs) consisting of N original binary spins and p continuous variables.
Revised: These lead to generalized Boltzmann Machines (BMs) consisting of N original binary spins and p continuous variables.

Original: These representations are not RBMs as the continuous variables interact with each other.
Revised: These representations are not RBMs as the continuous variables interact with each other.

Original: We present them for completeness.
Revised: We present them for completeness.

Original: “Overlap” Boltzmann Machine : Writing H ( s ) = − N 2 2 m T ( ξ T ξ ) − 1 m , we have
Revised: “Overlap” Boltzmann Machine : Writing H ( s ) = − N 2 2 m T ( ξ T ξ ) − 1 m , we have

Original: [Equation]
Revised: [Equation]

Original: Applying the gaussian integral identity,
Revised: Applying the gaussian integral identity,

Original: [Equation]
Revised: [Equation]

Original: from which we identify the BM Hamiltonian
Revised: from which we identify the BM Hamiltonian

Original: [Equation]
Revised: [Equation]

Original: This is the analog of Eq.
Revised: This is the analog of Eq.

Original: (6) in the main text for the “overlap” representation.
Revised: (6) in the main text for the “overlap” representation.

Original: Note we can also sum out the binary variables in Eq.
Revised: Note we can also sum out the binary variables in Eq.

Original: (C.2), which allows for  a saddle point approximation to  the free energy (as in Eq.
Revised: (C.2), which allows for an analogous expression to Eq. [Equation] Curiously, we note that we may perform a second Gaussian integral on Eq. (C.2), introducing new auxiliary variables τ ν to remove the interactions between the λ µ variables:

Original: [Equation]
Revised: [Equation]

Revised: det

Revised: [Equation]

Revised: Eq.

Revised: (C.5) describes a three-layer RBM with complex interactions between the λ µ and τ ν variables, a representation which could be useful in some contexts.

Original: “Projection” Boltzmann Machine : Proceeding as above but for H ( s ) = − 12 a T ( ξ T ξ ) a , one ﬁnds
Revised: “Projection” Boltzmann Machine : Proceeding as above but for H ( s ) = − 12 a T ( ξ T ξ ) a , one ﬁnds

Original: [Equation]
Revised: [Equation]

Original: which corresponds to the BM Hamiltonian
Revised: which corresponds to the BM Hamiltonian

Original: [Equation]
Revised: [Equation]

Original: The saddle point free energy in this case is
Revised: The analogous expression to Eq. (8) in this case is

Original: [Equation]
Revised: [Equation]

Original: D RBM TO HN DETAILS D. I NTEGRATING OUT THE HIDDEN VARIABLES
Revised: D RBM TO HN DETAILS D. I NTEGRATING OUT THE HIDDEN VARIABLES

Original: The explanation from Mehta et al.
Revised: The explanation from Mehta et al.

Original: (2019) for integrating out the hidden variables of an RBM is presented here for completeness.
Revised: (2019) for integrating out the hidden variables of an RBM is presented here for completeness.

Original: For a given binary-gaussian RBM deﬁned by H RBM ( s , λ ) (as in
Revised: For a given binary-gaussian RBM deﬁned by H RBM ( s , λ ) (as in

Original: Eq.
Revised: Eq.

Original: (9)), we have p ( s ) =
Revised: (9)), we have p ( s ) =

Original: Z − 1 (cid:82) e − βH RBM ( s , λ ) d λ .
Revised: Z − 1 (cid:82) e − βH RBM ( s , λ ) d λ .

Original: Consider also that p ( s ) =
Revised: Consider also that p ( s ) =

Original: Z − 1 e − βH ( s ) for some unknown H ( s ) .
Revised: Z − 1 e − βH ( s ) for some unknown H ( s ) .

Original: Equating these expressions gives,
Revised: Equating these expressions gives,

Original: [Equation]
Revised: [Equation]

Original: Decompose the argument of ln( · ) in Eq.
Revised: Decompose the argument of ln( · ) in Eq.

Original: (D.1) by deﬁning q µ ( λ µ ) , a gaussian with zero mean and variance β − 1 .
Revised: (D.1) by deﬁning q µ ( λ µ ) , a gaussian with zero mean and variance β − 1 .

Original: Writing t µ = β (cid:80)
Revised: Writing t µ = β (cid:80)

Original: i W iµ
Revised: i W iµ

Original: s i , one observes that the second term (up to a constant) is a sum of cumulant generating functions,
Revised: s i , one observes that the second term (up to a constant) is a sum of cumulant generating functions,

Original: i.e.
Revised: i.e.

Original: [Equation]
Revised: [Equation]

Original: These can be written as a cumulant expansion, K µ ( t µ ) =
Revised: These can be written as a cumulant expansion, K µ ( t µ ) =

Original: (cid:80) n =1 κ ( n ) µ t nµ n ! , where κ ( n ) µ = ∂K µ ∂t µ | t µ = is the n th cumulant of q µ .
Revised: (cid:80) n =1 κ ( n ) µ t nµ n ! , where κ ( n ) µ [Equation] is the n th cumulant of q µ .

Original: However, since q µ ( λ µ ) is gaussian, only the second term remains, leaving
Revised: However, since q µ ( λ µ ) is gaussian, only the second term remains, leaving

Original: K µ ( t µ ) = 1 β ( β (cid:80) i
Revised: K µ ( t µ [Equation] = [Equation]

Original: W
Revised: [Equation]

Original: iµ s i ) 2.
Revised: i [Equation]

Original: Putting this all together, we have
Revised: Putting this all together, we have

Original: [Equation]
Revised: [Equation]

Original: Note that in general, q µ ( λ µ ) need not be gaussian, in which case the resultant Hamiltonian H ( s ) can have higher order interactions (expressed via the cumulant expansion).
Revised: Note that in general, q µ ( λ µ ) need not be gaussian, in which case the resultant Hamiltonian H ( s ) can have higher order interactions (expressed via the cumulant expansion).

Original: D. A PPROXIMATE REVERSE MAPPING
Revised: D. A PPROXIMATE REVERSE MAPPING

Original: Suppose one has a trial solution B p =
Revised: Suppose one has a trial solution B p =

Original: W X to the approximate binarization problem Eq.
Revised: W X to the approximate binarization problem Eq.

Original: (12), with error matrix E ≡ B p − sgn ( B p ) .
Revised: (12), with error matrix E ≡ B p − sgn ( B p ) .

Original: We consider two cases depending on if W is orthogonal.
Revised: We consider two cases depending on if W is orthogonal.

Original: Case 1: If W is orthogonal, then starting from Eq.
Revised: Case 1: If W is orthogonal, then starting from Eq.

Original: (11) and applying Eq.
Revised: (11) and applying Eq.

Original: (13), we have J =
Revised: (13), we have J =

Original: W W T =
Revised: W W T =

Original: B p ( X T X ) − 1 B p .
Revised: B p ( X T X ) − 1 B Tp .

Original: Using I = W T W , we get
Revised: Using I = W T W , we get

Original: [Equation]
Revised: [Equation]

Revised: (D.4)

Original: Thus, the interactions between the visible units is the familiar projection rule used to store the approximately binary patterns B p .
Revised: Thus, the interactions between the visible units is the familiar projection rule used to store the approximately binary patterns B p .

Original: “Storage” means the patterns are stable ﬁxed points of the deterministic update rule s t +1 ≡ sgn ( Js t ) .
Revised: “Storage” means the patterns are stable ﬁxed points of the deterministic update rule s t +1 ≡ sgn ( Js t ) .

Original: We cannot initialize the network to a non-binary state.
Revised: We cannot initialize the network to a non-binary state.

Original: Therefore, the columns of B = sgn ( B p ) are our candidate patterns.
Revised: Therefore, the columns of B = sgn ( B p ) are our candidate patterns.

Original: To test if they are ﬁxed points, consider
Revised: To test if they are ﬁxed points, consider

Original: [Equation]
Revised: [Equation]

Original:  We need the error E to be such that JE will not alter the sign of B p .
Revised: ( B p − JE ) . We need the error E to be such that JE will not alter the sign of B p .

Original: Two sufﬁcient conditions are:
Revised: Two sufﬁcient conditions are:

Original: (a) small error: | ( JE ) iµ | < | ( B p ) iµ | , or
Revised: (a) small error: | ( JE ) iµ | < | ( B p ) iµ | , or

Original: (b) error with compatible sign: ( JE ) iµ
Revised: (b) error with compatible sign: ( JE ) iµ

Original: ( B p ) iµ < 0 .
Revised: ( B p ) iµ < 0 .

Original: When either of these conditions hold for each element, we have sgn ( JB ) = sgn
Revised: When either of these conditions hold for each element, we have sgn ( JB ) = sgn

Original: ( B p ) =
Revised: ( B p ) =

Original: B , so that the candidate patterns B are ﬁxed points.
Revised: B , so that the candidate patterns B are ﬁxed points.

Original: It remains to show that they are also stable (i.e. minima).
Revised: It remains to show that they are also stable (i.e. minima).

Original: Case 2: If W is not orthogonal but its singular values remain close to one, then L¨owdin Orthogonalization (also known as Symmetric Orthogonalization)
Revised: Case 2: If W is not orthogonal but its singular values remain close to one, then L¨owdin Orthogonalization (also known as Symmetric Orthogonalization)

Original: (L¨owdin, 1970) provides a way to preserve the HN mapping from Case 1 above.
Revised: (L¨owdin, 1970) provides a way to preserve the HN mapping from Case 1 above.

Original: Consider the SVD of W : W = U Σ V T .
Revised: Consider the SVD of W : W = U Σ V T .

Original: The closest matrix to W (in terms of Frobenius norm) with orthogonal columns is L = UV T , and the approximation W ≈ L is called the L¨owdin
Revised: The closest matrix to W (in terms of Frobenius norm) with orthogonal columns is L = UV T , and the approximation W ≈ L is called the L¨owdin

Original: Orthogonalization of W .
Revised: Orthogonalization of W .

Original: Note the approximation becomes exact when all the singular values are one.
Revised: Note the approximation becomes exact when all the singular values are one.

Original: We then write W W T ≈ UU T , and the orthogonal W approach of Case 1 can then be applied.
Revised: We then write W W T ≈ UU T , and the orthogonal W approach of Case 1 can then be applied.

Original: On the other hand, W may be strongly not orthogonal (singular values far from one).
Revised: On the other hand, W may be strongly not orthogonal (singular values far from one).

Original: If it is still full rank, then its pseudo-inverse W † =
Revised: If it is still full rank, then its pseudo-inverse W † =

Original: ( W T W ) − 1 W T =
Revised: ( W T W ) − 1 W T =

Original: V Σ − 1 U T is well-deﬁned.
Revised: V Σ − 1 U T is well-deﬁned.

Original: Repeating the steps from the orthogonal case, we note here X T X
Revised: Repeating the steps from the orthogonal case, we note here X T X

Original: =
Revised: =

Original: B Tp ( W † ) T W † B p .
Revised: B Tp ( W † ) T W † B p .

Original: Deﬁning C ≡ ( W † ) T W † =
Revised: Deﬁning C ≡ ( W † ) T W † =

Original: U Σ − 2 U T , we arrive at the corresponding result,
Revised: U Σ − 2 U T , we arrive at the corresponding result,

Original: [Equation]
Revised: [Equation]

Original: This is analogous to the projection rule but with a “correction factor” C .
Revised: This is analogous to the projection rule but with a “correction factor” C .

Original: However, it is not immediately clear how C affects pattern storage.
Revised: However, it is not immediately clear how C affects pattern storage.

Original: Given the resemblance between J Proj and Eq.
Revised: Given the resemblance between J Proj and Eq.

Original: (D.4)
Revised: (D.4)

Original: (relative to Eq.
Revised: (relative to Eq.

Original: (D.5))
Revised: (D.5))

Original: , we expect that RBMs trained with an orthogonality constraint on the weights may be easier to (approximately) map to HNs.
Revised: , we expect that RBMs trained with an orthogonality constraint on the weights may be easier to (approximately) map to HNs.

Original: E RBM TRAINING
Revised: E RBM TRAINING

Original: Consider a general binary-gaussian RBM with N visible and p hidden units.
Revised: Consider a general binary-gaussian RBM with N visible and p hidden units.

Original: The energy function
Revised: The energy function

Original: [Equation]
Revised: [Equation]

Original: First, we note the Gibbs-block update distributions for sampling one layer of a binary-gaussian RBM given the other (see e.g. Melchior et al.
Revised: First, we note the Gibbs-block update distributions for sampling one layer of a binary-gaussian RBM given the other (see e.g. Melchior et al.

Original: Visible units: p ( s i = 1 | λ ) = 1 1+ e − 2 βxi , where x i ≡ (cid:80) µ W iµ λ µ + b i deﬁnes input to s i ,
Revised: Visible units: p ( s i = 1 | λ ) = 1 1+ e − 2 βxi , where x i ≡ (cid:80) µ W iµ λ µ + b i deﬁnes input to s i ,

Original: Hidden units : p ( λ µ = λ | s ) ∼ N ( h µ , β − 1 ) , where h µ ≡ (cid:80) i
Revised: Hidden units : p ( λ µ = λ | s ) ∼ N ( h µ , β − 1 ) , where h µ ≡ (cid:80) i

Original: W iµ s i
Revised: W iµ s i

Original: + c µ deﬁnes the input to λ µ .
Revised: + c µ deﬁnes the input to λ µ .

Original: E.
Revised: E.

Original: G ENERATIVE TRAINING
Revised: G ENERATIVE TRAINING

Original: For completeness, we re-derive the binary-gaussian RBM weight updates, along the lines of Melchior et al.
Revised: For completeness, we re-derive the binary-gaussian RBM weight updates, along the lines of Melchior et al.

Original: We want to maximize L ≡ 1 M (cid:80) a ln p θ ( s a ) .
Revised: We want to maximize L ≡ 1 M (cid:80) a ln p θ ( s a ) .

Original: The contribution for a single datapoint s a has the form ln p θ ( s a )
Revised: The contribution for a single datapoint s a has the form ln p θ ( s a )

Original: =
Revised: =

Original: ln( C − 1 (cid:82) e − βH
Revised: ln( C − 1 (cid:82) e − βH

Original: ( s a , λ ) d λ ) − ln Z with C ≡ (2 π/β ) p/ 2 .
Revised: ( s a , λ ) d λ ) − ln Z with C ≡ (2 π/β ) p/ 2 .

Original: The gradient with respect to the model is
Revised: The gradient with respect to the model is

Original: [Equation]
Revised: [Equation]

Original: We are focused on the interlayer weights, with ∂H ( s , λ )
Revised: We are focused on the interlayer weights, with ∂H ( s , λ )

Original: [Equation]
Revised: [Equation]

Original: i λ µ , so 
Revised: i λ µ , so ∂ ln p

Original: [Equation]
Revised: [Equation]

Original:  The ﬁrst term is straightforward to compute: (cid:104) λ µ | s = s a (cid:105) model =
Revised: = [Equation] The ﬁrst term is straightforward to compute: (cid:104) λ µ | s = s a (cid:105) model =

Original: (cid:80) i W iµ ( s a )
Revised: (cid:80) i W iµ ( s a )

Original: i + c µ .
Revised: i + c µ .

Original: The second term is intractable and needs to be approximated.
Revised: The second term is intractable and needs to be approximated.

Original: We use contrastive divergence (Hinton, 2012): (cid:104)
Revised: We use contrastive divergence (Carreira-Perpinan & Hinton, 2005; Hinton, 2012): (cid:104)

Original: s i λ µ (cid:105) model
Revised: s i λ µ (cid:105) model

Original: ≈ s ( k ) i λ ( k ) µ .
Revised: ≈ s ( k ) i λ ( k ) µ .

Original: Here k denotes CD- k – k steps of Gibbs-block updates (introduced above) – from which s ( k ) i , λ ( k ) µ comprise the ﬁnal state.
Revised: Here k denotes CD- k – k steps of Gibbsblock updates (introduced above) – from which s ( k ) i , λ ( k ) µ comprise the ﬁnal state.

Original: We evaluate both terms over mini-batches of the training data to arrive at the weight update rule Eq.
Revised: We evaluate both terms over mini-batches of the training data to arrive at the weight update rule Eq.

Original: E. G ENERATIVE PERFORMANCE
Revised: E. G ENERATIVE PERFORMANCE

Original: We are interested in estimating the objective function L ≡ 1 M (cid:80) a ln p θ ( s a ) during training.
Revised: We are interested in estimating the objective function L ≡ 1 M (cid:80) a ln p θ ( s a ) during training.

Original: As above, we split L into two terms,
Revised: As above, we split L into two terms,

Original: [Equation]
Revised: [Equation]

Original: with C ≡ (2 π/β ) p/ 2 .
Revised: with C ≡ (2 π/β ) p/ 2 .

Original: The ﬁrst term evaluates to
Revised: The ﬁrst term evaluates to

Original: [Equation]
Revised: [Equation]

Original: which can be computed deterministically.
Revised: which can be computed deterministically.

Original: ln Z , on the other hand, needs to be estimated.
Revised: ln Z , on the other hand, needs to be estimated.

Original: For this we perform Annealed Importance Sampling (AIS)
Revised: For this we perform Annealed Importance Sampling (AIS)

Original: (Neal, 2001) on its continuous representation
Revised: (Neal, 2001) on its continuous representation

Original: [Equation]
Revised: [Equation]

Original: For AIS we need to specify the target distribution’s un-normalized log-probabilities
Revised: For AIS we need to specify the target distribution’s un-normalized log-probabilities

Original: [Equation]
Revised: [Equation]

Original: as well as an initial proposal distribution to anneal from, which we ﬁx as a p -dimensional unit normal distribution N ( 0 , I ) .
Revised: as well as an initial proposal distribution to anneal from, which we ﬁx as a p -dimensional unit normal distribution N ( 0 , I ) .

Revised: E. C LASSIFICATION PERFORMANCE

Revised: Here we provide extended data (Fig.

Revised: E.1) on the classiﬁcation performance shown in Fig.

Revised: As in the main text, color denotes initial condition (introduced in Section 3.1) and shape denotes the number of sub-patterns.

Revised: Here we train for each 100 epochs, which allows convergence of most of the curves.

Revised: We also include the “Hebbian” initialization (green curves).

Revised: Notably, the Hebbian initialization performs quite poorly in the classiﬁcation task (as compared to direct generative objective, Fig. 4).

Revised: In particular, for the 100 sub-patterns case, where the projection

Revised: HN performs best, the Hebbian curve trains very slowly (still not converged after 100 epochs) and lags behind even the 10 sub-pattern Hebbian curve for most of the training.

Revised: This emphasizes the beneﬁts of the projection rule HN over the Hebbian HN when the data is composed of correlated patterns, which applies to most real-world datasets.