Original: T HE K ANERVA M ACHINE : A G ENERATIVE D ISTRIBUTED M EMORY
Revised: T HE K ANERVA M ACHINE : A G ENERATIVE D ISTRIBUTED M EMORY

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them.
Revised: We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them.

Original: Inspired by Kanerva’s sparse distributed memory, it has a robust distributed reading and writing mechanism.
Revised: Inspired by Kanerva’s sparse distributed memory, it has a robust distributed reading and writing mechanism.

Original: The memory is analytically tractable, which enables optimal on-line compression via a Bayesian update-rule.
Revised: The memory is analytically tractable, which enables optimal on-line compression via a Bayesian update-rule.

Original: We formulate it as a hierarchical conditional generative model, where memory provides a rich data-dependent prior distribution.
Revised: We formulate it as a hierarchical conditional generative model, where memory provides a rich data-dependent prior distribution.

Original: Consequently, the top-down memory and bottom-up perception are combined to produce the code representing an observation.
Revised: Consequently, the top-down memory and bottom-up perception are combined to produce the code representing an observation.

Original: Empirically, we demonstrate that the adaptive memory signiﬁcantly improves generative models trained on both the Omniglot and CIFAR datasets.
Revised: Empirically, we demonstrate that the adaptive memory signiﬁcantly improves generative models trained on both the Omniglot and CIFAR datasets.

Original: Compared with the Differentiable Neural Computer (DNC) and its variants, our memory model has greater capacity and is signiﬁcantly easier to train.
Revised: Compared with the Differentiable Neural Computer (DNC) and its variants, our memory model has greater capacity and is signiﬁcantly easier to train

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent work in machine learning has examined a variety of novel ways to augment neural networks with fast memory stores.
Revised: Recent work in machine learning has examined a variety of novel ways to augment neural networks with fast memory stores.

Original: However, the basic problem of how to most efﬁciently use memory remains an open question.
Revised: However, the basic problem of how to most efﬁciently use memory remains an open question.

Original: For instance, the slot-based external memory in models like Differentiable Neural
Revised: For instance, the slot-based external memory in models like Differentiable Neural

Original: Computers (DNCs Graves et al.
Revised: Computers (DNCs Graves et al.

Original: (2016)) often collapses reading and writing into single slots, even though the neural network controller can in principle learn more distributed strategies.
Revised: (2016)) often collapses reading and writing into single slots, even though the neural network controller can in principle learn more distributed strategies.

Original: As as result, information is not shared across memory slots, and additional slots have to be recruited for new inputs, even if they are redundant with existing memories.
Revised: As as result, information is not shared across memory slots, and additional slots have to be recruited for new inputs, even if they are redundant with existing memories.

Original: Similarly, Matching Networks (Vinyals et al., 2016; Bartunov & Vetrov, 2016) and the Neural Episodic Controller (Pritzel et al., 2017) directly store embeddings of data.
Revised: Similarly, Matching Networks (Vinyals et al., 2016; Bartunov & Vetrov, 2016) and the Neural Episodic Controller (Pritzel et al., 2017) directly store embeddings of data.

Original: They therefore require the volume of memory to increase with the number of samples stored.
Revised: They therefore require the volume of memory to increase with the number of samples stored.

Original: In contrast, the Neural Statistician (Edwards & Storkey, 2016) summarises a dataset by averaging over their embeddings.
Revised: In contrast, the Neural Statistician (Edwards & Storkey, 2016) summarises a dataset by averaging over their embeddings.

Original: The resulting “statistics” are conveniently small, but a large amount of information may be dropped by the averaging process, which is at odds with the desire to have large memories that can capture details of past experience.
Revised: The resulting “statistics” are conveniently small, but a large amount of information may be dropped by the averaging process, which is at odds with the desire to have large memories that can capture details of past experience.

Original: Historically developed associative memory architectures provide insight into how to design efﬁcient memory structures that store data in overlapping representations.
Revised: Historically developed associative memory architectures provide insight into how to design efﬁcient memory structures that store data in overlapping representations.

Original: For example, the Hopﬁeld Net (Hopﬁeld, 1982) pioneered the idea of storing patterns in low-energy states in a dynamic system.
Revised: For example, the Hopﬁeld Net (Hopﬁeld, 1982) pioneered the idea of storing patterns in low-energy states in a dynamic system.

Original: This type of model is robust, but its capacity is limited by the number of recurrent connections, which is in turn constrained by the dimensionality of the input patterns.
Revised: This type of model is robust, but its capacity is limited by the number of recurrent connections, which is in turn constrained by the dimensionality of the input patterns.

Original: The Boltzmann Machine (Ackley et al., 1985) lifts this constraint by introducing latent variables, but at the cost of requiring slow reading and writing mechanisms (i.e. Gibbs sampling).
Revised: The Boltzmann Machine (Ackley et al., 1985) lifts this constraint by introducing latent variables, but at the cost of requiring slow reading and writing mechanisms (i.e. Gibbs sampling).

Original: This issue is resolved by Kanerva’s sparse distributed memory model (Kanerva, 1988), which affords fast reads and writes and dissociates capacity from the dimensionality of input by introducing addressing into a distributed memory store whose size is independent of the dimension of the data 1 .
Revised: This issue is resolved by Kanerva’s sparse distributed memory model (Kanerva, 1988), which affords fast reads and writes and dissociates capacity from the dimensionality of input by introducing addressing into a distributed memory store whose size is independent of the dimension of the data 1 .

Original: In this paper, we present a conditional generative memory model inspired by Kanerva’s sparse distributed memory.
Revised: In this paper, we present a conditional generative memory model inspired by Kanerva’s sparse distributed memory.

Original: We generalise Kanerva’s original model through learnable addresses and reparametrised latent variables (Rezende et al., 2014; Kingma & Welling, 2013; Bornschein et al., 2017).
Revised: We generalise Kanerva’s original model through learnable addresses and reparametrised latent variables (Rezende et al., 2014; Kingma & Welling, 2013; Bornschein et al., 2017).

Original: We solve the challenging problem of learning an effective memory writing operation by exploiting the analytically tractability of our memory model — we derive a Bayesian memory update rule that optimally trades-off preserving old content and storing new content.
Revised: We solve the challenging problem of learning an effective memory writing operation by exploiting the analytic tractability of our memory model — we derive a Bayesian memory update rule that optimally trades-off preserving old content and storing new content.

Original: The resulting hierarchical generative model has a memory dependent prior that quickly adapts to new data, providing top-down knowledge in addition to bottom-up perception from the encoder to form the latent code representing data.
Revised: The resulting hierarchical generative model has a memory dependent prior that quickly adapts to new data, providing top-down knowledge in addition to bottom-up perception from the encoder to form the latent code representing data.

Original: As a generative model, our proposal provides a novel way of enriching the often over-simpliﬁed priors in VAE-like models (Rezende et al., 2016) through a adaptive memory.
Revised: As a generative model, our proposal provides a novel way of enriching the often over-simpliﬁed priors in VAE-like models (Rezende et al., 2016) through a adaptive memory.

Original: As a memory system, our proposal offers an effective way to learn online distributed writing which provides effective compression and storage of complex data.
Revised: As a memory system, our proposal offers an effective way to learn online distributed writing which provides effective compression and storage of complex data.

Original:  B ACKGROUND : V ARIATIONAL A UTOENCODERS
Revised:  B ACKGROUND : V ARIATIONAL A UTOENCODERS

Original: Our memory architecture can be viewed as an extension the variational autoencoder (VAE)
Revised: Our memory architecture can be viewed as an extension the variational autoencoder (VAE)

Original: (Rezende et al., 2014; Kingma & Welling, 2013), where the prior is derived from an adaptive memory store.
Revised: (Rezende et al., 2014; Kingma & Welling, 2013), where the prior is derived from an adaptive memory store.

Original: A VAE has an observable variable x and a latent variable z .
Revised: A VAE has an observable variable x and a latent variable z .

Original: Its generative model is speciﬁed by a prior distribution p θ ( z ) and the conditional distribution p θ ( x | z ) .
Revised: Its generative model is speciﬁed by a prior distribution p θ ( z ) and the conditional distribution p θ ( x | z ) .

Original: The intractable posterior p θ ( z | x ) is approximated by a parameterised inference model q φ ( z | x ) .
Revised: The intractable posterior p θ ( z | x ) is approximated by a parameterised inference model q φ ( z | x ) .

Original: Throughout this paper, we use θ to represent the generative model’s parameters, and φ to represent the inference model’s parameters.
Revised: Throughout this paper, we use θ to represent the generative model’s parameters, and φ to represent the inference model’s parameters.

Original: All parameterised distributions are implemented as multivariate Gaussian distributions with diagonal covariance matrices, whose means and variances are outputs from neural networks as in (Rezende et al., 2014; Kingma & Welling, 2013).
Revised: All parameterised distributions are implemented as multivariate Gaussian distributions with diagonal covariance matrices, whose means and variances are outputs from neural networks as in (Rezende et al., 2014; Kingma & Welling, 2013).

Original: We assume a dataset with independently and identically distributed (iid) samples D
Revised: We assume a dataset with independently and identically distributed (iid) samples D

Original: = { x 1 , . . .
Revised: = { x 1 , . . .

Original: , x n , . . .
Revised: , x n , . . .

Original: , x N } .
Revised: , x N } .

Original: The objective of training a VAE is to maximise its log-likelihood E x
Revised: The objective of training a VAE is to maximise its log-likelihood E x

Original: ∼D
Revised: ∼D

Original: [ln p θ ( x )] .
Revised: [ln p θ ( x )] .

Original: This can be achieved by jointly optimising θ and φ for a variational lower-bound of the likelihood (omitting the expectation over all x for simplicity):
Revised: This can be achieved by jointly optimising θ and φ for a variational lower-bound of the likelihood (omitting the expectation over all x for simplicity):

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term can be interpreted as the negative reconstruction loss for reconstructing x using its approximated posterior sample from q φ ( z | x ) , and the second term as a regulariser that encourages the approximated posterior to be near the prior of z .
Revised: where the ﬁrst term can be interpreted as the negative reconstruction loss for reconstructing x using its approximated posterior sample from q φ ( z | x ) , and the second term as a regulariser that encourages the approximated posterior to be near the prior of z .

Original:  T HE K ANERVA M ACHINE
Revised:  T HE K ANERVA M ACHINE

Original: To introduce our model, we use the concept of an exchangeable episode X
Revised: To introduce our model, we use the concept of an exchangeable episode: X

Original: = { x 1 , . . .
Revised: [Equation] { x 1 , . . .

Original: , x t , . . .
Revised: , x t , . . .

Original: , x T } ⊂ D as a subset of the entire dataset, whose order does not matter.
Revised: , x T } ⊂ D is a subset of the entire dataset whose order does not matter.

Original: The objective of traininga Kanerva Machine is the expected conditional log-likelihood (Bornschein et al., 2017),
Revised: The objective of training is the expected conditional log-likelihood (Bornschein et al., 2017),

Original: [Equation]
Revised: [Equation]

Original: The equality utilises the conditional independence of x t given the memory M , which is equivalent to the assumption of an exchangeable episode X (Aldous, 1985).
Revised: The equality utilises the conditional independence of x t given the memory M , which is equivalent to the assumption of an exchangeable episode X (Aldous, 1985).

Original: We factorise the joint distribution of p ( X, M ) into the marginal distribution p ( X ) and the posterior p ( M | X ) , so that computing p ( M | X ) can be naturally interpreted as writing X into the memory.
Revised: We factorise the joint distribution of p ( X, M ) into the marginal distribution p ( X ) and the posterior p ( M | X ) , so that computing p ( M | X ) can be naturally interpreted as writing X into the memory.

Original: We propose this scenario as a general and principled way of formulating memory-based generative models, since J is directly related to the mutual information
Revised: We propose this scenario as a general and principled way of formulating memory-based generative models, since J is directly related to the mutual information

Original: I ( X ; M ) through I ( X ; M ) =
Revised: I ( X ; M ) through I ( X ; M ) =

Original: H ( X ) −
Revised: H ( X ) −

Original: H ( X | M ) =
Revised: H ( X | M ) =

Original: H ( X ) + (cid:82) p ( X, M )
Revised: H ( X ) + (cid:82) p ( X, M )

Original: ln p θ ( X | M ) d X d M = H ( X ) + J .
Revised: ln p θ ( X | M ) d X d M = H ( X ) + J .

Original: As the entropy of the data
Revised: As the entropy of the data

Original: H ( X ) is a constant, maximising J is equivalent to maximising I ( X ; M ) , the mutual information between the memory and the episode to store.
Revised: H ( X ) is a constant, maximising J is equivalent to maximising I ( X ; M ) , the mutual information between the memory and the episode to store.

Original: 3. T HE GENERATIVE MODEL
Revised: 3. T HE GENERATIVE MODEL

Original: We write the collection of latent variables corresponding to the observed episode X as Y =
Revised: We write the collection of latent variables corresponding to the observed episode X as Y =

Original: { y 1 , . . .
Revised: { y 1 , . . .

Original: , y t , . . .
Revised: , y t , . . .

Original: , y T } and Z = { z 1 , . . .
Revised: , y T } and Z = { z 1 , . . .

Original: , z t , . . .
Revised: , z t , . . .

Original: , z T } .
Revised: , z T } .

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 1 (left), the joint dis- tribution of the generative model can be factorised as
Revised: 1 (left), the joint dis- tribution of the generative model can be factorised as

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst equality uses the conditional independence of z t , y t , x t given M , shown by the "plates" in
Revised: The ﬁrst equality uses the conditional independence of z t , y t , x t given M , shown by the "plates" in

Original: Fig.
Revised: Fig.

Original: 1 (left).
Revised: 1 (left).

Original: The memory M is a K × C random matrix with the matrix variate Gaussian distribution (Gupta & Nagar, 1999):
Revised: The memory M is a K × C random matrix with the matrix variate Gaussian distribution (Gupta & Nagar, 1999):

Original: [Equation]
Revised: [Equation]

Original: x t z t y t
Revised: x t z t y t

Original: M
Revised: M

Original: T N x t z t y t
Revised: T N x t z t y t

Original: M
Revised: M

Original: T N x t z t y t
Revised: T N x t z t y t

Original: M
Revised: M

Original: T N where R is a K × C matrix as the mean of M , U is a K × K matrix that provides the covariance between rows of M , and V is a C × C matrix providing covariances between columns of M .
Revised: T N where R is a K × C matrix as the mean of M , U is a K × K matrix that provides the covariance between rows of M , and V is a C × C matrix providing covariances between columns of M .

Original: This distribution is equivalent to the multivariate Gaussian distribution of vectorised M : p (vec ( M ))
Revised: This distribution is equivalent to the multivariate Gaussian distribution of vectorised M : p (vec ( M ))

Original: =
Revised: =

Original: N (vec ( M ) | vec ( R ) ,U ⊗ V  ) , where vec ( · ) is the vectorisation operator and ⊗ denotes the Kronecker product.
Revised: N (vec ( M ) | vec ( R ) , V ⊗ U ) , where vec ( · ) is the vectorisation operator and ⊗ denotes the Kronecker product.

Original: We assume independence between the columns but not the rows of
Revised: We assume independence between the columns but not the rows of

Original: M , by ﬁxing V to be the identity matrix I C and allow the full degree of freedom for U .
Revised: M , by ﬁxing V to be the identity matrix I C and allow the full degree of freedom for U .

Original: Since our experiments suggest the covariance between rows is useful for coordinating memory access, this setting balances simplicity and performance (Fig. 10).
Revised: Since our experiments suggest the covariance between rows is useful for coordinating memory access, this setting balances simplicity and performance (Fig. 10).

Original: Accompany M is the address A , a K × S real-value matrix that is randomly initialised and is optimised through back-propagation.
Revised: Accompanying M are the addresses A , a K × S real-value matrix that is randomly initialised and is optimised through back-propagation.

Original: To avoid degeneracy, rows of A are normalised to have L2norms of 1 .
Revised: To avoid degeneracy, rows of A are normalised to have L2-norms of 1 .

Original: The addressing variable y t is used to compute the weights controlling memory access.
Revised: The addressing variable y t is used to compute the weights controlling memory access.

Original: As in VAEs, the prior p θ ( y t ) is an isotropic Gaussian distribution N ( 0 , 1 ) .
Revised: As in VAEs, the prior p θ ( y t ) is an isotropic Gaussian distribution N ( 0 , 1 ) .

Original: A learned projection b t = f
Revised: A learned projection b t = f

Original: ( y t ) then transforms y t into a S × 1 key vector.
Revised: ( y t ) then transforms y t into a S × 1 key vector.

Original: The K × 1 vector w t , as weights across the rows of M , is computed via the product:
Revised: The K × 1 vector w t , as weights across the rows of M , is computed via the product:

Original: [Equation]
Revised: [Equation]

Original: The projection f is implemented as an multi-layer perception (MLP), which transforms the distribution of y t , as well as w t , to potentially non-Gaussian distributions that may better suit addressing.
Revised: The projection f is implemented as a multi-layer perception (MLP), which transforms the distribution of y t , as well as w t , to potentially non-Gaussian distributions that may better suit addressing.

Original: The code z t is a learned representation that generates samples of x t through the parametrised conditional distribution p θ ( x t | z t ) .
Revised: The code z t is a learned representation that generates samples of x t through the parametrised conditional distribution p θ ( x t | z t ) .

Original: This distribution is tied for all t ∈ { 1 . . .
Revised: This distribution is tied for all t ∈ { 1 . . .

Original: T } .
Revised: T } .

Original: Importantly, instead of the isotropic Gaussian prior, z t has a memory dependent prior:
Revised: Importantly, instead of the isotropic Gaussian prior, z t has a memory dependent prior:

Original: [Equation]
Revised: [Equation]

Original: whose mean is a linear combination of memory rows, with a ﬁxed  identity covariance matrix.
Revised: whose mean is a linear combination of memory rows, with the noise covariance matrix is ﬁxed as an identity matrix by setting σ 2 = 1 , although it can be tuned to other values.

Original: This prior results in a much richer marginal distribution, because of its dependence on memory and the addressing variable y t through p θ ( z t | M ) =
Revised: This prior results in a much richer marginal distribution, because of its dependence on memory and the addressing variable y t through p θ ( z t | M ) =

Original: (cid:82) p θ ( z t | y t , M ) p θ ( y t )
Revised: (cid:82) p θ ( z t | y t , M ) p θ ( y t )

Original: d y t .
Revised: d y t .

Original: In our hierarchical model, M is a global latent variable for an episode that captures statistics of the entire episode (Bartunov & Vetrov, 2016; Edwards & Storkey, 2016), while the local latent variables y t and z t capture local statistics for data x t within an episode.
Revised: In our hierarchical model, M is a global latent variable for an episode that captures statistics of the entire episode (Bartunov & Vetrov, 2016; Edwards & Storkey, 2016), while the local latent variables y t and z t capture local statistics for data x t within an episode.

Original: To generate an episode of length T , we ﬁrst sample M once, then sample y t , z t , and x t sequentially for each of the T samples.
Revised: To generate an episode of length T , we ﬁrst sample M once, then sample y t , z t , and x t sequentially for each of the T samples.

Original: 3. T HE READING INFERENCE MODEL
Revised: 3. T HE READING INFERENCE MODEL

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 1 (central), the approximated posterior distribution is factorised using the conditional independence again as
Revised: 1 (central), the approximated posterior distribution is factorised using the conditional independence again as

Original: [Equation]
Revised: [Equation]

Original: q φ ( y t | x t ) is a parameterised approximate posterior distribution.
Revised: q φ ( y t | x t ) is a parameterised approximate posterior distribution.

Original: The posterior distribution q φ ( z t | x t , y t , M ) reﬁnes the (conditional) prior distribution p θ ( z t | y t , M ) with additional evidence fromx t.
Revised: The posterior distribution q φ ( z t | x t , y t , M ) reﬁnes the (conditional) prior distribution p θ ( z t | y t , M ) with additional evidence from t

Original: This parameterisedposterior takes the concatenation of x t and the meanof pθ ( z t | y t, M ) (eq. 6) as input.
Revised: This parameterised takes the concatenation of x t and the mean p z t | y t

Original: The constant variance of p θ ( z t | y t , M) is omitted.
Revised: M

Original: Similar to the generative model, q φ ( y t | x t ) is shared for all t ∈ { 1 . . . T } .
Revised: ( z M model

Original: 3. T HE WRITING INFERENCE MODEL
Revised: 3. T HE WRITING INFERENCE MODEL

Original: A central difﬁculty in updating memory is the trade-off between preserving old information and writing new information.
Revised: A central difﬁculty in updating memory is the trade-off between preserving old information and writing new information.

Original: It is well known that this trade-off can be balanced optimally through Bayes’ rule MacKay (2003).
Revised: It is well known that this trade-off can be balanced optimally through Bayes’ rule MacKay (2003).

Original: From the generative model perspective (eq. 2), it is natural to interpret memory writing as inference — computing the posterior distribution of memory p ( M | X ) .
Revised: From the generative model perspective (eq. 2), it is natural to interpret memory writing as inference — computing the posterior distribution of memory p ( M | X ) .

Original: This section considers both batch inference — directly computing p ( M | X ) and on-line inference — sequentially accumulating evidence from x 1 , . . .
Revised: This section considers both batch inference — directly computing p ( M | X ) and on-line inference — sequentially accumulating evidence from x 1 , . . .

Original: , x T .
Revised: , x T .

Original: Following Fig.
Revised: Following Fig.

Original: 1 (right), the approximated posterior distribution of memory can be written as
Revised: 1 (right), the approximated posterior distribution of memory can be written as

Original: [Equation]
Revised: [Equation]

Original: The last line uses one sample of y t , x t to approximate the intractable integral.
Revised: The last line uses one sample of y t , x t to approximate the intractable integral.

Original: The posterior of the addressing variable q φ ( y t | x t ) is the same as in section 3.2, and the posterior of code q φ ( z t | x t ) is a parameterised distribution.
Revised: The posterior of the addressing variable q φ ( y t | x t ) is the same as in section 3.2, and the posterior of code q φ ( z t | x t ) is a parameterised distribution.

Original: We use the short-hand p θ ( M | Y, Z ) for p θ ( M |{ y 1 , . . .
Revised: We use the short-hand p θ ( M | Y, Z ) for p θ ( M |{ y 1 , . . .

Original: , y T } , { z 1 , . . .
Revised: , y T } , { z 1 , . . .

Original: , z T } ) when Y, Z are sampled as described here.
Revised: , z T } ) when Y, Z are sampled as described here.

Original: We abuse notations in this section and use Z =
Revised: We abuse notation in this section and use Z =

Original: ( z 1 (cid:124) ; , . . .
Revised: ( z 1 (cid:124) ; , . . .

Original: , ; z T (cid:124) ) as a T × C matrix with all the observations in an episode, and W = ( w 1 (cid:124) ; . . .
Revised: , ; z T (cid:124) ) as a T × C matrix with all the observations in an episode, and W = ( w 1 (cid:124) ; . . .

Original: ; w T (cid:124) ) as a T × K matrix with all corresponding weights for addressing.
Revised: ; w T (cid:124) ) as a T × K matrix with all corresponding weights for addressing.

Original: Given the linear Gaussian model (eq. 6), the posterior of memory p θ ( M | Y, Z ) is analytically tractable, whose parameters R and U can be updated as follows:
Revised: Given the linear Gaussian model (eq. 6), the posterior of memory p θ ( M | Y, Z ) is analytically tractable, whose parameters R and U can be updated as follows:

Original: [Equation]
Revised: [Equation]

Original: where ∆ is the prediction error before updating the memory, Σ c is a T × C matrix providing the cross-covariance between Z and M ,  and Σ z is a T × T matrix that encodes the covariance for z 1 , . . .
Revised: where ∆ is the prediction error before updating the memory, Σ c is a T × C matrix providing the cross-covariance between Z and M , Σ ξ is a T × T diagonal matrix whose diagonal elements are the noise variance σ 2 and Σ z is a T × T matrix that encodes the covariance for z 1 , . . .

Original: , z T .
Revised: , z T .

Original: This update rule is derived from applying Bayes’ rule to the linear Gaussian model (Appendix E).
Revised: This update rule is derived from applying Bayes’ rule to the linear Gaussian model (Appendix E).

Original: The prior parameters of p ( M ) , R 0 and U 0 are trained through back-propagation.
Revised: The prior parameters of p ( M ) , R 0 and U 0 are trained through back-propagation.

Original: Therefore, the prior of M can learn the general structure of the entire dataset, while the posterior is left to adapt to features presented in a subset of data observed within a given episode.
Revised: Therefore, the prior of M can learn the general structure of the entire dataset, while the posterior is left to adapt to features presented in a subset of data observed within a given episode.

Original: The main cost of the update rule comes from inverting Σ z , which has a complexity of O ( T 3 ) .
Revised: The main cost of the update rule comes from inverting Σ z , which has a complexity of O ( T 3 ) .

Original: One may reduce the per-step cost with on-line update, by performing the update rule using one sample at a time — when X = x t , Σ z is a scalar which can be inverted trivially.
Revised: One may reduce the per-step cost via on-line updating, by performing the update rule using one sample at a time — when X = x t , Σ z is a scalar which can be inverted trivially.

Original: According to Bayes’ rule, updating using the entire episode at once is equivalent to performing the one-sample/on-line update iteratively for all observations in the episode.
Revised: According to Bayes’ rule, updating using the entire episode at once is equivalent to performing the one-sample/on-line update iteratively for all observations in the episode.

Original: Similarly, one can perform intermediate updates using mini-batch with size between 1 and T .
Revised: Similarly, one can perform intermediate updates using mini-batch with size between 1 and T .

Original: Another major cost in the update rule is the storage and multiplication of the memory’s row-covariance matrix U , with the complexity of O ( K 2 ) .
Revised: Another major cost in the update rule is the storage and multiplication of the memory’s row-covariance matrix U , with the complexity of O ( K 2 ) .

Original: Although restricting this covariance to diagonal can reduce this cost to O ( K ) , our experiments suggested this covariance is useful for coordinating memory accessing (Fig. 10).
Revised: Although restricting this covariance to diagonal can reduce this cost to O ( K ) , our experiments suggested this covariance is useful for coordinating memory accessing (Fig. 10).

Original: Moreover, the cost of O ( K 2 ) is usually small, since parameters of the model are dominated by the encoder and decoder.
Revised: Moreover, the cost of O ( K 2 ) is usually small, since parameters of the model are dominated by the encoder and decoder.

Original: Nevertheless, a future direction is to investigating low-rank approximation of U that better balance cost and performance.
Revised: Nevertheless, a future direction is to investigating low-rank approximation of U that better balance cost and performance.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: To train this model, we optimise a variational lower-bound of the conditional likelihood J (eq. 2), which can be derived in a fashion similar to standard VAEs:
Revised: To train this model, we optimise a variational lower-bound of the conditional likelihood J (eq. 2), which can be derived in a fashion similar to standard VAEs:

Original: [Equation]
Revised: [Equation]

Original: To maximise this lower bound, we sample y t , z t from q φ ( y t , z t | x t , M ) to approximate the inner expectation.
Revised: To maximise this lower bound, we sample y t , z t from q φ ( y t , z t | x t , M ) to approximate the inner expectation.

Original: For computational efﬁciency, we use a mean-ﬁeld approximation for the memory — using the mean R in the place of memory samples (since directly sampling M requires expensive
Revised: For computational efﬁciency, we use a mean-ﬁeld approximation for the memory — using the mean R in the place of memory samples (since directly sampling M requires expensive

Original: Cholesky decomposition of the non-diagonal matrix U ).
Revised: Cholesky decomposition of the non-diagonal matrix U ).

Revised: Alternatively, we can further exploit the analytical tractability of the Gaussian distribution to obtain distribution-based reading and writing operations (Appendix F).

Original: Inside the bracket, the ﬁrst term is the usual reconstruction error as in VAEs.
Revised: Inside the bracket, the ﬁrst term is the usual reconstruction error as in VAEs.

Original: The ﬁrst KL-divergence penalises complex addresses, and the second term penalises deviation of the code z t from the memory-based prior.
Revised: The ﬁrst KL-divergence penalises complex addresses, and the second term penalises deviation of the code z t from the memory-based prior.

Original: In this way, the memory learns useful representations that do not rely on complex addresses, and the bottom-up evidence only corrects top-down memory reading when necessary.
Revised: In this way, the memory learns useful representations that do not rely on complex addresses, and the bottom-up evidence only corrects top-down memory reading when necessary.

Original: 3. I TERATIVE S AMPLING
Revised: 3. I TERATIVE S AMPLING

Original: An important feature of Kanerva’s sparse distributed memory is its iterative reading mechanism, by which output from the model is fed back as input for several iterations.
Revised: An important feature of Kanerva’s sparse distributed memory is its iterative reading mechanism, by which output from the model is fed back as input for several iterations.

Original: Kanerva proved that the dynamics of iterative reading will decrease errors when the initial error is within a generous range, converging to a stored memory Kanerva (1988).
Revised: Kanerva proved that the dynamics of iterative reading will decrease errors when the initial error is within a generous range, converging to a stored memory Kanerva (1988).

Original: A similar iterative process is also available in our model, by repeatedly feeding-back the reconstruction ˆ x t .
Revised: A similar iterative process is also available in our model, by repeatedly feeding-back the reconstruction ˆ x t .

Original: This Gibbs-like sampling follows the loop in Fig.
Revised: This Gibbs-like sampling follows the loop in Fig.

Original: 1 (central).
Revised: 1 (central).

Original: While we cannot prove convergence, in our experiments iterative reading reliably improves denoising and sampling.
Revised: While we cannot prove convergence, in our experiments iterative reading reliably improves denoising and sampling.

Original: To understand this process, notice that knowledge about memory is helpful in reading, which suggests using q φ ( y t | x t , M ) instead of q φ ( y t | x t ) for addressing (section 3.2).
Revised: To understand this process, notice that knowledge about memory is helpful in reading, which suggests using q φ ( y t | x t , M ) instead of q φ ( y t | x t ) for addressing (section 3.2).

Original: Unfortunately, training a parameterised model with the whole matrix M as input can be prohibitively costly.
Revised: Unfortunately, training a parameterised model with the whole matrix M as input can be prohibitively costly.

Original: Nevertheless, it is well-known in the coding literature that such intractable posteriors that usually arise in non-tree graph (as Fig. 1) can be approximated efﬁciently by loopy belief-propagation, as has been used infamous algorithms like Turbo coding (Frey & MacKay, 1998).
Revised: Nevertheless, it is well-known in the coding literature that such intractable posteriors that usually arise in non-tree graph (as Fig. 1) can be approximated efﬁciently by loopy belief-propagation, as has been used in algorithms like Turbo coding (Frey & MacKay, 1998).

Original: Similarly, we believe iterative reading works in our model because q φ ( y t | x t ) models the local coupling between x t and y t well enough, so iterative sampling with the rest of the model is likely to converge to the true posterior q φ ( y t | x t , M ) .
Revised: Similarly, we believe iterative reading works in our model because q φ ( y t | x t ) models the local coupling between x t and y t well enough, so iterative sampling with the rest of the model is likely to converge to the true posterior q φ ( y t | x t , M ) .

Original: Future research will seek to better understand this process.
Revised: Future research will seek to better understand this process.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Details of our model implementation are described in Appendix C. We use straightforward encoder and decoder models in order to focus on evaluating the improvements provided by an adaptive memory.
Revised: Details of our model implementation are described in Appendix C. We use straightforward encoder and decoder models in order to focus on evaluating the improvements provided by an adaptive memory.

Original: In particular, we use the same model architecture for all experiments with both Omniglot and CIFAR dataset, changing only the the number of ﬁlters in the convolutional layers, memory size, and code size.
Revised: In particular, we use the same model architecture for all experiments with both Omniglot and CIFAR dataset, changing only the the number of ﬁlters in the convolutional layers, memory size, and code size.

Original: We always use the on-line version of the update rule (section 3.3).
Revised: We always use the on-line version of the update rule (section 3.3).

Original: The Adam optimiser was used for all training and required minimal tuning for our model (Kingma & Ba, 2014).
Revised: The Adam optimiser was used for all training and required minimal tuning for our model (Kingma & Ba, 2014).

Original: In all experiments, we report the value of variational lower bound (eq. 

Revised: In all experiments, we report the value of variational lower bound (eq.

Original: L divided by the length of episode T , so the per-sample value can be compared with the likelihood from existing models.
Revised: L divided by the length of episode T , so the per-sample value can be compared with the likelihood from existing models.

Original: We ﬁrst used the Omniglot dataset to test our model.
Revised: We ﬁrst used the Omniglot dataset to test our model.

Original: This dataset contains images of hand-written characters with 1623 different classes and 20 examples in each class Lake et al.
Revised: This dataset contains images of hand-written characters with 1623 different classes and 20 examples in each class Lake et al.

Original: This large variation creates challenges for models trying to capture the entire complex distribution.
Revised: This large variation creates challenges for models trying to capture the entire complex distribution.

Original: We use a 64 × 100 memory M , and a smaller 64 × 50 address matrix A .
Revised: We use a 64 × 100 memory M , and a smaller 64 × 50 address matrix A .

Original: For simplicity, we always randomly sample 32 images from the entire training set to form an “episode”, and ignore the class labels.
Revised: For simplicity, we always randomly sample 32 images from the entire training set to form an “episode”, and ignore the class labels.

Original: This represents a worst case scenario since the images in an episode will tend to have relatively little redundant information for compression.
Revised: This represents a worst case scenario since the images in an episode will tend to have relatively little redundant information for compression.

Original: We use a mini-batch size of 16, and optimise the variational lower-bound (eq. 12) using Adam with learning rate 1 × 10 − 4 .
Revised: We use a mini-batch size of 16, and optimise the variational lower-bound (eq. 12) using Adam with learning rate 1 × 10 − 4 .

Original: We also tested our model with the CIFAR dataset, in which each 32 × 32 × 3 real-valued colour image contains much more information than a binary omniglot pattern.
Revised: We also tested our model with the CIFAR dataset, in which each 32 × 32 × 3 real-valued colour image contains much more information than a binary omniglot pattern.

Original: Again, we discard all the label information and test our model in the unsupervised setting.
Revised: Again, we discard all the label information and test our model in the unsupervised setting.

Original: To accommodation the increased complexity of CIFAR, we use convolutional coders with 32 features at each layer, use alarger code size of 200 , and alarger 128 × 200 memory with 128 × 50 address matrix.
Revised: To accommodate the increased complexity of CIFAR, we use convolutional coders with 32 features at each layer, use a code size of 200 , and a 128 × 200 memory with 128 × 50 address matrix.

Original: All other settings are identical to experiments with Omniglot.
Revised: All other settings are identical to experiments with Omniglot.

Original: 4. C OMPARISON WITH VAE S
Revised: 4. C OMPARISON WITH VAE S

Original: We ﬁrst use the 28 × 28 binary Omniglot from Burda et al.
Revised: We ﬁrst use the 28 × 28 binary Omniglot from Burda et al.

Original: and follow the same split of 24, training and 8,070 test examples.
Revised: and follow the same split of 24, training and 8,070 test examples.

Original: We ﬁrst compare the training process of our model with a baseline
Revised: We ﬁrst compare the training process of our model with a baseline

Original: VAE model using the exact same encoder and decoder.
Revised: VAE model using the exact same encoder and decoder.

Original: Note that there is only a modest increase of parameters in the Kanerva Machine compared the VAE since the encoder and decoder dominates the model parameters.
Revised: Note that there is only a modest increase of parameters in the Kanerva Machine compared the VAE since the encoder and decoder dominates the model parameters.

Original: Fig.
Revised: Fig.

Original: 2 shows learning curves for our model along with those for the VAE trained on the Omniglot dataset.
Revised: 2 shows learning curves for our model along with those for the VAE trained on the Omniglot dataset.

Original: We plot 4 randomly initialised instances for each model.
Revised: We plot 4 randomly initialised instances for each model.

Original: The training is stable and insensitive to initialisation.
Revised: The training is stable and insensitive to initialisation.

Original: Fig.
Revised: Fig.

Original: 2 (left) shows that our model reached a signiﬁcantly lower negative variational lower-bound versus the VAE.
Revised: 2 (left) shows that our model reached a signiﬁcantly lower negative variational lower-bound versus the VAE.

Original: Fig.
Revised: Fig.

Original: 2 (central) and (right) further shows that the Kanerva Machine achieved signiﬁcantly better reconstruction and KL-divergence.
Revised: 2 (central) and (right) further shows that the Kanerva Machine achieved signiﬁcantly better reconstruction and KL-divergence.

Original: In particular, the KL-divergence of our model “dips” sharply from about the 2000th step, implying our model learned to use the memory to derive a more informative prior.
Revised: In particular, the KL-divergence of our model “dips” sharply from about the 2000th step, implying our model learned to use the memory to derive a more informative prior.

Original: Fig.
Revised: Fig.

Original: 11 conﬁrms this: the KL-divergence for z t has collapsed to near zero, showing that the top-down prior from memory q φ ( z t | y t , M ) provides most of the information for the code.
Revised: 11 conﬁrms this: the KL-divergence for z t has collapsed to near zero, showing that the top-down prior from memory q φ ( z t | y t , M ) provides most of the information for the code.

Original: This rich prior is achieved at the cost of an additional KL-divergence for y t (Fig.
Revised: This rich prior is achieved at the cost of an additional KL-divergence for y t (Fig.

Original: 11, right) which is still much lower than the KL-divergence for z t in a VAE.
Revised: 11, right) which is still much lower than the KL-divergence for z t in a VAE.

Original: Similar patterns of training curves are observed for CIFAR training (Fig. 12).
Revised: Similar patterns of training curves are observed for CIFAR training (Fig. 12).

Original: Gemici et al.
Revised: Gemici et al.

Original: (2017) also observed such
Revised: (2017) also observed such

Original: KL-divergence dips with a memory model.
Revised: KL-divergence dips with a memory model.

Original: They report the reduction in KL-divergence, rather than the reduction in reconstruction loss, signiﬁcantly improved sample quality, which we also observed in our experiments with Omniglot and CIFAR.
Revised: They report the reduction in KL-divergence, rather than the reduction in reconstruction loss, signiﬁcantly improved sample quality, which we also observed in our experiments with Omniglot and CIFAR.

Original: At the end of training, our VAE reached negative log-likelihood (NLL) of ≤ 112 .
Revised: At the end of training, our VAE reached negative log-likelihood (NLL) of ≤ 112 .

Original: 7 (the lower-bound of likelihood), which is worse than the state-of-the-art unconditioned generation that is achieved by rolling out 80 steps of a DRAW model (NLL of 95.5, Rezende et al., 2016), but comparable to results with IWAE training (NLL of 103.4, Burda et al., 2015).
Revised: 7 (the lower-bound of likelihood), which is worse than the state-of-the-art unconditioned generation that is achieved by rolling out 80 steps of a DRAW model (NLL of 95.5, Rezende et al., 2016), but comparable to results with IWAE training (NLL of 103.4, Burda et al., 2015).

Original: In contrast, with the same encoder and decoders, the Kanerva Machine achieve conditional NLL of 68.3.
Revised: In contrast, with the same encoder and decoders, the Kanerva Machine achieve conditional NLL of 68.3.

Original: It is not fair to directly compare our results with unconditional generative models since our model has the advantage of its memory contents.
Revised: It is not fair to directly compare our results with unconditional generative models since our model has the advantage of its memory contents.

Original: Nevertheless, the dramatic improvement of NLL demonstrates the power of incorporating an adaptive memory into generative models.
Revised: Nevertheless, the dramatic improvement of NLL demonstrates the power of incorporating an adaptive memory into generative models.

Original: Fig.
Revised: Fig.

Original: 3 (left) shows examples of reconstruction at the end of training; as a signature of our model, the weights were well distributed over the memory, illustrating that patterns written into the memory were superimposed on others.
Revised: 3 (left) shows examples of reconstruction at the end of training; as a signature of our model, the weights were well distributed over the memory, illustrating that patterns written into the memory were superimposed on others.

Original: iterations
Revised: iterations

Original: 4. O NE - SHOT GENERATION
Revised: 4. O NE - SHOT GENERATION

Original: We generalise “one-shot” generation from a single image (Rezende et al., 2016), or a few sample images from a limited set of classes (Edwards & Storkey, 2016; Bartunov & Vetrov, 2016), to a batch of images with many classes and samples.
Revised: We generalise “one-shot” generation from a single image (Rezende et al., 2016), or a few sample images from a limited set of classes (Edwards & Storkey, 2016; Bartunov & Vetrov, 2016), to a batch of images with many classes and samples.

Original: To better illustrate how samples are shaped by the conditioning data, in this section we use the same trained models, but test them using episodes with samples from only 2, 4 or 12 classes (omniglot characters) 2 .
Revised: To better illustrate how samples are shaped by the conditioning data, in this section we use the same trained models, but test them using episodes with samples from only 2, 4 or 12 classes (omniglot characters) 2 .

Original: Fig.
Revised: Fig.

Original: 4 compares samples from the VAE and the Kanerva Machine.
Revised: 4 compares samples from the VAE and the Kanerva Machine.

Original: While initial samples from our model (left most columns) are visually about as good as those from the VAE, the sample quality improved in consecutive iterations and the ﬁnal samples clearly reﬂects the statistics of the conditioning patterns.
Revised: While initial samples from our model (left most columns) are visually about as good as those from the VAE, the sample quality improved in consecutive iterations and the ﬁnal samples clearly reﬂects the statistics of the conditioning patterns.

Original: Most samples did not change much after the 6th iteration, suggesting the iterative sampling had converged.
Revised: Most samples did not change much after the 6th iteration, suggesting the iterative sampling had converged.

Original: Similar conditional samples from CIFAR are shown in Fig.
Revised: Similar conditional samples from CIFAR are shown in Fig.

Original: Notice  this approach, however, does not apply to VAEs, since VAEs do not have the structure we discussed in section 3.5.
Revised: Notice that this approach, however, does not apply to VAEs, since VAEs do not have the structure we discussed in section 3.5.

Original: To illustrate it in Figure 8 by feeding back output from VAEs as input in the next iterations, which shows the sample quality did not improve after iterations.
Revised: This is illustrated in Figure 8 by feeding back output from VAEs as input to the next iteration, which shows the sample quality did not improve after iterations.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4. D ENOSING AND INTERPOLATION
Revised: 4. D ENOSING AND INTERPOLATION

Original: To further examine generalisation, we input images corrupted by randomly positioned 12 × 12 blocks, and tested whether our model can recover the original image through iterative reading.
Revised: To further examine generalisation, we input images corrupted by randomly positioned 12 × 12 blocks, and tested whether our model can recover the original image through iterative reading.

Original: Our model was not trained on this task, but Fig.
Revised: Our model was not trained on this task, but Fig.

Original: 3 (right) shows that, over several iterations, input images can be recovered.
Revised: 3 (right) shows that, over several iterations, input images can be recovered.

Original: Due to high ambiguity, some cases (e.g., the second and last) ended up producing incorrect but still reasonable patterns.
Revised: Due to high ambiguity, some cases (e.g., the second and last) ended up producing incorrect but still reasonable patterns.

Original: The structure of our model affords interpretability of internal representations in memory.
Revised: The structure of our model affords interpretability of internal representations in memory.

Original: Since representations of data x are obtained from a linear combination of memory slots (eq. 6), we expect linear interpolations between address weights to be meaningful.
Revised: Since representations of data x are obtained from a linear combination of memory slots (eq. 6), we expect linear interpolations between address weights to be meaningful.

Original: We examined interpolations by computing 2 weight vectors from two random input images, and then linearly interpolating between these two vectors.
Revised: We examined interpolations by computing 2 weight vectors from two random input images, and then linearly interpolating between these two vectors.

Original: These vectors were then used to read z t from memory (eq. 6), which is then decoded to produce the interpolated images.
Revised: These vectors were then used to read z t from memory (eq. 6), which is then decoded to produce the interpolated images.

Original: Fig.
Revised: Fig.

Original: 7 in Appendix A shows that interpolating between these access weights indeed produces meaningful and smoothly changing images.
Revised: 7 in Appendix A shows that interpolating between these access weights indeed produces meaningful and smoothly changing images.

Original: 4. C OMPARISON WITH D IFFERENTIABLE N EURAL C OMPUTERS
Revised: 4. C OMPARISON WITH D IFFERENTIABLE N EURAL C OMPUTERS

Original: This section compares our model with a popular memory model, the Differentiable Neural Computer (DNC, Graves et al., 2016), and a variant of it, the Least Recent Used Architecture (LRUA, Santoro et al., 2016).
Revised: This section compares our model with a popular memory model, the Differentiable Neural Computer (DNC, Graves et al., 2016), and a variant of it, the Least Recently Used Architecture (LRUA, Santoro et al., 2016).

Original: We test them using the same episode storage and retrieval task as in previous experiments with Omniglot data.
Revised: We test them using the same episode storage and retrieval task as in previous experiments with Omniglot data.

Original: For a fair comparison, we ﬁt the DNC models into the same framework, as detailed in Appendix D. Fig.
Revised: For a fair comparison, we ﬁt the DNC models into the same framework, as detailed in Appendix D. Fig.

Original: 6 (left) illustrates the process of training the DNC and the Kanerva
Revised: 6 (left) illustrates the process of training the DNC and the Kanerva

Original: Machine.
Revised: Machine.

Original: Unfortunately, the LRUA never passed the loss level of 150, so weconclude it failed in our task and did not include it in the ﬁgure.
Revised: The LRUA did not passed the loss level of 150, so we did not include it in the ﬁgure.

Original: The DNC reached a test loss close to 100, but was very sensitive to hyper-parameters and random initialisation: only 2 out of 6 instances with the best hyper-parameter conﬁguration (batch size = 16, learning rate= 3 × 10 − 4 ) found by grid search reached this level.
Revised: The DNC reached a test loss close to 100, but was very sensitive to hyper-parameters and random initialisation: only 2 out of 6 instances with the best hyper-parameter conﬁguration (batch size = 16, learning rate= 3 × 10 − 4 ) found by grid search reached this level.

Original: On the other hand, the Kanerva Machine wasvery robust to these hyper-parameters, and worked well with batch size from 8 to 64, and learning rates from 3 × 10 − 5 to 3 × 10 − 4 .
Revised: On the other hand, the Kanerva Machine was robust to these hyper-parameters, and worked well with batch sizes between 8 and 64, and learning rates between 3 × 10 − 5 and 3 × 10 − 4 .

Original: Although it trained faster under the best conﬁguration of batch size 16 and learning rate 1 × 10 − 4 , the Kanerva Machine could eventually converge to at least below 70 test loss with all tested conﬁgurations.
Revised: The Kanerva Machine trained fastest with batch size 16 and learning rate 1 × 10 − 4 and eventually converged below 70 test loss with all tested conﬁgurations.

Original: Therefore, the Kanerva Machine is signiﬁcantly easier to train, thanks to principled reading and writing operations that do not depends on any model parameter.
Revised: Therefore, the Kanerva Machine is signiﬁcantly easier to train, thanks to principled reading and writing operations that do not depend on any model parameter.

Original: We next analysed the capacity of our model versus the DNC by examining the (lower bound of) likelihood when storing and then retrieving patterns from increasingly large episodes.
Revised: We next analysed the capacity of our model versus the DNC by examining the lower bound of then likelihood when storing and then retrieving patterns from increasingly large episodes.

Original: As above, these models are still trained with episodes containing 32 samples, but are tested on much larger episode.
Revised: As above, these models are still trained with episodes containing 32 samples, but are tested on much larger episodes.

Original: We tested our model with episodes containing different numbers of classes and thus varying amounts of redundancy.
Revised: We tested our model with episodes containing different numbers of classes and thus varying amounts of redundancy.

Original: Fig.
Revised: Fig.

Original: 6 (right) shows both models are able to exploit this redundancy, since episodes with fewer classes (but the same number of images) have lower reconstruction losses.
Revised: 6 (right) shows both models are able to exploit this redundancy, since episodes with fewer classes (but the same number of images) have lower reconstruction losses.

Original: Overall, the
Revised: Overall, the

Original: Kanerva Machine generalise well to larger episodes, and maintained a clear advantage over the DNC (as measured by the variational lower-bound).
Revised: Kanerva Machine generalises well to larger episodes, and maintained a clear advantage over the DNC (as measured by the variational lower-bound).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper, we present the Kanerva Machine, a novel memory model that combines slow-learning neural networks and a fast-adapting linear Gaussian model as memory.
Revised: In this paper, we present the Kanerva Machine, a novel memory model that combines slow-learning neural networks and a fast-adapting linear Gaussian model as memory.

Original: While our architecture is inspired by Kanerva’s seminal model, we have removed the assumption of a uniform data distribution by training a generative model that ﬂexibly learns the observed data distribution.
Revised: While our architecture is inspired by Kanerva’s seminal model, we have removed the assumption of a uniform data distribution by training a generative model that ﬂexibly learns the observed data distribution.

Original: By implementing memory as a generative model, we can retrieve unseen patterns from the memory through sampling.
Revised: By implementing memory as a generative model, we can retrieve unseen patterns from the memory through sampling.

Original: This phenomenon is consistent with the observation of constructive memory neuroscience experiments (Hassabis et al., 2007).
Revised: This phenomenon is consistent with the observation of constructive memory neuroscience experiments (Hassabis et al., 2007).

Original: Probabilistic interpretations of Kanerva’s model have been developed in previous works: Anderson (1989) explored a conditional probability interpretation of Kanerva’s sparse distributed memory, and generalised binary data to discrete data with more than two values.
Revised: Probabilistic interpretations of Kanerva’s model have been developed in previous works: Anderson (1989) explored a conditional probability interpretation of Kanerva’s sparse distributed memory, and generalised binary data to discrete data with more than two values.

Original: Abbott et al.
Revised: Abbott et al.

Original: (2013) provides an approximate Bayesian interpretation based on importance sampling.
Revised: (2013) provides an approximate Bayesian interpretation based on importance sampling.

Original: To our knowledge, our model is the ﬁrst to generalise Kanerva’s memory model to continuous, non-uniform data while maintaining an analytic form of Bayesian inference.
Revised: To our knowledge, our model is the ﬁrst to generalise Kanerva’s memory model to continuous, non-uniform data while maintaining an analytic form of Bayesian inference.

Original: Moreover, we demonstrate its potential in modern machine learning through integration with deep neural networks.
Revised: Moreover, we demonstrate its potential in modern machine learning through integration with deep neural networks.

Original: Other models have combined memory mechanisms with neural networks in a generative setting.
Revised: Other models have combined memory mechanisms with neural networks in a generative setting.

Original: For example, Li et al.
Revised: For example, Li et al.

Original: (2016) usedan attention to retrieve information from a set of trainable parameters in a memory matrix.
Revised: (2016) used attention to retrieve information from a set of trainable parameters in a memory matrix.

Original: Notably, the memory in this model is not updated following learning.
Revised: Notably, the memory in this model is not updated following learning.

Original: As a result, the memory does not quickly adapt to new data as in our model, and so is not suited to the kind of episode-based learning explored here.
Revised: As a result, the memory does not quickly adapt to new data as in our model, and so is not suited to the kind of episode-based learning explored here.

Original: Bornschein et al.
Revised: Bornschein et al.

Original: (2017) used discrete (categorical) random variables, instead of continuous attention, to address an external memory, and train the addressing mechanism, together with the rest of the generative model, though a variational objective.
Revised: (2017) used discrete (categorical) random variables, instead of continuous attention, to address an external memory, and train the addressing mechanism, together with the rest of the generative model, though a variational objective.

Original: However, the memory in their model is populated by storing images in the form of raw pixels.
Revised: However, the memory in their model is populated by storing images in the form of raw pixels.

Original: Although this provides a mechanism for fast adaptation, the cost of storing raw pixels may be overwhelming for large data sets.
Revised: Although this provides a mechanism for fast adaptation, the cost of storing raw pixels may be overwhelming for large data sets.

Original: Our model learns to to store information in a compressed form by taking advantage of statistical regularity in the images through the encoder at perceptual level, the learned addresses, and
Revised: Our model learns to to store information in a compressed form by taking advantage of statistical regularity in the images through the encoder at perceptual level, the learned addresses, and

Original: Bayes’ rule in the memory.
Revised: Bayes’ rule in the memory.

Original: Central to an effective memory model is the efﬁcient updating of memory.
Revised: Central to an effective memory model is the efﬁcient updating of memory.

Original: While various approaches to learning such updating mechanisms have been examined recently (Graves et al., 2016; Edwards &
Revised: While various approaches to learning such updating mechanisms have been examined recently (Graves et al., 2016; Edwards &

Original: Storkey, 2016; Santoro et al., 2016), we designed our model to employ an exact Bayes’ update-rule without compromising the ﬂexibility and expressive power of neural networks.
Revised: Storkey, 2016; Santoro et al., 2016), we designed our model to employ an exact Bayes’ update-rule without compromising the ﬂexibility and expressive power of neural networks.

Original: The compelling performance of our model and its scalable architecture suggests combining classical statistical models and neural networks may be a promising direction for novel memory models in machine learning
Revised: The compelling performance of our model and its scalable architecture suggests combining classical statistical models and neural networks may be a promising direction for novel memory models in machine learning

Original: B S PARSE D ISTRIBUTED M EMORY
Revised: B S PARSE D ISTRIBUTED M EMORY

Original: This section reviews Kanerva’s sparse distributed memory Kanerva (1988).
Revised: This section reviews Kanerva’s sparse distributed memory (Kanerva, 1988).

Original: For consistency with the rest of this paper, many of the notations are different from Kanerva’s description.
Revised: For consistency with the rest of this paper, many of the notations are different from Kanerva’s description.

Original: In contrast to many recent models, Kanerva’s memory model is characterised by its distributed reading and writing operations.
Revised: In contrast to many recent models, Kanerva’s memory model is characterised by its distributed reading and writing operations.

Original: The model has two main components: a ﬁxed table of addresses A pointing to a modiﬁable memory M .
Revised: The model has two main components: a ﬁxed table of addresses A pointing to a modiﬁable memory M .

Original: Both A and M have the same size of K × D , where K is the number of addresses that
Revised: Both A and M have the same size of K × D , where K is the number of addresses that

Original: and D is the input dimensionality.
Revised: and D is the input dimensionality.

Original: Kanerva assumes all the inputs are uniform random vectors y ∈ {− 1 , 1 } D .
Revised: Kanerva assumes all the inputs are uniform random vectors y ∈ {− 1 , 1 } D .

Original: Therefore, the ﬁxed addresses A i are uniformly randomly sampled from {− 1 , 1 } D to reﬂect the input statistics.
Revised: Therefore, the ﬁxed addresses A i are uniformly randomly sampled from {− 1 , 1 } D to reﬂect the input statistics.

Original: An input y is compared with each address A k in A through the Hamming distance.
Revised: An input y is compared with each address A k in A through the Hamming distance.

Original: For binary vectors a, b ∈ {− 1 , 1 } D , the Hamming distance can be written as h ( a, b ) =
Revised: For binary vectors a, b ∈ {− 1 , 1 } D , the Hamming distance can be written as h ( a, b ) =

Original: ( D − a · b ) where · represents inner product between two vectors.
Revised: ( D − a · b ) where · represents inner product between two vectors.

Original: An address k is selected when the hamming distance between x and A k is smaller than a threshold τ , so the selection can be summarised by the binary weight vector:
Revised: An address k is selected when the hamming distance between x and A k is smaller than a threshold τ , so the selection can be summarised by the binary weight vector:

Original: [Equation]
Revised: [Equation]

Original: During writing, a pattern x is stored into M by adding M k ← M k + w k x .
Revised: During writing, a pattern x is stored into M by adding M k ← M k + w k x .

Original: For reading, the memory contents pointed to by all the selected addresses are summed together to pass a threshold at 0 to produce a read out:
Revised: For reading, the memory contents pointed to by all the selected addresses are summed together to pass a threshold at 0 to produce a read out:

Original: [Equation]
Revised: [Equation]

Original: This reading process can be iterated several times by repeatedly feeding-back the output ˆ x as input.
Revised: This reading process can be iterated several times by repeatedly feeding-back the output ˆ x as input.

Original: It has been shown analytically by Kanerva that, when both K and D are large enough, a small portion of the addresses will always be selected, thus the operations are sparse and distributed.
Revised: It has been shown analytically by Kanerva that when both K and D are large enough, a small portion of the addresses will always be selected, thus the operations are sparse and distributed.

Original: Although an address’ content may be over-written many times, the stored vectors can be retrieved correctly.
Revised: Although an address’ content may be over-written many times, the stored vectors can be retrieved correctly.

Original: Moreover, Kanerva proved that even a signiﬁcantly corrupted query can be discovered from the memory through iterative reading.
Revised: Moreover, Kanerva proved that even a signiﬁcantly corrupted query can be discovered from the memory through iterative reading.

Original: However, the application of Kanerva’s model is restricted by the assumption of a uniform and binary data distribution, on which Kanerva’s analyses and bounds of performance rely Kanerva (1988).
Revised: However, the application of Kanerva’s model is restricted by the assumption of a uniform and binary data distribution, on which Kanerva’s analyses and bounds of performance rely (Kanerva, 1988).

Original: Unfortunately, this assumption is rarely true in practice, since real-world data typically lie on low-dimensional manifolds, and binary representation of data is less efﬁcient in high-level neural network implementations that are heavily optimised for ﬂoating-point numbers.
Revised: Unfortunately, this assumption is rarely true in practice, since real-world data typically lie on low-dimensional manifolds, and binary representation of data is less efﬁcient in high-level neural network implementations that are heavily optimised for ﬂoating-point numbers.

Original: C M ODEL DETAILS
Revised: C M ODEL DETAILS

Original: Figure 9 shows the architecture of our model compared with a standard VAE.
Revised: Figure 9 shows the architecture of our model compared with a standard VAE.

Original: For all experiments, we use a convolutional encoder to convert input images into 2 C embedding vectors e ( x t ) , where
Revised: For all experiments, we use a convolutional encoder to convert input images into 2 C embedding vectors e ( x t ) , where

Original: C is the code size (dimension of z t ).
Revised: C is the code size (dimension of z t ).

Original: The convolutional encoder has 3 consecutive blocks, where each block is a convolutional layer with 4 × 4 ﬁlter with stride 2, which reduces the input dimension, followed by a basic ResNet block without bottleneck (He et al., 2016).
Revised: The convolutional encoder has 3 consecutive blocks, where each block is a convolutional layer with 4 × 4 ﬁlter with stride 2, which reduces the input dimension, followed by a basic ResNet block without bottleneck (He et al., 2016).

Original: All the convolutional layers have the same number of ﬁlters, which is either 16 or 32 depending on the dataset.
Revised: All the convolutional layers have the same number of ﬁlters, which is either 16 or 32 depending on the dataset.

Original: The output from the blocks is ﬂattened and linearly projected to a 2 C dimensional vector.
Revised: The output from the blocks is ﬂattened and linearly projected to a 2 C dimensional vector.

Original: The convolutional decoder mirrors this structure with transposed convolutional layers.
Revised: The convolutional decoder mirrors this structure with transposed convolutional layers.

Original: All the “MLP” boxes in Fig.
Revised: All the “MLP” boxes in Fig.

Original: 9 are 2-layer multi-layer perceptron with ReLU non-linearity in between.
Revised: 9 are 2-layer multi-layer perceptron with ReLU non-linearity in between.

Original: We found that adding noise to the input into q φ ( y t | x t ) helped stabilise training, possibly by restricting the information in the addresses.
Revised: We found that adding noise to the input into q φ ( y t | x t ) helped stabilise training, possibly by restricting the information in the addresses.

Original: The exact magnitude of the added noise matters little, and we use Gaussian noise with zero mean and standard deviation of 0 . 2 for all experiments.
Revised: The exact magnitude of the added noise matters little, and we use Gaussian noise with zero mean and standard deviation of 0 . 2 for all experiments.

Original: We use Bernoulli likelihood function for Omniglot dataset, and Gaussian likelihood function for CIFAR.
Revised: We use Bernoulli likelihood function for Omniglot dataset, and Gaussian likelihood function for CIFAR.

Original: To avoid Gaussian likelihood collapsing, we added uniform noise U (0 , 1256 ) to CIFAR images during training.
Revised: To avoid Gaussian likelihood collapsing, we added uniform noise U (0 , 1256 ) to CIFAR images during training.

Original: D DNC DETAILS
Revised: D DNC DETAILS

Original: For a fair comparison, we wrap the differentiable neural computer (DNC) with the same interface as the Kanerva memory, so that it can simply replace the memory M in Fig.
Revised: For a fair comparison, we wrap the differentiable neural computer (DNC) with the same interface as the Kanerva memory so that it can simply replace the memory M in Fig.

Original: More speciﬁcally, the
Revised: More speciﬁcally, the

Original: DNC receives the addressing variable y t with the same size and sampled the same ways as described in the main text in reading and writing stages.
Revised: DNC receives the addressing variable y t with the same size and sampled the same ways as described in the main text in reading and writing stages.

Original: During writing it also receives z t sampled from q φ ( z t | x t ) as input, by concatenating y t and z t together as input into the memory controller.
Revised: During writing it also receives z t sampled from q φ ( z t | x t ) as input, by concatenating y t and z t together as input into the memory controller.

Original: Since DNCs do not have separated reading and writing stages, we separated this two process in our experiments by: during writing, we discard the read-out from the DNC, and only keep its state as the memory; during reading, we discard the state at each step, so it cannot be used for storing new information.
Revised: Since DNCs do not have separated reading and writing stages, we separated this two process in our experiments: during writing, we discard the read-out from the DNC, and only keep its state as the memory; during reading, we discard the state at each step so it cannot be used for storing new information.

Original: In addition, we use a 2-layer MLP with 200 hidden neurons and ReLU nonlinearity as the controller instead of the commonly used LSTM to avoid the recurrent state being used as memory and interference with DNC’s external memory.
Revised: In addition, we use a 2-layer MLP with 200 hidden neurons and ReLU nonlinearity as the controller instead of the commonly used LSTM to avoid the recurrent state being used as memory and interference with DNC’s external memory.

Original: Another issue with off-the-shelf DNC (Graves et al., 2016; Santoro et al., 2016) is that controllers may generate output bypassing the memory, which can be particularly confusing in our auto-encoding setting by simply ignoring the memory and functioning as a skip connection.
Revised: Another issue with off-the-shelf DNC (Graves et al., 2016; Santoro et al., 2016) is that controllers may generate output bypassing the memory, which can be particularly confusing in our auto-encoding setting by simply ignoring the memory and functioning as a skip connection.

Original: We avoid this situation by removing this controller output and ensure that the
Revised: We avoid this situation by removing this controller output and ensure that the

Original: DNC only reads-out from its memory.
Revised: DNC only reads-out from its memory.

Original: Further, to focus on the memory performance, we removethe x t e ( x t ) p ✓ ( x t | z t ) x t e ( x t ) q  ( z t | x t , y t , M ) p ✓ ( x t | z t ) q
Revised: Further, to focus on the memory performance, we remove x t e ( x t ) p ✓ ( x t | z t ) x t e ( x t ) q  ( z t | x t , y t , M ) p ✓ ( x t | z t ) q

Original:  ( y t | x t )
Revised:  ( y t | x t )

Original: M p ✓ ( z t | y t , M ))
Revised: M p ✓ ( z t | y t , M ))

Original: q  ( z t | x t ) conv
Revised: q  ( z t | x t ) conv

Original: MLP deconv
Revised: MLP deconv

Original: MLP
Revised: MLP

Original: MLP concat x t e ( x t ) q  ( y t | x t )
Revised: MLP concat x t e ( x t ) q  ( y t | x t )

Original: M MLP q  ( z t | x t ) MLP p ✓ ( x t | z t )
Revised: M MLP q  ( z t | x t ) MLP p ✓ ( x t | z t )

Original: M p ✓ ( z t | y t , M ))
Revised: M p ✓ ( z t | y t , M ))

Original: p ✓ ( y t )
Revised: p ✓ ( y t )

Original: VAE generation reading writing  bottom-up stream in our model that compensates  the memory.
Revised: [Figure] generation [Figure] writing the bottom-up stream in our model that compensates for the memory.

Original: Thisessentially means directly sample z t from p
Revised: This means directly sampling z t from p

Original: θ ( z t | y t , M ) , instead of p θ ( z t | x t , y t , M ) , for the decoder p θ ( x t | z t ) ,
Revised: θ ( z t | y t , M ) , instead of p θ ( z t | x t , y t , M ) , for the decoder p θ ( x t | z t ) ,

Original: forcing the model to reconstruct solely using read-outs from the memory.
Revised: forcing the model to reconstruct solely using read-outs from the memory.

Original: E D ERIVATION OF THE ONLINE UPDATE RULE

Original: Eq.

Original: 6 deﬁnes a linear Gaussian model.

Original: Using notations in the main paper, can write the joint distribution p (vec ( Z ) , vec( M ))

Original: =

Original: N (vec ( Z ) , vec( M ); µ j , Σ j ) , where

Original: [Equation]

Original: CIFAR training.
Revised: CIFAR training.

Original: Although the difference between the lower bound objective is smaller than that during Omniglot training, the general patterns of these curves are similar to those in Fig.
Revised: Although the difference between the lower bound objective is smaller than that during Omniglot training, the general patterns of these curves are similar to those in Fig.

Original: The relatively small difference in KL-divergence signiﬁcantly inﬂuences sample quality.
Revised: The relatively small difference in KL-divergence signiﬁcantly inﬂuences sample quality.

Original: Notice at the time of our submission, the training is continuing and the advantage of the Kanerva Machine over the
Revised: Notice at the time of our submission, the training is continuing and the advantage of the Kanerva Machine over the

Original: VAE is increasing.
Revised: VAE is increasing.

Revised: E D ERIVATION OF THE ONLINE UPDATE RULE

Original: We can then use the conditional formula for the Gaussian to derive the posterior distribution p (vec ( M ) | vec( Z
Revised: Eq. 6 deﬁnes a linear Gaussian model. Using notations in the main paper, can write the joint distribution p (vec ( Z ) , vec

Original: [Equation] 
Revised: [Equation] vec

Original: = N

Original: [Equation]
Revised: [Equation]

Original: vec
Revised: Z ) , vec( M ); µ j , Σ j ) , where

Original: [Equation]
Revised: [Equation]

Original: M

Original: [Equation]
Revised: [Equation]

Revised: We can then use the conditional formula for the Gaussian to derive the posterior distribution p (vec ( M ) | vec ( Z ))

Original:  µ p , Σ p ) , using the property Kronecker product:
Revised: = N (vec ( M ) ; µ p , Σ p ) , using the property Kronecker product:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: From properties of matrix variate Gaussian distribution, the above two equations can be re-arranged to the update rule in eq.
Revised: From properties of matrix variate Gaussian distribution, the above two equations can be re-arranged to the update rule in eq.

Original: 9 to 11.
Revised: 9 to 11.

Original: do sample compute
Revised: While the model we described in this paper works well using samples from q φ ( z t | x t ) for writing to the memory (section 3.3) and the mean-ﬁeld approximation during reading (section 3.4), here we describe an alternative that fully exploits the analytic tractability of the Gaussian distribution. To simplify notation, we use ψ = { R, U, V } for all parameters of the memory. For reading, eq. 6 can be replaced with a distribution that directly depends on ψ through the integral:

Original: ← f (
Revised: [Equation]

Original: Compute the weights w t ← b (cid:124) t · A ; read out µ
Revised: [Equation] For writing, the distribution q φ ( Z | X ) =

Original: ← w t (cid:124) · M ; sample end
Revised: (cid:81)

Original: ∆ ← Z − W R Σ c ← W U Σ
Revised: Tt = 1 q φ ( z t | x t ) =

Original: z ← W U W (cid:124)
Revised: N ( µ Q , Σ Q ) ( µ Q and Σ Q are functions of X ) can be incorporated into the Bayes’ update rule by analytically marginalising-out Z : [Equation] where we used Bayes’ rule and dropped the normalising constant p θ ( Z | Y )

Original: + Σ ξ
Revised: [Equation]

Original: R ← R + Σ c (cid:124) Σ − 1 z
Revised: and then replaced the equality with proportional-to accordingly. The last integral is: [Equation] [Equation]

Original: U ← U − Σ c (cid:124) Σ− 1 z 
Revised: where Σ z (cid:48) = W U W (cid:124)

Revised: + Σ ξ + Σ Q and p (cid:48) ( Z ) is a distribution of Z whose exact form is unimportant.

Revised: Therefore, eq. 20 shows that the posterior distribution of M is proportional to the product between the prior p θ ( M ) and the above likelihood term.

Revised: From inspection, we can see the update rule (eq. 911) needs to be modiﬁed by replacing Σ z with Σ z (cid:48) by adding the bottom-up uncertainty Σ Q .

Revised: G D ESCRIPTION OF THE A LGORITHM

Revised: Algorithm 1 Iterative Reading

Revised: Input: Memory M , a (potentially noisy) query x t , the number of iteration n

Revised: Output: An estimate of the noiseless ˆ x t initialise i

Revised: =

Revised: while i < n do sample y t ∼ q φ ( y t | x t ) compute the key b t ← f ( y t ) Compute the weights w t ← b (cid:124) t

Revised: · A read-out mean µ z ← w t (cid:124) ·

Revised: M sample z t ∼ p θ ( z t | x t , y t , M ) which takes µ z and x t as inputs sample the new query x t ∼ p θ ( x t | z t ) increment i ← i + 1 end while return ˆ x ← x t

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Output

Revised: [Table]

Revised: do

Revised: [Table]