Original: BayesPCN: A Continually Learnable Predictive Coding Associative Memory
Revised: BayesPCN: A Continually Learnable Predictive Coding Associative Memory

Original: Abstract
Revised: Abstract

Original: Associative memory plays an important role in human intelligence and its mechanisms have been linked to attention in machine learning.
Revised: Associative memory plays an important role in human intelligence and its mechanisms have been linked to attention in machine learning.

Original: While the machinelearning community’s interest in associative memories has recently been rekindled,most work has focused on memory recall ( read ) over memory learning ( write ).
Revised: While the machinelearning community’s interest in associative memories has recently been rekindled,most work has focused on memory recall ( read ) over memory learning ( write ).

Original: In this paper, we present BayesPCN, a hierarchical associative memory capable ofperforming continual one-shot memory writes without meta-learning.
Revised: In this paper, we present BayesPCN, a hierarchical associative memory capable ofperforming continual one-shot memory writes without meta-learning.

Original: Moreover,
Revised: Moreover,

Original: BayesPCN is able to gradually forget past observations ( forget ) to free its memory.
Revised: BayesPCN is able to gradually forget past observations ( forget ) to free its memory.

Original: Experiments show that BayesPCN can recall corrupted i.i d. high-dimensional dataobserved hundreds of “timesteps” ago without a significant drop in recall abilitycompared to the state-of-the-art offline-learned associative memory models
Revised: Experiments show that BayesPCN can recall corrupted i.i d. high-dimensional dataobserved hundreds of “timesteps” ago without a significant drop in recall abilitycompared to the state-of-the-art offline-learned associative memory models

Original:  Introduction
Revised:  Introduction

Original: One of the hallmarks of biological intelligence is the ability to robustly recall learned associations.
Revised: One of the hallmarks of biological intelligence is the ability to robustly recall learned associations.

Original: Biological agents are able to associate stimuli of various modalities and rely on their memories toperform everyday tasks from navigation to entity recognition.
Revised: Biological agents are able to associate stimuli of various modalities and rely on their memories toperform everyday tasks from navigation to entity recognition.

Original: Yet these agents are not only goodat recall but can quickly and continually form new memories from as little as a single experience.
Revised: Yet these agents are not only goodat recall but can quickly and continually form new memories from as little as a single experience.

Original: How do biological agents achieve this feat while mitigating catastrophic forgetting [French, 1999]?
Revised: How do biological agents achieve this feat while mitigating catastrophic forgetting [French, 1999]?

Original: And how can we build neural networks that demonstrate these properties?
Revised: And how can we build neural networks that demonstrate these properties?

Original: This paper addresses thesequestions, focusing on fast, continual learning of neural artifacts that perform robust recall of i.i.d.high-dimensional observations.
Revised: This paper addresses thesequestions, focusing on fast, continual learning of neural artifacts that perform robust recall of i.i.d.high-dimensional observations.

Original: Associative memory models have been of interest in machine learning for decades [Steinbuch,1961, Hopfield, 1982].
Revised: Associative memory models have been of interest in machine learning for decades [Steinbuch,1961, Hopfield, 1982].

Original: They have been used to solve a wide range of problems from sequenceprocessing [Schmidhuber, 1992, Graves et al., 2014, Schlag et al., 2021] to pattern detection [Widrichet al., 2020, 2021, Seidl et al., 2021].
Revised: They have been used to solve a wide range of problems from sequenceprocessing [Schmidhuber, 1992, Graves et al., 2014, Schlag et al., 2021] to pattern detection [Widrichet al., 2020, 2021, Seidl et al., 2021].

Original: Their study has allowed us to build models that replicateaspects of biological intelligence [Kanerva, 1988, Hawkins and Blakeslee, 2004] and provided novelperspectives on machine learning toolsets like dot-product attention, transformers, and linear layersof deep neural networks [Ramsauer et al., 2020, Bricken and Pehlevan, 2021, Irie et al., 2022].
Revised: Their study has allowed us to build models that replicateaspects of biological intelligence [Kanerva, 1988, Hawkins and Blakeslee, 2004] and provided novelperspectives on machine learning toolsets like dot-product attention, transformers, and linear layersof deep neural networks [Ramsauer et al., 2020, Bricken and Pehlevan, 2021, Irie et al., 2022].

Original: Our main aim is to recall exact memories from a neural network that can rapidly and continuallylearn from a stream of data.
Revised: Our main aim is to recall exact memories from a neural network that can rapidly and continuallylearn from a stream of data.

Original: Towards this end we introduce BayesPCN, a Bayesian take on thegenerative predictive coding network (GPCN)
Revised: Towards this end we introduce BayesPCN, a Bayesian take on thegenerative predictive coding network (GPCN)

Original: [Salvatori et al., 2021] that combines predictive codingand locally conjugate Bayesian updates.
Revised: [Salvatori et al., 2021] that combines predictive codingand locally conjugate Bayesian updates.

Original: There are currently a number of associative memory systemsthat are continually learnable [Hopfield, 1982, Kanerva, 1988, Hawkins and George, 2007, Ramsaueret al., 2020, Sharma et al., 2022].
Revised: There are currently a number of associative memory systemsthat are continually learnable [Hopfield, 1982, Kanerva, 1988, Hawkins and George, 2007, Ramsaueret al., 2020, Sharma et al., 2022].

Original: However, to our knowledge, BayesPCN is the first parametricassociative memory model to continually learn hundreds of very high-dimensional observations(> 10,000 dimensions) while maintaining good recall performance.
Revised: However, to our knowledge, BayesPCN is the first parametricassociative memory model to continually learn hundreds of very high-dimensional observations(> 10,000 dimensions) while maintaining good recall performance.

Original: In addition, it is also the firsthierarchical memory which after repeated forgetting recovers its original memory state.
Revised: In addition, it is also the firsthierarchical memory which after repeated forgetting recovers its original memory state.

Original: [Figure]
Revised: [Figure]

Original:  Background
Revised:  Background

Original: 2.1 Auto-Associative and Hetero-Associative Memory
Revised: 2.1 Auto-Associative and Hetero-Associative Memory

Original: Before we begin let us review two useful concepts: auto-associative memory and hetero-associativememory.
Revised: Before we begin let us review two useful concepts: auto-associative memory and hetero-associativememory.

Original: The former, auto-associative memory, refers to a memory system that is content addressablebut for which the query may be a corrupted version of the original data item.
Revised: The former, auto-associative memory, refers to a memory system that is content addressablebut for which the query may be a corrupted version of the original data item.

Original: One can think aboutauto-associative memory systems as a kind of approximate nearest neighbor lookup mechanism.
Revised: One can think aboutauto-associative memory systems as a kind of approximate nearest neighbor lookup mechanism.

Original: Hetero-associative memory systems are key-value stores, meaning that the recalled “value” ispotentially of a different type to the “key.”
Revised: Hetero-associative memory systems are key-value stores, meaning that the recalled “value” ispotentially of a different type to the “key.”

Original: Crucially, hetero-associative memory systems can, in largepart, be implemented by auto-associative memory systems by forming a “joint” object to memorizeby combining the key and value into one object.
Revised: Crucially, hetero-associative memory systems can, in largepart, be implemented by auto-associative memory systems by forming a “joint” object to memorizeby combining the key and value into one object.

Original: Hetero-associative queries can then be processedsimply by looking up the joint object “corrupted” by discarding the value.
Revised: Hetero-associative queries can then be processedsimply by looking up the joint object “corrupted” by discarding the value.

Original: 2.2 Generative Predictive Coding Network
Revised: 2.2 Generative Predictive Coding Network

Original: A generative predictive coding network (GPCN)
Revised: A generative predictive coding network (GPCN)

Original: [Salvatori et al., 2021] is an energy-based associativememory model.
Revised: [Salvatori et al., 2021] is an energy-based associativememory model.

Original: As illustrated in Figure 2, GPCN takes the form of a fully connected neuralnetwork whose bottom layer activations characterize the sensory inputs and higher layer activationscharacterize the internal representations of the sensory inputs.
Revised: As illustrated in Figure 2, GPCN takes the form of a fully connected neuralnetwork whose bottom layer activations characterize the sensory inputs and higher layer activationscharacterize the internal representations of the sensory inputs.

Original: GPCN’s read operation (recall)initializes the bottom layer activations to the input query vector and iteratively denoises the queryuntil a stored value is retrieved by minimizing its energy function w.r.t.
Revised: GPCN’s read operation (recall)initializes the bottom layer activations to the input query vector and iteratively denoises the queryuntil a stored value is retrieved by minimizing its energy function w.r.t.

Original: some or all of its neuronactivations.
Revised: some or all of its neuronactivations.

Original: GPCN’s write operation (learning) adapts the network parameters to shape the energyfunction such that the stored datapoints become the local minima of the energy function.
Revised: GPCN’s write operation (learning) adapts the network parameters to shape the energyfunction such that the stored datapoints become the local minima of the energy function.

Original: Before describing GPCN’s read and write mechanisms in more detail, we introduce the notationwe use in this paper.
Revised: Before describing GPCN’s read and write mechanisms in more detail, we introduce the notationwe use in this paper.

Original: Let M be a GPCN with L + 1 layers of neurons where 0 and L denotes thebottom and the top layer index respectively.
Revised: Let M be a GPCN with L + 1 layers of neurons where 0 and L denotes thebottom and the top layer index respectively.

Original: Then, let d l be the width of the l -th layer, x l ∈ R
Revised: Then, let d l be the width of the l -th layer, x l ∈ R

Original: d
Revised: d

Original: l bethe row vector denoting the l -th layer’s pre-activation function neuron values, and f l : R d l → R d lbe the l -th layer’s activation function, for instance ReLU.
Revised: l bethe row vector denoting the l -th layer’s pre-activation function neuron values, and f l : R d l → R d lbe the l -th layer’s activation function, for instance ReLU.

Original: In addition, let W l ∈ R d l +1 ×
Revised: In addition, let W l ∈ R d l +1 ×

Original: d l where0 ≤ l < L be the synaptic weight matrix between l + 1 -th and l -th layers and W L ∈ R
Revised: d l where0 ≤ l < L be the synaptic weight matrix between l + 1 -th and l -th layers and W L ∈ R

Original: d L be the “bias”row vector that influences the dynamics of the top layer neuron activities x L .
Revised: d L be the “bias”row vector that influences the dynamics of the top layer neuron activities x L .

Original: One can intuitivelythink of the model parameters W 0:
Revised: One can intuitivelythink of the model parameters W 0:

Original: L as the memory content and the activations x 0: L as the read stateof the memory.
Revised: L as the memory content and the activations x 0: L as the read stateof the memory.

Original: [Equation]
Revised: [Equation]

Original: x 0 t x 1 t x 2 t
Revised: x 0 t x 1 t x 2 t

Original: W
Revised: W

Original: W
Revised: W

Original: Wµ Σ
Revised: Wµ Σ

Original: R
Revised: R

Original: U
Revised: U

Original: R
Revised: R

Original: U 0 1 : T
Revised: U 0 1 : T

Original: GPCN’s read and write are facilitated by predictive coding (PC)
Revised: GPCN’s read and write are facilitated by predictive coding (PC)

Original: [Rao and Ballard, 1999], abiologically plausible learning rule that can approximate backpropagation [Millidge et al., 2020].
Revised: [Rao and Ballard, 1999], abiologically plausible learning rule that can approximate backpropagation [Millidge et al., 2020].

Original: PC’s energy function is the sum of prediction errors across all network layers.
Revised: PC’s energy function is the sum of prediction errors across all network layers.

Original: For GPCN, the choiceof prediction error is the squared Euclidean distance between the actual layer activations and theprediction of the same layer activations made by their immediate upper layer activations.
Revised: For GPCN, the choiceof prediction error is the squared Euclidean distance between the actual layer activations and theprediction of the same layer activations made by their immediate upper layer activations.

Original: This leadsto the following energy function.
Revised: This leadsto the following energy function.

Original: [Equation]
Revised: [Equation]

Original: GPCN’s read iteratively performs gradient descent on the energy w.r.t.
Revised: GPCN’s read iteratively performs gradient descent on the energy w.r.t.

Original: x 0: L after setting the initialvalues of x 0 to the memory query ¯ x 0 and fixing the values of W 0: L .
Revised: x 0: L after setting the initialvalues of x 0 to the memory query ¯ x 0 and fixing the values of W 0: L .

Original: GPCN’s write obtains the readoutput x 0:
Revised: GPCN’s write obtains the readoutput x 0:

Original: L 1: T for each of T i.i.d.
Revised: L 1: T for each of T i.i.d.

Original: datapoints to store x 01: T , takes a gradient step on the energy w.r.t.
Revised: datapoints to store x 01: T , takes a gradient step on the energy w.r.t.

Original: W 0: L while fixing x 0:
Revised: W 0: L while fixing x 0:

Original: L 1: T , and repeats until convergence.
Revised: L 1: T , and repeats until convergence.

Original: We now take a probabilistic perspective on GPCN.
Revised: We now take a probabilistic perspective on GPCN.

Original: One can view GPCN as a hierarchical generativemodel where all neuron activations are modeled by normally distributed random variables and the linkfunction between the subsequent layers is a layer-specific nonlinearity followed by a linear transform.
Revised: One can view GPCN as a hierarchical generativemodel where all neuron activations are modeled by normally distributed random variables and the linkfunction between the subsequent layers is a layer-specific nonlinearity followed by a linear transform.

Original: Then, GPCN’s read - the minimization of Equation 1 - is equivalent to maximizing the log density in
Revised: Then, GPCN’s read - the minimization of Equation 1 - is equivalent to maximizing the log density in

Original: Equation 2 w.r.t.
Revised: Equation 2 w.r.t.

Original: x 0 and x 1: L .
Revised: x 0 and x 1: L .

Original: [Equation]
Revised: [Equation]

Original: In other words, GPCN’s read performs iterated conditional modes [Besag, 1986] on the hierarchicalgenerative model in the neuron activation space (refer to Algorithms 3 and 4).
Revised: In other words, GPCN’s read performs iterated conditional modes [Besag, 1986] on the hierarchicalgenerative model in the neuron activation space (refer to Algorithms 3 and 4).

Original: In addition, GPCN’swrite can be viewed as an approximate maximization of the lower bound on log p ( x 01: T | W 0: L ) w.r.t.
Revised: In addition, GPCN’swrite can be viewed as an approximate maximization of the lower bound on log p ( x 01: T | W 0: L ) w.r.t.

Original: W 0:
Revised: W 0:

Original: L [Millidge et al., 2021].
Revised: L [Millidge et al., 2021].

Original: This follows from the fact that GPCN learns via PC.
Revised: This follows from the fact that GPCN learns via PC.

Original: While GPCN possesses impressive associative recall capabilities, the learning rule proposed in
Revised: While GPCN possesses impressive associative recall capabilities, the learning rule proposed in

Original: Salvatori et al.
Revised: Salvatori et al.

Original: [2021] is not amenable to continual learning.
Revised: [2021] is not amenable to continual learning.

Original: This means write operations can onlyoccur offline and incorporating new datapoints requires costly model refitting as well as the need toretain all observed datapoints.
Revised: This means write operations can onlyoccur offline and incorporating new datapoints requires costly model refitting as well as the need toretain all observed datapoints.

Original: BayesPCN’s foremost aim is to address this limitation while preserving
Revised: BayesPCN’s foremost aim is to address this limitation while preserving

Original: GPCN’s read capabilities as much as possible.
Revised: GPCN’s read capabilities as much as possible.

Original:  Memory Model
Revised:  Memory Model

Original: BayesPCN is structurally identical to GPCN, but unlike GPCN the network parameters W 0: Lare treated as latent random variables.
Revised: BayesPCN is structurally identical to GPCN, but unlike GPCN the network parameters W 0: Lare treated as latent random variables.

Original: This allows BayesPCN’s write operation to be cast as
Revised: This allows BayesPCN’s write operation to be cast as

Original: (approximate) posterior inference over W 0: L given new observations in a way similar to [Wu et al.,2018a].
Revised: (approximate) posterior inference over W 0: L given new observations in a way similar to [Wu et al.,2018a].

Original: In other words, storing T i.i.d.
Revised: In other words, storing T i.i.d.

Original: datapoints x 01: T into memory is equivalent to estimatingp ( W 0: L | x 01: T ) .
Revised: datapoints x 01: T into memory is equivalent to estimatingp ( W 0: L | x 01: T ) .

Original: Figure 2 illustrates the graphical model for BayesPCN.
Revised: Figure 2 illustrates the graphical model for BayesPCN.

Original: We now review the probabilistic structure of BayesPCN.
Revised: We now review the probabilistic structure of BayesPCN.

Original: We assume that, prior to any observations,the random variables x 0: L t and W 0: L (where t is the datapoint index) are distributed as follows
Revised: We assume that, prior to any observations,the random variables x 0: L t and W 0: L (where t is the datapoint index) are distributed as follows

Original: [Equation]
Revised: [Equation]

Original: where N ( µ, Σ) and MN ( R, U, V ) denotes normal and matrix normal distribution [Gupta and Nagar,2018] respectively.
Revised: where N ( µ, Σ) and MN ( R, U, V ) denotes normal and matrix normal distribution [Gupta and Nagar,2018] respectively.

Original: We remind the readers that the matrix normal distribution is a special kindof normal distribution.
Revised: We remind the readers that the matrix normal distribution is a special kindof normal distribution.

Original: The joint distribution over all random variables when there are T sensoryobservations is, due to the conditional independence of x 0: L t , t ∈ { 1 , ..., T } given W 0: L ,
Revised: The joint distribution over all random variables when there are T sensoryobservations is, due to the conditional independence of x 0: L t , t ∈ { 1 , ..., T } given W 0: L ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We note that BayesPCN’s structure leads to two desirable properties that become useful whenapproximating p ( W 0:
Revised: We note that BayesPCN’s structure leads to two desirable properties that become useful whenapproximating p ( W 0:

Original: L | x 01: T ) during the write operation.
Revised: L | x 01: T ) during the write operation.

Original: First, by exploiting the linear Gaussianmarginalization formula and conditional independence we can analytically compute the marginaldistribution over all activation vectors x 0: L t .
Revised: First, by exploiting the linear Gaussianmarginalization formula and conditional independence we can analytically compute the marginaldistribution over all activation vectors x 0: L t .

Original: [Equation]
Revised: [Equation]

Original: We stress that neither p ( x 0 t ) nor p ( x 0 t | W 0: L ) has an analytic form due to the presence of nonlinearities.
Revised: We stress that neither p ( x 0 t ) nor p ( x 0 t | W 0: L ) has an analytic form due to the presence of nonlinearities.

Original: In addition, if all activation vectors x 0: L t are observed, the posterior over the model parametersbecomes normally distributed due to the linear Gaussian posterior formula and the fact that W 0: Lare conditionally independent given x 0: L t (the analytical posterior formulae for Equation 9 are in
Revised: In addition, if all activation vectors x 0: L t are observed, the posterior over the model parametersbecomes normally distributed due to the linear Gaussian posterior formula and the fact that W 0: Lare conditionally independent given x 0: L t (the analytical posterior formulae for Equation 9 are in

Original: Appendix C).
Revised: Appendix C).

Original: [Equation]
Revised: [Equation]

Original: R l ∗ , U l ∗ , µ ∗ , Σ ∗ denote the sufficient statistics of the layer-wise posterior distributions given x lt , x l +1 t .
Revised: R l ∗ , U l ∗ , µ ∗ , Σ ∗ denote the sufficient statistics of the layer-wise posterior distributions given x lt , x l +1 t .

Original:  Memory Operations
Revised:  Memory Operations

Original: This section contains the memory operations that BayesPCN supports: read , write , and forget .
Revised: This section contains the memory operations that BayesPCN supports: read , write , and forget .

Original: Weuse the shorthand h
Revised: Weuse the shorthand h

Original: = x 1: L to denote all hidden layer neuron activations and W = W 0: L to denoteall network parameters.
Revised: = x 1: L to denote all hidden layer neuron activations and W = W 0: L to denoteall network parameters.

Original: 4.1 Read
Revised: 4.1 Read

Original: The read operation takes the corrupted query vector ¯ x 0 and retrieves the memory’s best guess of theassociated original datapoint x 0 ∗ .
Revised: The read operation takes the corrupted query vector ¯ x 0 and retrieves the memory’s best guess of theassociated original datapoint x 0 ∗ .

Original: As in GPCN, this is done by climbing the memory’s log densitylog p ( x 0 , h | x 0 1: T ) w.r.t.
Revised: As in GPCN, this is done by climbing the memory’s log densitylog p ( x 0 , h | x 0 1: T ) w.r.t.

Original: the neuron activations x 0 , h .
Revised: the neuron activations x 0 , h .

Original: To clarify, x 0 , h is the current memory read stateand is not directly related to the past observations x 01: T .
Revised: To clarify, x 0 , h is the current memory read stateand is not directly related to the past observations x 01: T .

Original: BayesPCN’s auto-associative recall performsiterated conditional modes on log p ( x 0 , h | x 01: T ) w.r.t.
Revised: BayesPCN’s auto-associative recall performsiterated conditional modes on log p ( x 0 , h | x 01: T ) w.r.t.

Original: x 0 , h and BayesPCN’s hetero-associative recallperforms gradient ascent on log p ( x 0 , h | x 01: T ) w.r.t.
Revised: x 0 , h and BayesPCN’s hetero-associative recallperforms gradient ascent on log p ( x 0 , h | x 01: T ) w.r.t.

Original: x 0 , h while keeping the known values of x 0fixed.
Revised: x 0 , h while keeping the known values of x 0fixed.

Original: Algorithm 3 and Algorithm 4 in Appendix A, the recall procedures taken from Salvatori et al.[2021] and rewritten from the probabilistic perspective, describe this in more detail.
Revised: Algorithm 3 and Algorithm 4 in Appendix A, the recall procedures taken from Salvatori et al.[2021] and rewritten from the probabilistic perspective, describe this in more detail.

Original: 4.2 Write
Revised: 4.2 Write

Original: The write operations aims to take the previous timestep posterior over the model
Revised: The write operations aims to take the previous timestep posterior over the model

Original: parametersp ( W | x 01: t − 1 ) and the current timestep observation x 0 t to return the current timestep posterior overthe model parameters p
Revised: parametersp ( W | x 01: t − 1 ) and the current timestep observation x 0 t to return the current timestep posterior overthe model parameters p

Original: ( W | x 01: t ) .
Revised: ( W | x 01: t ) .

Original: Unfortunately, the analytical form of p ( W | x 01: t ) is unavailable dueto the presence of nonlinearities in BayesPCN so one must resort to approximate posterior inference.
Revised: Unfortunately, the analytical form of p ( W | x 01: t ) is unavailable dueto the presence of nonlinearities in BayesPCN so one must resort to approximate posterior inference.

Original: We propose a sequential importance sampling posterior inference algorithm for estimating p ( W | x 01: t )where the particles are high-dimensional Gaussian distributions.
Revised: We propose a sequential importance sampling posterior inference algorithm for estimating p ( W | x 01: t )where the particles are high-dimensional Gaussian distributions.

Original: Let N ∈ N be the number ofsequential importance sampling particles and n ∈ { 1 , ..., N } be the particle index.
Revised: Let N ∈ N be the number ofsequential importance sampling particles and n ∈ { 1 , ..., N } be the particle index.

Original: The t -th timestepposterior estimate ˆ p ( W | x 01: t ) is recursively represented as a weighted Gaussian mixture of theform ˆ p ( W | x 01: t ) = (cid:80)
Revised: The t -th timestepposterior estimate ˆ p ( W | x 01: t ) is recursively represented as a weighted Gaussian mixture of theform ˆ p ( W | x 01: t ) = (cid:80)

Original: Nn =1 ω ( n ) p ( n ) ( W | x 01: t , h ( n ) 1: t ) where ω ( n ) is the n -th particle’s importanceweight and p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is the n -th particle’s activation conditioned posterior describedin Equation 8.
Revised: Nn =1 ω ( n ) p ( n ) ( W | x 01: t , h ( n ) 1: t ) where ω ( n ) is the n -th particle’s importanceweight and p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is the n -th particle’s activation conditioned posterior describedin Equation 8.

Original: We note that each p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is efficiently characterised by the sufficientstatistics µ ( n ) , Σ ( n ) , R 0: L − 1 , ( n ) , U 0:
Revised: We note that each p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is efficiently characterised by the sufficientstatistics µ ( n ) , Σ ( n ) , R 0: L − 1 , ( n ) , U 0:

Original: L − 1 , ( n ) .
Revised: L − 1 , ( n ) .

Original: Algorithm 1 describes the write operation in detail (see
Revised: Algorithm 1 describes the write operation in detail (see

Original: Appendix B for its justification).
Revised: Appendix B for its justification).

Original: x
Revised: x

Original: t − 1 , h ( n )
Revised: t − 1 , h ( n )

Original: t − 1 ) and weight ω
Revised: t − 1 ) and weight ω

Original: n ) t −
Revised: n ) t −

Original: in ˆ p
Revised: in ˆ p

Original: W
Revised: W

Original: x
Revised: x

Original: t − 1 ) do 4:
Revised: t − 1 ) do 4:

Original: Fit the parameters of the current timestep variational distribution: 5: ϕ = argmin ϕ D KL (cid:16) q ( n ) ϕ
Revised: Fit the parameters of the current timestep variational distribution: 5: ϕ = argmin ϕ D KL (cid:16) q ( n ) ϕ

Original: ( h t )
Revised: ( h t )

Original: ||
Revised: ||

Original: ˆ p ( n ) ( x 0 t , h t | x 01: t − 1 ) (cid:17) 6:
Revised: ˆ p ( n ) ( x 0 t , h t | x 01: t − 1 ) (cid:17) 6:

Original: Sample h ( n ) t : h ( n ) t ∼ q ( n ) ϕ ( h t ) 7:
Revised: Sample h ( n ) t : h ( n ) t ∼ q ( n ) ϕ ( h t ) 7:

Original: Update the model via linear Gaussian update conditioned on x 0 t and h ( n ) t : 8: p ( n ) ( W | x 01: t , h ( n ) 1: t )
Revised: Update the model via linear Gaussian update conditioned on x 0 t and h ( n ) t : 8: p ( n ) ( W | x 01: t , h ( n ) 1: t )

Original: ∝ p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) p ( n ) ( x 0 t , h ( n ) t | W ) 9:
Revised: ∝ p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) p ( n ) ( x 0 t , h ( n ) t | W ) 9:

Original: Compute the unnormalized particle weight: ˆ ω ( n ) t = ω ( n ) t − 1 p ( n ) ( x 0 t , h ( n ) t | x 01: t − 1 ) q ( n ) ϕ
Revised: Compute the unnormalized particle weight: ˆ ω ( n ) t = ω ( n ) t − 1 p ( n ) ( x 0 t , h ( n ) t | x 01: t − 1 ) q ( n ) ϕ

Original: ( h ( n ) t ) 10: end for 11: Set ˆ p ( W | x 01: t ) =
Revised: ( h ( n ) t ) 10: end for 11: Set ˆ p ( W | x 01: t ) =

Original: (cid:80) Nn
Revised: (cid:80) Nn

Original: = ˆ ω ( n ) t (cid:80)
Revised: = ˆ ω ( n ) t (cid:80)

Original: Nn ′ =
Revised: Nn ′ =

Original: 1 ˆ ω ( n ′ )
Revised: 1 ˆ ω ( n ′ )

Original: t p ( n ) ( W | x 01: t , h ( n ) 1: t )
Revised: t p ( n ) ( W | x 01: t , h ( n ) 1: t )

Original: While q ( n ) ( h t ) in principle can be any distribution, BayesPCN’s write sets it to q ( n ) ( h t ) =
Revised: While q ( n ) ( h t ) in principle can be any distribution, BayesPCN’s write sets it to q ( n ) ( h t ) =

Original: δ (cid:0) argmax h t log p ( x 0 t , h t | x 0: L t ) (cid:1) as is standard in predictive coding [Millidge et al., 2021].
Revised: δ (cid:0) argmax h t log p ( x 0 t , h t | x 0: L t ) (cid:1) as is standard in predictive coding [Millidge et al., 2021].

Original: Inaddition, we define the BayesPCN parameter prior (of an “empty” memory) as ˆ p ( W ) = 140 (cid:80)
Revised: Inaddition, we define the BayesPCN parameter prior (of an “empty” memory) as ˆ p ( W ) = 140 (cid:80)

Original: Nn =1 1 N p ( n ) ( W ) where µ ( n ) , R 0:
Revised: Nn =1 1 N p ( n ) ( W ) where µ ( n ) , R 0:

Original: L − 1 , ( n ) are initialized using Kaiming initialization.
Revised: L − 1 , ( n ) are initialized using Kaiming initialization.

Original: We chose thisprior over a non-mixture prior to obtain a multi-particle posterior approximation despite using a Diracvariational distribution.
Revised: We chose thisprior over a non-mixture prior to obtain a multi-particle posterior approximation despite using a Diracvariational distribution.

Original: We now direct our attention to obtaining log p ( n ) ( x 0 t , h | x 01: t − 1 ) used in the read operations fromˆ p ( W | x 01: t ) .
Revised: We now direct our attention to obtaining log p ( n ) ( x 0 t , h | x 01: t − 1 ) used in the read operations fromˆ p ( W | x 01: t ) .

Original: Using Equation 6, we obtain the approximate posterior predictive density
Revised: Using Equation 6, we obtain the approximate posterior predictive density

Original: [Equation]
Revised: [Equation]

Original: BayesPCN’s write can be seen as a special case of sequential imputation [Kong et al., 1994], anonline parameter inference algorithm that represents the parameter posterior conditioned on partialobservations as a mixture of parameter posteriors conditioned on complete observations.
Revised: BayesPCN’s write can be seen as a special case of sequential imputation [Kong et al., 1994], anonline parameter inference algorithm that represents the parameter posterior conditioned on partialobservations as a mixture of parameter posteriors conditioned on complete observations.

Original: In addition,whereas the parameter update proposed by Salvatori et al.
Revised: In addition,whereas the parameter update proposed by Salvatori et al.

Original: [2021] is doing local gradient descent foreach W l , BayesPCN’s parameter update is doing local conjugate normal-normal Bayesian update foreach W l .
Revised: [2021] is doing local gradient descent foreach W l , BayesPCN’s parameter update is doing local conjugate normal-normal Bayesian update foreach W l .

Original: Hence, like GPCN, BayesPCN’s synaptic weight update only utilizes local information.
Revised: Hence, like GPCN, BayesPCN’s synaptic weight update only utilizes local information.

Original: Lastly, while exact Bayesian updates are invariant to the order of observations, because we are usingpredictive coding to fit the variational distribution
Revised: Lastly, while exact Bayesian updates are invariant to the order of observations, because we are usingpredictive coding to fit the variational distribution

Original: BayesPCN’s write is technically not invariant tothe order of observations.
Revised: BayesPCN’s write is technically not invariant tothe order of observations.

Original: 4.3 Forget
Revised: 4.3 Forget

Original: We also propose a diffusion-based forgetting mechanism for BayesPCN.
Revised: We also propose a diffusion-based forgetting mechanism for BayesPCN.

Original: As more datapoints arewritten into a BayesPCN model, the model’s belief over the true parameter location R 1: L − 1 , µshifts and the uncertainty over its belief U 1: L − 1 , Σ decreases.
Revised: As more datapoints arewritten into a BayesPCN model, the model’s belief over the true parameter location R 1: L − 1 , µshifts and the uncertainty over its belief U 1: L − 1 , Σ decreases.

Original: This has the effect of making themodel’s parameters less adaptive to new datapoints and magnifying R 1: L − 1 , µ ’s norms as the prior’scontribution weakens.
Revised: This has the effect of making themodel’s parameters less adaptive to new datapoints and magnifying R 1: L − 1 , µ ’s norms as the prior’scontribution weakens.

Original: One can interpret this as the memory being overloaded.
Revised: One can interpret this as the memory being overloaded.

Original: The forget operationcounteracts this phenomenon by nudging the model’s belief over R 1: L − 1 , µ toward the prior (emptymemory state) and increasing its uncertainty U 1: L − 1 , Σ proportional to the forget strength parameterβ .
Revised: The forget operationcounteracts this phenomenon by nudging the model’s belief over R 1: L − 1 , µ toward the prior (emptymemory state) and increasing its uncertainty U 1: L − 1 , Σ proportional to the forget strength parameterβ .

Original: Algorithm 2 describes this in detail (the analytical diffusion formulae are in Appendix D).
Revised: Algorithm 2 describes this in detail (the analytical diffusion formulae are in Appendix D).

Original: ) = (cid:82) p ( W l
Revised: ) = (cid:82) p ( W l

Original: | W l , β ) p ( n ) ( W l | x ( n ) ,l 1: t , x ( n ) ,l +1 1: t ) dW l 7: end for 8:
Revised: | W l , β ) p ( n ) ( W l | x ( n ) ,l 1: t , x ( n ) ,l +1 1: t ) dW l 7: end for 8:

Original: Set p ( n ) ( W ′ | x 01: t , h ( n ) 1: t ) =
Revised: Set p ( n ) ( W ′ | x 01: t , h ( n ) 1: t ) =

Original: (cid:81) Ll =

Revised: (cid:81) Ll =

Original: p ( n ) ( W l ′ | x ( n ) ,l 1: t , x ( n ) ,l +1 1: t ) 9: end for 10: Set
Revised: p ( n ) ( W l ′ | x ( n ) ,l 1: t , x ( n ) ,l +1 1: t ) 9: end for 10: Set

Original: ˆ p ( W ′ | x 01: t ) =
Revised: ˆ p ( W ′ | x 01: t ) =

Original: (cid:80) Nn =1 ω ( n ) p ( n ) ( W ′ | x 01: t , h ( n ) 1: t )
Revised: (cid:80) Nn =1 ω ( n ) p ( n ) ( W ′ | x 01: t , h ( n ) 1: t )

Original: If applied an infinite number of times, forget reverts the posterior over the parameters ˆ p
Revised: If applied an infinite number of times, forget reverts the posterior over the parameters ˆ p

Original: ( W | x 01: t ) tothe prior ˆ p ( W ) due to the property of diffusion if the particle weights are uniform.
Revised: ( W | x 01: t ) tothe prior ˆ p ( W ) due to the property of diffusion if the particle weights are uniform.

Original: In practice, onewould apply forget periodically or when the memory starts to become overloaded from too muchdata and stops being performant.
Revised: In practice, onewould apply forget periodically or when the memory starts to become overloaded from too muchdata and stops being performant.

Original: Because forget erases stored information for all of datapointssomewhat evenly, the datapoints from the distant past that forget was applied to more would be, asthe name suggests, more forgotten than the recent datapoints that forget was applied to less.
Revised: Because forget erases stored information for all of datapointssomewhat evenly, the datapoints from the distant past that forget was applied to more would be, asthe name suggests, more forgotten than the recent datapoints that forget was applied to less.

Original:  Related Work
Revised:  Related Work

Original: We briefly review the most relevant work to BayesPCN, starting with the predictive coding associativememories.
Revised: We briefly review the most relevant work to BayesPCN, starting with the predictive coding associativememories.

Original: As covered in Section 2, GPCN [Salvatori et al., 2021] is the model that BayesPCN is builtupon.
Revised: As covered in Section 2, GPCN [Salvatori et al., 2021] is the model that BayesPCN is builtupon.

Original: Self-taught associative memory [Dovrolis, 2018, Smith et al., 2021] is a hierarchical predictivecoding memory that performs unsupervised continual learning using online clustering modules asbuilding blocks, unlike BayesPCN’s building blocks of artificial neurons.
Revised: Self-taught associative memory [Dovrolis, 2018, Smith et al., 2021] is a hierarchical predictivecoding memory that performs unsupervised continual learning using online clustering modules asbuilding blocks, unlike BayesPCN’s building blocks of artificial neurons.

Original: The sequential neuralcoding network [Ororbia et al., 2019, 2018], while not explicitly used as an associative memory,is also a continually learned predictive coding network similar in structure to GPCN that combatscatastrophic forgetting via task-dependent activation sparsity.
Revised: The sequential neuralcoding network [Ororbia et al., 2019, 2018], while not explicitly used as an associative memory,is also a continually learned predictive coding network similar in structure to GPCN that combatscatastrophic forgetting via task-dependent activation sparsity.

Original: BayesPCN on the other hand combats catastrophic forgetting by incorporating uncertainty over the synaptic weights.
Revised: BayesPCN on the other hand combats catastrophic forgetting by incorporating uncertainty over the synaptic weights.

Original: Moving on to general attractor-based memory models, Hopfield networks [Hopfield, 1982] andtheir modern variants [Krotov and Hopfield, 2016, Ramsauer et al., 2020, Krotov and Hopfield,2020, Krotov, 2021, Millidge et al., 2022] are biologically motivated auto-associative memoriesthat define the neuron activation dynamics via Lagrangian functions.
Revised: Moving on to general attractor-based memory models, Hopfield networks [Hopfield, 1982] andtheir modern variants [Krotov and Hopfield, 2016, Ramsauer et al., 2020, Krotov and Hopfield,2020, Krotov, 2021, Millidge et al., 2022] are biologically motivated auto-associative memoriesthat define the neuron activation dynamics via Lagrangian functions.

Original: Out of these models only
Revised: Out of these models only

Original: Krotov [2021]’s model is hierarchical, but theirs is not continually learned.
Revised: Krotov [2021]’s model is hierarchical, but theirs is not continually learned.

Original: BayesPCN generalizesthe modern Hopfield network and fits into the universal Hopfield network’s similarity-separationprojection paradigm for associative memories (refer to Appendix E).
Revised: BayesPCN generalizesthe modern Hopfield network and fits into the universal Hopfield network’s similarity-separationprojection paradigm for associative memories (refer to Appendix E).

Original: Sparse Distributed Memory(SDM)
Revised: Sparse Distributed Memory(SDM)

Original: [Kanerva, 1988] is a non-hierarchical memory that exploits high-dimensionality to performauto-associative and hetero-associative recall.
Revised: [Kanerva, 1988] is a non-hierarchical memory that exploits high-dimensionality to performauto-associative and hetero-associative recall.

Original: Hierarchical Temporal Memory (HTM)
Revised: Hierarchical Temporal Memory (HTM)

Original: [Hawkins and
Revised: [Hawkins and

Original: George, 2007, Cui et al., 2016] is a biologically constrained hierarchical memory that continually learns time-based patterns with sparse representations.
Revised: George, 2007, Cui et al., 2016] is a biologically constrained hierarchical memory that continually learns time-based patterns with sparse representations.

Original: HTM does not employ predictive coding andto our knowledge has not yet been tested on high-dimensional data.
Revised: HTM does not employ predictive coding andto our knowledge has not yet been tested on high-dimensional data.

Original: Relative to generative modeling and memory research, our work has connections to the Kanervamachine’s memory write as inference approach [Wu et al., 2018a,b] and the one-shot memory writefrom Bartunov et al.
Revised: Relative to generative modeling and memory research, our work has connections to the Kanervamachine’s memory write as inference approach [Wu et al., 2018a,b] and the one-shot memory writefrom Bartunov et al.

Original: [2019], Pham et al.
Revised: [2019], Pham et al.

Original: These papers’ models employ meta-learning unlikeaforementioned work.
Revised: These papers’ models employ meta-learning unlikeaforementioned work.

Original: We note that the Kanerva machine also applies conjugate normal-normal
Revised: We note that the Kanerva machine also applies conjugate normal-normal

Original: Bayesian update to adapt its top level memory matrix once the model is meta-learned via end-to-endgradient descent.
Revised: Bayesian update to adapt its top level memory matrix once the model is meta-learned via end-to-endgradient descent.

Original: On the other hand, BayesPCN applies conjugate normal-normal Bayesian update toall of its weight matrices and this as of itself constitutes learning.
Revised: On the other hand, BayesPCN applies conjugate normal-normal Bayesian update toall of its weight matrices and this as of itself constitutes learning.

Original:  Experiments
Revised:  Experiments

Original: This section assesses BayesPCN’s recall capabilities on image recovery tasks and examines the effectof forget on BayesPCN’s behaviour.
Revised: This section assesses BayesPCN’s recall capabilities on image recovery tasks and examines the effectof forget on BayesPCN’s behaviour.

Original: We used CIFAR

Revised: We used CIFAR

Original: [Krizhevsky et al., 2009] and Tiny ImageNet [Le and Yang, 2015] datasets for allexperiments, whose images are of size 3 × 32 × 32 and 3 × 64 × 64 respectively.
Revised: [Krizhevsky et al., 2009] and Tiny ImageNet [Le and Yang, 2015] datasets for allexperiments, whose images are of size 3 × 32 × 32 and 3 × 64 × 64 respectively.

Original: During the writephase of the experiments, BayesPCN models were given one image to store into their memoriesper timestep for up to 1024 images.
Revised: During the writephase of the experiments, BayesPCN models were given one image to store into their memoriesper timestep for up to 1024 images.

Original: Then, during the read phase of the experiments, we recordedthe pixel-wise mean squared error (MSE) between the write phase images and the models’ readoutputs given the corrupted write phase images as inputs.
Revised: Then, during the read phase of the experiments, we recordedthe pixel-wise mean squared error (MSE) between the write phase images and the models’ readoutputs given the corrupted write phase images as inputs.

Original: For reference, the pixel-wise MSE of 0 .
Revised: For reference, the pixel-wise MSE of 0 .

Original: 01is
Revised: 01is

Original: distinguishable to human eyes on close inspection of the images.
Revised: distinguishable to human eyes on close inspection of the images.

Original: BayesPCN models were compared against identity models (models that return their inputs as outputs),modern Hopfield networks (MHN)
Revised: BayesPCN models were compared against identity models (models that return their inputs as outputs),modern Hopfield networks (MHN)

Original: [Ramsauer et al., 2020], and vanilla GPCNs [Salvatori et al.
Revised: [Ramsauer et al., 2020], and vanilla GPCNs [Salvatori et al.

Original: 2021]all of which were offline learned.
Revised: 2021]all of which were offline learned.

Original: BayesPCN models were also compared against online GPCNs,
Revised: BayesPCN models were also compared against online GPCNs,

Original: GPCNs that naively applied their offline memory write to sequentially arriving observations andmemorized one image per timestep for up to 1024 images.
Revised: GPCNs that naively applied their offline memory write to sequentially arriving observations andmemorized one image per timestep for up to 1024 images.

Original: We used our own autodiff implementationof MHNs based on Appendix E. See Appendix F for additional experiment details.
Revised: We used our own autodiff implementationof MHNs based on Appendix E. See Appendix F for additional experiment details.

Original: 6.1 Auto-Associative and Hetero-Associative Read
Revised: 6.1 Auto-Associative and Hetero-Associative Read

Original: For the auto-associative recall experiments, the memory inputs at the read phase were the write phaseimages corrupted by pixel-wise white noise with standard deviation 0 .
Revised: For the auto-associative recall experiments, the memory inputs at the read phase were the write phaseimages corrupted by pixel-wise white noise with standard deviation 0 .

Original: For the hetero-associativerecall experiments, we tested two types of training image corruption schemes.
Revised: For the hetero-associativerecall experiments, we tested two types of training image corruption schemes.

Original: The first type, dropout,randomly blacked out 25% of the image pixels.
Revised: The first type, dropout,randomly blacked out 25% of the image pixels.

Original: The second type, mask, blacked out 25% of therightmost image pixels.
Revised: The second type, mask, blacked out 25% of therightmost image pixels.

Original: Image recovery under these settings is hetero-associative recall becauseone can view the non-blacked out pixels as the “key” and the blacked out pixels as the “value”.
Revised: Image recovery under these settings is hetero-associative recall becauseone can view the non-blacked out pixels as the “key” and the blacked out pixels as the “value”.

Original: Weexperimented with 4-layer BayesPCN models that have hidden layer sizes of 256, 512, and 1024,particle counts of 1 and 4, and activation functions ReLU and GELU.
Revised: Weexperimented with 4-layer BayesPCN models that have hidden layer sizes of 256, 512, and 1024,particle counts of 1 and 4, and activation functions ReLU and GELU.

Original: We searched over the samehyperparameters for offline and online 4-layer GPCNs aside from the particle count.
Revised: We searched over the samehyperparameters for offline and online 4-layer GPCNs aside from the particle count.

Original: Lastly, weevaluated 4-layer BayesPCN models, which we denote as BayesPCN ( forget ), that applied forgetwith strength β = 0 .
Revised: Lastly, weevaluated 4-layer BayesPCN models, which we denote as BayesPCN ( forget ), that applied forgetwith strength β = 0 .

Original: 001 to their weights once every 64 write operations.
Revised: 001 to their weights once every 64 write operations.

Original: Table 1 shows the average MSE between the ground truth images and the associative memoryread outputs for models trained on data sequences of different lengths, where the reported scorescome from the best models of each category.
Revised: Table 1 shows the average MSE between the ground truth images and the associative memoryread outputs for models trained on data sequences of different lengths, where the reported scorescome from the best models of each category.

Original: The results were averaged across three random seeds.
Revised: The results were averaged across three random seeds.

Original: Appendix H contains additional images that demonstrate how BayesPCN’s read performance scaleswith different hyperparameters, how BayesPCN gets better at reconstructing unseen images the moredatapoints it observes, and shows images sampled from BayesPCN by using it as a generative model.
Revised: Appendix H contains additional images that demonstrate how BayesPCN’s read performance scaleswith different hyperparameters, how BayesPCN gets better at reconstructing unseen images the moredatapoints it observes, and shows images sampled from BayesPCN by using it as a generative model.

Original: We found that for sequences of length ≤ 512, BayesPCN’s recall errors were consistently lowerthan online GPCN’s recall errors and were similar to offline GPCN’s recall errors.
Revised: We found that for sequences of length ≤ 512, BayesPCN’s recall errors were consistently lowerthan online GPCN’s recall errors and were similar to offline GPCN’s recall errors.

Original: In particular,
Revised: In particular,

Original: BayesPCN’s hetero-associative recall errors were orders of magnitude lower than online GPCN’serrors.
Revised: BayesPCN’s hetero-associative recall errors were orders of magnitude lower than online GPCN’serrors.

Original: As expected, BayesPCN ( forget ) performed worse than BayesPCN in this regime becauseinformation is being forgotten before the memory is nearing its storage capacity.
Revised: As expected, BayesPCN ( forget ) performed worse than BayesPCN in this regime becauseinformation is being forgotten before the memory is nearing its storage capacity.

Original: However, when thesequence length approached 1024, BayesPCN’s recall errors deteriorated more rapidly than online
Revised: However, when thesequence length approached 1024, BayesPCN’s recall errors deteriorated more rapidly than online

Original: GPCN’s errors and on the TinyImageNet white noise task became even worse than the identity model.
Revised: GPCN’s errors and on the TinyImageNet white noise task became even worse than the identity model.

Original: However, BayesPCN ( forget )’s recall errors deteriorated more gradually and overall was able tooutperform online GPCN’s recall errors on all tasks and sequence lengths.
Revised: However, BayesPCN ( forget )’s recall errors deteriorated more gradually and overall was able tooutperform online GPCN’s recall errors on all tasks and sequence lengths.

Original: We hypothesize that
Revised: We hypothesize that

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] 
Revised: [Table] [Figure] [Table]

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: BayesPCN’s sudden performance drop stems from the fact that it is not “removing” old informationunlike online GPCN and BayesPCN ( forget ) due to conjugate Bayesian update’s observation orderinvariance property.
Revised: BayesPCN’s sudden performance drop stems from the fact that it is not “removing” old informationunlike online GPCN and BayesPCN ( forget ) due to conjugate Bayesian update’s observation orderinvariance property.

Original: This causes BayesPCN to approach the overloaded memory regime, where recallof all data suffers, more abruptly once some amount of information has been stored.
Revised: This causes BayesPCN to approach the overloaded memory regime, where recallof all data suffers, more abruptly once some amount of information has been stored.

Original: GPCN and BayesPCN models overall found hetero-associative recall easier than auto-associativerecall, likely because the former fixes the key bits during memory read and thus discourages thealgorithm from wandering to less realistic x 0 values.
Revised: GPCN and BayesPCN models overall found hetero-associative recall easier than auto-associativerecall, likely because the former fixes the key bits during memory read and thus discourages thealgorithm from wandering to less realistic x 0 values.

Original: Counterintuitively, GPCN and BayesPCNmodels were better at Tiny ImageNet recall over CIFAR10 recall.
Revised: Counterintuitively, GPCN and BayesPCNmodels were better at Tiny ImageNet recall over CIFAR10 recall.

Original: We attribute this behaviour to thefact that for some models, high-dimensional data recall can ironically be easier than low-dimensionaldata recall because individual datapoints are further apart from one another in higher dimensions.
Revised: We attribute this behaviour to thefact that for some models, high-dimensional data recall can ironically be easier than low-dimensionaldata recall because individual datapoints are further apart from one another in higher dimensions.

Original: A surprising discovery from this experiment was how good MHN’s recall can be compared to GPCNand BayesPCN’s recall when the memory query has a moderate amount of noise.
Revised: A surprising discovery from this experiment was how good MHN’s recall can be compared to GPCNand BayesPCN’s recall when the memory query has a moderate amount of noise.

Original: Because MHNexplicitly retains past observations in a data matrix and its softmax inverse temperature was set veryhigh ( β = 10 , 000 ), MHN’s recall effectively selected the observation that had the highest dot-productsimilarity to the memory query in our experiments.
Revised: Because MHNexplicitly retains past observations in a data matrix and its softmax inverse temperature was set veryhigh ( β = 10 , 000 ), MHN’s recall effectively selected the observation that had the highest dot-productsimilarity to the memory query in our experiments.

Original: In Appendix G, we assess how MHN’s recallcompares to BayesPCN’s recall when the memory query corruption is much higher.
Revised: In Appendix G, we assess how MHN’s recallcompares to BayesPCN’s recall when the memory query corruption is much higher.

Original: As expected,because the observation with the highest dot-product similarity to the memory query is often not theright image to recall, BayesPCN significantly outperformed MHN in hetero-associative recall tasks.
Revised: As expected,because the observation with the highest dot-product similarity to the memory query is often not theright image to recall, BayesPCN significantly outperformed MHN in hetero-associative recall tasks.

Original: 6.2 Forgetting
Revised: 6.2 Forgetting

Original: This section investigates the forget operation’s effect on BayesPCN’s recall behaviour.
Revised: This section investigates the forget operation’s effect on BayesPCN’s recall behaviour.

Original: Specifically,we show that forget causes BayesPCN to better recall recent observations at the expense of olderobservations and that this effect is more pronounced the higher the forget strength is.
Revised: Specifically,we show that forget causes BayesPCN to better recall recent observations at the expense of olderobservations and that this effect is more pronounced the higher the forget strength is.

Original: We trainedsix structurally identical BayesPCN models on 1024 CIFAR10 images sequentially and appliedforget with strength β = 0 , 0 .
Revised: We trainedsix structurally identical BayesPCN models on 1024 CIFAR10 images sequentially and appliedforget with strength β = 0 , 0 .

Original: 02 to each model once after every 64 writeoperations.
Revised: 02 to each model once after every 64 writeoperations.

Original: Figure 3 compares the models’ recall MSE on 16 data batches of size 64 that were observed duringdifferent points of the training.
Revised: Figure 3 compares the models’ recall MSE on 16 data batches of size 64 that were observed duringdifferent points of the training.

Original: We see that BayesPCN with no forget ( β = 0) performs similarly onall datapoints regardless of their observation timestep, which is likely the consequence of conjugate
Revised: We see that BayesPCN with no forget ( β = 0) performs similarly onall datapoints regardless of their observation timestep, which is likely the consequence of conjugate

Original: Bayesian update’s observation order invariance.
Revised: Bayesian update’s observation order invariance.

Original: On the other hand, BayesPCNs with forget ( β > 0)consistently perform better on newer datapoints with this effect being strongly correlated with thevalue of β .
Revised: On the other hand, BayesPCNs with forget ( β > 0)consistently perform better on newer datapoints with this effect being strongly correlated with thevalue of β .

Original: As previously shown, some amount of forgetting during training can improve the overallperformance of the memory across all datapoints over no forgetting for long training sequences.
Revised: As previously shown, some amount of forgetting during training can improve the overallperformance of the memory across all datapoints over no forgetting for long training sequences.

Original:  Discussion
Revised:  Discussion

Original: One can intuitively think of BayesPCN read as an approximate nearest neighbour operation with alearned similarity metric and write as the continual learning of that metric.
Revised: One can intuitively think of BayesPCN read as an approximate nearest neighbour operation with alearned similarity metric and write as the continual learning of that metric.

Original: MHN on the other handuses the dot-product distance as its similarity metric [Millidge et al., 2022].
Revised: MHN on the other handuses the dot-product distance as its similarity metric [Millidge et al., 2022].

Original: This explains why MHNis not good at highly masked image recall because from a pixelwise dot-product distance perspective,highly masked image queries are often closer to unrelated images than to the ground truth images.
Revised: This explains why MHNis not good at highly masked image recall because from a pixelwise dot-product distance perspective,highly masked image queries are often closer to unrelated images than to the ground truth images.

Original: BayesPCN is closely related to the predictive coding theories of the brain’s memory system.
Revised: BayesPCN is closely related to the predictive coding theories of the brain’s memory system.

Original: Forexample, Barron et al.
Revised: Forexample, Barron et al.

Original: [2020] suggests that the hippocampus may correspond to the top layer of apredictive coding network and the neo-cortical areas may correspond to the rest of the predictivecoding network.
Revised: [2020] suggests that the hippocampus may correspond to the top layer of apredictive coding network and the neo-cortical areas may correspond to the rest of the predictivecoding network.

Original: Similarly, Dovrolis [2018] posits that their predictive coding network could serveas a model of the cortical column albeit without experimental results.
Revised: Similarly, Dovrolis [2018] posits that their predictive coding network could serveas a model of the cortical column albeit without experimental results.

Original: It is however unclear how
Revised: It is however unclear how

Original: BayesPCN’s locally conjugate Bayesian weight update can be realized in the brain unlike GPCN’s
Revised: BayesPCN’s locally conjugate Bayesian weight update can be realized in the brain unlike GPCN’s

Original: Hebbian weight update.
Revised: Hebbian weight update.

Original: Nevertheless, given that conjugate normal-normal Bayesian update’s posteriormean formula (Equation 27) can be seen as a preconditioned Hebbian weight update influenced by theinput and error neurons, there may be a biologically plausible way of approximating single particle
Revised: Nevertheless, given that conjugate normal-normal Bayesian update’s posteriormean formula (Equation 27) can be seen as a preconditioned Hebbian weight update influenced by theinput and error neurons, there may be a biologically plausible way of approximating single particle

Original: BayesPCN write .
Revised: BayesPCN write .

Original: In addition, there are similarities between BayesPCN’s diffusion-based forgetand the synaptic homeostasis hypothesis [Cirelli and Tononi, 2022], where during sleep synapticweight strengths are decreased on average to facilitate future learning.
Revised: In addition, there are similarities between BayesPCN’s diffusion-based forgetand the synaptic homeostasis hypothesis [Cirelli and Tononi, 2022], where during sleep synapticweight strengths are decreased on average to facilitate future learning.

Original: Aside from these biological connections, BayesPCN has many potential machine learning applications.
Revised: Aside from these biological connections, BayesPCN has many potential machine learning applications.

Original: One possible line of work is using it in conjunction with a neural controller that learns whatinformation to store into memory [Graves et al., 2014, Schlag et al., 2021].
Revised: One possible line of work is using it in conjunction with a neural controller that learns whatinformation to store into memory [Graves et al., 2014, Schlag et al., 2021].

Original: Using BayesPCN in thisway would confer the benefit of high query noise tolerance for neural controllers.
Revised: Using BayesPCN in thisway would confer the benefit of high query noise tolerance for neural controllers.

Original: Another line ofwork is using it to solve small data supervised learning problems, leveraging the fact that nearestneighbour methods work well when data availability is sparse [Ramsauer et al., 2020].
Revised: Another line ofwork is using it to solve small data supervised learning problems, leveraging the fact that nearestneighbour methods work well when data availability is sparse [Ramsauer et al., 2020].

Original: To conclude, we introduced BayesPCN, an associative memory model that can robustly retrieve pastobservations given various corrupted inputs, perform one-shot parameter update of a deep neuralnetwork to store new observations as attractors, and forget old observations to better store newer ones.
Revised: To conclude, we introduced BayesPCN, an associative memory model that can robustly retrieve pastobservations given various corrupted inputs, perform one-shot parameter update of a deep neuralnetwork to store new observations as attractors, and forget old observations to better store newer ones.

Original: We demonstrated that BayesPCN’s recall performance is comparable to its offline-learned counterparteven though it is continually learned.
Revised: We demonstrated that BayesPCN’s recall performance is comparable to its offline-learned counterparteven though it is continually learned.

Original: There are several intriguing extensions to BayesPCN that mayaddress some of its current limitations.
Revised: There are several intriguing extensions to BayesPCN that mayaddress some of its current limitations.

Original: As of now, BayesPCN read can take up to thousands ofactivation gradient descent iterations to obtain the best possible recall result when a lot of data hasbeen stored into memory.
Revised: As of now, BayesPCN read can take up to thousands ofactivation gradient descent iterations to obtain the best possible recall result when a lot of data hasbeen stored into memory.

Original: Supplementing BayesPCN read with amortized inference may greatlyreduce this computational requirement [Tschantz et al., 2022].
Revised: Supplementing BayesPCN read with amortized inference may greatlyreduce this computational requirement [Tschantz et al., 2022].

Original: In addition, BayesPCN read canreturn a nonsensical output if the memory query is too far away from the stored datapoints or if thememory is overloaded.
Revised: In addition, BayesPCN read canreturn a nonsensical output if the memory query is too far away from the stored datapoints or if thememory is overloaded.

Original: Investigating BayesPCN’s latent landscape may allow us design models withmore robust basin of attractions around the stored datapoints.
Revised: Investigating BayesPCN’s latent landscape may allow us design models withmore robust basin of attractions around the stored datapoints.

Original: Last but not least, extending BayesPCNbeyond memorizing i.i.d.
Revised: Last but not least, extending BayesPCNbeyond memorizing i.i.d.

Original: data to memorizing sequences like the human episodic memory would beof great interest.
Revised: data to memorizing sequences like the human episodic memory would beof great interest.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We mention that our memory can
Revised: [Yes] We mention that our memory can

Original: only store i.i.d.
Revised: only store i.i.d.

Original: data, questions of its biological plausibility compared to GPCN, andhow its performance rapidly declines relative to the baselines for longer sequences.(c)
Revised: data, questions of its biological plausibility compared to GPCN, andhow its performance rapidly declines relative to the baselines for longer sequences.(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We will releasethe code under the MIT License.(b)
Revised: [Yes] We will releasethe code under the MIT License.(b)

Original: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] Refer to Section 6 and Appendix F.
Revised: [Yes] Refer to Section 6 and Appendix F.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We did in Appendix F.
Revised: [Yes] We did in Appendix F.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?

Original: [No] We did not measure the totalcompute amount but listed the GPU type and the compute clusters used in Appendix F.4.
Revised: [No] We did not measure the totalcompute amount but listed the GPU type and the compute clusters used in Appendix F.4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Refer
Revised: [Yes] Refer

Original: to
Revised: to

Original: Section
Revised: Section

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Refer to Appendix F.
Revised: [Yes] Refer to Appendix F.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If
Revised: If

Original: you
Revised: you

Original: used crowdsourcing
Revised: used crowdsourcing

Original: or conducted research with human subjects.
Revised: or conducted research with human subjects.

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: B Derivation of BayesPCN write
Revised: B Derivation of BayesPCN write

Original: p ( W | x 01: t ) = 1 Z p ( W | x 01: t − 1 ) p ( x 0 t | W ) (12) =
Revised: p ( W | x 01: t ) = 1 Z p ( W | x 01: t − 1 ) p ( x 0 t | W ) (12) =

Original: Z p ( W | x 01: t − 1 ) E q ( h t )
Revised: Z p ( W | x 01: t − 1 ) E q ( h t )

Original: [ p ( x 0 t , h t | W ) q ( h t ) ] (13) ≈
Revised: [ p ( x 0 t , h t | W ) q ( h t ) ] (13) ≈

Original: Z [
Revised: Z [

Original: N (cid:88) n = ω ( n ) t − 1 p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 )]
Revised: N (cid:88) n = ω ( n ) t − 1 p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 )]

Original: E q ( h t )
Revised: E q ( h t )

Original: [ p ( x 0 t , h t | W ) q ( h t ) ] (14) =
Revised: [ p ( x 0 t , h t | W ) q ( h t ) ] (14) =

Original: Z
Revised: Z

Original: N (cid:88) n = ω ( n
Revised: N (cid:88) n = ω ( n

Original: [Equation]
Revised: [Equation]

Original: Equation 17 holds because
Revised: Equation 17 holds because

Original: [Equation]
Revised: [Equation]

Original: and Equation 18 holds because
Revised: and Equation 18 holds because

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: if S = 1 we get
Revised: if S = 1 we get

Original: [Equation]
Revised: [Equation]

Original: C Analytical Posterior Formulae for write
Revised: C Analytical Posterior Formulae for write

Original: Let z
Revised: Let z

Original: = f ( x ) .
Revised: = f ( x ) .

Original: Then, BayesPCN’s top layer parameter update given x 0: L is
Revised: Then, BayesPCN’s top layer parameter update given x 0: L is

Original: [Equation]
Revised: [Equation]

Original: 26) while the update for all other layers given x 0: L is
Revised: 26) while the update for all other layers given x 0: L is

Original: [Equation]
Revised: [Equation]

Original: D Analytical Diffusion Formulae for forget
Revised: D Analytical Diffusion Formulae for forget

Original: Let R 1: L − 1, U 1: L − 1, µ 0 , Σ 0 be the memory prior parameters and β be the forget strength.
Revised: Let R 1: L − 1, U 1: L − 1, µ 0 , Σ 0 be the memory prior parameters and β be the forget strength.

Original: Then,
Revised: Then,

Original: BayesPCN’s top layer diffusion update is
Revised: BayesPCN’s top layer diffusion update is

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: while the update for all other layers is
Revised: while the update for all other layers is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: E Connections to Hopfield Networks
Revised: E Connections to Hopfield Networks

Original: We show that modern Hopfield network’s recall is equivalent to the recall of a BayesPCN model with
Revised: We show that modern Hopfield network’s recall is equivalent to the recall of a BayesPCN model with

Original: L = 0 .
Revised: L = 0 .

Original: BayesPCN’s activation log density is defined as log p ( x 0 , h ) =
Revised: BayesPCN’s activation log density is defined as log p ( x 0 , h ) =

Original: log (cid:16)(cid:80) N n =1 ω ( n ) p ( n ) ( x 0 , h ) (cid:17) ,and its gradient w.r.t.
Revised: log (cid:16)(cid:80) N n =1 ω ( n ) p ( n ) ( x 0 , h ) (cid:17) ,and its gradient w.r.t.

Original: the activations is∇ x 0 , h log p ( x 0 , h ) =
Revised: the activations is∇ x 0 , h log p ( x 0 , h ) =

Original: N (cid:88) n =
Revised: N (cid:88) n =

Original: [Equation]
Revised: [Equation]

Original: exp(log
Revised: exp(log

Original: [Equation]
Revised: [Equation]

Original: ( n
Revised: ( n

Original: [Equation]
Revised: [Equation]

Original: p ( n
Revised: p ( n

Original: [Equation]
Revised: [Equation]

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: , h ))
Revised: , h ))

Original: (cid:80) Nn
Revised: (cid:80) Nn

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: h )) ∇ x
Revised: h )) ∇ x

Original: [Equation]
Revised: [Equation]

Original: h log p ( n ) ( x
Revised: h log p ( n ) ( x

Original: [Equation]
Revised: [Equation]

Original: h )
Revised: h )

Original: [Equation]
Revised: [Equation]

Original: On the other hand, MHN’s energy is
Revised: On the other hand, MHN’s energy is

Original: [Equation]
Revised: [Equation]

Original: where q ∈ R 1 × d k is the query row vector, K ∈ R N × d k is the key matrix, and K j ∈ R 1 × d k is thej-th key row vector.
Revised: where q ∈ R 1 × d k is the query row vector, K ∈ R N × d k is the key matrix, and K j ∈ R 1 × d k is thej-th key row vector.

Original: q and K correspond to x 0 and W 0 in our paper’s notation.
Revised: q and K correspond to x 0 and W 0 in our paper’s notation.

Original: The negative of theenergy can be converted to the following Gaussian mixture log density.
Revised: The negative of theenergy can be converted to the following Gaussian mixture log density.

Original: [Equation]
Revised: [Equation]

Original: Recall in both BayesPCN and MHN is gradient ascent on the above log density.
Revised: Recall in both BayesPCN and MHN is gradient ascent on the above log density.

Original: When we take thegradient with respect to the input vector q , we recover Equation 33.
Revised: When we take thegradient with respect to the input vector q , we recover Equation 33.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ■
Revised: ■

Original: We conclude that recall in Modern Hopfield Network is equivalent to recall under our framework,which is gradient descent on the log joint of a normal mixture w.r.t.
Revised: We conclude that recall in Modern Hopfield Network is equivalent to recall under our framework,which is gradient descent on the log joint of a normal mixture w.r.t.

Original: neuron activations, where thereare no hidden layers ( L = 0 ).
Revised: neuron activations, where thereare no hidden layers ( L = 0 ).

Original: However training our model with L = 0 does not lead to the samememory update as the suggested training procedure of MHN, which is setting each key vector K j tosome observed datapoint.
Revised: However training our model with L = 0 does not lead to the samememory update as the suggested training procedure of MHN, which is setting each key vector K j tosome observed datapoint.

Original: Universal Hopfield network [Millidge et al., 2022] proposes a framework for single-shot associativememory that decomposes recall into three components: similarity function, separation function, andprojection matrix.
Revised: Universal Hopfield network [Millidge et al., 2022] proposes a framework for single-shot associativememory that decomposes recall into three components: similarity function, separation function, andprojection matrix.

Original: Let x =
Revised: Let x =

Original: (¯ x 0 , h ) be the row vector of all initial network activations when give thequery ¯ x 0 .
Revised: (¯ x 0 , h ) be the row vector of all initial network activations when give thequery ¯ x 0 .

Original: Equation 33 suggests that BayesPCN read ’s implementation of those components is perparticle weighted log joint log ω ( n ) p ( n ) ( x ) of the query ¯ x 0 for the similarity function, softmax forthe separation function, and the matrix
Revised: Equation 33 suggests that BayesPCN read ’s implementation of those components is perparticle weighted log joint log ω ( n ) p ( n ) ( x ) of the query ¯ x 0 for the similarity function, softmax forthe separation function, and the matrix

Original: [Equation]
Revised: [Equation]

Original:  for the projection matrix where γis the learning rate.
Revised:  for the projection matrix where γis the learning rate.

Original: We can accommodate the fact that BayesPCN’s read does iterated conditionalmodes by zeroing out the fixed variable gradients in the projection matrix.
Revised: We can accommodate the fact that BayesPCN’s read does iterated conditionalmodes by zeroing out the fixed variable gradients in the projection matrix.

Original: F Additional Experiment Details
Revised: F Additional Experiment Details

Original: All GPCN and BayesPCN models had σ W = 1 , σ x = 0 . 01 , and used Adam with learning rate0 .
Revised: All GPCN and BayesPCN models had σ W = 1 , σ x = 0 . 01 , and used Adam with learning rate0 .

Original: 01 as the neuron activation gradient descent optimizer.
Revised: 01 as the neuron activation gradient descent optimizer.

Original: All energy minimization w.r.t.
Revised: All energy minimization w.r.t.

Original: the neuronactivations took 500 gradient steps during the write phase.
Revised: the neuronactivations took 500 gradient steps during the write phase.

Original: During the read phase, the outer loopiterated conditional mode in Algorithm 3 was repeated 30 times and the energy minimization in
Revised: During the read phase, the outer loopiterated conditional mode in Algorithm 3 was repeated 30 times and the energy minimization in

Original: Algorithm 4 took 30 × 500 gradient steps.
Revised: Algorithm 4 took 30 × 500 gradient steps.

Original: MHN models had β = 10 , 000 (equivalent to σ x = 0 . 01 )and used Adam with learning rate 1 . 

Revised: MHN models had β = 10 , 000 (equivalent to σ x = 0 . 01 ),used Adam with learning rate 1 .

Original: as the gradient descent optimizer. All experiments were run on CIFAR10 and/or Tiny ImageNet datasets (both of which have the MIT License) and the image pixelvalues were normalized to fall between [ − 1 , 1] .
Revised: as the gradient descent optimizer, and performed gradient-descentbased recall similar to BayesPCN based on the connection from Appendix E. All experiments wererun on CIFAR10 and/or Tiny ImageNet datasets (both of which have the MIT License) and theimage pixel values were normalized to fall between [ − 1 , 1] .

Original: Online GPCNs took a single gradient step w.r.t.
Revised: Online GPCNs took a single gradientstep w.r.t.

Original: thenetwork weights after the hidden activations converged per observation, a treatment consistent withthat of the fast weight memory in Schlag et al.
Revised: the network weights after the hidden activations converged per observation, a treatmentconsistent with that of the fast weight memory in Schlag et al.

Original: BayesPCN forget models had four hiddenlayers of width 1024, a single particle, and GELU activations.
Revised: BayesPCN forget modelshad four hidden layers of width 1024, a single particle, and GELU activations.

Original: All hyperparameter ranges were chosenbased on Salvatori et al.
Revised: All hyperparameterranges were chosen based on Salvatori et al.

Original: [2021]’s experiments and GPCN/BayesPCN’s empirical results.
Revised: [2021]’s experiments and GPCN/BayesPCN’s empiricalresults.

Original: Lastly,experiments used NVIDIA Tesla V100 GPUs and were run on the university’s internal clusters.
Revised: Lastly, experiments used NVIDIA Tesla V100 GPUs and were run on the university’s internalclusters.

Original: [Table]
Revised: [Table]

Original: G BayesPCN vs MHN Recall on Highly Noised Queries
Revised: G BayesPCN vs MHN Recall on Highly Noised Queries

Original: This section reports the recall performance of MHN and BayesPCN models on high query noiseassociative recall tasks.
Revised: This section reports the recall performance of MHN and BayesPCN models on high query noiseassociative recall tasks.

Original: The experiment setup is similar to that of Section 6 aside from the fact thatthe white noise tasks had the noise standard deviation set to 0 .
Revised: The experiment setup is similar to that of Section 6 aside from the fact thatthe white noise tasks had the noise standard deviation set to 0 .

Original: 8 instead of 0 .
Revised: 8 instead of 0 .

Original: 2 , the dropout tasksrandomly blacked out 75% of the pixels instead of 25% , and the masking tasks blacked out 75% ofthe rightmost pixels instead of 25% .
Revised: 2 , the dropout tasksrandomly blacked out 75% of the pixels instead of 25% , and the masking tasks blacked out 75% ofthe rightmost pixels instead of 25% .

Original: Because the high query noise task is harder, we show the recallresult after the models observed 16, 32, 64, and 128 datapoints.
Revised: Because the high query noise task is harder, we show the recallresult after the models observed 16, 32, 64, and 128 datapoints.

Original: BayesPCN models had four hiddenlayers of width 256, a single particle, and GELU activations.
Revised: BayesPCN models had four hiddenlayers of width 256, a single particle, and GELU activations.

Original: MHNs again used β = 10 , 000 .
Revised: MHNs again used β = 10 , 000 .

Original: [Table]
Revised: [Table]

Original: H BayesPCN Scaling, Generalization, and Sampling
Revised: H BayesPCN Scaling, Generalization, and Sampling

Revised: [Figure]

Original: H.1 Scaling
Revised: H.1 Scaling

Original:  Figure 5 qualitatively demonstrates how BayesPCN’s read scales with the number of stored datapointsfor the CIFAR10 recall tasks.
Revised: [Figure] Figure 5 qualitatively demonstrates how BayesPCN’s read scales with the number of stored datapointsfor the CIFAR10 recall tasks.

Original: BayesPCN models are able to output images that are very close tothe original image even when the inputs are significantly corrupted.
Revised: BayesPCN models are able to output images that are very close tothe original image even when the inputs are significantly corrupted.

Original: As the number of observationsincreases, the read operation’s basin of attraction decreases as it is still able to reconstruct the originalimage but gets worse at recovering the original image given a corrupted version of it.
Revised: As the number of observationsincreases, the read operation’s basin of attraction decreases as it is still able to reconstruct the originalimage but gets worse at recovering the original image given a corrupted version of it.

Original: Figure 6 illustrates how BayesPCN’s recall accuracy scales with the network width and depth.
Revised: Figure 6 illustrates how BayesPCN’s recall accuracy scales with the network width and depth.

Original: Arecall is considered correct if the MSE between the ground truth and the recalled data is less than0 .
Revised: Arecall is considered correct if the MSE between the ground truth and the recalled data is less than0 .

Original: We find that the increased network width and depth is generally helpful across all tasks.
Revised: We find that the increased network width and depth is generally helpful across all tasks.

Original: Wenote that because the plots depict recall accuracy not MSE, if the memory performance generallydeclines and the average recall MSE exceeds 0 .
Revised: Wenote that because the plots depict recall accuracy not MSE, if the memory performance generallydeclines and the average recall MSE exceeds 0 .

Original: 01 , this can lead to very low accuracy even if theactual recall MSE is not much greater than 0 .
Revised: 01 , this can lead to very low accuracy even if theactual recall MSE is not much greater than 0 .

Original: [Figure]
Revised: We also investigate the effect of σ W and σ x hyperparameters on BayesPCN’s scaling properties. Since σ W determines the prior uncertainty and σ x determines the observation noise, higher σ Wand lower σ x reduces the prior’s impact and increases the new observation’s impact on the networkweight’s posterior. Hence, we can control the memory write strength by modulating σ W , σ x . Table 4 describes the CIFAR10 recall results of nine structurally identical BayesPCN models withfour hidden layers of size 1024, a single particle, and GELU activations but with different valuesof σ W and σ x .

Revised: We observe that lower σ W and higher σ x tend to alleviate the memory overloadingbehaviour, where recall MSE explodes for longer data sequences due to more and more informationbeing written into memory without forgetting.

Original: .2 Generalization
Revised: We hypothesize that this is so because those choicesencourage synaptic weights’ Frobenius norms to remain small. As an aside, the white noise recall MSE of BayesPCN with σ W = 0 . 5 , σ x = 0 . 05 decreased as moredatapoints were observed from sequence length 64 and onward.

Revised: On visual inspection, we foundthat the model’s auto-associative recall outputs for both observed and unobserved inputs becameless blurry as more datapoints were written into memory.

Revised: We hypothesize that the model learned togeneralize at some point of its training.

Revised: [Table]

Revised: Lastly, we investigate whether a very deep BayesPCN model can perform better on moderate noiserecall tasks compared to the 4 layer BayesPCN model we used in our experiments.

Revised: Table 5 detailsthe recall MSE of 4 layer BayesPCN (BayesPCN L4) and 8 layer BayesPCN (BayesPCN L8), bothwith a single particle, hidden layer width of

Original: [Figure]
Revised: [Table]

Revised: and GELU activation function.

Revised: We find that theirperformances are almost identical, which shows that having more than 4 layers does not help much inthe moderate noise associative recall task.

Revised: However, having less than 4 layers appears to adverselyaffect the recall performance according to Figure

Original:  Figure 7 illustrates BayesPCN’s read outputs for unseen image queries after different number ofdatapoints have been stored into memory.
Revised: [Table] H.2 Generalization Figure 7 illustrates BayesPCN’s read outputs for unseen image queries after different number ofdatapoints have been stored into memory.

Original: As BayesPCN observes more data, it learns to “generalize”and gets better at reconstructing and even mildly removing white noise from unseen images.
Revised: As BayesPCN observes more data, it learns to “generalize”and gets better at reconstructing and even mildly removing white noise from unseen images.

Original: Thiscan be attributed to the model continual learning its internal representation that better “describe” thedata distribution.
Revised: Thiscan be attributed to the model continual learning its internal representation that better “describe” thedata distribution.

Original: We expected this behaviour to occur since S-NCN [Ororbia et al., 2019], a modelsimilar in structure to GPCN, could continually learn to perform discriminative tasks.[Figure]
Revised: We expected this behaviour to occur since S-NCN [Ororbia et al., 2019], a modelsimilar in structure to GPCN, could continually learn to perform discriminative tasks.

Original: H.3 Sampling
Revised: H.3 Sampling

Original: [Figure]

Original: Both GPCN and BayesPCN at the core are as much generative models as they are associativememories.
Revised: Both GPCN and BayesPCN at the core are as much generative models as they are associativememories.

Original: We examine the quality of ancestral sampling samples from both models in Figure 8.
Revised: We examine the quality of ancestral sampling samples from both models in Figure 8.

Original: When ancestral sampling, BayesPCN did not marginalize out the synaptic weights and instead fixedthem to the mean parameters of p ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) .
Revised: When ancestral sampling, BayesPCN did not marginalize out the synaptic weights and instead fixedthem to the mean parameters of p ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) .

Original: [Figure]

Original: We find that both GPCN and BayesPCN samples are superpositions of the training images.
Revised: We find that both GPCN and BayesPCN samples are superpositions of the training images.

Original: However,as BayesPCN is trained on more and more observations, its sample quality quickly deteriorates.
Revised: However,as BayesPCN is trained on more and more observations, its sample quality quickly deteriorates.

Original: Wehypothesize that the poor sample quality for both GPCN and BayesPCN stems from the approximatenature of their parameter estimation.
Revised: Wehypothesize that the poor sample quality for both GPCN and BayesPCN stems from the approximate [Figure] nature of their parameter estimation.

Original: For example, BayesPCN’s particle count would have to be muchgreater than 4 to accurately capture the true posterior p ( W | x 01: t − 1 ) using its sequential importancesampling estimate and the variational distribution over the hidden activations should not be Diracdistributed.
Revised: For example, BayesPCN’s particle count would have to be muchgreater than 4 to accurately capture the true posterior p ( W | x 01: t − 1 ) using its sequential importancesampling estimate and the variational distribution over the hidden activations should not be Diracdistributed.

Original: However, we note that Ororbia and Kifer [2022] has successfully trained predictivecoding networks to be good generative models.
Revised: However, we note that Ororbia and Kifer [2022] has successfully trained predictivecoding networks to be good generative models.