Original: BayesPCN: A Continually Learnable Predictive Coding Associative Memory
Revised: BayesPCN: A Continually Learnable Predictive Coding Associative Memory

Original: Abstract
Revised: Abstract

Original: Associative memory plays an important role in human intelligence and its mechanisms have been linked to attention in machine learning.
Revised: Associative memory plays an important role in human intelligence and its mechanisms have been linked to attention in machine learning.

Original: While the machinelearning community’s interest in associative memories has recently been rekindled,most work has focused on memory recall ( read ) over memory learning ( write ).
Revised: While the machine learning community’s interest in associative memories has recently been rekindled, most work has focused on memory recall ( read ) over memory learning ( write ).

Original: In this paper, we present BayesPCN, a hierarchical associative memory capable ofperforming continual one-shot memory writes without meta-learning.
Revised: In this paper, we present BayesPCN, a hierarchical associative memory capable of performing continual one-shot memory writes without meta-learning.

Original: Moreover,
Revised: Moreover,

Original: BayesPCN is able to gradually forget past observations ( forget ) to free its memory.
Revised: BayesPCN is able to gradually forget past observations ( forget ) to free its memory.

Original: Experiments show that BayesPCN can recall corrupted i.i d. high-dimensional dataobserved hundreds of “timesteps” ago without a significant drop in recall abilitycompared to the state-of-the-art offline-learned associative memory models
Revised: Experiments show that BayesPCN can recall corrupted i.i.d. high-dimensional data observed hundreds to a thousand “timesteps” ago without a large drop in recall ability compared to the state-of-the-art offline-learned associative memory models.

Original:  Introduction
Revised:  Introduction

Original: One of the hallmarks of biological intelligence is the ability to robustly recall learned associations.
Revised: One of the hallmarks of biological intelligence is the ability to robustly recall learned associations.

Original: Biological agents are able to associate stimuli of various modalities and rely on their memories toperform everyday tasks from navigation to entity recognition.
Revised: Biological agents are able to associate stimuli of various modalities and rely on their memories to perform everyday tasks from navigation to entity recognition.

Original: Yet these agents are not only goodat recall but can quickly and continually form new memories from as little as a single experience.
Revised: Yet these agents are not only good at recall but can quickly and continually form new memories from as little as a single experience.

Original: How do biological agents achieve this feat while mitigating catastrophic forgetting [French, 1999]?
Revised: How do biological agents achieve this feat while mitigating catastrophic forgetting [French, 1999]?

Original: And how can we build neural networks that demonstrate these properties?
Revised: And how can we build neural networks that demonstrate these properties?

Original: This paper addresses thesequestions, focusing on fast, continual learning of neural artifacts that perform robust recall of i.i.d.high-dimensional observations.
Revised: This paper addresses these questions, focusing on fast, continual learning of neural artifacts that perform robust recall of i.i d. high-dimensional observations.

Original: Associative memory models have been of interest in machine learning for decades [Steinbuch,1961, Hopfield, 1982].
Revised: Associative memory models have been of interest in machine learning for decades [Steinbuch, 1961, Hopfield, 1982].

Original: They have been used to solve a wide range of problems from sequenceprocessing [Schmidhuber, 1992, Graves et al., 2014, Schlag et al., 2021] to pattern detection [Widrichet al., 2020, 2021, Seidl et al., 2021].
Revised: They have been used to solve a wide range of problems from sequence processing [Schmidhuber, 1992, Graves et al., 2014, Schlag et al., 2021] to pattern detection [Widrich et al., 2020, 2021, Seidl et al., 2021].

Original: Their study has allowed us to build models that replicateaspects of biological intelligence [Kanerva, 1988, Hawkins and Blakeslee, 2004] and provided novelperspectives on machine learning toolsets like dot-product attention, transformers, and linear layersof deep neural networks [Ramsauer et al., 2020, Bricken and Pehlevan, 2021, Irie et al., 2022].
Revised: Their study has allowed us to build models that replicate aspects of biological intelligence [Kanerva, 1988, Hawkins and Blakeslee, 2004] and provided novel perspectives on machine learning toolsets like dot-product attention, transformers, and linear layers of deep neural networks [Ramsauer et al., 2020, Bricken and Pehlevan, 2021, Irie et al., 2022].

Original: Our main aim is to recall exact memories from a neural network that can rapidly and continuallylearn from a stream of data.
Revised: Our main aim is to recall exact memories from a neural network that can rapidly and continually learn from a stream of data.

Original: Towards this end we introduce BayesPCN, a Bayesian take on thegenerative predictive coding network (GPCN)
Revised: Towards this end we introduce BayesPCN, a Bayesian take on the generative predictive coding network (GPCN)

Original: [Salvatori et al., 2021] that combines predictive codingand locally conjugate Bayesian updates.
Revised: [Salvatori et al., 2021] that combines predictive coding and locally conjugate Bayesian updates.

Original: There are currently a number of associative memory systemsthat are continually learnable [Hopfield, 1982, Kanerva, 1988, Hawkins and George, 2007, Ramsaueret al., 2020, Sharma et al., 2022].
Revised: There are currently a number of associative memory systems that are continually learnable [Hopfield, 1982, Kanerva, 1988, Hawkins and George, 2007, Ramsauer et al., 2020, Sharma et al., 2022].

Original: However, to our knowledge, BayesPCN is the first parametricassociative memory model to continually learn hundreds ofvery high-dimensional observations(> 10,000 dimensions) while maintaining good recall performance.
Revised: However, to our knowledge, BayesPCN is the first parametric associative [Figure] memory model to continually learn hundreds of high-dimensional observations while maintaining good recall performance on inputs with nontrivial amounts of corruption.

Original: In addition, it is also the firsthierarchical memory which after repeated forgetting recovers its original memory state.[Figure]
Revised: In addition, it is also the first hierarchical memory which after repeated forgetting recovers its original memory state.

Original:  Background 
Revised:  Background 2.1 Auto-Associative and Hetero-Associative Memory

Original: 2.1 Auto-Associative and Hetero-Associative Memory

Original: Before we begin let us review two useful concepts: auto-associative memory and hetero-associativememory.
Revised: Before we begin let us review two useful concepts: auto-associative memory and hetero-associative memory.

Original: The former, auto-associative memory, refers to a memory system that is content addressablebut for which the query may be a corrupted version of the original data item.
Revised: The former, auto-associative memory, refers to a memory system that is content addressable but for which the query may be a corrupted version of the original data item.

Original: One can think aboutauto-associative memory systems as a kind of approximate nearest neighbor lookup mechanism.
Revised: One can think about auto-associative memory systems as a kind of approximate nearest neighbor lookup mechanism.

Original: Hetero-associative memory systems are key-value stores, meaning that the recalled “value” ispotentially of a different type to the “key.”
Revised: Hetero-associative memory systems are key-value stores, meaning that the recalled “value” is potentially of a different type to the “key.”

Original: Crucially, hetero-associative memory systems can, in largepart, be implemented by auto-associative memory systems by forming a “joint” object to memorizeby combining the key and value into one object.
Revised: Crucially, hetero-associative memory systems can, in large part, be implemented by auto-associative memory systems by forming a “joint” object to memorize by combining the key and value into one object.

Original: Hetero-associative queries can then be processedsimply by looking up the joint object “corrupted” by discarding the value.
Revised: Hetero-associative queries can then be processed simply by looking up the joint object “corrupted” by discarding the value.

Original: 2.2 Generative Predictive Coding Network
Revised: 2.2 Generative Predictive Coding Network

Original: A generative predictive coding network (GPCN)
Revised: A generative predictive coding network (GPCN)

Original: [Salvatori et al., 2021] is an energy-based associativememory model.
Revised: [Salvatori et al., 2021] is an energy-based associative memory model that has set the state-of-the-art on a number of image associative recall tasks.

Original: As illustrated in Figure 2, GPCN takes the form of a fully connected neuralnetwork whose bottom layer activations characterize the sensory inputs and higher layer activationscharacterize the internal representations of the sensory inputs.
Revised: As illustrated in Figure 2, GPCN takes the form of a fully connected neural network whose bottom layer activations characterize the sensory inputs and higher layer activations characterize the internal representations of the sensory inputs.

Original: GPCN’s read operation (recall)initializes the bottom layer activations to the input query vector and iteratively denoises the queryuntil a stored value is retrieved by minimizing its energy function w.r.t.
Revised: GPCN’s read operation (recall) initializes the bottom layer activations to the input query vector and iteratively denoises the query until a stored value is retrieved by minimizing its energy function w.r.t.

Original: some or all of its neuronactivations.
Revised: some or all of its neuron activations.

Original: GPCN’s write operation (learning) adapts the network parameters to shape the energyfunction such that the stored datapoints become the local minima of the energy function.
Revised: GPCN’s write operation (learning) adapts the network parameters to shape the energy function such that the stored datapoints become the local minima of the energy function.

Original: Before describing GPCN’s read and write mechanisms in more detail, we introduce the notationwe use in this paper.
Revised: Before describing GPCN’s read and write mechanisms in more detail, we introduce the notation we use in this paper.

Original: Let M be a GPCN with L + 1 layers of neurons where 0 and L denotes thebottom and the top layer index respectively.
Revised: Let M be a GPCN with L + 1 layers of neurons where 0 and L denotes the bottom and the top layer index respectively.

Original: Then, let d l be the width of the l -th layer, x l ∈ R
Revised: Then, let d l be the width of the l -th layer, x l ∈ R

Original: d
Revised: d

Original: l bethe row vector denoting the l -th layer’s pre-activation function neuron values, and f l : R d l → R dlbe the l -th layer’s activation function, for instance ReLU. In addition, let W l ∈ R d l +1 × d l where0 ≤ l < L be the synaptic weight matrix between l + 1 -th and l-th layers and W L ∈ R d L be the “bias”row vector that influences the dynamics of the top layer neuron activities x L .
Revised: l be the row vector denoting the l -th layer’s pre-activation function neuron values, and f l : R d l → R d l

Original: One can intuitivelythink of the model parameters W 0:

Original: L as the memory content and the activations x 0: L as the read stateof the memory.

Original: [Equation]
Revised: [Equation]

Original: x 0 t x 1 t x 2 t
Revised: x 0 t x 1 t x 2 t

Original: W
Revised: W

Original: W
Revised: W

Original: Wµ Σ
Revised: Wµ Σ

Original: R
Revised: R

Original: U
Revised: U

Original: R
Revised: R

Original: U 0 1 : T 
Revised: U 0 1 : T be the l -th layer’s activation function, for instance ReLU. In addition, let W l ∈ R d l +1 × d l where 0 ≤ l < L be the synaptic weight matrix between l + 1 -th and l -th layers and W L ∈ R d L be the “bias” row vector that influences the dynamics of the top layer neuron activities x L .

Revised: One can intuitively think of the model parameters W 0:

Revised: L as the memory content and the activations x 0: L as the read state of the memory.

Original: GPCN’s read and write are facilitated by predictive coding (PC)
Revised: GPCN’s read and write are facilitated by predictive coding (PC)

Original: [Rao and Ballard, 1999], abiologically plausible learning rule that can approximate backpropagation [Millidge et al., 2020].
Revised: [Rao and Ballard, 1999], a biologically plausible learning rule that can approximate backpropagation [Millidge et al., 2020].

Original: PC’s energy function is the sum of prediction errors across all network layers.
Revised: PC’s energy function is the sum of prediction errors across all network layers.

Original: For GPCN, the choiceof prediction error is the squared Euclidean distance between the actual layer activations and theprediction of the same layer activations made by their immediate upper layer activations.
Revised: For GPCN, the choice of prediction error is the squared Euclidean distance between the actual layer activations and the prediction of the same layer activations made by their immediate upper layer activations.

Original: This leadsto the following energy function.
Revised: This leads to the following energy function.

Original: [Equation]
Revised: [Equation]

Original: GPCN’s readiteratively performs  gradient descent on the energy w.r.t.
Revised: GPCN’s read performs alternating gradient descent on the energy w.r.t.

Original: x 0: L after setting the initialvalues of x 0 to the memory query ¯ x 0 and fixing the values of W 0: L .
Revised: x 0 and x 1: L after setting the initial values of x 0 to the memory query ¯ x 0 and fixing the values of W 0: L .

Original: GPCN’s write obtains the readoutput x 0:
Revised: GPCN’s write obtains the read output x 0:

Original: L 1: T for each of T i.i.d.
Revised: L 1: T for each of T i.i.d.

Original: datapoints to store x 01: T , takes a gradient step on the energy w.r.t.
Revised: datapoints to store x 01: T , takes a gradient step on the energy w.r.t.

Original: W 0: L while fixing x 0: L 1: T , and repeats until convergence.
Revised: W 0: L while fixing x 0: L 1: T , and repeats until convergence.

Original: We now take a probabilistic perspective on GPCN.
Revised: We now take a probabilistic perspective on GPCN.

Original: One can view GPCN as a hierarchical generativemodel where all neuron activations are modeled by normally distributed random variables and the linkfunction between the subsequent layers is a layer-specific nonlinearity followed by a linear transform.
Revised: One can view GPCN as a hierarchical generative model where all neuron activations are modeled by normally distributed random variables and the link function between the subsequent layers is a layer-specific nonlinearity followed by a linear transform.

Original: Then, GPCN’s read - the minimization of Equation 1 - is equivalent to maximizing the log density in
Revised: Then, GPCN’s read - the minimization of Equation 1 - is equivalent to maximizing the log density in

Original: Equation 2 w.r.t.
Revised: Equation 2 w.r.t.

Original: x 0 and x 1: L .
Revised: x 0 and x 1: L .

Original: [Equation]
Revised: [Equation]

Original: In other words, GPCN’s read performs iterated conditional modes [Besag, 1986] on the hierarchicalgenerative model in the neuron activation space (refer to Algorithms 3 and 4).
Revised: In other words, GPCN’s read performs iterated conditional modes [Besag, 1986] on the hierarchical generative model in the neuron activation space (refer to Algorithms 3 and 4).

Original: In addition, GPCN’swrite can be viewed as an approximate maximization of the lower bound on log p ( x 01: T | W 0: L ) w.r.t.
Revised: In addition, GPCN’s write can be viewed as an approximate maximization of the lower bound on log p ( x 01: T | W 0: L ) w.r.t.

Original: W 0:
Revised: W 0:

Original: L [Millidge et al., 2021].
Revised: L [Millidge et al., 2021].

Original: This follows from the fact that GPCN learns via PC.
Revised: This follows from the fact that GPCN learns via PC.

Original: While GPCN possesses impressive associative recall capabilities, the learning rule proposed in
Revised: While GPCN possesses impressive associative recall capabilities, the learning rule proposed in

Original: Salvatori et al.
Revised: Salvatori et al.

Original: [2021] is not amenable to continual learning.
Revised: [2021] is not amenable to continual learning.

Original: This means write operations can onlyoccur offline and incorporating new datapoints requires costly  model refitting as well as the need toretain all observed datapoints.
Revised: Due to its lack of safeguards against catastrophic forgetting, continually learning GPCN leads to a forgetting of information associated with older observations or requires costly offline model refitting on all observations whenever a new datapoint is observed.

Original: BayesPCN’s foremost aim is to address this limitation while preserving
Revised: BayesPCN’s foremost aim is to address this limitation while preserving

Original: GPCN’s read capabilities as much as possible.
Revised: GPCN’s read capabilities as much as possible.

Original:  Memory Model
Revised:  Memory Model

Original: BayesPCN is structurally identical to GPCN, but unlike GPCN the network parameters W 0: Lare treated as latent random variables.
Revised: BayesPCN is structurally identical to GPCN, but unlike GPCN the network parameters W 0: L are treated as latent random variables.

Original: This allows BayesPCN’s write operation to be cast as (approximate) posterior inference over W 0: L given new observations in a way similar to [Wu et al.,2018a].
Revised: This allows BayesPCN’s write operation to be cast as (approximate) posterior inference over W 0: L given new observations in a way similar to [Wu et al., 2018a].

Original: In other words, storing T i.i.d.
Revised: In other words, storing T i.i.d.

Original: datapoints x 01: T into memory is equivalent to estimatingp ( W 0: L | x 01: T ) .
Revised: datapoints x 01: T into memory is equivalent to estimating p ( W 0: L | x 01: T ) .

Original: Figure 2 illustrates the graphical model for BayesPCN.
Revised: Figure 2 illustrates the graphical model for BayesPCN.

Original: We now review the probabilistic structure of BayesPCN.
Revised: We now review the probabilistic structure of BayesPCN.

Original: We assume that, prior to any observations,the random variables x 0: L t and W 0: L (where t is the datapoint index) are distributed as follows
Revised: We assume that, prior to any observations, the random variables x 0: L t and W 0: L (where t is the datapoint index) are distributed as follows

Original: [Equation]
Revised: [Equation]

Original: where N ( µ, Σ) and MN ( R, U, V ) denotes normal and matrix normal distribution [Gupta and Nagar,2018] respectively.
Revised: where N ( µ, Σ) and MN ( R, U, V ) denotes normal and matrix normal distribution [Gupta and Nagar, 2018] respectively.

Original: We remind the readers that the matrix normal distribution is aspecial kindof normal distribution.
Revised: We remind the readers that the matrix normal distribution is a normal distribution for matrix-variate data that compactly specifies the matrix elements’ means, row-wise covariances, and column-wise covariances.

Original: The joint distribution over all random variables when there are T sensoryobservations is, due to the conditional independence of x 0: L t , t ∈ { 1 , ..., T } given W 0: L , 
Revised: The joint distribution over all random variables when there are T sensory observations is, due to the conditional independence of x 0: L t , t ∈ { 1 , ..., T } given W 0: L , p ( x 0: L 1: T , W 0: L [Equation] =

Original: [Equation]
Revised: [Equation]

Revised: T (cid:89)

Original: [Equation]
Revised: [Equation]

Original: We note that BayesPCN’s structure leads to two desirable properties that become useful whenapproximating p ( W 0:
Revised: We note that BayesPCN’s structure leads to two desirable properties that become useful when approximating p ( W 0:

Original: L | x 01: T ) during the write operation.
Revised: L | x 01: T ) during the write operation.

Original: First, by exploiting the linear Gaussianmarginalization formula and conditional independence we can analytically compute the marginaldistribution over all activation vectors x 0: L t .
Revised: First, by exploiting the linear Gaussian marginalization formula and conditional independence we can analytically compute the marginal distribution over all activation vectors x 0: L t .

Original: [Equation]
Revised: [Equation]

Original: We stress that neither p ( x 0 t ) nor p ( x 0 t | W 0: L ) has an analytic form due to the presence of nonlinearities.
Revised: We stress that neither p ( x 0 t ) nor p ( x 0 t | W 0: L ) has an analytic form due to the presence of nonlinearities.

Original: In addition, if all activation vectors x 0: L t are observed, the posterior over the model parametersbecomes normally distributed due to the linear Gaussian posterior formula and the fact that W 0: Lare conditionally independent given x 0: L t (the analytical posterior formulae for Equation 9 are in
Revised: In addition, if all activation vectors x 0: L t are observed, the posterior over the model parameters becomes normally distributed due to the linear Gaussian posterior formula and the fact that W 0: L are conditionally independent given x 0: L t (the analytical posterior formulae for Equation 9 are in

Original: Appendix C).
Revised: Appendix C).

Original: [Equation]
Revised: [Equation]

Original:  R l ∗ , U l ∗ , µ ∗ , Σ ∗  denote the sufficient statistics of the layer-wise  posterior distributions given x lt , x l +1 t .
Revised: We use R l ∗ , U l ∗ , µ ∗ , Σ ∗ to denote the sufficient statistics of the layer-wise network weight posterior distributions given x lt , x l +1 t .

Original:  Memory Operations
Revised:  Memory Operations

Original: This section contains the memory operations that BayesPCN supports: read , write , and forget .
Revised: This section contains the memory operations that BayesPCN supports: read , write , and forget .

Original: Weuse the shorthand h
Revised: We use the shorthand h

Original: = x 1: L to denote all hidden layer neuron activations and W = W 0: L to denoteall network parameters.
Revised: = x 1: L to denote all hidden layer neuron activations and W = W 0: L to denote all network parameters.

Original: 4.1 Read
Revised: 4.1 Read

Original: The read operation takes the corrupted query vector ¯ x 0 and retrieves the memory’s best guess of theassociated original datapoint x 0 ∗ .
Revised: The read operation takes the corrupted query vector ¯ x 0 and retrieves the memory’s best guess of the associated original datapoint x 0 ∗ .

Original: As in GPCN, this is done by  climbing the memory’s log densitylog p ( x 0 , h | x 0 1: T ) w.r.t.
Revised: As in GPCN, this is done by hill climbing the memory’s log density log p ( x 0 , h | x 01: T ) w.r.t.

Original: the neuron activations x 0 , h .
Revised: the neuron activations x 0 , h .

Original: To clarify, x 0 , h is the current memory read stateand is not directly related to the past observations x 01: T .
Revised: To clarify, x 0 , h is the current memory read state and is not directly related to the past observations x 01: T .

Original: BayesPCN’s auto-associative recall performsiterated conditional modes on log p ( x 0 , h | x 01: T ) w.r.t.
Revised: BayesPCN’s auto-associative recall performs iterated conditional modes on log p ( x 0 , h | x 01: T ) w.r.t.

Original: x 0 , h and BayesPCN’s hetero-associative recallperforms gradient ascent on log p ( x 0 , h | x 01: T ) w.r.t.
Revised: x 0 , h and BayesPCN’s hetero-associative recall performs gradient ascent on log p ( x 0 , h | x 01: T ) w.r.t.

Original: x 0 , h while keeping the known values of x 0fixed.
Revised: x 0 , h while keeping the known values of xfixed.

Original: Algorithm 3 and Algorithm 4 in Appendix A, the recall procedures taken from Salvatori et al.[2021] and rewritten from the probabilistic perspective, describe this in more detail.
Revised: Algorithm 3 and Algorithm 4 in Appendix A, the recall procedures taken from Salvatori et al. [2021] and rewritten from the probabilistic perspective, describe this in more detail.

Original: 4.2 Write
Revised: 4.2 Write

Original: The write operations aims to take the previous timestep posterior over the model 
Revised: The write operations aims to take the previous timestep posterior over the model parameters p

Original: parametersp ( W | x 01: t − 1 ) and the current timestep observation x 0 t to return the current timestep posterior overthe model parameters p
Revised: ( W | x 01: t − 1 ) and the current timestep observation x 0 t to return the current timestep posterior over the model parameters p

Original: ( W | x 01: t ) .
Revised: ( W | x 01: t ) .

Original: Unfortunately, the analytical form of p ( W | x 01: t ) is unavailable dueto the presence of nonlinearities in BayesPCN so one must resort to approximate posterior inference.
Revised: Unfortunately, the analytical form of p ( W | x 01: t ) is unavailable due to the presence of nonlinearities in BayesPCN so one must resort to approximate posterior inference.

Original: We propose a sequential importance sampling posterior inference algorithm for estimating p ( W | x 01: t )where the particles are high-dimensional Gaussian distributions.
Revised: We propose a sequential importance sampling posterior inference algorithm for estimating p ( W | x 01: t ) where the particles are high-dimensional Gaussian distributions.

Original: Let N ∈ N be the number ofsequential importance sampling particles and n ∈ { 1 , ..., N } be the particle index.
Revised: Let N ∈ N be the number of sequential importance sampling particles and n ∈ { 1 , ..., N } be the particle index.

Original: The t -th timestepposterior estimate ˆ p ( W | x 01: t ) is recursively represented as a weighted Gaussian mixture of theform ˆ p ( W | x 01: t ) =(cid:80)
Revised: The t -th timestep posterior estimate ˆ p ( W | x 01: t ) is recursively represented as a weighted Gaussian mixture of the form ˆ p ( W | x 01: t ) =

Original:  Nn =1 ω ( n ) p ( n ) ( W | x 01: t , h ( n ) 1: t ) where ω ( n ) is the n -th particle’s importanceweight and p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is the n -th particle’s activation conditioned posterior describedin Equation 8.
Revised: (cid:80) Nn =1 ω ( n ) p ( n ) ( W | x 01: t , h ( n ) 1: t ) where ω ( n ) is the n -th particle’s importance weight and p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is the n -th particle’s activation conditioned posterior described in Equation 8.

Original: We note that each p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is efficiently characterised by the sufficientstatistics µ ( n ) , Σ ( n ) , R 0: L − 1 , ( n ) , U 0:
Revised: We note that each p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is efficiently characterised by the sufficient statistics µ ( n ) , Σ ( n ) , R 0: L − 1 , ( n ) , U 0:

Original: L − 1 , ( n ) .
Revised: L − 1 , ( n ) .

Original: Algorithm 1 describes the write operation in detail (see
Revised: Algorithm 1 describes the write operation in detail (see

Original: Appendix B for its justification).x
Revised: Appendix B for its justification).

Original: t − 1 , h ( n ) t − 1 ) and weight ω n ) t − in ˆ p
Revised: do

Original: W

Original: x t − 1 ) do 4: Fit the parameters of the current timestep variational distribution: 5: ϕ = argmin ϕ D KL (cid:16) q (n ) ϕ ( h t)
Revised: D KL (cid:16) q ( t

Original: ||
Revised: ||

Original: ˆ p( n) ( x 0 t , h t | x01: t − 1 ) (cid:17) 6:
Revised: ˆ p n ( x 0 t , h t | x t − Sample h t : q

Original: Sample h ( n ) t : h ( n ) t ∼ q ( n ) ϕ ( h t ) 7: Update the model via linear Gaussian update conditioned on x 0 t and h ( n ) t : 8: p ( n ) ( W | x 01: t , h ( n ) 1: t )
Revised: Compute

Original: ∝ p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) p ( n ) ( x 0 t , h ( n ) t | W ) 9: Compute the unnormalized particle weight: ˆ ω ( n ) t =ω ( n ) t − 1 p ( n) ( x 0 t , h ( n ) t | x 01: t − 1 ) q ( n ) ϕ
Revised: = ( n ) t − 1 p ( n

Original: (  h ( n ) t) 10: end for 11: Set ˆ p ( W | x 01: t  ) = (cid:80) Nn
Revised: ( x 0 t , h ( n ) t | x 01: t − 1 ) q ( n ) ϕ ( h ( n ) t ) 10: end

Original: = ˆ ω ( n ) t (cid:80) Nn ′ = 1 ˆ ω ( n ′ ) t p ( n ) ( W | x 01: t , h ( n ) 1: t )
Revised: [Equation]

Original: While q ( n ) ( h t ) in principle can be any distribution, BayesPCN’s write sets it to q ( n ) ( h t ) =
Revised: While q ( n ) ( h t ) in principle can be any distribution, BayesPCN’s write sets it to q ( n ) ( h t ) =

Original: δ (cid:0) argmax h t log p ( x 0 t , h t | x 0: L t ) (cid:1) as is standard in predictive coding [Millidge et al., 2021].
Revised: δ (cid:0) argmax h t log p ( x 0 t , h t | x 0: L t ) (cid:1) as is standard in predictive coding [Millidge et al., 2021].

Original: Inaddition, we define the BayesPCN parameter prior (of an “empty” memory) as ˆ p ( W ) =140 (cid:80)
Revised: In addition, we define the BayesPCN parameter prior (of an “empty” memory) as ˆ p ( W ) = (cid:80)

Original: Nn =1 1 N p ( n ) ( W ) where µ ( n ) , R 0:
Revised: Nn =1 1 N p ( n ) ( W ) where µ ( n ) , R 0:

Original: L − 1 , ( n ) are initialized using Kaiming initialization.
Revised: L − 1 , ( n ) are initialized using Kaiming initialization.

Original: We chose thisprior over a non-mixture prior to obtain a multi-particle posterior approximation despite using a Diracvariational distribution.
Revised: We chose this prior over a non-mixture prior to obtain a multi-particle posterior approximation despite using a Dirac variational distribution.

Original: We now direct our attention to obtaining log p ( n ) ( x 0 t , h | x 01: t − 1 ) used in the read operations fromˆ p ( W | x 01: t ) .
Revised: We now direct our attention to obtaining log p ( n ) ( x 0 t , h | x 01: t − 1 ) used in the read operations from ˆ p ( W | x 01: t ) .

Original: Using Equation 6, we obtain the approximate posterior predictive density
Revised: Using Equation 6, we obtain the approximate posterior predictive density

Original: [Equation]
Revised: [Equation]

Original: BayesPCN’s write can be seen as a special case of sequential imputation [Kong et al., 1994], anonline parameter inference algorithm that represents the parameter posterior conditioned on partialobservations as a mixture of parameter posteriors conditioned on complete observations.
Revised: BayesPCN’s write can be seen as a special case of sequential imputation [Kong et al., 1994], an online parameter inference algorithm that represents the parameter posterior conditioned on partial observations as a mixture of parameter posteriors conditioned on complete observations.

Original: In addition,whereas the parameter update proposed by Salvatori et al.
Revised: In addition, whereas the parameter update proposed by Salvatori et al.

Original: [2021] is doing local gradient descent foreach W l , BayesPCN’s parameter update is doing local conjugate normal-normal Bayesian update foreach W l .
Revised: [2021] is doing local gradient descent for each W l , BayesPCN’s parameter update is doing local conjugate normal-normal Bayesian update for each W l .

Original: Hence, like GPCN, BayesPCN’s synaptic weight update only utilizes local information.
Revised: Hence, like GPCN, BayesPCN’s synaptic weight update only utilizes local information.

Original: Lastly, while exact Bayesian updates are invariant to the order of observations, because we are usingpredictive coding to fit the variational distribution BayesPCN’s write is technically not invariant tothe order of observations.
Revised: Lastly, while exact Bayesian updates are invariant to the order of observations, because we are using predictive coding to fit the variational distribution BayesPCN’s write is technically not invariant to the order of observations.

Original: 4.3 Forget
Revised: 4.3 Forget

Original: We also propose a diffusion-based forgetting mechanism for BayesPCN.
Revised: We also propose a diffusion-based forgetting mechanism for BayesPCN.

Original: As more datapoints arewritten into a BayesPCN model, the model’s belief over the true parameter location R 1: L − 1 , µshifts and the uncertainty over its belief U 1: L − 1 , Σ decreases.
Revised: As more datapoints are written into a BayesPCN model, the model’s belief over the true parameter location R 1: L − 1 , µ shifts and the uncertainty over its belief U 1: L − 1 , Σ decreases.

Original: This has the effect of making themodel’s parameters less adaptive to new datapoints and  magnifying R 1: L − 1 , µ ’s norms as the prior’scontribution weakens.
Revised: This has the effect of making the model’s parameters less adaptive to new datapoints and typically magnifying R 1: L − 1 , µ ’s norms as the prior’s contribution weakens.

Original: One can interpret this as the memory being overloaded.
Revised: One can interpret this as the memory being overloaded.

Original: The forget operationcounteracts this phenomenon by nudging the model’s belief over R 1: L − 1 , µ toward the prior (emptymemory state) and increasing its uncertainty U 1: L − 1 , Σ proportional to the forget strength parameterβ .
Revised: The forget operation counteracts this phenomenon by nudging the model’s belief over R 1: L − 1 , µ toward the prior (empty memory state) and increasing its uncertainty U 1: L − 1 , Σ proportional to the forget strength parameter β .

Original: Algorithm 2 describes this in detail (the analytical diffusion formulae are in Appendix D). 
Revised: Algorithm 2 describes this in detail (the analytical diffusion formulae are in Appendix D). do

Revised: Obtain the one-step diffused layer weight distribution p

Original:  ) = (cid:82) p ( W l  | W l , β ) p ( n ) ( W l | x ( n ) ,l 1: t , x ( n ) ,l +11: t ) dW l7: end for 8:
Revised: | n l t x n l t ) = (cid:82) p ( W l ′ | W l , β ) p ( n ) ( W l | x ( n ) ,l 1: t , x ( n ) ,l +1 t ) dW l

Original: Set p ( n ) ( W′ | x 01: t , h ( n ) 1: t ) =
Revised: Set p ( n ) ( W | x 01: t , h ( n ) 1: t ) =

Original: (cid:81) Ll =

Revised: (cid:81) Ll =

Original: p ( n ) ( W l ′ | x ( n ) ,l1: t , x ( n) ,l +1 1: t ) 9: end for 10: Set ˆ p ( W ′ | x 01: t ) =
Revised: p ( n ) ( W l ′ | x ( n ) ,l t , x ( n ,l + 10: Set ˆ p ( W ′ | x 01: t ) =

Original: (cid:80) Nn =1 ω ( n ) p ( n ) ( W ′ | x 01: t , h ( n ) 1: t )
Revised: (cid:80) Nn =1 ω ( n ) p ( n ) ( W ′ | x 01: t , h ( n ) 1: t )

Original: If applied an infinite number of times, forget reverts the posterior over the parameters ˆ p
Revised: If applied an infinite number of times, forget reverts the posterior over the parameters ˆ p

Original: ( W | x 01: t ) tothe prior ˆ p ( W ) due to the property of diffusion if the particle weights are uniform.
Revised: ( W | x 01: t ) to the prior ˆ p ( W ) due to the property of diffusion if the particle weights are uniform.

Original: In practice, onewould apply forget periodically or when the memory starts to become overloaded from too muchdata and stops being performant.
Revised: In practice, one would apply forget periodically or when the memory starts to become overloaded from too much data and stops being performant.

Original: Because forget erases stored information for all of datapointssomewhat evenly, the datapoints from the distant past that forget was applied to more would be, asthe name suggests, more forgotten than the recent datapoints that forget was applied to less.
Revised: Because forget erases stored information for all of datapoints somewhat evenly, the datapoints from the distant past that forget was applied to more would be, as the name suggests, more forgotten than the recent datapoints that forget was applied to less.

Original:  Related Work
Revised:  Related Work

Original: We briefly review the most relevant work to BayesPCN, starting with the predictive coding associativememories.
Revised: We briefly review the most relevant work to BayesPCN, starting with the predictive coding networks.

Original: As covered in Section 2, GPCN [Salvatori et al., 2021] is the model that BayesPCN is builtupon.
Revised: As covered in Section 2, GPCN [Salvatori et al., 2021] is the model that BayesPCN is built upon.

Original: Self-taught associative memory [Dovrolis, 2018, Smith et al., 2021] is a hierarchical predictivecoding memory that performs unsupervised continual learning using online clustering modules asbuilding blocks, unlike BayesPCN’s building blocks of artificial neurons.
Revised: Self-taught associative memory [Dovrolis, 2018, Smith et al., 2021] is a hierarchical predictive coding memory that performs unsupervised continual learning using online clustering modules as building blocks, unlike BayesPCN’s building blocks of artificial neurons.

Original: The sequential neuralcoding network [Ororbia et al., 2019, 2018], while not explicitly used as an associative memory,is also a continually learned predictive coding network similar in structure to GPCN that combatscatastrophic forgetting via task-dependent activation sparsity.
Revised: The sequential neural coding network [Ororbia et al., 2019, 2018], while not explicitly used as an associative memory, is also a continually learned predictive coding network similar in structure to GPCN that combats catastrophic forgetting via task-dependent activation sparsity.

Original: BayesPCN on the other hand combats catastrophic forgetting by incorporating uncertainty over the synaptic weights.
Revised: BayesPCN on the other hand combats catastrophic forgetting by incorporating uncertainty over the synaptic weights.

Original: Moving on to general attractor-based memory models, Hopfield networks [Hopfield, 1982] andtheir modern variants [Krotov and Hopfield, 2016, Ramsauer et al., 2020, Krotov and Hopfield,2020, Krotov, 2021, Millidge et al., 2022] are biologically motivated auto-associative memoriesthat define the neuron activation dynamics via Lagrangian functions.
Revised: Moving on to general attractor-based memory models, Hopfield networks [Hopfield, 1982] and their modern variants [Krotov and Hopfield, 2016, Ramsauer et al., 2020, Krotov and Hopfield, 2020, Krotov, 2021, Millidge et al., 2022] are biologically motivated auto-associative memories that define the neuron activation dynamics via Lagrangian functions.

Original: Out of these models only
Revised: Out of these models only

Original: Krotov [2021]’s model is hierarchical, but theirs is not continually learned.
Revised: Krotov [2021]’s model is hierarchical, but theirs is not continually learned.

Original: BayesPCN generalizesthe modern Hopfield network and fits into the universal Hopfield network’s similarity-separationprojection paradigm for associative memories (refer to Appendix E).
Revised: BayesPCN generalizes the modern Hopfield network and fits into the universal Hopfield network’s similarity-separationprojection paradigm for associative memories (refer to Appendix E).

Original: Sparse Distributed Memory(SDM)
Revised: Sparse Distributed Memory (SDM)

Original: [Kanerva, 1988] is a non-hierarchical memory that exploits high-dimensionality to performauto-associative and hetero-associative recall.
Revised: [Kanerva, 1988] is a non-hierarchical memory that exploits high-dimensionality to perform auto-associative and hetero-associative recall.

Original: Hierarchical Temporal Memory (HTM)
Revised: Hierarchical Temporal Memory (HTM)

Original: [Hawkins and
Revised: [Hawkins and

Original: George, 2007, Cui et al., 2016] is a biologically constrained hierarchical memory that continually learns time-based patterns with sparse representations.
Revised: George, 2007, Cui et al., 2016] is a biologically constrained hierarchical memory that continually learns time-based patterns with sparse representations.

Original: HTM does not employ predictive coding andto our knowledge has not yet been tested on high-dimensional data.
Revised: HTM does not employ predictive coding and to our knowledge has not yet been tested on high-dimensional data.

Original: Relative to generative modeling and memory research, our work has connections to the Kanervamachine’s memory write as inference approach [Wu et al., 2018a,b] and the one-shot memory writefrom Bartunov et al.
Revised: Relative to generative modeling and memory research, our work has connections to the Kanerva machine’s memory write as inference approach [Wu et al., 2018a,b] and the one-shot memory write from Bartunov et al.

Original: [2019], Pham et al.
Revised: [2019], Pham et al.

Original: These papers’ models employ meta-learning unlikeaforementioned work.
Revised: These papers’ models employ meta-learning unlike aforementioned work.

Original: We note that the Kanerva machine also applies conjugate normal-normal
Revised: We note that the Kanerva machine also applies conjugate normal-normal

Original: Bayesian update to adapt its top level memory matrix once the model is meta-learned via end-to-endgradient descent.
Revised: Bayesian update to adapt its top level memory matrix once the model is meta-learned via end-to-end gradient descent.

Original: On the other hand, BayesPCN applies conjugate normal-normal Bayesian update toall of its weight matrices and this as of itself constitutes learning.
Revised: On the other hand, BayesPCN applies conjugate normal-normal Bayesian update to all of its weight matrices and this as of itself constitutes learning.

Original:  Experiments
Revised:  Experiments

Original: This section assesses BayesPCN’s recall capabilities on image recovery tasks and examines the effectof forget on BayesPCN’s behaviour.
Revised: This section assesses BayesPCN’s recall capabilities on image recovery tasks and examines the effect of forget on BayesPCN’s behaviour.

Original: We used CIFAR

Revised: We used CIFAR

Original: [Krizhevsky et al., 2009] and Tiny ImageNet [Le and Yang, 2015] datasets for allexperiments, whose images are of size 3 × 32 × 32 and 3 × 64 × 64 respectively.
Revised: [Krizhevsky et al., 2009] and Tiny ImageNet [Le and Yang, 2015] datasets for all experiments, whose images are of size 3 × 32 × 32 and 3 × 64 × 64 respectively.

Original: During the writephase of the experiments, BayesPCN models were given one image to store into their memoriesper timestep for up to 1024 images.
Revised: During the write phase of the experiments, BayesPCN models were given one image to store into their memories per timestep for up to 1024 images.

Original: Then, during the read phase of the experiments, we recordedthe pixel-wise mean squared error (MSE) between the write phase images and the models’ readoutputs given the corrupted write phase images as inputs.
Revised: Then, during the read phase of the experiments, we recorded the pixel-wise mean squared error (MSE) between the write phase images and the models’ read outputs given the corrupted write phase images as inputs.

Original: For reference, the pixel-wise MSE of 0 . 01is distinguishable to human eyes on close inspection of the images.
Revised: For reference, the pixel-wise MSE of 0 .is distinguishable to human eyes on close inspection of the images.

Original: BayesPCN models were compared against identity models (models that return their inputs as outputs),modern Hopfield networks (MHN)
Revised: BayesPCN models were compared against identity models (models that return their inputs as outputs), modern Hopfield networks (MHN)

Original: [Ramsauer et al., 2020], and vanilla GPCNs [Salvatori et al. 2021]all of which were offline learned.
Revised: [Ramsauer et al., 2020], and vanilla GPCNs [Salvatori et al., 2021] all of which were offline learned.

Revised: We note that GPCNs have set the state-of-the-art on CIFAR10 and

Revised: TinyImageNet image recall tasks.

Original: BayesPCN models were also compared against online GPCNs,
Revised: BayesPCN models were also compared against online GPCNs,

Original: GPCNs that naively applied their offline memory write to sequentially arriving observations andmemorized one image per timestep for up to 1024 images.
Revised: GPCNs that naively applied their offline memory write to sequentially arriving observations and memorized one image per timestep for up to 1024 images.

Original: We used our own autodiff implementationof MHNs based on Appendix E. See Appendix F for additional experiment details.
Revised: We used our own autodiff implementation of MHNs based on Appendix E. See Appendix F for additional experiment details.

Original: 6.1 Auto-Associative and Hetero-Associative Read
Revised: 6.1 Auto-Associative and Hetero-Associative Read

Original: For the auto-associative recall experiments, the memory inputs at the read phase were the write phaseimages corrupted by pixel-wise white noise with standard deviation 0 .
Revised: For the auto-associative recall experiments, the memory inputs at the read phase were the write phase images corrupted by pixel-wise white noise with standard deviation 0 .

Original: For the hetero-associativerecall experiments, we tested two types of training image corruption schemes.
Revised: For the hetero-associative recall experiments, we tested two types of training image corruption schemes.

Original: The first type, dropout,randomly blacked out 25% of the image pixels.
Revised: The first type, dropout, randomly blacked out 25% of the image pixels.

Original: The second type, mask, blacked out 25% of therightmost image pixels.
Revised: The second type, mask, blacked out 25% of the rightmost image pixels.

Original: Image recovery under these settings is hetero-associative recall becauseone can view the non-blacked out pixels as the “key” and the blacked out pixels as the “value”.
Revised: Image recovery under these settings is hetero-associative recall because one can view the non-blacked out pixels as the “key” and the blacked out pixels as the “value”.

Original: Weexperimented with 4-layer BayesPCN models that have hidden layer sizes of 256, 512, and 1024,particle counts of 1 and 4, and activation functions ReLU and GELU.
Revised: We experimented with 4-layer BayesPCN models that have hidden layer sizes of 256, 512, and 1024, particle counts of 1 and 4, and activation functions ReLU and GELU.

Original: We searched over the samehyperparameters for offline and online 4-layer GPCNs aside from the particle count.
Revised: We searched over the same hyperparameters for offline and online 4-layer GPCNs aside from the particle count.

Original: Lastly,weevaluated 4-layer BayesPCN models, which we denote as BayesPCN ( forget ), that applied forgetwith strength β = 0 .
Revised: Lastly, we [Figure] [Table] [Figure]

Original: 001 to their weights once every 64 write operations.
Revised: [Table] [Figure] [Table] [Figure]

Original:  Table 1 shows the average MSE between the ground truth images and the associative memoryread outputs for models trained on data sequences of different lengths, where the reported scorescome from the best models of each category.
Revised: [Table] Table 1 shows the average MSE between the ground truth images and the associative memory read outputs for models trained on data sequences of different lengths, where the reported scores come from the best models of each category.

Original: The results were averaged across three random seeds.
Revised: The results were averaged across three random seeds.

Original: Appendix H contains additional images that demonstrate how  BayesPCN’s read performance scaleswith different hyperparameters, how BayesPCN gets better at reconstructing unseen images the moredatapoints it observes, and shows images sampled from BayesPCN by using it as a generative model.
Revised: Appendix I details how different hyperparameter values affect BayesPCN’s read performance.

Original: We found that for sequences of length ≤ 512, BayesPCN’s recall errors were consistently lowerthan online GPCN’s recall errors and were similar to offline GPCN’s recall errors.
Revised: We found that for sequences of length ≤ 512, BayesPCN’s recall errors were consistently lower than online GPCN’s recall errors and in general close to offline GPCN’s recall errors.

Original: In particular,
Revised: In particular,

Original: BayesPCN’s hetero-associative recall errors were orders of magnitude lower than online GPCN’serrors.
Revised: BayesPCN’s hetero-associative recall errors were orders of magnitude lower than online GPCN’s errors.

Original: As expected, BayesPCN ( forget ) performed worse than BayesPCN in this regime becauseinformation is being forgotten before the memory is nearing its storage capacity.
Revised: As expected, BayesPCN ( forget ) performed worse than BayesPCN in this regime because information is being forgotten before the memory is nearing its storage capacity.

Original: However, when thesequence length approached 1024, BayesPCN’s  recall errors deteriorated more rapidly than online GPCN’s errors and on the TinyImageNet white noise task became even worse than the identity model.
Revised: However, when the sequence length approached 1024, BayesPCN’s auto-associative recall errors deteriorated more rapidly than online GPCN’s errors and on the TinyImageNet white noise task became even worse than the identity model.

Original: However, BayesPCN ( forget )’s recall errors deteriorated more gradually and overall was able tooutperform online GPCN’s recall errors on all tasks and sequence lengths.We hypothesize that [Figure] [Table]
Revised: On the flipside, BayesPCN ( forget )’s recall errors deteriorated more gradually and overall was able to outperform online GPCN’s recall errors on all tasks and sequence lengths.

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Table] BayesPCN’s sudden performance drop stems from the fact that it is not “removing” old informationunlike online GPCN and BayesPCN ( forget ) due to conjugate Bayesian update’s observation orderinvariance property.
Revised: We hypothesize that BayesPCN’s sudden performance drop stems from the fact that it is not “removing” old information unlike online GPCN and BayesPCN ( forget ) due to conjugate Bayesian update’s observation order invariance property.

Original: This causes BayesPCN to approach the overloaded memory regime, where recallof all data suffers, more abruptly once some amount of information has been stored.
Revised: This causes BayesPCN to approach the overloaded memory regime, where recall of all data suffers, more abruptly once enough information has been stored.

Original: GPCN and BayesPCN models overall found hetero-associative recall easier than auto-associativerecall, likely because the former fixes the key bits during memory read and thus discourages thealgorithm from wandering to less realistic x 0 values.
Revised: GPCN and BayesPCN models overall found hetero-associative recall easier than auto-associative recall, likely because the former fixes the key bits during memory read and thus discourages the algorithm from wandering to less realistic x 0 values.

Original: Counterintuitively, GPCN and BayesPCNmodels were better at Tiny ImageNet recall over CIFAR10 recall.
Revised: Counterintuitively, GPCN and BayesPCN models were better at Tiny ImageNet recall over CIFAR10 recall.

Original: We attribute this behaviour to thefact that for some models, high-dimensional data recall can ironically be easier than low-dimensionaldata recall because individual datapoints are further apart from one another in higher dimensions.
Revised: We attribute this behaviour to the fact that for some models, high-dimensional data recall can ironically be easier than low-dimensional data recall because individual datapoints are further apart from one another in higher dimensions.

Original: A surprising discovery from this experiment was how good MHN’s recall can be compared to GPCNand BayesPCN’s recall when the memory query has a moderate amount of noise.
Revised: A surprising discovery from this experiment was how good MHN’s recall can be compared to GPCN and BayesPCN’s recall when the memory query had a moderate amount of noise, a result inconsistent with the findings from Salvatori et al.

Original: Because MHNexplicitly retains past observations in a data matrix and its softmax inverse temperature was set veryhigh ( β = 10 , 000 ), MHN’s recall effectively selected the observation that had the highest dot-productsimilarity to the memory query in our experiments.
Revised: Because MHN explicitly retains past observations in a data matrix and its softmax inverse temperature was set very high ( β = 10 , 000 ), MHN’s recall effectively selected the observation that had the highest dot-product similarity to the memory query in our experiments.

Original: In Appendix G, we assess how MHN’s recallcompares to BayesPCN’s recall when the memory query corruption is much higher.
Revised: In Appendix G, we assess how MHN’s recall compares to BayesPCN’s recall when the memory query corruption is much higher.

Original: As expected,because the observation with the highest dot-product similarity to the memory query is often not theright image to recall, BayesPCN significantly outperformed MHN in hetero-associative recall tasks.
Revised: As expected, because the observation with the highest dot-product similarity to the memory query is often not the right image to recall, BayesPCN significantly outperformed MHN in hetero-associative recall tasks.

Original: 6.2 Forgetting
Revised: 6.2 Forgetting

Original: This section investigates the forget operation’s effect on BayesPCN’s recall behaviour.
Revised: This section investigates the forget operation’s effect on BayesPCN’s recall behaviour.

Original: Specifically,we show that forget causes BayesPCN to better recall recent observations at the expense of olderobservations and that this effect is more pronounced the higher the forget strength is.
Revised: Specifically, we show that forget causes BayesPCN to better recall recent observations at the expense of older observations and that this effect is more pronounced the higher the forget strength is.

Original: We trainedsix structurally identical BayesPCN models on 1024 CIFAR10 images sequentially and appliedforget with strength β = 0 , 0 . 
Revised: We trained six structurally identical BayesPCN models on 1024 CIFAR10 images sequentially and applied forget with strength β = 0 , 0 . 001 , 0 . 002 , 0 .

Original: 02 to each model once after every 64 writeoperations.
Revised: 02 to each model once after every 64 write operations.

Original: Figure 3 compares the models’ recall MSE on 16 data batches of size 64 that were observed duringdifferent points of the training.
Revised: Figure 3 compares the models’ recall MSE on 16 data batches of size 64 that were observed during different points of the training once the models have observed all 1024 datapoints.

Original: We see that BayesPCN with no forget ( β = 0) performs similarly onall datapoints regardless of their observation timestep, which is likely the consequence of conjugate Bayesian update’s observation order invariance.
Revised: We see that BayesPCN with no forget ( β = 0) performs similarly on all datapoints regardless of their observation timestep, which is likely the consequence of conjugate Bayesian update’s observation order invariance.

Original: On the other hand, BayesPCNs with forget ( β > 0)consistently perform better on newer datapoints with this effect being strongly correlated with thevalue of β .
Revised: On the other hand, BayesPCNs with forget ( β > 0) consistently perform better on newer datapoints with this effect being strongly correlated with the value of β .

Original: As previously shown, some amount of forgetting during training can improve the overallperformance of the memory across all datapoints over no forgetting for long training sequences.
Revised: As previously shown, some amount of forgetting during training can improve the overall performance of the memory across all datapoints over no forgetting for long training sequences.

Revised: Lastly, the higher the forget strength was the lower the recall error on the most recent data batch was for all tasks and forget strength values.

Original:  Discussion
Revised:  Discussion

Original: One can intuitively think of BayesPCN read as an approximate nearest neighbour operation with alearned similarity metric and write as the continual learning of that metric.
Revised: One can intuitively think of BayesPCN read as an approximate nearest neighbour operation with a learned similarity metric and write as the continual learning of that metric.

Original: MHN on the other handuses the dot-product distance as its similarity metric [Millidge et al., 2022].
Revised: MHN on the other hand uses the dot-product distance as its similarity metric [Millidge et al., 2022].

Original: This explains why MHNis not good at highly masked image recall because from a pixelwise dot-product distance perspective,highly masked image queries are often closer to unrelated images than to the ground truth images.
Revised: This explains why MHN is not good at highly masked image recall because from a pixelwise dot-product distance perspective, highly masked image queries are often closer to unrelated images than to the ground truth images.

Original: BayesPCN is closely related to the predictive coding theories of the brain’s memory system.
Revised: BayesPCN is closely related to the predictive coding theories of the brain’s memory system.

Original: Forexample, Barron et al.
Revised: For example, Barron et al.

Original: [2020] suggests that the hippocampus may correspond to the top layer of apredictive coding network and the neo-cortical areas may correspond to the rest of the predictivecoding network.
Revised: [2020] suggests that the hippocampus may correspond to the top layer of a predictive coding network and the neo-cortical areas may correspond to the rest of the predictive coding network.

Original: Similarly, Dovrolis [2018] posits that their predictive coding network could serveas a model of the cortical column albeit without experimental results.
Revised: Similarly, Dovrolis [2018] posits that their predictive coding network could serve as a model of the cortical column albeit without experimental results.

Original: It is however unclear how  BayesPCN’s  locally conjugate Bayesian weight update can be realized in the brain unlike GPCN’s Hebbian weight update.
Revised: It is however unclear how exactly BayesPCN’s activation gradient descent and locally conjugate Bayesian weight update can be realized in the brain.

Original: Nevertheless, given that conjugate normal-normal Bayesian update’s posteriormean formula (Equation 27) can be seen as a preconditioned Hebbian weight update influenced by theinput and error neurons, there may be a biologically plausible way of approximating single particle

Original: BayesPCN write . In addition, there are similarities between BayesPCN’s diffusion-based forgetand the synaptic homeostasis hypothesis [Cirelli and Tononi, 2022], where during sleep synapticweight strengths are decreased on average to facilitate future learning.
Revised: Lastly, there are similarities between BayesPCN’s diffusion-based forget and the synaptic homeostasis hypothesis [Cirelli and Tononi, 2022], where during sleep synaptic weight strengths are decreased on average to facilitate future learning.

Original: Aside from these biological connections, BayesPCN has many potential machine learning applications.
Revised: Aside from these biological connections, BayesPCN has many potential machine learning applications.

Original: One possible line of work is using it in conjunction with a neural controller that learns whatinformation to store into memory [Graves et al., 2014, Schlag et al., 2021].
Revised: One possible line of work is using it in conjunction with a neural controller that learns what information to store into memory [Graves et al., 2014, Schlag et al., 2021].

Original: Using BayesPCN in thisway would confer the benefit of high query noise tolerance for neural controllers.
Revised: Using BayesPCN in this way would confer the benefit of high query noise tolerance for neural controllers.

Original: Another line ofwork is using it to solve small data supervised learning problems, leveraging the fact that nearestneighbour methods work well when data availability is sparse [Ramsauer et al., 2020].
Revised: Another line of work is using it to solve small data supervised learning problems, leveraging the fact that nearest neighbour methods work well when data availability is sparse [Ramsauer et al., 2020].

Original: To conclude, we introduced BayesPCN, an associative memory model that can robustly retrieve pastobservations given various corrupted inputs, perform one-shot parameter update of a deep neuralnetwork to store new observations as attractors, and forget old observations to better store newer ones.
Revised: To conclude, we introduced BayesPCN, an associative memory model that can robustly retrieve past observations given various corrupted inputs, perform one-shot parameter update of a deep neural network to store new observations as attractors, and forget old observations to better store newer ones.

Original: We demonstrated that BayesPCN’s recall performance is comparable to its offline-learned counterparteven though it is continually learned.
Revised: We demonstrated that BayesPCN’s recall performance is not significantly worse than the recall performance of its offline-learned counterpart even though it is continually learned.

Original: There are several intriguing extensions to BayesPCN that mayaddress some of itscurrent limitations.
Revised: There are several intriguing extensions to BayesPCN that may address some of its limitations.

Original: As of now, BayesPCN read can take up to thousands ofactivation gradient descent iterations to obtain the bestpossible recall result when a lot of data hasbeen stored into memory.
Revised: Currently, BayesPCN read can take up to thousands of activation gradient descent iterations to obtain the best recall result when a lot of data has been stored into memory.

Original: Supplementing BayesPCN read with amortized inference may greatlyreduce this computational requirement [Tschantz et al., 2022].
Revised: Supplementing BayesPCN read with amortized inference may greatly reduce this computational requirement [Tschantz et al., 2022].

Original: In addition, BayesPCN read canreturn a nonsensical output if the memory query is too far away from the stored datapoints or if thememory is overloaded.
Revised: In addition, BayesPCN read can return a nonsensical output if the memory query is too far away from the stored datapoints or if the memory is overloaded.

Original: Investigating BayesPCN’s latent landscape may allow us design models withmore robust basin of attractions around the stored datapoints.
Revised: Investigating BayesPCN’s latent landscape may allow us design models with more robust basin of attractions around the stored datapoints that can easily be reached via activation gradient descent.

Original: Last but not least, extending BayesPCNbeyond memorizing i.i.d.
Revised: Last but not least, extending BayesPCN beyond memorizing i.i.d.

Original: data to memorizing sequences like the human episodic memory would beof great interest.
Revised: data to memorizing sequences like the human episodic memory would be of great interest.

Original: Checklist
Revised: Acknowledgments and Disclosure of Funding

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work? [Yes] We mention that our memory can only store i.i.d.
Revised: We would like to thank Boyan Beronov for the helpful discussions.

Original: data, questions of its biological plausibility compared to GPCN, andhow its performance rapidly declines relative to the baselines for longer sequences.(c) Did you discuss any potential negative societal impacts ofyour work? [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: In addition, we acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada

Original: [Yes] If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A]
Revised: CIFAR AI Chairs Program, and the Intel Parallel Computing Centers program.

Original: (b) Did you include complete proofs of all theoretical results? [N/A]3. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: Additional support was provided by UBC’s Composites Research Network (CRN), and Data Science Institute (DSI).

Original: [Yes] We will releasethe code under the MIT License.(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)? [Yes] Refer to Section 6 and Appendix F. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: This research was enabled in part by technical support and computational resources provided by

Original: [Yes] We did in Appendix F.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)? [No] We did not measure the totalcompute amount but listed the GPU type and the compute clusters used in Appendix F.4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: WestGrid (www.westgrid.ca), Compute Canada (www.computecanada.ca), and Advanced Research

Original: [Yes] Refer to Section (b) Did you mention the license of the assets?
Revised: Computing at the University of British Columbia (arc.ubc.ca).

Original: [Yes] Refer to Appendix F. (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: B Derivation of BayesPCN write

Original: using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: p ( W |

Original: [N/A] If you
Revised: [Equation]

Original: used crowdsourcing or conducted research with human subjects. (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: 01: t ) =

Original: [N/A]
Revised: [Equation]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: Z

Original: spent on participant compensation? [N/A] B Derivation of BayesPCN write p ( W | x 01: t ) = 1 Z p ( W | x 01: t − 1 ) p ( x 0 t | W ) (12) =
Revised: [Equation]

Original: Z p ( W | x 01: t− 1 ) E q ( h t )
Revised: [Equation] | [Equation] 01: t

Original:  [ p ( x 0 t , h t | W ) q ( h t ) ] (13) ≈
Revised: [Equation] 1 ) p ( x 0 t | W ) (12) = Z p ( W | x 01: t − 1 ) E q ( h t ) [ p ( x 0 t , h t | W ) q ( h t ) ] (13) ≈

Original: Z [
Revised: Z [

Original: N (cid:88) n = ω ( n ) t − 1 p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 )]
Revised: N (cid:88) n = ω ( n ) t − 1 p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 )]

Original: E q ( h t )
Revised: E q ( h t )

Original: [ p ( x 0 t , h t | W ) q ( h t ) ] (14) =
Revised: [ p ( x 0 t , h t | W ) q ( h t ) ] (14) =

Original: Z
Revised: Z

Original: N (cid:88) n = ω ( n 
Revised: N (cid:88) n = ω ( n ) t − 1 p ( [Equation] [Equation] [Equation]

Revised: t

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Equation 17 holds because
Revised: [Equation] [Equation] [ p ( [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Revised: h t |

Revised: [Equation]

Revised: ] (15) ≈

Revised: Z

Revised: N

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: n )

Revised: [Equation]

Revised: (cid:88) i =

Revised: [Equation]

Revised: p ( n ) (

Revised: [Equation]

Revised: | x

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: h ( n )

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: [Equation]

Original: and Equation 18 holds because
Revised: ( x [Equation] [Equation] t |

Original: [Equation]
Revised: [Equation]

Revised: ) q (

Revised: [Equation]

Revised: [Equation]

Original: Finally
Revised: [Equation] ) t ) , h ( [Equation] ( n ) ( h

Revised: [Equation]

Revised: ) (16) =

Revised: Z

Revised: S

Revised: N (cid:88) n =

Revised: S (cid:88) i = ω ( n ) t

Revised: [Equation]

Revised: 1 p ( n ) (

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: h ( n

Revised: [Equation]

Revised: i ) t | x 01:

Revised: [Equation]

Revised: − 1 ) q ( n ) (

Revised: [Equation]

Revised: ( n

Revised: [Equation]

Revised: i ) t ) p ( n ) ( W

Revised: [Equation]

Revised: x 01: t , h ( n,i ) 1: t ) (17) =

Revised: N (cid:88) n =

Revised: S (cid:88) i = ω ( n ) t − 1 p ( n ) ( x 0 t , h ( n,i ) t | x 01: t − 1 ) q ( n ) ( h ( n,i ) t ) (cid:80) Nn ′ =1 (cid:80) Si ′

Revised: =1 ω ( n ′ )

Revised: t − 1 p ( n ′ )

Revised: ( x 0 t , h ( n ′ ,i ′ )

Revised: t | x 01: t − 1 ) q ( n ′ )

Revised: ( h ( n ′ ,i ′ )

Revised: t ) p ( n ) ( W | x 01: t , h ( n,i ) 1: t ) (18) =

Revised: N (cid:88) n =

Revised: S (cid:88) i = ω ( n,i ) t p ( n ) ( W | x 01: t , h ( n,i ) 1: t

Revised: [Equation]

Revised: Equation 17 holds because

Original: [Equation]
Revised: [Equation]

Revised: and Equation 18 holds because

Revised: [Equation]

Revised: Finally

Revised: [Equation]

Original: if S = 1 we get
Revised: if S = 1 we get

Original: [Equation]
Revised: [Equation]

Original: C Analytical Posterior Formulae for write
Revised: C Analytical Posterior Formulae for write

Original: Let z
Revised: Let z

Original: = f ( x ) .
Revised: = f ( x ) .

Original: Then, BayesPCN’s top layer parameter update given x 0: L is
Revised: Then, BayesPCN’s top layer parameter update given x 0: L is

Original: [Equation]
Revised: [Equation]

Original: 26) while the update for all other layers given x 0: L is
Revised: 26) while the update for all other layers given x 0: L is

Original: [Equation]
Revised: [Equation]

Original: D Analytical Diffusion Formulae for forget
Revised: D Analytical Diffusion Formulae for forget

Original: Let R 1: L − 1, U 1: L − 1, µ 0 , Σ 0 be the memory prior parameters and β be the forget strength.
Revised: Let R 1: L − 1, U 1: L − 1, µ 0 , Σ 0 be the memory prior parameters and β be the forget strength.

Original: Then,
Revised: Then,

Original: BayesPCN’s top layer diffusion update is
Revised: BayesPCN’s top layer diffusion update is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: while the update for all other layers is
Revised: while the update for all other layers is

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: E Connections to Hopfield Networks
Revised: E Connections to Hopfield Networks

Original: We show that modern Hopfield network’s recall is equivalent to the recall of a BayesPCN model with
Revised: We show that modern Hopfield network’s recall is equivalent to the recall of a BayesPCN model with

Original: L = 0 .
Revised: L = 0 .

Original: BayesPCN’s activation log density is defined as log p ( x 0 , h ) =
Revised: BayesPCN’s activation log density is defined as log p ( x 0 , h ) =

Original: log (cid:16)(cid:80) N n =1 ω ( n ) p ( n ) ( x 0 , h ) (cid:17) ,and its gradient w.r.t.
Revised: log (cid:16)(cid:80) N n =1 ω ( n ) p ( n ) ( x 0 , h ) (cid:17) , and its gradient w.r.t.

Original: the activations is∇ x 0 , h log p( x 0 , h ) = N (cid:88) n = [Equation] exp(log
Revised: the activations is ∇ x 0 , h log p

Original: [Equation]

Original: ( n

Original: [Equation]

Original: p ( n

Original: [Equation]

Original: ( x

Original: [Equation]

Original: , h ))

Original: (cid:80) Nn

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: p
Revised: x

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation]
Revised: [Equation]

Original: h )) ∇ x [Equation] h log p ( n ) ( x [Equation]
Revised: On the other hand, MHN’s energy is

Original: h )

Original: [Equation]
Revised: [Equation]

Original: On the other hand, MHN’s energy is

Original: [Equation]
Revised: [Equation]

Original: where q ∈ R 1 × d k is the query row vector, K ∈ R N × d k is the key matrix, and K j ∈ R 1 × d k is thej-th key row vector.
Revised: where q ∈ R 1 × d k is the query row vector, K ∈ R N × d k is the key matrix, and K j ∈ R 1 × d k is the j-th key row vector.

Original: q and K correspond to x 0 and W 0 in our paper’s notation.
Revised: q and K correspond to x 0 and W 0 in our paper’s notation.

Original: The negative of theenergy can be converted to the following Gaussian mixture log density.
Revised: The negative of the energy can be converted to the following Gaussian mixture log density.

Original: [Equation]
Revised: [Equation]

Original: Recall in both BayesPCN and MHN is gradient ascent on the above log density.
Revised: Recall in both BayesPCN and MHN is gradient ascent on the above log density.

Original: When we take thegradient with respect to the input vector q , we recover Equation 33.
Revised: When we take the gradient with respect to the input vector q , we recover Equation 33.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ■ We conclude that recall in Modern Hopfield Network is equivalent to recall under our framework,which is gradient descent on the log joint of a normal mixture w.r.t.
Revised: We conclude that recall in Modern Hopfield Network is equivalent to recall under our framework, which is gradient descent on the log joint of a normal mixture w.r.t.

Original: neuron activations, where thereare no hidden layers ( L = 0 ).
Revised: neuron activations, where there are no hidden layers ( L = 0 ).

Original: However training our model with L = 0 does not lead to the samememory update as the suggested training procedure of MHN, which is setting each key vector K j tosome observed datapoint.
Revised: However training our model with L = 0 does not lead to the same memory update as the suggested training procedure of MHN, which is setting each key vector K j to some observed datapoint.

Original: Universal Hopfield network [Millidge et al., 2022] proposes a framework for single-shot associativememory that decomposes recall into three components: similarity function, separation function, andprojection matrix.
Revised: Universal Hopfield network [Millidge et al., 2022] proposes a framework for single-shot associative memory that decomposes recall into three components: similarity function, separation function, and projection matrix.

Original: Let x = (¯ x 0 , h ) be the row vector of all initial network activations when give thequery ¯ x 0 .
Revised: Let x = (¯ x 0 , h ) be the row vector of all initial network activations when give the query ¯ x 0 .

Original: Equation 33 suggests that BayesPCN read ’s implementation of those components is perparticle weighted log joint log ω ( n ) p ( n ) ( x ) of the query ¯ x 0 for the similarity function, softmax forthe separation function, and the matrix
Revised: Equation 33 suggests that BayesPCN read ’s implementation of those components is per particle weighted log joint log ω ( n ) p ( n ) ( x ) of the query ¯ x 0 for the similarity function, softmax for the separation function, and the matrix

Original: [Equation]
Revised: [Equation]

Original:  for the projection matrix where γis the learning rate.
Revised:  for the projection matrix where γ is the learning rate.

Original: We can accommodate the fact that BayesPCN’s read does iterated conditionalmodes by zeroing out the fixed variable gradients in the projection matrix.
Revised: We can accommodate the fact that BayesPCN’s read does iterated conditional modes by zeroing out the fixed variable gradients in the projection matrix.

Original: F Additional Experiment Details
Revised: F Additional Experiment Details

Original: All GPCN and BayesPCN models had σ W = 1 , σ x = 0 . 01 , and used Adam with learning rate0 .
Revised: All GPCN and BayesPCN models had σ W = 1 , σ x = 0 . 01 , and used Adam with learning rate 0 .

Original: 01 as the neuron activation gradient descent optimizer.
Revised: 01 as the neuron activation gradient descent optimizer.

Original: All energy minimization w.r.t.
Revised: All energy minimization w.r.t.

Original: the neuronactivations took 500 gradient steps during the write phase.
Revised: the neuron activations took 500 gradient steps during the write phase.

Original: During the read phase, the outer loopiterated conditional mode in Algorithm 3 was repeated 30 times and the energy minimization in
Revised: During the read phase, the outer loop iterated conditional mode in Algorithm 3 was repeated 30 times and the energy minimization in

Original: Algorithm 4 took 30 × 500 gradient steps.
Revised: Algorithm 4 took 30 × 500 gradient steps.

Original: MHN models had β = 10 , 000 (equivalent to σ x = 0 . 01 ),used Adam with learning rate 1 . 

Revised: MHN models had β = 10 , 000 (equivalent to σ x = 0 . 01 ), used Adam with learning rate 1 .

Original: as the gradient descent optimizer, and performed gradient-descentbased recall similar to BayesPCN based on the connection from Appendix E. All experiments wererun on CIFAR10 and/or Tiny ImageNet datasets (both of which have the MIT License) and theimage pixel values were normalized to fall between [ − 1 , 1] .
Revised: as the gradient descent optimizer, and performed gradient-descent based recall similar to BayesPCN based on the connection from Appendix E. All experiments were run on CIFAR10 and/or Tiny ImageNet datasets (both of which have the MIT License) and the image pixel values were normalized to fall between [ − 1 , 1] .

Revised: Offline GPCNs received 4000 iterations of network weight gradient descent steps.

Original: Online GPCNs took a single gradientstep w.r.t.
Revised: Online GPCNs took a single gradient step w.r.t.

Original: the network weights after the hidden activations converged per observation, a treatmentconsistent with that of the fast weight memory in Schlag et al.
Revised: the network weights after the hidden activations converged per observation, a treatment consistent with that of the fast weight memory in Schlag et al.

Original: BayesPCN forget modelshad four hidden layers of width 1024, a single particle, and GELU activations.
Revised: BayesPCN forget models had four hidden layers of width 1024, a single particle, and GELU activations.

Original: All hyperparameterranges were chosen based on Salvatori et al.
Revised: All hyperparameter ranges were chosen based on Salvatori et al.

Original: [2021]’s experiments and GPCN/BayesPCN’s empiricalresults.
Revised: [2021]’s experiments and GPCN/BayesPCN’s empirical results.

Original: Lastly, experiments used NVIDIA Tesla V100 GPUs and were run on the university’s internalclusters.
Revised: All experiments used NVIDIA Tesla V100 GPUs and were run on the university’s internal clusters.

Revised: Training the most expensive GPCN model (hidden layer width of 1024) and BayesPCN model (hidden layer width of 1024, 4 particles) on 1024 observations took 20 hours and 3 hours respectively.

Revised: Evaluating the most expensive GPCN and BayesPCN models on all tasks took 25 minutes andhours respectively

Original: [Table]
Revised: [Table]

Original: G BayesPCN vs MHN Recall on Highly Noised Queries
Revised: G BayesPCN vs MHN Recall on Highly Noised Queries

Original: This section reports the recall performance of MHN and BayesPCN models on high query noiseassociative recall tasks.
Revised: This section reports the recall performance of MHN and BayesPCN models on high query noise associative recall tasks.

Original: The experiment setup is similar to that of Section 6 aside from the fact thatthe white noise tasks had the noise standard deviation set to 0 .
Revised: The experiment setup is similar to that of Section 6 aside from the fact that the white noise tasks had the noise standard deviation set to 0 .

Original: 8 instead of 0 .
Revised: 8 instead of 0 .

Original: 2 , the dropout tasksrandomly blacked out 75% of the pixels instead of 25% , and the masking tasks blacked out 75% ofthe rightmost pixels instead of 25% .
Revised: 2 , the dropout tasks randomly blacked out 75% of the pixels instead of 25% , and the masking tasks blacked out 75% of the rightmost pixels instead of 25% .

Original: Because the high query noise task is harder, we show the recallresult after the models observed 16, 32, 64, and 128 datapoints.
Revised: Because the high query noise task is harder, we show the recall result after the models observed 16, 32, 64, and 128 datapoints.

Original: BayesPCN models had four hiddenlayers of width 256, a single particle, and GELU activations.
Revised: BayesPCN models had four hidden layers of width 256, a single particle, and GELU activations.

Original: MHNs again used β = 10 , 000 .
Revised: MHNs again used β = 10 , 000 .

Original: [Table]
Revised: [Table]

Original: H BayesPCN Scaling, Generalization, and Sampling
Revised: H Additional Qualitative Results

Original: [Figure] H.1 Scaling

Original: [Figure] Figure 5 qualitatively demonstrates how BayesPCN’s read scales with the number of stored datapointsfor the CIFAR10 recall tasks.
Revised: Figure 5 qualitatively demonstrates how BayesPCN’s read scales with the number of stored datapoints for the CIFAR10 recall tasks.

Original: BayesPCN models are able to output images that are very close tothe original image even when the inputs are significantly corrupted.
Revised: BayesPCN models are able to output images that are very close to the original image even when the inputs are significantly corrupted.

Original: As the number of observationsincreases, the read operation’s basin of attraction decreases as it is still able to reconstruct the originalimage but gets worse at recovering the original image given a corrupted version of it.
Revised: As the number of observations increases, the read operation is still able to reconstruct the original image but gets worse at recovering the original image given a corrupted version of it.

Revised: I Effects of BayesPCN Hyperparameters I.1 Network Width and Depth

Original: Figure 6 illustrates how BayesPCN’s recall accuracy scales with the network width and depth.
Revised: Figure 6 and Table 4 illustrate how BayesPCN’s recall accuracy and MSE scale with the network width and depth.

Original: Arecall is considered correct if the MSE between the ground truth and the recalled data is less than0 .
Revised: A recall is considered correct if the MSE between the ground truth and the recalled data is less than 0 . 01 .

Revised: BayesPCN models had GELU activation functions, σ W = 1 , σ x = 0 .

Revised: 01 , and a single particle.

Original: We find that the increased network width and depth is generally helpful across all tasks.
Revised: We found that the increased network width was helpful across all tasks.

Revised: Increased network depth was helpful when moving from network depth of 2 to 4, but moving from network depth of 4 to 8 had no noticeable impact across all tasks.

Original: Wenote that because the plots depict recall accuracy not MSE, if the memory performance generallydeclines and the average recall MSE exceeds 0 .
Revised: We note that because the Figure 6 depicts recall accuracy not MSE, if the memory performance generally declines and the average recall MSE exceeds 0 .

Original: 01 , this can lead to very low accuracy even if theactual recall MSE is not much greater than 0 .
Revised: 01 , this can lead to very low accuracy even if the actual recall MSE is not much greater than 0 .

Revised: I.2 Network Weight Prior Uncertainty and Observation Noise

Original: We also investigate the effect of σ W and σ x hyperparameters on BayesPCN’s scaling properties.
Revised: We also investigate the effect of σ W and σ x hyperparameters on BayesPCN’s scaling properties.

Original: Since σ W determines the prior uncertainty and σ x determines the observation noise, higher σ Wand lower σ x reduces the prior’s impact and increases the new observation’s impact on the networkweight’s posterior.
Revised: Since σ W determines the prior uncertainty and σ x determines the observation noise, higher σ W and lower σ x reduces the prior’s impact and increases the new observation’s impact on the network weight’s posterior.

Original: Hence, we can control the memory write strength by modulating σ W , σ x .
Revised: Hence, we can control the memory write strength by modulating σ W , σ x .

Original: Table 4 describes the CIFAR10 recall results of nine structurally identical BayesPCN models withfour hidden layers of size 1024, a single particle, and GELU activations but with different valuesof σ W and σ x .
Revised: Table 5 describes the CIFAR10 recall results of nine structurally identical BayesPCN models with four hidden layers of size 1024, a single particle, and GELU activations but with different values of σ W and σ x .

Original: We observe that lower σ W and higher σ x tend to alleviate the memory overloadingbehaviour, where recall MSE explodes for longer data sequences due to more and more informationbeing written into memory without forgetting. We hypothesize that this is so because those choicesencourage synaptic weights’ Frobenius norms to remain small.
Revised: We observe that lower σ W and higher σ x tend to alleviate the memory overloading behaviour. We hypothesize that this is the case because lower σ W and higher σ x encourage the synaptic weights’ Frobenius norms to remain small, causing activation gradient descent more stable.

Original: As an aside, the white noise recall MSE of BayesPCNwith σ W = 0 .
Revised: [Table] BayesPCN

Original: 5 , σ x = 0 .
Revised: [Table]

Original: 05 decreased as moredatapoints were observed from sequence length 64 and onward. On visual inspection, we foundthat the model’s auto-associative recall outputs forboth observed and unobserved inputs becameless blurry as more datapoints were written into memory. We hypothesize that the model learned togeneralize at some point of its training.
Revised: associative memory read outputs for the high query noise white noise, dropout, and mask tasks on CIFAR10 and Tiny ImageNet datasets.

Original: [Table]
Revised: [Table]

Original: Lastly, we investigate whether a very deep BayesPCN model can perform better on moderate noiserecall tasks compared to the 4 layer BayesPCN model we used in our experiments.

Original: Table 5 detailsthe recall MSE of 4 layer BayesPCN (BayesPCN L4) and 8 layer BayesPCN (BayesPCN L8), bothwith a single particle, hidden layer width of

Original: [Table]
Revised: [Table]

Original:  and GELU activation function.
Revised: As an aside, the white noise recall MSE of BayesPCN with σ x = 0 . 05 and σ W ∈ { 0 . 5 , 1 . 0 } decreased as more datapoints were observed from sequence length 64 and onward.

Original: We find that theirperformances are almost identical, which shows that having more than 4 layers does not help much inthe moderate noise associative recall task. However, having less than 4 layers appears to adverselyaffect the recall performance according to Figure
Revised: On visual inspection, we found that the model’s auto-associative recall outputs for both observed and unobserved inputs became less blurry as more datapoints were written into memory. We hypothesize that the model learned to generalize at some point of its training

Original: [Table]
Revised: [Table]

Original: H.2 Generalization
Revised: J BayesPCN Generalization

Original: Figure 7 illustrates BayesPCN’s read outputs for unseen image queries after different number ofdatapoints have been stored into memory.
Revised: Figure 7 illustrates BayesPCN’s read outputs for unseen image queries after different number of datapoints have been stored into memory.

Original: As BayesPCN observes more data, it learns to “generalize”and gets better at reconstructing and even mildly removing white noise from unseen images.
Revised: As BayesPCN observes more data, it learns to “generalize” and gets better at reconstructing and even mildly removing white noise from unseen images.

Original: Thiscan be attributed to the model continual learning its internal representation that better “describe” thedata distribution.
Revised: This can be attributed to the model continual learning its internal representation that better “describe” the data distribution.

Original: We expected this behaviour to occur since S-NCN [Ororbia et al., 2019], a modelsimilar in structure to GPCN, could continually learn to perform discriminative tasks.
Revised: We expected this behaviour to occur since S-NCN [Ororbia et al., 2019], a model similar in structure to GPCN, could continually learn to perform discriminative tasks.

Original: H.3 Sampling
Revised: K BayesPCN Sampling

Original: Both GPCN and BayesPCN at the core are as much generative models as they are associativememories.
Revised: Both GPCN and BayesPCN at the core are as much generative models as they are associative memories.

Original: We examine the quality of ancestral sampling samples from both models in Figure 8.
Revised: We examine the quality of ancestral sampling samples from both models in Figure 8.

Original: When ancestral sampling, BayesPCN did not marginalize out the synaptic weights and instead fixedthem to the mean parameters of p ( W | x01: t − 1 , h ( n ) 1: t − 1 ) .
Revised: When ancestral sampling, BayesPCN did not marginalize out the synaptic weights and instead fixed them to the mean parameters of p ( W | x t − 1 , h ( n ) 1: t − 1 ) .

Revised: [Figure]

Original: We find that both GPCN and BayesPCN samples are superpositions of the training images.
Revised: We find that both GPCN and BayesPCN samples are superpositions of the training images.

Original: However,as BayesPCN is trained on more and more observations, its sample quality quickly deteriorates.
Revised: However, as BayesPCN is trained on more and more observations, its sample quality quickly deteriorates.

Original: Wehypothesize that the poor sample quality for both GPCN and BayesPCN stems from the approximate[Figure] nature of their parameter estimation.
Revised: We hypothesize that the poor sample quality for both GPCN and BayesPCN stems from the approximate nature of their parameter estimation.

Original: For example, BayesPCN’s particle count would have to be muchgreater than 4 to accurately capture the true posterior p ( W | x 01: t − 1 ) using its sequential importancesampling estimate and the variational distribution over the hidden activations should not be Diracdistributed.
Revised: For example, BayesPCN’s particle count would have to be much greater than 4 to accurately capture the true posterior p ( W | x 01: t − 1 ) using its sequential importance sampling estimate and the variational distribution over the hidden activations should not be Dirac distributed.

Original: However, we note that Ororbia and Kifer [2022] has successfully trained predictivecoding networks to be good generative models.
Revised: However, we note that Ororbia and Kifer [2022] has successfully trained predictive coding networks to be good generative models.