Original: BayesPCN: A Continually Learnable Predictive Coding Associative Memory
Revised: BayesPCN: A Continually Learnable Predictive Coding Associative Memory

Original: Abstract
Revised: Abstract

Original: Associative memory plays an important role in human intelligence and its mechanisms have been linked to attention in machine learning.
Revised: Associative memory plays an important role in human intelligence and its mechanisms have been linked to attention in machine learning.

Original: While the machine learning community’s interest in associative memories has recently been rekindled, most work has focused on memory recall ( read ) over memory learning ( write ).
Revised: While the machine learning community’s interest in associative memories has recently been rekindled, most work has focused on memory recall ( read ) over memory learning ( write ).

Original: In this paper, we present BayesPCN, a hierarchical associative memory capable of performing continual one-shot memory writes without meta-learning.
Revised: In this paper, we present BayesPCN, a hierarchical associative memory capable of performing continual one-shot memory writes without meta-learning.

Original: Moreover,
Revised: Moreover,

Original: BayesPCN is able to gradually forget past observations ( forget ) to free its memory.
Revised: BayesPCN is able to gradually forget past observations ( forget ) to free its memory.

Original: Experiments show that BayesPCN can recall corrupted i.i.d.
Revised: Experiments show that BayesPCN can recall corrupted i.i.d.

Original: high-dimensional data observed hundreds to a thousand “timesteps” ago without a large drop in recall ability compared to the state-of-the-art offline-learned parametric memory models.
Revised: high-dimensional data observed hundreds to a thousand “timesteps” ago without a large drop in recall ability compared to the state-of-the-art offline-learned parametric memory models.

Original:  Introduction
Revised:  Introduction

Original: One of the hallmarks of biological intelligence is the ability to robustly recall learned associations.
Revised: One of the hallmarks of biological intelligence is the ability to robustly recall learned associations.

Original: Biological agents are able to associate stimuli of various modalities and rely on their memories to perform everyday tasks from navigation to entity recognition.
Revised: Biological agents are able to associate stimuli of various modalities and rely on their memories to perform everyday tasks from navigation to entity recognition.

Original: Yet these agents are not only good at recall but can quickly and continually form new memories from as little as a single experience.
Revised: Yet these agents are not only good at recall but can quickly and continually form new memories from as little as a single experience.

Original: How do biological agents achieve this feat while mitigating catastrophic forgetting [French, 1999]?
Revised: How do biological agents achieve this feat while mitigating catastrophic forgetting [French, 1999]?

Original: And how can we build neural networks that demonstrate these properties?
Revised: And how can we build neural networks that demonstrate these properties?

Original: This paper addresses these questions, focusing on fast, continual learning of neural artifacts that perform robust recall of i.i d. high-dimensional observations.
Revised: This paper addresses these questions, focusing on fast, continual learning of neural artifacts that perform robust recall of i.i d. high-dimensional observations.

Original: Associative memory models have been of interest in machine learning for decades [Steinbuch, 1961, Hopfield, 1982].
Revised: Associative memory models have been of interest in machine learning for decades [Steinbuch, 1961, Hopfield, 1982].

Original: They have been used to solve a wide range of problems from sequence processing [Schmidhuber, 1992, Graves et al., 2014, Schlag et al., 2021] to pattern detection [Widrich et al., 2020, 2021, Seidl et al., 2021].
Revised: They have been used to solve a wide range of problems from sequence processing [Schmidhuber, 1992, Graves et al., 2014, Schlag et al., 2021] to pattern detection [Widrich et al., 2020, 2021, Seidl et al., 2021].

Original: Their study has allowed us to build models that replicate aspects of biological intelligence [Kanerva, 1988, Hawkins and Blakeslee, 2004] and provided novel perspectives on machine learning toolsets like dot-product attention, transformers, and linear layers of deep neural networks [Ramsauer et al., 2020, Bricken and Pehlevan, 2021, Irie et al., 2022].
Revised: Their study has allowed us to build models that replicate aspects of biological intelligence [Kanerva, 1988, Hawkins and Blakeslee, 2004] and provided novel perspectives on machine learning toolsets like dot-product attention, transformers, and linear layers of deep neural networks [Ramsauer et al., 2020, Bricken and Pehlevan, 2021, Irie et al., 2022].

Original: Our main aim is to recall exact memories from a neural network that can rapidly and continually learn from a stream of data.
Revised: Our main aim is to recall exact memories from a neural network that can rapidly and continually learn from a stream of data.

Original: Towards this end we introduce BayesPCN, a Bayesian take on the generative predictive coding network (GPCN)
Revised: Towards this end we introduce BayesPCN, a Bayesian take on the generative predictive coding network (GPCN)

Original: [Salvatori et al., 2021] that combines predictive coding and locally conjugate Bayesian updates 1 .
Revised: [Salvatori et al., 2021] that combines predictive coding and locally conjugate Bayesian updates 1 .

Original: There are currently a number of associative memory systems that are continually learnable [Hopfield, 1982, Kanerva, 1988, Hawkins and George, 2007, Ramsauer et al., 2020, Sharma et al., 2022].
Revised: There are currently a number of associative memory systems that are continually learnable [Hopfield, 1982, Kanerva, 1988, Hawkins and George, 2007, Ramsauer et al., 2020, Sharma et al., 2022].

Original: However, to our knowledge, BayesPCN is the first parametric associative memory model to continually learn hundreds of high-dimensional observations while maintaining
Revised: However, to our knowledge, BayesPCN is the first parametric associative memory model to continually learn hundreds of high-dimensional observations while maintaining

Original: [Figure]
Revised: [Figure]

Original: good recall performance on inputs with nontrivial amounts of corruption.
Revised: good recall performance on inputs with nontrivial amounts of corruption.

Original: In addition, it is also the first hierarchical memory which after repeated forgetting recovers its original memory state.
Revised: In addition, it is also the first hierarchical memory which after repeated forgetting recovers its original memory state.

Original:  Background 2.1 Auto-Associative and Hetero-Associative Memory
Revised:  Background 2.1 Auto-Associative and Hetero-Associative Memory

Original: Before we begin let us review two useful concepts: auto-associative memory and hetero-associative memory.
Revised: Before we begin let us review two useful concepts: auto-associative memory and hetero-associative memory.

Original: The former, auto-associative memory, refers to a memory system that is content addressable but for which the query may be a corrupted version of the original data item.
Revised: The former, auto-associative memory, refers to a memory system that is content addressable but for which the query may be a corrupted version of the original data item.

Original: One can think about auto-associative memory systems as a kind of approximate nearest neighbor lookup mechanism.
Revised: One can think about auto-associative memory systems as a kind of approximate nearest neighbor lookup mechanism.

Original: Hetero-associative memory systems are key-value stores, meaning that the recalled “value” is potentially of a different type to the “key.”
Revised: Hetero-associative memory systems are key-value stores, meaning that the recalled “value” is potentially of a different type to the “key.”

Original: Crucially, hetero-associative memory systems can, in large part, be implemented by auto-associative memory systems by forming a “joint” object to memorize by combining the key and value into one object.
Revised: Crucially, hetero-associative memory systems can, in large part, be implemented by auto-associative memory systems by forming a “joint” object to memorize by combining the key and value into one object.

Original: Hetero-associative queries can then be processed simply by looking up the joint object “corrupted” by discarding the value.
Revised: Hetero-associative queries can then be processed simply by looking up the joint object “corrupted” by discarding the value.

Original: 2.2 Generative Predictive Coding Network
Revised: 2.2 Generative Predictive Coding Network

Original: A generative predictive coding network (GPCN)
Revised: A generative predictive coding network (GPCN)

Original: [Salvatori et al., 2021] is an energy-based associative memory model that has set the state-of-the-art on a number of image associative recall tasks.
Revised: [Salvatori et al., 2021] is an energy-based associative memory model that has set the state-of-the-art on a number of image associative recall tasks.

Original: As illustrated in Figure 2, GPCN takes the form of a fully connected neural network whose bottom layer activations characterize the sensory inputs and higher layer activations characterize the internal representations of the sensory inputs.
Revised: As illustrated in Figure 2, GPCN takes the form of a fully connected neural network whose bottom layer activations characterize the sensory inputs and higher layer activations characterize the internal representations of the sensory inputs.

Original: GPCN’s read operation (recall) initializes the bottom layer activations to the input query vector and iteratively denoises the query until a stored value is retrieved by minimizing its energy function w.r.t.
Revised: GPCN’s read operation (recall) initializes the bottom layer activations to the input query vector and iteratively denoises the query until a stored value is retrieved by minimizing its energy function w.r.t.

Original: some or all of its neuron activations.
Revised: some or all of its neuron activations.

Original: GPCN’s write operation (learning) adapts the network parameters to shape the energy function such that the stored datapoints become the local minima of the energy function.
Revised: GPCN’s write operation (learning) adapts the network parameters to shape the energy function such that the stored datapoints become the local minima of the energy function.

Original: Before describing GPCN’s read and write mechanisms in more detail, we introduce the notation we use in this paper.
Revised: Before describing GPCN’s read and write mechanisms in more detail, we introduce the notation we use in this paper.

Original: Let M be a GPCN with L + 1 layers of neurons where 0 and L denotes the bottom and the top layer index respectively.
Revised: Let M be a GPCN with L + 1 layers of neurons where 0 and L denotes the bottom and the top layer index respectively.

Original: Then, let d l be the width of the l -th layer, x l ∈ R
Revised: Then, let d l be the width of the l -th layer, x l ∈ R

Original: d
Revised: d

Original: l be the row vector denoting the l -th layer’s pre-activation function neuron values, and f l : R d l → R d l be the l -th layer’s activation function, for instance ReLU.
Revised: l be the row vector denoting the l -th layer’s pre-activation function neuron values, and f l : R d l → R d l be the l -th layer’s activation function, for instance ReLU.

Original: In addition, let W l ∈ R d l +1 ×
Revised: In addition, let W l ∈ R d l +1 ×

Original: d l where
Revised: d l where

Original: [Equation]
Revised: [Equation]

Original: x 0 t x 1 t x 2 t
Revised: x 0 t x 1 t x 2 t

Original: W
Revised: W

Original: W
Revised: W

Original: Wµ Σ
Revised: Wµ Σ

Original: R
Revised: R

Original: U
Revised: U

Original: R
Revised: R

Original: U 0 1 : T
Revised: U 0 1 : T

Original: 0 ≤ l < L be the synaptic weight matrix between l + 1 -th and l -th layers and W L ∈ R
Revised: 0 ≤ l < L be the synaptic weight matrix between l + 1 -th and l -th layers and W L ∈ R

Original: d L be the “bias” row vector that influences the dynamics of the top layer neuron activities x L .
Revised: d L be the “bias” row vector that influences the dynamics of the top layer neuron activities x L .

Original: One can intuitively think of the model parameters W 0:
Revised: One can intuitively think of the model parameters W 0:

Original: L as the memory content and the activations x 0: L as the read state of the memory.
Revised: L as the memory content and the activations x 0: L as the read state of the memory.

Original: GPCN’s read and write are facilitated by predictive coding (PC)
Revised: GPCN’s read and write are facilitated by predictive coding (PC)

Original: [Rao and Ballard, 1999], a biologically plausible learning rule that can approximate backpropagation [Millidge et al., 2020].
Revised: [Rao and Ballard, 1999], a biologically plausible learning rule that can approximate backpropagation [Millidge et al., 2020].

Original: PC’s energy function is the sum of prediction errors across all network layers.
Revised: PC’s energy function is the sum of prediction errors across all network layers.

Original: For GPCN, the choice of prediction error is the squared Euclidean distance between the actual layer activations and the prediction of the same layer activations made by their immediate upper layer activations.
Revised: For GPCN, the choice of prediction error is the squared Euclidean distance between the actual layer activations and the prediction of the same layer activations made by their immediate upper layer activations.

Original: This leads to the following energy function.
Revised: This leads to the following energy function.

Original: [Equation]
Revised: [Equation]

Original: GPCN’s read performs gradient descent on the energy w.r.t.
Revised: GPCN’s read performs gradient descent on the energy w.r.t.

Original: some or all of x 0 and x 1: L after setting the initial values of x 0 to the memory query ¯ x 0 and fixing the values of W 0: L .
Revised: some or all of x 0 and x 1: L after setting the initial values of x 0 to the memory query ¯ x 0 and fixing the values of W 0: L .

Original: GPCN’s write obtains the read output x 0:
Revised: GPCN’s write obtains the read output x 0:

Original: L 1: T for each of T i.i.d.
Revised: L 1: T for each of T i.i.d.

Original: datapoints to store x 01: T , takes a gradient step on the energy w.r.t.
Revised: datapoints to store x 01: T , takes a gradient step on the energy w.r.t.

Original: W 0:
Revised: W 0:

Original: L while fixing x 0: L 1: T , and repeats until convergence.
Revised: L while fixing x 0: L 1: T , and repeats until convergence.

Original: We now take a probabilistic perspective on GPCN.
Revised: We now take a probabilistic perspective on GPCN.

Original: One can view GPCN as a hierarchical generative model where all neuron activations are modeled by normally distributed random variables and the link function between the subsequent layers is a layer-specific nonlinearity followed by a linear transform.
Revised: One can view GPCN as a hierarchical generative model where all neuron activations are modeled by normally distributed random variables and the link function between the subsequent layers is a layer-specific nonlinearity followed by a linear transform.

Original: Then, GPCN’s read - the minimization of Equation 1 - is equivalent to maximizing the log density in
Revised: Then, GPCN’s read - the minimization of Equation 1 - is equivalent to maximizing the log density in

Original: Equation 2 w.r.t.
Revised: Equation 2 w.r.t.

Original: x 0 and x 1: L .
Revised: x 0 and x 1: L .

Original: [Equation]
Revised: [Equation]

Original: In other words, GPCN’s read performs iterated conditional modes [Besag, 1986] on the hierarchical generative model in the neuron activation space (refer to Algorithms 3 and 4).
Revised: In other words, GPCN’s read performs iterated conditional modes [Besag, 1986] on the hierarchical generative model in the neuron activation space (refer to Algorithms 3 and 4).

Original: In addition, GPCN’s write can be viewed as an approximate maximization of the lower bound on log p ( x 01: T | W 0: L ) w.r.t.
Revised: In addition, GPCN’s write can be viewed as an approximate maximization of the lower bound on log p ( x 01: T | W 0: L ) w.r.t.

Original: W 0:
Revised: W 0:

Original: L [Millidge et al., 2021].
Revised: L [Millidge et al., 2021].

Original: This follows from the fact that GPCN learns via PC.
Revised: This follows from the fact that GPCN learns via PC.

Original: While GPCN possesses impressive associative recall capabilities, the learning rule proposed in
Revised: While GPCN possesses impressive associative recall capabilities, the learning rule proposed in

Original: Salvatori et al.
Revised: Salvatori et al.

Original: [2021] is not amenable to continual learning.
Revised: [2021] is not amenable to continual learning.

Original: Due to its lack of safeguards against catastrophic forgetting, continually learning GPCN leads to a forgetting of information associated with older observations or requires a costly offline model refitting on all observations whenever a new datapoint is observed.
Revised: Due to its lack of safeguards against catastrophic forgetting, continually learning GPCN leads to a forgetting of information associated with older observations or requires a costly offline model refitting on all observations whenever a new datapoint is observed.

Original: BayesPCN’s foremost aim is to address this limitation while preserving
Revised: BayesPCN’s foremost aim is to address this limitation while preserving

Original: GPCN’s read capabilities as much as possible.
Revised: GPCN’s read capabilities as much as possible.

Original:  Memory Model
Revised:  Memory Model

Original: BayesPCN is structurally identical to GPCN, but unlike GPCN the network parameters W 0: L are treated as latent random variables.
Revised: BayesPCN is structurally identical to GPCN, but unlike GPCN the network parameters W 0: L are treated as latent random variables.

Original: This allows BayesPCN’s write operation to be cast as (approximate) posterior inference over W 0: L given new observations in a way similar to [Wu et al., 2018a].
Revised: This allows BayesPCN’s write operation to be cast as (approximate) posterior inference over W 0: L given new observations in a way similar to [Wu et al., 2018a].

Original: In other words, storing T i.i.d.
Revised: In other words, storing T i.i.d.

Original: datapoints x 01: T into memory is equivalent to estimating p ( W 0: L | x 01: T ) .
Revised: datapoints x 01: T into memory is equivalent to estimating p ( W 0: L | x 01: T ) .

Original: Figure 2 illustrates the graphical model for BayesPCN.
Revised: Figure 2 illustrates the graphical model for BayesPCN.

Original: We now review the probabilistic structure of BayesPCN.
Revised: We now review the probabilistic structure of BayesPCN.

Original: We assume that, prior to any observations, the random variables x 0: L t and W 0: L (where t is the datapoint index) are distributed as follows
Revised: We assume that, prior to any observations, the random variables x 0: L t and W 0: L (where t is the datapoint index) are distributed as follows

Original: [Equation]
Revised: [Equation]

Original: where N ( µ, Σ) and MN ( R, U, V ) denotes normal and matrix normal distribution [Gupta and Nagar, 2018] respectively.
Revised: where N ( µ, Σ) and MN ( R, U, V ) denotes normal and matrix normal distribution [Gupta and Nagar, 2018] respectively.

Original: We remind the readers that the matrix normal distribution is a normal distribution for matrix-variate data that compactly specifies the matrix elements’ means, row-wise covariances, and column-wise covariances.
Revised: We remind the readers that the matrix normal distribution is a normal distribution for matrix-variate data that compactly specifies the matrix elements’ means, row-wise covariances, and column-wise covariances.

Original: The joint distribution over all random variables when there are T sensory observations is, due to the conditional independence of x 0: L t , t ∈ { 1 , ..., T } given W 0: L , p ( x 0: L 1: T , W 0:
Revised: The joint distribution over all random variables when there are T sensory observations is, due to the conditional independence of x 0: L t , t ∈ { 1 , ..., T } given W 0: L , p ( x 0: L 1: T , W 0:

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: T (cid:89)
Revised: T (cid:89)

Original: [Equation]
Revised: [Equation]

Original: We note that BayesPCN’s structure leads to two desirable properties that become useful when approximating p ( W 0:
Revised: We note that BayesPCN’s structure leads to two desirable properties that become useful when approximating p ( W 0:

Original: L | x 01: T ) during the write operation.
Revised: L | x 01: T ) during the write operation.

Original: First, by exploiting the linear Gaussian marginalization formula and conditional independence we can analytically compute the marginal distribution over all activation vectors x 0: L t .
Revised: First, by exploiting the linear Gaussian marginalization formula and conditional independence we can analytically compute the marginal distribution over all activation vectors x 0: L t .

Original: [Equation]
Revised: [Equation]

Original: We stress that neither p ( x 0 t ) nor p ( x 0 t | W 0: L ) has an analytic form due to the presence of nonlinearities.
Revised: We stress that neither p ( x 0 t ) nor p ( x 0 t | W 0: L ) has an analytic form due to the presence of nonlinearities.

Original: In addition, if all activation vectors x 0: L t are observed, the posterior over the model parameters becomes normally distributed due to the linear Gaussian posterior formula and the fact that W 0: L are conditionally independent given x 0: L t (the analytical posterior formulae for Equation 9 are in
Revised: In addition, if all activation vectors x 0: L t are observed, the posterior over the model parameters becomes normally distributed due to the linear Gaussian posterior formula and the fact that W 0: L are conditionally independent given x 0: L t (the analytical posterior formulae for Equation 9 are in

Original: Appendix C).
Revised: Appendix C).

Original: [Equation]
Revised: [Equation]

Original: We use R l ∗ , U l ∗ , µ ∗ , Σ ∗ to denote the sufficient statistics of the layer-wise network weight posterior distributions given x lt , x l +1 t .
Revised: We use R l ∗ , U l ∗ , µ ∗ , Σ ∗ to denote the sufficient statistics of the layer-wise network weight posterior distributions given x lt , x l +1 t .

Original:  Memory Operations
Revised:  Memory Operations

Original: This section contains the memory operations that BayesPCN supports: read , write , and forget .
Revised: This section contains the memory operations that BayesPCN supports: read , write , and forget .

Original: We use the shorthand h
Revised: We use the shorthand h

Original: = x 1: L to denote all hidden layer neuron activations and W = W 0: L to denote all network parameters.
Revised: = x 1: L to denote all hidden layer neuron activations and W = W 0: L to denote all network parameters.

Original: 4.1 Read
Revised: 4.1 Read

Original: The read operation takes the corrupted query vector ¯ x 0 and retrieves the memory’s best guess of the associated original datapoint x 0 ∗ .
Revised: The read operation takes the corrupted query vector ¯ x 0 and retrieves the memory’s best guess of the associated original datapoint x 0 ∗ .

Original: As in GPCN, this is done by hill climbing the memory’s log density log p ( x 0 , h | x 01: T ) w.r.t.
Revised: As in GPCN, this is done by hill climbing the memory’s log density log p ( x 0 , h | x 01: T ) w.r.t.

Original: the neuron activations x 0 , h .
Revised: the neuron activations x 0 , h .

Original: To clarify, x 0 , h is the current memory read state and is not directly related to the past observations x 01: T .
Revised: To clarify, x 0 , h is the current memory read state and is not directly related to the past observations x 01: T .

Original: BayesPCN’s auto-associative recall performs iterated conditional modes on log p ( x 0 , h | x 01: T ) w.r.t.
Revised: BayesPCN’s auto-associative recall performs iterated conditional modes on log p ( x 0 , h | x 01: T ) w.r.t.

Original: x 0 , h and BayesPCN’s hetero-associative recall performs gradient ascent on log p ( x 0 , h | x 01: T ) w.r.t.
Revised: x 0 , h and BayesPCN’s hetero-associative recall performs gradient ascent on log p ( x 0 , h | x 01: T ) w.r.t.

Original: x 0 , h while keeping the known values of xfixed.
Revised: x 0 , h while keeping the known values of xfixed.

Original: Algorithm 3 and Algorithm 4 in Appendix A, the recall procedures taken from Salvatori et al.
Revised: Algorithm 3 and Algorithm 4 in Appendix A, the recall procedures taken from Salvatori et al.

Original: [2021] and rewritten from the probabilistic perspective, describe this in more detail.
Revised: [2021] and rewritten from the probabilistic perspective, describe this in more detail.

Original: 4.2 Write
Revised: 4.2 Write

Original: The write operations aims to take the previous timestep posterior over the model parameters p
Revised: The write operations aims to take the previous timestep posterior over the model parameters p

Original: ( W | x 01: t − 1 ) and the current timestep observation x 0 t to return the current timestep posterior over the model parameters p
Revised: ( W | x 01: t − 1 ) and the current timestep observation x 0 t to return the current timestep posterior over the model parameters p

Original: ( W | x 01: t ) .
Revised: ( W | x 01: t ) .

Original: Unfortunately, the analytical form of p ( W | x 01: t ) is unavailable due to the presence of nonlinearities in BayesPCN so one must resort to approximate posterior inference.
Revised: Unfortunately, the analytical form of p ( W | x 01: t ) is unavailable due to the presence of nonlinearities in BayesPCN so one must resort to approximate posterior inference.

Original: We propose a sequential importance sampling posterior inference algorithm for estimating p ( W | x 01: t ) where the particles are high-dimensional Gaussian distributions.
Revised: We propose a sequential importance sampling posterior inference algorithm for estimating p ( W | x 01: t ) where the particles are high-dimensional Gaussian distributions.

Original: Let N ∈ N be the number of sequential importance sampling particles and n ∈ { 1 , ..., N } be the particle index.
Revised: Let N ∈ N be the number of sequential importance sampling particles and n ∈ { 1 , ..., N } be the particle index.

Original: The t -th timestep posterior estimate ˆ p ( W | x 01: t ) is recursively represented as a weighted Gaussian mixture of the form ˆ p ( W | x 01: t ) =
Revised: The t -th timestep posterior estimate ˆ p ( W | x 01: t ) is recursively represented as a weighted Gaussian mixture of the form ˆ p ( W | x 01: t ) =

Original: (cid:80) Nn =1 ω ( n ) p ( n ) ( W | x 01: t , h ( n ) 1: t ) where ω ( n ) is the n -th particle’s importance weight and p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is the n -th particle’s activation conditioned posterior described in Equation 8.
Revised: (cid:80) Nn =1 ω ( n ) p ( n ) ( W | x 01: t , h ( n ) 1: t ) where ω ( n ) is the n -th particle’s importance weight and p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is the n -th particle’s activation conditioned posterior described in Equation 8.

Original: We note that each p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is efficiently characterised by the sufficient statistics µ ( n ) , Σ ( n ) , R 0: L − 1 , ( n ) , U 0:
Revised: We note that each p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 ) is efficiently characterised by the sufficient statistics µ ( n ) , Σ ( n ) , R 0: L − 1 , ( n ) , U 0:

Original: L − 1 , ( n ) .
Revised: L − 1 , ( n ) .

Original: Algorithm 1 describes the write operation in detail (see
Revised: Algorithm 1 describes the write operation in detail (see

Original: Appendix B for its justification).
Revised: Appendix B for its justification).

Original: do
Revised: do

Original: D KL (cid:16) q ( t
Revised: D KL (cid:16) q ( t

Original: ||
Revised: ||

Original: ˆ p n
Revised: ˆ p n

Original: ( x 0 t , h t | x t −
Revised: ( x 0 t , h t | x t −

Original: Sample h t : q
Revised: Sample h t : q

Original: Compute
Revised: Compute

Original: =
Revised: =

Original: ( n ) t − 1 p ( n
Revised: ( n ) t − 1 p ( n

Original: ( x 0 t , h ( n ) t | x 01: t − 1 ) q ( n ) ϕ
Revised: ( x 0 t , h ( n ) t | x 01: t − 1 ) q ( n ) ϕ

Original: ( h ( n ) t ) 10: end
Revised: ( h ( n ) t ) 10: end

Original: [Equation]
Revised: [Equation]

Original: While q ( n ) ( h t ) in principle can be any distribution, BayesPCN’s write sets it to q ( n ) ( h t ) =
Revised: While q ( n ) ( h t ) in principle can be any distribution, BayesPCN’s write sets it to q ( n ) ( h t ) =

Original: δ (cid:0) argmax h t log p ( x 0 t , h t | x 0: L t ) (cid:1) as is standard in predictive coding [Millidge et al., 2021].
Revised: δ (cid:0) argmax h t log p ( x 0 t , h t | x 0: L t ) (cid:1) as is standard in predictive coding [Millidge et al., 2021].

Original: In addition, we define the BayesPCN parameter prior (of an “empty” memory) as ˆ p ( W ) =
Revised: In addition, we define the BayesPCN parameter prior (of an “empty” memory) as ˆ p ( W ) =

Original: (cid:80)
Revised: (cid:80)

Original: Nn =1 1 N p ( n ) ( W ) where µ ( n ) , R 0:
Revised: Nn =1 1 N p ( n ) ( W ) where µ ( n ) , R 0:

Original: L − 1 , ( n ) are initialized using Kaiming initialization.
Revised: L − 1 , ( n ) are initialized using Kaiming initialization.

Original: We chose this prior over a non-mixture prior to obtain a multi-particle posterior approximation despite using a Dirac variational distribution.
Revised: We chose this prior over a non-mixture prior to obtain a multi-particle posterior approximation despite using a Dirac variational distribution.

Original: We now direct our attention to obtaining log p ( n ) ( x 0 t , h | x 01: t − 1 ) used in the read operations from ˆ p ( W | x 01: t ) .
Revised: We now direct our attention to obtaining log p ( n ) ( x 0 t , h | x 01: t − 1 ) used in the read operations from ˆ p ( W | x 01: t ) .

Original: Using Equation 6, we obtain the approximate posterior predictive density
Revised: Using Equation 6, we obtain the approximate posterior predictive density

Original: [Equation]
Revised: [Equation]

Original: BayesPCN’s write can be seen as a special case of sequential imputation [Kong et al., 1994], an online parameter inference algorithm that represents the parameter posterior conditioned on partial observations as a mixture of parameter posteriors conditioned on complete observations.
Revised: BayesPCN’s write can be seen as a special case of sequential imputation [Kong et al., 1994], an online parameter inference algorithm that represents the parameter posterior conditioned on partial observations as a mixture of parameter posteriors conditioned on complete observations.

Original: In addition, whereas the parameter update proposed by Salvatori et al.
Revised: In addition, whereas the parameter update proposed by Salvatori et al.

Original: [2021] is doing local gradient descent for each W l , BayesPCN’s parameter update is doing local conjugate normal-normal Bayesian update for each W l .
Revised: [2021] is doing local gradient descent for each W l , BayesPCN’s parameter update is doing local conjugate normal-normal Bayesian update for each W l .

Original: Hence, like GPCN, BayesPCN’s synaptic weight update only utilizes local information.
Revised: Hence, like GPCN, BayesPCN’s synaptic weight update only utilizes local information.

Original: Lastly, while exact Bayesian updates are invariant to the order of observations, because we are using predictive coding to fit the variational distribution BayesPCN’s write is technically not invariant to the order of observations.
Revised: Lastly, while exact Bayesian updates are invariant to the order of observations, because we are using predictive coding to fit the variational distribution BayesPCN’s write is technically not invariant to the order of observations.

Original: 4.3 Forget
Revised: 4.3 Forget

Original: We also propose a diffusion-based forgetting mechanism for BayesPCN.
Revised: We also propose a diffusion-based forgetting mechanism for BayesPCN.

Original: As more datapoints are written into a BayesPCN model, the model’s belief over the true parameter location R 1: L − 1 , µ shifts and the uncertainty over its belief U 1: L − 1 , Σ decreases.
Revised: As more datapoints are written into a BayesPCN model, the model’s belief over the true parameter location R 1: L − 1 , µ shifts and the uncertainty over its belief U 1: L − 1 , Σ decreases.

Original: This has the effect of making the model’s parameters less adaptive to new datapoints and typically magnifying R 1: L − 1 , µ ’s norms as the prior’s contribution weakens.
Revised: This has the effect of making the model’s parameters less adaptive to new datapoints and typically magnifying R 1: L − 1 , µ ’s norms as the prior’s contribution weakens.

Original: One can interpret this as the memory being overloaded.
Revised: One can interpret this as the memory being overloaded.

Original: The forget operation counteracts this phenomenon by nudging the model’s belief over R 1: L − 1 , µ toward the prior (empty memory state) and increasing its uncertainty U 1: L − 1 , Σ proportional to the forget strength parameter β .
Revised: The forget operation counteracts this phenomenon by nudging the model’s belief over R 1: L − 1 , µ toward the prior (empty memory state) and increasing its uncertainty U 1: L − 1 , Σ proportional to the forget strength parameter β .

Original: Algorithm 2 describes this in detail (the analytical diffusion formulae are in Appendix D).
Revised: Algorithm 2 describes this in detail (the analytical diffusion formulae are in Appendix D).

Original: do
Revised: do

Original: Obtain the one-step diffused layer weight distribution p
Revised: Obtain the one-step diffused layer weight distribution p

Original: | n l t x n l t ) =
Revised: | n l t x n l t ) =

Original: (cid:82) p ( W l ′ | W l , β ) p ( n ) ( W l | x ( n ) ,l 1: t , x ( n ) ,l +1 t )
Revised: (cid:82) p ( W l ′ | W l , β ) p ( n ) ( W l | x ( n ) ,l 1: t , x ( n ) ,l +1 t )

Original: dW l
Revised: dW l

Original: Set p ( n ) ( W
Revised: Set p ( n ) ( W

Original: | x 01: t , h ( n ) 1: t ) =
Revised: | x 01: t , h ( n ) 1: t ) =

Original: (cid:81) Ll =

Revised: (cid:81) Ll =

Original: p ( n ) ( W l ′ | x ( n ) ,l t , x ( n
Revised: p ( n ) ( W l ′ | x ( n ) ,l t , x ( n

Original: ,l +
Revised: ,l +

Original: 10: Set ˆ p ( W ′ | x 01: t ) =
Revised: 10: Set ˆ p ( W ′ | x 01: t ) =

Original: (cid:80) Nn =1 ω ( n ) p ( n ) ( W ′ | x 01: t , h ( n ) 1: t )
Revised: (cid:80) Nn =1 ω ( n ) p ( n ) ( W ′ | x 01: t , h ( n ) 1: t )

Original: If applied an infinite number of times, forget reverts the posterior over the parameters ˆ p
Revised: If applied an infinite number of times, forget reverts the posterior over the parameters ˆ p

Original: ( W | x 01: t ) to the prior ˆ p ( W ) due to the property of diffusion if the particle weights are uniform.
Revised: ( W | x 01: t ) to the prior ˆ p ( W ) due to the property of diffusion if the particle weights are uniform.

Original: In practice, one would apply forget periodically or when the memory starts to become overloaded from too much data and stops being performant.
Revised: In practice, one would apply forget periodically or when the memory starts to become overloaded from too much data and stops being performant.

Original: Because forget erases stored information for all of datapoints somewhat evenly, the datapoints from the distant past that forget was applied to more would be, as the name suggests, more forgotten than the recent datapoints that forget was applied to less.
Revised: Because forget erases stored information for all of datapoints somewhat evenly, the datapoints from the distant past that forget was applied to more would be, as the name suggests, more forgotten than the recent datapoints that forget was applied to less.

Original:  Related Work
Revised:  Related Work

Original: We briefly review the most relevant work to BayesPCN, starting with the predictive coding networks.
Revised: We briefly review the most relevant work to BayesPCN, starting with the predictive coding networks.

Original: As covered in Section 2, GPCN [Salvatori et al., 2021] is the model that BayesPCN is built upon.
Revised: As covered in Section 2, GPCN [Salvatori et al., 2021] is the model that BayesPCN is built upon.

Original: Self-taught associative memory [Dovrolis, 2018, Smith et al., 2021] is a hierarchical predictive coding memory that performs unsupervised continual learning using online clustering modules as building blocks, unlike BayesPCN’s building blocks of artificial neurons.
Revised: Self-taught associative memory [Dovrolis, 2018, Smith et al., 2021] is a hierarchical predictive coding memory that performs unsupervised continual learning using online clustering modules as building blocks, unlike BayesPCN’s building blocks of artificial neurons.

Original: The sequential neural coding network [Ororbia et al., 2019, 2018], while not explicitly used as an associative memory, is also a continually learned predictive coding network similar in structure to GPCN that combats catastrophic forgetting via task-dependent activation sparsity.
Revised: The sequential neural coding network [Ororbia et al., 2019, 2018], while not explicitly used as an associative memory, is also a continually learned predictive coding network similar in structure to GPCN that combats catastrophic forgetting via task-dependent activation sparsity.

Original: BayesPCN on the other hand combats catastrophic forgetting by incorporating uncertainty over the synaptic weights.
Revised: BayesPCN on the other hand combats catastrophic forgetting by incorporating uncertainty over the synaptic weights.

Original: Moving on to general attractor-based memory models, Hopfield networks [Hopfield, 1982] and their modern variants [Krotov and Hopfield, 2016, Ramsauer et al., 2020, Krotov and Hopfield, 2020, Krotov, 2021, Millidge et al., 2022] are biologically motivated auto-associative memories that define the neuron activation dynamics via Lagrangian functions.
Revised: Moving on to general attractor-based memory models, Hopfield networks [Hopfield, 1982] and their modern variants [Krotov and Hopfield, 2016, Ramsauer et al., 2020, Krotov and Hopfield, 2020, Krotov, 2021, Millidge et al., 2022] are biologically motivated auto-associative memories that define the neuron activation dynamics via Lagrangian functions.

Original: Out of these models only
Revised: Out of these models only

Original: Krotov [2021]’s model is hierarchical, but theirs is not continually learned.
Revised: Krotov [2021]’s model is hierarchical, but theirs is not continually learned.

Original: BayesPCN generalizes the modern Hopfield network and fits into the universal Hopfield network’s similarity-separationprojection paradigm for associative memories (refer to Appendix E).
Revised: BayesPCN generalizes the modern Hopfield network and fits into the universal Hopfield network’s similarity-separationprojection paradigm for associative memories (refer to Appendix E).

Original: Sparse Distributed Memory (SDM)
Revised: Sparse Distributed Memory (SDM)

Original: [Kanerva, 1988] is a non-hierarchical memory that exploits high-dimensionality to perform auto-associative and hetero-associative recall.
Revised: [Kanerva, 1988] is a non-hierarchical memory that exploits high-dimensionality to perform auto-associative and hetero-associative recall.

Original: Hierarchical Temporal Memory (HTM)
Revised: Hierarchical Temporal Memory (HTM)

Original: [Hawkins and
Revised: [Hawkins and

Original: George, 2007, Cui et al., 2016] is a biologically constrained hierarchical memory that continually learns time-based patterns with sparse representations.
Revised: George, 2007, Cui et al., 2016] is a biologically constrained hierarchical memory that continually learns time-based patterns with sparse representations.

Original: HTM does not employ predictive coding and to our knowledge has not yet been tested on high-dimensional data.
Revised: HTM does not employ predictive coding and to our knowledge has not yet been tested on high-dimensional data.

Original: Relative to generative modeling and memory research, our work has connections to the Kanerva machine’s memory write as inference approach [Wu et al., 2018a,b] and the one-shot memory write from Bartunov et al.
Revised: Relative to generative modeling and memory research, our work has connections to the Kanerva machine’s memory write as inference approach [Wu et al., 2018a,b] and the one-shot memory write from Bartunov et al.

Original: [2019], Pham et al.
Revised: [2019], Pham et al.

Original: These papers’ models employ meta-learning unlike aforementioned work.
Revised: These papers’ models employ meta-learning unlike aforementioned work.

Original: We note that the Kanerva machine also applies conjugate normal-normal
Revised: We note that the Kanerva machine also applies conjugate normal-normal

Original: Bayesian update to adapt its top level memory matrix once the model is meta-learned via end-to-end gradient descent.
Revised: Bayesian update to adapt its top level memory matrix once the model is meta-learned via end-to-end gradient descent.

Original: On the other hand, BayesPCN applies conjugate normal-normal Bayesian update to all of its weight matrices and this as of itself constitutes learning.
Revised: On the other hand, BayesPCN applies conjugate normal-normal Bayesian update to all of its weight matrices and this as of itself constitutes learning.

Original:  Experiments
Revised:  Experiments

Original: This section assesses BayesPCN’s recall capabilities on image recovery tasks and examines the effect of forget on BayesPCN’s behaviour.
Revised: This section assesses BayesPCN’s recall capabilities on image recovery tasks and examines the effect of forget on BayesPCN’s behaviour.

Original: We used CIFAR

Revised: We used CIFAR

Original: [Krizhevsky et al., 2009] and Tiny ImageNet [Le and Yang, 2015] datasets for all experiments, whose images are of size 3 × 32 × 32 and 3 × 64 × 64 respectively.
Revised: [Krizhevsky et al., 2009] and Tiny ImageNet [Le and Yang, 2015] datasets for all experiments, whose images are of size 3 × 32 × 32 and 3 × 64 × 64 respectively.

Original: During the write phase of the experiments, BayesPCN models were given one image to store into their memories per timestep for up to 1024 images.
Revised: During the write phase of the experiments, BayesPCN models were given one image to store into their memories per timestep for up to 1024 images.

Original: Then, during the read phase of the experiments, we recorded the pixel-wise mean squared error (MSE) between the write phase images and the models’ read outputs given the corrupted write phase images as inputs.
Revised: Then, during the read phase of the experiments, we recorded the pixel-wise mean squared error (MSE) between the write phase images and the models’ read outputs given the corrupted write phase images as inputs.

Original: For reference, the pixel-wise MSE of 0 .is distinguishable to human eyes on close inspection of the images.
Revised: For reference, the pixel-wise MSE of 0 .is distinguishable to human eyes on close inspection of the images.

Original: BayesPCN models were compared against identity models (models that return their inputs as outputs), modern Hopfield networks (MHN)
Revised: BayesPCN models were compared against identity models (models that return their inputs as outputs), modern Hopfield networks (MHN)

Original: [Ramsauer et al., 2020], and vanilla GPCNs [Salvatori et al., 2021] all of which were offline learned.
Revised: [Ramsauer et al., 2020], and vanilla GPCNs [Salvatori et al., 2021] all of which were offline learned.

Original: We note that GPCNs have set the state-of-the-art on CIFAR10 and
Revised: We note that GPCNs have set the state-of-the-art on CIFAR10 and

Original: TinyImageNet image recall tasks.
Revised: TinyImageNet image recall tasks.

Original: BayesPCN models were also compared against online GPCNs,
Revised: BayesPCN models were also compared against online GPCNs,

Original: GPCNs that naively applied their offline memory write to sequentially arriving observations and memorized one image per timestep for up to 1024 images.
Revised: GPCNs that naively applied their offline memory write to sequentially arriving observations and memorized one image per timestep for up to 1024 images.

Original: We used our own autodiff implementation of MHNs based on Appendix E. See Appendix F for additional experiment details.
Revised: We used our own autodiff implementation of MHNs based on Appendix E. See Appendix F for additional experiment details.

Original: 6.1 Auto-Associative and Hetero-Associative Read
Revised: 6.1 Auto-Associative and Hetero-Associative Read

Original: For the auto-associative recall experiments, the memory inputs at the read phase were the write phase images corrupted by pixel-wise white noise with standard deviation 0 .
Revised: For the auto-associative recall experiments, the memory inputs at the read phase were the write phase images corrupted by pixel-wise white noise with standard deviation 0 .

Original: For the hetero-associative recall experiments, we tested two types of training image corruption schemes.
Revised: For the hetero-associative recall experiments, we tested two types of training image corruption schemes.

Original: The first type, dropout, randomly blacked out 25% of the image pixels.
Revised: The first type, dropout, randomly blacked out 25% of the image pixels.

Original: The second type, mask, blacked out 25% of the rightmost image pixels.
Revised: The second type, mask, blacked out 25% of the rightmost image pixels.

Original: Image recovery under these settings is hetero-associative recall because one can view the non-blacked out pixels as the “key” and the blacked out pixels as the “value”.
Revised: Image recovery under these settings is hetero-associative recall because one can view the non-blacked out pixels as the “key” and the blacked out pixels as the “value”.

Original: We experimented with 4-layer BayesPCN models that have hidden layer sizes of 256, 512, and 1024, particle counts of 1 and 4, and activation functions ReLU and GELU.
Revised: We experimented with 4-layer BayesPCN models that have hidden layer sizes of 256, 512, and 1024, particle counts of 1 and 4, and activation functions ReLU and GELU.

Original: We searched over the same hyperparameters for offline and online 4-layer GPCNs aside from the particle count.
Revised: We searched over the same hyperparameters for offline and online 4-layer GPCNs aside from the particle count.

Original: Lastly, we
Revised: Lastly, we

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Table 1 shows the average MSE between the ground truth images and the associative memory read outputs for models trained on data sequences of different lengths, where the reported scores come from the best models of each category.
Revised: Table 1 shows the average MSE between the ground truth images and the associative memory read outputs for models trained on data sequences of different lengths, where the reported scores come from the best models of each category.

Original: The results were averaged across three random seeds.
Revised: The results were averaged across three random seeds.

Original: Appendix I details how different hyperparameter values affect BayesPCN’s read performance.
Revised: Appendix I details how different hyperparameter values affect BayesPCN’s read performance.

Original: BayesPCN’s recall errors were on par with offline GPCN’s recall errors and were orders of magnitude lower than online GPCN’s recall errors on all hetero-associative recall tasks and sequence lengths.
Revised: BayesPCN’s recall errors were on par with offline GPCN’s recall errors and were orders of magnitude lower than online GPCN’s recall errors on all hetero-associative recall tasks and sequence lengths.

Original: BayesPCN ( forget ) performed worse than BayesPCN in these settings but still outperformed online
Revised: BayesPCN ( forget ) performed worse than BayesPCN in these settings but still outperformed online

Original: GPCN on all hetero-associative recall tasks and sequence lengths.
Revised: GPCN on all hetero-associative recall tasks and sequence lengths.

Original: On auto-associative recall tasks with sequences of length ≤ 512 , BayesPCN’s recall errors were marginally higher than offline
Revised: On auto-associative recall tasks with sequences of length ≤ 512 , BayesPCN’s recall errors were marginally higher than offline

Original: GPCN’s recall errors but lower than online GPCN’s recall errors.
Revised: GPCN’s recall errors but lower than online GPCN’s recall errors.

Original: However, when the sequence length approached 1024, BayesPCN performed worse than online GPCN and on the TinyImageNet autoassociative recall task became even worse than the identity model.
Revised: However, when the sequence length approached 1024, BayesPCN performed worse than online GPCN and on the TinyImageNet autoassociative recall task became even worse than the identity model.

Original: We hypothesize that BayesPCN’s sudden performance drop stems from the fact that it is not “removing” old information unlike online
Revised: We hypothesize that BayesPCN’s sudden performance drop stems from the fact that it is not “removing” old information unlike online

Original: GPCN and BayesPCN ( forget ) due to the conjugate Bayesian update’s observation order invariance property.
Revised: GPCN and BayesPCN ( forget ) due to the conjugate Bayesian update’s observation order invariance property.

Original: This causes BayesPCN to approach the overloaded memory regime, where recall of all data suffers, once enough information has been stored.
Revised: This causes BayesPCN to approach the overloaded memory regime, where recall of all data suffers, once enough information has been stored.

Original: Because BayesPCN ( forget ) deliberately removes old information and thus is farther away from the overloaded memory regime, it was able to outperform online GPCN on all auto-associative recall tasks and sequence lengths.
Revised: Because BayesPCN ( forget ) deliberately removes old information and thus is farther away from the overloaded memory regime, it was able to outperform online GPCN on all auto-associative recall tasks and sequence lengths.

Original: GPCN and BayesPCN models overall found hetero-associative recall easier than auto-associative recall, likely because the former fixes the key bits during memory read and thus discourages the algorithm from wandering to less realistic x 0 values.
Revised: GPCN and BayesPCN models overall found hetero-associative recall easier than auto-associative recall, likely because the former fixes the key bits during memory read and thus discourages the algorithm from wandering to less realistic x 0 values.

Original: Counterintuitively, GPCN and BayesPCN models were better at Tiny ImageNet recall over CIFAR10 recall.
Revised: Counterintuitively, GPCN and BayesPCN models were better at Tiny ImageNet recall over CIFAR10 recall.

Original: We attribute this behaviour to the fact that for some models, high-dimensional data recall can ironically be easier than low-dimensional data recall because individual datapoints are further apart from one another in higher dimensions.
Revised: We attribute this behaviour to the fact that for some models, high-dimensional data recall can ironically be easier than low-dimensional data recall because individual datapoints are further apart from one another in higher dimensions.

Original: A surprising discovery from this experiment was how good MHN’s recall can be compared to GPCN and BayesPCN’s recall when the memory query had a moderate amount of noise, a result inconsistent with the findings from Salvatori et al.
Revised: A surprising discovery from this experiment was how good MHN’s recall can be compared to GPCN and BayesPCN’s recall when the memory query had a moderate amount of noise, a result inconsistent with the findings from Salvatori et al.

Original: Because MHN explicitly retains past observations in a data matrix and its softmax inverse temperature was set very high ( β = 10 , 000 ), MHN’s recall effectively selected the observation that had the highest dot-product similarity to the memory query in our experiments.
Revised: Because MHN explicitly retains past observations in a data matrix and its softmax inverse temperature was set very high ( β = 10 , 000 ), MHN’s recall effectively selected the observation that had the highest dot-product similarity to the memory query in our experiments.

Original: In Appendix G, we assess how MHN’s recall compares to BayesPCN’s recall when the memory query corruption is much higher.
Revised: In Appendix G, we assess how MHN’s recall compares to BayesPCN’s recall when the memory query corruption is much higher.

Original: As expected, because the observation with the highest dot-product similarity to the memory query is often not the right image to recall, BayesPCN significantly outperformed MHN in hetero-associative recall tasks.
Revised: As expected, because the observation with the highest dot-product similarity to the memory query is often not the right image to recall, BayesPCN significantly outperformed MHN in hetero-associative recall tasks.

Original: 6.2 Forgetting
Revised: 6.2 Forgetting

Original: This section investigates the forget operation’s effect on BayesPCN’s recall behaviour.
Revised: This section investigates the forget operation’s effect on BayesPCN’s recall behaviour.

Original: Specifically, we show that forget causes BayesPCN to better recall recent observations at the expense of older observations and that this effect is more pronounced the higher the forget strength is.
Revised: Specifically, we show that forget causes BayesPCN to better recall recent observations at the expense of older observations and that this effect is more pronounced the higher the forget strength is.

Original: We trained six structurally identical BayesPCN models on 1024 CIFAR10 images sequentially and applied forget with strength β = 0 , 0 . 001 , 0 . 002 , 0 .
Revised: We trained six structurally identical BayesPCN models on 1024 CIFAR10 images sequentially and applied forget with strength β = 0 , 0 . 001 , 0 . 002 , 0 .

Original: 02 to each model once after every 64 write operations.
Revised: 02 to each model once after every 64 write operations.

Original: Figure 3 compares the models’ recall MSE on 16 data batches of size 64 that were observed during different points of the training once the models have observed all 1024 datapoints.
Revised: Figure 3 compares the models’ recall MSE on 16 data batches of size 64 that were observed during different points of the training once the models have observed all 1024 datapoints.

Original: We see that BayesPCN with no forget ( β = 0) performs similarly on all datapoints regardless of their observation timestep, which is likely the consequence of conjugate Bayesian update’s observation order invariance.
Revised: We see that BayesPCN with no forget ( β = 0) performs similarly on all datapoints regardless of their observation timestep, which is likely the consequence of conjugate Bayesian update’s observation order invariance.

Original: On the other hand, BayesPCNs with forget ( β > 0) consistently perform better on newer datapoints with this effect being strongly correlated with the value of β .
Revised: On the other hand, BayesPCNs with forget ( β > 0) consistently perform better on newer datapoints with this effect being strongly correlated with the value of β .

Original: As previously shown, some amount of forgetting during training can improve the overall performance of the memory across all datapoints over no forgetting for long training sequences.
Revised: As previously shown, some amount of forgetting during training can improve the overall performance of the memory across all datapoints over no forgetting for long training sequences.

Original: Lastly, the higher the forget strength was the lower the recall error on the most recent data batch was for all tasks and forget strength values.
Revised: Lastly, the higher the forget strength was the lower the recall error on the most recent data batch was for all tasks and forget strength values.

Original:  Discussion
Revised:  Discussion

Original: One can intuitively think of BayesPCN read as an approximate nearest neighbour operation with a learned similarity metric and write as the continual learning of that metric.
Revised: One can intuitively think of BayesPCN read as an approximate nearest neighbour operation with a learned similarity metric and write as the continual learning of that metric.

Original: MHN on the other hand uses the dot-product distance as its similarity metric [Millidge et al., 2022].
Revised: MHN on the other hand uses the dot-product distance as its similarity metric [Millidge et al., 2022].

Original: This explains why MHN is not good at highly masked image recall because from a pixelwise dot-product distance perspective, highly masked image queries are often closer to unrelated images than to the ground truth images.
Revised: This explains why MHN is not good at highly masked image recall because from a pixelwise dot-product distance perspective, highly masked image queries are often closer to unrelated images than to the ground truth images.

Original: BayesPCN is closely related to the predictive coding theories of the brain’s memory system.
Revised: BayesPCN is closely related to the predictive coding theories of the brain’s memory system.

Original: For example, Barron et al.
Revised: For example, Barron et al.

Original: [2020] suggests that the hippocampus may correspond to the top layer of a predictive coding network and the neo-cortical areas may correspond to the rest of the predictive coding network.
Revised: [2020] suggests that the hippocampus may correspond to the top layer of a predictive coding network and the neo-cortical areas may correspond to the rest of the predictive coding network.

Original: Similarly, Dovrolis [2018] posits that their predictive coding network could serve as a model of the cortical column albeit without experimental results.
Revised: Similarly, Dovrolis [2018] posits that their predictive coding network could serve as a model of the cortical column albeit without experimental results.

Original: It is however unclear how exactly BayesPCN’s activation gradient descent and locally conjugate Bayesian weight update can be realized in the brain.
Revised: It is however unclear how exactly BayesPCN’s activation gradient descent and locally conjugate Bayesian weight update can be realized in the brain.

Original: Lastly, there are similarities between BayesPCN’s diffusion-based forget and the synaptic homeostasis hypothesis [Cirelli and Tononi, 2022], where during sleep synaptic weight strengths are decreased on average to facilitate future learning.
Revised: Lastly, there are similarities between BayesPCN’s diffusion-based forget and the synaptic homeostasis hypothesis [Cirelli and Tononi, 2022], where during sleep synaptic weight strengths are decreased on average to facilitate future learning.

Original: Aside from these biological connections, BayesPCN has many potential machine learning applications.
Revised: Aside from these biological connections, BayesPCN has many potential machine learning applications.

Original: One possible line of work is using it in conjunction with a neural controller that learns what information to store into memory [Graves et al., 2014, Schlag et al., 2021].
Revised: One possible line of work is using it in conjunction with a neural controller that learns what information to store into memory [Graves et al., 2014, Schlag et al., 2021].

Original: Using BayesPCN in this way would confer the benefit of high query noise tolerance for neural controllers.
Revised: Using BayesPCN in this way would confer the benefit of high query noise tolerance for neural controllers.

Original: Another line of work is using it to solve small data supervised learning problems, leveraging the fact that nearest neighbour methods work well when data availability is sparse [Ramsauer et al., 2020].
Revised: Another line of work is using it to solve small data supervised learning problems, leveraging the fact that nearest neighbour methods work well when data availability is sparse [Ramsauer et al., 2020].

Original: To conclude, we introduced BayesPCN, an associative memory model that can robustly retrieve past observations given various corrupted inputs, perform one-shot parameter update of a deep neural network to store new observations as attractors, and forget old observations to better store newer ones.
Revised: To conclude, we introduced BayesPCN, an associative memory model that can robustly retrieve past observations given various corrupted inputs, perform one-shot parameter update of a deep neural network to store new observations as attractors, and forget old observations to better store newer ones.

Original: We demonstrated that BayesPCN’s recall performance is in most cases not significantly worse than the recall performance of its offline-learned counterpart even though it is continually learned.
Revised: We demonstrated that BayesPCN’s recall performance is in most cases not significantly worse than the recall performance of its offline-learned counterpart even though it is continually learned.

Original: There are several intriguing extensions to BayesPCN that may address some of its limitations.
Revised: There are several intriguing extensions to BayesPCN that may address some of its limitations.

Original: Currently,
Revised: Currently,

Original: BayesPCN read can take up to thousands of activation gradient descent iterations to obtain the best recall result when a lot of data has been stored into memory.
Revised: BayesPCN read can take up to thousands of activation gradient descent iterations to obtain the best recall result when a lot of data has been stored into memory.

Original: Supplementing BayesPCN read with amortized inference may greatly reduce this computational requirement [Tschantz et al., 2022].
Revised: Supplementing BayesPCN read with amortized inference may greatly reduce this computational requirement [Tschantz et al., 2022].

Original: In addition, BayesPCN read can return a nonsensical output if the memory query is too far away from the stored datapoints or if the memory is overloaded.
Revised: In addition, BayesPCN read can return a nonsensical output if the memory query is too far away from the stored datapoints or if the memory is overloaded.

Original: Investigating BayesPCN’s latent landscape may allow us design models with more robust basin of attractions that can easily be reached via activation gradient descent.
Revised: Investigating BayesPCN’s latent landscape may allow us design models with more robust basin of attractions that can easily be reached via activation gradient descent.

Original: Last but not least, extending BayesPCN beyond memorizing i.i.d.
Revised: Last but not least, extending BayesPCN beyond memorizing i.i.d.

Original: data to memorizing sequences like the human episodic memory would be of great interest.
Revised: data to memorizing sequences like the human episodic memory would be of great interest.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We would like to thank Boyan Beronov for the helpful discussions.
Revised: We would like to thank Boyan Beronov for the helpful discussions.

Original: In addition, we acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada
Revised: In addition, we acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada

Original: CIFAR AI Chairs Program, and the Intel Parallel Computing Centers program.
Revised: CIFAR AI Chairs Program, and the Intel Parallel Computing Centers program.

Original: Additional support was provided by UBC’s Composites Research Network (CRN), and Data Science Institute (DSI).
Revised: Additional support was provided by UBC’s Composites Research Network (CRN), and Data Science Institute (DSI).

Original: This research was enabled in part by technical support and computational resources provided by
Revised: This research was enabled in part by technical support and computational resources provided by

Original: WestGrid (www.westgrid.ca), Compute Canada (www.computecanada.ca), and Advanced Research
Revised: WestGrid (www.westgrid.ca), Compute Canada (www.computecanada.ca), and Advanced Research

Original: Computing at the University of British Columbia (arc.ubc.ca).B Derivation of BayesPCN write p ( W | [Equation]
Revised: Computing at the University of British Columbia (arc.ubc.ca).

Original: 01: t ) =

Original: [Equation]

Original: Z

Original: [Equation]

Original: [Equation]

Original: |

Original: [Equation]

Original: 01: t

Original: [Equation]

Original: 1 ) p ( x 0 t | W ) (12) =

Original: Z p ( W | x 01: t − 1 ) E q ( h t )

Original: [ p ( x 0 t , h t | W ) q ( h t ) ] (13) ≈

Original: Z [

Original: N (cid:88) n = ω ( n ) t − 1 p ( n ) ( W | x 01: t − 1 , h ( n ) 1: t − 1 )]

Original: E q ( h t )

Original: [ p ( x 0 t , h t | W ) q ( h t ) ] (14) =

Original: Z

Original: N (cid:88) n = ω ( n ) t − 1 p (

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: t

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: t

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [ p (

Original: [Equation]

Original: t

Original: [Equation]

Original: h t |

Original: [Equation]

Original: ] (15) ≈

Original: Z

Original: N

Original: [Equation]

Original: n

Original: [Equation]

Original: n )

Original: [Equation]

Original: (cid:88) i =

Original: [Equation]

Original: p ( n ) (

Original: [Equation]

Original: | x

Original: [Equation]

Original: t

Original: [Equation]

Original: h ( n )

Original: [Equation]

Original: −

Original: [Equation]

Original: [Equation]

Original: ( x

Original: [Equation]

Original: [Equation]

Original: t |

Original: [Equation]

Original: ) q (

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ) t ) , h (

Original: [Equation]

Original: ( n ) ( h

Original: [Equation]

Original: ) (16) =

Original: Z

Original: S

Original: N (cid:88) n =

Original: S (cid:88) i = ω ( n ) t

Original: [Equation]

Original: 1 p ( n ) (

Original: [Equation]

Original: t

Original: [Equation]

Original: h ( n

Original: [Equation]

Original: i ) t | x 01:

Original: [Equation]

Original: − 1 ) q ( n ) (

Original: [Equation]

Original: ( n

Original: [Equation]

Original: i ) t ) p ( n ) ( W

Original: [Equation]

Original: x 01: t , h ( n,i ) 1: t ) (17) =

Original: N (cid:88) n =

Original: S (cid:88) i = ω ( n ) t − 1 p ( n ) ( x 0 t , h ( n,i ) t | x 01: t − 1 ) q ( n ) ( h ( n,i ) t ) (cid:80) Nn ′ =1 (cid:80) Si ′

Original: =1 ω ( n ′ )

Original: t − 1 p ( n ′ )

Original: ( x 0 t , h ( n ′ ,i ′ )

Original: t | x 01: t − 1 ) q ( n ′ )

Original: ( h ( n ′ ,i ′ )

Original: t ) p ( n ) ( W | x 01: t , h ( n,i ) 1: t ) (18) =

Original: N (cid:88) n =

Original: S (cid:88) i = ω ( n,i ) t p ( n ) ( W | x 01: t , h ( n,i ) 1: t

Original: [Equation]

Original: Equation 17 holds because

Original: [Equation]

Original: and Equation 18 holds because

Original: [Equation]

Original: Finally

Original: [Equation]

Original: if S = 1 we get

Original: [Equation]

Original: C Analytical Posterior Formulae for write

Original: Let z

Original: = f ( x ) .

Original: Then, BayesPCN’s top layer parameter update given x 0: L is

Original: [Equation]

Original: 26) while the update for all other layers given x 0: L is

Original: [Equation]

Original: D Analytical Diffusion Formulae for forget

Original: Let R 1: L − 1, U 1: L − 1, µ 0 , Σ 0 be the memory prior parameters and β be the forget strength.

Original: Then,

Original: BayesPCN’s top layer diffusion update is

Original: [Equation]

Original: while the update for all other layers is

Original: [Equation]

Original: E Connections to Hopfield Networks

Original: We show that modern Hopfield network’s recall is equivalent to the recall of a BayesPCN model with

Original: L = 0 .

Original: BayesPCN’s activation log density is defined as log p ( x 0 , h ) =

Original: log (cid:16)(cid:80) N n =1 ω ( n ) p ( n ) ( x 0 , h ) (cid:17) , and its gradient w.r.t.

Original: the activations is ∇ x 0 , h log p

Original: [Equation]

Original: x

Original: [Equation]

Original: [Equation]

Original: On the other hand, MHN’s energy is

Original: [Equation]

Original: [Equation]

Original: where q ∈ R 1 × d k is the query row vector, K ∈ R N × d k is the key matrix, and K j ∈ R 1 × d k is the j-th key row vector.

Original: q and K correspond to x 0 and W 0 in our paper’s notation.

Original: The negative of the energy can be converted to the following Gaussian mixture log density.

Original: [Equation]

Original: Recall in both BayesPCN and MHN is gradient ascent on the above log density.

Original: When we take the gradient with respect to the input vector q , we recover Equation 33.

Original: [Equation]

Original: where

Original: [Equation]

Original: We conclude that recall in Modern Hopfield Network is equivalent to recall under our framework, which is gradient descent on the log joint of a normal mixture w.r.t.

Original: neuron activations, where there are no hidden layers ( L = 0 ).

Original: However training our model with L = 0 does not lead to the same memory update as the suggested training procedure of MHN, which is setting each key vector K j to some observed datapoint.

Original: Universal Hopfield network [Millidge et al., 2022] proposes a framework for single-shot associative memory that decomposes recall into three components: similarity function, separation function, and projection matrix.

Original: Let x = (¯ x 0 , h ) be the row vector of all initial network activations when give the query ¯ x 0 .

Original: Equation 33 suggests that BayesPCN read ’s implementation of those components is per particle weighted log joint log ω ( n ) p ( n ) ( x ) of the query ¯ x 0 for the similarity function, softmax for the separation function, and the matrix

Original: [Equation]

Original:  for the projection matrix where γ is the learning rate.

Original: We can accommodate the fact that BayesPCN’s read does iterated conditional modes by zeroing out the fixed variable gradients in the projection matrix.

Original: F Additional Experiment Details

Original: All GPCN and BayesPCN models had σ W = 1 , σ x = 0 . 01 , and used Adam with learning rate 0 .

Original: 01 as the neuron activation gradient descent optimizer.

Original: All energy minimization w.r.t.

Original: the neuron activations took 500 gradient steps during the write phase.

Original: During the read phase, the outer loop iterated conditional mode in Algorithm 3 was repeated 30 times and the energy minimization in

Original: Algorithm 4 took 30 × 500 gradient steps.

Original: [-MHN models had β = 10 , 000 (equivalent to σ x = 0 . 01 ), used Adam with learning rate 1 . 
-]
Revised: [-MHN models had β = 10 , 000 (equivalent to σ x = 0 . 01 ), used Adam with learning rate 1 . 
-]

Original: as the gradient descent optimizer, and performed gradient-descent based recall similar to BayesPCN based on the connection from Appendix E. All experiments were run on CIFAR10 and/or Tiny ImageNet datasets (both of which have the MIT License) and the image pixel values were normalized to fall between [ − 1 , 1] .

Original: Offline GPCNs received 4000 iterations of network weight gradient descent steps.

Original: Online GPCNs took a single gradient step w.r.t.

Original: the network weights after the hidden activations converged per observation, a treatment consistent with that of the fast weight memory in Schlag et al.

Original: BayesPCN forget models had four hidden layers of width 1024, a single particle, and GELU activations.

Original: All hyperparameter ranges were chosen based on Salvatori et al.

Original: [2021]’s experiments and GPCN/BayesPCN’s empirical results.

Original: All experiments used NVIDIA Tesla V100 GPUs and were run on the university’s internal clusters.

Original: Training the most expensive GPCN model (hidden layer width of 1024) and BayesPCN model (hidden layer width of 1024, 4 particles) on 1024 observations took 20 hours and 3 hours respectively.

Original: Evaluating the most expensive GPCN and BayesPCN models on all tasks took 25 minutes andhours respectively

Original: [Table]

Original: G BayesPCN vs MHN Recall on Highly Noised Queries

Original: This section reports the recall performance of MHN and BayesPCN models on high query noise associative recall tasks.

Original: The experiment setup is similar to that of Section 6 aside from the fact that the white noise tasks had the noise standard deviation set to 0 .

Original: 8 instead of 0 .

Original: 2 , the dropout tasks randomly blacked out 75% of the pixels instead of 25% , and the masking tasks blacked out 75% of the rightmost pixels instead of 25% .

Original: Because the high query noise task is harder, we show the recall result after the models observed 16, 32, 64, and 128 datapoints.

Original: BayesPCN models had four hidden layers of width 256, a single particle, and GELU activations.

Original: MHNs again used β = 10 , 000 .

Original: [Table]

Original: H Additional Qualitative Results

Original: Figure 5 qualitatively demonstrates how BayesPCN’s read scales with the number of stored datapoints for the CIFAR10 recall tasks.

Original: BayesPCN models are able to output images that are very close to the original image even when the inputs are significantly corrupted.

Original: As the number of observations increases, the read operation is still able to reconstruct the original image but gets worse at recovering the original image given a corrupted version of it.

Original: I Effects of BayesPCN Hyperparameters I.1 Network Width and Depth

Original: Figure 6 and Table 4 illustrate how BayesPCN’s recall accuracy and MSE scale with the network width and depth.

Original: A recall is considered correct if the MSE between the ground truth and the recalled data is less than 0 . 01 .

Original: BayesPCN models had GELU activation functions, σ W = 1 , σ x = 0 .

Original: 01 , and a single particle.

Original: We found that the increased network width was helpful across all tasks.

Original: Increased network depth was helpful when moving from network depth of 2 to 4, but moving from network depth of 4 to 8 had no noticeable impact across all tasks.

Original: We note that because the Figure 6 depicts recall accuracy not

Original: MSE, if the memory performance generally declines and the average recall MSE exceeds 0 .

Original: 01 , this can lead to very low accuracy even if the actual recall MSE is not much greater than 0 .

Original: I.2 Network Weight Prior Uncertainty and Observation Noise

Original: We also investigate the effect of σ W and σ x hyperparameters on BayesPCN’s scaling properties.

Original: Since σ W determines the prior uncertainty and σ x determines the observation noise, higher σ W and lower σ x reduces the prior’s impact and increases the new observation’s impact on the network weight’s posterior.

Original: Hence, we can control the memory write strength by modulating σ W , σ x .

Original: Table 5 describes the CIFAR10 recall results of nine structurally identical BayesPCN models with four hidden layers of size 1024, a single particle, and GELU activations but with different values of σ W and σ x .

Original: We observe that lower σ W and higher σ x tend to alleviate the memory overloading behaviour.

Original: We hypothesize that this is the case because lower σ W and higher σ x encourage the synaptic weights’ Frobenius norms to remain small, causing activation gradient descent more stable.

Original: [Table]

Original: BayesPCN

Original: [Table]

Original: associative memory read outputs for the high query noise white noise, dropout, and mask tasks on CIFAR10 and Tiny ImageNet datasets.

Original: [Table]

Original: [Table]

Original: As an aside, the white noise recall MSE of BayesPCN with σ x = 0 .

Original: 05 and σ W ∈ { 0 . 5 , 1 . 0 } decreased as more datapoints were observed from sequence length 64 and onward.

Original: On visual inspection, we found that the model’s auto-associative recall outputs for both observed and unobserved inputs became less blurry as more datapoints were written into memory.

Original: We hypothesize that the model learned to generalize at some point of its training

Original: [Table]

Original: J BayesPCN Generalization

Original: Figure 7 illustrates BayesPCN’s read outputs for unseen image queries after different number of datapoints have been stored into memory.

Original: As BayesPCN observes more data, it learns to “generalize” and gets better at reconstructing and even mildly removing white noise from unseen images.

Original: This can be attributed to the model continual learning its internal representation that better “describe” the data distribution.

Original: We expected this behaviour to occur since S-NCN [Ororbia et al., 2019], a model similar in structure to GPCN, could continually learn to perform discriminative tasks.

Original: K BayesPCN Sampling

Original: Both GPCN and BayesPCN at the core are as much generative models as they are associative memories.

Original: We examine the quality of ancestral sampling samples from both models in Figure 8.

Original: When ancestral sampling, BayesPCN did not marginalize out the synaptic weights and instead fixed them to the mean parameters of p ( W | x t − 1 , h ( n ) 1: t − 1 ) .

Original: [Figure]

Original: We find that both GPCN and BayesPCN samples are superpositions of the training images.

Original: However, as BayesPCN is trained on more and more observations, its sample quality quickly deteriorates.

Original: We hypothesize that the poor sample quality for both GPCN and BayesPCN stems from the approximate nature of their parameter estimation.

Original: For example, BayesPCN’s particle count would have to be much greater than 4 to accurately capture the true posterior p ( W | x 01: t − 1 ) using its sequential importance sampling estimate and the variational distribution over the hidden activations should not be Dirac distributed.

Original: However, we note that Ororbia and Kifer [2022] has successfully trained predictive coding networks to be good generative models.