Original: C ONVOLUTIONAL B IPARTITE A TTRACTOR N ETWORKS
Revised: C ONVOLUTIONAL B IPARTITE A TTRACTOR N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In human perception and cognition, a fundamental operation that brains perform is interpretation : constructing coherent neural states from noisy, incomplete, and intrinsically ambiguous evidence.
Revised: In human perception and cognition, a fundamental operation that brains perform is interpretation : constructing coherent neural states from noisy, incomplete, and intrinsically ambiguous evidence.

Original: The problem of interpretation is well matched to an early and often overlooked architecture, the attractor network—a recurrent neural net that performs constraint satisfaction, imputation of missing features, and clean up of noisy data via energy minimization dynamics.
Revised: The problem of interpretation is well matched to an early and often overlooked architecture, the attractor network—a recurrent neural net that performs constraint satisfaction, imputation of missing features, and clean up of noisy data via energy minimization dynamics.

Original: We revisit attractor nets in light of modern deep learning methods and propose a convolutional bipartite architecture with a novel training loss, activation function, and connectivity constraints.
Revised: We revisit attractor nets in light of modern deep learning methods and propose a convolutional bipartite architecture with a novel training loss, activation function, and connectivity constraints.

Original: We tackle larger problems than have been previously explored with attractor nets and demonstrate their potential for image completion and super-resolution.
Revised: We tackle larger problems than have been previously explored with attractor nets and demonstrate their potential for image completion and super-resolution.

Original: We argue that this architecture is better motivated than ever-deeper feedforward models and is a viable alternative to more costly sampling-based generative methods on a range of supervised and unsupervised tasks.
Revised: We argue that this architecture is better motivated than ever-deeper feedforward models and is a viable alternative to more costly sampling-based generative methods on a range of supervised and unsupervised tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Under ordinary conditions, human visual perception is quick and accurate.
Revised: Under ordinary conditions, human visual perception is quick and accurate.

Original: Studying circumstances that give rise to slow or inaccurate perception can help reveal the underlying mechanisms of visual information processing.
Revised: Studying circumstances that give rise to slow or inaccurate perception can help reveal the underlying mechanisms of visual information processing.

Original: Recent investigations of occluded (Tang et al., 2018) and empirically challenging (Kar et al., 2019) scenes have led to the conclusion that recurrent brain circuits can play a critical role in object recognition.
Revised: Recent investigations of occluded (Tang et al., 2018) and empirically challenging (Kar et al., 2019) scenes have led to the conclusion that recurrent brain circuits can play a critical role in object recognition.

Original: Further, recurrence can improve the classiﬁcation performance of deep nets (Tang et al., 2018;
Revised: Further, recurrence can improve the classiﬁcation performance of deep nets (Tang et al., 2018;

Original: Nayebi et al., 2018), speciﬁcally for the same images with which humans and animals have the most difﬁculty (Kar et al., 2019).
Revised: Nayebi et al., 2018), speciﬁcally for the same images with which humans and animals have the most difﬁculty (Kar et al., 2019).

Original: Recurrent dynamics allow the brain to perform pattern completion , constructing a coherent neural state from noisy, incomplete, and intrinsically ambiguous evidence.
Revised: Recurrent dynamics allow the brain to perform pattern completion , constructing a coherent neural state from noisy, incomplete, and intrinsically ambiguous evidence.

Original: This interpretive process is well matched to attractor networks ( ANs ) (Hopﬁeld, 1982; 1984; Krotov and Hopﬁeld, 2016; Zemel and Mozer, 2001), a class of dynamical neural networks that converge to ﬁxed-point attractor states (Figure 1a).
Revised: This interpretive process is well matched to attractor networks ( ANs ) (Hopﬁeld, 1982; 1984; Krotov and Hopﬁeld, 2016; Zemel and Mozer, 2001), a class of dynamical neural networks that converge to ﬁxed-point attractor states (Figure 1a).

Original: Given evidence in the form of a static input, an AN settles to an asymptotic state—an interpretation or completion—that is as consistent as possible with the evidence and with implicit knowledge embodied in the network connectivity.
Revised: Given evidence in the form of a static input, an AN settles to an asymptotic state—an interpretation or completion—that is as consistent as possible with the evidence and with implicit knowledge embodied in the network connectivity.

Original: We show examples from our model in Figure 1b.
Revised: We show examples from our model in Figure 1b.

Original: ANs have played a pivotal role in characterizing computation in the brain (Amit, 1992; McClelland and Rumelhart, 1981), not only perception (e.g., Sterzer and Kleinschmidt, 2007), but also language (Stowe et al., 2018) and awareness (Mozer, 2009).
Revised: ANs have played a pivotal role in characterizing computation in the brain (Amit, 1992; McClelland and Rumelhart, 1981), not only perception (e.g., Sterzer and Kleinschmidt, 2007), but also language (Stowe et al., 2018) and awareness (Mozer, 2009).

Original: We revisit attractor nets in light of modern deep learning methods and propose a convolutional bipartite architecture for pattern completion tasks with a novel training loss, activation function, and connectivity constraints.
Revised: We revisit attractor nets in light of modern deep learning methods and propose a convolutional bipartite architecture for pattern completion tasks with a novel training loss, activation function, and connectivity constraints.

Original: [Figure]
Revised: [Figure]

Original: 𝑾 𝑾′
Revised: 𝑾 𝑾′

Original: [Figure]
Revised: [Figure]

Original:  B ACKGROUND AND R ELATED R ESEARCH
Revised:  B ACKGROUND AND R ELATED R ESEARCH

Original: Although ANs have been mostly neglected in the recent literature, attractor-like dynamics can be seen in many models.
Revised: Although ANs have been mostly neglected in the recent literature, attractor-like dynamics can be seen in many models.

Original: For example, clustering and denoising autoencoders are used to clean up internal states and improve the robustness of deep models (Liao et al., 2016;
Revised: For example, clustering and denoising autoencoders are used to clean up internal states and improve the robustness of deep models (Liao et al., 2016;

Original: Tang et al., 2018; Lamb et al., 2019).
Revised: Tang et al., 2018; Lamb et al., 2019).

Original: In a range of image-processing domains, e.g., denoising, inpainting, and super-resolution, performance gains are realized by constructing deeper and deeper architectures (e.g., Lai et al., 2018).
Revised: In a range of image-processing domains, e.g., denoising, inpainting, and super-resolution, performance gains are realized by constructing deeper and deeper architectures (e.g., Lai et al., 2018).

Original: State-of-the-art results are often obtained using deep recursive architectures that replicate layers and weights (Kim et al., 2016; Tai et al., 2017), effectively implementing an unfolded-in-time recurrent net.
Revised: State-of-the-art results are often obtained using deep recursive architectures that replicate layers and weights (Kim et al., 2016; Tai et al., 2017), effectively implementing an unfolded-in-time recurrent net.

Original: This approach is sensible because image processing tasks are fundamentally constraint satisfaction problems: the value of any pixel depends on the values of its neighborhood, and iterative processing is required to converge on mutually consistent activation patterns.
Revised: This approach is sensible because image processing tasks are fundamentally constraint satisfaction problems: the value of any pixel depends on the values of its neighborhood, and iterative processing is required to converge on mutually consistent activation patterns.

Original: Because ANs are speciﬁcally designed to address constraint-satisfaction problems, our goal is to re-examine them from a modern deep-learning perspective.
Revised: Because ANs are speciﬁcally designed to address constraint-satisfaction problems, our goal is to re-examine them from a modern deep-learning perspective.

Original: Interest in ANs seems to be narrow for two reasons.
Revised: Interest in ANs seems to be narrow for two reasons.

Original: First, in both early (Hopﬁeld, 1982; 1984) and recent (Li et al., 2015; Wu et al., 2018a;b; Chaudhuri and Fiete, 2017) work, ANs are characterized as content-addressable memories: activation vectors are stored and can later be retrieved with only partial information.
Revised: First, in both early (Hopﬁeld, 1982; 1984) and recent (Li et al., 2015; Wu et al., 2018a;b; Chaudhuri and Fiete, 2017) work, ANs are characterized as content-addressable memories: activation vectors are stored and can later be retrieved with only partial information.

Original: However, memory retrieval does not well characterize the model’s capabilities: like its probabilistic sibling the Boltzmann machine (Hinton, 2007; Welling et al., 2005), the AN is a general computational architecture for supervised and unsupervised learning.
Revised: However, memory retrieval does not well characterize the model’s capabilities: like its probabilistic sibling the Boltzmann machine (Hinton, 2007; Welling et al., 2005), the AN is a general computational architecture for supervised and unsupervised learning.

Original: Second, ANs have been limited by training procedures.
Revised: Second, ANs have been limited by training procedures.

Original: In Hopﬁeld’s work, ANs are trained with a simple procedure—an outer product (Hebbian) rule—which cannot accommodate hidden units and the representational capacity they provide.
Revised: In Hopﬁeld’s work, ANs are trained with a simple procedure—an outer product (Hebbian) rule—which cannot accommodate hidden units and the representational capacity they provide.

Original: Recent explorations have considered stronger training procedures (e.g., Wu et al., 2018b; Liao et al., 2018); however, as for all recurrent nets, training is complicated by the issue of vanishing/exploding gradients.
Revised: Recent explorations have considered stronger training procedures (e.g., Wu et al., 2018b; Liao et al., 2018); however, as for all recurrent nets, training is complicated by the issue of vanishing/exploding gradients.

Original: To facilitate training and increase the computational power of ANs, we propose a set of extensions to the architecture and training procedures.
Revised: To facilitate training and increase the computational power of ANs, we propose a set of extensions to the architecture and training procedures.

Original: ANs are related to several popular architectures.
Revised: ANs are related to several popular architectures.

Original: Autoencoding models such as the VAE (Kingma and
Revised: Autoencoding models such as the VAE (Kingma and

Original: Welling, 2013) and denoising autoencoders (Vincent et al., 2008) can be viewed as approximating one step of attractor dynamics, directing the input toward the training data manifold (Alain et al., 2012).
Revised: Welling, 2013) and denoising autoencoders (Vincent et al., 2008) can be viewed as approximating one step of attractor dynamics, directing the input toward the training data manifold (Alain et al., 2012).

Original: These models can be applied recursively, though convergence is not guaranteed, nor is improvement in output quality over iterations.
Revised: These models can be applied recursively, though convergence is not guaranteed, nor is improvement in output quality over iterations.

Original: Flow-based generative models (FBGMs)
Revised: Flow-based generative models (FBGMs)

Original: (e.g., Dinh et al., 2016) are invertible density-estimation models that can map between observations and latent states.
Revised: (e.g., Dinh et al., 2016) are invertible density-estimation models that can map between observations and latent states.

Original: Whereas
Revised: Whereas

Original: FBGMs require invertibility of mappings, ANs require only a weaker constraint that weights in one direction are the transpose of the weights in the other direction.
Revised: FBGMs require invertibility of mappings, ANs require only a weaker constraint that weights in one direction are the transpose of the weights in the other direction.

Original: Energy-based models (EBMs) are also density-estimation models that learn a mapping from input data to energies and are trained to assign low energy values to the data manifold (LeCun et al., 2006; Han et al., 2018; Xie et al., 2016; Du and Mordatch, 2019).
Revised: As currently formulated, energy-based models (EBMs) are feedforward density-estimation models that learn a mapping from input data to energies and are trained to assign low energy values to the data manifold (LeCun et al., 2006; Han et al., 2018; Xie et al., 2016; Du and Mordatch, 2019).

Original: Whereas AN dynamics are determined by an implicit energy function, the EBM dynamics are driven by optimizing or sampling from an explicit energy function.
Revised: Whereas AN dynamics are determined by an implicit energy function, the EBM dynamics are driven by optimizing or sampling from an explicit energy function.

Original: In the AN, lowering the energy for some states raises it for others, whereas the explicit EBM energy function requires well-chosen negative samples to ensure it discriminates likely from unlikely states.
Revised: In the AN, lowering the energy for some states raises it for others, whereas the explicit EBM energy function requires well-chosen negative samples to ensure it discriminates likely from unlikely states.

Original: Although the EBM and FBGM seem well suited for synthesis and generation tasks, due to their probabilistic underpinnings, we show that ANs can  be used forconditional generation (maximum likelihood completion) tasks.
Revised: Although the EBM and FBGM seem well suited for synthesis and generation tasks due to their probabilistic underpinnings, we show that ANs can also be used for generation (maximum-likelihood completion) tasks.

Original:  C ONVOLUTIONAL B IPARTITE A TTRACTOR N ETS 
Revised:  C ONVOLUTIONAL B IPARTITE A TTRACTOR N ETS (CBAN S )

Original: Various types of recurrent nets have been shown to converge to activation ﬁxed points, including fully interconnected networks of asynchronous binary units (Hopﬁeld, 1982) and networks of continuousvalued units operating in continuous time (Hopﬁeld, 1984).
Revised: Various types of recurrent nets have been shown to converge to activation ﬁxed points, including fully interconnected networks of asynchronous binary units (Hopﬁeld, 1982) and networks of continuousvalued units operating in continuous time (Hopﬁeld, 1984).

Original: Most relevant to modern deep learning,
Revised: Most relevant to modern deep learning,

Original: Koiran (1994) identiﬁed convergence conditions for synchronous update of continuous-valued units in discrete time: given a network with state x , parallel updates of the full state with the standard activation rule,
Revised: Koiran (1994) identiﬁed convergence conditions for synchronous update of continuous-valued units in discrete time: given a network with state x , parallel updates of the full state with the standard activation rule,

Original: [Equation]
Revised: [Equation]

Original: will asymptote at either a ﬁxed point or a limit cycle of 2.
Revised: will asymptote at either a ﬁxed point or a limit cycle of 2.

Original: Sufﬁcient conditions for this result are: initial x ∈ [ − 1 , +1] n , W = W T , w ii ≥ 0 , and f ( . )
Revised: Sufﬁcient conditions for this result are: initial x ∈ [ − 1 , +1] n , W = W T , w ii ≥ 0 , and f ( . )

Original: piecewise continuous and strictly increasing with lim
Revised: piecewise continuous and strictly increasing with lim

Original: η →±∞ f ( η ) = ± 1 .
Revised: η →±∞ f ( η ) = ± 1 .

Original: The proof is cast in terms of an energy function,
Revised: The proof is cast in terms of an energy function,

Original: [Equation]
Revised: [Equation]

Original: With f ≡ tanh , we have the barrier function:
Revised: With f ≡ tanh , we have the barrier function:

Original: [Equation]
Revised: [Equation]

Original: To ensure a ﬁxed point (no limit cycle > 1), asynchronous updates are sufﬁcient because the solution of ∂E/∂x i = 0 is the standard update for unit
Revised: To ensure a ﬁxed point (no limit cycle > 1), asynchronous updates are sufﬁcient because the solution of ∂E/∂x i = 0 is the standard update for unit

Original: i (Equation 1).
Revised: i (Equation 1).

Original: Because the energy function additively factorizes for units that have no direct connections, parallel updates of these units still ensure non-increasing energy, and hence attainment of a ﬁxed point.
Revised: Because the energy function additively factorizes for units that have no direct connections, parallel updates of these units still ensure non-increasing energy, and hence attainment of a ﬁxed point.

Original: We adopt the bipartite architecture of a stacked restricted Boltzmann machine (Hinton and Salakhutdinov, 2006), with bidirectional symmetric connections between adjacent layers of units and no connectivity within a layer (Figure 1c).
Revised: We adopt the bipartite architecture of a stacked restricted Boltzmann machine (Hinton and Salakhutdinov, 2006), with bidirectional symmetric connections between adjacent layers of units and no connectivity within a layer (Figure 1c).

Original: We distinguish between visible layers, which contain inputs and/or outputs of the net, and hidden layers.
Revised: We distinguish between visible layers, which contain inputs and/or outputs of the net, and hidden layers.

Original: The bipartite architecture allows for units within a layer to be updated in parallel while guaranteeing strictly non-increasing energy and attainment of a local energy minimum.
Revised: The bipartite architecture allows for units within a layer to be updated in parallel while guaranteeing strictly non-increasing energy and attainment of a local energy minimum.

Original: We thus perform layerwise updating of units, deﬁning one iteration as a sweep from one end of the architecture to the other and back.
Revised: We thus perform layerwise updating of units, deﬁning one iteration as a sweep from one end of the architecture to the other and back.

Original: The 8-step update sequence for the architecture in Figure 1c is shown above the network.
Revised: The 8-step update sequence for the architecture in Figure 1c is shown above the network.

Original: 3. C ONVOLUTIONAL W EIGHT C ONSTRAINTS
Revised: 3. C ONVOLUTIONAL W EIGHT C ONSTRAINTS

Original: Weight constraints required for convergence can be achieved within a convolutional architecture as well (Figure 1d).
Revised: Weight constraints required for convergence can be achieved within a convolutional architecture as well (Figure 1d).

Original: In a feedforward convolutional architecture, the connectivity from layer l to l + 1 is represented by weights W l = { w lqrab } , where q and r are channel indices in the destination ( l + 1 ) and source ( l ) layers, respectively, and a and b specify the relative coordinate within the kernel, such that the weight w lqrab modulates the input to the unit in layer l + 1 , channel q , absolute position ( α, β ) —denoted x l +1 qαβ —from the unit x lr,α + a,β + b .
Revised: In a feedforward convolutional architecture, the connectivity from layer l to l + 1 is represented by weights W l = { w lqrab } , where q and r are channel indices in the destination ( l + 1 ) and source ( l ) layers, respectively, and a and b specify the relative coordinate within the kernel, such that the weight w lqrab modulates the input to the unit in layer l + 1 , channel q , absolute position ( α, β ) —denoted x l +1 qαβ —from the unit x lr,α + a,β + b .

Original: If W l +1 = { w l +1 qrab } denotes the reverse weights to channel q in layer l from channel r in layer l + 1 , symmetry requires that
Revised: If W l +1 = { w l +1 qrab } denotes the reverse weights to channel q in layer l from channel r in layer l + 1 , symmetry requires that

Original: [Equation]
Revised: [Equation]

Original: This follows from the fact that the weights are translation invariant: the reverse mapping from x l +1 q,α,β to x lr,α + a,β + b has the same weight as from x l +1 q,α − a,β − b to x lr,α,β , embodied in Equation 4.
Revised: This follows from the fact that the weights are translation invariant: the reverse mapping from x l +1 q,α,β to x lr,α + a,β + b has the same weight as from x l +1 q,α − a,β − b to x lr,α,β , embodied in Equation 4.

Original: Implementation of the weight constraint is simple: W l is unconstrained, and W l +1 is obtained by transposing the ﬁrst two tensor dimensions of W l and ﬂipping the indices of the last two.
Revised: Implementation of the weight constraint is simple: W l is unconstrained, and W l +1 is obtained by transposing the ﬁrst two tensor dimensions of W l and ﬂipping the indices of the last two.

Original: The convolutional bipartite architecture has energy function:
Revised: The convolutional bipartite architecture has energy function:

Original: [Equation]
Revised: [Equation]

Original: where x l is the activation in layer l , b l are the channel biases, and ρ ( . ) is the barrier function (Equation 3), ‘ ∗ ’ is the convolution operator, and ‘ • ’ is the element-wise sum of the Hadamard product of tensors.
Revised: where x l is the activation in layer l , b l are the channel biases, and ρ ( . ) is the barrier function (Equation 3), ‘ ∗ ’ is the convolution operator, and ‘ • ’ is the element-wise sum of the Hadamard product of tensors.

Original: The factor of 12 ordinarily found in energy functions is not present in the ﬁrst term because, in contrast to Equation 2, each second-order term in x appears only once.
Revised: The factor of 12 ordinarily found in energy functions is not present in the ﬁrst term because, in contrast to Equation 2, each second-order term in x appears only once.

Original: For a similar formulation in stacked restricted Boltzmann machines, see Lee et al.
Revised: For a similar formulation in stacked restricted Boltzmann machines, see Lee et al.

Original: 3. L OSS F UNCTIONS
Revised: 3. L OSS F UNCTIONS

Original: Evidence provided to the CBAN  consists of activation constraints on a subset of the visible units.
Revised: Evidence provided to our Convolutional Bipartite Attractor Net (hereafter, CBAN ) consists of activation constraints on a subset of the visible units.

Original: The CBAN is trained to ﬁll-in or complete the activation pattern over the visible state.
Revised: The CBAN is trained to ﬁll-in or complete the activation pattern over the visible state.

Original: The manner in which evidence constrains activations depends on the nature of the evidence.
Revised: The manner in which evidence constrains activations depends on the nature of the evidence.

Original: In a scenario where all features are present but potentially noisy, one should treat them as soft constraints that can be overridden by the model; in a scenario where the evidence features are reliable but other features are entirely missing, one should treat the evidence as hard constraints.
Revised: In a scenario where all features are present but potentially noisy, one should treat them as soft constraints that can be overridden by the model; in a scenario where the evidence features are reliable but other features are entirely missing, one should treat the evidence as hard constraints.

Original: We have focused on this latter scenario in our simulations, although we discuss the use of soft constraints in Appendix A. For a hard constraint, we clamp the visible units to the value of the evidence, meaning that activation is set to the observed value and not allowed to change.
Revised: We have focused on this latter scenario in our simulations, although we discuss the use of soft constraints in Appendix A. For a hard constraint, we clamp the visible units to the value of the evidence, meaning that activation is set to the observed value and not allowed to change.

Original: Energy is minimized conditioned on the clamped values.
Revised: Energy is minimized conditioned on the clamped values.

Original: One extension to clamping is to replicate all visible units and designate one set as input, clamped to the evidence, and one set as output, which serves as the network read out.
Revised: One extension to clamping is to replicate all visible units and designate one set as input, clamped to the evidence, and one set as output, which serves as the network read out.

Original: We considered using the evidence to initialize the visible state, but initialization is inadequate to anchor the visible state and it wanders.
Revised: We considered using the evidence to initialize the visible state, but initialization is inadequate to anchor the visible state and it wanders.

Original: We also considered using the evidence as a ﬁxed bias on the input to the visible state, but redundancy of the bias and top-down signals from the hidden layer can prevent the CBAN from achieving the desired activations.
Revised: We also considered using the evidence as a ﬁxed bias on the input to the visible state, but redundancy of the bias and top-down signals from the hidden layer can prevent the CBAN from achieving the desired activations.

Original: An obvious loss function is squared error, L SE =
Revised: An obvious loss function is squared error, L SE =

Original: (cid:80)
Revised: (cid:80)

Original: i ||
Revised: i ||

Original: v
Revised: v

Original: i − y
Revised: i − y

Original: i ||
Revised: i ||

Original: 2 , where i is an index over visible units, v is the visible state, and y is the target visible state.
Revised: 2 , where i is an index over visible units, v is the visible state, and y is the target visible state.

Original: However, this loss misses out on a key source of error.
Revised: However, this loss misses out on a key source of error.

Original: The clamped units have zero error under this loss.
Revised: The clamped units have zero error under this loss.

Original: Consequently, we replace v i with ˜ v i , the value that unit i would take were it unclamped, i.e., free to take on a value consistent with the hidden units driving it:
Revised: Consequently, we replace v i with ˜ v i , the value that unit i would take were it unclamped, i.e., free to take on a value consistent with the hidden units driving it:

Original: [Equation]
Revised: [Equation]

Original: An alternative loss, related to the contrastive loss of the Boltzmann machine (see Appendix B), explicitly aims to ensure that the energy of the current state is higher than that of the target state.
Revised: An alternative loss, related to the contrastive loss of the Boltzmann machine (see Appendix B), explicitly aims to ensure that the energy of the current state is higher than that of the target state.

Original: With x = ( y , h ) being the complete state with all visible units clamped at their target values and the hidden units in some conﬁguration h , and ˜ x
Revised: With x = ( y , h ) being the complete state with all visible units clamped at their target values and the hidden units in some conﬁguration h , and ˜ x

Original: = (˜ v , h ) being the complete state with the visible units unclamped, one can deﬁne the loss
Revised: = (˜ v , h ) being the complete state with the visible units unclamped, one can deﬁne the loss

Original: [Equation]
Revised: [Equation]

Original: We apply this loss by allowing the net to iterate for some number of steps given a partially clamped input, yielding a hidden state that is a plausible candidate to generate the target visible state.
Revised: We apply this loss by allowing the net to iterate for some number of steps given a partially clamped input, yielding a hidden state that is a plausible candidate to generate the target visible state.

Original: Note that ρ ( y i ) is constant and although it does not factor into the gradient computation, it helps interpret
Revised: Note that ρ ( y i ) is constant and although it does not factor into the gradient computation, it helps interpret

Original: L ∆ E : when L ∆ E = 0 , ˜ v = y .
Revised: L ∆ E : when L ∆ E = 0 , ˜ v = y .

Original: This loss is curious in that it is a function not just of the visible state, but, through the term f − 1 (˜ v i ) , it directly depends on the hidden state in the adjacent layer and the weights between these layers.
Revised: This loss is curious in that it is a function not just of the visible state, but, through the term f − 1 (˜ v i ) , it directly depends on the hidden state in the adjacent layer and the weights between these layers.

Original: A variant on L ∆ E is based on the observation that the goal of training is only to make the two energies equal, suggesting a soft hinge loss:
Revised: A variant on L ∆ E is based on the observation that the goal of training is only to make the two energies equal, suggesting a soft hinge loss:

Original: [Equation]
Revised: [Equation]

Original: Both energy-based losses have an interpretation under the Boltzmann distribution:
Revised: Both energy-based losses have an interpretation under the Boltzmann distribution:

Original: L ∆ E is related to the conditional likelihood ratio of the clamped to unclamped visible state, and L ∆ E + is related to the conditional probability of the clamped versus unclamped visible state:
Revised: L ∆ E is related to the conditional likelihood ratio of the clamped to unclamped visible state, and L ∆ E + is related to the conditional probability of the clamped versus unclamped visible state:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: 3. P REVENTING VANISHING / EXPLODING GRADIENTS
Revised: 3. P REVENTING VANISHING / EXPLODING GRADIENTS

Original: Although gradient descent is a more powerful method to train the CBAN than Hopﬁeld’s Hebb rule or the Boltzmann machine’s contrastive loss, vanishing and exploding gradients are a concern as with any recurrent net (Hochreiter et al., 2001), particularly in the CBAN which may take 50 steps to fully relax.
Revised: Although gradient descent is a more powerful method to train the CBAN than Hopﬁeld’s Hebb rule or the Boltzmann machine’s contrastive loss, vanishing and exploding gradients are a concern as with any recurrent net (Hochreiter et al., 2001), particularly in the CBAN which may take 50 steps to fully relax.

Original: We address the gradient issue in two ways: through intermediate training signals and through a soft sigmoid activation function.
Revised: We address the gradient issue in two ways: through intermediate training signals and through a soft sigmoid activation function.

Original: The aim of the CBAN is to produce a stable interpretation asymptotically.
Revised: The aim of the CBAN is to produce a stable interpretation asymptotically.

Original: The appropriate way to achieve this is to apply the loss once activation converges.
Revised: The appropriate way to achieve this is to apply the loss once activation converges.

Original: However, the loss can be applied prior to convergence as well, essentially training the net to achieve convergence as quickly as possible, while also introducing loss gradients deep inside the unrolled net.
Revised: However, the loss can be applied prior to convergence as well, essentially training the net to achieve convergence as quickly as possible, while also introducing loss gradients deep inside the unrolled net.

Original: Assume a stability criterion θ that determines the iteration t ∗ at which the net has effectively converged:
Revised: Assume a stability criterion θ that determines the iteration t ∗ at which the net has effectively converged:

Original: [Equation]
Revised: [Equation]

Original: Training can be logically separated into pre- and post-convergence phases, which we will refer to as transient and stationary .
Revised: Training can be logically separated into pre- and post-convergence phases, which we will refer to as transient and stationary .

Original: In the stationary phase, the Almeida/Pineda algorithm (Pineda, 1987; Almeida, 1987) leverages the fact that activation is constant over iterations, permitting a computationally efﬁcient gradient calculation with low memory requirements.
Revised: In the stationary phase, the Almeida/Pineda algorithm (Pineda, 1987; Almeida, 1987) leverages the fact that activation is constant over iterations, permitting a computationally efﬁcient gradient calculation with low memory requirements.

Original: In the transient phase, the loss can be injected at each step, which is exactly the temporal-difference method TD(1)
Revised: In the transient phase, the loss can be injected at each step, which is exactly the temporal-difference method TD(1)

Original: (Sutton, 1988).
Revised: (Sutton, 1988).

Original: Casting training as temporal-difference learning, one might consider other values of λ in TD( λ ); for example, TD(0) trains the model to predict the visible state at the next time step, encouraging the model to reach the target state as quickly as feasible while not penalizing it for being unable to get to the target immediately.
Revised: Casting training as temporal-difference learning, one might consider other values of λ in TD( λ ); for example, TD(0) trains the model to predict the visible state at the next time step, encouraging the model to reach the target state as quickly as feasible while not penalizing it for being unable to get to the target immediately.

Original: Any of the losses, L SE , L ∆ E , and L ∆ E + , can be applied with a weighted mixture of training in the stationary and transient phases.
Revised: Any of the losses, L SE , L ∆ E , and L ∆ E + , can be applied with a weighted mixture of training in the stationary and transient phases.

Original: Although we do not report systematic experiments in this article, we consistently ﬁnd that transient training with λ = 1 is as efﬁcient and effective as weighted mixtures including stationary-phase-only training, and that λ
Revised: Although we do not report systematic experiments in this article, we consistently ﬁnd that transient training with λ = 1 is as efﬁcient and effective as weighted mixtures including stationary-phase-only training, and that λ

Original: = 1 outperforms any λ < 1 .
Revised: = 1 outperforms any λ < 1 , likely due to moving targets during training.

Original: In our results, we thus conduct  simulations with transient-phase training and λ = 1 .
Revised: We thus conduct all simulations with transient-phase training and λ = 1 .

Original: We propose a second method of avoiding vanishing gradients speciﬁcally due to sigmoidal activation functions: a leaky sigmoid , analogous to a leaky ReLU, which allows gradients to propagate through the net more freely.
Revised: We propose a second method of avoiding vanishing gradients speciﬁcally due to sigmoidal activation functions: a leaky sigmoid , analogous to a leaky ReLU, which allows gradients to propagate through the net more freely.

Original: The leaky sigmoid has activation and barrier functionsf
Revised: The leaky sigmoid has activation and barrier functions

Original: [Equation]
Revised: [Equation]

Original: Parameter α speciﬁes the slope of the piecewise linear function outside the | x | < 1 interval.
Revised: Parameter α speciﬁes the slope of the piecewise linear function outside the | x | < 1 interval.

Original: As α → 0 , loss gradients become ﬂat and the CBAN fails to train well.
Revised: As α → 0 , loss gradients become ﬂat and the CBAN fails to train well.

Original: As α → 1 , activation magnitudes can blow up and the CBAN fails to reach a ﬁxed point.
Revised: As α → 1 , activation magnitudes can blow up and the CBAN fails to reach a ﬁxed point.

Original: In Appendix C, we show that convergence to a ﬁxed point is guaranteed when α || W || 1 , ∞ < 1 , where || W || 1 , ∞ =
Revised: In Appendix C, we show that convergence to a ﬁxed point is guaranteed when α || W || 1 , ∞ < 1 , where || W || 1 , ∞ =

Original: max i ||
Revised: max i ||

Original: w
Revised: w

Original: i
Revised: i

Original: || 1 .
Revised: || 1 .

Original: In practice, we have found that restricting W is unnecessary and α = 0 .
Revised: In practice, we have found that restricting W is unnecessary and α = 0 .

Original: 2 works well.
Revised: 2 works well.

Original:  S IMULATIONS
Revised:  S IMULATIONS

Original: We report on a series of simulation studies of increasing complexity.
Revised: We report on a series of simulation studies of increasing complexity.

Original: First, we explore fully connected bipartite attractor net (FBAN) on a bar imputation task and then supervised MNIST image completion and classiﬁcation.
Revised: First, we explore fully connected bipartite attractor net (FBAN) on a bar imputation task and then supervised MNIST image completion and classiﬁcation.

Original: Second, we apply CBAN to unsupervised image completion tasks on Omniglot and CIFAR-10 and compare CBAN to CBAN-variants and denoising-VAEs.
Revised: Second, we apply CBAN to unsupervised image completion tasks on Omniglot and CIFAR-10 and compare CBAN to CBAN-variants and denoising-VAEs.

Original: Lastly, we revision
Revised: Lastly, we revision

Original: CBAN for the task of super-resolution and report promising results against competing models, such as DRCN and LapSRN.
Revised: CBAN for the task of super-resolution and report promising results against competing models, such as DRCN and LapSRN.

Original: Details of architectures, parameters, and training are in Appendix D. 
Revised: Details of architectures, parameters, and training are in Appendix D. All models are trained via SGD (back propagation through time).

Original: 4. B AR T ASK
Revised: 4. B AR T ASK

Original: We studied a simple inference task on partial images that have exactly one correct interpretation.
Revised: We studied a simple inference task on partial images that have exactly one correct interpretation.

Original: Images are 5 × 5 binary pixel arrays consisting of two horizontal bars or two vertical bars.
Revised: Images are 5 × 5 binary pixel arrays consisting of two horizontal bars or two vertical bars.

Original: Twenty distinct images exist, shown in the top row of Figure 2.
Revised: Twenty distinct images exist, shown in the top row of Figure 2.

Original: A subset of pixels is provided as evidence; examples are shown in the bottom row of Figure 2.
Revised: A subset of pixels is provided as evidence; examples are shown in the bottom row of Figure 2.

Original: The task is to ﬁll in the masked pixels.
Revised: The task is to ﬁll in the masked pixels.

Original: Evidence is generated such that only one consistent completion exists.
Revised: Evidence is generated such that only one consistent completion exists.

Original: In some cases, a bar must be inferred without any white pixels as evidence (e.g., second column from the right).
Revised: In some cases, a bar must be inferred without any white pixels as evidence (e.g., second column from the right).

Original: In other cases, the local evidence is consistent with both vertical and horizontal bars (e.g., ﬁrst column from left).
Revised: In other cases, the local evidence is consistent with both vertical and horizontal bars (e.g., ﬁrst column from left).

Original: An FBAN with one layer of 50 hidden units is sufﬁcient for the task.
Revised: An FBAN with one layer of 50 hidden units is sufﬁcient for the task.

Original: Evidence is generated randomly on each trial, and evaluating on 10k random states after training, the model is 99.995% correct.
Revised: Evidence is generated randomly on each trial. Evaluating on 10k random states, the model is 99.995% correct.

Original: The middle row in Figure 2 shows the FBAN response after one iteration.
Revised: The middle row in Figure 2 shows the FBAN response after one iteration.

Original: The net comes close to performing the task in a single shot, but after a second iteration of clean up and the asymptotic state is shown in the top row.
Revised: The net comes close to performing the task in a single shot, but after a second iteration of clean up and the asymptotic state is shown in the top row.

Original: Figure 3 shows some visible-hidden weights learned by the FBAN.
Revised: Figure 3 shows some visible-hidden weights learned by the FBAN.

Original: Each 5 × 5 array depicts weights to/from one hidden unit.
Revised: Each 5 × 5 array depicts weights to/from one hidden unit.

Original: Weight sign and magnitude are indicated by coloring and area of the 
Revised: Weight sign and magnitude are indicated by coloring and area of the square, respectively.

Revised: Units appear to select one row and one column, either with the same or opposite

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: polarity.

Revised: Same-polarity weights within a row or column induce coherence among pixels.

Revised: Opposite polarity weights between a row and a column allow the pixel at the intersection to activate either the row/column depending on the sign of the unit’s activation.

Original: 4. S UPERVISED MNIST
Revised: 4. S UPERVISED MNIST

Original: We trained an FBAN with two hidden layers on a supervised version of MNIST in which the visible state consists of a 28 × 28 array for an MNIST digit and an additional vector to code the class label.
Revised: We trained an FBAN with two hidden layers on a supervised version of MNIST in which the visible state consists of a 28 × 28 array for an MNIST digit and an additional vector to code the class label.

Original: For the sake of graphical convenience, we allocate 28 units to the label, using the ﬁrst 20 by redundantly coding the class label in pairs of units, and ignoring the ﬁnal 8 units.
Revised: For the sake of graphical convenience, we allocate 28 units to the label, using the ﬁrst 20 by redundantly coding the class label in pairs of units, and ignoring the ﬁnal 8 units.

Original: Our architecture had 812 inputs, 200 units in the ﬁrst hidden layer, and 50 units in the second.
Revised: Our architecture had 812 inputs, 200 units in the ﬁrst hidden layer, and 50 units in the second.

Original: During training, all bits of the label were masked as well as one-third of image pixels.
Revised: During training, all bits of the label were masked as well as one-third of image pixels.

Original: The image was masked with thresholded Perlin coherent noise (Perlin, 1985), which produces missing patches that are far more difﬁcult to ﬁll in than the isolated pixels produced by Bernoulli masking.
Revised: The image was masked with thresholded Perlin coherent noise (Perlin, 1985), which produces missing patches that are far more difﬁcult to ﬁll in than the isolated pixels produced by Bernoulli masking.

Original: Figure 4 shows evidence provided to FBAN for 20 random test set items in the third row.
Revised: Figure 4 shows evidence provided to FBAN for 20 random test set items in the third row.

Original: The red masks indicate unobserved pixels; the other pixels are clamped in the visible state.
Revised: The red masks indicate unobserved pixels; the other pixels are clamped in the visible state.

Original: The unobserved pixels include those representing the class label, coded in the bottom row of the pixel array.
Revised: The unobserved pixels include those representing the class label, coded in the bottom row of the pixel array.

Original: The top row of the Figure shows the target visible representation, with class labels indicated by the isolated white pixels.
Revised: The top row of the Figure shows the target visible representation, with class labels indicated by the isolated white pixels.

Original: Even though the training loss treats all pixels as equivalent, the FBAN does learn to classify unlabeled images.
Revised: Even though the training loss treats all pixels as equivalent, the FBAN does learn to classify unlabeled images.

Original: On the test set, the model achieves a classiﬁcation accuracy of 87.5% on
Revised: On the test set, the model achieves a classiﬁcation accuracy of 87.5% on

Original: Perlin-masked test images and 89.9% on noise-free test images.
Revised: Perlin-masked test images and 89.9% on noise-free test images.

Original: Note that the 20 pixels indicating class membership are no different than any other missing pixels in the input.
Revised: Note that the 20 pixels indicating class membership are no different than any other missing pixels in the input.

Original: The model learns to classify by virtue of the systematic relationship between images and labels.
Revised: The model learns to classify by virtue of the systematic relationship between images and labels.

Original: We can train the model with fully observed images and fully unobserved labels, and its performance is like that of any fully-connected MNIST classiﬁer, achieving an accuracy of 98.5%.
Revised: We can train the model with fully observed images and fully unobserved labels, and its performance is like that of any fully-connected MNIST classiﬁer, achieving an accuracy of 98.5%.

Original: The FBAN does an excellent job of ﬁlling in missing features in Figure 4 and in further examples in
Revised: The FBAN does an excellent job of ﬁlling in missing features in Figure 4 and in further examples in

Original: Appendix E. The FBAN’s interpretations of the input seem to be respectable in comparison to other recent recurrent associative memory models (Figures 8a,b).
Revised: Appendix F. The FBAN’s interpretations of the input seem to be respectable in comparison to other recent recurrent associative memory models (Figures 8a,b).

Original: We mean no disrespect of other research efforts—which have very different foci than ours—but merely wish to indicate we are obtaining state-of-the-art results for associative memory models.
Revised: We mean no disrespect of other research efforts—which have very different foci than ours—but merely wish to indicate we are obtaining state-of-the-art results for associative memory models.

Original: Figure 8c shows some weights between visible and hidden units.
Revised: Figure 8c shows some weights between visible and hidden units.

Original: Note that the weights link image pixels with multiple digit labels.
Revised: Note that the weights link image pixels with multiple digit labels.

Original: These weights stand apart from the usual hidden representations found in feedforward classiﬁcation networks.
Revised: These weights stand apart from the usual hidden representations found in feedforward classiﬁcation networks.

Original: 4. U NSUPERVISED O MNIGLOT
Revised: 4. U NSUPERVISED O MNIGLOT

Original: We trained a CBAN with the Omniglot images (Lake et al., 2015). Omniglot consists ofmultiple instances of 1623 characters from 50 different alphabets.
Revised: We trained a CBAN with the Omniglot data set (Lake et al., 2015) which consists of instances of 1623 characters from 50 different alphabets.

Original: The CBAN has one visible layer containing the character image, 28 × 28 × 1 , and three successive hidden layers with dimensions 28 × 28 × 128 , 14 × 14 × 256 , and 7 × 7 × 256 , all with average pooling between the layers and ﬁlters of size 3 × 3 .
Revised: The CBAN has one visible layer containing the character image, 28 × 28 × 1 , and three successive hidden layers with dimensions 28 × 28 × 128 , 14 × 14 × 256 , and 7 × 7 × 256 , all with average pooling between the layers and ﬁlters of size 3 × 3 .

Original: Other network parameters and training details are presented in Appendix D. To experiment with a different type of masking, we used random square patches of diameter 3–6, which remove on averageroughly 30% of the white pixels in the image.
Revised: Other network parameters and training details are presented in Appendix D. To vary the masking, we used random square patches of diameter 3–6, which remove on average 30% of the white pixels in the image.

Original: We compared our CBAN to variants with critical properties removed: one without weight symmetry ( CBAN-asym ) and one in which the TD(1) training procedure is substituted for a standard squared 
Revised: We compared our CBAN to variants with critical properties removed: one without weight symmetry ( CBAN-asym ) and one in which the TD(1) training procedure is substituted for a standard squared loss at the ﬁnal step ( CBAN-noTD ). We also compare to a convolutional denoising VAE ( CD-VAE ), which takes the masked image as input and outputs the completion. The CBAN with symmetric

Original: [Figure]
Revised: [Figure]

Original: Qualitatively, CBAN produces the best image reconstructions (Figure 5).
Revised: Qualitatively, CBAN produces the best image reconstructions (Figure 5; additional completions in

Revised: Appendix F).

Original: CBAN-asym and CBANnoTD tend to hallucinate additional strokes; and CBAN-noTD and CD-VAE produce less crisp edges.
Revised: CBAN-asym and CBAN-noTD tend to hallucinate additional strokes; and CBAN-noTD and CD-VAE produce less crisp edges.

Original: Quantitatively, we assess models with two measures of reconstruction quality, peak signal-to-noise ratio ( PSNR ) and structural similarity ( SSIM , Wang et al., 2004); larger is better on each measure.
Revised: Quantitatively, we assess models with two measures of reconstruction quality, peak signal-to-noise ratio ( PSNR ) and structural similarity ( SSIM , Wang et al., 2004); larger is better on each measure.

Original: CBAN is strictly superior to the alternatives on both measures (Figure 5, right panel).
Revised: CBAN is strictly superior to the alternatives on both measures (Figure 5, right panel).

Original: CBAN completions are not merely memorized instances; the CBAN has learned structural regularities of the images, allowing it to ﬁll in big gaps in images that—with the missing pixels—are typically uninterpretable by both classiﬁers and humans.Additional CBAN image completion examples for
Revised: CBAN completions are not merely memorized instances; the CBAN has learned structural regularities of the images, allowing it to ﬁll in big gaps in images that—with the missing pixels—are typically uninterpretable by both classiﬁers and humans.

Original: Omniglot can be found in Appendix E.

Original: 4. U NSUPERVISED CIFAR- 
Revised: 4. U NSUPERVISED CIFAR-

Original: We trained a CBAN with one visible and three hidden layers on CIFAR-10 images.
Revised: We trained a CBAN with one visible and three hidden layers on CIFAR-10 images.

Original: The visible layer is the size of the input image, 32 × 32 × 3 .
Revised: The visible layer is the size of the input image, 32 × 32 × 3 .

Original: The successive hidden layers had dimensions 32 × 32 × 40 , 16 × 16 × 120 , and 8 × 8 × 440 , all with ﬁlters of size 3 × 3 and average pooling between the hidden layers.
Revised: The successive hidden layers had dimensions 32 × 32 × 40 , 16 × 16 × 120 , and 8 × 8 × 440 , all with ﬁlters of size 3 × 3 and average pooling between the hidden layers.

Original: Further details of architecture and training can be found in Appendix D.
Revised: Further details of architecture and training can be found in Appendix D.

Original: Figure 6 shows qualitative and quantitative comparisons of alternative models.
Revised: Figure 6 shows qualitative and quantitative comparisons of alternative models.

Original: Here, CBAN-asym performs about the same as CBAN.
Revised: Here, CBAN-asym performs about the same as CBAN.

Original: However, CBAN-asym typically attains bi-phasic limit cycles, and CBAN-asym sometimes produces splotchy artifacts in background regions (e.g., third image from left).
Revised: However, CBAN-asym typically attains bi-phasic limit cycles, and CBAN-asym sometimes produces splotchy artifacts in background regions (e.g., third image from left).

Original: CBAN-noTD and the CD-VAE are clearly inferior to CBAN.
Revised: CBAN-noTD and the CD-VAE are clearly inferior to CBAN.

Original: Additional CBAN image completions can be found in Appendix E. [Figure] [Table]
Revised: Additional CBAN image completions can be found in Appendix F.

Original: 4. S UPER - RESOLUTION
Revised: 4. S UPER - RESOLUTION

Original: Deep learning models have proliferated in many domains of image processing, perhaps none more than image super-resolution, which is concerned with recovering a high-resolution image from a  low-resolution image.
Revised: Deep learning models have proliferated in many domains of image processing, perhaps none more than image super-resolution, which is concerned with recovering a high-resolution image from a [Figure] [Table] low-resolution image.

Original: Many specialized architectures have been developed, and although common test data sets exist, comparisons are not as simple as one would hope due to subtle differences in methodology.
Revised: Many specialized architectures have been developed, and although common test data sets exist, comparisons are not as simple as one would hope due to subtle differences in methodology.

Original: (For example, even the baseline method, bicubic interpolation, yields different results depending on the implementation.)
Revised: (For example, even the baseline method, bicubic interpolation, yields different results depending on the implementation.)

Original: We set out to explore the feasibility of using CBANs for super-resolution.
Revised: We set out to explore the feasibility of using CBANs for super-resolution.

Original: Our architecture processes 40 × 40 color image patches, and the visible state included both the low- and high-resolution images, with the low-resolution version clamped and the high-resolution version read out from the net.
Revised: Our architecture processes 40 × 40 color image patches, and the visible state included both the low- and high-resolution images, with the low-resolution version clamped and the high-resolution version read out from the net.

Original: Details can be found in Appendix D.
Revised: Details can be found in Appendix D.

Original: Table 1 presents two measures of performance, SSIM and PSNR, for the CBAN and various published alternatives.
Revised: Table 1 presents two measures of performance, SSIM and PSNR, for the CBAN and various published alternatives.

Original: CBAN beats the baseline, bicubic interpolation, on both measures, and performs well on
Revised: CBAN beats the baseline, bicubic interpolation, on both measures, and performs well on

Original: SSIM against some leading contenders (even beating LapSRN and DRCN on Set14 and Urban100 ), but poorly on PSNR.
Revised: SSIM against some leading contenders (even beating LapSRN and DRCN on Set14 and Urban100 ), but poorly on PSNR.

Original: It is common for PSNR and SSIM to be in opposition: SSIM rewards crisp edges, PSNR rewards averaging toward the mean.
Revised: It is common for PSNR and SSIM to be in opposition: SSIM rewards crisp edges, PSNR rewards averaging toward the mean.

Original: The border sharpening and contrast enhancement that produce good perceptual quality and a high SSIM score (see Figure 7) are due to the fact that
Revised: The border sharpening and contrast enhancement that produce good perceptual quality and a high SSIM score (see Figure 7) are due to the fact that

Original: CBAN comes to an interpretation of the images: it imposes edges and textures in order to make the features mutually consistent.
Revised: CBAN comes to an interpretation of the images: it imposes edges and textures in order to make the features mutually consistent.

Original: We believe that CBAN warrants further investigation for superresolution; regardless of whether it becomes the winner in this competitive ﬁeld, one can argue that it is performing a different type of computation than feedforward models like LapSRN and DRCN.
Revised: We believe that CBAN warrants further investigation for superresolution; regardless of whether it becomes the winner in this competitive ﬁeld, one can argue that it is performing a different type of computation than feedforward models like LapSRN and DRCN.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Revised: This article revisits attractor nets, which are traditionally fully interconnected RNNs trained with recurrent back propagation Liao et al.

Revised: (2018) or the Hebbian rule (Hopﬁeld, 1984).

Revised: Our key contribution is to endow nets with a combination of properties which allows them to tackle difﬁcult image completion problems; these properties include: convolutional weight constraints, novel loss functions, and methods for preventing vanishing/exploding gradients.

Original: In comparison to recent published results on image completion with attractor networks, our CBAN produces far more impressive results (see Appendix, Figure 8, for a contrast).
Revised: In comparison to recent published results on image completion with attractor networks, our CBAN produces far more impressive results (see Appendix, Figure 8, for a contrast).

Original: The computational cost and challenge of training CBANs is no greater than those of training deep feedforward nets.
Revised: The computational cost and challenge of training CBANs is no greater than those of training deep feedforward nets.

Original: CBANs seem to produce crisp images, on par with those produced by generative (e.g., energy- and ﬂow-based) models.
Revised: CBANs seem to produce crisp images, on par with those produced by generative (e.g., energy- and ﬂow-based) models.

Original: CBANs have potential to be applied in many contexts involving data interpretation, with the virtue that the computational resources they bring to bear on a task is dynamic and dependent on the difﬁculty of interpreting a given input.
Revised: CBANs have potential to be applied in many contexts involving data interpretation, with the virtue that the computational resources they bring to bear on a task is dynamic and dependent on the difﬁculty of interpreting a given input.

Original: Although this article has focused on convolutional networks that have attractor dynamics between levels of representation, we have recently recognized the value of architectures that are fundamentally feedforward with attractor dynamics within a level.
Revised: Although this article has focused on convolutional networks that have attractor dynamics between levels of representation, we have recently recognized the value of architectures that are fundamentally feedforward with attractor dynamics within a level.

Original: Our current research explores this variant of the CBAN as a biologically plausible account of intralaminar lateral inhibition.
Revised: Our current research explores this variant of the CBAN as a biologically plausible account of intralaminar lateral inhibition.

Original: A PPENDIX A U SING EVIDENCE
Revised: A PPENDIX A U SING EVIDENCE

Original: The CBAN is probed with an observation —a constraint on the activation of a subset of visible units.
Revised: The CBAN is probed with an observation —a constraint on the activation of a subset of visible units.

Original: For any visible unit, we must specify how an observation is used to constrain activation.
Revised: For any visible unit, we must specify how an observation is used to constrain activation.

Original: The possibilities include:
Revised: The possibilities include:

Original: • The unit is clamped , meaning that the unit activation is set to the observed value and is not allowed to change.
Revised: • The unit is clamped , meaning that the unit activation is set to the observed value and is not allowed to change.

Original: Convergence is still guaranteed, and the energy is minimized conditional on the clamped value.
Revised: Convergence is still guaranteed, and the energy is minimized conditional on the clamped value.

Original: However, clamping a unit has the disadvantage that any error signal back propagated to the unit will be lost (because changing the unit’s input does not change its output).
Revised: However, clamping a unit has the disadvantage that any error signal back propagated to the unit will be lost (because changing the unit’s input does not change its output).

Original: • The unit is initialized to the observed value, instead of 0.
Revised: • The unit is initialized to the observed value, instead of 0.

Original: This scheme has the disadvantage that activation dynamics can cause the network to wander away from the observed state.
Revised: This scheme has the disadvantage that activation dynamics can cause the network to wander away from the observed state.

Original: This problem occurs in practice and the consequences are so severe it is not a viable approach.
Revised: This problem occurs in practice and the consequences are so severe it is not a viable approach.

Original: • In principle, we might try an activation rule which sets the visible unit’s activation to be a convex combination of the observed value and the value that would be obtained via activation dynamics: α × observed + (1 − α ) × f
Revised: • In principle, we might try an activation rule which sets the visible unit’s activation to be a convex combination of the observed value and the value that would be obtained via activation dynamics: α × observed + (1 − α ) × f

Original: ( net input ) .
Revised: ( net input ) .

Original: With α = 1 this is simply the clamping scheme; with α = 0 and appropriate start state, this is just the initialization scheme.
Revised: With α = 1 this is simply the clamping scheme; with α = 0 and appropriate start state, this is just the initialization scheme.

Original: • The unit has an external bias proportional to the observation.
Revised: • The unit has an external bias proportional to the observation.

Original: In this scenario, the net input to a visible unit is:
Revised: In this scenario, the net input to a visible unit is:

Original: [Equation]
Revised: [Equation]

Original: where e j ∝ observation.
Revised: where e j ∝ observation.

Original: The initial activation can be either 0 or the observation.
Revised: The initial activation can be either 0 or the observation.

Original: One concern with this scheme is that the ideal input to a unit will depend on whether or not the unit has this additional bias.
Revised: One concern with this scheme is that the ideal input to a unit will depend on whether or not the unit has this additional bias.

Original: For this reason the magnitude of the bias should probably be small.
Revised: For this reason the magnitude of the bias should probably be small.

Original: However, in order to have an impact, the bias must be larger.
Revised: However, in order to have an impact, the bias must be larger.

Original: • We might replicate all visible units and designate one set for input (clamped) and one set for output (unclamped).
Revised: • We might replicate all visible units and designate one set for input (clamped) and one set for output (unclamped).

Original: The input is clamped to the observation (which may be zero).
Revised: The input is clamped to the observation (which may be zero).

Original: The output is allowed to settle.
Revised: The output is allowed to settle.

Original: The hidden layer(s) would synchronize the inputs and outputs, but it could handle noisy inputs, which isn’t possible with clamping.
Revised: The hidden layer(s) would synchronize the inputs and outputs, but it could handle noisy inputs, which isn’t possible with clamping.

Original: Essentially,
Revised: Essentially,

Original: the input would serve as a bias, but on the hidden units, not on the inputs directly.
Revised: the input would serve as a bias, but on the hidden units, not on the inputs directly.

Original: In practice, we have found that external biases work but are not as effective as clamping.
Revised: In practice, we have found that external biases work but are not as effective as clamping.

Original: Partial clamping with 0 < α < 1 has partial effectiveness relative to clamping.
Revised: Partial clamping with 0 < α < 1 has partial effectiveness relative to clamping.

Original: And initialization is not effective; the state wanders from the initialized values.
Revised: And initialization is not effective; the state wanders from the initialized values.

Original: However, the replicated-visible scheme seems very promising and should be explored further.
Revised: However, the replicated-visible scheme seems very promising and should be explored further.

Original: B L OSS FUNCTIONS
Revised: B L OSS FUNCTIONS

Original: The training procedure for a Boltzmann machine aims to maximize the likelihood of the training data, which consist of a set of observations over the visible units.
Revised: The training procedure for a Boltzmann machine aims to maximize the likelihood of the training data, which consist of a set of observations over the visible units.

Original: The complete states in a Boltzmann machine occur with probabilities speciﬁed by
Revised: The complete states in a Boltzmann machine occur with probabilities speciﬁed by

Original: [Equation]
Revised: [Equation]

Original: where T is a computational temperature and the likelihood of a visible state is obtained by marginalizing over the hidden states.
Revised: where T is a computational temperature and the likelihood of a visible state is obtained by marginalizing over the hidden states.

Original: Raising the likelihood of a visible state is achieved by lowering its energy.
Revised: Raising the likelihood of a visible state is achieved by lowering its energy.

Original: The Boltzmann machine learning algorithm has a contrastive loss: it tries to maximize the energy of states with the visible units clamped to training observations and minimize the energy of states with the visible units unclamped and free to take on whatever values they want.
Revised: The Boltzmann machine learning algorithm has a contrastive loss: it tries to maximize the energy of states with the visible units clamped to training observations and minimize the energy of states with the visible units unclamped and free to take on whatever values they want.

Original: This contrastive loss is an example of an energy-based loss , which expresses the training objective in terms of the network energies.
Revised: This contrastive loss is an example of an energy-based loss , which expresses the training objective in terms of the network energies.

Original: In our model, we will deﬁne an energy-based loss via matched pairs of states: x is a state with the visible units clamped to observed values, and ˜ x is a state in which the visible units are unclamped, i.e., they are free take on values consistent with the hidden units driving them.
Revised: In our model, we will deﬁne an energy-based loss via matched pairs of states: x is a state with the visible units clamped to observed values, and ˜ x is a state in which the visible units are unclamped, i.e., they are free take on values consistent with the hidden units driving them.

Original: Although ˜ x could be any unclamped state, it will be most useful for training if it is related to x (i.e., it is a good point of contrast).
Revised: Although ˜ x could be any unclamped state, it will be most useful for training if it is related to x (i.e., it is a good point of contrast).

Original: To achieve this relationship, we propose to compute (˜ x , x ) pairs by:
Revised: To achieve this relationship, we propose to compute (˜ x , x ) pairs by:

Original: Clamp some portion of the visible units with a training example.
Revised: Clamp some portion of the visible units with a training example.

Original: Run the net to some iteration, at which point the full hidden state is h .
Revised: Run the net to some iteration, at which point the full hidden state is h .

Original: (The point of this step is to identify a hidden state that is a plausible candidate to generate the target visible state.)
Revised: (The point of this step is to identify a hidden state that is a plausible candidate to generate the target visible state.)

Original: Set x to be the complete state in which the hidden component of the state is h and the visible component is the target visible state.
Revised: Set x to be the complete state in which the hidden component of the state is h and the visible component is the target visible state.

Original: Set ˜ x to be the complete state in which the hidden component of the state is h and the visible component is the fully unclamped activation pattern that would be obtained by propagating activities from the hidden units to the (unclamped) visible units.
Revised: Set ˜ x to be the complete state in which the hidden component of the state is h and the visible component is the fully unclamped activation pattern that would be obtained by propagating activities from the hidden units to the (unclamped) visible units.

Original: Note that the contrastive pair at this iteration, (˜ x i , x i ) , are states close to the activation trajectory that the network is following.
Revised: Note that the contrastive pair at this iteration, ( ˜ x i , x i ) , are states close to the activation trajectory that the network is following.

Original: We might train the net only after it has reached convergence, but we’ve found that deﬁning the loss for every iteration i up until convergence improves training performance.
Revised: We might train the net only after it has reached convergence, but we’ve found that deﬁning the loss for every iteration i up until convergence improves training performance.

Original: B. L OSS 1: T HE DIFFERENCE OF ENERGIES
Revised: B. L OSS 1: T HE DIFFERENCE OF ENERGIES

Original: [Equation]
Revised: [Equation]

Revised: v i ) − ρ (˜ v i ) with ρ ( s ) = 1 2(1 + s ) ln(1 + s ) + 1 2(1 − s ) ln(1 − s )

Original: This reduction depends on x and ˜ x sharing the same hidden state, a bipartite architecture in which visible and hidden are interconnected, all visible-to-visible connections are zero, a tanh activation function, f , for all units, and symmetric weights. 
Revised: This reduction depends on x and ˜ x sharing the same hidden state, a bipartite architecture in which visible and hidden are interconnected, all visible-to-visible connections are zero, a tanh activation function, f , for all units, and symmetric weights. The goal of this loss is to move ˜ x toward x .

Revised: Once they become identical, adjusting their relative energies is not beneﬁcial.

Revised: Thus, we propose using a hinge version of the loss:

Original: B. L OSS 2: T HE CONDITIONAL PROBABILITY OF CORRECT RESPONSE
Revised: [Equation] . L OSS 2: T HE CONDITIONAL PROBABILITY OF CORRECT RESPONSE

Original: This loss aims to maximize the log probability of the clamped state conditional on the choice between unclamped and clamped states.
Revised: This loss aims to maximize the log probability of the clamped state conditional on the choice between unclamped and clamped states.

Original: Framed as a loss, we have a negative log likelihood:
Revised: Framed as a loss, we have a negative log likelihood:

Original: [Equation]
Revised: [Equation]

Original: The last step is attained using the Boltzmann distribution (Equation 7).
Revised: The last step is attained using the Boltzmann distribution (Equation 7).

Original: C P ROOF OF CONVERGENCE OF CBAN WITH LEAKY SIGMOID ACTIVATION FUNCTION
Revised: C P ROOF OF CONVERGENCE OF CBAN WITH LEAKY SIGMOID ACTIVATION FUNCTION

Original: (1) x fCWxtb xelR f is applied element wise xCzH
Revised: (1) x fCWxtb xelR f is applied element wise xCzH

Original: I 2 sfee f
Revised: I 2 sfee f

Original: Z laze
Revised: Z laze

Original: L Z
Revised: L Z

Original: I
Revised: I

Original: Z where o a
Revised: Z where o a

Original: I
Revised: I

Original: Assume llblla.si bglelHxtH Eltm where mso
Revised: Assume llblla.si bglelHxtH Eltm where mso

Original: Vgillwyll
Revised: Vgillwyll

Original: Er
Revised: Er

Original: Then for
Revised: Then for

Original: Hx HIM to hold we need llfcwxttbdla ltm
Revised: Hx HIM to hold we need llfcwxttbdla ltm

Original: Vglfcwj xth.pl
Revised: Vglfcwj xth.pl

Original: Itm if 12we are done if
Revised: Itm if 12we are done if

Original: Zanalogously 2 a l (2) f
Revised: Zanalogously 2 a l (2) f

Original: z _LEZ lx
Revised: z _LEZ lx

Original: WE by
Revised: WE by

Original: I
Revised: I

Original: E L Hugh H
Revised: E L Hugh H

Original: Xena l l
Revised: Xena l l

Original: E L ration we want the last term to be at most Itm
Revised: E L ration we want the last term to be at most Itm

Original: Lv
Revised: Lv

Original: Itm 11 E l 1M
Revised: Itm 11 E l 1M

Original: Are
Revised: Are

Original: Ci ar m drei
Revised: Ci ar m drei

Original: Let maxgllwyll
Revised: Let maxgllwyll

Original: In fact there is a degree of freedom here so we can simply use c
Revised: In fact there is a degree of freedom here so we can simply use c

Original: Xr
Revised: Xr

Original: I as the only param in the analysis so long as we also re parameterize b accordingly
Revised: I as the only param in the analysis so long as we also re parameterize b accordingly

Original: In conclusion given c xr
Revised: In conclusion given c xr

Original: I (3) the region of convergence must include the hypercube i
Revised: I (3) the region of convergence must include the hypercube i

Original: Ea
Revised: Ea

Original: HE if the barrier is set to be smaller than the above region no convergence is guaranteed
Revised: HE if the barrier is set to be smaller than the above region no convergence is guaranteed

Original: D N ETWORK ARCHITECTURES AND HYPERPARAMETERS D. B AR TASK
Revised: D N ETWORK ARCHITECTURES AND HYPERPARAMETERS D. B AR TASK

Original: Our architecture was a fully connected bipartite attractor net (FBAN) with one visible layer and two hidden layers having 48 and 24 channels.
Revised: Our architecture was a fully connected bipartite attractor net (FBAN) with one visible layer and two hidden layers having 48 and 24 channels.

Original: We trained using L ∆ E + with the transient TD(1) procedure, deﬁning network stability as the condition in which all changes in unit activation on successive iterations are less than 0.01 for a given input, tanh activation functions, batches of 20 examples (the complete data set), with masks randomly generated on each epoch subject to the constraint that only one completion is consistent with the evidence.
Revised: We trained using L ∆ E + with the transient TD(1) procedure, deﬁning network stability as the condition in which all changes in unit activation on successive iterations are less than 0.01 for a given input, tanh activation functions, batches of 20 examples (the complete data set), with masks randomly generated on each epoch subject to the constraint that only one completion is consistent with the evidence.

Original: Weights between layers l and l + 1 and the biases in layer l are initialized from a mean-zero Gaussian with standard deviation 0 .
Revised: Weights between layers l and l + 1 and the biases in layer l are initialized from a mean-zero Gaussian with standard deviation 0 .

Original: 1( 12 n l + 12 n l +1 + 1) − 12 , where n l is the number of units in layer l .
Revised: 1( 12 n l + 12 n l +1 + 1) − 12 , where n l is the number of units in layer l .

Original: Optimization is via stochastic gradient descent with an initial learning rate of 0.01, dropped to .001; the gradients in a given layer of weights are Lrenormalized to be 1.0 for a batch of examples, which we refer to as SGD-L2 .
Revised: Optimization is via stochastic gradient descent with an initial learning rate of 0.01, dropped to .001; the gradients in a given layer of weights are Lrenormalized to be 1.0 for a batch of examples, which we refer to as SGD-L2 .

Original: D. MNIST
Revised: D. MNIST

Original: Our architecture was a fully connected bipartite attractor net (FBAN) with one visible layer to one hidden layer with 200 units to a second hidden layer with 50.
Revised: Our architecture was a fully connected bipartite attractor net (FBAN) with one visible layer to one hidden layer with 200 units to a second hidden layer with 50.

Original: We trained using L ∆ E + with the transient TD(1) procedure, deﬁning network stability as the condition in which all changes in unit activation on successive iterations are less than 0.01 for a given input, tanh activation functions, batches of 250 examples.
Revised: We trained using L ∆ E + with the transient TD(1) procedure, deﬁning network stability as the condition in which all changes in unit activation on successive iterations are less than 0.01 for a given input, tanh activation functions, batches of 250 examples.

Original: Masks are generated randomly for each example on each epoch.
Revised: Masks are generated randomly for each example on each epoch.

Original: The masks were produced by generating Perlin noise, frequency 7, thresholded such that one third of the pixels were obscured.
Revised: The masks were produced by generating Perlin noise, frequency 7, thresholded such that one third of the pixels were obscured.

Original: Weights between layers l and l + 1 and the biases in layer l are initialized from a mean-zero Gaussian with standard deviation 0 .
Revised: Weights between layers l and l + 1 and the biases in layer l are initialized from a mean-zero Gaussian with standard deviation 0 .

Original: 1( 12 n l + 12 n l +1 + 1) − 12 , where n l is the number of units in layer l .
Revised: 1( 12 n l + 12 n l +1 + 1) − 12 , where n l is the number of units in layer l .

Original: Optimization is via stochastic gradient descent with learning rate 0.01; the gradients in a given layer of weights are L ∞ renormalized to be 1.0 for a batch of examples, which we refer to as SGD-Linf .
Revised: Optimization is via stochastic gradient descent with learning rate 0.01; the gradients in a given layer of weights are L ∞ renormalized to be 1.0 for a batch of examples, which we refer to as SGD-Linf .

Original: Target activations scaled to lie in [-0.999,0.999].
Revised: Target activations scaled to lie in [-0.999,0.999].

Original: [Figure]
Revised: [Figure]

Original: D. CIFAR-
Revised: D. O MNIGLOT

Original: The network architecture consists of four layers: one visible layer and three hidden layers.
Revised: The network architecture consists of four layers: one visible layer and three hidden layers.

Original: The visible layer dimensions match the input image dimensions: (32 , 32 , 3) .
Revised: The visible layer dimensions match the input image dimensions: (28 , 28 , 1) .

Original: The channel dimensions of the three hidden layers increase by 40, 120, and 440, respectively.
Revised: The channel dimensions of the three hidden layers increase by 128, 256, and 512, respectively.

Original: We used ﬁlter sizes of 3 × 3 between all layers.
Revised: We used ﬁlter sizes of 3 × 3 between all layers.

Original: Beyond the ﬁrst hidden layer, we introduce a 2 × 2 average pooling operation followed by half-padded convolution going from layer l to layer l + 1 , and a half-padded convolution followed by a 2 × 2 nearest-neighbor interpolation going from layer l + 1 to layer l .
Revised: Beyond the ﬁrst hidden layer, we introduce a 2 × 2 average pooling operation followed by half-padded convolution going from layer l to layer l + 1 , and a half-padded convolution followed by a 2 × 2 nearest-neighbor interpolation going from layer l + 1 to layer l .

Original: Consequently, the spatial dimensions of the hidden states, from lowest to highest, are (32,32), (16,16) and (8,8).
Revised: Consequently, the spatial dimensions of the hidden states, from lowest to highest, are (28,28), (14,14) and (7,7).

Original: A trainable bias is applied per-channel to each layer.
Revised: A trainable bias is applied per-channel to each layer.

Original: All biases are initialized to 0, whereas kernel weights are
Revised: All biases are initialized to 0, whereas kernel weights are

Original: Gaussian initialized with a standard deviation of 0.0001.
Revised: Gaussian initialized with a standard deviation of 0.01.

Original: The CBAN used tanh activation functions and L SE with TD(1) transient training, as described in the main text.
Revised: The CBAN used tanh activation functions and L SE with TD(1) transient training, as described in the main text.

Original: We trained our model on 50,000 images from the CIFAR10 dataset (test set 10,000).
Revised: We trained our model on 15,424 images from the Omniglot dataset (test set: 3856).

Original: The images are noised by online-generation of Perlin noise that masks 40% of the  image.
Revised: The images are noised by online-generation of squares that mask 20-40% of the white pixels in the image.

Original: We optimized our mean-squared error objective using Adam.
Revised: We optimized our mean-squared error objective using Adam.

Original: The learning rate is initially set to 0.0005 and then decreased manually by a factor of 10 every 20 epochs beyond training epoch 150.
Revised: The learning rate is initially set to 0. and then decreased manually by a factor of 10 every 20 epochs after training epoch 100.

Original: For each batch, the network runs until the state stabilizes, where the condition for stabilization is speciﬁed as the maximum absolute difference of the full network states between stabilization steps t and t + 1 being less than 0.01.
Revised: For each batch, the network runs until the state stabilizes, where the condition for stabilization is speciﬁed as the maximum absolute difference of the full network states between stabilization steps i and i+1 being less than 0.01.

Original: The maximum number of stabilization steps was set to 100; the average stabilization iteration per batch over the course of training was 50 stabilization steps.
Revised: The maximum number of stabilization steps was set to 100; the average stabilization iteration per batch over the course of training was 50 stabilization steps.

Revised: Masks were formed by selecting patches of diameter 3–6 uniformly, in random, possibly overlapping locations, stopping when at least 25% of the white pixels have been masked.

Revised: Our DC-VAE is based on code from https://github.com/sksq96/pytorch-vae/blob/master/vaecnn.ipynb and followed the method of Im et al.

Original: D. O MNIGLOT 
Revised: D. CIFAR-

Original: The network architecture consists of four layers: one visible layer and three hidden layers.
Revised: The network architecture consists of four layers: one visible layer and three hidden layers.

Original: The visible layer dimensions match the input image dimensions: (28 , 28 , 1) .
Revised: The visible layer dimensions match the input image dimensions: (32 , 32 , 3) .

Original: The channel dimensions of the three hidden layers increase by 128, 256, and 512, respectively.
Revised: The channel dimensions of the three hidden layers increase by 40, 120, and 440, respectively.

Original: We used ﬁlter sizes of 3 × 3 between all layers.
Revised: We used ﬁlter sizes of 3 × 3 between all layers.

Original: Beyond the ﬁrst hidden layer, we introduce a 2 × 2 average pooling operation followed by half-padded convolution going from layer l to layer l + 1 , and a half-padded convolution followed by a 2 × 2 nearest-neighbor interpolation going from layer l + 1 to layer l .
Revised: Beyond the ﬁrst hidden layer, we introduce a 2 × 2 average pooling operation followed by half-padded convolution going from layer l to layer l + 1 , and a half-padded convolution followed by a 2 × 2 nearest-neighbor interpolation going from layer l + 1 to layer l .

Original: Consequently, the spatial dimensions of the hidden states, from lowest to highest, are (28,28), (14,14) and (7,7).
Revised: Consequently, the spatial dimensions of the hidden states, from lowest to highest, are (32,32), (16,16) and (8,8).

Original: A trainable bias is applied per-channel to each layer.
Revised: A trainable bias is applied per-channel to each layer.

Original: All biases are initialized to 0, whereas kernel weights are
Revised: All biases are initialized to 0, whereas kernel weights are

Revised: Gaussian initialized with a standard deviation of 0.0001.

Revised: The CBAN used tanh activation functions and L SE with TD(1) transient training, as described in the main text.

Original: We trained our model on 15,424 images from the Omniglot dataset (test set: 3856).
Revised: We trained our model on 50,000 images from the CIFAR10 dataset (test set 10,000).

Original: The images are noised by online-generation of squares that mask 20-40% of thewhite pixels in the image.
Revised: The images are noised by online-generation of Perlin noise that masks 40% of the image.

Original: We optimized our mean-squared error objective using Adam.
Revised: We optimized our mean-squared error objective using Adam.

Original: The learning rate is initially set to 0. and then decreased manually by a factor of 10 every 20 epochs after training epoch 100.
Revised: The learning rate is initially set to 0.0005 and then decreased manually by a factor of 10 every 20 epochs beyond training epoch 150.

Original: For each batch, the network runs until the state stabilizes, where the condition for stabilization is speciﬁed as the maximum absolute difference of the full network states between stabilization steps i and i+1 being less than 0.01.
Revised: For each batch, the network runs until the state stabilizes, where the condition for stabilization is speciﬁed as the maximum absolute difference of the full network states between stabilization steps t and t + 1 being less than 0.01.

Original: The maximum number of stabilization steps was set to 100; the average stabilization iteration per batch over the course of training was 50 stabilization steps.
Revised: The maximum number of stabilization steps was set to 100; the average stabilization iteration per batch over the course of training was 50 stabilization steps.

Original: Masks were formed by selecting patches of diameter 3–6 uniformly, in random, possibly overlapping locations, stopping when at least 25% of the white pixels have been masked.

Original: D. S UPER - RESOLUTION
Revised: D. S UPER - RESOLUTION

Original: The network architecture consists of four layers: one visible layer and three hidden layers.
Revised: The network architecture consists of four layers: one visible layer and three hidden layers.

Original: The visible layer spatial dimensions match the input patch dimensions, but consists of 6 channels: (40 , 40 , 6) .
Revised: The visible layer spatial dimensions match the input patch dimensions, but consists of 6 channels: (40 , 40 , 6) .

Original: The low-resolution evidence patch is clamped to the bottom 3 channels of the visible state; the topchannels of the visible state serve as the unclamped output against which the high-resolution target patch is compared and loss is computed as a mean-squared error.
Revised: The low-resolution evidence patch is clamped to the bottom 3 channels of the visible state; the topchannels of the visible state serve as the unclamped output against which the high-resolution target patch is compared and loss is computed as a mean-squared error.

Original: The channel dimensions of the three hidden layers are 300, 300, and 300.
Revised: The channel dimensions of the three hidden layers are 300, 300, and 300.

Original: We used ﬁlter sizes of 5 × 5 between all layers.
Revised: We used ﬁlter sizes of 5 × 5 between all layers.

Original: All convolutions are half-padded and no average pooling operations are introduced in the SR network scheme.
Revised: All convolutions are half-padded and no average pooling operations are introduced in the SR network scheme.

Original: Consequently, the spatial dimensions of the hidden states remain constant and match the input patches of (40 , 40) .
Revised: Consequently, the spatial dimensions of the hidden states remain constant and match the input patches of (40 , 40) .

Original: A trainable bias is applied per-channel to each layer.
Revised: A trainable bias is applied per-channel to each layer.

Original: All biases are initialized to 0, whereas kernel weights are Gaussian initialized with a standard deviation of 0.001.
Revised: All biases are initialized to 0, whereas kernel weights are Gaussian initialized with a standard deviation of 0.001.

Original: We trained our model on 91 images from the T91 dataset (Yang et al., 2010) scaled at × 2.
Revised: We trained our model on 91 images from the T91 dataset (Yang et al., 2010) scaled at × 2.

Original: We optimized our mean-squared error objective using Adam.
Revised: We optimized our mean-squared error objective using Adam.

Original: The learning rate is initially set to 0.
Revised: The learning rate is initially set to 0.

Original: and then decreased by a factor of 10 every 10 epochs.
Revised: and then decreased by a factor of 10 every 10 epochs.

Original: The stability conditions described in the CBAN models for CIFAR-10 and Omniglot are repeated for the SR task, except the stability threshold was set to 0.1 half way through training.
Revised: The stability conditions described in the CBAN models for CIFAR-10 and Omniglot are repeated for the SR task, except the stability threshold was set to 0.1 half way through training.

Original: We evaluated on four test datasets at × 2 scaling: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), DSB100 (Martin et al., 2001), and Urban100 (Huang et al., 2015).
Revised: We evaluated on four test datasets at × 2 scaling: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), DSB100 (Martin et al., 2001), and Urban100 (Huang et al., 2015).

Original: E A DDITIONAL RESULTS
Revised: E A DDITIONAL E XPERIMENTS E. C ONVERGENCE T HRESHOLD

Revised: The convergence threshold operates within the forward iterations of network training and determines the point at which the network’s state has satisfactorily converged. Smaller thresholds will result in longer training time, whereas large thresholds will yield faster training times. However, the higher thresholds may negatively impact network performance.

Original: target completion evidence dream target completion evidence dream target completion evidence dream target completion evidence dream
Revised: We examined the effects of convergence thresholds on network performance by training two identical CBAN models on the Omniglot dataset; see § D.3 for network architecture and training details. The only difference between the models is the convergence threshold: one model uses a convergence threshold of 0.01, whereas the other uses 1.0. Our experimental results are shown in Table 2. We ﬁnd that the higher convergence threshold provides both a signiﬁcant improvement in training efﬁciency, with an increased performance on PSNR, SSIM, and L1 reconstruction error metrics.

Revised: The average number of forward iterations drops from 45 towhen changing from a convergence threshold of 0.01 to 1.0.

Revised: E. L AYER U PDATING S CHEMES

Revised: Salakhutdinov and Hinton (2009) apply odd-even rules for layer updating in their work on Restricted

Revised: Boltzmann Machines (RBMs).

Revised: We empirically evaluate the odd-even update rule and compare it against a simple up-down update rule.

Revised: The up-down update rule sequentially updates the layers,

Revised: [Table]

Revised: starting from the bottom and moving to the top, and then back down.

Revised: We explored the two alternative update rules by training two CBAN models on the Omniglot dataset; one with the up-down update rule and the other with the odd-even update rule.

Revised: See § D.3 for detailed network architecture and training details.

Revised: Although the odd-even rule is suggested to be a more biologically-plausible mechanism, our results show the up-down rule achieving superior performance, measured by PSNR, SSIM, and

Revised: L1 reconstruction loss (see Table 3).

Revised: In addition, a qualitative comparison on image completion demonstrates that the odd-even rule is less efﬁcient when compared to the up-down rule, and more frequently fails to fully complete the task (see Figure 9).

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: target completion evidence dream target completion evidence dream target completion evidence dream target completion evidence dream