Original: M ONOTONE DEEP B OLTZMANN MACHINES
Revised: M ONOTONE DEEP B OLTZMANN MACHINES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep Boltzmann machines refer to deep multi-layered probabilistic models, governed by a pairwise energy function that describes the likelihood of all variables in the network.
Revised: Deep Boltzmann machines refer to deep multi-layered probabilistic models, governed by a pairwise energy function that describes the likelihood of all variables in the network.

Original: Due to the difficulty of inference in such systems, they have given way largely to restricted deep Boltzmann machines (which do not permit intralayer or skip connections).
Revised: Due to the difficulty of inference in such systems, they have given way largely to restricted deep Boltzmann machines (which do not permit intralayer or skip connections).

Original: In this paper, we propose a class of model that allows for exact, efficient mean-field inference and learning in general deep Boltzmann machines.
Revised: In this paper, we propose a class of model that allows for exact, efficient mean-field inference and learning in general deep Boltzmann machines.

Original: To do so, we use the tools of the recently proposed monotone
Revised: To do so, we use the tools of the recently proposed monotone

Original: Deep
Revised: Deep

Original: Equilibrium (DEQ) Model, an implicit-depth deep network that always guarantees the existence and uniqueness of its fixed points.
Revised: Equilibrium (DEQ) Model, an implicit-depth deep network that always guarantees the existence and uniqueness of its fixed points.

Original: We show that, for a class of general deep Boltzmann machine, the mean-field fixed point can be considered as the equivalent fixed point of a monotone DEQ, which gives us a recipe for deriving an efficient mean-field inference procedure with global convergence guarantees.
Revised: We show that, for a class of general deep Boltzmann machine, the mean-field fixed point can be considered as the equivalent fixed point of a monotone DEQ, which gives us a recipe for deriving an efficient mean-field inference procedure with global convergence guarantees.

Original: We apply this approach to simple deep convolutional Boltzmann architectures and demonstrate that it allows for tasks such as the joint completion and classification of images, all within a single deep probabilistic setting.
Revised: We apply this approach to simple deep convolutional Boltzmann architectures and demonstrate that it allows for tasks such as the joint completion and classification of images, all within a single deep probabilistic setting.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: This paper considers (deep) Boltzmann machines, which are pairwise energy-based probabilistic models.
Revised: This paper considers (deep) Boltzmann machines, which are pairwise energy-based probabilistic models.

Original: Theses models specify a joint distribution over variables x given by the density
Revised: Theses models specify a joint distribution over variables x given by the density

Original: [Equation]
Revised: [Equation]

Original: where each x 1: n denotes a discrete random variable over k i possible values, represented as a onehot encoding x i ∈ { 0 , 1 } k i ; E denotes the set of edges in the model; Φ i,j ∈ R k i × k j represents pairwise potential; and b i ∈
Revised: where each x 1: n denotes a discrete random variable over k i possible values, represented as a onehot encoding x i ∈ { 0 , 1 } k i ; E denotes the set of edges in the model; Φ i,j ∈ R k i × k j represents pairwise potential; and b i ∈

Original: R k i represents unary potential.
Revised: R k i represents unary potential.

Original: Depending on context, these models are typically referred to as pairwise Markov random fields (MRFs) (Koller & Friedman, 2009), or (potentially deep) Boltzmann machines (Goodfellow et al., 2016; Salakhutdinov & Hinton, 2009;
Revised: Depending on context, these models are typically referred to as pairwise Markov random fields (MRFs) (Koller & Friedman, 2009), or (potentially deep) Boltzmann machines (Goodfellow et al., 2016; Salakhutdinov & Hinton, 2009;

Original: Hinton, 2002).
Revised: Hinton, 2002).

Original: In the above setting each x i may represent an observed or unobserved value, and there can be substantial structure within the variables; for instance, the collection of variables x may (and indeed will, in the main settings we consider in this paper) consist of several different “layers” in a joint convolutional structure, leading to the deep convolutional Boltzmann machine (Norouzi et al., 2009).
Revised: In the above setting each x i may represent an observed or unobserved value, and there can be substantial structure within the variables; for instance, the collection of variables x may (and indeed will, in the main settings we consider in this paper) consist of several different “layers” in a joint convolutional structure, leading to the deep convolutional Boltzmann machine (Norouzi et al., 2009).

Original: In this paper, we propose a new parameterization and algorithmic approach to approximate inference of these probabilistic models.
Revised: In this paper, we propose a new parameterization and algorithmic approach to approximate inference of these probabilistic models.

Original: There are two main contributions: First, we define a generic parameterization of the pairwise kernel function Φ , that can represent a general Boltzmann machine.
Revised: There are two main contributions: First, we define a generic parameterization of the pairwise kernel function Φ , that can represent a general Boltzmann machine.

Original: Our parametrization is flexible enough to incorporate almost all operators and network topology, including fully-connected layers, convolution operators, and skip-connections, etc.
Revised: Our parametrization is flexible enough to incorporate almost all operators and network topology, including fully-connected layers, convolution operators, and skip-connections, etc.

Original: Through the lens of the recently developed monotone DEQ (Winston & Kolter, 2020), we constraint Φ to satisfy certain monotonicity conditions through training and inference; this thereby assures that the mean-field approximation will always have a unique, globally-optimal fixed point under this parameterization.
Revised: Through the lens of the recently developed monotone DEQ (Winston & Kolter, 2020), we constraint Φ to satisfy certain monotonicity conditions through training and inference; this thereby assures that the mean-field approximation will always have a unique, globally-optimal fixed point under this parameterization.

Original: Second, although previous works (Kr¨ahenb¨uhl & Koltun, 2013; Baqu´e et al., 2016) have made approaches to parallel mean-field updates, they either require strong conditions on ϕ , or fail to converge to the true mean-field distribution.
Revised: Second, although previous works (Kr¨ahenb¨uhl & Koltun, 2013; Baqu´e et al., 2016) have made approaches to parallel mean-field updates, they either require strong conditions on ϕ , or fail to converge to the true mean-field distribution.

Original: We provide a properly-damped mean-field update method, based upon a generic proximal operator, which is guaranteed to converge to the mean-field fixed point, even if applied in parallel to all random variables simultaneously.
Revised: We provide a properly-damped mean-field update method, based upon a generic proximal operator, which is guaranteed to converge to the mean-field fixed point, even if applied in parallel to all random variables simultaneously.

Original: Although there is no exact closed form solution of this proximal operator, we derive a very efficient Newton-based implementation.
Revised: Although there is no exact closed form solution of this proximal operator, we derive a very efficient Newton-based implementation.

Original: To estimate the parameters of our model, we follow the marginal-based loss minimization approach in Kr ¨ ahenb ¨ uhl & Koltun (2013);
Revised: To estimate the parameters of our model, we follow the marginal-based loss minimization approach in Kr ¨ ahenb ¨ uhl & Koltun (2013);

Original: Domke (2013), where the objective is to directly maximize the likelihood induced by the approximated mean-field distribution.
Revised: Domke (2013), where the objective is to directly maximize the likelihood induced by the approximated mean-field distribution.

Original: Such an approach grants us the ability to update parameters via taking gradient steps of the proposed mean-field iterations.
Revised: Such an approach grants us the ability to update parameters via taking gradient steps of the proposed mean-field iterations.

Original: With the proposed approaches, we perform both learning and inference for a deep convolutional, multi-resolution Boltzmann machine, and apply the network to model MNIST and CIFAR-10 pixels and their classes conditioned on partially observed images.
Revised: With the proposed approaches, we perform both learning and inference for a deep convolutional, multi-resolution Boltzmann machine, and apply the network to model MNIST and CIFAR-10 pixels and their classes conditioned on partially observed images.

Original: Such joint probabilistic modelling allows us to simultaneously impute missing pixels and predict the class.
Revised: Such joint probabilistic modelling allows us to simultaneously impute missing pixels and predict the class.

Original: While these are naturally a smallscale problem, we emphasize that performing joint probabilistic inference over a complete model of this type is a relatively high-dimensional task as far as traditional mean-field inference is concerned.
Revised: While these are naturally a smallscale problem, we emphasize that performing joint probabilistic inference over a complete model of this type is a relatively high-dimensional task as far as traditional mean-field inference is concerned.

Original: We also compare our inference method to previous ones and demonstrate different convergence properties of each.
Revised: We also compare our inference method to previous ones and demonstrate different convergence properties of each.

Original: We conclude by highlighting limitations and directions for future work with the method.
Revised: We conclude by highlighting limitations and directions for future work with the method.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: This paper builds upon three main avenues of work: 1) deep equilibrium models, especially their convergent version, the monotone DEQ; 2) the broad topic of energy-based deep model and Boltzmann machines in particular; and 3) work on concave potentials and parallel methods for mean-field inference.
Revised: This paper builds upon three main avenues of work: 1) deep equilibrium models, especially their convergent version, the monotone DEQ; 2) the broad topic of energy-based deep model and Boltzmann machines in particular; and 3) work on concave potentials and parallel methods for mean-field inference.

Original: We discuss each of these below.
Revised: We discuss each of these below.

Original: Equilibrium models and their provable convergence The DEQ model was first proposed by Bai et al.
Revised: Equilibrium models and their provable convergence The DEQ model was first proposed by Bai et al.

Original: Based on the observation that a neural network z t +1 =
Revised: Based on the observation that a neural network z t +1 =

Original: σ ( Wz t
Revised: σ ( Wz t

Original: + Ux + b ) with input injection x usually converges to a fixed point, they modeled an effectively infinite-depth network with input injection directly via its fixed point: z ∗ =
Revised: + Ux + b ) with input injection x usually converges to a fixed point, they modeled an effectively infinite-depth network with input injection directly via its fixed point: z ∗ =

Original: σ ( Wz ∗ + Ux + b ) .
Revised: σ ( Wz ∗ + Ux + b ) .

Original: Its backpropagation is done through the implicit function theorem and only requires constant memory.
Revised: Its backpropagation is done through the implicit function theorem and only requires constant memory.

Original: Bai et al.
Revised: Bai et al.

Original: (2020) also showed that the multiscale DEQ models achieve near state-of-the-art performances on many large-scale tasks.
Revised: (2020) also showed that the multiscale DEQ models achieve near state-of-the-art performances on many large-scale tasks.

Original: Winston & Kolter (2020) later presented a parametrization of the DEQ (denoted as monDEQ) that guarantees provable convergence to a unique fixed point, using monotone operator theory.
Revised: Winston & Kolter (2020) later presented a parametrization of the DEQ (denoted as monDEQ) that guarantees provable convergence to a unique fixed point, using monotone operator theory.

Original: Specifically, they parameterize W in a way that I − W ⪰ mI (called m -strongly monotone) is always satisfied during training for some m > 0 ; they convert nonlinearities into proximal operators (which include ReLU, tanh, etc.), and show that using existing splitting methods like forward-backward and Peaceman-Rachford can provably find the unique fixed point.
Revised: Specifically, they parameterize W in a way that I − W ⪰ mI (called m -strongly monotone) is always satisfied during training for some m > 0 ; they convert nonlinearities into proximal operators (which include ReLU, tanh, etc.), and show that using existing splitting methods like forward-backward and Peaceman-Rachford can provably find the unique fixed point.

Original: Markov random field (MRF) and its variants MRF is a form of energy-based model, which model joint probabilities of the form p θ ( x ) =
Revised: Markov random field (MRF) and its variants MRF is a form of energy-based model, which model joint probabilities of the form p θ ( x ) =

Original: exp ( − E θ ( x )) /Z
Revised: exp ( − E θ ( x )) /Z

Original: θ for an energy function E θ .
Revised: θ for an energy function E θ .

Original: A common type of MRF is the Boltzmann machine, the most successful variant of which is the restricted
Revised: A common type of MRF is the Boltzmann machine, the most successful variant of which is the restricted

Original: Boltzmann machines (RBM) (Hinton, 2002) and its deep (multi-layer) variant (Salakhutdinov &
Revised: Boltzmann machines (RBM) (Hinton, 2002) and its deep (multi-layer) variant (Salakhutdinov &

Original: Hinton, 2009).
Revised: Hinton, 2009).

Original: Particularly, RBMs define E θ ( v, h )
Revised: Particularly, RBMs define E θ ( v, h )

Original: = − a ⊤ v − b ⊤ h − v ⊤ Wh , where θ =
Revised: = − a ⊤ v − b ⊤ h − v ⊤ Wh , where θ =

Original: W, a, b } , v is the set of visible variables, and h is the set of latent variables.
Revised: W, a, b } , v is the set of visible variables, and h is the set of latent variables.

Original: It is usually trained using the contrastive-divergence algorithm, and its inference can be done efficiently by a block mean-field approximation.
Revised: It is usually trained using the contrastive-divergence algorithm, and its inference can be done efficiently by a block mean-field approximation.

Original: However, a particular restriction of RBMs is that there can be no intra-layer connections, that is, each variable in v (resp.
Revised: However, a particular restriction of RBMs is that there can be no intra-layer connections, that is, each variable in v (resp.

Original: h ) is independent conditioned on h (resp.
Revised: h ) is independent conditioned on h (resp.

Original: v ).
Revised: v ).

Original: A deep RBM allows different layers of hidden nodes, but there cannot be intra-layer connections.
Revised: A deep RBM allows different layers of hidden nodes, but there cannot be intra-layer connections.

Original: By contrast, our formulation allows intra-layer connections and is therefore is more expressive in this respect.
Revised: By contrast, our formulation allows intra-layer connections and is therefore is more expressive in this respect.

Original: See fig.
Revised: See fig.

Original: 1 for the network topology of RBM, deep RBM, and general BM (we also use the term general deep BM interchangeably to emphasize the existence of deep structure).
Revised: 1 for the network topology of RBM, deep RBM, and general BM (we also use the term general deep BM interchangeably to emphasize the existence of deep structure).

Original: Wu et al.
Revised: Wu et al.

Original: (2016) proposed a deep parameterization of MRF, but their setting only considers a grid of hidden variables h
Revised: (2016) proposed a deep parameterization of MRF, but their setting only considers a grid of hidden variables h

Original: and the connections among hidden units are restricted to the neighboring nodes.
Revised: and the connections among hidden units are restricted to the neighboring nodes.

Original: Therefore, it is a special case of our parameterization (although their learning algorithm is orthogonal to ours).
Revised: Therefore, it is a special case of our parameterization (although their learning algorithm is orthogonal to ours).

Original: Numerous works also try to combine deep neural networks with conditional random fields (CRF) (Kr¨ahenb¨uhl & Koltun, 2013; Schwartz et al., 2017; Zheng et al., 2015).
Revised: Numerous works also try to combine deep neural networks with conditional random fields (CRF) (Kr¨ahenb¨uhl & Koltun, 2013; Schwartz et al., 2017; Zheng et al., 2015).

Original: These models use neural networks for producing either inputs or parameters of their CRFs, whereas our network itself is a probabilistic graphical model.
Revised: These models use neural networks for producing either inputs or parameters of their CRFs, whereas our network itself is a probabilistic graphical model.

Original: We remark the readers upon this fundamental difference.
Revised: We remark the readers upon this fundamental difference.

Original: Parallel and convergent mean-field It is well-known that mean-field updates converge locally using a coordinate ascent algorithm (Blei et al., 2017).
Revised: Parallel and convergent mean-field It is well-known that mean-field updates converge locally using a coordinate ascent algorithm (Blei et al., 2017).

Original: However, local convergence is only guaranteed if the update is applied sequentially.
Revised: However, local convergence is only guaranteed if the update is applied sequentially.

Original: Nonetheless, several works have proposed techniques to parallelize updates.
Revised: Nonetheless, several works have proposed techniques to parallelize updates.

Original: Kr¨ahenb¨uhl & Koltun (2013) proposed a concave-convex procedure (CCCP) to minimize the KL divergence between the true distribution and the mean-field variational family.
Revised: Kr¨ahenb¨uhl & Koltun (2013) proposed a concave-convex procedure (CCCP) to minimize the KL divergence between the true distribution and the mean-field variational family.

Original: To achieve efficient inference, they use a concave approximation to the pairwise kernel, and their fast update rule only converges if the kernel function is concave.
Revised: To achieve efficient inference, they use a concave approximation to the pairwise kernel, and their fast update rule only converges if the kernel function is concave.

Original: Later, Baqu ´ e et al.
Revised: Later, Baqu ´ e et al.

Original: (2016) derived a similar parallel damped forward iteration to ours that provably converges without the concave potential constraint.
Revised: (2016) derived a similar parallel damped forward iteration to ours that provably converges without the concave potential constraint.

Original: However, unlike our approach, they do not use a parameterization which ensures a global mean-field optimum, and their algorithm therefore may not converge to the actual fixed point of the mean-field updates.
Revised: However, unlike our approach, they do not use a parameterization which ensures a global mean-field optimum, and their algorithm therefore may not converge to the actual fixed point of the mean-field updates.

Original: This is because Baqu ´ e et al.
Revised: This is because Baqu ´ e et al.

Original: (2016) used the prox 1 f proximal operator (described below), whereas we derive the prox αf operator to guarantee global convergence when doing mean-field updates in parallel.
Revised: (2016) used the prox 1 f proximal operator (described below), whereas we derive the prox αf operator to guarantee global convergence when doing mean-field updates in parallel.

Original: What’s more, Baqu ´ e et al.
Revised: What’s more, Baqu ´ e et al.

Original: (2016) focused only on inference over prescribed potentials, and not on training the (fully parameterized) potentials as we do here.
Revised: (2016) focused only on inference over prescribed potentials, and not on training the (fully parameterized) potentials as we do here.

Original: L ˆ e-Huu & Alahari (2021) brought up a generalized Frank-Wolfe based framework for mean-field updates which include the methods proposed by Baqu ´
Revised: L ˆ e-Huu & Alahari (2021) brought up a generalized Frank-Wolfe based framework for mean-field updates which include the methods proposed by Baqu ´

Original: e et al.
Revised: e et al.

Original: ; Kr ¨ ahenb ¨ uhl & Koltun (2013).
Revised: ; Kr ¨ ahenb ¨ uhl & Koltun (2013).

Original: Their results only guarantee global convergence to a local optimal.
Revised: Their results only guarantee global convergence to a local optimal.

Original:  M ONOTONE DEEP B OLTZMANN MACHINES AND APPROXIMATE INFERENCE
Revised:  M ONOTONE DEEP B OLTZMANN MACHINES AND APPROXIMATE INFERENCE

Original: In this section, we present the main technical contributions of this work.
Revised: In this section, we present the main technical contributions of this work.

Original: We begin by presenting a parameterization of the pairwise potential in a Boltzmann machine that guarantees the monotonicity condition.
Revised: We begin by presenting a parameterization of the pairwise potential in a Boltzmann machine that guarantees the monotonicity condition.

Original: We then illustrate the connection between a (joint) mean-field inference fixed point and the fixed point of our monotone Boltzmann machine and discuss how deep structured networks can be implemented in this form practically; this establishes that, under the monotonicity conditions on Φ , there exists a unique globally-optimal mean-field fixed point.
Revised: We then illustrate the connection between a (joint) mean-field inference fixed point and the fixed point of our monotone Boltzmann machine and discuss how deep structured networks can be implemented in this form practically; this establishes that, under the monotonicity conditions on Φ , there exists a unique globally-optimal mean-field fixed point.

Original: Finally, we present an efficient parallel method for computing this mean-field fixed point, again motivated by the machinery of monotone DEQs and operator splitting methods.
Revised: Finally, we present an efficient parallel method for computing this mean-field fixed point, again motivated by the machinery of monotone DEQs and operator splitting methods.

Original: 3. A MONOTONE PARAMETERIZATION OF GENERAL B OLTZMANN MACHINES
Revised: 3. A MONOTONE PARAMETERIZATION OF GENERAL B OLTZMANN MACHINES

Original: In this section, we show how to parameterize our probabilistic model in a way that the pairwise potentials satisfy I − Φ ⪰ mI , which will be used later to show the existence of a unique meanfield fixed point.
Revised: In this section, we show how to parameterize our probabilistic model in a way that the pairwise potentials satisfy I − Φ ⪰ mI , which will be used later to show the existence of a unique meanfield fixed point.

Original: Additionally, since Φ defines a graphical model that has no self-loop, we further require Φ to be a block hollow matrix (that is, the k
Revised: Additionally, since Φ defines a graphical model that has no self-loop, we further require Φ to be a block hollow matrix (that is, the k

Original: i × k
Revised: i × k

Original: i diagonal blocks corresponding to each variable must be zero).
Revised: i diagonal blocks corresponding to each variable must be zero).

Original: While both these conditions on Φ are convex constraints, in practice it would be extremely difficult to project a generic set of weights onto this constraint set under an ordinary parameterization of the network.
Revised: While both these conditions on Φ are convex constraints, in practice it would be extremely difficult to project a generic set of weights onto this constraint set under an ordinary parameterization of the network.

Original: Thus, we instead advocate for a non-convex parameterization of the network weights, but one which guarantees that the monotonicity condition is always satisfied, without any constraint on the weights in the parameterization.
Revised: Thus, we instead advocate for a non-convex parameterization of the network weights, but one which guarantees that the monotonicity condition is always satisfied, without any constraint on the weights in the parameterization.

Original: Specifically, define the block matrix
Revised: Specifically, define the block matrix

Original: [Equation]
Revised: [Equation]

Original: with A i ∈ R d × k i matrices for each variables, and where d can be some arbitrarily chosen dimension.
Revised: with A i ∈ R d × k i matrices for each variables, and where d can be some arbitrarily chosen dimension.

Original: Then let ˆ A i be a spectrally-normalized version of A
Revised: Then let ˆ A i be a spectrally-normalized version of A

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i.e., a version of A i normalized such that its largest singular value is at most √ 1 − m (note that we can compute the spectral norm of A i as ∥ A i ∥ 2 = ∥
Revised: i.e., a version of A i normalized such that its largest singular value is at most √ 1 − m (note that we can compute the spectral norm of A i as ∥ A i ∥ 2 = ∥

Original: A Ti A i ∥ 1 / 2 2 , which involves computing the singular values of only a k
Revised: A Ti A i ∥ 1 / 2 2 , which involves computing the singular values of only a k

Original: i × k
Revised: i × k

Original: i matrix, and thus is very fast in practice).
Revised: i matrix, and thus is very fast in practice).

Original: We define the ˆ A matrix analogously as the block version of these normalized matrices.
Revised: We define the ˆ A matrix analogously as the block version of these normalized matrices.

Original: Then we propose to parameterize Φ as
Revised: Then we propose to parameterize Φ as

Original: [Equation]
Revised: [Equation]

Original: where blkdiag denotes the block-diagonal portion of the matrix along the k i × k
Revised: where blkdiag denotes the block-diagonal portion of the matrix along the k i × k

Original: i block.
Revised: i block.

Original: Put another way, this parameterizes Φ as
Revised: Put another way, this parameterizes Φ as

Original: [Equation]
Revised: [Equation]

Original: As the following simple theorem shows, this parameterization guarantees both hollowness of the Φ matrix and monotonicity of I − Φ , for any value of the A matrix.
Revised: As the following simple theorem shows, this parameterization guarantees both hollowness of the Φ matrix and monotonicity of I − Φ , for any value of the A matrix.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: For any choice of parameters A , under the parametrization equation 4 above, we have that 1) Φ ii = 0 for all i = 1 , . . . , n , and 2) I − Φ ⪰ mI .
Revised: For any choice of parameters A , under the parametrization equation 4 above, we have that 1) Φ ii = 0 for all i = 1 , . . . , n , and 2) I − Φ ⪰ mI .

Original: Proof.
Revised: Proof.

Original: Block hollowness of the matrix follows immediately from construction.
Revised: Block hollowness of the matrix follows immediately from construction.

Original: To establish monotonicity, note that
Revised: To establish monotonicity, note that

Original: [Equation]
Revised: [Equation]

Original: This last property always holds by the construction of ˆ A i .
Revised: This last property always holds by the construction of ˆ A i .

Original: 3. M EAN - FIELD INFERENCE AS A MONOTONE DEQ
Revised: 3. M EAN - FIELD INFERENCE AS A MONOTONE DEQ

Original: In this section, we formally present how to formulate the mean-field inference as a DEQ update.
Revised: In this section, we formally present how to formulate the mean-field inference as a DEQ update.

Original: Recall from before that we are modelling a distribution of the form eq.
Revised: Recall from before that we are modelling a distribution of the form eq.

Original: We are interested in approximating the conditional distribution p ( x h | x o ) , where o and h denote the observed and hidden variables respectively, with a factored distribution q ( x h )
Revised: We are interested in approximating the conditional distribution p ( x h | x o ) , where o and h denote the observed and hidden variables respectively, with a factored distribution q ( x h )

Original: =
Revised: =

Original: (cid:81)
Revised: (cid:81)

Original: i ∈
Revised: i ∈

Original: h q i ( x i ) .
Revised: h q i ( x i ) .

Original: Here, the standard mean-field updates (which minimize the KL divergence between q ( x h ) and p ( x h | x o ) over the single distribution q i ( x i ) ) are given by the following equation,
Revised: Here, the standard mean-field updates (which minimize the KL divergence between q ( x h ) and p ( x h | x o ) over the single distribution q i ( x i ) ) are given by the following equation,

Original: [Equation]
Revised: [Equation]

Original: where overloading notation slightly, we let q j ( x j ) denote a one-hot encoding of the observed value for any j ∈
Revised: where overloading notation slightly, we let q j ( x j ) denote a one-hot encoding of the observed value for any j ∈

Original: o (see e.g., Koller & Friedman (2009) for a full derivation).
Revised: o (see e.g., Koller & Friedman (2009) for a full derivation).

Original: The essence of the above updates is a characterization of the joint fixed point to mean-field inference.
Revised: The essence of the above updates is a characterization of the joint fixed point to mean-field inference.

Original: For simplicity of notation, defining q =
Revised: For simplicity of notation, defining q =

Original: [ q 1 ( x 1 ) q 2 ( x 2 ) . .
Revised: [ q 1 ( x 1 ) q 2 ( x 2 ) . .

Original: T .
Revised: T .

Original: We see that q h is a joint fixed point of all the mean-field updates if and only if
Revised: We see that q h is a joint fixed point of all the mean-field updates if and only if

Original: [Equation]
Revised: [Equation]

Original: where x o analogously denotes the stacked one-hot encoding of the observed variables.
Revised: where x o analogously denotes the stacked one-hot encoding of the observed variables.

Original: We briefly recall the monotone DEQ framework of Winston & Kolter (2020).
Revised: We briefly recall the monotone DEQ framework of Winston & Kolter (2020).

Original: Given input vector x , a monotone DEQ computes the fixed point
Revised: Given input vector x , a monotone DEQ computes the fixed point

Original: z
Revised: z

Original: ⋆ ( x ) that satisfies the equilibrium equation
Revised: ⋆ ( x ) that satisfies the equilibrium equation

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Winston & Kolter (2020) showed that if: 1) σ is given by a proximal operator 1 σ ( x )
Revised: Winston & Kolter (2020) showed that if: 1) σ is given by a proximal operator 1 σ ( x )

Original: = prox 1 f ( x ) for some convex closed proper (CCP) f , and 2) if we have the monotonicity condition I − W ⪰ mI (in the positive semidefinite sense) for some m > 0 , then for any x there exists a unique fixed point
Revised: = prox 1 f ( x ) for some convex closed proper (CCP) f , and 2) if we have the monotonicity condition I − W ⪰ mI (in the positive semidefinite sense) for some m > 0 , then for any x there exists a unique fixed point

Original: z ⋆ ( x ) , which can be computed through standard operator splitting methods, such as forwardbackward splitting.
Revised: z ⋆ ( x ) , which can be computed through standard operator splitting methods, such as forwardbackward splitting.

Original: We now state our main claim of this subsection, that under certain conditions the mean-field fixed point can be viewed as the fixed point of an analogous DEQ.
Revised: We now state our main claim of this subsection, that under certain conditions the mean-field fixed point can be viewed as the fixed point of an analogous DEQ.

Original: This is formalized in the following proposition.
Revised: This is formalized in the following proposition.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Suppose that the pairwise kernel Φ satisfies I − Φ ⪰ mI 2 for m > 0 .
Revised: Suppose that the pairwise kernel Φ satisfies I − Φ ⪰ mI 2 for m > 0 .

Original: Then the mean-field fixed point
Revised: Then the mean-field fixed point

Original: [Equation]
Revised: [Equation]

Original: ) corresponds to the fixed point of a monotone DEQ model.
Revised: ) corresponds to the fixed point of a monotone DEQ model.

Original: Specifically, this implies that for any x o , there exists a unique, globally-optimal fixed point of the mean-field distribution q h .
Revised: Specifically, this implies that for any x o , there exists a unique, globally-optimal fixed point of the mean-field distribution q h .

Original: As the monotonicity condition of the monotone DEQ is assumed in the proposition, the proof of the proposition rests entirely in showing that the softmax operator is given by prox 1 f for some CCP f .
Revised: As the monotonicity condition of the monotone DEQ is assumed in the proposition, the proof of the proposition rests entirely in showing that the softmax operator is given by prox 1 f for some CCP f .

Original: Specifically, as shown in (Kr ¨ ahenb ¨ uhl & Koltun, 2013), this is the case for f (
Revised: Specifically, as shown in (Kr ¨ ahenb ¨ uhl & Koltun, 2013), this is the case for f (

Original: [Equation]
Revised: [Equation]

Original: i.e., the restriction of the entropy minus squared norm to the simplex (note that even though we are subtracting a squared norm term it is straightforward to show that this function is convex, since the second derivatives are given by 

Revised: i.e., the restriction of the entropy minus squared norm to the simplex (note that even though we are subtracting a squared norm term it is straightforward to show that this function is convex, since the second derivatives are given by

Original: /z i − 1 , which is always non-negative over its domain).
Revised: /z i − 1 , which is always non-negative over its domain).

Original: 3. P RACTICAL CONSIDERATIONS WHEN MODELLING MONOTONE B OLTZMANN MACHINES
Revised: 3. P RACTICAL CONSIDERATIONS WHEN MODELLING MONOTONE B OLTZMANN MACHINES

Original: The construction in section 3.1 guarantees monotonicity of the resulting pairwise probabilistic model.
Revised: The construction in section 3.1 guarantees monotonicity of the resulting pairwise probabilistic model.

Original: However, instantiating the model in practice, where the variables represent hidden units of a deep architecture (i.e., representing multi-channel image tensors with pairwise potentials defined by convolutional operators), requires substantial subtlety and care in implementation.
Revised: However, instantiating the model in practice, where the variables represent hidden units of a deep architecture (i.e., representing multi-channel image tensors with pairwise potentials defined by convolutional operators), requires substantial subtlety and care in implementation.

Original: In this setting, we do not want to actually represent A explicitly, but rather determine a method for multiplying Av and A T v for some vector v (as we see in section 3.2, this is all that is required for the parallel mean-field inference method we propose).
Revised: In this setting, we do not want to actually represent A explicitly, but rather determine a method for multiplying Av and A T v for some vector v (as we see in section 3.2, this is all that is required for the parallel mean-field inference method we propose).

Original: This means that certain blocks of A are typically parameterized as convolutional layers, with convolution and transposed convolution operators as the main units of computation.
Revised: This means that certain blocks of A are typically parameterized as convolutional layers, with convolution and transposed convolution operators as the main units of computation.

Original: More specifically, we typically want to partition the full set of hidden units into some K distinct sets
Revised: More specifically, we typically want to partition the full set of hidden units into some K distinct sets

Original: [Equation]
Revised: [Equation]

Original: where e.g., q i would be best represented as a height × width × groups × cardinality tensor (i.e., a collection of a multiple hidden units corresponding to different locations in a typical deep network hidden layer).
Revised: where e.g., q i would be best represented as a height × width × groups × cardinality tensor (i.e., a collection of a multiple hidden units corresponding to different locations in a typical deep network hidden layer).

Original: Note that here q i is not the same as q i ( x i ) , but rather the collection of many different individual variables.
Revised: Note that here q i is not the same as q i ( x i ) , but rather the collection of many different individual variables.

Original: These q i terms can be related to each other via different operators, and a natural manner of parameterizing
Revised: These q i terms can be related to each other via different operators, and a natural manner of parameterizing

Original: A in this case is as an interconnected set of a convolutional or dense operators.
Revised: A in this case is as an interconnected set of a convolutional or dense operators.

Original: To represent the pairwise interactions, we can create a similarly-factored matrix A , e.g., one of the form
Revised: To represent the pairwise interactions, we can create a similarly-factored matrix A , e.g., one of the form

Original: [Equation]
Revised: [Equation]

Original: where e.g., A ij is a (possibly strided) convolution mapping between the tensors representing q j and q i .
Revised: where e.g., A ij is a (possibly strided) convolution mapping between the tensors representing q j and q i .

Original: In this case, we emphasize that A ij is not the kernel matrix that one “slides” along the variables.
Revised: In this case, we emphasize that A ij is not the kernel matrix that one “slides” along the variables.

Original: Instead, A ij is the linear mapping as if we write the convolution as a matrix-matrix multiplication.
Revised: Instead, A ij is the linear mapping as if we write the convolution as a matrix-matrix multiplication.

Original: For example, a 2D convolution with stride 1 can be expressed as a doubly block circulant matrix (the case is more complicated when different striding is allowed).
Revised: For example, a 2D convolution with stride 1 can be expressed as a doubly block circulant matrix (the case is more complicated when different striding is allowed).

Original: This parametrization is effectively a general Boltzmann machine, since each random variable in eq.
Revised: This parametrization is effectively a general Boltzmann machine, since each random variable in eq.

Original: (12) can interactive with any other variables except for itself.
Revised: (12) can interactive with any other variables except for itself.

Original: Varying A ij , the formulation in eq.
Revised: Varying A ij , the formulation in eq.

Original: (13) is rich enough for any types of architectures including convolutions, fully-connected layers, and skip-connections, etc.
Revised: (13) is rich enough for any types of architectures including convolutions, fully-connected layers, and skip-connections, etc.

Original: An illustration of one possible network structure is shown in fig.
Revised: An illustration of one possible network structure is shown in fig.

Original: The precise details of how one computes the block diagonal elements of A T A , and how one normalizes the proper diagonal blocks (which, we emphasize, still just requires computing the singular values of matrices whose size is the cardinality of a single q i ( x i ) ) are somewhat involved, so we defer a complete description to the Appendix (and accompanying code).
Revised: The precise details of how one computes the block diagonal elements of A T A , and how one normalizes the proper diagonal blocks (which, we emphasize, still just requires computing the singular values of matrices whose size is the cardinality of a single q i ( x i ) ) are somewhat involved, so we defer a complete description to the Appendix (and accompanying code).

Original: The larger takeaway message, though, is that it is possible to parameterize complex convolutional multi-scale Boltzmann machines, all while ensuring monotonicity .
Revised: The larger takeaway message, though, is that it is possible to parameterize complex convolutional multi-scale Boltzmann machines, all while ensuring monotonicity .

Original: 3. E FFICIENT PARALLEL SOLVING FOR THE MEAN - FIELD FIXED POINT
Revised: 3. E FFICIENT PARALLEL SOLVING FOR THE MEAN - FIELD FIXED POINT

Original: Although the monotonicity of Φ guarantees the existence of a unique solution, it does not necessarily guarantee that the simple iteration
Revised: Although the monotonicity of Φ guarantees the existence of a unique solution, it does not necessarily guarantee that the simple iteration

Original: [Equation]
Revised: [Equation]

Original: will converge to this solution.
Revised: will converge to this solution.

Original: Instead, to guarantee convergence, one needs to apply the damped iteration (see, e.g. (Winston & Kolter, 2020))
Revised: Instead, to guarantee convergence, one needs to apply the damped iteration (see, e.g. (Winston & Kolter, 2020))

Original: [Equation]
Revised: [Equation]

Original: The damped forward-backward iteration converges linearly to the unique fixed point if α ≤ 2 m/L 2 , assuming I − Φ is m -strongly monotone and L -Lipschitz (Ryu & Boyd, 2016).
Revised: The damped forward-backward iteration converges linearly to the unique fixed point if α ≤ 2 m/L 2 , assuming I − Φ is m -strongly monotone and L -Lipschitz (Ryu & Boyd, 2016).

Original: Crucially, this update can be formed in parallel over all the variables in the network: we do not require a coordinate descent approach as is typically needed by mean-field inference.
Revised: Crucially, this update can be formed in parallel over all the variables in the network: we do not require a coordinate descent approach as is typically needed by mean-field inference.

Original: The key issue, though is that while prox 1 f ( x ) = softmax( x ) for f defined as in eq.
Revised: The key issue, though is that while prox 1 f ( x ) = softmax( x ) for f defined as in eq.

Original: (11), in general this does not hold for α ̸ = 1 .
Revised: (11), in general this does not hold for α ̸ = 1 .

Original: Indeed, for α ̸ = 1 , there is no closed form solution to the proximal operation, and computing the solution is substantially more involved.
Revised: Indeed, for α ̸ = 1 , there is no closed form solution to the proximal operation, and computing the solution is substantially more involved.

Original: Specifically, computing this proximal operator involves solving the optimization problem
Revised: Specifically, computing this proximal operator involves solving the optimization problem

Original: [Equation]
Revised: [Equation]

Original: The following theorem, proved in the Appendix, characterizes the solution to this problem for α ∈ (0 , 1) (although it is also possible to compute solutions for α > 1 , this is not needed in practice, as it corresponds to a “negatively damped” update, and it is typically better to simply use the softmax update in such cases).
Revised: The following theorem, proved in the Appendix, characterizes the solution to this problem for α ∈ (0 , 1) (although it is also possible to compute solutions for α > 1 , this is not needed in practice, as it corresponds to a “negatively damped” update, and it is typically better to simply use the softmax update in such cases).

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Given f as defined in eq.
Revised: Given f as defined in eq.

Original: (11) , α ∈ (0 , 1) , and x ∈ R k , the proximal operator prox αf ( x ) is given by
Revised: (11) , α ∈ (0 , 1) , and x ∈ R k , the proximal operator prox αf ( x ) is given by

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ R is the unique solution chosen to ensure that the resulting (cid:80) i prox αf ( x i ) = 1 , and where W ( · ) is the principal branch of the Lambert W function.
Revised: where λ ∈ R is the unique solution chosen to ensure that the resulting (cid:80) i prox αf ( x i ) = 1 , and where W ( · ) is the principal branch of the Lambert W function.

Original: In practice, however, this is not the most numerically stable method for computing the proximal operator, especially for small α , owing to the large term inside the exponential.
Revised: In practice, however, this is not the most numerically stable method for computing the proximal operator, especially for small α , owing to the large term inside the exponential.

Original: Computing the proximal operation efficiently is somewhat involved, though briefly, we define the alternative function
Revised: Computing the proximal operation efficiently is somewhat involved, though briefly, we define the alternative function

Original: [Equation]
Revised: [Equation]

Original: and show how to directly compute g ( y ) using Halley’s method (note that Halley’s method is also the preferred manner to computing the Lambert W function itself numerically (Corless et al., 1996)).
Revised: and show how to directly compute g ( y ) using Halley’s method (note that Halley’s method is also the preferred manner to computing the Lambert W function itself numerically (Corless et al., 1996)).

Original: Finding the prox operator then requires that we find λ such that (cid:80) ki =1 exp( g ( x i + λ ))
Revised: Finding the prox operator then requires that we find λ such that (cid:80) ki =1 exp( g ( x i + λ ))

Original: = 1 .
Revised: = 1 .

Original: This can be done via (one-dimensional) root finding with Newton’s method, which is guaranteed to always find a solution here, owing to the fact that this function is convex monotonic for λ ∈ ( −∞ , 1) .
Revised: This can be done via (one-dimensional) root finding with Newton’s method, which is guaranteed to always find a solution here, owing to the fact that this function is convex monotonic for λ ∈ ( −∞ , 1) .

Original: We can further compute the gradients of the g function and of the proximal operator itself via implicit differentiation (i.e., we can do it analytically without requiring unrolling the Newton or Halley iteration).
Revised: We can further compute the gradients of the g function and of the proximal operator itself via implicit differentiation (i.e., we can do it analytically without requiring unrolling the Newton or Halley iteration).

Original: We describe the details in the Appendix, and include an efficient PyTorch function implementation in the supplementary material.
Revised: We describe the details in the Appendix, and include an efficient PyTorch function implementation in the supplementary material.

Original: Comparison to Winston & Kolter (2020)
Revised: Comparison to Winston & Kolter (2020)

Original: Although this work uses the same monotonicity constraint as in Winston & Kolter (2020), our result further requires the linear module Φ to be hollow, and extend their work to the softmax nonlinear operator as well.
Revised: Although this work uses the same monotonicity constraint as in Winston & Kolter (2020), our result further requires the linear module Φ to be hollow, and extend their work to the softmax nonlinear operator as well.

Original: These extensions introduce significant complications, but also enable us to interpret our network as a probabilistic model, while the network in Winston & Kolter (2020) cannot.
Revised: These extensions introduce significant complications, but also enable us to interpret our network as a probabilistic model, while the network in Winston & Kolter (2020) cannot.

Original: 3. T RAINING CONSIDERATIONS
Revised: 3. T RAINING CONSIDERATIONS

Original: Finally, we discuss approaches for training these monotone Boltzmann machines, exploiting their efficient approach to mean-field inference.
Revised: Finally, we discuss approaches for training these monotone Boltzmann machines, exploiting their efficient approach to mean-field inference.

Original: Probabilistic models are typically trained via approximate likelihood maximization, and since the mean-field approximation is based upon a particular likelihood approximation, it may seem most natural to use this same approximation to train parameters.
Revised: Probabilistic models are typically trained via approximate likelihood maximization, and since the mean-field approximation is based upon a particular likelihood approximation, it may seem most natural to use this same approximation to train parameters.

Original: In practice, however, this is often a suboptimal approach.
Revised: In practice, however, this is often a suboptimal approach.

Original: Specifically, because our forward inference procedure ultimately uses mean-field inference, it is better to train the model directly to output the correct marginals, when running this mean-field procedure .
Revised: Specifically, because our forward inference procedure ultimately uses mean-field inference, it is better to train the model directly to output the correct marginals, when running this mean-field procedure .

Original: This is known as a marginalbased loss (Domke, 2013).
Revised: This is known as a marginalbased loss (Domke, 2013).

Original: In the context of monotone Boltzmann machines, this procedure has a particularly convenient form, as it corresponds roughly to the “typical” training of DEQ.
Revised: In the context of monotone Boltzmann machines, this procedure has a particularly convenient form, as it corresponds roughly to the “typical” training of DEQ.

Original: In more detail, suppose we are given a sample x ∈ X (i.e., at training time the entire sample is given), along with a specification of the “observed” and “hidden” sets, o and h respectively.
Revised: In more detail, suppose we are given a sample x ∈ X (i.e., at training time the entire sample is given), along with a specification of the “observed” and “hidden” sets, o and h respectively.

Original: Note that the choice of observed and hidden sets is potentially up to the algorithm designer, and can effectively allows one to train our model in a “self-supervised” fashion, where the goal is to predict some unobserved components from others.
Revised: Note that the choice of observed and hidden sets is potentially up to the algorithm designer, and can effectively allows one to train our model in a “self-supervised” fashion, where the goal is to predict some unobserved components from others.

Original: In practice, however, one typically wants to design hidden and observed portions congruent with the eventual use of the model: e.g., if one is using the model for classification, then at training time it makes sense for the label to be “hidden” and the input to be “observed.”
Revised: In practice, however, one typically wants to design hidden and observed portions congruent with the eventual use of the model: e.g., if one is using the model for classification, then at training time it makes sense for the label to be “hidden” and the input to be “observed.”

Original: Given this sample, we first solve the mean-field inference problem to find q ⋆h ( x h ) such that
Revised: Given this sample, we first solve the mean-field inference problem to find q ⋆h ( x h ) such that

Original: [Equation]
Revised: [Equation]

Original: For this sample, we know that the true value of the hidden states is given by x h .
Revised: For this sample, we know that the true value of the hidden states is given by x h .

Original: Thus, we can apply some loss function ℓ ( q ⋆h , x h ) between the prediction and true value, and update parameters of the model θ = { A , b } using their gradients
Revised: Thus, we can apply some loss function ℓ ( q ⋆h , x h ) between the prediction and true value, and update parameters of the model θ = { A , b } using their gradients

Original: [Equation]
Revised: [Equation]

Revised: with g ( q ⋆h ) ≡ prox αf ((1 − α ) q ∗ h

Revised: + α

Revised: ( Φ hh q ∗ h

Revised: +

Original: with g ( q ⋆h ) ≡ softmax ( Φ hh q ⋆h + Φ ho x o + b o ) and where the last equality comes from the standard application of the implicit function theorem as typical in DEQs or monotone DEQs.
Revised: Φ ho x o + b h )) . and where the last equality comes from the standard application of the implicit function theorem as typical in DEQs or monotone DEQs.

Original: This backward pass can also be computed via an iterative approach, and here the details exactly mirror that of Winston & Kolter (2020).
Revised: This backward pass can also be computed via an iterative approach, and here the details exactly mirror that of Winston & Kolter (2020).

Original: As a final note, we also mention that owning to the restricted range of weights allowed by the monotonicty constraint, the actual output marginals
Revised: As a final note, we also mention that owning to the restricted range of weights allowed by the monotonicty constraint, the actual output marginals

Original: q
Revised: q

Original: i ( x i ) are often more uniform in distribution than desired.
Revised: i ( x i ) are often more uniform in distribution than desired.

Original: Thus, we typically apply the loss to a scaled marginal 
Revised: Thus, we typically apply the loss to a scaled marginal ˜ q

Original: [Equation] where τ i ∈ R
Revised: i ( x i ) ∝ q i ( x i ) τ

Original:  + is a variable-dependent learnable temperature parameter.
Revised: i (20) where τ i ∈ R + is a variable-dependent learnable temperature parameter.

Original: Importantly, we emphasize that this is only done after convergence to the mean-field solution, and thus only applies to the marginals to which we apply a loss: the actual internal iterations of mean-field cannot have such a scaling, as it would violate the monotonicity condition.
Revised: Importantly, we emphasize that this is only done after convergence to the mean-field solution, and thus only applies to the marginals to which we apply a loss: the actual internal iterations of mean-field cannot have such a scaling, as it would violate the monotonicity condition.

Original:  E XPERIMENTAL EVALUATION
Revised:  E XPERIMENTAL EVALUATION

Original: We evaluate the performance of our model primarily on the MNIST dataset.
Revised: We evaluate the performance of our model primarily on the MNIST dataset.

Original: Our model is able to approximate any conditional distribution.
Revised: Our model is able to approximate any conditional distribution.

Original: Here we demonstrate how to model missing pixels and the class digits conditioned on the observed pixels, as well as model missing pixels conditioned on class digits and observed pixels.
Revised: Here we demonstrate how to model missing pixels and the class digits conditioned on the observed pixels, as well as model missing pixels conditioned on class digits and observed pixels.

Original: Although MNIST is of course a small-scale problem, the goal here is to demonstrate joint inference and learning over what is still a reasonably-sized joint model, considering the number of hidden units.
Revised: Although MNIST is of course a small-scale problem, the goal here is to demonstrate joint inference and learning over what is still a reasonably-sized joint model, considering the number of hidden units.

Original: Nonetheless, the current experiment is admittedly largely a demonstration of the proposed method rather than a full accounting of its performance, a point we highlight in the subsequent section as well.
Revised: Nonetheless, the current experiment is admittedly largely a demonstration of the proposed method rather than a full accounting of its performance, a point we highlight in the subsequent section as well.

Original: After describing the MNIST experiments, we detail a similar experiment on CIFAR-10 to demonstrate the potential for scaling the approach.
Revised: After describing the MNIST experiments, we detail a similar experiment on CIFAR-10 to demonstrate the potential for scaling the approach.

Original: We also show how our inference method differs from the previous ones.
Revised: We also show how our inference method differs from the previous ones.

Original: On the joint imputation and classification task, we train models using our updates and the updates in Kr ¨ ahenb ¨ uhl & Koltun (2013); Baqu´e et al.
Revised: On the joint imputation and classification task, we train models using our updates and the updates in Kr ¨ ahenb ¨ uhl & Koltun (2013); Baqu ´ e et al.

Original: (2016), and inference each using all three update methods, with and without the monotonicity constraint.
Revised: (2016), and inference each using all three update methods, with and without the monotonicity constraint.

Original: We show that our inference method is superior in both performance and convergence speed.
Revised: We show that our inference method is superior in both performance and convergence speed.

Original: Numerically, we also demonstrate that the other two methods could either diverge or not converging to the actual mean-field fixed point.
Revised: Numerically, we also demonstrate that the other two methods could either diverge or not converging to the actual mean-field fixed point.

Original: All deferred experiments and details can be found in the appendix.
Revised: All deferred experiments and details can be found in the appendix.

Original: Experiment setup The original MNIST dataset has one channel representing the gray-scale intensity, ranging between 0 and 1 .
Revised: Experiment setup The original MNIST dataset has one channel representing the gray-scale intensity, ranging between 0 and 1 .

Original: Here we adopt the strategy of Van Oord et al.
Revised: Here we adopt the strategy of Van Oord et al.

Original: (2016) to convert this continuous distribution to a discrete one.
Revised: (2016) to convert this continuous distribution to a discrete one.

Original: We bin the intensity evenly to 4 categories { 0 , . . .
Revised: We bin the intensity evenly to 4 categories { 0 , . . .

Original: , 3 } , and for each channel uses a one-hot encoding of the category so that the input data has shape 4 × 28 × 28 .
Revised: , 3 } , and for each channel uses a one-hot encoding of the category so that the input data has shape 4 × 28 × 28 .

Original: We remark that the number of categories is chosen arbitrarily and can be any integer.
Revised: We remark that the number of categories is chosen arbitrarily and can be any integer.

Original: For image with missing data, if the pixels are randomly masked, we always mask each pixel off independently with probability 60% , such that in expectation only 40% pixels are observed.
Revised: For image with missing data, if the pixels are randomly masked, we always mask each pixel off independently with probability 60% , such that in expectation only 40% pixels are observed.

Original: If a whole patch of pixels is masked, we randomly pick a 14 × 14 patch.
Revised: If a whole patch of pixels is masked, we randomly pick a 14 × 14 patch.

Original: To make the model class richer, in the patch case we lift the monotonicity constraint, and the model converges regardless.
Revised: To make the model class richer, in the patch case we lift the monotonicity constraint, and the model converges regardless.

Original: We also conduct the same set of MNIST experiments using a 3-layer DBM and include the results in the appendix.
Revised: We also conduct the same set of MNIST experiments using a 3-layer DBM and include the results in the appendix.

Original: Results on imputation and classification Training the above described model for 40 epochs with 40% missing pixels leads to a 92 .
Revised: Results on imputation and classification Training the above described model for 40 epochs with 40% missing pixels leads to a 92 .

Original: 95% test accuracy, and the image reconstruction result is shown in fig.
Revised: 95% test accuracy, and the image reconstruction result is shown in fig.

Original: 4d.
Revised: 4d.

Original: Based on the reconstructed images and our classification results, our model successfully recovers the unimodal distribution conditioned on the observed pixels.
Revised: Based on the reconstructed images and our classification results, our model successfully recovers the unimodal distribution conditioned on the observed pixels.

Original: Despite working simultaneously on two different tasks, our model can simply be trained jointly, taking full advantages of the existing auto-differentiation frameworks without additional burden.
Revised: Despite working simultaneously on two different tasks, our model can simply be trained jointly, taking full advantages of the existing auto-differentiation frameworks without additional burden.

Original: It is worth noting that while the autoregressive models of Van Oord et al.
Revised: It is worth noting that while the autoregressive models of Van Oord et al.

Original: (2016) are comparable to ours, they require sampling pixels one-by-one in a sequential order at inference time, whereas our model does not assume an underlying sequence and can generate all pixels at once.
Revised: (2016) are comparable to ours, they require sampling pixels one-by-one in a sequential order at inference time, whereas our model does not assume an underlying sequence and can generate all pixels at once.

Original: Comparing to the differently-parameterized monDEQ in Winston & Kolter (2020), whose linear module suffers from drastically increasing condition number (hence in later epochs taking around 20 steps to converge, even with tuned α ), our parameterization produces a much nicer convergence pattern: the average number of forward iterations over the 40 training epochs is less than 6 steps, see fig.
Revised: Comparing to the differently-parameterized monDEQ in Winston & Kolter (2020), whose linear module suffers from drastically increasing condition number (hence in later epochs taking around 20 steps to converge, even with tuned α ), our parameterization produces a much nicer convergence pattern: the average number of forward iterations over the 40 training epochs is less than 6 steps, see fig.

Original: When the missing pixels form
Revised: When the missing pixels form

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: consecutive patches, our model reconstructs readable digits despite potentially large chunk of missing pixels (fig. 4e).
Revised: consecutive patches, our model reconstructs readable digits despite potentially large chunk of missing pixels (fig. 4e).

Original: Meanwhile, if the model is given the image labels as input injections, our model performs conditionaly generation fairly well (fig.
Revised: Meanwhile, if the model is given the image labels as input injections, our model performs conditionaly generation fairly well (fig.

Original: 4f).
Revised: 4f).

Original: These results demonstrate the flexibility of our parameterization for modelling different conditional distributions.
Revised: These results demonstrate the flexibility of our parameterization for modelling different conditional distributions.

Original: CIFAR-

Revised: CIFAR-

Original: Experiment We additionally conduct an experiment on the simultaneous tasks of image pixel imputation and label prediction given partially observed features.
Revised: Experiment We additionally conduct an experiment on the simultaneous tasks of image pixel imputation and label prediction given partially observed features.

Original: Model architecture and training details are given in the appendix.
Revised: Model architecture and training details are given in the appendix.

Original: With 50 % of the pixels observed, the model obtains 58 % test accuracy, and can impute the missing pixels effectively (see fig. 5).
Revised: With 50 % of the pixels observed, the model obtains 58 % test accuracy, and can impute the missing pixels effectively (see fig. 5).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we give a monotone parameterization for general Boltzmann machines, and connect its mean-field fixed point to a monotone DEQ model.
Revised: In this work, we give a monotone parameterization for general Boltzmann machines, and connect its mean-field fixed point to a monotone DEQ model.

Original: We provide a mean-field update method that is proven to be globally convergent.
Revised: We provide a mean-field update method that is proven to be globally convergent.

Original: Our parameterization allows for full parallelization of mean-field updates without restricting the potential function to be concave, thus addressing issues with prior approaches.
Revised: Our parameterization allows for full parallelization of mean-field updates without restricting the potential function to be concave, thus addressing issues with prior approaches.

Original: Moreover, we allow complicated and hierarchical structures among the variables and show how to efficiently implement them.
Revised: Moreover, we allow complicated and hierarchical structures among the variables and show how to efficiently implement them.

Original: For parameter learning, we directly optimize the marginalbased loss over the mean-field variational family, circumventing the intractability of computing the partition function.
Revised: For parameter learning, we directly optimize the marginalbased loss over the mean-field variational family, circumventing the intractability of computing the partition function.

Original: Our model is evaluated on the MNIST and CIFAR-10 dataset for simultaneously predicting with missing data and imputing the missing data itself.
Revised: Our model is evaluated on the MNIST and CIFAR-10 dataset for simultaneously predicting with missing data and imputing the missing data itself.

Original: There are several potential future directions.
Revised: There are several potential future directions.

Original: First, the monotone parameterization in theorem 3.1 is not an if-and-only-if statement, hence potentially making our model more restrictive than necessary.
Revised: First, the monotone parameterization in theorem 3.1 is not an if-and-only-if statement, hence potentially making our model more restrictive than necessary.

Original: The second direction is that although we have a fairly efficient implementation of prox αf , it is still slower than normal nonlinearities like ReLU or softmax.
Revised: The second direction is that although we have a fairly efficient implementation of prox αf , it is still slower than normal nonlinearities like ReLU or softmax.

Original: It is an interesting direction to more efficiently scale these models.
Revised: It is an interesting direction to more efficiently scale these models.

Original: A A PPENDIX A. D EFERRED PROOFS
Revised: A A PPENDIX A. D EFERRED PROOFS

Original: Proof of theorem 3.2.
Revised: Proof of theorem 3.2.

Original: By definition, the proximal operator induced by f (the same f in eq.
Revised: By definition, the proximal operator induced by f (the same f in eq.

Original: and α solves the following optimization problem:
Revised: and α solves the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: of which the KKT condition is
Revised: of which the KKT condition is

Original: [Equation]
Revised: [Equation]

Original: We have that µ i = 0 is feasible and the first equation of the above KKT condition can be massaged as
Revised: We have that µ i = 0 is feasible and the first equation of the above KKT condition can be massaged as

Original: [Equation]
Revised: [Equation]

Original: where W is the lambert W function.
Revised: where W is the lambert W function.

Original: Notice here z i > 0 .
Revised: Notice here z i > 0 .

Original: Our primal problem is convex and Slater’s condition holds.
Revised: Our primal problem is convex and Slater’s condition holds.

Original: Hence, we conclude that
Revised: Hence, we conclude that

Original: [Equation]
Revised: [Equation]

Original: A. C ONVOLUTION NETWORK
Revised: A. C ONVOLUTION NETWORK

Original: It is clear that the monotone parameterization in section 3 directly applies to fully-connected networks, and all the related quantities can be calculated easily.
Revised: It is clear that the monotone parameterization in section 3 directly applies to fully-connected networks, and all the related quantities can be calculated easily.

Original: Nonetheless, the real power of the DEQ model comes in when we use more sophisticated linear operators like convolutions.
Revised: Nonetheless, the real power of the DEQ model comes in when we use more sophisticated linear operators like convolutions.

Original: In the context of Boltzmann machines, the convolution operator gives edge potentials beneficial structures.
Revised: In the context of Boltzmann machines, the convolution operator gives edge potentials beneficial structures.

Original: For example, when modeling the joint probability of pixels in an image, it is intuitive that only the nearby pixels depend closely on each other.
Revised: For example, when modeling the joint probability of pixels in an image, it is intuitive that only the nearby pixels depend closely on each other.

Original: Let A ∈ R
Revised: Let A ∈ R

Original: k
Revised: k

Original: × k × r ×
Revised: × k × r ×

Original: r denote a convolutional tensor with kernel size r and channel size k , let x denote some input.
Revised: r denote a convolutional tensor with kernel size r and channel size k , let x denote some input.

Original: For a convolution with stride 1 , the block diagonal elements of A T A simply form a 1 × 1 convolution.
Revised: For a convolution with stride 1 , the block diagonal elements of A T A simply form a 1 × 1 convolution.

Original: In particular, we apply the convolutions
Revised: In particular, we apply the convolutions

Original: [Equation]
Revised: [Equation]

Original: where ˜ A is a 1 × 1 convolution given by
Revised: where ˜ A is a 1 × 1 convolution given by

Original: [Equation]
Revised: [Equation]

Original: We can normalize by the spectral norm of ˜ A term to ensure strong monotonicity.
Revised: We can normalize by the spectral norm of ˜ A term to ensure strong monotonicity.

Original: Since ˜ A can be rewritten as a k × k matrix and k is usually small, its spectral norm can be easily calculated.
Revised: Since ˜ A can be rewritten as a k × k matrix and k is usually small, its spectral norm can be easily calculated.

Original: It takes more effort to work out convolutions with stride other than 1 .
Revised: It takes more effort to work out convolutions with stride other than 1 .

Original: Specifically, the block diagonal terms do not form a 1 × 1 convolution anymore, instead, the computation varies depending on the location.
Revised: Specifically, the block diagonal terms do not form a 1 × 1 convolution anymore, instead, the computation varies depending on the location.

Original: It is easier to see the computation directly in the accompanying code.
Revised: It is easier to see the computation directly in the accompanying code.

Original: Grouped channels It is crucial to introduce the concept of grouped channels, which allows us to represent multiple categorical variables in a single location, such as the three categorical variables representing the three (binned) color channels of an RGB pixel.
Revised: Grouped channels It is crucial to introduce the concept of grouped channels, which allows us to represent multiple categorical variables in a single location, such as the three categorical variables representing the three (binned) color channels of an RGB pixel.

Original: In this case, each of the three
Revised: In this case, each of the three

Original: RGB channels will be represented by a different group of k channels representing the k bins.
Revised: RGB channels will be represented by a different group of k channels representing the k bins.

Original: The grouping is achieved by having the nonlinearity function (softmax) applied to each group separately.
Revised: The grouping is achieved by having the nonlinearity function (softmax) applied to each group separately.

Original: We remark that the convolutions themselves are not grouped, otherwise none of the red pixels would interact with green or blue pixels, etc.
Revised: We remark that the convolutions themselves are not grouped, otherwise none of the red pixels would interact with green or blue pixels, etc.

Original: Instead, we want all RGB channels to interact with each other (except that channel i at position ( j, k ) does not interact with itself).
Revised: Instead, we want all RGB channels to interact with each other (except that channel i at position ( j, k ) does not interact with itself).

Original: That means in eq.
Revised: That means in eq.

Original: (4), the blkdiag( ˆ A T ˆ A ) is grouped in the following way.
Revised: (4), the blkdiag( ˆ A T ˆ A ) is grouped in the following way.

Original: Recall that this block diagonal term has element of size k i ×
Revised: Recall that this block diagonal term has element of size k i ×

Original: k
Revised: k

Original: i for i ∈ [ n ] .
Revised: i for i ∈ [ n ] .

Original: This parameterization has only 1 group.
Revised: This parameterization has only 1 group.

Original: With g groups, the element of the block diagonal matrix then has size k i 

Revised: With g groups, the element of the block diagonal matrix then has size k i

Original: × k i 1 , . . .
Revised: × k i 1 , . . .

Original: k
Revised: k

Original: i g
Revised: i g

Original: × k
Revised: × k

Original: i g for i ∈ [ n ] , where (cid:80) j ∈
Revised: i g for i ∈ [ n ] , where (cid:80) j ∈

Original: [ g ] k i j = k i .
Revised: [ g ] k i j = k i .

Original: We also observe empirically that grouping the latent variables improves the performance.
Revised: We also observe empirically that grouping the latent variables improves the performance.

Original: A. E FFICIENT COMPUTATION OF prox αf
Revised: A. E FFICIENT COMPUTATION OF prox αf

Original: The solution to the proximal operator in damped forward iteration given in theorem 3.2 involves the
Revised: The solution to the proximal operator in damped forward iteration given in theorem 3.2 involves the

Original: Lambert W function, which does not attain an analytical solution.
Revised: Lambert W function, which does not attain an analytical solution.

Original: In this section, we show how to efficiently calculate the nonlinearity σ ( x i ) , as well as its Jacobian matrix for backward iteration.
Revised: In this section, we show how to efficiently calculate the nonlinearity σ ( x i ) , as well as its Jacobian matrix for backward iteration.

Original: Let f ( y ) =
Revised: Let f ( y ) =

Original: α 1 − α W (cid:0) 1 − αα exp (cid:0) yα − 1 (cid:1)(cid:1) , and we have
Revised: α 1 − α W (cid:0) 1 − αα exp (cid:0) yα − 1 (cid:1)(cid:1) , and we have

Original: [Equation]
Revised: [Equation]

Original: where the last equality uses the identity log( W ( x ))
Revised: where the last equality uses the identity log( W ( x ))

Original: =
Revised: =

Original: log x − W ( x ) .
Revised: log x − W ( x ) .

Original: Rewrite
Revised: Rewrite

Original: W (cid:0) 1 − α α exp (cid:0) y α − 1 (cid:1)(cid:1) =
Revised: W (cid:0) 1 − α α exp (cid:0) y α − 1 (cid:1)(cid:1) =

Original: f ( y ) 1 − α α and massage the terms, we have that solving log f ( y ) is equivalent to finding the root of
Revised: f ( y ) 1 − α α and massage the terms, we have that solving log f ( y ) is equivalent to finding the root of

Original: [Equation]
Revised: [Equation]

Original: Direct calculation shows that h ′ ( x ) = − α − (1 − α ) e x and h ′′ ( x ) = − (1 − α ) e x .
Revised: Direct calculation shows that h ′ ( x ) = − α − (1 − α ) e x and h ′′ ( x ) = − (1 − α ) e x .

Original: Note here y is the input and it is known to us, and x is a scalar.
Revised: Note here y is the input and it is known to us, and x is a scalar.

Original: Hence we can efficiently solve the root finding problem using Halley’s method.
Revised: Hence we can efficiently solve the root finding problem using Halley’s method.

Original: For backpropagation, we need dxdy , which can be computed by implicit differentiation:
Revised: For backpropagation, we need dxdy , which can be computed by implicit differentiation:

Original: [Equation]
Revised: [Equation]

Original: Now we can find λ s.t (cid:80) i z i
Revised: Now we can find λ s.t (cid:80) i z i

Original: = 1 using Newton’s method on g ( λ ) =
Revised: = 1 using Newton’s method on g ( λ ) =

Original: (cid:80) i e log( f ( x i + λ ))
Revised: (cid:80) i e log( f ( x i + λ ))

Original: − 1 = 0 .
Revised: − 1 = 0 .

Original: Note this is still a one-dimensional optimization problem.
Revised: Note this is still a one-dimensional optimization problem.

Original: A direct calculation shows that dgdλ =
Revised: A direct calculation shows that dgdλ =

Original: (cid:80) i e log( f ( x i + λ ))
Revised: (cid:80) i e log( f ( x i + λ ))

Original: d log( f ( x i + λ ) dλ , and above we have already calculated that
Revised: d log( f ( x i + λ ) dλ , and above we have already calculated that

Original: [Equation]
Revised: [Equation]

Original: For backward computation, by the chain rule, we have:
Revised: For backward computation, by the chain rule, we have:

Original: [Equation]
Revised: [Equation]

Original: where the last step is derived by implicit differentiation.
Revised: where the last step is derived by implicit differentiation.

Original: Now to get dλ/dx i , notice that by applying the implicit function theorem on p ( x, λ ( x ))
Revised: Now to get dλ/dx i , notice that by applying the implicit function theorem on p ( x, λ ( x ))

Original: = (cid:80) i e log( f ( x i + λ ))
Revised: = (cid:80) i e log( f ( x i + λ ))

Original: − 1 = 0 , we get
Revised: − 1 = 0 , we get

Original: [Equation]
Revised: [Equation]

Original: Thus we have all the terms computed, which finishes the derivation.
Revised: Thus we have all the terms computed, which finishes the derivation.

Original: B A DDITIONAL E XPERIMENTS AND D ETAILS
Revised: B A DDITIONAL E XPERIMENTS AND D ETAILS

Original: Here we provide the model architectures and experiment details omitted in the main text.
Revised: Here we provide the model architectures and experiment details omitted in the main text.

Original: Model architecture For MNIST experiments, using the notation in eq.
Revised: Model architecture For MNIST experiments, using the notation in eq.

Original: (13), we design a 20 -layer deep monotone DEQ with the following structure:
Revised: (13), we design a 20 -layer deep monotone DEQ with the following structure:

Original: [Equation]
Revised: [Equation]

Original: where A 11 is a 20 × 20 × 3 × 3 convolution, A 22 is a 40 × 40 × 3 × 3 convolution, A 21 is a 40 × 20 × 3 ×convolution with stride 2 , A 33 is a 80 × 80 × 3 × 3 convolution, A 31 is a 80 × 20 × 3 × 3 convolution with stride 4 , A 32 is a 80 × 40 × 3 × 3 convolution with stride 2 , A 43 is a (80 · 7 · 7) × 10 dense linear layer, and A 44 is a 10 × 10 dense linear layer.
Revised: where A 11 is a 20 × 20 × 3 × 3 convolution, A 22 is a 40 × 40 × 3 × 3 convolution, A 21 is a 40 × 20 × 3 ×convolution with stride 2 , A 33 is a 80 × 80 × 3 × 3 convolution, A 31 is a 80 × 20 × 3 × 3 convolution with stride 4 , A 32 is a 80 × 40 × 3 × 3 convolution with stride 2 , A 43 is a (80 · 7 · 7) × 10 dense linear layer, and A 44 is a 10 × 10 dense linear layer.

Original: The corresponding variable q as in eq.
Revised: The corresponding variable q as in eq.

Original: (12) then has 4 elements of shape (20 × 28 × 28) , (40 × 14 × 14) , (80 × 7 × 7) , (10 × 1) .
Revised: (12) then has 4 elements of shape (20 × 28 × 28) , (40 × 14 × 14) , (80 × 7 × 7) , (10 × 1) .

Original: When applying the proximal operator to q , we use 1 , 10 , 20 , 1 as their number of groups, respectively.
Revised: When applying the proximal operator to q , we use 1 , 10 , 20 , 1 as their number of groups, respectively.

Original: Training details and hyperparameters Treating the image reconstruction as a dense classification task, we use cross-entropy loss and class weights 1 − β 1 − β ni with β = 0 .
Revised: Training details and hyperparameters Treating the image reconstruction as a dense classification task, we use cross-entropy loss and class weights 1 − β 1 − β ni with β = 0 .

Original: (Cui et al., 2019), where n i is the number of times pixels with intensity i appear in the hidden pixels.
Revised: (Cui et al., 2019), where n i is the number of times pixels with intensity i appear in the hidden pixels.

Original: For classification, we use standard cross-entropy loss.
Revised: For classification, we use standard cross-entropy loss.

Original: To enable joint training, we put equal weight of 0 .
Revised: To enable joint training, we put equal weight of 0 .

Original: 5 on both task losses and backpropagate through their sum.
Revised: 5 on both task losses and backpropagate through their sum.

Original: For both tasks, we put τ i Φ q ∗ i into the crossentropy loss as logits, as described in eq.
Revised: For both tasks, we put τ i Φ q ∗ i into the crossentropy loss as logits, as described in eq.

Original: Since mean-field approximation is (conditionally) unimodal, the scaling grants us the ability to model more extreme distributions.
Revised: Since mean-field approximation is (conditionally) unimodal, the scaling grants us the ability to model more extreme distributions.

Original: To achieve faster damped forward-backward iteration, we implement Anderson acceleration (Walker & Ni, 2011), and stop the fixed point update as soon as the relative difference between two iterations (that is, ∥ q t +1 − q t ∥ /
Revised: To achieve faster damped forward-backward iteration, we implement Anderson acceleration (Walker & Ni, 2011), and stop the fixed point update as soon as the relative difference between two iterations (that is, ∥ q t +1 − q t ∥ /

Original: ∥ q t ∥ ) is less than 0 .
Revised: ∥ q t ∥ ) is less than 0 .

Original: 01 , unless we hit a maximum number of 50 allowed iterations.
Revised: 01 , unless we hit a maximum number of 50 allowed iterations.

Original: For prox αf and the damped iteration, we set α = 0 .
Revised: For prox αf and the damped iteration, we set α = 0 .

Original: 125 (Although one can tune down α whenever the iterations do not converge, empirically this never happens on our task).
Revised: 125 (Although one can tune down α whenever the iterations do not converge, empirically this never happens on our task).

Original: We use the Adam optimizer with learning rate 0 .
Revised: We use the Adam optimizer with learning rate 0 .

Original: Our models are trained on one GeForce GTX 2080 Ti GPU.
Revised: Our models are trained on one GeForce GTX 2080 Ti GPU.

Original: Comparison to past inference methods We conduct numerical experiments to compare our inference updating method to the ones proposed by Kr¨ahenb¨uhl & Koltun (2013); Baqu´e et al.
Revised: Comparison to past inference methods We conduct numerical experiments to compare our inference updating method to the ones proposed by Kr¨ahenb¨uhl & Koltun (2013); Baqu´e et al.

Original: (2016), denoted as Kr¨ahenb¨uhl’s and Baqu´e’s respectively.
Revised: (2016), denoted as Kr¨ahenb¨uhl’s and Baqu´e’s respectively.

Original: Kr¨ahenb¨uhl’s fast concave-convex procedure (CCCP) essentially decomposes to eq.
Revised: Kr¨ahenb¨uhl’s fast concave-convex procedure (CCCP) essentially decomposes to eq.

Original: (14), the un-damped mean-field update with softmax.
Revised: (14), the un-damped mean-field update with softmax.

Original: This update only converges provably when Φ is concave.
Revised: This update only converges provably when Φ is concave.

Original: Baqu ´ e’s inference method can be written as
Revised: Baqu ´ e’s inference method can be written as

Original: [Equation]
Revised: [Equation]

Original: This algorithm provably converges despite the property of the pairwise kernel function.
Revised: This algorithm provably converges despite the property of the pairwise kernel function.

Original: However, this procedure converges in the sense that the variational free energy keeps decreasing.
Revised: However, this procedure converges in the sense that the variational free energy keeps decreasing.

Original: Therefore their fixed point may not be the true mean-field distribution eq.
Revised: Therefore their fixed point may not be the true mean-field distribution eq.

Original: In this experiment, we train the
Revised: In this experiment, we train the

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: models using three different updating methods, and perform inference using three methods as well, with and without the monotonicity condition.
Revised: models using three different updating methods, and perform inference using three methods as well, with and without the monotonicity condition.

Original: Kr¨ahenb¨uhl’s
Revised: Kr¨ahenb¨uhl’s

Original: and Baqu´e’s methods often do not converge in the backward pass (there’s no theoretical guarantees neither).
Revised: and Baqu´e’s methods often do not converge in the backward pass (there’s no theoretical guarantees neither).

Original: To rule out the impact of the backward iteration, during training we directly update use the gradient of the forward pass, instead of using a backward gradient hook to compute eq.
Revised: To rule out the impact of the backward iteration, during training we directly update use the gradient of the forward pass, instead of using a backward gradient hook to compute eq.

Original: Figure 7 and fig.
Revised: Figure 7 and fig.

Original: 8 demonstrate how the three update methods impute missing pixels when trained with different update rules, with and without the monotonicity condition, respectively.
Revised: 8 demonstrate how the three update methods impute missing pixels when trained with different update rules, with and without the monotonicity condition, respectively.

Original: Kr ¨ ahenb ¨ uhl’s usually does not converge when the model is trained with our method or
Revised: Kr ¨ ahenb ¨ uhl’s usually does not converge when the model is trained with our method or

Original: Baqu ´ e’s, whereas the other two methods impute the missing pixels well.
Revised: Baqu ´ e’s, whereas the other two methods impute the missing pixels well.

Original: The classification results are presented in table 3 and table 4.
Revised: The classification results are presented in table 3 and table 4.

Original: Notice that when trained with our method or Baqu ´ e’s, the convergence issue of Kr ¨ ahenb ¨ uhl’s leads to horrible classification accuracy.
Revised: Notice that when trained with our method or Baqu ´ e’s, the convergence issue of Kr ¨ ahenb ¨ uhl’s leads to horrible classification accuracy.

Original: Our method is superior to other inference methods when the model is trained in a different update fashion.
Revised: Our method is superior to other inference methods when the model is trained in a different update fashion.

Original: For example, if the model is trained by using Kr¨ahenb¨uhl’s, it makes sense that the model performs the best if the inference is also Kr ¨ ahenb ¨ uhl’s since the parameters are biased toward that particular inference method.
Revised: For example, if the model is trained by using Kr¨ahenb¨uhl’s, it makes sense that the model performs the best if the inference is also Kr ¨ ahenb ¨ uhl’s since the parameters are biased toward that particular inference method.

Original: However, our method in this case outperforms Baqu ´ e’s.
Revised: However, our method in this case outperforms Baqu ´ e’s.

Original: After these methods halt and return q Th , we run one more iteration of
Revised: After these methods halt and return q Th , we run one more iteration of

Original: [Equation]
Revised: [Equation]

Original: and record the relative update residual ∥ q T +

Revised: and record the relative update residual ∥ q T +

Original: h − q Th ∥ / ∥ q Th ∥ for randomly selected 4000 MNIST images.
Revised: h − q Th ∥ / ∥ q Th ∥ for randomly selected 4000 MNIST images.

Original: The results are listed in table 1 and table 2.
Revised: The results are listed in table 1 and table 2.

Original: To alleviate the effect of numerical issues, we strength the convergence condition to either the relative residual is less than 10 − 3 or the number of iterations exceeds 100 steps.
Revised: To alleviate the effect of numerical issues, we strength the convergence condition to either the relative residual is less than 10 − 3 or the number of iterations exceeds 100 steps.

Original: It appears in table 1 and table 2 that although our method has a much lower residual compare to
Revised: It appears in table 1 and table 2 that although our method has a much lower residual compare to

Original: Baqu´e’s, both of them seem small and convergent.
Revised: Baqu´e’s, both of them seem small and convergent.

Original: This is because the “optimal” fixed point in this setting on MNIST might be unique and both methods happen to converge to the same point.
Revised: This is because the “optimal” fixed point in this setting on MNIST might be unique and both methods happen to converge to the same point.

Original: However, this is in general not true.
Revised: However, this is in general not true.

Original: We compare our method vs Baqu´e’s on 400 randomly selected MNIST test images with 40% pixels observed, and perform mean-field update until the relative residual of [0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 , 0 .
Revised: We compare our method vs Baqu´e’s on 400 randomly selected MNIST test images with 40% pixels observed, and perform mean-field update until the relative residual of [0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 , 0 .

Original: 001 , 0 . 0005 , 0 . 0001] is reached (without step constraint), respectively.
Revised: 001 , 0 . 0005 , 0 . 0001] is reached (without step constraint), respectively.

Original: Then we measure the TV distance between the distributions computed by these two methods on the remaining 60% pixels, as well as the convergence speed.
Revised: Then we measure the TV distance between the distributions computed by these two methods on the remaining 60% pixels, as well as the convergence speed.

Original: The results are demonstrated in fig.
Revised: The results are demonstrated in fig.

Original: One can see that when the model is trained (using Kr¨ahenb¨uhl’s, fig. 9a), the TV distance converges to 0 as the tolerance decreases.
Revised: One can see that when the model is trained (using Kr¨ahenb¨uhl’s, fig. 9a), the TV distance converges to 0 as the tolerance decreases.

Original: However, when the model is just initialized (but still constrained to be monotone), the TV distance remains large (fig. 6c).
Revised: However, when the model is just initialized (but still constrained to be monotone), the TV distance remains large (fig. 6c).

Original: Even though in this case the optimal fixed point may be unique, our method is still superior to Baqu´e’s: it takes us less iterations till convergence, despite whether the model is trained or not..
Revised: Even though in this case the optimal fixed point may be unique, our method is still superior to Baqu´e’s: it takes us less iterations till convergence, despite whether the model is trained or not..

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: K r¨ a h e nb ¨ uh l
Revised: K r¨ a h e nb ¨ uh l

Original: B a qu ´ e
Revised: B a qu ´ e

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: CIFAR-10 model architecture Architecture is the same as for the MNIST experiments with the following exceptions: A 11 is a 20 × 20 × 3 × 3 convolution, A 22 is a 24 × 24 × 3 × 3 convolution,
Revised: CIFAR-10 model architecture Architecture is the same as for the MNIST experiments with the following exceptions: A 11 is a 20 × 20 × 3 × 3 convolution, A 22 is a 24 × 24 × 3 × 3 convolution,

Original: A 21 is a 24 × 20 × 3 × 3 convolution with stride 2 , A 33 is a 48 × 48 × 3 × 3 convolution, A 31 is a 48 × 20 × 3 × 3 convolution with stride 4 , A 32 is a 48 × 24 × 3 × 3 convolution with stride 2 , A 43 is a (48 · 8 · 8) × 10 dense linear layer, and A 44 is a 10 × 10 dense linear layer.
Revised: A 21 is a 24 × 20 × 3 × 3 convolution with stride 2 , A 33 is a 48 × 48 × 3 × 3 convolution, A 31 is a 48 × 20 × 3 × 3 convolution with stride 4 , A 32 is a 48 × 24 × 3 × 3 convolution with stride 2 , A 43 is a (48 · 8 · 8) × 10 dense linear layer, and A 44 is a 10 × 10 dense linear layer.

Original: The corresponding variable q as in eq.
Revised: The corresponding variable q as in eq.

Original: (12) then has 4 elements of shape (60 × 32 × 32) , (24 × 16 × 16) , (48 × 8 × 8) , (10 × 1) .
Revised: (12) then has 4 elements of shape (60 × 32 × 32) , (24 × 16 × 16) , (48 × 8 × 8) , (10 × 1) .

Original: When applying the proximal operator to q , we use 1 , 6 , 12 , 1 as their number of groups, respectively.
Revised: When applying the proximal operator to q , we use 1 , 6 , 12 , 1 as their number of groups, respectively.

Original: CIFAR-10 training details Training details are the same as for the MNIST experiments with the following exceptions: The model is trained for 100 epochs using standard data augmentation.
Revised: CIFAR-10 training details Training details are the same as for the MNIST experiments with the following exceptions: The model is trained for 100 epochs using standard data augmentation.

Original: During the first 10 epochs, the weight on the reconstruction loss is ramped up from 0.0 to 0.5 and the weight on the classification loss ramped down from 1.0 to 0.5.
Revised: During the first 10 epochs, the weight on the reconstruction loss is ramped up from 0.0 to 0.5 and the weight on the classification loss ramped down from 1.0 to 0.5.

Original: Also during the first 20 epochs, the percentage of observation pixels is ramped down from 100 % to 50 % .
Revised: Also during the first 20 epochs, the percentage of observation pixels is ramped down from 100 % to 50 % .

Original: C C OMPARISON TO RBM/DBM
Revised: C C OMPARISON TO RBM/DBM

Original: We conduct the same set of experiments on MNIST using DBM for comparison.
Revised: We conduct the same set of experiments on MNIST using DBM for comparison.

Original: We use a 3layer DBM where the first hidden layer has 300 neurons, and the last hidden layer (representing the digits) has 10 neurons, amounting to in total 238,200 parameters.
Revised: We use a 3layer DBM where the first hidden layer has 300 neurons, and the last hidden layer (representing the digits) has 10 neurons, amounting to in total 238,200 parameters.

Original: Our proposed model in the
Revised: Our proposed model in the

Original: MNIST experiments uses 165,300 parameters.
Revised: MNIST experiments uses 165,300 parameters.

Original: For image imputation, we randomly mask off 60% pixels, or a randomly selected 14 × 14 patch; the results are shown in fig.
Revised: For image imputation, we randomly mask off 60% pixels, or a randomly selected 14 × 14 patch; the results are shown in fig.

Original: In the experiment with 60% of pixels randomly masked, we also test
Revised: In the experiment with 60% of pixels randomly masked, we also test

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: the model on predicting the actual digit simultaneously.
Revised: the model on predicting the actual digit simultaneously.

Original: The test accuracy is 93 .
Revised: The test accuracy is 93 .

Original: Our model achieves comparable test accuracy ( 92 .
Revised: Our model achieves comparable test accuracy ( 92 .

Original: 95% ) and imputation compared to this DBM, given fewer parameters and despite the the monotonicity constraint of our model.
Revised: 95% ) and imputation compared to this DBM, given fewer parameters and despite the the monotonicity constraint of our model.

Original: The DBM is trained using CD -1 algorithm for 100 epochs with a batch size of 128 and learning rate of 0 . 01 .
Revised: The DBM is trained using CD -1 algorithm for 100 epochs with a batch size of 128 and learning rate of 0 . 01 .

Original: For imputation and classification, the DBM uses Gibbs sampling of 100 steps, although the quality of the imputed image and test accuracy are insensitive to the number of steps.
Revised: For imputation and classification, the DBM uses Gibbs sampling of 100 steps, although the quality of the imputed image and test accuracy are insensitive to the number of steps.