Original: HyperSPNs: Compact and Expressive Probabilistic Circuits
Revised: HyperSPNs: Compact and Expressive Probabilistic Circuits

Original: Abstract
Revised: Abstract

Original: Probabilistic circuits (PCs) are a family of generative models which allows for the computation of exact likelihoods and marginals of its probability distributions.
Revised: Probabilistic circuits (PCs) are a family of generative models which allows for the computation of exact likelihoods and marginals of its probability distributions.

Original: PCs are both expressive and tractable, and serve as popular choices for discrete density estimation tasks.
Revised: PCs are both expressive and tractable, and serve as popular choices for discrete density estimation tasks.

Original: However, large PCs are susceptible to overﬁtting, and only a few regularization strategies (e.g., dropout, weight-decay) have been explored.
Revised: However, large PCs are susceptible to overﬁtting, and only a few regularization strategies (e.g., dropout, weight-decay) have been explored.

Original: We propose HyperSPNs: a new paradigm of generating the mixture weights of large PCs using a small-scale neural network.
Revised: We propose HyperSPNs: a new paradigm of generating the mixture weights of large PCs using a small-scale neural network.

Original: Our framework can be viewed as a soft weight-sharing strategy, which combines the greater expressiveness of large models with the better generalization and memoryfootprint properties of small models.
Revised: Our framework can be viewed as a soft weight-sharing strategy, which combines the greater expressiveness of large models with the better generalization and memoryfootprint properties of small models.

Original: We show the merits of our regularization strategy on two stateof-the-art PC families introduced in recent literature – RAT-SPNs and EiNETs – and demonstrate generalization improvements in both models on a suite of density estimation benchmarks in both discrete and continuous domains.
Revised: We show the merits of our regularization strategy on two stateof-the-art PC families introduced in recent literature – RAT-SPNs and EiNETs – and demonstrate generalization improvements in both models on a suite of density estimation benchmarks in both discrete and continuous domains.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: One of the core motivations for building models of probability distributions from data is to reason about the data distribution.
Revised: One of the core motivations for building models of probability distributions from data is to reason about the data distribution.

Original: For example, given a model of the distribution over driving routes, we may like to reason about the probability that a driver will take a certain route given that there is congestion on a certain street.
Revised: For example, given a model of the distribution over driving routes, we may like to reason about the probability that a driver will take a certain route given that there is congestion on a certain street.

Original: Or, given a distribution over weather conditions, we may like to reason about the probability of it raining k times in the next year.
Revised: Or, given a distribution over weather conditions, we may like to reason about the probability of it raining k times in the next year.

Original: Traditional probabilistic models such as Categorical/Gaussian Mixture Models or Hidden Markov Models (HMMs) are well-equipped to answer these types of queries with exact probabilities that are consistent with the model distribution [1].
Revised: Traditional probabilistic models such as Categorical/Gaussian Mixture Models or Hidden Markov Models (HMMs) are well-equipped to answer these types of queries with exact probabilities that are consistent with the model distribution [1].

Original: But, their learned model distribution may not accurately approximate the data distribution due to their simplicity.
Revised: But, their learned model distribution may not accurately approximate the data distribution due to their simplicity.

Original: On the other end of the spectrum, the modern wave of deep generative models has largely focused on learning accurate approximations of the true data distribution, at the cost of tractability.
Revised: On the other end of the spectrum, the modern wave of deep generative models has largely focused on learning accurate approximations of the true data distribution, at the cost of tractability.

Original: High capacity models such as autoregressive or ﬂow models can mimic the true data distribution with great ﬁdelity [30, 12, 20].
Revised: High capacity models such as autoregressive or ﬂow models can mimic the true data distribution with great ﬁdelity [30, 12, 20].

Original: However, they are not designed for reasoning about the probability distribution beyond computation of likelihoods.
Revised: However, they are not designed for reasoning about the probability distribution beyond computation of likelihoods.

Original: More illustrative of this trend are
Revised: More illustrative of this trend are

Original: GANs and EBMs, which are expressive enough to sample high-quality images, but give up even the ability to compute exact likelihoods [8, 13].
Revised: GANs and EBMs, which are expressive enough to sample high-quality images, but give up even the ability to compute exact likelihoods [8, 13].

Original: To maintain the ability to reason about the distribution induced by our model, we need to explore within tractable probabilistic models.
Revised: To maintain the ability to reason about the distribution induced by our model, we need to explore within tractable probabilistic models.

Original: Of these model families, probabilistic circuits (e.g. Sum Product Networks) are one of the most expressive and general, subsuming shallow mixture models, and HMMs while maintaining their tractable properties [5, 24, 31].
Revised: Of these model families, probabilistic circuits (e.g. Sum Product Networks) are one of the most expressive and general, subsuming shallow mixture models, and HMMs while maintaining their tractable properties [5, 24, 31].

Original: Their expressive power comes from their hierarchical / deep architecture, allowing them to express a large number of modes in their distribution.
Revised: Their expressive power comes from their hierarchical / deep architecture, allowing them to express a large number of modes in their distribution.

Original: Their tractability comes from global constraints imposed in their network structure, enabling efﬁcient and exact computation of likelihoods, marginals, and more.
Revised: Their tractability comes from global constraints imposed in their network structure, enabling efﬁcient and exact computation of likelihoods, marginals, and more.

Original: Probabilistic circuits (PC) are a popular choice for density estimation [32, 16] and approximate inference in discrete settings [27].
Revised: Probabilistic circuits (PC) are a popular choice for density estimation [32, 16] and approximate inference in discrete settings [27].

Original: Given the status of probabilistic circuits as one of the most expressive models among tractable probabilistic model families, recent works have looked into pushing the limits of expressivity of probabilistic circuits [14, 23, 22].
Revised: Given the status of probabilistic circuits as one of the most expressive models among tractable probabilistic model families, recent works have looked into pushing the limits of expressivity of probabilistic circuits [14, 23, 22].

Original: Naturally, the bigger and deeper the probabilistic circuit, the greater the expressiveness of the model.
Revised: Naturally, the bigger and deeper the probabilistic circuit, the greater the expressiveness of the model.

Original: This has led to a trend of building large probabilistic circuits via design choices such as tensorizing model parameters, forgoing structure learning, and more.
Revised: This has led to a trend of building large probabilistic circuits via design choices such as tensorizing model parameters, forgoing structure learning, and more.

Original: For example, [22] report the training of probabilistic circuits with 9 .
Revised: For example, [22] report the training of probabilistic circuits with 9 .

Original: 4 M parameters.
Revised: 4 M parameters.

Original: However, larger probabilistic circuits are more susceptible to overﬁtting.
Revised: However, larger probabilistic circuits are more susceptible to overﬁtting.

Original: In addition, the so-called double-descent phenomenon [17] (wherein highly overparameterized models exhibit low generalization gap) has yet to be observed in probabilistic circuits.
Revised: In addition, the so-called double-descent phenomenon [17] (wherein highly overparameterized models exhibit low generalization gap) has yet to be observed in probabilistic circuits.

Original: This puts even more importance on effective regularization strategies for training large probabilistic circuits.
Revised: This puts even more importance on effective regularization strategies for training large probabilistic circuits.

Original: Recent works have used dropout [23] and weight decay [33], but there has not been much exploration of other regularization strategies.
Revised: Recent works have used dropout [23] and weight decay [33], but there has not been much exploration of other regularization strategies.

Original: In this work, we propose HyperSPNs, which regularize by aggressively limiting the degrees of freedom of the model.
Revised: In this work, we propose HyperSPNs, which regularize by aggressively limiting the degrees of freedom of the model.

Original: Drawing inspiration from HyperNetworks [10], we generate the weights of the probabilistic circuit via a small-scale external neural network.
Revised: Drawing inspiration from HyperNetworks [10], we generate the weights of the probabilistic circuit via a small-scale external neural network.

Original: More precisely, we partition the parameters (mixture weights) of the PC into several sectors.
Revised: More precisely, we partition the parameters (mixture weights) of the PC into several sectors.

Original: For each sector, we learn a low-dimensional embedding, then map that to the parameters of the sector using the neural network.
Revised: For each sector, we learn a low-dimensional embedding, then map that to the parameters of the sector using the neural network.

Original: The generated parameters still structurally form a PC, so we retain the same ability to reason about the probability distribution induced by the HyperSPN.
Revised: The generated parameters still structurally form a PC, so we retain the same ability to reason about the probability distribution induced by the HyperSPN.

Original: HyperSPNs combine the greater expressiveness of large probabilistic circuits with the better generalization of models with fewer degrees of freedom.
Revised: HyperSPNs combine the greater expressiveness of large probabilistic circuits with the better generalization of models with fewer degrees of freedom.

Original: The external neural network has much fewer parameters than the original PC, effectively regularizing the PC through a soft weight-sharing strategy.
Revised: The external neural network has much fewer parameters than the original PC, effectively regularizing the PC through a soft weight-sharing strategy.

Original: As a bonus, the memory requirement for storing the model is much smaller, and the memory requirement for evaluating the model can also be drastically reduced by materializing sectors of the PC “on the ﬂy”.
Revised: As a bonus, the memory requirement for storing the model is much smaller, and the memory requirement for evaluating the model can also be drastically reduced by materializing sectors of the PC “on the ﬂy”.

Original: We verify the empirical performance of HyperSPNs on density estimation tasks for the Twenty Datasets benchmark [11], the Amazon
Revised: We verify the empirical performance of HyperSPNs on density estimation tasks for the Twenty Datasets benchmark [11], the Amazon

Original: Baby Registries benchmark [7], and the Street View House
Revised: Baby Registries benchmark [7], and the Street View House

Original: Numbers (SVHN) [18] dataset, showing generalization improvements over competing regularization strategies in all three settings.
Revised: Numbers (SVHN) [18] dataset, showing generalization improvements over competing regularization strategies in all three settings.

Original: Our results suggest that HyperSPNs are a promising framework for training probabilistic circuits.
Revised: Our results suggest that HyperSPNs are a promising framework for training probabilistic circuits.

Original:  BACKGROUND
Revised:  BACKGROUND

Original: Probabilistic circuits (PCs) are a family of generative models known for their tractability properties [5, 24, 31].
Revised: Probabilistic circuits (PCs) are a family of generative models known for their tractability properties [5, 24, 31].

Original: They support efﬁcient and exact inference routines for computing likelihoods and marginals/conditionals of their probability distribution.
Revised: They support efﬁcient and exact inference routines for computing likelihoods and marginals/conditionals of their probability distribution.

Original: The internals of a PC consist of a network (edge-weighted Directed Acyclic Graph) of sum and product nodes, stacked in a deep/hierarchical manner and evaluated from leaves to root.
Revised: The internals of a PC consist of a network (edge-weighted Directed Acyclic Graph) of sum and product nodes, stacked in a deep/hierarchical manner and evaluated from leaves to root.

Original: PCs gain their tractability by enforcing various structural constraints on their network, such as decomposability, which restricts children of the same product node to have disjoint scopes [4, 5].
Revised: PCs gain their tractability by enforcing various structural constraints on their network, such as decomposability, which restricts children of the same product node to have disjoint scopes [4, 5].

Original: Different combination of constraints leading to different types of PCs, including Arithmetic Circuits [5], and most notably Sum Product Networks (SPNs) [24].
Revised: Different combination of constraints leading to different types of PCs, including Arithmetic Circuits [5], and most notably Sum Product Networks (SPNs) [24].

Original: The probability of an input x on the PC is deﬁned to be the value g root ( x ) obtained at the root of the PC after evaluating the network topologically bottom-up.
Revised: The probability of an input x on the PC is deﬁned to be the value g root ( x ) obtained at the root of the PC after evaluating the network topologically bottom-up.

Original: Internally, edges leading into a sum node are weighted with mixture probabilities α (that sum to 1 ), and edges leading into a product node are unweighted.
Revised: Internally, edges leading into a sum node are weighted with mixture probabilities α (that sum to 1 ), and edges leading into a product node are unweighted.

Original: Leaf nodes at the base of the network contain leaf distributions l .
Revised: Leaf nodes at the base of the network contain leaf distributions l .

Original: The nodes of the PC are evaluated as follows, with ch ( i ) denoting the children of node i :
Revised: The nodes of the PC are evaluated as follows, with ch ( i ) denoting the children of node i :

Original: [Equation]
Revised: [Equation]

Original: As long as the leaf distributions are tractable and the network structure satisﬁes the constraints of decomposability and smoothness, the output at g root is guaranteed to be normalized, and corresponding marginals/conditionals can be evaluated with linear-time exact inference algorithms [4, 5].
Revised: As long as the leaf distributions are tractable and the network structure satisﬁes the constraints of decomposability and smoothness, the output at g root is guaranteed to be normalized, and corresponding marginals/conditionals can be evaluated with linear-time exact inference algorithms [4, 5].

Original: The network structure of a PC refers to the set of nodes and edge connections in the DAG, excluding the edge weights.
Revised: The network structure of a PC refers to the set of nodes and edge connections in the DAG, excluding the edge weights.

Original: There are various ways of constructing the network structure of PCs, e.g., through the use of domain knowledge or through search-based methods [5, 15, 6, 26].
Revised: There are various ways of constructing the network structure of PCs, e.g., through the use of domain knowledge or through search-based methods [5, 15, 6, 26].

Original: Recently, there has been a trend of abstracting away individual sum and product nodes, and instead dealing directly with layers that satisfy the structural constraints necessary for tractability [23, 22, 27].
Revised: Recently, there has been a trend of abstracting away individual sum and product nodes, and instead dealing directly with layers that satisfy the structural constraints necessary for tractability [23, 22, 27].

Original: Prescribing a network structure through a stacking of these layers leads to memory and hardware efﬁciencies that allow us to scale up the training of PCs to millions of edges.
Revised: Prescribing a network structure through a stacking of these layers leads to memory and hardware efﬁciencies that allow us to scale up the training of PCs to millions of edges.

Original: Given a PC with a ﬁxed network structure, we can learn its edge weights to best ﬁt the probability distribution of data.
Revised: Given a PC with a ﬁxed network structure, we can learn its edge weights to best ﬁt the probability distribution of data.

Original: We will refer to the full set of mixture probabilities leading into sum nodes in the PC as its parameters.
Revised: We will refer to the full set of mixture probabilities leading into sum nodes in the PC as its parameters.

Original: These parameters can be learned using methods such as
Revised: These parameters can be learned using methods such as

Original: Expectation-Maximization (EM) [21], signomial program formulations [34], or simply via (projected) gradient descent.
Revised: Expectation-Maximization (EM) [21], signomial program formulations [34], or simply via (projected) gradient descent.

Original: EM has been a well-studied choice for optimizing
Revised: EM has been a well-studied choice for optimizing

Original: PCs [21, 22], although it requires hand-derived updates that can be tedious and sometimes even incorrect [24].
Revised: PCs [21, 22], although it requires hand-derived updates that can be tedious and sometimes even incorrect [24].

Original: In this work, we rely on gradient descent, which is straightforward and works out-of-the-box on new families of models [33].
Revised: In this work, we rely on gradient descent, which is straightforward and works out-of-the-box on new families of models [33].

Original: Learning a PC with a large number of parameters can be prone to overﬁtting.
Revised: Learning a PC with a large number of parameters can be prone to overﬁtting.

Original: One regularization strategy that has been proposed is “probabilistic dropout” [23], which takes inspiration from the standard dropout technique in neural networks [28] and randomly ignores a subset of edges leading into sum nodes.
Revised: One regularization strategy that has been proposed is “probabilistic dropout” [23], which takes inspiration from the standard dropout technique in neural networks [28] and randomly ignores a subset of edges leading into sum nodes.

Original: Weight decay has also been commonly used as another form of regularization [33].
Revised: Weight decay has also been commonly used as another form of regularization [33].

Original: Unfortunately, there has been little exploration done around regularization strategies, and some current approaches only work in limited settings (e.g., dropout has been limited to settings that use PCs as a discriminative classiﬁer [23]).
Revised: Unfortunately, there has been little exploration done around regularization strategies, and some current approaches only work in limited settings (e.g., dropout has been limited to settings that use PCs as a discriminative classiﬁer [23]).

Original: [Figure]
Revised: [Figure]

Original: × × ×
Revised: × × ×

Original: [Figure]
Revised: [Figure]

Original: 2.1 RAT-SPNS
Revised: 2.1 RAT-SPNS

Original: Two particular PC structures have been recently proposed, namely RAT-SPNs [23] and EiNETs [22], both of which scale to millions of parameters.
Revised: Two particular PC structures have been recently proposed, namely RAT-SPNs [23] and EiNETs [22], both of which scale to millions of parameters.

Original: They are conceptually similar, the main difference being that EiNETs are designed for better GPU performance.
Revised: They are conceptually similar, the main difference being that EiNETs are designed for better GPU performance.

Original: To ground our discussion around regularizing large PCs, we will describe our methods just on
Revised: To ground our discussion around regularizing large PCs, we will describe our methods just on

Original: RAT-SPNs, although we also apply our methods on EiNETs in later experiments too.
Revised: RAT-SPNs, although we also apply our methods on EiNETs in later experiments too.

Original: In RAT-SPNs, the input variables are organized into a binary tree, with one leaf in the binary tree corresponding to each variable.
Revised: In RAT-SPNs, the input variables are organized into a binary tree, with one leaf in the binary tree corresponding to each variable.

Original: We start from the base of the SPN and construct its full structure by stacking sum layers and product layers.
Revised: We start from the base of the SPN and construct its full structure by stacking sum layers and product layers.

Original: For each leaf node in the binary tree, we create one fully connected sum layer on top of the
Revised: For each leaf node in the binary tree, we create one fully connected sum layer on top of the

Original: SPN leaves.
Revised: SPN leaves.

Original: For each internal node in the binary tree, we create one product layer that merges its two children layers in the SPN, and then stack one fully connected sum layer on top.
Revised: For each internal node in the binary tree, we create one product layer that merges its two children layers in the SPN, and then stack one fully connected sum layer on top.

Original: We illustrate this construction in Figure 1.
Revised: We illustrate this construction in Figure 1.

Original: Finally,
Revised: Finally,

Original: RAT-SPNs create so-called replicas of this structure, by repeating the construction but with permuted orderings of input variables in the binary tree leaves.
Revised: RAT-SPNs create so-called replicas of this structure, by repeating the construction but with permuted orderings of input variables in the binary tree leaves.

Original: The replicas are then combined under one top-level mixture distribution (for readability, Figure 1 does not show any replicas).
Revised: The replicas are then combined under one top-level mixture distribution (for readability, Figure 1 does not show any replicas).

Original: The size of each sum layer in the RAT-SPN is determined by a single constant k .
Revised: The size of each sum layer in the RAT-SPN is determined by a single constant k .

Original: For convenience, we also let the size of each product layer be k .
Revised: For convenience, we also let the size of each product layer be k .

Original: This fully speciﬁes the structure of our SPN.
Revised: This fully speciﬁes the structure of our SPN.

Original: For the rest of the paper, it is helpful to abstractly group the parameters of the RAT-SPN into disjoint sectors.
Revised: For the rest of the paper, it is helpful to abstractly group the parameters of the RAT-SPN into disjoint sectors.

Original: Corresponding to the root of the binary tree, we have a sector with 1 × k parameters.
Revised: Corresponding to the root of the binary tree, we have a sector with 1 × k parameters.

Original: Corresponding to each internal node in the binary tree, we have a sector with k × k parameters.
Revised: Corresponding to each internal node in the binary tree, we have a sector with k × k parameters.

Original: Finally, corresponding to each leaf in the binary tree, we have a sector with k × l parameters, where l is the number of SPN leaf distributions per variable (we use l = 2 unless otherwise stated).
Revised: Finally, corresponding to each leaf in the binary tree, we have a sector with k × l parameters, where l is the number of SPN leaf distributions per variable (we use l = 2 unless otherwise stated).

Original: As shown in Figure 1, for n variables (where n is a power of 2 ) and r replicas, this corresponds to 2 n − 1 sectors totaling O ( rnk 2 ) parameters.
Revised: As shown in Figure 1, for n variables (where n is a power of 2 ) and r replicas, this corresponds to 2 n − 1 sectors totaling O ( rnk 2 ) parameters.

Original: In particular, for each replica there are ( n − 2) k
Revised: In particular, for each replica there are ( n − 2) k

Original: 2 + (2 n + 1) k parameters.
Revised: 2 + (2 n + 1) k parameters.

Original: For example, in a problem domain with 1000 variable, using layer size k = 10 and replica r = 10 leads to a RAT-SPN of around 1 M parameters.
Revised: For example, in a problem domain with 1000 variable, using layer size k = 10 and replica r = 10 leads to a RAT-SPN of around 1 M parameters.

Original:  HYPERSPNS
Revised:  HYPERSPNS

Original: We propose HyperSPNs, an approach of regularizing probabilistic circuits with a large number of parameters by limiting the degrees of freedom of the model using an external neural network.
Revised: We propose HyperSPNs, an approach of regularizing probabilistic circuits with a large number of parameters by limiting the degrees of freedom of the model using an external neural network.

Original: We describe our method concretely on RATSPNs, which we will refer to as simply SPNs.
Revised: We describe our method concretely on RATSPNs, which we will refer to as simply SPNs.

Original: To regularize the SPN, we introduce a smaller-scale external neural network to generate the parameters of the SPN (see
Revised: To regularize the SPN, we introduce a smaller-scale external neural network to generate the parameters of the SPN (see

Original: Figure 2).
Revised: Figure 2).

Original: We associate each of the sectors of the SPN with a sector embedding.
Revised: We associate each of the sectors of the SPN with a sector embedding.

Original: For each sector, we map the embedding to the parameters of the sector via the neural network.
Revised: For each sector, we map the embedding to the parameters of the sector via the neural network.

Original: Then, we use these mapped parameters directly as the parameters of the SPN, re-normalize such that parameters leading into a sum node  add up to 1 , and evaluate the SPN as usual.
Revised: Then, we use these mapped parameters directly as the parameters of the SPN, re-normalize the parameters leading into a sum node to add up to 1 , and evaluate the SPN as usual.

Original: We visualize the computation graph of this process in Figure 3.
Revised: The computation graph is shown in Figure 3.

Original: We materialize the SPN in the forward pass, and propagate the gradients through the SPN in the backward pass.
Revised: We materialize the SPN in the forward pass, and propagate the gradients through the SPN in the backward pass.

Original: We optimize the neural network and embeddings jointly using gradient descent.
Revised: We optimize the neural network and embeddings jointly using gradient descent.

Original: HyperSPNs can aggressively reduce the degrees of freedom of the original SPN.
Revised: HyperSPNs can aggressively reduce the degrees of freedom of the original SPN.

Original: Using embeddings of size h , the number of learnable parameters is reduced to O (( rn + k 2 ) h ) .
Revised: Using embeddings of size h , the number of learnable parameters is reduced to O (( rn + k 2 ) h ) .

Original: For the same above example with n = 1000 , k = 10 , r = 10 , choosing embeddings of dimension h = 5 reduces the number of learnable parameters 20 fold, from around 1 M down to around 50 k parameters.
Revised: For the same above example with n = 1000 , k = 10 , r = 10 , choosing embeddings of dimension h = 5 reduces the number of learnable parameters 20 fold, from around 1 M down to around 50 k parameters.

Original: The lower degrees of freedom mitigates overﬁtting of the model.
Revised: The lower degrees of freedom mitigates overﬁtting of the model.

Original: At the same time, the materialized SPN has a large number of mixture weights, and thus can express distribution families with a greater number of clusters compared to a small SPN with the same degrees of freedom.
Revised: At the same time, the materialized SPN has a large number of mixture weights, and thus can express distribution families with a greater number of clusters compared to a small SPN with the same degrees of freedom.

Original: We visualize this with a motivating example in Figure 4, where we represent a dataset using a grid that is split recursively for each variable (we only draw the splits for the ﬁrst 2 variables; the rest is implied).
Revised: We visualize this with a motivating example in Figure 4, where we represent a dataset using a grid that is split recursively for each variable (we only draw the splits for the ﬁrst 2 variables; the rest is implied).

Original: Intuitively, the number of parameters of the
Revised: Intuitively, the number of parameters of the

Original: SPN is tied closely to the number of clusters in the distribution.
Revised: SPN is tied closely to the number of clusters in the distribution.

Original: As a result, training a large SPN can produce many clusters that overﬁt to the training points in blue (Figure 4a), and training a small SPN can lead to coarse factorizations and inﬂexible clusters (Figure 4b).
Revised: As a result, training a large SPN can produce many clusters that overﬁt to the training points in blue (Figure 4a), and training a small SPN can lead to coarse factorizations and inﬂexible clusters (Figure 4b).

Original: Both cases can lead to poor generalization on the testing points in orange.
Revised: Both cases can lead to poor generalization on the testing points in orange.

Original: On the other hand, the HyperSPN uses a soft weight-sharing strategy by storing its parameters as low-dimensional embeddings and decoding them through a small-scale shared neural network.
Revised: On the other hand, the HyperSPN uses a soft weight-sharing strategy by storing its parameters as low-dimensional embeddings and decoding them through a small-scale shared neural network.

Original: This allows the model to learn a high number of clusters without suffering from too high of a degree of freedom.
Revised: This allows the model to learn a high number of clusters without suffering from too high of a degree of freedom.

Original: In Figure 4c, we visualize the soft weight-sharing as ﬁxing the shape of the cluster-triples within each quadrant, while still giving the model control over the center position of each cluster-triple.
Revised: In Figure 4c, we visualize the soft weight-sharing as ﬁxing the shape of the cluster-triples within each quadrant, while still giving the model control over the center position of each cluster-triple.

Original: In this way, HyperSPNs can improve generalization compared to a large SPN with the same number of mixture weights and a small SPN with the same degrees of freedom.
Revised: In this way, HyperSPNs can improve generalization compared to a large SPN with the same number of mixture weights and a small SPN with the same degrees of freedom.

Revised: We verify this phenomenon on a hand-crafted dataset in the Appendix, and also observe similar improvements on real world datasets in our experiments.

Original: 3.1 MEMORY-EFFICIENT EVALUATION
Revised: 3.1 MEMORY-EFFICIENT EVALUATION

Original: Another merit of HyperSPNs is the ability for low-memory storage of the model, since we only need to store the embeddings and the neural network, which together have much fewer parameters than the SPN.
Revised: Another merit of HyperSPNs is the ability for low-memory storage of the model, since we only need to store the embeddings and the neural network, which together have much fewer parameters than the SPN.

Original: Importantly, our abstraction of the SPN structure into sectors allows us to forgo storage of the nodes and edges in the SPN network.
Revised: Importantly, our abstraction of the SPN structure into sectors allows us to forgo storage of the nodes and edges in the SPN network.

Original: Moreover, not only can we be memory-efﬁcient during storage , but we can also be memory-efﬁcient during evaluation.
Revised: Moreover, not only can we be memory-efﬁcient during storage , but we can also be memory-efﬁcient during evaluation.

Original: To do so, we materialize sectors of the SPN on the ﬂy, and erase them from memory when they become unnecessary.
Revised: To do so, we materialize sectors of the SPN on the ﬂy, and erase them from memory when they become unnecessary.

Original: The sectors of the SPN are structured as a binary tree, soto evaluate each sector we need all the input values leading
Revised: The sectors of the SPN are structured as a binary tree, so

Original: [Figure]
Revised: [Figure]

Original:  into the sector.
Revised: to evaluate each sector we need all the input values leading into the sector.

Original: These input values correspond to the output values (each a vector of size k ) of its children sectors.
Revised: These input values correspond to the output values (each a vector of size k ) of its children sectors.

Original: Evaluating the binary tree bottom-up depth-wise is possible, but is also costly since the leaf layer has n sectors leading to
Revised: Evaluating the binary tree bottom-up depth-wise is possible, but is also costly since the leaf layer has n sectors leading to

Original: O ( nk ) cost for storing the output values of children sectors.
Revised: O ( nk ) cost for storing the output values of children sectors.

Original: Rather, we can proceed greedily, evaluating leaf sectors leftto-right and propagating values up a sector as soon as all of its children sectors have been evaluated.
Revised: Rather, we can proceed greedily, evaluating leaf sectors leftto-right and propagating values up a sector as soon as all of its children sectors have been evaluated.

Original: A simple recursive argument shows that this procedure stores only O ( k log n ) intermediate values at a time, and is provably optimal.
Revised: A simple recursive argument shows that this procedure stores only O ( k log n ) intermediate values at a time, and is provably optimal.

Original: Proposition 1.
Revised: Proposition 1.

Original: Evaluating a complete binary-tree computation graph requires Θ(log n ) memory, where n is the number of leaves in the complete binary-tree, and the memory cost of storing each intermediate value and performing each computation is assumed to be a constant.
Revised: Evaluating a complete binary-tree computation graph requires Θ(log n ) memory, where n is the number of leaves in the complete binary-tree, and the memory cost of storing each intermediate value and performing each computation is assumed to be a constant.

Original: Proof.
Revised: Proof.

Original: Perform the sequence of computations based on a post-order traversal of the binary tree, erasing children values from memory when their parent value has been computed.
Revised: Perform the sequence of computations based on a post-order traversal of the binary tree, erasing children values from memory when their parent value has been computed.

Original: This process requires storing at most log n + 1 intermediate values at any time.
Revised: This process requires storing at most log n + 1 intermediate values at any time.

Original: To show optimality, let C ( i ) denote the optimal memory cost needed to compute the value for a node i .
Revised: To show optimality, let C ( i ) denote the optimal memory cost needed to compute the value for a node i .

Original: For non-leaf nodes i , Let j 1 be the ﬁrst child of i that we begin to process, and j 2 be the second.
Revised: For non-leaf nodes i , Let j 1 be the ﬁrst child of i that we begin to process, and j 2 be the second.

Original: Since we processed j 1 ﬁrst (which requires at least 1 unit of memory throughout), we have C ( i ) ≥ 1 + C
Revised: Since we processed j 1 ﬁrst (which requires at least 1 unit of memory throughout), we have C ( i ) ≥ 1 + C

Original: ( j 2 ) .
Revised: ( j 2 ) .

Original: As the base case, for leaf nodes i , we have C ( i ) = 1 .
Revised: As the base case, for leaf nodes i , we have C ( i ) = 1 .

Original: Thus, the optimal memory cost for the whole computation is at least
Revised: Thus, the optimal memory cost for the whole computation is at least

Original: C ( root ) = log n + 1 .
Revised: C ( root ) = log n + 1 .

Original: Corollary 1.
Revised: Corollary 1.

Original: Evaluating a HyperSPN with an underlying
Revised: Evaluating a HyperSPN with an underlying

Original: RAT-SPN structure takes O ( k log n + h ( k 2 + rn )) memory.
Revised: RAT-SPN structure takes O ( k log n + h ( k 2 + rn )) memory.

Original: The cost in Corollary 1 can be broken down into two parts, where O ( k log n ) memory is needed for storing intermediate values in the binary-tree computation graph, and
Revised: The cost in Corollary 1 can be broken down into two parts, where O ( k log n ) memory is needed for storing intermediate values in the binary-tree computation graph, and

Original: O ( h ( rn + k 2 )) memory is needed for storing the HyperSPN parameters and materializing the SPN one sector at a time.
Revised: O ( h ( rn + k 2 )) memory is needed for storing the HyperSPN parameters and materializing the SPN one sector at a time.

Original: The takeaway is that both storing and evaluating a HyperSPN require less memory than storing and evaluating the underlying SPN, which require O ( rnk 2 ) memory.
Revised: The takeaway is that both storing and evaluating a HyperSPN require less memory than storing and evaluating the underlying SPN, which require O ( rnk 2 ) memory.

Original: This memory efﬁciency arises from our key insight of using sector abstractions to enable 1) encoding the SPN parameters as embeddings, 2) forgoing storage of the nodes and edges of the SPN structure, and 3) mapping the SPN to a binary-tree computation graph that allows for low-memory evaluation.
Revised: This memory efﬁciency arises from our key insight of using sector abstractions to enable 1) encoding the SPN parameters as embeddings, 2) forgoing storage of the nodes and edges of the SPN structure, and 3) mapping the SPN to a binary-tree computation graph that allows for low-memory evaluation.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: We experiment with HyperSPN on three sets of density estimation tasks: the Twenty Datasets benchmark, the Amazon Baby Registries benchmark, and the Street View House
Revised: We experiment with HyperSPN on three sets of density estimation tasks: the Twenty Datasets benchmark, the Amazon Baby Registries benchmark, and the Street View House

Original: Numbers (SVHN) dataset.
Revised: Numbers (SVHN) dataset.

Original: All runs were done using a single
Revised: All runs were done using a single

Original: GPU.
Revised: GPU.

Original: We primarily compare with weight decay as the competing regularization method.
Revised: We primarily compare with weight decay as the competing regularization method.

Original: We also include comparisons with smaller SPNs that have the same degrees of freedom as our HyperSPN, and with results reported from other works in literature.
Revised: We also include comparisons with smaller SPNs that have the same degrees of freedom as our HyperSPN, and with results reported from other works in literature.

Original: We omit results on dropout because in our experimentation (and as suggested by [23]), dropout on
Revised: We omit results on dropout because in our experimentation (and as suggested by [23]), dropout on

Original: SPNs worked poorly for these generative density estimation tasks.
Revised: SPNs worked poorly for these generative density estimation tasks.

Original: In all three sets of tasks, we ﬁnd that HyperSPNs give better generalization performance than weight decay, when comparing using the same underlying SPN structure and optimizer.
Revised: In all three sets of tasks, we ﬁnd that HyperSPNs give better generalization performance than weight decay, when comparing using the same underlying SPN structure and optimizer.

Original: Furthermore, HyperSPNs outperforms results reported in literature on EiNETs trained using the same optimization method (Adam), and is competitive against results reported on stochastic EM (sEM).
Revised: Furthermore, HyperSPNs outperforms results reported in literature on EiNETs trained using the same optimization method (Adam), and is competitive against results reported on stochastic EM (sEM).

Original: 4.1 TWENTY DATASETS
Revised: 4.1 TWENTY DATASETS

Original: The Twenty Datasets benchmark [11] is a standard benchmark used in comparing PC models.
Revised: The Twenty Datasets benchmark [11] is a standard benchmark used in comparing PC models.

Original: Our experiment aims to
Revised: Our experiment aims to

Original: [Table]
Revised: [Table]

Original: compare the use of HyperSPN against the baseline of regularization via weight decay, using the same underlying SPN structure for both.
Revised: compare the use of HyperSPN against the baseline of regularization via weight decay, using the same underlying SPN structure for both.

Original: We use the RAT-SPN structure described in Section 2.1, choosing layer size parameter k = 5 and replicas r = 50 , randomizing the variable orders for each replica.
Revised: We use the RAT-SPN structure described in Section 2.1, choosing layer size parameter k = 5 and replicas r = 50 , randomizing the variable orders for each replica.

Original: The number of learnable parameters in this SPN structure is shown in the # Params column under Weight
Revised: The number of learnable parameters in this SPN structure is shown in the # Params column under Weight

Original: Decay in Table 1.
Revised: Decay in Table 1.

Original: The HyperSPN uses the exact same underlying SPN structure, along with an external neural network (a 2 layer MLP of width 20 ) and embeddings of dimension h = 5 .
Revised: The HyperSPN uses the exact same underlying SPN structure, along with an external neural network (a 2 layer MLP of width 20 ) and embeddings of dimension h = 5 .

Original: The number of learnable parameters in the HyperSPN is around 5 times smaller, as shown in the # Params column under HyperSPN.
Revised: The number of learnable parameters in the HyperSPN is around 5 times smaller, as shown in the # Params column under HyperSPN.

Original: In Table 1, we show in bold the better of the results between
Revised: In Table 1, we show in bold the better of the results between

Original: Weight Decay and HyperSPN.
Revised: Weight Decay and HyperSPN.

Original: We see that HyperSPN outperforms Weight Decay on all but two of the datasets.
Revised: We see that HyperSPN outperforms Weight Decay on all but two of the datasets.

Original: On top of that, storing and evaluating the HyperSPN is much more memory-efﬁcient, as shown by the reduced number of parameters and as described by the techniques in Sec 3.1.
Revised: On top of that, storing and evaluating the HyperSPN is much more memory-efﬁcient, as shown by the reduced number of parameters and as described by the techniques in Sec 3.1.

Original: On the 2 right-most columns, we directly copy over results reported in literature.
Revised: On the 2 right-most columns, we directly copy over results reported in literature.

Original: We compare with training EiNETs on Adam (the same optimizer as our setup) [22], and training RAT-SPNs on sEM (a similar network structure as our setup) [23].
Revised: We compare with training EiNETs on Adam (the same optimizer as our setup) [22], and training RAT-SPNs on sEM (a similar network structure as our setup) [23].

Original: We see that HyperSPN also compares favorably to these reported results (the best values underlined).
Revised: We see that HyperSPN also compares favorably to these reported results (the best values underlined).

Original: To illustrate the regularization effects of HyperSPN, we plot training curves on the training and validation data for the Plants and for Pumsb-star datasets.
Revised: To illustrate the regularization effects of HyperSPN, we plot training curves on the training and validation data for the Plants and for Pumsb-star datasets.

Original: We use the same
Revised: We use the same

Original: HyperSPN model described above.
Revised: HyperSPN model described above.

Original: For clarity, let M denote the size of its underlying SPN and m denote its true number of learnable parameters (those in its external neural network and embeddings).
Revised: For clarity, let M denote the size of its underlying SPN and m denote its true number of learnable parameters (those in its external neural network and embeddings).

Original: We then construct two SPNs – one with size M (SPN-Large) and one with size m (SPN-Small) – and overlay their training curves in Figure 5.
Revised: We then construct two SPNs – one with size M (SPN-Large) and one with size m (SPN-Small) – and overlay their training curves in Figure 5.

Original: For both plots in Figure 5a and 5b, SPN-Large suffers from overﬁtting on the training data (presumably due to the large degrees of freedom), and SPN-Small ﬁts the data poorly (presumably limited by the size of the SPN).
Revised: For both plots in Figure 5a and 5b, SPN-Large suffers from overﬁtting on the training data (presumably due to the large degrees of freedom), and SPN-Small ﬁts the data poorly (presumably limited by the size of the SPN).

Original: Instead, HyperSPN strikes the balance between the expressiveness of a large underlying
Revised: Instead, HyperSPN strikes the balance between the expressiveness of a large underlying

Original: SPN, and regularization properties from the compact neural network with low degrees of freedom.
Revised: SPN, and regularization properties from the compact neural network with low degrees of freedom.

Original: This translates to better generalization on the validation data shown in the plots of Figure 5, and on the testing data shown in Table 1.
Revised: This translates to better generalization on the validation data shown in the plots of Figure 5, and on the testing data shown in Table 1.

Original: 4.2 AMAZON BABY REGISTRIES
Revised: 4.2 AMAZON BABY REGISTRIES

Original: We conduct similar experiments on the Amazon Baby Registries dataset [7].
Revised: We conduct similar experiments on the Amazon Baby Registries dataset [7].

Original: We repeat the same experimental setup
Revised: We repeat the same experimental setup

Original: [Figure]
Revised: [Figure]

Original: of maximizing the log-likelihood of data, and use the same model structure for HyperSPN and Weight Decay as their respective structures described in the previous section.
Revised: of maximizing the log-likelihood of data, and use the same model structure for HyperSPN and Weight Decay as their respective structures described in the previous section.

Original: In Table 2, we see that HyperSPN clearly outperforms
Revised: In Table 2, we see that HyperSPN clearly outperforms

Original: Weight Decay as a regularization technique, giving better test log-likelihoods (in bold) while using around 5 times fewer trainable parameters.
Revised: Weight Decay as a regularization technique, giving better test log-likelihoods (in bold) while using around 5 times fewer trainable parameters.

Original: On the right-most column we include results as reported in literature [33] on training
Revised: On the right-most column we include results as reported in literature [33] on training

Original: EiNETs using sEM.
Revised: EiNETs using sEM.

Original: Although the comparison with this right-most column is hard to interpret due to the differences in model architecture, optimization procedure, and regularization, we still note that HyperSPNs compare favorably on the majority of the datasets (best values underlined).
Revised: Although the comparison with this right-most column is hard to interpret due to the differences in model architecture, optimization procedure, and regularization, we still note that HyperSPNs compare favorably on the majority of the datasets (best values underlined).

Original: The experimental results on the Amazon Baby Registries benchmark conﬁrm our ﬁndings in the previous subsection.
Revised: The experimental results on the Amazon Baby Registries benchmark conﬁrm our ﬁndings in the previous subsection.

Original: On both tasks, HyperSPNs generalizes much better than
Revised: On both tasks, HyperSPNs generalizes much better than

Original: Weight Decay when compared on the same underlying SPN structure, while enabling more memory-efﬁcient storage and evaluation of the model.
Revised: Weight Decay when compared on the same underlying SPN structure, while enabling more memory-efﬁcient storage and evaluation of the model.

Original: 4.3 STREET VIEW HOUSE NUMBERS (SVHN)
Revised: 4.3 STREET VIEW HOUSE NUMBERS (SVHN)

Original: Lastly, we experiment on the SVHN dataset [18].
Revised: Lastly, we experiment on the SVHN dataset [18].

Original: We use the infrastructure provided by [22], which groups the train/valid/test data into 100 clusters, then learns a separate
Revised: We use the infrastructure provided by [22], which groups the train/valid/test data into 100 clusters, then learns a separate

Original: PC for each cluster.
Revised: PC for each cluster.

Original: The ﬁnal density estimator is a combination of all the individual PCs under one mixture distribution.
Revised: The ﬁnal density estimator is a combination of all the individual PCs under one mixture distribution.

Original: Here, we study the performance of each individual PC on each cluster separately, comparing between weight decay and HyperSPN when trained using Adam.
Revised: Here, we study the performance of each individual PC on each cluster separately, comparing between weight decay and HyperSPN when trained using Adam.

Original: The underlying PC structure is an EiNET with leaves that hold continuous distributions [22].
Revised: The underlying PC structure is an EiNET with leaves that hold continuous distributions [22].

Original: To introduce soft weightsharing of the HyperSPN on the EiNET, we partition the parameters into sectors of size 40 , which is the number of sum nodes used in each layer in the setup of [22].
Revised: To introduce soft weightsharing of the HyperSPN on the EiNET, we partition the parameters into sectors of size 40 , which is the number of sum nodes used in each layer in the setup of [22].

Original: We use an embedding size of h = 8 , which reduces the number of learnable edge-parameters in the EiNET by about 5 fold.
Revised: We use an embedding size of h = 8 , which reduces the number of learnable edge-parameters in the EiNET by about 5 fold.

Original: In Figure 6, we plot the difference between HyperSPN and weight decay in log-likelihood on the test set, sorted in increasing order for readability.
Revised: In Figure 6, we plot the difference between HyperSPN and weight decay in log-likelihood on the test set, sorted in increasing order for readability.

Original: The higher the value, the more improvement we observe when using HyperSPN as the regularization method.
Revised: The higher the value, the more improvement we observe when using HyperSPN as the regularization method.

Original: In around 70% of the clusters, we see an improvement in generalization, and on average the HyperSPN achieves an improvement in log-likelihood of 13 ± 5 .
Revised: In around 70% of the clusters, we see an improvement in generalization, and on average the HyperSPN achieves an improvement in log-likelihood of 13 ± 5 .

Original: Our experiments suggest that HyperSPNs give a superior regularization strategy for PCs compared to the primary alternative of weight decay.
Revised: Our experiments suggest that HyperSPNs give a superior regularization strategy for PCs compared to the primary alternative of weight decay.

Original: HyperSPNs also lead to better performance than other reported approaches in literature that use the same optimizer (Adam), as well as other reported methods that use sEM.
Revised: HyperSPNs also lead to better performance than other reported approaches in literature that use the same optimizer (Adam), as well as other reported methods that use sEM.

Original: Our experimental results are general, showing that HyperSPNs are applicable across three standard benchmarks, across PCs with discrete and continuous leaf distributions, and across different state-of-the-art structures such as RAT-SPNs and EiNETs.
Revised: Our experimental results are general, showing that HyperSPNs are applicable across three standard benchmarks, across PCs with discrete and continuous leaf distributions, and across different state-of-the-art structures such as RAT-SPNs and EiNETs.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: Many existing works focus on structure learning for probabilistic circuits, covering a wide range of PC types [6, 26, 25, 14].
Revised: Many existing works focus on structure learning for probabilistic circuits, covering a wide range of PC types [6, 26, 25, 14].

Original: These approaches often regularize by limiting the complexity of the structure [6, 32].
Revised: These approaches often regularize by limiting the complexity of the structure [6, 32].

Original: However, structure learning in general can be tedious and difﬁcult to tune [23].
Revised: However, structure learning in general can be tedious and difﬁcult to tune [23].

Original: Recent works have embraced a more deep-learning-like framework by prescribing the structure of the PC and learning only the model parameters [23, 22].
Revised: Recent works have embraced a more deep-learning-like framework by prescribing the structure of the PC and learning only the model parameters [23, 22].

Original: This framework leads to the training of large PCs with millions of parameters.
Revised: This framework leads to the training of large PCs with millions of parameters.

Original: However, little exploration has been done around regularization in this new parameter-centric framework.
Revised: However, little exploration has been done around regularization in this new parameter-centric framework.

Original: For generative settings, dropout was suggested as unsuitable [23], leaving only weight decay as the primary parameter regularization method.
Revised: For generative settings, dropout was suggested as unsuitable [23], leaving only weight decay as the primary parameter regularization method.

Original: In our work, we propose a new regularization strategy via HyperSPNs, showing large improvements 
Revised: In our work, we propose a new regularization strategy via HyperSPNs, showing large improvements in

Original: in generalization.

Original: [Table]
Revised: [Table]

Original: Finally, the idea of generating parameters with an external neural network comes from HyperNetworks [10], which were introduced as an effective architecture for recurrent models.
Revised: Finally, the idea of generating parameters with an external neural network comes from HyperNetworks [10], which were introduced as an effective architecture for recurrent models.

Original: Soft weight-sharing strategies in general [19] have also shown good empirical results [2] and proven useful for neural network compression [29].
Revised: Soft weight-sharing strategies in general [19] have also shown good empirical results [2] and proven useful for neural network compression [29].

Original:  CONCLUSION
Revised:  CONCLUSION

Original: We propose HyperSPNs, a new paradigm of training large probabilistic circuits using a small-scale external neural network to generate the parameters of the probabilistic circuit.
Revised: We propose HyperSPNs, a new paradigm of training large probabilistic circuits using a small-scale external neural network to generate the parameters of the probabilistic circuit.

Original: HyperSPNs provide regularization by aggressively reducing the degrees of freedom through a soft weight-sharing strategy.
Revised: HyperSPNs provide regularization by aggressively reducing the degrees of freedom through a soft weight-sharing strategy.

Original: We show empirical improvements in the generalization performance of PCs compared to competing regularization methods, across both discrete and continuous density estimation tasks, and across different state-of-the-art PC structures.
Revised: We show empirical improvements in the generalization performance of PCs compared to competing regularization methods, across both discrete and continuous density estimation tasks, and across different state-of-the-art PC structures.

Original: As a bonus, HyperSPNs enable memory-efﬁcient storage and evaluation, combining the expressiveness of large PCs with the compactness of small models.
Revised: As a bonus, HyperSPNs enable memory-efﬁcient storage and evaluation, combining the expressiveness of large PCs with the compactness of small models.

Original: Limitations and Future Work Our analysis of the memory efﬁciency of HyperSPNs is limited their storage and evaluation, since training HyperSPNs involves manipulating of the gradients through the underlying SPN.
Revised: Limitations and Future Work Our analysis of the memory efﬁciency of HyperSPNs is limited their storage and evaluation, since training HyperSPNs involves manipulating of the gradients through the underlying SPN.

Original: Future work can look into combining HyperSPNs with techniques from memory-efﬁcient back propagation [3, 9] to train very large SPNs that do not ﬁt in memory.
Revised: Future work can look into combining HyperSPNs with techniques from memory-efﬁcient back propagation [3, 9] to train very large SPNs that do not ﬁt in memory.

Original: A EXPERIMENTAL DETAILS
Revised: A HAND-DESIGNED EXAMPLE

Revised: To examine the merits of HyperSPNs as discussed in Section 3, we construct a hand-designed dataset to test the three types of models described in Figure 4: SPN-Large,

Revised: SPN-Small, and HyperSPN.

Revised: The hand-designed dataset is procedurally generated with 256 binary variables andinstances, broken into train/valid/test splits at 70 / 10 / 20% .

Revised: The generation procedure is designed such that the correlation between variable i and j is dependent on the path length between leaves i and j of a complete binary tree over the 256 variables.

Revised: The exact details can be found in our code.

Revised: SPN-Large has the same number of SPN edges as the HyperSPN, while SPN-Small has roughly the same number of trainable parameters as HyperSPN.

Revised: Both SPN-Large and

Revised: SPN-Small are regularized via weight-decay.

Revised: As we can see in Table 3, HyperSPN gives the best generalization performance on the test split of our hand-designed dataset, when compared to standard SPNs with either similar number of

Revised: SPN edges (SPN-Large) and similar number of trainable parameters (SPN-Small).

Revised: [Table]

Revised: B EXPERIMENTAL DETAILS

Original: Here, we provide more details on our experimental setup.
Revised: Here, we provide more details on our experimental setup.

Original: In
Revised: In

Original: Table 3, we give the hyperparameters used for training our models on the Twenty Datasets and Amazon Baby Registries benchmarks.
Revised: Table 4, we give the hyperparameters used for training our models on the Twenty Datasets and Amazon Baby Registries benchmarks.

Original: For both methods we do early stopping by training until the validation performance plateaus/declines (we train some up to 80 k steps).
Revised: For both methods we do early stopping by training until the validation performance plateaus/declines (we train some up to 80 k steps).

Original: Then we take the version of the model that performed best on the validation set, and use it for evaluation on the test set.
Revised: Then we take the version of the model that performed best on the validation set, and use it for evaluation on the test set.

Original: Recall that for the normal SPN, we take gradient descent steps on the mixture weights of the SPN.
Revised: Recall that for the normal SPN, we take gradient descent steps on the mixture weights of the SPN.

Original: For the HyperSPN, we take gradient descent steps on the parameters of the external neural network.
Revised: For the HyperSPN, we take gradient descent steps on the parameters of the external neural network.

Original: Empirically, we found that higher learning rates are more suitable for the SPN (e.g. 2e-2), and lower learning rates are more suitable for the HyperSPN (e.g. 5e-3).
Revised: Empirically, we found that higher learning rates are more suitable for the SPN (e.g. 2e-2), and lower learning rates are more suitable for the HyperSPN (e.g. 5e-3).

Original: [Table] For the SVHN experiment, we build on the code provided at https://github.com/cambridge-mlg/ 
Revised: For the SVHN experiment, we build on the code provided at https://github.com/cambridge-mlg/ [Table]

Original: EinsumNetworks .
Revised: EinsumNetworks .

Original: We train each method for 80 epochs using Adam, and also use early stopping based on the validation set.
Revised: We train each method for 80 epochs using Adam, and also use early stopping based on the validation set.

Original: The weight decay hyperparameter used for the weights of the models are the same as that shown in Table 3, and we scale down the learning rate for both models to 2eand 5e-4, respectively.
Revised: The weight decay hyperparameter used for the weights of the models are the same as that shown in Table 4, and we scale down the learning rate for both models to 2eand 5e-4, respectively.

Original: We found the slower learning rate to be more suitable for both models on this benchmark, with
Revised: We found the slower learning rate to be more suitable for both models on this benchmark, with

Original: HyperSPNs still giving the better performance.
Revised: HyperSPNs still giving the better performance.

Original: For the Twenty Datasets and Amazon Baby Registries, the leaf nodes are binary indicator random variables, hence there are no trainable parameters.
Revised: For the Twenty Datasets and Amazon Baby Registries, the leaf nodes are binary indicator random variables, hence there are no trainable parameters.

Original: For SVHN, the leaf nodes are factorized Gaussians.
Revised: For SVHN, the leaf nodes are factorized Gaussians.

Original: The training of the leaf distributions was kept the same for Weight Decay and HyperSPNs (i.e. for HyperSPNs, we do not generate the parameters of the leaf distributions using the external neural network).
Revised: The training of the leaf distributions was kept the same for Weight Decay and HyperSPNs (i.e. for HyperSPNs, we do not generate the parameters of the leaf distributions using the external neural network).