Original: Sparse Probabilistic Circuits via Pruning and Growing
Revised: Sparse Probabilistic Circuits via Pruning and Growing

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Probabilistic circuits (PCs) are a tractable representation of probability distributionsallowing for exact and efﬁcient computation of likelihoods and marginals.
Revised: Probabilistic circuits (PCs) are a tractable representation of probability distributionsallowing for exact and efficient computation of likelihoods and marginals.

Original: Therehas been signiﬁcant recent progress on improving the scale and expressiveness of
Revised: Therehas been significant recent progress on improving the scale and expressiveness of

Original: PCs.
Revised: PCs.

Original: However, PC training performance plateaus as model size increases.
Revised: However, PC training performance plateaus as model size increases.

Original: We discover that most capacity in existing large PC structures is wasted: fully-connectedparameter layers are only sparsely used.
Revised: We discover that most capacity in existing large PC structures is wasted: fully-connectedparameter layers are only sparsely used.

Original: We propose two operations: pruningand growing , that exploit the sparsity of PC structures.
Revised: We propose two operations: pruningand growing , that exploit the sparsity of PC structures.

Original: Speciﬁcally, the pruningoperation removes unimportant sub-networks of the PC for model compression andcomes with theoretical guarantees.
Revised: Specifically, the pruningoperation removes unimportant sub-networks of the PC for model compression andcomes with theoretical guarantees.

Original: The growing operation increases model capacityby increasing the dimensions of latent states.
Revised: The growing operation increases model capacityby increasing the dimensions of latent states.

Original: By alternatingly applying pruningand growing, we increase the capacity that is meaningfully used, allowing us tosigniﬁcantly scale up PC learning.
Revised: By alternatingly applying pruningand growing, we increase the capacity that is meaningfully used, allowing us tosignificantly scale up PC learning.

Original: Empirically, our learner achieves state-of-the-artlikelihoods on MNIST-family image datasets and an Penn Tree Bank language datacompared to other PC learners and less tractable deep generative models such asﬂow-based models and variational autoencoders (VAEs).
Revised: Empirically, our learner achieves state-of-the-artlikelihoods on MNIST-family image datasets and an Penn Tree Bank language datacompared to other PC learners and less tractable deep generative models such asflow-based models and variational autoencoders (VAEs).

Original:  Introduction
Revised:  Introduction

Original:  Probabilistic circuits (PCs) [33, 2] are a unifying framework to abstract from a multitudeof tractable probabilistic models.
Revised: [Figure] Probabilistic circuits (PCs) [37, 2] are a unifying framework to abstract from a multitudeof tractable probabilistic models.

Original: The keyproperty that separates PCs from other deepgenerative models such as ﬂow-based models [24] and VAEs [13] is their tractability .
Revised: The keyproperty that separates PCs from other deepgenerative models such as flow-based models [25] and VAEs [14] is their tractability .

Original: Itenables them to compute various queries, including marginal probabilities, exactly and efﬁciently [34].
Revised: Itenables them to compute various queries, including marginal probabilities, exactly and efficiently [38].

Original: Therefore, PCs are increasinglyused in inference-demanding applications suchas enforcing algorithmic fairness [1, 3], makingpredictions under missing data [12] and datacompression [19].
Revised: Therefore, PCs are increasinglyused in inference-demanding applications suchas enforcing algorithmic fairness [1, 3], makingpredictions under missing data [13], and datacompression [20].

Original: Recent advanced of PC learning [28], regularization [30, 18] and efﬁcient parallelism implementation [25] have been pushing the limits of PC’sexpressivity and scalability such that PCs can even match the performance of less tractable deepgenerative models such as Flows and VAEs.
Revised: Recent advancement of PC learning [30], regularization [33, 19] and efficient parallelism implementation [27] have been pushing the limits of PC’s expressivity and scalability such that PCs can even match the performance of less tractable deepgenerative models such as Flows and VAEs.

Original: This lead to a trend of building larger PCs.
Revised: This lead to a trend of building larger PCs.

Original: However, PCtraining performance plateaus as model size increases.
Revised: However, PCtraining performance plateaus as model size increases.

Original: This indicates that to go even further, simplyscaling up might not sufﬁce, and we need to be better at using the capacity available.
Revised: This indicates that to go even further, simplyscaling up might not suffice, and we need to be better at using the capacity available.

Original: We discover that this might be caused by the fact that the capacity of large PCs are wasted.
Revised: We discover that this might be caused by the fact that the capacity of large PCs are wasted.

Original: Asshown in Figure 1, most parameters in a PC with 2.18M parameters have close-to-zero values, whichhave little effect to the PC distribution.
Revised: Asshown in Figure 1, most parameters in a PC with 2.18M parameters have close-to-zero values, whichhave little effect on the PC distribution.

Original: Since existing PC structures usually have fully-connectedparameter layers [18, 28], this indicates that the parameter values only are sparsely used.
Revised: Since existing PC structures usually have fully-connectedparameter layers [19, 30], this indicates that the parameter values only are sparsely used.

Original: In this work, we propose to better exploit the sparsity of large PC models by two structure learningprimitives — pruning and growing .
Revised: In this work, we propose to better exploit the sparsity of large PC models by two structure learningprimitives — pruning and growing .

Original: Speciﬁcally, the goal of the pruning operation is to identify andremove unimportant sub-networks of a PC.
Revised: Specifically, the goal of the pruning operation is to identify andremove unimportant sub-networks of a PC.

Original: This is done by quantifying the importance of PC edgesw.r.t.
Revised: This is done by quantifying the importance of PC edgesw.r.t.

Original: a dataset using circuit ﬂows , a theoretically-grounded metric that upper bounds the drop oflog-likelihood caused by pruning.
Revised: a dataset using circuit flows , a theoretically-grounded metric that upper bounds the drop oflog-likelihood caused by pruning.

Original: Compared to L1 regularization, the proposed pruning operator ismore informed by the PC semantics, and hence quantiﬁes the global effects of pruning much moreeffectively.
Revised: Compared to L1 regularization, the proposed pruning operator ismore informed by the PC semantics, and hence quantifies the global effects of pruning much moreeffectively.

Original: Empirically, the proposed pruning method achieves a compression rate of 80-98% with atmost 1% drop in likelihood on various PCs.
Revised: Empirically, the proposed pruning method achieves a compression rate of 80-98% with atmost 1% drop in likelihood on various PCs.

Original: The proposed growing operation increases the model size by copying its existing components andinjecting noise.
Revised: The proposed growing operation increases the model size by copying its existing components andinjecting noise.

Original: In particular, when applied to PCs compressed by the pruning operation, growingproduces larger PCs that can be optimized to achieve better performance.
Revised: In particular, when applied to PCs compressed by the pruning operation, growingproduces larger PCs that can be optimized to achieve better performance.

Original: Applying pruning andgrowing iteratively can greatly reﬁne the structure and parameters of a PC.
Revised: Applying pruning andgrowing iteratively can greatly refine the structure and parameters of a PC.

Original: Empirically, the loglikelihoods metric can improve from 2% to 10% after a few iterations.
Revised: Empirically, the loglikelihoods metric can improve from 2% to 10% after a few iterations.

Original: Compared to existing PClearners as well as less tractable deep generative models such as VAEs and ﬂow-based models, ourproposed method achieves state-of-the-art density estimation results on image datasets including
Revised: Compared to existing PClearners as well as less tractable deep generative models such as VAEs and flow-based models, ourproposed method achieves state-of-the-art density estimation results on image datasets including

Original: MNIST, EMNIST, FashionMNIST, and the Penn Tree Bank language modeling task.
Revised: MNIST, EMNIST, FashionMNIST, and the Penn Tree Bank language modeling task.

Original:  Probabilistic Circuits
Revised:  Probabilistic Circuits

Original: Probabilistic circuits (PCs) [33, 2] model probability distributions with a structured computationgraph.
Revised: Probabilistic circuits (PCs) [37, 2] model probability distributions with a structured computationgraph.

Original: They are an umbrella term for a large family of tractable probabilistic models includingarithmetic circuits [7, 8], sum-product networks(SPN) [27], cutset networks [28], and-or searchspaces [22], and probabilistic sentential decision diagrams [15].
Revised: They are an umbrella term for a large family of tractable probabilistic models includingarithmetic circuits [7, 8], sum-product networks (SPNs) [29], cutset networks [30], and-or searchspaces [23], and probabilistic sentential decision diagrams [16].

Original: The syntax and semantics of PCs aredeﬁned as follows.
Revised: The syntax and semantics of PCs aredefined as follows.

Original: Deﬁnition 1 (Probabilistic Circuit) .
Revised: Definition 1 (Probabilistic Circuit) .

Original: A PC C :=( G , ✓ ) represents a joint probability distribution p ( X )over random variables X through a directed acyclic (computation) graph (DAG)
Revised: A PC C :=( G , θ ) represents a joint probability distribution p ( X )over random variables X through a directed acyclic (computation) graph (DAG)

Original: G parameterized by✓ .
Revised: G parameterized byθ .

Original: Similar to neural networks, each node in the DAG deﬁnes a computational unit.
Revised: Similar to neural networks, each node in the DAG defines a computational unit.

Original: Speciﬁcally, the
Revised: Specifically, the

Original: DAG G consists of three types of units — input , sum , and product .
Revised: DAG G consists of three types of units — input , sum , and product .

Original: Every leaf node in G is an inputunit; every inner unit n (i.e., sum or product) receives inputs from its children in ( n ) , and computesoutput , which encodes a probability distribution p n deﬁned recursively as follows:
Revised: Every leaf node in G is an inputunit; every inner unit n (i.e., sum or product) receives inputs from its children in ( n ) , and computesoutput , which encodes a probability distribution p n defined recursively as follows:

Original: [Equation]
Revised: [Equation]

Original: where f n ( x ) is a univariate input distribution (e.g, Gaussian, Categorical), and ✓ c | n denotes theparameter that corresponds to edge ( n, c ) in the DAG.
Revised: where f n ( x ) is a univariate input distribution (e.g, Gaussian, Categorical), and θ c | n denotes theparameter that corresponds to edge ( n, c ) in the DAG.

Original: For every sum unit n , its input parameters sumup to one, i.e., P c 2 in ( n ) ✓ c | n = 1 .
Revised: For every sum unit n , its input parameters sumup to one, i.e., (cid:80) c ∈ in ( n ) θ c | n = 1 .

Original: Intuitively, a product unit deﬁnes a factorized distribution over itsinputs, and a sum unit represents a mixture over its input distributions with weights { ✓ c | n : c 2 in ( n ) } .
Revised: Intuitively, a product unit defines a factorized distribution over itsinputs, and a sum unit represents a mixture over its input distributions with weights { θ c | n : c ∈ in ( n ) } .

Original: Finally, the probability distribution of a PC (i.e., p C ) is deﬁned as the distribution represented by itsroot unit r (i.e., p r ( x ) ), that is, its output neuron.
Revised: Finally, the probability distribution of a PC (i.e., p C ) is defined as the distribution represented by itsroot unit r (i.e., p r ( x ) ), that is, its output neuron.

Original: The size of a PC, denoted |C| = | ✓ | , is the numberof parameters in C .
Revised: The size of a PC, denoted |C| = | θ | , is the numberof parameters in C .

Original: We assume w.l.o.g that a PC alternates between layers of sum and product unitsbefore reaching its inputs.
Revised: We assume w.l.o.g that a PC alternates between layers of sum and product unitsbefore reaching its inputs.

Original: Figure 2 shows an example of PC.
Revised: Figure 2 shows an example of PC.

Original: As hinted by it deﬁnition, computing the (log)likelihood of PC C given a sample x is simply evaluatingits computation units in G in a feedforward manner following Equation 1.
Revised: As hinted by it definition, computing the (log)likelihood of PC C given a sample x is simply evaluatingits computation units in G in a feedforward manner following Equation 1.

Original: The key property that separates PCs from other deep probabilistic models such as ﬂows [10] and
Revised: The key property that separates PCs from other deep probabilistic models such as flows [10] and

Original: VAEs [13] is their tractability , which is the ability to exactly and efﬁciently answer various probabilistic queries.
Revised: VAEs [14] is their tractability , which is the ability to exactly and efficiently answer various probabilistic queries.

Original: This paper focuses on PCs that support linear time (w.r.t.
Revised: This paper focuses on PCs that support linear time (w.r.t.

Original: model size) marginal probability
Revised: model size) marginal probability

Original: Z
Revised: Z

Original: X
Revised: X

Original: Z
Revised: Z

Original: X
Revised: X

Original: X 4 X(a) ...0 .0 ...
Revised: X 4 X(a) ...0 .0 ...

Original: X 1 ⇠ B ( . 1) X 3 ⇠ B ( . 2) . X 1 ⇠ B ( . 7) X 3 ⇠ B ( . 3) .
Revised: X 1 ∼B ( . 1) X 3 ∼B ( . 2) . X 1 ∼B ( . 7) X 3 ∼B ( . 3) .

Original: X 2 ⇠ B ( . 6)
Revised: X 2 ∼B ( . 6)

Original: X 4 ⇠ B ( . 
Revised: X 4 ∼B ( . 8) ..

Original: X 2 ⇠ B ( . 1) .
Revised: X 2 ∼B ( . 1) .

Original: X 4 ⇠ B ( . 2) .(b) computation, as they are increasingly used in downstream applications such as data compression [20]and making predictions under missing data [12], and also achieve on-par expressiveness [20, 18, 17].
Revised: X 4 ∼B ( . 2) .(b) computation, as they are increasingly used in downstream applications such as data compression [21]and making predictions under missing data [13], and also achieve on-par expressiveness [21, 19, 18].

Original: To support efﬁcient marginal inference, PCs need to be smooth and decomposable .
Revised: To support efficient marginal inference, PCs need to be smooth and decomposable .

Original: Deﬁnition 2 (Smoothness and Decomposability [9]) .
Revised: Definition 2 (Smoothness and Decomposability [9]) .

Original: For a PC, the scope  ( n ) of a PC unit n is theinput variables that a unit depends on; then, (1) a product unit is decomposable if its children havedisjoint scope; (2) a sum unit is smooth if its children have identical scope.
Revised: For a PC, the scope ϕ ( n ) of a PC unit n is theinput variables that a unit depends on; then, (1) a product unit is decomposable if its children havedisjoint scope; (2) a sum unit is smooth if its children have identical scope.

Original: A PC is decomposable ifall of its produce units are decomposable; a PC is smooth if all of its sum units are smooth.
Revised: A PC is decomposable ifall of its product units are decomposable; a PC is smooth if all of its sum units are smooth.

Original: Decomposability ensures that every product unit encodes a well-deﬁned factorized distribution overdisjoint sets of variables; smoothness ensures that the mixture components of every sum units arewell-deﬁned over the same set of variables.
Revised: Decomposability ensures that every product unit encodes a well-defined factorized distribution overdisjoint sets of variables; smoothness ensures that the mixture components of every sum units arewell-defined over the same set of variables.

Original: Both structural properties will be the key to guaranteeingthe effectiveness of the structure learning algorithms proposed in the following sections.
Revised: Both structural properties will be the key to guaranteeingthe effectiveness of the structure learning algorithms proposed in the following sections.

Original:  Probabilistic Circuit Model Compression via Pruning
Revised:  Probabilistic Circuit Model Compression via Pruning

Original: Figure 1 shows that most parameters in a large PC are very small.
Revised: Figure 1 shows that most parameters in a large PC are very small.

Original: Given that these parameters areweights associated with mixture (sum unit) components, there are edges and sub-circuits that havelittle impact on the sum unit output.
Revised: Given that these parameters areweights associated with mixture (sum unit) components, there are edges and sub-circuits that havelittle impact on the sum unit output.

Original: Hence, by pruning away these unimportant components, it ispossible to signiﬁcantly reduce model size while maximally retaining model expressiveness.
Revised: Hence, by pruning away these unimportant components, it ispossible to significantly reduce model size while maximally retaining model expressiveness.

Original: Let C \E denote the PC C where the edges in E are removed from the circuit, and each sum unit isrenormalized such that its parameters sum to one.
Revised: Let C \E denote the PC C where the edges in E are removed from the circuit, and each sum unit isrenormalized such that its parameters sum to one.

Original: Given a PC C and a dataset D , our goal is toefﬁciently identify a set of k edges E such that the performance gap between the pruned PC C \E andthe original PC C is minimized:
Revised: Given a PC C and a dataset D , our goal is toefficiently identify a set of k edges E such that the performance gap between the pruned PC C \E andthe original PC C is minimized:

Original: [Equation]
Revised: [Equation]

Original: such that E ✓ { ( n, c ) : ✓ c | n 2 ✓ } and |E| =k,
Revised: such that E ⊆ { ( n, c ) : θ c | n ∈ θ } and |E| =

Original: [Equation] [Equation] where LL ( D , C ) = 1 |D| P x 2 D log p C ( x ) is the averaged log-likelihood of PC C given dataset D .
Revised: k, (2) where LL ( D , C ) = 1 |D| (cid:80) x ∈D log p C ( x ) is the averaged log-likelihood of PC C given dataset D .

Original: The edges E are chosen among all parameterized edges (i.e., all input edges of sum units).
Revised: The edges E are chosen among all parameterized edges (i.e., all input edges of sum units).

Original: Figure 3billustrates the result of pruning ﬁve (red) edges from the PC in Figure 3a.
Revised: Figure 3billustrates the result of pruning five (red) edges from the PC in Figure 3a.

Original: Pruning by parameters.
Revised: Pruning by parameters.

Original: The parameter value statistics in Figure 1 suggest that a natural criteriafor pruning edges is the magnitude of its corresponding parameter ✓ c | n .
Revised: The parameter value statistics in Figure 1 suggest that a natural criteriafor pruning edges is the magnitude of its corresponding parameter θ c | n .

Original: This leads to thee E P ARAMheuristic, which selects k edges with the smallest parameters.
Revised: This leads to the E P ARAMheuristic, which selects k edges with the smallest parameters.

Original: However, edge parameters themselvesare insufﬁcient to quantify the importance of inputs to a sum unit in the entire PC’s distribution.
Revised: However, edge parameters themselvesare insufficient to quantify the importance of inputs to a sum unit in the entire PC’s distribution.

Original: Theparameters of a sum unit are normalized to be 1 so they only contain local information about themixture components.
Revised: Theparameters of a sum unit are normalized to be 1 so they only contain local information about themixture components.

Original: Speciﬁcally, ✓ c | n merely deﬁnes the relative importance of edge ( n, c ) in thejoint distribution represented by its corresponding sum unit n .
Revised: Specifically, θ c | n merely defines the relative importance of edge ( n, c ) in thejoint distribution represented by its corresponding sum unit n .

Original: Figure 4a illustrates what happenswhen the edge with the smallest parameter is pruned from the PC in Figure 2.
Revised: Figure 4a illustrates what happenswhen the edge with the smallest parameter is pruned from the PC in Figure 2.

Original: However, as shown
Revised: However, as shown

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] in Figure 4b [Figure] pruning another edge delivers better likelihoods as it accounts more for the “globalinﬂuence” of edges on the PC’s output.
Revised: in Figure 4b, pruning another edge delivers better likelihoods as it accounts more for the “globalinfluence” of edges on the PC’s output.

Original: This global inﬂuence is highly related to PC semantics andwe will introduce it next along with its corresponding heuristics E F LOW .
Revised: This global influence is highly related to PC semantics andwe will introduce it next along with its corresponding heuristics E F LOW .

Original: Pruning by generative signiﬁcance.
Revised: Pruning by generative significance.

Original: A more informed pruning strategy needs to consider theglobal impact of edges on the distribution represented by the output of the PC.
Revised: A more informed pruning strategy needs to consider theglobal impact of edges on the distribution represented by the output of the PC.

Original: To achieve this,instead of viewing the distribution p C in a feedforward manner following Equation 1, we quantifythe signiﬁcance of a unit or edge by the probability that it will be “activated” when drawing samplesfrom the PC.
Revised: To achieve this,instead of viewing the distribution p C in a feedforward manner following Equation 1, we quantifythe significance of a unit or edge by the probability that it will be “activated” when drawing samplesfrom the PC.

Original: Indeed, if the presence of an edge is hardly ever relevant to the generative samplingprocess, removing it will not signiﬁcantly affect the PC’s distribution.
Revised: Indeed, if the presence of an edge is hardly ever relevant to the generative samplingprocess, removing it will not significantly affect the PC’s distribution.

Original: Algorithm 1 shows that the PC sampling process proceeds in a top-down manner: a queue Q isinitialized with the root unit (line 1).
Revised: Algorithm 1 shows that the PC sampling process proceeds in a top-down manner: a queue Q isinitialized with the root unit (line 1).

Original: The algorithm then processes every unit in the queue until it isempty.
Revised: The algorithm then processes every unit in the queue until it isempty.

Original: For a sum unit n (lines 6-7), the sampler randomly adds one of its input units to the queueaccording to the categorical distribution deﬁned by sum parameters { ✓ c | n : c 2 in ( n ) } ; for a productunit (line 5), all its inputs are added to the queue; for an input unit n deﬁned on variable X (line 4),the algorithm randomly samples value x according to its input distribution.
Revised: For a sum unit n (lines 6-7), the sampler randomly adds one of its input units to the queueaccording to the categorical distribution defined by sum parameters { θ c | n : c ∈ in ( n ) } ; for a productunit (line 5), all its inputs are added to the queue; for an input unit n defined on variable X (line 4),the algorithm randomly samples value x according to its input distribution.

Original: Algorithm 1 is designedto sample instances following the PC distribution, therefore the probability of adding a unit n to thequeue Q is the probability that n will be sampled, which provides a good measure of the importanceof unit n to the PC as a whole.
Revised: Algorithm 1 is designedto sample instances following the PC distribution, therefore the probability of adding a unit n to thequeue Q is the probability that n will be sampled, which provides a good measure of the importanceof unit n to the PC as a whole.

Original: Following this intuition, pruning edges with the least probability ofappearing in Q , which we call the top-down probability of n , deﬁnes a reasonable pruning strategy.
Revised: Following this intuition, pruning edges with the least probability ofappearing in Q , which we call the top-down probability of n , defines a reasonable pruning strategy.

Original: Deﬁnition 3 (Top-down probability) .
Revised: Definition 3 (Top-down probability) .

Original: The top-down probability of each unit in a PC C with parameters✓ is deﬁned as follows.
Revised: The top-down probability of each unit in a PC C with parametersθ is defined as follows.

Original: The top-down probability of the output unit r is one: p TD r, ✓ ( r ) = 1 .
Revised: The top-down probability of the output unit r is one: p TD r, θ ( r ) = 1 .

Original: Thetop-down probability of sum and product units n is deﬁned recursively as follows:
Revised: Thetop-down probability of sum and product units n is defined recursively as follows:

Original: [Equation]
Revised: [Equation]

Original: where pa sum ( n ) and pa prod ( n ) are the sum and product units that take n as input, respectively.
Revised: where pa sum ( n ) and pa prod ( n ) are the sum and product units that take n as input, respectively.

Original: Thetop-down probability of a sum edge ( n, c ) is deﬁned as p TD r, ✓ ( n, c )=
Revised: Thetop-down probability of a sum edge ( n, c ) is defined as p TD r, θ ( n, c )

Original: ✓ c | n · p TD r, ✓ ( n ) .
Revised: = θ c | n · p TD r, θ ( n ) .

Original: Input
Revised: Input

Original: X Output :

Original: Following this inductive deﬁnition, the top-down probability of all PC units and sum edges can becomputed in a single backward pass over the PC’s computation graph.
Revised: Following this inductive definition, the top-down probability of all PC units and sum edges can becomputed in a single backward pass over the PC’s computation graph.

Original: Pruning by circuit ﬂows.
Revised: Pruning by circuit flows.

Original: Despite its ability to capture global information of PC parameters,top-down probability is not tailored to a speciﬁc dataset.
Revised: Despite its ability to capture global information of PC parameters,top-down probability is not tailored to a specific dataset.

Original: The top-down probability p TD C , ✓ ( n ) representsthe probability of reaching n in an unconditional sampling process.
Revised: The top-down probability p TD C , θ ( n ) representsthe probability of reaching n in an unconditional sampling process.

Original: However, the pruning objectiveof Equation 2 requires measuring the probability of reaching certain units/edges in the samplingprocess given that the sampled instance is some x 2 D .
Revised: However, the pruning objectiveof Equation 2 requires measuring the probability of reaching certain units/edges in the samplingprocess given that the sampled instance is some x ∈ D .

Original: To bridge this gap, we deﬁne circuit ﬂow asa sample-dependent version of the top-down probability.
Revised: To bridge this gap, we define circuit flow asa sample-dependent version of the top-down probability.

Original: Deﬁnition 4 (Circuit Flow 1 ) .
Revised: Definition 4 (Circuit Flow 1 ) .

Original: For a given C
Revised: For a given C

Original: =
Revised: =

Original: ( G , ✓ ) and input x , let ✓ x denote a new set ofparameters such that ✓ x c | n is the probability of component c in the mixture represented by sum unit nafter observing sample x .
Revised: ( G , θ ) and input x , let θ x denote a new set ofparameters such that θ x c | n is the probability of component c in the mixture represented by sum unit nafter observing sample x .

Original: The node ﬂow F n ( x ) of a unit n is then deﬁned as the top-down probabilityunder this reparameterization of the circuit:
Revised: The node flow F n ( x ) of a unit n is then defined as the top-down probabilityunder this reparameterization of the circuit:

Original: [Equation]
Revised: [Equation]

Original: where✓ x c | n =
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Similarly, the edge ﬂow F n,c ( x ) w.r.t.
Revised: Similarly, the edge flow F n,c ( x ) w.r.t.

Original: PC C and sample x is deﬁned by p TD r, ✓ x ( n, c ) .
Revised: PC C and sample x is defined by p TD r, θ x ( n, c ) .

Original: We further deﬁne
Revised: We further define

Original: F n,c ( D ) =
Revised: F n,c ( D ) =

Original: P x 2 D F n,c ( x ) as the aggregate ﬂow over D .
Revised: (cid:80) x ∈D F n,c ( x ) as the aggregate flow over D .

Original: By using the sample-conditioned parameters ✓ x , F n ( x ) deﬁnes the probability of reaching unit n in
Revised: By using the sample-conditioned parameters θ x , F n ( x ) defines the probability of reaching unit n in

Original: Algorithm 1, given that the sampled instance is x .
Revised: Algorithm 1, given that the sampled instance is x .

Original: Therefore, edge ﬂow F n,c ( x ) is a natural metricof the importance of edge ( n, c ) given x .
Revised: Therefore, edge flow F n,c ( x ) is a natural metricof the importance of edge ( n, c ) given x .

Original: Intuitively, the circuit ﬂow measures how many expectedsamples “ﬂow” through certain edges.
Revised: Intuitively, the circuit flow measures how many expectedsamples “flow” through certain edges.

Original: We name E F LOW as the heuristic that prunes edges with thesmallest aggregate ﬂow.
Revised: We name E F LOW as the heuristic that prunes edges with thesmallest aggregate flow.

Original: Empirical Analysis.
Revised: Empirical Analysis.

Original: Figure 5a compares the effect of pruning heuristics E P ARAM , E F LOW , aswell as an uninformed strategy, prune randomly, which we denote as E R AND .
Revised: Figure 5a compares the effect of pruning heuristics E P ARAM , E F LOW , aswell as an uninformed strategy, prune randomly, which we denote as E R AND .

Original: It shows that both
Revised: It shows that both

Original: E P ARAM and E F LOW are reasonable pruning strategy, however, as we increase the percentage ofpruned parameters, E F LOW has less log-likelihoods drop compared with E P ARAM .
Revised: E P ARAM and E F LOW are reasonable pruning strategy, however, as we increase the percentage ofpruned parameters, E F LOW has less log-likelihoods drop compared with E P ARAM .

Original: Using E F LOWheuristics we can pruning up to 80% of the parameters without much log-likelihoods drop.
Revised: Using E F LOWheuristics we can pruning up to 80% of the parameters without much log-likelihoods drop.

Original: Asshown in Figure 5b, the parameter distribution is more balanced after pruning compared to Figure 1,indicating a higher signiﬁcance of each edge.
Revised: Asshown in Figure 5b, the parameter distribution is more balanced after pruning compared to Figure 1,indicating a higher significance of each edge.

Original: Section 6 will provide more empirical results.
Revised: Section 6 will provide more empirical results.

Original: Beforethat, we ﬁrst theoretically verify the effectiveness of the E F LOW heuristic in the next section.
Revised: Beforethat, we first theoretically verify the effectiveness of the E F LOW heuristic in the next section.

Original:  Bounding and Approximating the Loss of Likelihood
Revised:  Bounding and Approximating the Loss of Likelihood

Original: This section establish a theoretical upper bound on the loglikelihood drop  LL ( D , C , E ) =
Revised: This section establish a theoretical upper bound on the loglikelihood drop ∆ LL ( D , C , E ) =

Original: LL ( D , C )  LL ( D , C \E ) (cf.
Revised: LL ( D , C ) − LL ( D , C \E ) (cf.

Original: Equation 2) caused by pruning away edges E .
Revised: Equation 2) caused by pruning away edges E .

Original: Interestingly, the E F LOWheuristic proposed in the previous section is a good approximation of the derived upper bound.
Revised: Interestingly, the E F LOWheuristic proposed in the previous section is a good approximation of the derived upper bound.

Original:  We start from the case of pruning one edge (i.e., k = 1 in Equation 2).
Revised: [Figure] (c) We start from the case of pruning one edge (i.e., k = 1 in Equation 2).

Original: In this case, the loss oflikelihood can be quantiﬁed exactly using ﬂows and edge parameters:
Revised: In this case, the loss oflikelihood can be quantified exactly using flows and edge parameters:

Original: [Figure]

Original: Theorem 1 (Log-likelihood drop of pruning one edge) .
Revised: Theorem 1 (Log-likelihood drop of pruning one edge) .

Original: For a PC C and a dataset D , the loss oflog-likelihood by pruning away edge ( n, c ) is
Revised: For a PC C and a dataset D , the loss oflog-likelihood by pruning away edge ( n, c ) is

Original: [Equation]
Revised: [Equation]

Original: See proof in Appendix B.1.
Revised: See proof in Appendix B.1.

Original: By computing the second term from Theorem 1, we can pick the edgewith the smallest log-likelihood drop.
Revised: By computing the second equation from Theorem 1, we can pick the edgewith the smallest log-likelihood drop.

Original: Additionally, the third term characterizes the log-likelihooddrop without re-normalizing parameters of ✓ ·| n .
Revised: Additionally, the third equation characterizes the log-likelihooddrop without re-normalizing parameters of θ ·| n .

Original: It suggests pruning the edge with smallest edge ﬂow.
Revised: It suggests pruning the edge with smallest edge flow.

Original: A key insight from Theorem 1 is that the log-likelihood drop depends explicitly on the edge ﬂow
Revised: A key insight from Theorem 1 is that the log-likelihood drop depends explicitly on the edge flow

Original: F n,c ( x ) and unit ﬂow F n ( x ) .
Revised: F n,c ( x ) and unit flow F n ( x ) .

Original: This matches the intuition from Section 3 that the circuit ﬂow of anedge is sufﬁcient to quantify its importance in the PC.
Revised: This matches the intuition from Section 3 that the circuit flow of anedge is sufficient to quantify its importance in the PC.

Original: Next, we bound the drop of pruning multiple edges.
Revised: Next, we bound the drop of pruning multiple edges.

Original: Theorem 2 (Log-likelihood drop of pruning multiple edges) .
Revised: Theorem 2 (Log-likelihood drop of pruning multiple edges) .

Original: Let C be a PC and D be a dataset.
Revised: Let C be a PC and D be a dataset.

Original: Forany set of edges E in C , if 8 x 2 D , P ( n,c ) 2 E F n,c ( x ) < 1 , the log-likelihood drop by pruning away Eis bounded and approximated by
Revised: Forany set of edges E in C , if ∀ x ∈D , (cid:80) ( n,c ) ∈E F n,c ( x ) < 1 , the log-likelihood drop by pruning away Eis bounded and approximated by

Original: [Equation]
Revised: [Equation]

Original: Proof of this theorem is provided in Appendix B.2.
Revised: Proof of this theorem is provided in Appendix B.2.

Original: We ﬁrst look at the second term of Equation 3.
Revised: We first look at the second equation of Equation 3.

Original: Although it provides an upper bound to the performance drop, it cannot be used as a pruning heuristicsince the bound does not decompose over edges.
Revised: Although it provides an upper bound to the performance drop, it cannot be used as a pruning heuristicsince the bound does not decompose over edges.

Original: And hence ﬁnding the set of edges with thelowest score requires evaluating the bound exponentially (w.r.t.
Revised: And hence finding the set of edges with thelowest score requires evaluating the bound exponentially (w.r.t.

Original: k )
Revised: k )

Original: many times.
Revised: many times.

Original: Therefore, we doan additional approximation step of the bound via Taylor expansion, which leads to the third termof Equation 3.
Revised: Therefore, we do anadditional approximation step of the bound via Taylor expansion, which leads to the third equationof Equation 3.

Original: This approximation matches the E F LOW heuristic by a constant factor 1 / |D| , whichtheoretically justiﬁes the effectiveness of the heuristic.
Revised: This approximation matches the E F LOW heuristic by a constant factor 1 / |D| , whichtheoretically justifies the effectiveness of the heuristic.

Original: As shown in Figure 5c, the approximate bound( E F LOW heuristic) matches closely to the actual log-likelihood drop.
Revised: As shown in Figure 5c, the approximate bound( E F LOW heuristic) matches closely to the actual log-likelihood drop.

Original:  Scalable Structure Learning
Revised:  Scalable Structure Learning

Original: The pruning operator beneﬁts two aspects of PCs.
Revised: The pruning operator benefits two aspects of PCs.

Original: First, as shown in Figure 5b, model parameters aremore balanced after pruning.
Revised: First, as shown in Figure 5b, model parameters aremore balanced after pruning.

Original: Next, pruning removes sub-circuits with negligible contribution to themodel distribution.
Revised: Next, pruning removes sub-circuits with negligible contribution to themodel distribution.

Original: If we treat PC as hierarchical mixtures of components, pruning can be regardedas an implicit structure learning step that removes the “unimportant” components for each mixture.
Revised: If we treat PC as hierarchical mixtures of components, pruning can be regardedas an implicit structure learning step that removes the “unimportant” components for each mixture.

Original: However, since pruning only decreases model capacity, it is difﬁcult to get a more expressive PC thanthe original one.
Revised: However, since pruning only decreases model capacity, it is impossible to get a more expressive PC than the original one.

Original: To mitigate this problem, we propose a growing operation to increase thecapacityof a PC by introducing more components for each mixture.
Revised: To mitigate this problem, we propose a growing operation to increase the

Original: Pruning and growing together deﬁnes anscalable structure learning algorithm for PCs.

Original: Input : a PC C , Gaussian noisy variance 
Revised: Input : a PC C , Gaussian noisy variance σ

Original: Output : a PC C ’ after growing operation 1 old2new  a dictionary mapping input PC units n 2 C to units of the growed PC 2 foreach n 2 C do // visit children before parents if n is an input unit then old2new [ n ]  ( n, deepcopy ( n )) else chs _ 1 , chs _[ [ c ][0]  c  ( n )] , [ old2new [ c ][1] for c in 
Revised: Output : a PC C ’ after growing operation 1 old2new ← a dictionary mapping input PC units n ∈ C to units of the growed PC 2 foreach n ∈ C do // visit children before parents if n is an input unit then old2new [ n ] ← ( n, deepcopy ( n )) else chs _ 1 , chs _ 2 ← [ old2new [ c ][0] for c in in ( n )] , [ old2new [ c ][1] for c in ( n )] if n is a product unit then old2new [ n ] ← ( (cid:78) ( chs _ 1 ) , (cid:78) ( chs _ 2 ))

Original: ( n )] if n is a product unit thenold2new [ n] ( N ( chs _ 1) , 
Revised: else if n is a sum unit then n 1 , n 2 ← (cid:76)

Original: N ( chs _ 2 ))
Revised: _ 2 ]) θ | i ← normalize

Original: else if n is a sum unit then n1 ,  n 
Revised: | n , θ | n ]) ) ϵ i in [1 , 2] old2new [

Original: L ([ chs _ 1 , chs _ 2 ]) , L ([ chs _ 1 , chs _ 2 ]) ✓ | n i normalize ([ ✓ | n , ✓ | n ]) ⇥ ✏ ) ✏ ⇠ N ( 1 ,  2 ) for i in [1 , 2] old2new [ n ] (
Revised: ] ←

Original: 2 ) 11 return old2new [ r ][ // r is theroot unit of C 
Revised: 2 ) 11 return old2new [ r ] is the unit of C capacity of a PC by introducing more components for each mixture.

Revised: Pruning and growing togetherdefines an scalable structure learning algorithm for PCs.

Original: Growing Operation.
Revised: Growing Operation.

Original: Growing deﬁnes an operator that increases model size by coping its existingcomponents and injecting noise.
Revised: Growing defines an operator that increases model size by coping its existingcomponents and injecting noise.

Original: As shown in Figure 3, after applying the growing operation on theoriginal PC in Figure 3b, we can get a new grown PC as in Figure 3c.
Revised: As shown in Figure 3, after applying the growing operation on theoriginal PC in Figure 3b, we can get a new grown PC as in Figure 3c.

Original: Speciﬁcally, the growingoperation is applied to units, edges, and parameters respectively: (1) for units, growing operates onevery PC unit n and create another copy n new ; (2) for edges, the sum edge ( n, c ) from the original
Revised: Specifically, the growingoperation is applied to units, edges, and parameters respectively: (1) for units, growing operates onevery PC unit n and create another copy n new ; (2) for edges, the sum edge ( n, c ) from the original

Original: PC (Figure 3b) are copied three times to the grown PC (Figure 3c): from new parent to new child( n new , c new ) , from old parent to new child ( n, c new ) , and from new parent to old child ( n new , c ) ; theproduct edge is simply connecting new copied sum units; (3) for parameters, new parameter ✓ new c | n area noisy copy from old parameter ✓ c | n , that is ✓ new c | n ✏ · ✓ c | n where ✏ ⇠ N (1 ,  2 ) and  2 controlsthe Gaussian noise variance.
Revised: PC (Figure 3b) are copied three times to the grown PC (Figure 3c): from new parent to new child( n new , c new ) , from old parent to new child ( n, c new ) , and from new parent to old child ( n new , c ) ; theproduct edge is simply connecting new copied sum units; (3) for parameters, new parameter θ new c | n area noisy copy from old parameter θ c | n , that is θ new c | n ← ϵ · θ c | n where ϵ ∼ N (1 , σ 2 ) and σ 2 controlsthe Gaussian noise variance.

Original: Gaussian noise is added to the copied parameters such that after weapply the growing operation, parameter learning algorithms can ﬁnd diverse parameters for differentcopies.
Revised: Gaussian noise is added to the copied parameters such that after weapply the growing operation, parameter learning algorithms can find diverse parameters for differentcopies.

Original: After a growing operation, the PC size is 4 times the original PC size.
Revised: After a growing operation, the PC size is 4 times the original PC size.

Original: Algorithm 2 shows afeed-forward implementation of the growing operation.
Revised: Algorithm 2 shows afeed-forward implementation of the growing operation.

Original: Structure Learning through Pruning and Growing.
Revised: Structure Learning through Pruning and Growing.

Original: The proposed pruning and growing algorithms can be applied iteratively to reﬁne the structure and parameters of an initial PC.
Revised: The proposed pruning and growing algorithms can be applied iteratively to refine the structure and parameters of an initial PC.

Original: Speciﬁcally,since the growing operator increases the number of PC parameters by a factor 4, applying growingafter pruning 75% edges from an initial PC keeps the number of parameters unchanged.
Revised: Specifically,since the growing operator increases the number of PC parameters by a factor 4, applying growingafter pruning 75% edges from an initial PC keeps the number of parameters unchanged.

Original: We makeuse of these two operations and propose a joint structure and parameter learning algorithm for PCs.
Revised: We makeuse of these two operations and propose a joint structure and parameter learning algorithm for PCs.

Original: Speciﬁcally, starting from an initial PC, we apply 75% pruning, growing, and parameter learningiteratively until convergence.
Revised: Specifically, starting from an initial PC, we apply 75% pruning, growing, and parameter learningiteratively until convergence.

Original: We utilize HCLT [18] as initial PC structure as it has the state-of-the-artlikelihood performance.
Revised: We utilize HCLT [19] as initial PC structure as it has the state-of-the-artlikelihood performance.

Original: Note that this structure learning pipeline can be applied to any PC structure.
Revised: Note that this structure learning pipeline can be applied to any PC structure.

Original: Parameter Estimation.
Revised: Parameter Estimation.

Original: We use a stochastic mini-batch version of Expectation-Maximizationoptimization [1].
Revised: We use a stochastic mini-batch version of Expectation-Maximizationoptimization [1].

Original: Speciﬁcally, at each iteration, we draw a mini-batch of samples D B , computeaggregated circuit ﬂows F n,c ( D B ) and F n ( D B ) of these samples (E-step), and then compute newparameter ✓ new c | n = F n,c ( D B ) / F n ( D B ) , and update the targeting parameter with a learning rate↵ : ✓ t + ↵✓ new + (1  ↵ ) ✓ t (M-step).
Revised: Specifically, at each iteration, we draw a mini-batch of samples D B , computeaggregated circuit flows F n,c ( D B ) and F n ( D B ) of these samples (E-step), and then compute newparameter θ new c | n = F n,c ( D B ) / F n ( D B ) , and update the targeting parameter with a learning rateα : θ t +1 ← αθ new + (1 − α ) θ t (M-step).

Original: Empirically this approach converges faster and is moreregularized compared to full-batch EM.
Revised: Empirically this approach converges faster and is more

Original: Parallelism Computation.
Revised: Parallelism Computation.

Original: Existing approaches to scale up learning and inferences of PCs such as
Revised: Existing approaches to scale up learning and inferences of PCs such as

Original: Einsum networks [25] utilize fully connected parametrized layers (Figure 3a) of PC structures suchas HCLT [18] and RatSPN [26].
Revised: Einsum networks [27] utilize fully connected parametrized layers (Figure 3a) of PC structures suchas HCLT [19] and RatSPN [28].

Original: These structures can be easily vectorized to utilize deep learningpackages such as PyTorch.
Revised: These structures can be easily vectorized to utilize deep learningpackages such as PyTorch.

Original: However, the sparse structure learned by pruning and growing is noteasily vectorized as a dense matrix operation.
Revised: However, the sparse structure learned by pruning and growing is noteasily vectorized as a dense matrix operation.

Original: We therefore implement customized GPU kernels toparallelize the computation of parameter learning and inferences based on Juice.jl [6], an open-source
Revised: We therefore implement customized GPU kernels toparallelize the computation of parameter learning and inferences based on Juice.jl [6], an open-source

Original: Julia package for learning PCs.
Revised: Julia package for learning PCs.

Original: The kernels segment PC units into layers such that the units in eachlayer are independent thus the computation can be fully parallelized in the GPU.
Revised: The kernels segment PC units into layers such that the units in eachlayer are independent thus the computation can be fully parallelized in the GPU.

Original: As a result, we cantrain PCs with millions of parameters in less than half an hour.
Revised: As a result, we cantrain PCs with millions of parameters in less than half an hour.

Original:  Experiments
Revised:  Experiments

Original: We now evaluate our proposed method pruning and growing on two different sets of density estimationbenchmarks: (1) the MNIST-family image generation datasets including MNIST [16], EMNIST [4],and FashionMNIST [35]; (2) character level Penn Tree Bank language modeling task [21].
Revised: We now evaluate our proposed method pruning and growing on two different sets of density estimationbenchmarks: (1) the MNIST-family image generation datasets including MNIST [17], EMNIST [4],and FashionMNIST [39]; (2) character level Penn Tree Bank language modeling task [22].

Original: We ﬁrst report the best results we get on image datasets and language modeling tasks via structurelearning proposed in Section 5 (Section 6.1) and then show the effect of pruning and growingoperations via two detail experimental settings as two different constrained optimization problem:ﬁnd the smallest PC given the same performance via model compression and ﬁnd the best PC giventhe same size via structure learning (Section 6.2).
Revised: We first report the best results we get on image datasets and language modeling tasks via structurelearning proposed in Section 5 (Section 6.1) and then show the effect of pruning and growingoperations via two detailed experimental settings as two different constrained optimization problem:find the smallest PC given the same performance via model compression and find the best PC giventhe same size via structure learning (Section 6.2).

Original: Settings.
Revised: Settings.

Original: For all experiments, we use hidden Chow-Liu Tree (HCLT)
Revised: For all experiments, we use hidden Chow-Liu Tree (HCLT)

Original: [18] with the number oflatent states ranging between { 16 , 32 , 64 , 128 } as initial PC structures.
Revised: [19] with the number oflatent states ranging between { 16 , 32 , 64 , 128 } as initial PC structures.

Original: We train the parametersof PCs with stochastic mini-batch EM algorithms (cf.
Revised: We train the parametersof PCs with stochastic mini-batch EM algorithms (cf.

Original: Section 5).
Revised: Section 5).

Original: We perform early stoppingand hyperparameter search using validation set and report results on the test set.
Revised: We perform early stoppingand hyperparameter search using validation set and report results on the test set.

Original: Please refer to
Revised: Please refer to

Original: Appendix C for more details.
Revised: Appendix C for more details.

Original: We use mean test set bits-per-dimension (bpd) as evaluation criteria,where bpd ( D , C ) =
Revised: We use mean test set bits-per-dimension (bpd) as evaluation criteria,where bpd ( D , C ) =

Original:  LL ( D , C ) / (log(2) ·|D| · m ) and m is the number of features in dataset D .
Revised: −LL ( D , C ) / (log(2) · m ) and m is the number of features in dataset D .

Original: 6.1 Density Estimation Benchmarks
Revised: 6.1 Density Estimation Benchmarks

Original: Image Datasets.
Revised: Image Datasets.

Original: The MNIST-family datasets contains gray scale pixel image with size n =28 ⇥ 28and each pixel taking between [0 , 255]( k =256) .
Revised: The MNIST-family datasets contains gray scale pixel image with size n =28 × 28and each pixel taking between [0 , 255]( k =256) .

Original: We split out 5% from training set as validation set.
Revised: We split out 5% from training set as validation set.

Original: We compare with 2 competitive PC learning algorithms: HCLT [18] and RatSPN [26], one ﬂow-basedmodel: IDF [11], and 3 VAE based methods: BitSwap [14], BB-ANS [31], and McBits [29].
Revised: We compare with 2 competitive PC learning algorithms: HCLT [19] and RatSPN [28], one flow-basedmodel: IDF [12], and 3 VAE based methods: BitSwap [15], BB-ANS [34], and McBits [32].

Original: Asa fair comparison, we implement RatSPN structure ourselves and used the same training pipelineand EM optimizer as our method.
Revised: As a faircomparison, we implement RatSPN structure ourselves and use the same training pipeline and EMoptimizer as our method.

Original: Note that EinsumNet [25] also uses RatSPN structures but with PyTorch implementation so its comparison is subsumed by comparison of RatSPN.
Revised: Note that EinsumNet [27] also uses RatSPN structures but with PyTorchimplementation so its comparison is subsumed by comparison of RatSPN.

Original: All 7 methods aretested on MNIST, 4 splits of EMNIST and FashionMNIST.
Revised: All 7 methods are testedon MNIST, 4 splits of EMNIST and FashionMNIST.

Original: As shown in Table 1, the best results arebold.
Revised: As shown in Table 1, the best results are bold.

Original: We see that our proposed method signiﬁcantly outperforms all other baselines on all datasets,and establishes new state-of-the-art results among PCs, ﬂows, and VAE models.
Revised: We see that our proposed method significantly outperforms all other baselines on all datasets, andestablishes new state-of-the-art results among PCs, flows, and VAE models.

Original: More experimentsdetails are in Appendix C.
Revised: More experiments detailsare in Appendix C.

Original: [Table]
Revised: [Table]

Original: Language Modeling Task.
Revised: Language Modeling Task.

Original: We use the Penn Tree Bank dataset with standard processingfrom Mikolov et al.
Revised: We use the Penn Tree Bank dataset with standard processingfrom Mikolov et al.

Original: [23], which contains around 5M characters and a character-level vocabularysize of k = 50 .
Revised: [24], which contains around 5M characters and a character-level vocabularysize of k = 50 .

Original: The data is split into sentences with a maximum sequence length of n = 288 .
Revised: The data is split into sentences with a maximum sequence length of n = 288 .

Original: Wecompare with 3 competitive normalizing ﬂow based models: Bipartite ﬂow [32] and latent ﬂows [36]including AF/SCF and IAF/SCF, since they are the only comparable work with non-autoregressivelanguage modeling.
Revised: Wecompare with 3 competitive normalizing flow based models: Bipartite flow [35] and latent flows [40]including AF/SCF and IAF/SCF, since they are the only comparable work with non-autoregressivelanguage modeling.

Original: As shown in Tab. 2, the proposed method outperforms all 3 baselines
Revised: As shown in Tab. 2, the proposed method outperforms all 3 baselines.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 6.2 Evaluating Pruning and Growing
Revised: 6.2 Evaluating Pruning and Growing

Original: What is the Smallest PC for the Same Likelihood?
Revised: What is the Smallest PC for the Same Likelihood?

Original: We evaluation the ability of pruning operationsbased on circuit ﬂows (Section 3) to do effective model compression by iteratively pruning k % ofthe PC parameters and then ﬁne-tuning them until the ﬁnal training log-likelihood does not decreaseby more than 1% .
Revised: We evaluation the ability of pruning operationsbased on circuit flows (Section 3) to do effective model compression by iteratively pruning k fractionof the PC parameters and then fine-tuning them until the final training log-likelihood does not decreaseby more than 1% .

Original: Speciﬁcally, we take k% ranging between { 0 . 05 , 0 . 1 , 0 . 3 } .
Revised: Specifically, we take k ranging between { 0 . 05 , 0 . 1 , 0 . 3 } .

Original: As shown in Figure 6,we can achieve a compression rate of 80-98% with negligible performance loss on PCs.
Revised: As shown in Figure 6,we can achieve a compression rate of 80-98% with negligible performance loss on PCs.

Original: Besides, byﬁxing the number of latent parameters (x-axis) and comparing bpp across different number of latentstates (legend), we discover that compressing a large PC to a get smaller PC has better likelihoodscompared to directly training a HCLT with the same number of parameters from scratch, due to thesparsity of compressed PC structures, as well as the smarter way to ﬁnd good parameters: ﬁnding abetter PC with larger size and compress it to smaller one. 
Revised: Besides, byfixing the number of latent parameters (x-axis) and comparing bpp across different number of latentstates (legend), we discover that compressing a large PC to a get smaller PC has better likelihoodscompared to directly training a HCLT with the same number of parameters from scratch, due to thesparsity of compressed PC structures, as well as the smarter way to find good parameters: finding abetter PC with larger size and compress it to smaller one. [Figure]

Original: What is the Best PC Given the Same Size?
Revised: What is the Best PC Given the Same Size?

Original: We evaluate structure learning methods combiningpruning and growing proposed in Section 5.
Revised: We evaluate structure learning methods combiningpruning and growing proposed in Section 5.

Original: Starts from initial HCLT, we iteratively prune 75% of theparameters, growing again, and ﬁnetuning until meeting the stopping criteria.
Revised: Starting from initial HCLT, we iteratively prune 75%of the parameters, growing again, and finetuning until meeting the stopping criteria.

Original: As shown in Figure 7,our method consistently improve the likelihoods of initial PCs for different number of latent statesamong all datasets. s
Revised: As shown in Figure 7, our method consistently improve the likelihoods of initial PCs for different number of latentstates among all datasets. [Figure]

Original:  Related Work and Conclusions
Revised:  Related Work and Conclusions

Original: Improving the expressiveness of PCs have been a central topic in the literature.
Revised: Improving the expressiveness of PCs have been a central topic in the literature.

Original: A recent trend isto construct PCs with good initial structures.
Revised: A recent trend isto construct PCs with good initial structures.

Original: For example, EiNets [25] and RAT-SPNs [26] userandomly generated structures,and HCLTs [18] deﬁne PCs dependent on the pairwise correlation onvariables.
Revised: For example, EiNets [27] and RAT-SPNs [28] userandomly generated structures, HCLTs [19] and ID-SPNs [31] define PCs dependent on the pairwisecorrelation on variables, learns direct and indirect variable correlations among variable.

Original: Alternatively, several works propose structure learning algorithms that iteratively modify 
Revised: Alternatively,several works propose structure learning algorithms that iteratively modify PC structures to betterfit the data [5, 18]. There are also works aiming for better regularized parameters of PC such as

Original: PC structures to better ﬁt the data [5, 17]
Revised: HyperSPN [33]. Pruning and growing have been introduced in deep neural networks to exploitsparsity [11]. Similar strategies can be adopted in probabilistic circuits. [26] prunes decision treesaccording to validation accuracy reduction.

Original:  We propose structure learning of PCs by combining pruning and growing operations to exploit thesparsity of PC structures.
Revised: ResSPNs uses on the lottery ticket hypothesis for weightpruning to gain more compact PC [36]. We propose structure learning of PCs by combining pruning and growing operations to exploit thesparsity of PC structures.

Original: We show signiﬁcant empirical improvements in the density estimation tasksof PCs compared to existing PC learners and competing ﬂow-based models and VAEs.
Revised: We show significant empirical improvements in the density estimation tasksof PCs compared to existing PC learners and competing flow-based models and VAEs.

Original: Moreover,
Revised: Moreover,

Original: Sparse PC provides parallelism computation of parameter learning and inferences. 
Revised: Sparse PC provides parallelism computation of parameter learning and inferences. Checklist For all authors...

Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Revised: contributions and scope?

Revised: [Yes] Contributions are clearly stated in the third and fourthparagraphs of the introduction.

Revised: (b) Did you describe the limitations of your work?

Revised: [Yes] The proposed learning algorithms

Revised: are designed for smooth and decomposable PCs.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [N/A]

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Revised: [Yes]2.

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] All theoremsin the main text formally state all assumptions.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] Proofs are includedin the appendix.3.

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] Experiment

Revised: setup is detailed in the main text and the appendix.(b)

Revised: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Revised: [Yes] Training details are provided in the appendix.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [N/A](d)

Revised: Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] Details about computationalresources can be found in the appendix.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] We cited all PC

Revised: learning algorithms as well as the adopted datasets.

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] We specified that the used algorithms

Revised: are publicly available.

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes]

Revised: We included our code in the supplementary material.

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [N/A]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Revised: information or offensive content?

Revised: [N/A]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: applicable?

Revised: [N/A]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: spent on participant compensation?

Revised: [N/A]

Revised: A Pseudocode

Revised: In this section, we list the detailed algorithms of pruning operation (Section 3), circuit flows computation (Definition 4), and mini-batch Expectation Maximization (Section 5).

Revised: Algorithm 3 shows how to prune k percentage edges from PC C following heuristic h .

Revised: Algorithm

Revised: Input : a non-deterministic PC C , heuristic h deciding which edge to prune, h can be E F LOW ,

Revised: E R AND , or E P ARAM , percentage of edges to prune k

Revised: Output : a PC C ’ after pruned 1 old2new ← mapping from input PC n ∈ C to pruned PC 2 s ( n, c ) ← compute a score for each edge ( n, c ) based on heuristic h 3 f ( n, c ) ← false 4 f ( n, c ) ← true if s ( n, c ) ranks the last k 5 // visit

Revised: children before parents

Revised: 6 foreach n ∈ C do if n is a leaf then then

Revised: Algorithm 4 computes the circuit flows of a sample x given PC C with parameters θ though oneforward pass (line 1) and one backward pass (line 2-8).

Revised: in backward order do flow [ n ] ← (cid:80) g ∈ pa ( n ) flow [ g ] if n is a sum node then ∀ c ∈ in ( n ) , flow [ n, c ] ← θ c | n p [ c ] p [ n ] flow [ n ] else ∀ c ∈ in ( n ) , flow [ n, c ] ← flow [ n ]

Revised: Algorithm 5 shows the pipeline of mini-batches Expectation Maximization algorithm given PC C ,dataset D , batch size B and learning rate α .

Revised: B Proofs

Revised: In this section, we provide detailed proofs of Theorem 1 (Section B.1) and Theorem 2 (Section B.2).

Revised: B.1 Pruning One Edge over One Example

Revised: Lemma 1 (Pruning One Edge Log-Likelihood Lower Bound) .

Revised: For a PC C and a sample x , the lossof log-likelihood by pruning away edge ( n, c ) is

Revised: [Equation]

Revised: Input : a PC C ; dataset D ; batch size B ; learning rate α

Revised: Output : parameters θ estimated from D 1 θ ← random initialization 2 For root

Revised: [ n ] ←3 while converged or early stopped do

Revised: D ′ ← B random samples from D flow ← (cid:80) x ∈D ′

Revised: CircuitFlow ( C , θ , x ) for sum unit n and its child c do θ newc | n ← flow [ n, c ] / flow [ n ] θ c | n ← αθ ( new ) c | n + (1 − α )

Revised: θ c | n

Revised: Proof.

Revised: For notation simplicit, denote the probability of units m (resp. n ) in the original (resp.

Revised: pruned)

Revised: PC given sample x as p m ( x ) (resp.

Revised: p ′ n ( x ) ).

Revised: As a slight extension of Definition 4, we define F n ( x ; m )as the flow of unit n w.r.t.

Revised: the PC rooted at m .

Revised: The proof proceeds by induction over the PC’s root unit.

Revised: That is, we first consider pruning ( n, c ) w.r.t.the PC rooted at n .

Revised: Then, in the induction step, we prove that if the lemma holds for PC rooted at m ,then it also holds for PC rooted at any parent unit of m .

Revised: Instead of directly proving the statement in

Revised: Lemma 1, we first prove that for any root node m , the following holds:

Revised: [Equation]

Revised: Base case: pruning an edge of the root unit.

Revised: That is, the root unit of the PC is n .

Revised: In this case, we have

Revised: [Equation]

Revised: where θ ′ c | n denotes the normalized parameter corresponding to edge ( n, c ) in the pruned PC.

Revised: Specifically, we have

Revised: [Equation]

Revised: For notation simplicity, denote θ := θ c | n .

Revised: Plug in the above definition into Equation 5, we have

Revised: [Equation]

Revised: where ( a ) follows from the fact that F n ( x ; n ) = 1 and F n,c ( x ; n ) = θ c | n p c ( x ) /p n ( x ) .

Revised: Inductive case #1: suppose Equation 4 holds for m .

Revised: If product unit d is a parent of m , we show that

Revised: Equation 4 also holds for d :

Revised: [Equation]

Revised: where ( a ) is the inductive step that applies Equation 6; ( b ) follows from the fact that (note that d is aproduct unit) F n ( x ; m )

Revised: =

Revised: F n ( x ; d ) and F n,c ( x ; m ) =

Revised: F n,c ( x ; d ) .

Revised: Inductive case #2: for sum unit d , suppose Equation 4 holds for m ,

Revised: where m ∈

Revised: A iff m ∈ in ( d ) andm is an ancester of n and c .

Revised: Assume all other children of d are not ancestoer of n , we show that

Revised: Equation 4 also holds for d :

Revised: [Equation]

Revised: ) − θ 1 − θ (cid:19) =

Revised: θ m | d · F n ( x ; m ) · p m ( x ) · (cid:18)1 − θ

Revised: F n,c ( x ; d )

Revised: F n ( x ; d ) − θ 1 − θ (cid:19)

Revised: =

Revised: θ m | d · F n ( x ; d ) · (cid:80) m ′ ∈ in ( d ) θ m ′ |

Revised: d p m ′ ( x ) θ m | d p m ( x ) · p m ( x ) · (cid:18)1 − θ

Revised: F n,c ( x ; d )

Revised: F n ( x ; d ) − θ 1 − θ (cid:19) =

Revised: F n ( x ; d ) · (cid:18) (cid:88) m ′ ∈ in ( d ) θ m ′

Revised: | d p m ′ ( x ) (cid:19) · (cid:18)1 − θ

Revised: F n,c ( x ; d )

Revised: F n ( x ; d ) − θ 1 − θ (cid:19) =

Revised: F n ( x ; d ) · p d ( x ) · (cid:18)1 − θ

Revised: F n,c ( x ; d )

Revised: F n ( x ; d ) − θ 1 − θ (cid:19) .

Revised: Therefore, following Equation 4 for root r , we havep r ( x ) − p ′

Revised: r ( x ) p r ( x ) = 1 − θ F n,c ( x ; r ) − θ 1 − θF n ( x ; r ) ⇔ p ′

Revised: r ( x ) p r ( x ) = 1 + θ 1 − θF n ( x ; r ) − 1 − θ F n,c ( x ; r )

Revised: Therefore, we have∆ LL ( { x } , C , { ( n, c ) } )

Revised: = log p r ( x ) − log p ′

Revised: r ( x ) =|D| (cid:88) x ∈D log (cid:18) 1 − θ c |

Revised: [Equation]

Revised: where ( a ) follows from the fact that F n,c ( x ) ≤ F n ( x ) .

Revised: Theorem 1 follows directly from Lemma 1 by noting that for any dataset D , ∆ LL ( D , C , { ( n, c ) } )

Revised: =1 |D| ∆ LL ( { x } , C , { ( n, c ) } ) .

Revised: B.2 Pruning Multiple Edge

Revised: Proof.

Revised: Similar to the proof of Lemma 1, we prove Theorem 2 by induction.

Revised: Different from Lemma 1,we induce a slightly different objective:

Revised: [Equation]

Revised: where des ( n ) is the set of descendent units of n .

Revised: Base case: the base case follows directly from the proof of Lemma

Revised: [Equation]

Revised: and lead to the conclusion in

Revised: Equation 6.

Revised: Inductive case #1: suppose for all children of a product unit d , Equation 7 holds, we show that

Revised: Equation 7 also holds for d :

Revised: [Equation]

Revised: where ( a ) uses the definition that p d ( x ) = (cid:81) m ∈ in ( d ) p m ( x ) .

Revised: Inductive case #2: suppose for all children of a sum unit d , Equation 7 holds, we show that Equation 7also holds for d :

Revised: [Equation]

Revised: where ( a ) follows from the base case of the induction.

Revised: Next, we focus on the first term of the aboveequation:(cid:88)

Revised: [Equation]

Revised: where the derivation of the last inequality follows from the corresponding steps in the proof of

Revised: Lemma 1.

Revised: Therefore, from Equation 7, we can conclude that

Revised: [Equation]

Revised: Finally, we prove the approximation step in Equation 3.

Revised: Let ϵ ( · ) =

Revised: (cid:80) ( n,c ) ∈E F n,c ( · ) ∈ [0 , 1) .

Revised: Wehave,

Revised: [Equation]

Revised: C Experiments Details

Revised: Hardware specifications All experiments are performed on a server with 32 CPUs, 126G Memory,and NVIDIA RTX A5000 GPUs with 26G Memory.

Revised: In all experiments, we only use a single GPU onthe server.

Revised: C.1 Datasets

Revised: For MNIST-family datasets, we split 5% of training set as validation set for early stopping.

Revised: For Penn

Revised: Tree Bank dataset, we follow the setting in Mikolov et al.

Revised: [24] to split a training, validation, and testset.

Revised: Table 3 lists the all the dataset statistics.

Revised: test ).

Revised: [Table]

Revised: C.2 Learning Hidden Chow-Liu Trees

Revised: HCLT structures.

Revised: Adopting hidden chow liu tree (HCLT) PC architecture as in Liu and Van den

Revised: Broeck [19], we reimplement the learning process to speed it up and use a different training pipelineand hyper-parameters tuning.

Revised: EM parameter learning We adopt the EM parameter learning algorithm introduced in Choi et al.[3], which computes the EM update target parameters using circuit flows.

Revised: We use a stochasticmini-batches EM algorithm.

Revised: Denoting θ new as the EM update target computed from a mini-batch ofsamples, and we update the targeting parameter with a learning rate α : θ t +1 ← αθ new + (1 − α ) θ t .α is piecewise-linearly annealed from [1 . 0 , 0 . 1] , [0 . 1 , 0 . 01] , [0 . 01 , 0 . 001] , and each piece is trained Tepochs.

Revised: Hyper-parameters searching.

Revised: For all the experiments, the hyper-parameters are searched from

Revised: • h ∈ { 8 , 16 , 32 , 64 , 128 , 256 } , the hidden size of HCLT structures;

Revised: • γ ∈ { 0 . 0001 , 0 . 001 , 0 . 01 , 0 .

Revised: 1 , 1 . 0 } , Laplace smoothing factor;

Revised: • B ∈ { 128 , 256 , 512 , 1024 } , batch-size in mini-batches EM algorithm;

Revised: • α piecewise-linearly annealed from [1 . 0 , 0 . 1] , [0 . 1 , 0 . 01] , [0 . 01 , 0 . 001] , where each piece iscalled one mini-batch EM phase.

Revised: Usually the algorithm will start to overfit as validation setand stop at the third phase;

Revised: • T = 100 , number of epochs for each mini-batch EM phase.

Revised: The PC size is quadratically growing with hidden size h , thus it is inefficient to do a grid searchamong the entire hyper-parameters space.

Revised: What we do is to fist do a grid search when h = 8 orh = 16 to find the best Laplace smoothing factor γ and batch-size B for each dataset, and thenfix γ and B to train a PC with larger hidden size h ∈ { 32 , 64 , 128 , 256 } .

Revised: The best tuned B is in{ 256 , 512 } , which is different for different hidden size h , and the best tuned γ is 0 .

Revised: C.3 Details of Section 6.

Revised: Sparse PC (ours).

Revised: Given HCLT learned in Section C.2 as initial PC, we use the structure learningprocess proposed in Section 5.

Revised: Specifically, starts from initial HCLT, for each iteration, we (1) prune75% of the PC parameters, and (2) grow PC size with Gaussian variance ϵ , (3) finetuing PC usingmini-batches EM parameter learning with learning rate α .

Revised: We prune and grow PC iteratively until thevalidation set likelihood is overfitted .

Revised: The hyper-parameters are searched from

Revised: • ϵ ∈ { 0 . 1 , 0 . 3 , 0 . 5 } , Gaussian variance in growing operation;

Revised: • α , piecewise-linearly annealed from [0 . 1 , 0 . 01] , [0 . 01 , 0 . 001] ;

Revised: • T = 50 , number of epochs for each mini-batch EM phase;

Revised: • for γ and B , we use the tuned best number from Section C.2.

Revised: HCLT.

Revised: The HLCT experiments in Table 1 are performed following the original paper (Code https://github.com/UCLA-StarAI/Tractable-PC-Regularization ), which is different from theleaning pipeline we use as our inital PC (Section C.2).

Revised: SPN.

Revised: We reimplement the SPN architecture ourselves following Peharz et al.

Revised: [28] and train it withthe same mini-batch pipeline as HCLT.

Revised: IDF.

Revised: We run all experiments with the code in the GitHub repo provided by the authors.

Revised: We adoptan IDF model with the following hyperparameters: 8 flow layers per level; 2 levels; densenets withdepth 6 and 512 channels; base learning rate 0.001; learning rate decay 0.999.

Revised: The algorithm adoptsan CPU-based entropy coder rANS.

Revised: BitSwap.

Revised: We train all models using the following author-provided script: https://github.com/fhkingma/bitswap/blob/master/model/mnist_train .

Revised: BB-ANS.

Revised: All experiments are performed using the following official code https://github.com/bits-back/bits-back .

Revised: McBits.

Revised: All experiments are performed using the following official code https://github.com/ryoungj/mcbits .

Revised: C.4 Details of Section 6.2

Revised: For all experiments in Section 6.2, we use the best tuned γ and B from Section C.2 and hidden size hranging from { 16 , 32 , 64 , 128 } .

Revised: For experiments “What is the Smallest PC for the Same Likelihood?”,the hyper-parameters are searched from

Revised: • k ∈ { 0 . 05 , 0 . 1 , 0 . 3 } , percentage of parameters to prune each iteration;

Revised: • α , piecewise-linearly annealed from [0 . 3 , 0 . 1] , [0 . 1 , 0 . 01] , [0 . 01 , 0 . 001] ;

Revised: • T = 50 , number of epochs for each mini-batch EM phase;

Revised: For experiments “What is the Best PC Given the Same Size?”

Revised: , we use the same setting as in

Revised: Section C.3.