Original: Probabilistic Generating Circuits
Revised: Probabilistic Generating Circuits

Original: Abstract
Revised: Abstract

Original: Generating functions, which are widely used in combinatorics and probability theory, encode function values into the coefﬁcients of a polynomial.
Revised: Generating functions, which are widely used in combinatorics and probability theory, encode function values into the coefﬁcients of a polynomial.

Original: In this paper, we explore their use as a tractable probabilistic model, and propose probabilistic generating circuits (PGCs) for their efﬁcient representation.
Revised: In this paper, we explore their use as a tractable probabilistic model, and propose probabilistic generating circuits (PGCs) for their efﬁcient representation.

Original: PGCs  strictly subsume many existing tractable probabilistic models, including determinantal point processes (DPPs), probabilistic circuits (PCs) such as sum-product networks, and tractable graphical models.
Revised: PGCs are strictly more expressive efﬁcient than many existing tractable probabilistic models, including determinantal point processes (DPPs), probabilistic circuits (PCs) such as sum-product networks, and tractable graphical models.

Original: We contend that PGCs are not just a theoretical framework that uniﬁes vastly different existing models, but also show huge potential in modeling realistic data.
Revised: We contend that PGCs are not just a theoretical framework that uniﬁes vastly different existing models, but also show great potential in modeling realistic data.

Original: We exhibit a simple class of PGCs that are not trivially subsumed by simple combinations of PCs and DPPs, and obtain competitive performance on a suite of density estimation benchmarks.
Revised: We exhibit a simple class of PGCs that are not trivially subsumed by simple combinations of PCs and DPPs, and obtain competitive performance on a suite of density estimation benchmarks.

Original: We also highlight PGCs’ connection to the theory of strongly Rayleigh distributions.
Revised: We also highlight PGCs’ connection to the theory of strongly Rayleigh distributions.

Original: 1. Introduction
Revised: 1. Introduction

Original: Probabilistic modeling is an important task in machine learning.
Revised: Probabilistic modeling is an important task in machine learning.

Original: Scaling up such models is a key challenge: probabilistic inference quickly becomes intractable as the models become large and sophisticated (Roth, 1996).
Revised: Scaling up such models is a key challenge: probabilistic inference quickly becomes intractable as the models become large and sophisticated (Roth, 1996).

Original: Central to this effort is the development of tractable probabilistic models (TPMs) that guarantee tractable probabilistic inference in the size of the model yet can efﬁciently represent a wide range of probability distributions.
Revised: Central to this effort is the development of tractable probabilistic models (TPMs) that guarantee tractable probabilistic inference in the size of the model, yet can efﬁciently represent a wide range of probability distributions.

Original: There has been a proliferation of different classes of TPMs.
Revised: There has been a proliferation of different classes of TPMs.

Original: Examples include boundedtreewidth graphical models (Meila & Jordan, 2000), determinantal point processes (Borodin & Rains, 2005; Kulesza & Taskar, 2012) and various probabilistic circuits (Darwiche, 2009;
Revised: Examples include boundedtreewidth graphical models (Meila & Jordan, 2000), determinantal point processes (Borodin & Rains, 2005; Kulesza & Taskar, 2012), and various probabilistic circuits (Dar- wiche, 2009;

Original: Kisa et al., 2014; Vergari et al., 2020) such as< sum-product networks (Poon & Domingos, 2011).
Revised: Kisa et al., 2014; Vergari et al., 2020) such as sum-product networks (Poon & Domingos, 2011).

Original: Ideally, we want our probabilistic models to be as expressive efﬁcient (Martens & Medabalimi, 2014) as possible, meaning that they can efﬁciently represent as many classes of distributions as possible, and adapt to a wider spectrum of realistic applications.
Revised: Ideally, we want our probabilistic models to be as expressive efﬁcient (Martens & Medabalimi, 2014) as possible, meaning that they can efﬁciently represent as many classes of distributions as possible, and adapt to a wider spectrum of realistic applications.

Original: Often, however, stronger expressive power comes at the expense of tractability: fewer restrictions can make a model more expressive, but it can also make probabilistic inference intractable.
Revised: Often, however, stronger expressive power comes at the expense of tractability: fewer restrictions can make a model more expressive efﬁcient, but it can also make probabilistic inference intractable.

Original: We therefore raise the following central research question of this paper: Does there exist a class of tractable probabilistic models that is strictly more expressive efﬁcient than current TPMs?
Revised: We therefore raise the following central research question of this paper: Does there exist a class of tractable probabilistic models that is strictly more expressive efﬁcient than current TPMs?

Original: All aforementioned models are usually seen as representing probability mass functions : they take assignments to random variables as input and output likelihoods.
Revised: All aforementioned models are usually seen as representing probability mass functions : they take assignments to random variables as input and output likelihoods.

Original: In contrast, especially in the ﬁeld of probability theory, it is also common to represent distributions as probability generating polynomials (or generating polynomials for short).
Revised: In contrast, especially in the ﬁeld of probability theory, it is also common to represent distributions as probability generating polynomials (or generating polynomials for short).

Original: Generating polynomials are a powerful mathematical tool, but they have not yet found direct use as a probabilistic machine learning representation that permits tractable probabilistic inference.
Revised: Generating polynomials are a powerful mathematical tool, but they have not yet found direct use as a probabilistic machine learning representation that permits tractable probabilistic inference.

Original: We make the key observation that the marginal probabilities (including likelihoods) for a probability distribution can be computed by evaluating its generating polynomial in a particular way.
Revised: We make the key observation that the marginal probabilities (including likelihoods) for a probability distribution can be computed by evaluating its generating polynomial in a particular way.

Original: Based on this observation, we propose probability generating circuits (PGCs), a class of probabilistic models that represent probability generating polynomials compactly as directed acyclic graphs.
Revised: Based on this observation, we propose probabilistic generating circuits (PGCs), a class of probabilistic models that represent probability generating polynomials compactly as directed acyclic graphs.

Original: PGCs provide a partly positive answer to our research question: they are the ﬁrst known class of TPMs that (strictly) subsume probabilistic circuits (PCs) such as sum-product networks (Poon & Domingos, 2011) and determinantal point processes (DPPs) while supporting tractable marginal inference.
Revised: PGCs provide a partly positive answer to our research question: they are the ﬁrst known class of TPMs that are strictly more expressive efﬁcient than decomposable probabilistic circuits (PCs), in particular, sum-product networks, and determinantal point processes (DPPs) while supporting tractable marginal inference.

Original: Section 2 formally deﬁnes PGCs and establishes their tractability by presenting an efﬁcient algorithm for computing marginals.
Revised: Section 2 formally deﬁnes PGCs and establishes their tractability by presenting an efﬁcient algorithm for computing marginals.

Original: Section 3 demonstrates the expressive power of PGCs by showing that they subsume PCs andDPPs.
Revised: Section 3 demonstrates the expressive power of PGCs by showing that they subsume PCs and

Original: Section 4 shows that there are PGCs that are not trivially subsumed by simple combinations of PCs and DPPs.

Original: Sectionevaluates PGCs on standard density estimation benchmarks: even simple PGCs outperform other TPM learners on half of the datasets. Then, Section 6 highlights PGCs’ connection to strongly Rayleigh distributions. Section 7 summarizes [Figure]
Revised: DPPs while remaining strictly more expressive efﬁcient.

Original: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: Section 4 shows that there are PGCs that cannot be represented by PCs with DPPs as leaves.

Original: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: Section 5 evaluates

Original: [Table] [Figure] [Table] [Figure]
Revised: PGCs on standard density estimation benchmarks: even the

Original: [Table]
Revised: [Table]

Original: (a)
Revised: (a)

Original: (c) Probabilistic mass circuit
Revised: (c) Probabilistic mass circuit

Original: L β =
Revised: L β =

Original: X
Revised: X

Original: X
Revised: X

Original: X 3 (cid:34) (cid:35)
Revised: X 3 (cid:34) (cid:35)

Original: X 

Revised: X

Original: X 
 X
Revised: X 
 X

Original: K β =
Revised: K β =

Original: X
Revised: X

Original: X
Revised: X

Original: X 3 (cid:34) (cid:35) 0 .0 .
Revised: X 3 (cid:34) (cid:35) 0 .0 .

Original: X 1 0 .0 .
Revised: X 1 0 .0 .

Original: X 2 0 .
Revised: X 2 0 .

Original: X(d) Kernel L β and marginal kernel K β for a DPP  the paper and motivates future research directions.
Revised: X(d) Kernel L β and marginal kernel K β for a DPP simplest PGCs outperform other TPM learners on half of the datasets. Then, Section 6 highlights PGCs’ connection to strongly Rayleigh distributions. Section 7 summarizes the paper and motivates future research directions.

Original: 2. Probabilistic Generating Circuits
Revised: 2. Probabilistic Generating Circuits

Original: In this section we establish probabilistic generating circuits (PGCs) as a class of tractable probabilistic models.
Revised: In this section we establish probabilistic generating circuits (PGCs) as a class of tractable probabilistic models.

Original: We ﬁrst introduce generating polynomials as a representation for probability distributions and propose PGCs for their compact representations.
Revised: We ﬁrst introduce generating polynomials as a representation for probability distributions and propose PGCs for their compact representations.

Original: Then, we show that marginal probabilities for a PGC can be computed efﬁciently.
Revised: Then, we show that marginal probabilities for a PGC can be computed efﬁciently.

Original: 2.1. Probability Generating Polynomials
Revised: 2.1. Probability Generating Polynomials

Original: It is a common technique in combinatorics to encode sequences as generating polynomials .
Revised: It is a common technique in combinatorics to encode sequences as generating polynomials .

Original: In particular, probability distributions over binary random variables can be represented by probability generating polynomials .
Revised: In particular, probability distributions over binary random variables can be represented by probability generating polynomials .

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Let Pr( · ) be a probability distribution over binary random variables X 1 , X 2 , . . .
Revised: Let Pr( · ) be a probability distribution over binary random variables X 1 , X 2 , . . .

Original: , X n , then the probability-generating polynomial (or generating polynomial for short) for the distribution is deﬁned as
Revised: , X n , then the probability generating polynomial (or generating polynomial for short) for the distribution is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (1) where α S =
Revised: where α S =

Original: Pr( { X i = 1 } i ∈ S , { X i = 0 } i/ ∈ S ) and z S = (cid:81) i ∈
Revised: Pr( { X i = 1 } i ∈ S , { X i = 0 } i/ ∈ S ) and z S = (cid:81) i ∈

Original: S z i .
Revised: S z i .

Original: As an illustrating example, we consider the probability distribution Pr β speciﬁed as a table in Figure 1a.
Revised: As an illustrating example, we consider the probability distribution Pr β speciﬁed as a table in Figure 1a.

Original: By deﬁnition, the generating polynomial for distribution Pr β is given by
Revised: By deﬁnition, the generating polynomial for distribution Pr β is given by

Original: [Equation]
Revised: [Equation]

Original: We see from Equation 2 that the generating polynomial for a distribution simply “enumerates” all possible variable assignments term-by-term, and the coefﬁcient of each term corresponds to the probability of an assignment.
Revised: We see from Equation 2 that the generating polynomial for a distribution simply “enumerates” all possible variable assignments term-by-term, and the coefﬁcient of each term corresponds to the probability of an assignment.

Original: The probability for the assignment X 1 = 0 , X 2 = 1 , X 3 = 1 , for example, is 0 . 48 , which corresponds to the coefﬁcient of the term z 2 z 3 .
Revised: The probability for the assignment X 1 = 0 , X 2 = 1 , X 3 = 1 , for example, is 0 . 48 , which corresponds to the coefﬁcient of the term z 2 z 3 .

Original: That is, given an assignment, we can evaluate its probability by directly reading off the coefﬁcient for the corresponding term.
Revised: That is, given an assignment, we can evaluate its probability by directly reading off the coefﬁcient for the corresponding term.

Original: We can also evaluate marginal probabilities by summing over the coefﬁcients for a set of terms.
Revised: We can also evaluate marginal probabilities by summing over the coefﬁcients for a set of terms.

Original: For example, the marginal probability Pr( X 2 = 0 , X 3 = 0) is given by Pr( X 1 = 0 , X 2 = 0 , X 3 = 0)
Revised: For example, the marginal probability Pr( X 2 = 0 , X 3 = 0) is given by Pr( X 1 = 0 , X 2 = 0 , X 3 = 0)

Original: +
Revised: +

Original: Pr( X 1 = 1 , X 2 = 0 , X 3 = 0) , which corresponds to the sum of the constant term and the coefﬁcient for the term z 1 .
Revised: Pr( X 1 = 1 , X 2 = 0 , X 3 = 0) , which corresponds to the sum of the constant term and the coefﬁcient for the term z 1 .

Original: 2.2. Compactly Representing Generating Polynomials
Revised: 2.2. Compactly Representing Generating Polynomials

Original: Equation 2 also illustrates that the size of a term-by-term representation for a generating polynomial is exponential in the number of variables.
Revised: Equation 2 also illustrates that the size of a term-by-term representation for a generating polynomial is exponential in the number of variables.

Original: As the number of variables increases, it quickly becomes impractical to compute probabilities by extracting coefﬁcients from these polynomials.
Revised: As the number of variables increases, it quickly becomes impractical to compute probabilities by extracting coefﬁcients from these polynomials.

Original: Hence, to turn generating polynomials into tractable models, we need a data structure to represent them more efﬁciently.
Revised: Hence, to turn generating polynomials into tractable models, we need a data structure to represent them more efﬁciently.

Original: We thus introduce a new class of probabilistic circuits called probabilistic generating circuits to represent generating polynomials compactly as directed acyclic graphs (DAGs).
Revised: We thus introduce a new class of probabilistic circuits called probabilistic generating circuits to represent generating polynomials compactly as directed acyclic graphs (DAGs).

Original: We ﬁrst present the formal deﬁnition for PGCs.
Revised: We ﬁrst present the formal deﬁnition for PGCs.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: A probabilistic generating circuit (PGC) is a directed acyclic graph consisting of three types of nodes:
Revised: A probabilistic generating circuit (PGC) is a directed acyclic graph consisting of three types of nodes:

Original: Sum nodes (cid:76) with weighted edges to children; 2.
Revised: Sum nodes (cid:76) with weighted edges to children; 2.

Original: Product nodes (cid:78) with unweighted edges to children; 3.
Revised: Product nodes (cid:78) with unweighted edges to children; 3.

Original: Leaf nodes , which are z i or constants.
Revised: Leaf nodes , which are z i or constants.

Original: A PGC has one node of out-degree0 (edges are directed from children to parents), and we refer to it as the root of the PGC.
Revised: A PGC has one node of out-degree (edges are directed from children to parents), and we refer to it as the root of the PGC.

Original: The size of a PGC is the number of edges in it.
Revised: The size of a PGC is the number of edges in it.

Original: Each node in a PGC represents a polynomial, (i) each leaf in a PGC represents the polynomial z i or a constant, (ii) each sum node represents the weighted sum over the polynomials represented by its children, and (iii) each product node represents the unweighted product over the polynomials represented by its children.
Revised: Each node in a PGC represents a polynomial, (i) each leaf in a PGC represents the polynomial z i or a constant, (ii) each sum node represents the weighted sum over the polynomials represented by its children, and (iii) each product node represents the unweighted product over the polynomials represented by its children.

Original: The polynomial represented by a PGC is the polynomial represented by its root.
Revised: The polynomial represented by a PGC is the polynomial represented by its root.

Original: We have now fully speciﬁed the syntax of PGCs, but a PGC with valid syntax does not necessarily have valid semantics .
Revised: We have now fully speciﬁed the syntax of PGCs, but a PGC with valid syntax does not necessarily have valid semantics .

Original: Because of the presence of negative parameters, it is not guaranteed that the polynomial represented by a PGC is a probability generating polynomial: it might contain terms that are not multiafﬁne or have negative coefﬁcients (e.g.
Revised: Because of the presence of negative parameters, it is not guaranteed that the polynomial represented by a PGC is a probability generating polynomial: it might contain terms that are not multiafﬁne or have negative coefﬁcients (e.g.

Original: − 1 .
Revised: − 1 .

Original: 2 z 1 z 32 ).
Revised: 2 z 1 z 32 ).

Original: In practice, however, we show in later sections ways of constructing PGC structures that are guaranteed to have valid semantics for any parameterization.
Revised: In practice, however, we show in Sectionthat, by certain compositional operations, we can construct PGCs that are guaranteed to have valid semantics for any parameterization.

Original: Continuing our example, we observe that the generating polynomial g β in Equation 2 can be re-written as:
Revised: Continuing our example, we observe that the generating polynomial g β in Equation 2 can be re-written as:

Original: [Equation]
Revised: [Equation]

Original: Based on Equation 3, we can immediately construct a PGC that compactly represents g β , as shown in Figure 1b.
Revised: Based on Equation 3, we can immediately construct a PGC that compactly represents g β , as shown in Figure 1b.

Original: In this way, generating polynomials for high-dimensional distributions may become feasible to represent by PGCs.
Revised: In this way, generating polynomials for high-dimensional distributions may become feasible to represent by PGCs.

Original: 2.3. Tractable Inference with PGCs
Revised: 2.3. Tractable Inference with PGCs

Original: We now show that the computation of marginals is tractable for PGCs.
Revised: We now show that the computation of marginals is tractable for PGCs.

Original: As brieﬂy mentioned in Section 2.1, we can compute probabilities by extracting the coefﬁcients of generating polynomials, which is much trickier when they are represented as deeply-nested DAGs; as shown in Figure 1b, it is impossible to directly read off any coefﬁcient.
Revised: As brieﬂy mentioned in Section 2.1, we can compute probabilities by extracting the coefﬁcients of generating polynomials, which is much trickier when they are represented as deeply-nested DAGs; as shown in Figure 1b, it is impossible to directly read off any coefﬁcient.

Original: We circumvent this problem by making the key observation that we can “zero-out” the terms we don’t want from generating polynomials by evaluating them in a certain way.
Revised: We circumvent this problem by making the key observation that we can “zero-out” the terms we don’t want from generating polynomials by evaluating them in a certain way.

Original: For example, when evaluating a marginal probability with X 1 set to 0 , we zero-out all terms that contain z 1 by setting z 1 to 0 .
Revised: For example, when evaluating a marginal probability with X 1 set to 0 , we zero-out all terms that contain z 1 by setting z 1 to 0 .

Original: We generalize this idea as follows.
Revised: We generalize this idea as follows.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let g ( z 1 , . . .
Revised: Let g ( z 1 , . . .

Original: , z n ) be a probability generating polynomial for Pr( · ) , then for A, B ⊆ { 1 , . .
Revised: , z n ) be a probability generating polynomial for Pr( · ) , then for A, B ⊆ { 1 , . .

Original: . , n } with
Revised: . , n } with

Original: A ∩ B = ∅ , the marginal probability can be computed by:
Revised: A ∩ B = ∅ , the marginal probability can be computed by:

Original: Pr( { X i = 1 } i ∈
Revised: Pr( { X i = 1 } i ∈

Original: A , { X i = 0 } i ∈ B ) =
Revised: A , { X i = 0 } i ∈ B ) =

Original: coef | A | ( g ( { z i = t }
Revised: coef | A | ( g ( { z i = t }

Original: i ∈
Revised: i ∈

Original: A , { z i = 0 } i ∈ B , { z i = 1 }
Revised: A , { z i = 0 } i ∈ B , { z i = 1 }

Original: i/ ∈
Revised: i/ ∈

Original: A ∪ B )) , where t is an indeterminate for polynomials, and coef k ( g ( t )) denotes the coefﬁcient for the term t k in g ( t ) .
Revised: A ∪ B )) , where t is an indeterminate for polynomials, and coef k ( g ( t )) denotes the coefﬁcient for the term t k in g ( t ) .

Original: Lemma 1 basically says that for a generating polynomial, its marginals can be computed by evaluating the generating polynomial in the polynomial ring R [ t ] .
Revised: Lemma 1 basically says that for a generating polynomial, its marginals can be computed by evaluating the generating polynomial in the polynomial ring R [ t ] .

Original: This result naturally extends to PGCs with a bottom-up pass: we ﬁrst evaluate the leaf nodes to t , 1 or 0 based on the assignment; then, for the sum nodes, we compute the weighted sum over the polynomials that their children evaluate to; for the product nodes, we compute the product over the polynomials that their children evaluate to.
Revised: With a bottom-up pass, this result naturally extends to generating polynomials represented as PGCs: we ﬁrst evaluate the leaf nodes to t ,or 0 based on the assignment; then, for the sum nodes, we compute the weighted sum over the polynomials that their children evaluate to; for the product nodes, we compute the product over the polynomials that their children evaluate to.

Original: Note that as we are taking sums and products over univariate polynomials of degree n , the time complexities for the naive algorithms are O ( n ) and O ( n 2 ) , respectively.
Revised: Note that, as we are taking sums and products over univariate polynomials of degree n , the time complexities for the naive algorithms are O ( n ) and O ( n 2 ) , respectively.

Original: It is not hard to see that computing marginal probabilities is tractable for PGCs.
Revised: In light of this, it is not hard to see that computing marginal probabilities is polynomial-time with respect to the size of the PGCs.

Original: Theorem 1.
Revised: Theorem 1.

Original: For PGCs of size m representing distributions on n binary random variables, marginal probabilities (including likelihoods) are computable in time
Revised: For PGCs of size m representing distributions on n binary random variables, marginal probabilities (including likelihoods) are computable in time

Original: O ( mn log n log log n ) .
Revised: O ( mn log n log log n ) .

Original: The O ( mn log n log log n ) complexity in the theorem consists of two parts: the O ( m ) part is the time complexity of a bottom-up pass for a PGC of size m and the
Revised: The O ( mn log n log log n ) complexity in the theorem consists of two parts: the O ( m ) part is the time complexity of a bottom-up pass for a PGC of size m and the

Original: O ( n log n log log n ) part is contributed by the time complexity for computing the product of two degree- n polynomials with fast Fourier transform (Sch¨onhage & Strassen, 1971; Cantor & Kaltofen, 1991).
Revised: O ( n log n log log n ) part is contributed by the time complexity of computing the product of two degree- n polynomials with fast Fourier transform (Sch ¨ onhage & Strassen, 1971; Cantor & Kaltofen, 1991).

Original: 3. PGCs Subsume Other Probabilistic Models
Revised: 3. PGCs Subsume Other Probabilistic Models

Original: To this point, we have introduced PGCs as a probabilistic model and shown that they support tractable marginals.
Revised: To this point, we have introduced PGCs as a probabilistic model and shown that they support tractable marginals.

Original: Next, we show that PGCs are at least as expressive efﬁcient as other TPMs by showing that PGCs tractably subsume decomposable probabilistic circuits and determinantal point processes.
Revised: Next, we show that PGCs are strictly more expressive efﬁcient than other TPMs by showing that PGCs tractably subsume decomposable probabilistic circuits and determinantal point processes.

Original: 3.1. PGCs Subsume Other Probabilistic Circuits
Revised: 3.1. PGCs Subsume Other Probabilistic Circuits

Original: We start by introducing the basics of probabilistic circuits (Vergari et al., 2020; Choi et al., 2020).
Revised: We start by introducing the basics of probabilistic circuits (Vergari et al., 2020; Choi et al., 2020).

Revised: Just like PGCs, each PC also represents a polynomial with respect to variables

Revised: X i and X i .

Original: The syntax of probabilistic circuits (PCs) is basically the same as PGCs except for the following:(1) the variables in PCs are X
Revised: The syntax of probabilistic circuits (PCs) is basically the same as PGCs except for the following: the variables in PCs are X

Original: i s and X
Revised: i s and X

Original: i s rather than z i s; they are inherently different in the sense that X i s and X
Revised: i s rather than z i s; they are inherently different in the sense that X i s and X

Original: i s are  random variables where z i s arepurely symbolic formal objects.
Revised: i s are the random variables themselves, while z i s are symbolic formal objects; 2.

Original: (2) the edge weights of PCs must be non-negative. Unlike PGCs,  all of the existing PCs represent probability mass polynomials, so we sometimes refer to them as probabilistic mass circuits .
Revised: the edge weights of PCs must be non-negative; 3. unlike PGCs, which represent probability generating polynomials, all of the existing PCs represent probability mass functions (as polynomials), so we sometimes refer to them as probabilistic mass circuits .

Original: Figure 1c shows an example PC that represents the distribution Pr β .
Revised: Figure 1c shows an example PC that represents the distribution Pr β .

Original: For a given assignment X = x , the PC
Revised: For a given assignment X = x , the PC

Original: A evaluates to a number A ( x ) , which is obtained by (i) replacingvariable leaves X i  by x i , (ii) replacingvariable leaves X i  by 1 − x i , (iii) evaluating product nodes as taking the product over their children, and (iv) evaluating sum nodes as taking a weighted sum over their children.
Revised: A evaluates to a number A ( x ) , which is obtained by (i) replacing X i variable leaves by x i , (ii) replacing X i variable leaves by 1 − x i , (iii) evaluating product nodes as taking the product over their children, and (iv) evaluating sum nodes as taking a weighted sum over their children.

Original: Finally, a PC A with variable leaves X =
Revised: Finally, a PC A with variable leaves X =

Original: ( X 1 , . . .
Revised: ( X 1 , . . .

Original: , X n ) represents the probability distribution
Revised: , X n ) represents the probability distribution

Original: Pr( X = x ) ∝ A ( x ) .
Revised: Pr( X = x ) ∝ A ( x ) .

Original: For an arbitrary PC, most probabilistic inference tasks, including marginals and MAP inference, are computationally hard in the circuit size.
Revised: For an arbitrary PC, most probabilistic inference tasks, including marginals and MAP inference, are computationally hard in the circuit size.

Original: In order to guarantee the efﬁcient evaluation of queries it is therefore necessary to impose further constraints on the structure of the circuit.
Revised: In order to guarantee the efﬁcient evaluation of queries it is therefore necessary to impose further constraints on the structure of the circuit.

Original: In this paper we consider two well-known structural properties of probabilistic circuits (Darwiche & Marquis, 2002; Choi et al., 2020):
Revised: In this paper we consider two well-known structural properties of probabilistic circuits (Darwiche & Marquis, 2002; Choi et al., 2020):

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: For a PC, we denote the input variables that a node depends on as its scope ; then,
Revised: For a PC, we denote the input variables that a node depends on as its scope ; then,

Original: A (cid:78) node is decomposable if the scopes of its children are disjoint.
Revised: A (cid:78) node is decomposable if the scopes of its children are disjoint.

Original: A (cid:76) node is smooth if the scopes of its children are the same.
Revised: A (cid:76) node is smooth if the scopes of its children are the same.

Original: A PC is decomposable if all of its (cid:78) nodes are decomposable; a PC is smooth if all of its (cid:76) nodes are smooth.
Revised: A PC is decomposable if all of its (cid:78) nodes are decomposable; a PC is smooth if all of its (cid:76) nodes are smooth.

Original: Let A be a PC over X 1 , . . .
Revised: Let A be a PC over X 1 , . . .

Original: , X n .
Revised: , X n .

Original: If A is decomposable and smooth, then we can efﬁciently compute its marginals: for disjoint A, B ⊆ { 1 , . . . , n } the marginal probability
Revised: If A is decomposable and smooth, then we can efﬁciently compute its marginals: for disjoint A, B ⊆ { 1 , . . . , n } the marginal probability

Original: Pr( { X i = 1 } i ∈
Revised: Pr( { X i = 1 } i ∈

Original: A , { X i = 0 } i ∈ B ) is given by the evaluation of A with the following inputs.
Revised: A , { X i = 0 } i ∈ B ) is given by the evaluation of A with the following inputs.

Original: [Equation]
Revised: [Equation]

Original: The class of decomposable PCs subsumes many TPMs as subclasses.
Revised: Many TPMs are certain forms of decomposable PCs.

Original: Examples include sum-product networks (SPNs) (Poon & Domingos, 2011; Peharz et al., 2019), AndOr graphs (Mateescu et al., 2008), probabilistic sentential decision diagrams (PSDDs) (Kisa et al., 2014), arithmetic circuits (Darwiche, 2009), cutset networks (Rahman & Gogate, 2016) and bounded-treewidth graphical models (Meila & Jordan, 2000) such as Chow-Liu trees (Chow & Liu, 1968) and hidden Markov models (Rabiner & Juang, 1986).
Revised: Examples include sum-product networks (SPNs) (Poon & Domingos, 2011; Peharz et al., 2019), And-Or graphs (Mateescu et al., 2008), probabilistic sentential decision diagrams (PSDDs) (Kisa et al., 2014), arithmetic circuits (Darwiche, 2009), cutset networks (Rahman & Gogate, 2016) and bounded-treewidth graphical models (Meila & Jordan, 2000) such as Chow-Liu trees (Chow & Liu, 1968) and hidden Markov models (Rabiner & Juang, 1986).

Original: A decomposable PC can always be “smoothed” (i.e. transformed into a smooth and decomposable PC) in polynomial time with respect to its size (Darwiche, 2001;
Revised: A decomposable PC can always be “smoothed” (i.e. transformed into a smooth and decomposable PC) in polynomial time with respect to its size (Darwiche, 2001;

Original: Shih et al., 2019).
Revised: Shih et al., 2019).

Original: Hence, when we are trying to show that decomposable PCs can be transformed into equivalent PGCs in polynomial time, we can always assume without loss of generality that decomposable PCs are also smooth.
Revised: Hence, when we are trying to show that decomposable PCs can be transformed into equivalent PGCs in polynomial time, we can always assume without loss of generality that decomposable PCs are also smooth.

Original: Our ﬁrst observation is that the probability mass functions represented by smooth and decomposable PCs are very similar to the corresponding generating polynomials:
Revised: Our ﬁrst observation is that the probability mass functions represented by smooth and decomposable PCs are very similar to the corresponding generating polynomials:

Original: Proposition 1.
Revised: Proposition 1.

Original: Let A be a smooth and decomposable PC that represents the probability distribution Pr over random variables X 1 , . . .
Revised: Let A be a smooth and decomposable PC that represents the probability distribution Pr over random variables X 1 , . . .

Original: , X n .
Revised: , X n .

Original: Then A represents a probability mass polynomial of the form: 
Revised: Then A represents a probability mass polynomial of the form: m ( z [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: where α S =
Revised: where α S =

Original: Pr( { X i = 1 } i ∈ S , { X i = 0 } i/ ∈ S ) .
Revised: Pr( { X i = 1 } i ∈ S , { X i = 0 } i/ ∈ S ) .

Original: Note that Equation 4 is closely related to the network polynomials  deﬁned for Bayesian Networks (Darwiche, 2003).
Revised: Note that Equation 4 is closely related to the network polynomials (Darwiche, 2003) deﬁned for Bayesian Networks.

Original: By comparing Equation 4 to Equation 1 in the deﬁnition of generating circuits, we ﬁnd that they are basically the same, except for the absence of the negative literals
Revised: By comparing Equation 4 to Equation 1 in the deﬁnition of generating circuits, we ﬁnd that they look very similar, except for the absence of the negative random variables

Original: X i in Equation 1, which gives us the following corollary:
Revised: X i in Equation 1, which gives us the following corollary:

Original: Corollary 1.
Revised: Corollary 1.

Original: Let A be a smooth and decomposable PC.
Revised: Let A be a smooth and decomposable PC.

Original: By replacing all X i in A by 1 and X i by z i , we obtain a PGC that represents the same distribution.
Revised: By replacing all X i in A by 1 and X i by z i , we obtain a PGC that represents the same distribution.

Original: Corollary 1 establishes that PGCs subsume decomposable
Revised: Corollary 1 establishes that PGCs subsume decomposable

Original: PCs and in turn, the TPMs subsumed by decomposable PCs.
Revised: PCs and in turn, the TPMs subsumed by decomposable PCs.

Original: This raises the question of whether PGCs are strictly more expressive efﬁcient and we give a positive answer:
Revised: This raises the question of whether PGCs are strictly more expressive efﬁcient and we give a positive answer:

Original: Theorem 2.
Revised: Theorem 2.

Original: PGCs are strictly more expressive efﬁcient than decomposable PCs; that is, there exists a class of probability distributions that can be represented by polynomial-size
Revised: PGCs are strictly more expressive efﬁcient than decomposable PCs; that is, there exists a class of probability distributions that can be represented by polynomial-size

Original: PGCs but the size of any decomposable PCs that represent them is at least exponential.
Revised: PGCs but the sizes of any decomposable PCs that represent the distributions are at least exponential in the number of random variables.

Original: We take determinantal point processes (DPPs) as this separating class  and prove Theorem 2 by showing the following two results: (1) DPPs, in general, cannot betractably represented by  decomposable PCs; (2) DPPs are tractably subsumed by PGCs.
Revised: We take determinantal point processes (DPPs) as this separating class of distributions and prove Theorem 2 by showing the following two results: (1) DPPs, in general, cannot be represented by polynomial-size decomposable PCs, and (2) DPPs are tractably subsumed by PGCs.

Original: The ﬁrst result has already been proved ina previous works:
Revised: The ﬁrst result has already been proved in previous works:

Original: Theorem 3 (Zhang et al.
Revised: Theorem 3 (Zhang et al.

Original: ; Martens & Medabalimi (2014)) .
Revised: ; Martens & Medabalimi (2014)) .

Original: There exists a class of DPPs such that the the size of any decomposable PCs that represent them is exponential with respect to the number of random variables.
Revised: There exists a class of DPPs such that the the size of any decomposable PCs that represent them is exponential in the number of random variables.

Original: In the next section, we complete this proof by showing that any DPP can be represented by a PGC of polynomial size, with respect to the number of random variables.
Revised: In the next section, we complete this proof by showing that any DPP can be represented by a PGC of polynomial size in the number of random variables.

Original: 3.2. PGCs subsume Determinantal Point Processes
Revised: 3.2. PGCs subsume Determinantal Point Processes

Original: In this section, we show that determinantal point processes (DPPs) can be tractably represented by PGCs.
Revised: In this section, we focus on showing that determinantal point processes (DPPs) can be tractably represented by PGCs.

Original:  At a highlevel, a unique property of DPPs is that they are tractable representations of probability distributions that express global negative dependence , which makes them very useful in many applications (Mariet & Sra, 2016), such as document and video summarization (Chao et al., 2015; Lin & Bilmes, 2012), recommender systems (Zhou et al., 2010), and object retrieval (Affandi et al., 2014).
Revised: We start by introducing the basics for DPPs. At a high level, a unique property of DPPs is that they are tractable representations of probability distributions that express global negative dependence , which makes them very useful in many applications (Mariet & Sra, 2016), such as document and video summarization (Chao et al., 2015; Lin & Bilmes, 2012), recommender systems (Zhou et al., 2010), and object retrieval (Affandi et al., 2014).

Original: In machine learning, DPPs are most often represented by means of an L-ensemble (Borodin & Rains, 2005):
Revised: In machine learning, DPPs are most often represented by means of an L-ensemble (Borodin & Rains, 2005):

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: A probability distribution Pr over n binary random variables X
Revised: A probability distribution Pr over n binary random variables X

Original: =
Revised: =

Original: ( X 1 , . . .
Revised: ( X 1 , . . .

Original: , X n ) is an L-ensemble if there exists a (symmetric) positive semideﬁnite matrix
Revised: , X n ) is an L-ensemble if there exists a (symmetric) positive semideﬁnite matrix

Original: L ∈ R n × n such that for all x = ( x 1 , . .
Revised: L ∈ R n × n such that for all x = ( x 1 , . .

Original: , x n ) ∈ { 0 , 1 } n ,
Revised: , x n ) ∈ { 0 , 1 } n ,

Revised: [Equation]

Original: Pr( X = x ) ∝ det( L x ) , (5) where
Revised: X [Equation] det [Equation]

Revised: where

Original: L x =
Revised: L x =

Original: [ L ij ] x i =1 ,x j =1 denotes the submatrix of L indexed by those i, j where x i = 1 and x j = 1 .
Revised: [ L ij ] x i =1 ,x j =1 denotes the submatrix of L indexed by those i, j where x i = 1 and x j = 1 .

Original: The matrix
Revised: The matrix

Original: L is called the kernel for the L-ensemble.
Revised: L is called the kernel for the L-ensemble.

Original: To ensure that the distribution is properly normalized, it is necessary to divide
Revised: To ensure that the distribution is properly normalized, it is necessary to divide

Original: Equation 5 by det( L + I ) , where I is the n × n identity matrix (Kulesza & Taskar, 2012).
Revised: Equation 5 by det( L + I ) , where I is the n × n identity matrix (Kulesza & Taskar, 2012).

Original: Consider again the example distribution Pr β .
Revised: Consider again the example distribution Pr β .

Original: It is actually a DPP whose kernel is given by the matrix L β in Figure 1d.
Revised: It is actually a DPP whose kernel is given by the matrix L β in Figure 1d.

Original: The probability of the assignment X
Revised: The probability of the assignment X

Original: =
Revised: =

Original: (1 , 0 , 1) , for example, is given by
Revised: (1 , 0 , 1) , for example, is given by

Original: [Equation]
Revised: [Equation]

Original: To compute marginal probabilities for L-ensembles, we also need marginal kernels , which characterize DPPs in general, as an alternative to L-ensemble kernels.
Revised: To compute marginal probabilities for L-ensembles, we also need marginal kernels , which characterize DPPs in general, as an alternative to L-ensemble kernels.

Original: Deﬁnition 5.
Revised: Deﬁnition 5.

Original: A probability distribution Pr is a DPP over n binary random variables X 1 , . . .
Revised: A probability distribution Pr is a DPP over n binary random variables X 1 , . . .

Original: , X n if there exists a positive semdideﬁnite matrix K ∈ R n × n such that for all
Revised: , X n if there exists a positive semdideﬁnite matrix K ∈ R n × n such that for all

Original: A ⊆ { 1 , . . . , n }
Revised: A ⊆ { 1 , . [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where K A =
Revised: where K A =

Original: [ K ij ] i ∈ A,
Revised: [ K ij ] i ∈ A,

Original: j ∈
Revised: j ∈

Original: A denotes the submatrix of K indexed by elements in A .
Revised: A denotes the submatrix of K indexed by elements in A .

Original: The marginal kernel K β for the L-ensemble that represents the distribution Pr β is shown in Figure 1d, along with its kernel L β .
Revised: The marginal kernel K β for the L-ensemble that represents the distribution Pr β is shown in Figure 1d, along with its kernel L β .

Original: One can use a generalized version of Equation 6 to compute the marginal probabilities Pr(( X i =1)
Revised: One can use a generalized version of Equation 6 to compute the marginal probabilities Pr(( X i =1)

Original: i ∈
Revised: i ∈

Original: A , ( X j = 0) j ∈ B ) efﬁciently, where A, B ⊆ { 1 , . . . , n } .
Revised: A , ( X j = 0) j ∈ B ) efﬁciently, where A, B ⊆ { 1 , . . . , n } .

Original: We refer to
Revised: We refer to

Original: Kulesza & Taskar (2012) for further details.
Revised: Kulesza & Taskar (2012) for further details.

Original: PCs and DPPs support tractable marginals in strikingly different ways, and we wonder whether these two tractable languages can be captured by a uniﬁed framework.
Revised: PCs and DPPs support tractable marginals in strikingly different ways, and we wonder whether these two tractable languages can be captured by a uniﬁed framework.

Original: Previous works have shown that DPPs cannotbe tractably represented by various subclasses of PCs including SPNs (Martens &
Revised: As mentioned in Section 3.1, it has already been proved that PCs cannot tractably represent DPPs in general.

Original: Medabalimi, 2014; Zhang et al., 2020).

Original: Here, by showing that PGCs  tractably subsume DPPs, we provide a positive answer to this open problem.
Revised: We now show that PGCs also tractably subsume DPPs, providing a positive answer to this open problem.

Original: The key to constructing a PGC representation for DPPs is that their generating polynomials can be written as determinants over polynomial rings.
Revised: The key to constructing a PGC representation for DPPs is that their generating polynomials can be written as determinants over polynomial rings.

Original: Lemma 2 (Borcea et al.
Revised: Lemma 2 (Borcea et al.

Original: The generating polynomial for an L-ensemble with kernel L is given by
Revised: The generating polynomial for an L-ensemble with kernel L is given by

Original: [Equation]
Revised: [Equation]

Original: With Z = diag ( z 1 , . . .
Revised: With Z = diag ( z 1 , . . .

Original: , z n ) , the generating polynomial for a DPP with marginal kernel K is given by
Revised: , z n ) , the generating polynomial for a DPP with marginal kernel K is given by

Original: [Equation]
Revised: [Equation]

Original: Note that the generating polynomials presented in Lemmaare just mathematical objects; to use them as tractable models, we need to represent them in the framework of PGCs.
Revised: Note that the generating polynomials presented in Lemmaare just mathematical objects; to use them as tractable models, we need to represent them in the framework of PGCs.

Original: So let us examine Equations (7) and (8) in detail.
Revised: So let us examine Equations (7) and (8) in detail.

Original: The entries in the matrices LZ
Revised: The entries in the matrices LZ

Original: + I
Revised: + I

Original: and I − K
Revised: and I − K

Original: + KZ are degree-one univariate polynomials, which can be easily represented as
Revised: + KZ are degree-one univariate polynomials, which can be easily represented as

Original: PGCs.
Revised: PGCs.

Original: Thus, to compactly represent DPPs’ generating polynomials as PGCs, we only need to compactly represent the determinant function as a PGC.
Revised: Thus, to compactly represent DPPs’ generating polynomials as PGCs, we only need to compactly represent the determinant function as a PGC.

Original: There are a variety of polynomial-time division-free algorithms for computing determinants over rings (Bird, 2011;
Revised: There are a variety of polynomial-time division-free algorithms for computing determinants over rings (Bird, 2011;

Original: Samuelson, 1942; Berkowitz, 1984; Mahajan & Vinay, 1997) and we take Bird’s algorithm (Bird, 2011) as an example.
Revised: Samuelson, 1942; Berkowitz, 1984; Mahajan & Vinay, 1997) and we take Bird’s algorithm (Bird, 2011) as an example.

Original: Bird’s algorithm is simply an iteration of certain matrix multiplications and requires O ( nM ( n )) additions and multiplications, where M ( n ) is the number of basic operations needed for a matrix multiplication.
Revised: Bird’s algorithm is simply an iteration of certain matrix multiplications and requires O ( nM ( n )) additions and multiplications, where M ( n ) is the number of basic operations needed for a matrix multiplication.

Original: We conservatively assume that M ( n ) is upper-bounded by n 3 .
Revised: We conservatively assume that M ( n ) is upper-bounded by n 3 .

Original: Thus when we encode Bird’s algorithm as a PGC, the PGC contains at most
Revised: Thus when we encode Bird’s algorithm as a PGC, the PGC contains at most

Original: O ( n 4 ) sum and product nodes, each with a constant number of edges.
Revised: O ( n 4 ) sum and product nodes, each with a constant number of edges.

Original: Together with Lemma 2, it follows that DPPs are tractably subsumed by PGCs.
Revised: Together with Lemma 2, it follows that DPPs are tractably subsumed by PGCs.

Original: Theorem 4.
Revised: Theorem 4.

Original: Any DPP over n binary random variables can be represented by a PGC of size O ( n 4 ) .
Revised: Any DPP over n binary random variables can be represented by a PGC of size O ( n 4 ) .

Original: We conclude this section with the following remarks:
Revised: We conclude this section with the following remarks:

Revised: DPPs cannot represent any positive dependence; for example, Pr( X i = 1 , X j = 1) >

Revised: Pr( X i = 1) Pr( X j = 1) can never happen for a DPP.

Revised: On the other hand, since PGCs are fully general, they are strictly more expressive than DPPs.

Original: In practice, when representing DPPs in the language of PGCs, we do not need to explicitly construct the sum nodes and product nodes to form the circuit structures.
Revised: In practice, when representing DPPs in the language of PGCs, we do not need to explicitly construct the sum nodes and product nodes to form the circuit structures.

Original: Recall from Lemma 1 that marginals are tractable as long as we can efﬁciently evaluate the PGCs over polynomial rings.
Revised: Recall from Lemma 1 that marginals are tractable as long as we can efﬁciently evaluate the PGCs over polynomial rings.

Original: Thus we can simply apply Bird’s algorithm, for example, to compute the determinant.
Revised: Thus we can simply apply Bird’s algorithm, for example, to compute the determinants from Lemma 2.

Original: Since nonsymmetric DPPs (Gartrell et al., 2019) are deﬁned in the same way as standard DPPs, except that their kernel Ldoes not need to be symmetric, they are also tractably subsumed by PGCs.
Revised: Since nonsymmetric DPPs (Gartrell et al., 2019) are deﬁned in the same way as standard DPPs, except for their kernels L need not be symmetric, they are also tractably subsumed by PGCs.

Original: 4. Beyond PCs and DPPs
Revised: 4. Beyond PCs and DPPs

Original: In the previous section we have demonstrated the expressive power of PGCs by showing that they subsume decomposable PCs and DPPs.
Revised: In the previous section we have demonstrated the expressive power of PGCs by showing that they are strictly more expressive efﬁcient than decomposable PCs and DPPs.

Original: It is well-known, however, that PCs can use arbitrary families of tractable distributions at their leaves, including DPPs.
Revised: It is well-known, however, that PCs can use arbitrary families of tractable distributions at their leaves, including DPPs.

Original: Thus, our ﬁndings in the previous section did not rule out the possibility that PGCs could be represented by such a combination of PCs over standard TPMs such as DPPs. In this section,to show that PGCs contain classes of models that are strictly more expressive, we construct a simple class of PGCs that are not trivially subsumed by such simple combinations of PCs and DPPs.
Revised: In this section, we construct a simple class of PGCs that are more interesting than PCs with DPP leaves.

Original: 4.1. Basic Compositional Operations for PGCs
Revised: 4.1. Basic Compositional Operations for PGCs

Original: We start by deﬁning the sum , product and hierarchical composition operations for PGCs.
Revised: We start by deﬁning the sum , product and hierarchical composition operations for PGCs.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let A, B ⊂ N + ; denote { z i } i ∈
Revised: Let A, B ⊂ N + ; denote { z i } i ∈

Original: A by z A and { X i } i ∈
Revised: A by z A and { X i } i ∈

Original: A by X A .
Revised: A by X A .

Original: Let f ( z A ) and g ( z B ) be the generating Pr " Pr $
Revised: Let f ( z A ) and g ( z B ) be the generating polynomials for distributions

Original: Pr ! ! $" …! $# ! "
Revised: Pr f ( X A ) and Pr g ( X B ) , then,

Original: = det(' + ) diag - " , - # , … , - $ ) % " % $ % ! polynomials for distributions Pr f ( X A ) and Pr g ( X B ) , then, (Sum) : let α ∈ [0 , 1] , then αf + (1 − α ) g is the generating polynomial for the probability distribution Pr sum wherePr sum
Revised: Sum : let α ∈ [0 , 1] , then αf + (1 − α ) g is the generating polynomial for the probability distribution Pr sum where

Original: [Equation]
Revised: [Equation]

Original: X A = a , X B =

Original: b [Equation] (Product) : if A and B are disjoint (i.e. f and g depend on disjoint sets of variables), then fg is the generating polynomial for the probability distribution Pr prod where 
Revised: Product : if A and B are disjoint (i.e. f and g depend on disjoint sets of variables), then fg is the generating polynomial for the probability distribution Pr prod where Pr prod ( X A = a , X B = b ) =

Original: Pr 
Revised: Pr f

Original: [Equation]
Revised: ( X A = a )

Original:  The sum and product operations described above are basically the same as those for PCs: the sum distribution Pr sum is just a mixture over two distributions Pr f and Pr g , and the product distribution Pr prod is the point-wise product of
Revised: Pr g ( X B = b ) . The sum and product operations described above are basically the same as those for PCs: the sum distribution Pr sum is just a mixture over two distributions Pr f and Pr g , and the product distribution Pr prod is the point-wise product of

Original: Pr f and Pr g .
Revised: Pr f and Pr g .

Original: The hierarchical composition is much more interesting.
Revised: The hierarchical composition is much more interesting.

Original: Proposition 3 (hierarchical composition) .
Revised: Proposition 3 (hierarchical composition) .

Original: Let Pr g be a probability distribution with generating polynomial g ( z 1 , . . .
Revised: Let Pr g be a probability distribution with generating polynomial g ( z 1 , . . .

Original: , z n ) .
Revised: , z n ) .

Original: Let A 1 , . . .
Revised: Let A 1 , . . .

Original: , A n be disjoint subsets of N + and f 1 ( z A 1 ) , . . .
Revised: , A n be disjoint subsets of N + and f 1 ( z A 1 ) , . . .

Original: , f n ( z A n ) be generating polynomials for
Revised: , f n ( z A n ) be generating polynomials for

Original: Pr i .
Revised: Pr i .

Original: We deﬁne the hierarchical composition of g and f i s byg
Revised: We deﬁne the hierarchical composition of g and f i s by

Original: [Equation]
Revised: [Equation]

Original: g [Equation] = f i , which is the generating polynomial obtained by substituting z i in g by f i s. In particular, g comp is a well-deﬁned generating polynomial that represents a valid probability distribution.
Revised: which is the generating polynomial obtained by substituting z i in g by f i s. In particular, g comp is a well-deﬁned generating polynomial that represents a valid probability distribution.

Original: Unlike the sum and product operations, the hierarchical composition operation for PGCs does not have an immediate  analogue for PCs.
Revised: Unlike the sum and product operations, the hierarchical composition operation for PGCs does not have an immediate [Figure] analogue for PCs.

Original: This operation is a simple yet powerful way to construct classes of PGCs; Figure 2 shows an example, which we illustrate in detail in the next section.
Revised: This operation is a simple yet powerful way to construct classes of PGCs; Figure 2 shows an example, which we illustrate in detail in the next section.

Original: 4.2. A Separating Example
Revised: 4.2. A Separating Example

Original: Now we construct a separating class of PGCs that are not trivially subsumed bysimple combinations of PCs and DPPs.
Revised: Now we construct a simple class of PGCs that are not trivially subsumed by PCs with DPPs as leaves.

Original: Figure 2 gives an outline of its structure.
Revised: Figure 2 gives an outline of its structure.

Original: We construct a model of a probability distribution over n random variables X 1 , . . .
Revised: We construct a model of a probability distribution over n random variables X 1 , . . .

Original: , X n .
Revised: , X n .

Original: For simplicity we assume n = mk and partition the variables into m parts, each with k variables.
Revised: For simplicity we assume n = mk and partition the variables into m parts, each with k variables.

Original: Changing notation, we write { X 11 , . . .
Revised: Changing notation, we write { X 11 , . . .

Original: , X 1 k } , . .
Revised: , X 1 k } , . .

Original: , { X m 1 , . . .
Revised: , { X m 1 , . . .

Original: , X mk } .
Revised: , X mk } .

Original: For 1 ≤ i ≤ m , let Pr i be a PGC over the random variables
Revised: For 1 ≤ i ≤ m , let Pr i be a PGC over the random variables

Original: X i 1 , . . .
Revised: X i 1 , . . .

Original: , X ik with generating polynomial f i ( z i 1 , . . .
Revised: , X ik with generating polynomial f i ( z i 1 , . . .

Original: , z ik ) .
Revised: , z ik ) .

Original: Let Pr L be a DPP with kernel L and generating polynomial g L ( z 1 , . .
Revised: Let Pr L be a DPP with kernel L and generating polynomial g L ( z 1 , . .

Original: z m ) .
Revised: z m ) .

Original: Then, the generating polynomial g δ = g L (cid:12)(cid:12) z i = f
Revised: Then, the generating polynomial g δ = g L (cid:12)(cid:12) z i = f

Original: i , namely the hierarchical composition of g L and f i , deﬁnes a PGC Pr δ , which we refer to as a determinantal PGC (DetPGC).
Revised: i , namely the hierarchical composition of g L and f i , deﬁnes a PGC Pr δ , which we refer to as a determinantal PGC (DetPGC).

Original: DPPs are great at modeling negative dependencies but cannot represent positive dependencies between variables: for the DPP Pr L , Pr L ( X i
Revised: DPPs are great at modeling negative dependencies but cannot represent positive dependencies between variables: for the DPP Pr L , Pr L ( X i

Original: = 1 , X j = 1) ≥ Pr L ( X i =
Revised: = 1 , X j = 1) > Pr L ( X i =

Original: 1)Pr L ( X j = 1) can never happen.
Revised: 1)Pr L ( X j = 1) can never happen.

Original: Our construction of
Revised: Our construction of

Original: DetPGCs aims to equip a DPP model to capture local positive dependencies.
Revised: DetPGCs aims to equip a DPP model to capture local positive dependencies.

Original: To understand how DetPGCs actually behave, we compute the marginal probability Pr δ ( X ik = 1 , X jl = 1) by Lemma 1.
Revised: To understand how DetPGCs actually behave, we compute the marginal probability Pr δ ( X ik = 1 , X jl = 1) by Lemma 1.

Original: When X ik and X jl belong to the same group (i.e. i = j ):

Original: [Equation]

Original: ; that is, when two variables belong to the same group, the dependencies between them are dominated by the leaf PGC

Original: Pr i , giving space for positive dependencies.

Original: When X ik and X jl belong to different groups (i.e. i(cid:54) = j ):
Revised: When X ik and X jl belong to the same group (i.e. i = j ):

Original: Pr

Original: [Equation]

Original: ( X ik = 1 , X jl = 1) ≤ Pr

Original: [Equation]

Original: ( X ik = 1)Pr

Original: [Equation]

Original: ( X jl = 1); that is, random variables from different clusters are still negatively dependent, just like variables in the DPP Pr L .

Original: We stress that we construct DetPGCs merely to illustrate how the ﬂexibility of PGCs permits us to develop TPMs that capture structure that is beyond the reach of the standard suite of TPMs, including PCs, DPPs, and standard combinations thereof.

Original: We are not proposing DetPGCs as an “optimal”

Original: PGC structure for probabilistic modeling.

Original: Nevertheless, as we will see, even the simple DetPGC model may be a better model than PCs or DPPs for some kinds of real-world data.

Original: DPP
Revised: DPP

Original: Strudel
Revised: Strudel

Original: EiNet
Revised: EiNet

Original: SimplePGC nltcs −

Original: [Table]

Original: −

Original: [Table]

Original: [Table]

Original: [Table] msnbc [Table] −
Revised: MT

Original: [Table]

Original: −

Original: [Table]

Original: −

Original: [Table] † kdd − [Table] −
Revised: SimplePGC

Original: [Table]

Original: [Table]

Original: −

Original: [Table]

Original: plants

Original: [Table]

Original: [Table]

Original: [Table]

Original: audio

Original: [Table]
Revised: [Table]

Original: jester [Table] netﬂix [Table]
Revised: bbc

Original: accidents

Original: [Table]
Revised: [Table]

Original: retail

Original: [Table]
Revised: [Table]

Original: dna

Original: [Table]
Revised: [Table]

Original: kosarek

Original: [Table]

Original: msweb

Original: [Table]

Original: book

Original: [Table]

Original: movie [Table] [Table] webkb
Revised: ad

Original: [Table]
Revised: [Table]

Original: reuters

Original: [Table]

Original: 20ng [Table] bbc [Table]
Revised: −

Original: ad

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: (a Results

Original: DPP
Revised: DPP

Original: Strudel EiNet
Revised: Strudel EiNet

Original:  SimplePGCapparel
Revised: MT SimplePGC

Original: [Table]
Revised: [Table]

Original: bath
Revised: bath

Original: [Table]
Revised: [Table]

Original: bedding
Revised: toys

Original: [Table]
Revised: [Table]

Original: carseats

Original: [Table]
Revised: [Figure]

Original: diaper [Table] feeding
Revised: Results

Original: [Table]
Revised: [Equation]

Original: furniture [Table] gifts
Revised: that is, when two variables belong to the same group, the dependencies between them are dominated by Pr i , giving space for positive dependencies.

Original: [Table] health [Table]
Revised: When X ik and X jl belong to different groups (i.e. i (cid:54) =

Original: media
Revised: j ):

Original: [Table]
Revised: [Equation]

Original: safety
Revised: that is, random variables from different groups are still negatively dependent, just like variables in the DPP Pr L . We stress that we construct DetPGCs merely to illustrate how the ﬂexibility of PGCs permits us to develop TPMs that capture structure that is beyond the reach of the standard suite of TPMs, including PCs, DPPs, and standard combinations thereof. We are not proposing DetPGCs as an “optimal” PGC structure for probabilistic modeling.

Original: [Table]
Revised: Nevertheless, as we will see, even the simple DetPGC model may be a better model than PCs or DPPs for some kinds of real-world data.

Original: 5. Experiments
Revised: 5. Experiments

Original: This section evaluates PGCs’ ability to model real data on density estimation benchmarks.
Revised: This section evaluates PGCs’ ability to model real data on density estimation benchmarks.

Original: We use a weighted sum over
Revised: We use a weighted sum over

Original: DetPGCs as our model.
Revised: DetPGCs as our model.

Original: This simple method achieves stateof-the-art performance on half the benchmarks, illustrating the potential of PGCs in real-world applications.
Revised: This simple method achieves stateof-the-art performance on half the benchmarks, illustrating the potential of PGCs in real-world applications.

Original: 5.1. Datasets
Revised: 5.1. Datasets

Original: We evaluate PGCs on two density estimation benchmarks:
Revised: We evaluate PGCs on two density estimation benchmarks:

Original: Twenty Datasets (Van Haaren & Davis, 2012), which contains 20 real-world datasets ranging from retail to biology.
Revised: Twenty Datasets (Van Haaren & Davis, 2012), which contains 20 real-world datasets ranging from retail to biology.

Original: These datasets have been used to evaluate various tractable probabilistic models (Liang et al., 2017;
Revised: These datasets have been used to evaluate various tractable probabilistic models (Liang et al., 2017;

Original: Dang et al., 2020; Peharz et al., 2020).
Revised: Dang et al., 2020; Peharz et al., 2020).

Original: Amazon Baby Registries , which contains 15 datasets,each representing a collection of registries or “baskets” of baby products from a speciﬁc category such as “apparel” and “bath”. 
Revised: Amazon Baby Registries , which contains 15 datasets,each representing a collection of registries or “baskets” of baby products from a speciﬁc category such as “apparel” and “bath”. We randomly split each dataset into train

Original: We randomly split each dataset into train (70%), valid(10%) and test (20%) sets.
Revised: %), valid and test (20%) sets.

Original: This benchmark has been commonly used to evaluate DPP learners (Gillenwater et al., 2014; Mariet & Sra Gartrell 
Revised: This benchmark has been commonly used to evaluate DPP learners (Gillenwater et al., 2014; Mariet & Sra, 2015; Gartrell et al., 2019).

Original: et al

Original: 5.2. Model Structure
Revised: 5.2. Model Structure

Original: The model we use in our experiments is a weighted sum over
Revised: The model we use in our experiments is a weighted sum over

Original: DetPGCs, the example constructed in Section 4.2, which we refer to as SimplePGCs.
Revised: DetPGCs, the example constructed in Section 4.2, which we refer to as SimplePGCs.

Original: Recall from Figure 2 that a
Revised: Recall from Figure 2 that a

Original: DetPGC is the hierarchical composition of a DPP and some “leaf” PGCs Pr 1 , . . .
Revised: DetPGC is the hierarchical composition of a DPP and some “leaf” PGCs Pr 1 , . . .

Original: , Pr m .
Revised: , Pr m .

Original: For SimplePGC, we make the simplest choice by setting the Pr i s to be the fully general
Revised: For SimplePGC, we make the simplest choice by setting the Pr i s to be the fully general

Original: PGCs.
Revised: PGCs.

Original: In addition, to partition the input variables into m groups as shown in Figure 2, we use a simple greedy algorithm that aims at putting pairs of positively dependent variables into the same groups.
Revised: In addition, to partition the input variables into m groups as shown in Figure 2, we use a simple greedy algorithm that aims at putting pairs of positively dependent variables into the same groups.

Original: The structure of a
Revised: The structure of a

Original: SimplePGC is also governed by two hyperparameters: the number of DetPGCs in the weighted sum (denoted by C ) and the maximum number of variables (i.e. k in Figure 2) allowed in each group (denoted by K ).
Revised: SimplePGC is also governed by two hyperparameters: the number of DetPGCs in the weighted sum (denoted by C ) and the maximum number of variables (i.e. k in Figure 2) allowed in each group (denoted by K ).

Original: We tune C and K by a grid search over the following ranges: K ∈ { 1 , 2 , 5 , 7 } and C ∈ { 1 , 4 , 7 , 10 , 20 } .
Revised: We tune C and K by a grid search over the following ranges: K ∈ { 1 , 2 , 5 , 7 } and C ∈ { 1 , 4 , 7 , 10 , 20 } .

Original: Note that our model reduces to a mixture over DPPs when K = 1 .
Revised: Note that our model reduces to a mixture over DPPs when K = 1 .

Original: We implement SimplePGC in PyTorch and learn the parameters by maximum likelihood estimation (MLE).
Revised: We implement SimplePGC in PyTorch and learn the parameters by maximum likelihood estimation (MLE).

Original: In particular, we use Adam (Kingma & Ba, 2014) as the optimizing algorithm to minimize the negative log likelihoods of the training sets.
Revised: In particular, we use Adam (Kingma & Ba, 2014) as the optimizing algorithm to minimize the negative log likelihoods given the training sets.

Original: Regularization is done by setting the weight decay parameter in Adam.
Revised: Regularization is done by setting the weight decay parameter in Adam.

Original: For further details regarding the construction and implementation of SimplePGC, please see the Appendix.
Revised: For further details regarding the construction and implementation of SimplePGCs, please see the Appendix.

Original: 5.3. Baselines
Revised: 5.3. Baselines

Original: We compare SimplePGC against four baselines: DPPs,
Revised: We compare SimplePGC against four baselines: DPPs,

Original: Strudel,  Mixture of Trees and Einsum Networks.
Revised: Strudel, Einsum Networks and Mixture of Trees.

Original: DPP : As mentioned, SimplePGC reduces to a mixture over
Revised: DPP : As mentioned, SimplePGC reduces to a mixture over

Original: DPPs when the hyperparameter K = 1 .
Revised: DPPs when the hyperparameter K = 1 .

Original: DPPs are included as a sanity check; in particular, we expect PGCs to beat DPPs on most datasets and be at least as good on all datasets.
Revised: DPPs are learned via SGD. We expect PGCs to outperform DPPs on most datasets and be at least as good on all datasets.

Original: Strudel : Strudel (Dang et al., 2020) is an algorithm for learning  structured-decomposable PCs.
Revised: Strudel : Strudel (Dang et al., 2020) is an algorithm for learning the circuit structure of structured-decomposable PCs.

Original: We include them as one of the state-of-the-art  density estimators.
Revised: We include them as one of the state-of-the-art tractable density estimators.

Original: Einsum Networks : Einsum Networks (Peharz et al., 2020) (EiNets) is a deep-learning style implementation design for PCs.
Revised: Einsum Networks : Einsum Networks (Peharz et al., 2020) (EiNets) are a deep-learning-style implementation design for PCs.

Original: Compared to Strudel, EiNets are more related to
Revised: Compared to Strudel, EiNets are more related to

Original: SimplePGC in the sense that they are both ﬁxed-structure models where only parameters are learned.
Revised: SimplePGC in the sense that they are both ﬁxed-structure models where only parameters are learned.

Original: 5.4. Results and Analysis
Revised: The hyperparameters are chosen by a grid search as suggested by Peharz et al. Mixture of Trees The Mixture of Trees (Meila & Jordan, 2000) (MT) model is a mixture model over Chow-Liu trees (Chow & Liu, 1968). MTs are included as a representative of tractable graphical models with simple yet expressive structures.

Revised: For learning, we run the mtlearn algorithm implemented in the Libra-tk library (Lowd &

Revised: Rooshenas, 2015); the number of components in MT is chosen by a grid search from 2 to 30 with step size 2, as suggested by Rooshenas & Lowd (2014).

Revised: 5.4. Results and Analysis

Original: Figure 3 shows the experiment results.
Revised: Figure 3 shows the experiment results.

Original: As a sanity check, we ﬁrst compare SimplePGC against DPPs.
Revised: We ﬁrst compare SimplePGC against DPPs.

Original: On both benchmarks, SimplePGC performs signiﬁcantly better than DPPs on almost every dataset except for 4 datasets from the Amazon Baby Registries benchmark, where SimplePGC performs at least as well as DPPs.
Revised: On both benchmarks, SimplePGC performs signiﬁcantly better than DPPs on almost every dataset except for 4 datasets from the Amazon Baby Registries benchmark, where SimplePGC performs at least as well as DPPs.

Original: Overall, SimplePGC achieves competitive performance when compared against Strudel and EiNet  on both benchmarks.
Revised: Overall, SimplePGC achieves competitive performance when compared against Strudel, EiNet and MT on both benchmarks.

Original: On the Twenty Datasets benchmark, SimplePGC obtains better average log-likelihood than either Strudel or EiNet  on 18 out of the 20 datasets and, in particular, SimplePGC obtains higher log-likelihood than both of them on 3 datasets.
Revised: On the Twenty Datasets benchmark, SimplePGC obtains better average log-likelihood than at least one of the baselines (Strudel, EiNet and MT) on 19 out of the 20 datasets and, in particular, SimplePGC obtains higher log-likelihood than all of them on 2 datasets.

Original: Such results are remarkable, given the fact that SimplePGC is just  simple hand-crafted PGC architecture with little ﬁne-tuning, while Strudel and EiNets follow from a long line of research aiming to perform well on exactly the Twenty Datasets benchmark.
Revised: Such results are remarkable, given the fact that SimplePGC is just a simple hand-crafted PGC architecture with little ﬁnetuning, while Strudel, EiNet and MT follow from a long line of research aiming to perform well on exactly the Twenty Datasets benchmark.

Original: The performance of SimplePGC on the Amazon Baby Registries benchmark is even more impressive: SimplePGC beats both Strudel and EiNet on 11 out of 15 datasets and beats either one of them on 13 datasets.
Revised: The performance of SimplePGC on the Amazon Baby Registries benchmark is even more impressive: SimplePGC beats all of baselines on 11 out of 15 datasets and beats at least one of them on all datasets.

Revised: One possible reason that

Revised: SimplePGC performs much better than the other baselines on this benchmark is because these datasets exhibit relatively strong negative dependence and SimplePGCs’ DPPlike structure allows them to capture negative dependence well.

Original: We also conducted one-sample t-test for the results; for further details please refer to the Appendix.
Revised: We also conducted one-sample t-test for the results; for further details please refer to the Appendix.

Original: 6. PGCs and Strongly Rayleigh Distributions
Revised: 6. PGCs and Strongly Rayleigh Distributions

Original: At a high level, the study of PCs and graphical models mainly focuses on constructing classes of models that guarantee tractable exact inference.
Revised: At a high level, the study of PCs and graphical models mainly focuses on constructing classes of models that guarantee tractable exact inference.

Original: A separate line of research in probabilistic machine learning, however, aims at identifying classes of distributions that support tractable sampling, where generating polynomials play an essential role.
Revised: A separate line of research in probabilistic machine learning, however, aims at identifying classes of distributions that support tractable sampling, where generating polynomials play an essential role.

Original: For example, a well-studied class of distributions are the strongly
Revised: For example, a well-studied class of distributions are the strongly

Original: Rayleigh (SR) distributions (Borcea et al., 2009; Li et al., 2016), which were ﬁrst deﬁned in the ﬁeld of probability theory for studying negative dependence:
Revised: Rayleigh (SR) distributions (Borcea et al., 2009; Li et al., 2016), which were ﬁrst deﬁned in the ﬁeld of probability theory for studying negative dependence:

Original: Deﬁnition 6.
Revised: Deﬁnition 6.

Original: A polynomial f ∈ R [ z 1 , . . .
Revised: A polynomial f ∈ R [ z 1 , . . .

Original: , z n ] is real stable if whenever  Im ( z i ) > 0 for 1 ≤ i ≤ n , f ( z 1 , . . .
Revised: , z n ] is real stable if whenever the imaginary part Im ( z i ) > 0 for 1 ≤ i ≤ n , f ( z 1 , . . .

Original: , z n ) (cid:54)
Revised: , z n ) (cid:54)

Original: = 0 .
Revised: = 0 .

Original: We say that a distribution over X 1 , . . .
Revised: We say that a distribution over X 1 , . . .

Original: , X n is strongly Rayleigh (SR) if its generating polynomial is real stable.
Revised: , X n is strongly Rayleigh (SR) if its generating polynomial is real stable.

Original: SR distributions contain many important subclasses such as DPPs and the spanning tree (forest) distributions, which have various applications.
Revised: SR distributions contain many important subclasses such as DPPs and the spanning tree/forest distributions, which have various applications.

Original: From Section 3.2, we already know that PGCs can compactly represent DPPs.
Revised: From Section 3.2, we already know that PGCs can compactly represent DPPs.

Original: We now show that PGCs can represent spanning tree  distributions.
Revised: We now show that PGCs can represent spanning tree distributions in polynomial-size. We ﬁrst deﬁne the spanning tree distributions.

Original: Let G =
Revised: Let G =

Original: ( V, E ) be a connected graph with vertex set V = { 1 , . .
Revised: ( V, E ) be a connected graph with vertex set V = { 1 , . .

Original: . , n } and edge set E .
Revised: . , n } and edge set E .

Original: Associate to each edge e ∈ E a variable z e and a weight w e ∈ R ≥ 0 .
Revised: Associate to each edge e ∈ E a variable z e and a weight w e ∈ R ≥ 0 .

Original: If e = { i, j } , let A e be the n × n matrix where A ii =
Revised: If e = { i, j } , let A e be the n × n matrix where A ii =

Original: A jj = 1 , A ij =
Revised: A jj = 1 , A ij =

Original: A ji = − 1 and all other entries equal to 0 .
Revised: A ji = − 1 and all other entries equal to 0 .

Original: Then the weighted Laplacian of G is given by L ( G )
Revised: Then the weighted Laplacian of G is given by L ( G )

Original: = (cid:80) e ∈ E w e z e A e , and by the Principal Minors Matrix-Tree Theorem (Chaiken &
Revised: = (cid:80) e ∈ E w e z e A e , By the Principal Minors Matrix-Tree Theorem (Chaiken &

Original: Kleitman, 1978), 
Revised: Kleitman, 1978), f G

Original: [Equation]
Revised: [Equation]

Revised: det

Revised: [Equation]

Revised: [Equation]

Original:  is the (un-normalized) generating polynomial for the distribution (denote it by Pr G supported on the spanning trees of
Revised: (cid:88) T [Figure] spanning tree of G w edges ( T ) z edges ( T ) is the (un-normalized) generating polynomial for the spanning tree distribution, and we denote it by Pr G .

Original: G ; here L ( G ) \{ i } denotes the principal minor of L ( G ) by removing its i th row and column.
Revised: Here L ( G ) \{ i } denotes the principal minor of L ( G ) by removing its i th row and column.

Original: As presented in the equation above,  the probability of each spanning tree is proportional to the product of its edge weights.
Revised: As shown in the equation above, Pr G is supported on the spanning trees of G , and the probability of each spanning tree is proportional to the product of its edge weights.

Original: Pr G is a strongly Rayleigh distribution (Borcea et al., 2009), and, to the best of our knowledge, it is not a DPP unless the edge weights are the same.
Revised: Pr G is a strongly Rayleigh distribution (Borcea et al., 2009), and, to the best of our knowledge, it is not a DPP unless the edge weights are the same.

Original: By the same argument as in Section 3.2, we claim that Pr G can be tractably represented by PGCs.
Revised: By the same argument as in Section 3.2, we claim that Pr G can be tractably represented by PGCs.

Original: Thus, we see that there is another natural class of SR distributions – spanning tree distributions – that can be represented by PGCs.
Revised: Thus, we see that there is another natural class of SR distributions – spanning tree distributions – that can be represented by PGCs.

Original: More generally, generating polynomials play a key role in the study of a number of other classes of distributions, include the Ising model (Jerrum & Sinclair, 1993), exponentiated strongly Rayleigh (ESR) distributions (Mariet et al., 2018) and strongly log-concave (SLC) distributions (Robinson et al., 2019).
Revised: More generally, generating polynomials play a key role in the study of a number of other classes of distributions, including the Ising model (Jerrum & Sinclair, 1993), exponentiated strongly Rayleigh (ESR) distributions (Mariet et al., 2018) and strongly log-concave (SLC) distributions (Robinson et al., 2019).

Original: Speciﬁcally, most of these distributions are naturally characterized by their generating polynomials rather than probability mass functions.
Revised: Speciﬁcally, most of these distributions are naturally characterized by their generating polynomials rather than probability mass functions.

Original: This poses a major barrier to linking them to other probabilistic models.
Revised: This poses a major barrier to linking them to other probabilistic models.

Original: Thus by showing that PGCs can tractably represent certain subclasses of SR distributions, we present PGCs as a prospective avenue for bridging this gap.
Revised: Thus by showing that PGCs can tractably represent certain subclasses of SR distributions, we present PGCs as a prospective avenue for bridging this gap.

Original: Although we conjecture that not all SR distributions can be represented by polynomial-size PGCs, we believe that the subclasses of the above distributions that have concise parameterizations should be representable by PGCs.
Revised: Although we conjecture that not all SR distributions can be represented by polynomial-size PGCs, we believe that the subclasses of the above distributions that have concise parameterizations should be representable by PGCs.

Original: Establishing this for various families is a direction for future work.
Revised: Establishing this for various families is a direction for future work.

Original: 7. Conclusion and Perspectives
Revised: 7. Conclusion and Perspectives

Original: We conclude by summarizing our contributions and highlighting future research directions.
Revised: We conclude by summarizing our contributions and highlighting future research directions.

Original: In this paper, we study the use of probability generating polynomials as a data structure for representing probability distributions.
Revised: In this paper, we study the use of probability generating polynomials as a data structure for representing probability distributions.

Original: We showed that their representation as circuits are a TPM, and  strictly subsume existing families of TPMs.
Revised: We showed that their representation as circuits are a TPM, and are strictly more expressive efﬁcient than existing families of TPMs.

Original: Indeed, even a simple example family of distributions that can be represented by PGCs but not PCs or DPPs obtains state-of-the-art performance as a probabilistic model on some datasets.
Revised: Indeed, even a simple example family of distributions that can be represented by PGCs but not PCs or DPPs obtains state-of-the-art performance as a probabilistic model on some datasets.

Original: To facilitate the general use of PGCs for probabilistic modeling a facinating direction for future work is to build efﬁcient structure learning or ‘architecture search’ algorithms for
Revised: To facilitate the general use of PGCs for probabilistic modeling, a facinating direction for future work is to build efﬁcient structure learning or ‘architecture search’ algorithms for

Original: PGCs.
Revised: PGCs.

Original: Theoretically, the main mathematical advantage of generating polynomials was the variety of properties they reveal about a distribution.
Revised: Theoretically, the main mathematical advantage of generating polynomials was the variety of properties they reveal about a distribution.

Original: This raises the question of whether there are other kinds of useful queries we can support efﬁciently with PGCs and where truly lies the boundary of tractable probabilistic inference. 
Revised: This raises the question of whether there are other kinds of useful queries we can support efﬁciently with PGCs, and where truly lies the boundary of tractable probabilistic inference. Acknowledgements

Revised: We thank the reviewers for their detailed and thoughtful feedback and efforts towards improving this paper.

Revised: We thank

Revised: Steven Holtzen, Antonio Vergari and Zhe Zeng for helpful feedback and discussion.

Revised: This work is partially supported by

Original:  A. Proofs
Revised: NSF grants #IIS-1943641, #IIS-1633857, #CCF-1837129, #CCF-1718380, #IIS-1908287, and #IIS-1939677, DARPA XAI grant #N66001-17-2-4032, Sloan and UCLA Samueli Fellowships, and gifts from Intel and Facebook Research. A. Proofs

Original: Proof for Lemma 1.
Revised: Proof for Lemma 1.

Original: We write the generating polynomial for Pr as g ( z 1 , . . .
Revised: We write the generating polynomial for Pr as g ( z 1 , . . .

Original: , z n ) = (cid:80) S ⊂ [ n ] α S z S ; then,
Revised: , z n ) = (cid:80) S ⊂ [ n ] α S z S ; then,

Original: Pr( { X i = 1 } i ∈
Revised: Pr( { X i = 1 } i ∈

Original: A , { X i = 0 } i ∈ B ))
Revised: A , { X i = 0 } i ∈ B ))

Original: = (cid:88)
Revised: = (cid:88)

Original: A ⊂ S,B ∩ S = ∅ α S
Revised: A ⊂ S,B ∩ S = ∅ α S

Original: Besides, we also have
Revised: Besides, we also have

Original: [Equation]
Revised: [Equation]

Original: given the assignment asgn :
Revised: given the assignment asgn :

Original: = {{ z i = t } i ∈
Revised: = {{ z i = t } i ∈

Original: A , { z i = 0 } i ∈ B , { z i = 1 }
Revised: A , { z i = 0 } i ∈ B , { z i = 1 }

Original: i/ ∈
Revised: i/ ∈

Original: A ∪ B } ; that is, to prove the Proposition, we only need to show that coef | A | ( z S (cid:12)(cid:12) asgn )
Revised: A ∪ B } ; that is, to prove the Proposition, we only need to show that coef | A | ( z S (cid:12)(cid:12) asgn )

Original: = 1 for S ⊂ [ n ] where A ⊂ S and B ∩ S = ∅ ; and coef | A | ( z S (cid:12)(cid:12) asgn )
Revised: = 1 for S ⊂ [ n ] where A ⊂ S and B ∩ S = ∅ ; and coef | A | ( z S (cid:12)(cid:12) asgn )

Original: = 

Revised: =

Original: otherwise.
Revised: otherwise.

Original: Case 1.
Revised: Case 1.

Original: Assume A ⊂ S and B ∩ S = ∅ ; then z S (cid:12)(cid:12) asgn
Revised: Assume A ⊂ S and B ∩ S = ∅ ; then z S (cid:12)(cid:12) asgn

Original: =
Revised: =

Original: t | A | ; hence coef | A | ( z S (cid:12)(cid:12) asgn ) = 1 .
Revised: t | A | ; hence coef | A | ( z S (cid:12)(cid:12) asgn ) = 1 .

Original: Case 2.
Revised: Case 2.

Original: Assume A (cid:54)⊂ S or B ∩ S (cid:54) = ∅ ; if B ∩ S (cid:54) = ∅ , then z S (cid:12)(cid:12) asgn = 0 ; done.
Revised: Assume A (cid:54)⊂ S or B ∩ S (cid:54) = ∅ ; if B ∩ S (cid:54) = ∅ , then z S (cid:12)(cid:12) asgn = 0 ; done.

Original: Now we assume A (cid:54)⊂ S .
Revised: Now we assume A (cid:54)⊂ S .

Original: In this case, z S (cid:12)(cid:12) asgn =
Revised: In this case, z S (cid:12)(cid:12) asgn =

Original: t | S ∩ A | .
Revised: t | S ∩ A | .

Original: It follows from S ∩ A (cid:40)
Revised: It follows from S ∩ A (cid:40)

Original: A that | S ∩
Revised: A that | S ∩

Original: A | < | A | , which implies that coef | A | ( z S (cid:12)(cid:12) asgn ) = 0 .
Revised: A | < | A | , which implies that coef | A | ( z S (cid:12)(cid:12) asgn ) = 0 .

Original: Proof for Proposition 1.
Revised: Proof for Proposition 1.

Original: Let A be a decomposable and smooth PC.
Revised: Let A be a decomposable and smooth PC.

Original: Without loss of generality we assume A is normalized.
Revised: Without loss of generality we assume A is normalized.

Original: For each node u in A , we deﬁne I u
Revised: For each node u in A , we deﬁne I u

Original: = { i : X i or X
Revised: = { i : X i or X

Original: i ∈ scope ( u ) } and denote the polynomial that u represents by m u .
Revised: i ∈ scope ( u ) } and denote the polynomial that u represents by m u .

Original: We ﬁrst prove the following intermediate result by a bottom-up induction on A :
Revised: We ﬁrst prove the following intermediate result by a bottom-up induction on A :

Original: [Equation]
Revised: [Equation]

Original: where the α S are some non-negative numbers depending on the node u .
Revised: where the α S are some non-negative numbers depending on the node u .

Original: Case 1.
Revised: Case 1.

Original: If u is a leaf node X i or X i , then m u is X i or X i ; done.
Revised: If u is a leaf node X i or X i , then m u is X i or X i ; done.

Original: Case 2.
Revised: Case 2.

Original: If u is a sum node with children { v i } 1 ≤ i ≤ k and weights { w i } 1 ≤ i ≤ k . m u =
Revised: If u is a sum node with children { v i } 1 ≤ i ≤ k and weights { w i } 1 ≤ i ≤ k . m u =

Original: [Equation]
Revised: [Equation]

Original: 1 ≤ i ≤ k w i m v i .
Revised: 1 ≤ i ≤ k w i m v i .

Original: By smoothness,
Revised: By smoothness,

Original: I v i
Revised: I v i

Original: = I u for all i .
Revised: = I u for all i .

Original: Then, by the induction hypothesis:
Revised: Then, by the induction hypothesis:

Original: [Equation]
Revised: [Equation]

Original: Case 3.
Revised: Case 3.

Original: If u is a product node with children { v i } 1 ≤ i ≤ k .
Revised: If u is a product node with children { v i } 1 ≤ i ≤ k .

Original: Then, by decomposability, I v 1 , . .
Revised: Then, by decomposability, I v 1 , . .

Original: , I v k are pairwise disjoint; in particular, for each S ⊂ I u , S can be uniquely decomposed into S 1 ⊂
Revised: , I v k are pairwise disjoint; in particular, for each S ⊂ I u , S can be uniquely decomposed into S 1 ⊂

Original: I v 1 , . . .
Revised: I v 1 , . . .

Original: , S k ⊂ I v k .
Revised: , S k ⊂ I v k .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: ≤ i ≤ k (cid:88)
Revised: ≤ i ≤ k (cid:88)

Original: S i ⊂ I vi α iS (cid:89) j ∈ S
Revised: S i ⊂ I vi α iS (cid:89) j ∈ S

Original: i
Revised: i

Original: X j (cid:89) j/ ∈ S
Revised: X j (cid:89) j/ ∈ S

Original: i
Revised: i

Original: X j = (cid:88)
Revised: X j = (cid:88)

Original: S 1 ⊂
Revised: S 1 ⊂

Original: I v 1 ,...,S k ⊂ I vk   (cid:89) 1 ≤ i ≤ k
Revised: I v 1 ,...,S k ⊂ I vk   (cid:89) 1 ≤ i ≤ k

Original: α iS i 
Revised: α iS i 

Original:  (cid:89) 1 ≤ i ≤ k   (cid:89)
Revised:  (cid:89) 1 ≤ i ≤ k   (cid:89)

Original: j ∈ S
Revised: j ∈ S

Original: i
Revised: i

Original: X j (cid:89) j/ ∈ S
Revised: X j (cid:89) j/ ∈ S

Original: i
Revised: i

Original: X j 
Revised: X j 

Original:  =
Revised:  =

Original: (cid:88)
Revised: (cid:88)

Original: S ⊂
Revised: S ⊂

Original: I u   (cid:89) 1 ≤ i ≤ k α iS i 
Revised: I u   (cid:89) 1 ≤ i ≤ k α iS i 

Original:  (cid:89) j ∈ S
Revised:  (cid:89) j ∈ S

Original: X j (cid:89) j/ ∈ S
Revised: X j (cid:89) j/ ∈ S

Original: X j ;
Revised: X j ;

Original: with
Revised: with

Original: I u
Revised: I u

Original: = I v 1 ∪ · · · ∪ I v
Revised: = I v 1 ∪ · · · ∪ I v

Original: n a disjoint
Revised: n a disjoint

Original: union.
Revised: union.

Original: Hence the mass polynomial represented by A is given by 3 :
Revised: Hence the mass polynomial represented by A is given by:

Original: [Equation]
Revised: [Equation]

Original: By plugging in { X i = 1 } i ∈ S , { X i = 0 } i/ ∈ S , it immediately follows that α S =
Revised: By plugging in { X i = 1 } i ∈ S , { X i = 0 } i/ ∈ S , it immediately follows that α S =

Original: Pr( { X i = 1 } i ∈ S , { X i = 0 } i/ ∈ S ) .
Revised: Pr( { X i = 1 } i ∈ S , { X i = 0 } i/ ∈ S ) .

Original: Proof for Proposition 2.
Revised: Proof for Proposition 2.

Original: Let A, B ⊂ N + ; let f = (cid:80) S ⊂
Revised: Let A, B ⊂ N + ; let f = (cid:80) S ⊂

Original: A β S z S and g = (cid:80) S ⊂ B γ S z S be the normalized probability generating polynomials for distributions Pr f ( X A ) and Pr g ( X B ) , respectively.
Revised: A β S z S and g = (cid:80) S ⊂ B γ S z S be the normalized probability generating polynomials for distributions Pr f ( X A ) and Pr g ( X B ) , respectively.

Original: Case 1 ( Sum ).
Revised: Case 1 ( Sum ).

Original: First, we view f and g as polynomials over { z i } i ∈
Revised: First, we view f and g as polynomials over { z i } i ∈

Original: A ∪ B by setting β S = 0 ∀ S (cid:54)⊂ A , and γ S = 0 ∀ S (cid:54)⊂ B , which is equivalent to, from the perspective of probability distributions, viewing Pr f and Pr g as distributions over X A ∪ B such that
Revised: A ∪ B by setting β S = 0 ∀ S (cid:54)⊂ A , and γ S = 0 ∀ S (cid:54)⊂ B , which is equivalent to, from the perspective of probability distributions, viewing Pr f and Pr g as distributions over X A ∪ B such that

Original: [Equation]
Revised: [Equation]

Original: ( X A = a ) , if b i = 0 for i ∈
Revised: ) , if b [Equation] 0 for i ∈

Revised: [Equation]

Original: ( A ∪ B ) \ A 0 , otherwise and
Revised: A ∪ B ) \ A 0 , otherwise and

Original: Pr g ( X A = a , X B = b ) = (cid:40)
Revised: [Equation]

Original: Pr g ( X B = b ) , if a i = 0 for i ∈ 
Revised: ) , if a i [Equation] 0 for i ∈ [Equation]

Original: ( A ∪ B ) \ B 0 , otherwise
Revised: A ∪ B ) \ B 0 , otherwise

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where αβ S +(1 − α ) γ S are clearly non-negative, and (cid:80) S
Revised: where αβ S +(1 − α ) γ S are clearly non-negative, and (cid:80) S

Original: ⊂
Revised: ⊂

Original: A ∪ B αβ S +(1 − α ) γ S = α (cid:80) S 
Revised: A ∪ B αβ S +(1 − α ) γ S = α (cid:80) S ⊂ A ∪ B β S

Original: [Equation]

Original: β S +(1 − α ) (cid:80) S ⊂
Revised: +(1 − α ) (cid:80) S ⊂

Original: A ∪ B γ S =
Revised: A ∪ B γ S =

Original: α + (1 − α )
Revised: α + (1 − α )

Original: = 1 .
Revised: = 1 .

Original: That is, αf + (1 − α ) g is a valid probability generating polynomial for a distribution, Pr sum .
Revised: That is, αf + (1 − α ) g is a valid probability generating polynomial for a distribution, Pr sum .

Original: For assignments X A = a and X B = b with no conﬂict ( A and B are not necessarily disjoint), let S =
Revised: For assignments X A = a and X B = b with no conﬂict ( A and B are not necessarily disjoint), let S =

Original: { i ∈ A : a i = 1 } ∪ { i ∈ B : b i = 1 } .
Revised: { i ∈ A : a i = 1 } ∪ { i ∈ B : b i = 1 } .

Original: By deﬁnition, Pr sum ( X A = a , X B = b ) is given by the coefﬁcient of the term z S , which is
Revised: By deﬁnition, Pr sum ( X A = a , X B = b ) is given by the coefﬁcient of the term z S , which is

Original: [Equation]
Revised: [Equation]

Original:  for short.
Revised: ) for short.

Original: Case 2 ( Product ).
Revised: Case 2 ( Product ).

Original: We assume A ∩ B = ∅ .
Revised: We assume A ∩ B = ∅ .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: As A and B are disjoint, z S z T are multiafﬁne.
Revised: As A and B are disjoint, z S z T are multiafﬁne.

Original: On top of that, β S γ T ≥ 0 and (cid:80) S ⊂ A,T ⊂ B β S γ T = (cid:80) S ⊂
Revised: On top of that, β S γ T ≥ 0 and (cid:80) S ⊂ A,T ⊂ B β S γ T = (cid:80) S ⊂

Original: A β S (cid:80) T ⊂ B γ T = 1 .
Revised: A β S (cid:80) T ⊂ B γ T = 1 .

Original: Thus, fg is a valid probability generating polynomial for a distribution, Pr prod .
Revised: Thus, fg is a valid probability generating polynomial for a distribution, Pr prod .

Original: For assignments X A = a , X B = b , we set S a = { i ∈
Revised: For assignments X A = a , X B = b , we set S a = { i ∈

Original: A : a i = 1 } and S b = { i ∈ B : b i = 1 } .
Revised: A : a i = 1 } and S b = { i ∈ B : b i = 1 } .

Original: Then, by deﬁnition,
Revised: Then, by deﬁnition,

Original: Pr prod ( X A = a , X B = b ) is given by the coefﬁcient of the term z S a ∪ S b , which is β S a γ S b =
Revised: Pr prod ( X A = a , X B = b ) is given by the coefﬁcient of the term z S a ∪ S b , which is β S a γ S b =

Original: Pr f
Revised: Pr f

Original: ( X A = a )
Revised: ( X A = a )

Original: Pr f
Revised: Pr f

Original: ( X B = b ) .
Revised: ( X B = b ) .

Original: Proof for Proposition 3.
Revised: Proof for Proposition 3.

Original: Let g ( z 1 , . . .
Revised: Let g ( z 1 , . . .

Original: , z n ) be a normalized probability generating polynomial.
Revised: , z n ) be a normalized probability generating polynomial.

Original: Let A 1 , . . .
Revised: Let A 1 , . . .

Original: , A n be disjoint subsets of N + and f 1 ( z A 1 ) , . . .
Revised: , A n be disjoint subsets of N + and f 1 ( z A 1 ) , . . .

Original: , f n ( z A n ) be normalized generating polynomials.
Revised: , f n ( z A n ) be normalized generating polynomials.

Original: Write g
Revised: Write g

Original: =
Revised: =

Original: (cid:80) S
Revised: (cid:80) S

Original: ⊂{ 1 ,...,n } α S z S .
Revised: ⊂{ 1 ,...,n } α S z S .

Original: Then,g (cid:12)(cid:12) z i = f i = (cid:88) S ⊂{ 1 ,...,n } α S (cid:89) i ∈
Revised: Then,

Original: S f i
Revised: [Equation]

Original: It follows from Proposition 2 (product operation) that (cid:81) i ∈
Revised: It follows from Proposition 2 (product operation) that (cid:81) i ∈

Original: S
Revised: S

Original: f i are valid generating polynomials for S ⊂ { 1 , . . . , n } ; again by Proposition 2 (sum operation), g (cid:12)(cid:12) z i = f i
Revised: f i are valid generating polynomials for S ⊂ { 1 , . . . , n } ; again by Proposition 2 (sum operation), g (cid:12)(cid:12) z i = f i

Original: = (cid:80) S
Revised: = (cid:80) S

Original: ⊂{ 1 ,...,n } α S (cid:81) i ∈ S
Revised: ⊂{ 1 ,...,n } α S (cid:81) i ∈ S

Original: f i is a valid generating polynomial.
Revised: f i is a valid generating polynomial.

Original: B. Experiments B.1. The Construction of SimplePGC
Revised: B. Experiments B.1. The Construction of SimplePGC

Original: A SimplePGC is a weighted sum over several DetPGCs, which are deﬁned in Section 4.2.
Revised: A SimplePGC is a weighted sum over several DetPGCs, which are deﬁned in Section 4.2.

Original: The structure of a SimplePGC is governed by two hyperparameters, the number of DetPGCs in the weighted sum (denoted by C ), and the maximum number of the variables (i.e. k in Figure 2) in the leaf distributions of the DetPGCs (denoted by K ).
Revised: The structure of a SimplePGC is governed by two hyperparameters, the number of DetPGCs in the weighted sum (denoted by C ), and the maximum number of the variables (i.e. k in Figure 2) in the leaf distributions of the DetPGCs (denoted by K ).

Original: Partitioning Variables To construct SimplePGC, we ﬁrst partition the variables X 1 , . . .
Revised: Partitioning Variables To construct SimplePGC, we ﬁrst partition the variables X 1 , . . .

Original: , X n into several groups.
Revised: , X n into several groups.

Original: The idea is, as shown in Section 4.2, for a DetPGC, variables from different groups have to be negatively dependent, so we want to put pairs of variables that are positively dependent in the same group.
Revised: The idea is, as shown in Section 4.2, for a DetPGC, variables from different groups have to be negatively dependent, so we want to put pairs of variables that are positively dependent in the same group.

Original: Given some training examples D 1 , . . .
Revised: Given some training examples D 1 , . . .

Original: , D l , we estimate the probabilities Pr
Revised: , D l , we estimate the probabilities Pr

Original: ( X i = 1) and Pr( X i = 1 , X j = 1) by counting; in particular, we set
Revised: ( X i = 1) and Pr( X i = 1 , X j = 1) by counting; in particular, we set

Original: Pr( event ) =
Revised: Pr( event ) =

Original: | D i where event is true | l .
Revised: | D i where event is true | l .

Original: Then, inspired by the deﬁnition of pairwise mutual information , when Pr( X i = 1 , X j = 1) >
Revised: Then, inspired by the deﬁnition of pairwise mutual information , when Pr( X i = 1 , X j = 1) >

Original: 0 , we use the quantity
Revised: 0 , we use the quantity

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  to measure the degree of positive dependence between X i and X j .
Revised: (9) to measure the degree of positive dependence between X i and X j .

Original: Note that X i and X j are positively dependent if w ij > 0 .
Revised: Note that X i and X j are positively dependent if w ij > 0 .

Original: Then we partition the variables into groups by the following greedy algorithm. 
Revised: Then we partition the variables into groups by the following greedy algorithm. Algorithm

Original:  Input: variables { X i } 1 ≤ i ≤ n , weights { w ij } 1 ≤ i<j ≤ n as deﬁned by Equation (9)
Revised: Variables Input: variables { X i } 1 ≤ i ≤ n , weights { w ij } 1 ≤ i<j ≤ n as deﬁned by Equation (9)

Original: Output: function group that maps X i to the group that X i belongs to
Revised: Output: function group that maps X i to the group that X i belongs to

Original: Initialization: group ( X i ) ← {

Original: W

Original:  > 0 } < ≤ n sort W in descending order for w ij in W do union ← group ( X i ) ∪ group ( X j ) if | union | ≤ K then group ( X i ) ← union group ( X j ) ← union end if end for
Revised: Initialization: group ( X i ) ← { X i } , W ← { w ij > 0 } 1 ≤ i<j ≤ n sort W in descending order for w ij in W do union ← group ( X i ) ∪ group ( X j ) if | union | ≤ K then group ( X i ) ← union group ( X j ) ← union end if end for

Original: Leaf PGCs After we partition the variables in to groups, we feed them to the leaf PGCs Pr i , as shown in Figure 2.
Revised: Leaf PGCs After we partition the variables in to groups, we feed them to the leaf PGCs Pr i , as shown in Figure 2.

Original: Pr i can be any PGCs; for SimplePGCs, we make the simplest choice by setting them to be the fully general distributions.
Revised: Pr i can be any PGCs; for SimplePGCs, we make the simplest choice by setting them to be the fully general distributions.

Original: For the leaf distribution Pr i over variables { X i 1 , . . .
Revised: For the leaf distribution Pr i over variables { X i 1 , . . .

Original: , X ik } , we let
Revised: , X ik } , we let

Original: [Equation]
Revised: [Equation]

Original: be its generating polynomial with parameters θ i,S and normalizing constant Z i .
Revised: be its generating polynomial with parameters θ i,S and normalizing constant Z i .

Original: Pr i s are fully general except for the constraint that all-zero assignment must have zero probability; this constraint is nothing but a trick that makes implementation easier.
Revised: Pr i s are fully general except for the constraint that all-zero assignment must have zero probability; this constraint is nothing but a trick that makes implementation easier.

Original: B.2. Supplementary Experiment Results
Revised: B.2. Supplementary Experiment Results

Original: In addition to the experiment results presented in Figure [Table] we also conducted one-sample t-tests.
Revised: In addition to the experiment results presented in Figure 3, we also conducted one-sample t-tests.

Original: The results of the statistical test are shown in Figure 4.
Revised: The results of the statistical test are shown in Figure 4.

Original: For the Twenty Datasets benchmark, the log-likelihood of Strudel is only statistically better than
Revised: For the Twenty Datasets benchmark, the log-likelihood of Strudel is only statistically better than

Original: SimplePGC on 6 out of 20 datasets and the log-likelihood of EiNets is only statistically better than SimplePGC on 2 out of  20 datasets.
Revised: SimplePGC on 6 out of 20 datasets; the log-likelihood of EiNets is only statistically better than SimplePGC on 2 out of datasets; the log-likelihood of MT is statistically better than SimplePGC on 7 out of 20 datasets.

Original: For the Amazon Baby Registries benchmark, the log-likelihoods of Strudel and EiNets  are statistically better than SimplePGC on none of the datasets.
Revised: For the Amazon Baby Registries benchmark, the log-likelihoods of Strudel, EiNets and MT are statistically better than SimplePGC on none of the datasets.

Revised: It would also help if we estimate the conﬁdence intervals of the average test log-likelihoods via cross validation.

Revised: However, since we have four baselines and not all of their learning algorithms were designed to be fast, the computation cost for estimating the conﬁdence intervals would be infeasible.

Original: [Table]
Revised: [Table]

Revised: .=

Revised: =

Revised: =

Revised: msweb − 9 .− 10 .− 9 .− 9 .

Revised: <

Revised: = = book − 34 .−

Revised: 34 .−

Revised: 34 .−

Revised: 34 .=

Revised: =

Revised: =

Revised: movie − 53 .− 51 .− 54 .−

Revised: 53 .=

Revised: = =

Revised: webkb − 154 .−

Original: 157 .−  155 .= 
Revised: 157 .− 156 .− 155 .= =

Original: =
Revised: =

Original: reuters − 86 .− 87 .−
Revised: reuters − 86 .− 87 .−

Revised: 85 .−

Original: 87 .=
Revised: 87 .=

Original: =  20ng − 153 .− 153 .− 154  .=
Revised: = < 20ng − 153 .− 153 .− 154 .− 154 .=

Original: =  bbc − 256 .−
Revised: = = bbc − 256 .−

Original: 248 .−  254 .=
Revised: 248 .− 261 .− 254 .=

Original: =  ad − 16 .−
Revised: = = ad − 16 .−

Original: 26 .− 21 .
Revised: 26 .− 16 .−

Original: < > 
Revised: < > <

Revised: [Table]

Revised: [Table]

Original: Strudel
Revised: Strudel

Original: EiNet
Revised: EiNet

Revised: MT

Original: SimplePGC vs. Strudel vs. EiNet  apparel − 9 .− 9 .− 9 .
Revised: SimplePGC vs. Strudel vs. EiNet vs. MT apparel − 9 .− 9 .− 9 .− 9 .>

Original: > =  bath − 8 .− 8 .− 8  .= >  bedding − 8 .− 8 .− 8  .= = 
Revised: = > bath − 8 .− 8 .− 8 .− 8 .= > > bedding − 8 .− 8 .− 8 .− 8 .= = >

Original: carseats − 4 .− 4 .− 4 .
Revised: carseats − 4 .− 4 .− 4 .− 4 .>

Original: > =  diaper − 9 .− 9 .− 9  .>
Revised: = > diaper − 9 .− 9 .− 9 .− 9 .>

Original: =  feeding − 11 .− 
Revised: = > feeding − 11 .− 11 .−

Original: 11 .− 11 .
Revised: 11 .− 11 .

Original: > =
Revised: > =

Original:  furniture − 4 .− 4 .− 4  .= 
Revised: = furniture − 4 .− 4 .− 4 .− 4 .= =

Original: =
Revised: =

Original: gear − 9 .− 9 .− 9  .= >  gifts − 3 .− 3 .− 3 .=
Revised: gear − 9 .− 9 .− 9 .− 9 .= > > gifts [Table] [Table]

Revised: [Table]

Original: =  health − 7 .− 7 .− 7 .> > media− 7 .− 7 .− 7 .=
Revised: = = = health [Table] [Table] > > > media

Original:  = moms − 3 .− 3 .−
Revised: [Table] = = = moms [Table]

Original: 3 .=
Revised: [Table]

Original: =
Revised: =

Original:  safety − 4 .− 4 .− 4 .> > strollers − 5 .− 5 .− 5 .=
Revised: = = safety [Table] [Table] [Table]

Original:  = 
Revised: > > = strollers [Table] [Table] [Table]

Original:  toys − 7 .− 7 .− 7 .= >
Revised: = = > toys [Table] [Table] = > >

Revised: [Figure]