Original: Semantic Probabilistic Layers for Neuro-Symbolic Learning
Revised: Semantic Probabilistic Layers for Neuro-Symbolic Learning

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We design a predictive layer for structured-output prediction (SOP) that can beplugged into any neural network guaranteeing its predictions are consistent with aset of predeﬁned symbolic constraints.
Revised: We design a predictive layer for structured-output prediction (SOP) that can be plugged into any neural network guaranteeing its predictions are consistent with a set of predeﬁned symbolic constraints.

Original: Our S emantic P robabilistic L ayer (SPL)can model intricate correlations, and hard constraints, over a structured outputspace all while being amenable to end-to-end learning via maximum likelihood.
Revised: Our S emantic P robabilistic L ayer (SPL) can model intricate correlations, and hard constraints, over a structured output space all while being amenable to end-to-end learning via maximum likelihood.

Original: SPLs combine exact probabilistic inference with logical reasoning in a cleanand modular way, learning complex distributions and restricting their support tosolutions of the constraint.
Revised: SPLs combine exact probabilistic inference with logical reasoning in a clean and modular way, learning complex distributions and restricting their support to solutions of the constraint.

Original: As such, they can faithfully, and efﬁciently, modelcomplex SOP tasks beyond the reach of alternative neuro-symbolic approaches.
Revised: As such, they can faithfully, and efﬁciently, model complex SOP tasks beyond the reach of alternative neuro-symbolic approaches.

Original: We empirically demonstrate that SPLs outperform these competitors in terms ofaccuracy on challenging SOP tasks including hierarchical multi-label classiﬁcation,pathﬁnding and preference learning, while retaining perfect constraint satisfaction. 
Revised: We empirically demonstrate that SPLs outperform these competitors in terms of accuracy on challenging SOP tasks including hierarchical multi-label classiﬁcation, pathﬁnding and preference learning, while retaining perfect constraint satisfaction. Our code is made publicly available on Github at github.com/KareemYousrii/SPL.

Original:  Introduction
Revised:  Introduction

Original: Modularity is among the major factors that propelled the Cambrian explosion of deep learning [34].
Revised: Modularity is among the major factors that propelled the Cambrian explosion of deep learning [34].

Original: By stacking multiple differentiable layers together, practitioners are able to train deep classiﬁers inan end-to-end fashion with little effort.
Revised: By stacking multiple differentiable layers together, practitioners are able to train deep classiﬁers in an end-to-end fashion with little effort.

Original: However, despite its ﬂexibility, this modular approach tolearning does not guarantee that the predictions of these models conform to our expectations asto what makes sense .
Revised: However, despite its ﬂexibility, this modular approach to learning does not guarantee that the predictions of these models conform to our expectations as to what makes sense .

Original: On the contrary, unconstrained deep classiﬁers are notorious for leading topredictions that are inconsistent with the logical constraints governing an underlying domain.
Revised: On the contrary, unconstrained deep classiﬁers are notorious for leading to predictions that are inconsistent with the logical constraints governing an underlying domain.

Original: This is even more evident in, and crucial for, structured output prediction (SOP) tasks, whereclassiﬁers have to predict hundreds of mutually constrained labels [70, 7].
Revised: This is even more evident in, and crucial for, structured output prediction (SOP) tasks, where classiﬁers have to predict hundreds of mutually constrained labels [72, 7].

Original: Consider for example aclassical SOP task such as multi-label classiﬁcation (MLC) [71].
Revised: Consider for example a classical SOP task such as multi-label classiﬁcation (MLC) [73].

Original: Learning a multi-label classiﬁer thatdisregards the correlations among labels, e.g., by considering them fully independent given the inputs,yields sub-optimal results [5].
Revised: Learning a multi-label classiﬁer that disregards the correlations among labels, e.g., by considering them fully independent given the inputs, yields sub-optimal results [5].

Original: In more challenging tasks such as hierarchical MLC (HMLC) [68] orpathﬁnding [57], leveraging the domain’s logical constraints (encoding, e.g., the label hierarchy oracyclicity and connectedness of a path) at training time can improve prediction accuracy [42], but itcannot guarantee that the predictions are always consistent with the constraints at inference time [31].
Revised: In more challenging tasks such as hierarchical MLC (HMLC) [70] or pathﬁnding [59], leveraging the domain’s logical constraints (encoding, e.g., the label hierarchy or acyclicity and connectedness of a path) at training time can improve prediction accuracy [43], but it cannot guarantee that the predictions are always consistent with the constraints at inference time [31].

Original: Fig.
Revised: Fig.

Original: 1 illustrates this problem in the context of pathﬁnding: constraint-unaware neural networks systematically fail to predict label conﬁgurations that form a valid path. 
Revised: 1 illustrates this problem in the context of pathﬁnding: constraint-unaware neural networks systematically fail to predict label conﬁgurations that form a valid path. In many safety-critical 36th Conference on Neural Information Processing Systems (NeurIPS 2022). G ROUND T RUTH

Original: In many safety-criticalscenarios such as protein function [60] and interaction prediction [62], and drug discovery [19, 23],predicting inconsistent solutions can not only be harmful but also highly expensive [4, 33].
Revised: [Figure] scenarios such as protein function [62] and interaction prediction [64], and drug discovery [19, 23], predicting inconsistent solutions can not only be harmful but also highly expensive [4, 33].

Original: , due to their discrete nature, injecting logical constraints into deep neural networks while retaining modularity and differentiability is extremely challenging, as demonstrated in the neuro-symbolic learning literature [63].
Revised: Unsurprisingly, due to their discrete nature, injecting logical constraints into deep neural networks while retaining modularity and differentiability is extremely challenging, as demonstrated in the neuro-symbolic learning literature [65].

Original: One such attempt has been to learn neural networks that satisfy the logical constraints by explicitly minimizing a differentiable loss term, the probability that the networks violates the constraint for any given prediction.
Revised: One such attempt has been to learn neural networks that satisfy the logical constraints by explicitly minimizing a differentiable loss term, the probability that the networks violates the constraint for any given prediction.

Original: And while successful, such approaches 
Revised: And while successful, such approaches do not guarantee consistency of the predictions at test time.

Original: [Figure]
Revised: More recently, researchers have proposed predictive layers that do guarantee consistency, but these are restricted to speciﬁc kinds of symbolic knowledge [31, 69] or become intractable for even moderately complex logical constraints [37].

Original: ET [Figure] [Figure] SPL
Revised: Motivated by these observations, we introduce a novel S emantic P robabilistic L ayer (SPL) for modeling intricate correlations, and logical constraints on the labels of the output space in a modular and probabilistically sound manner.

Original: [Figure] logical Motivated by these observations, introduce a
Revised: It does so by leveraging recent advancements in the literature on probabilistic circuits [75, 12].

Original: S emantic P robabilistic L
Revised: The key features of SPL are that, on the one hand, it can be used as a drop-in replacement for common predictive layers of deep nets like sigmoid layers, and on the other, it guarantees the output’s consistency with any prespeciﬁed logical constraints. Importantly, SPL also supports efﬁcient inference and – perhaps surprisingly – does not complicate training.

Original: Contributions.
Revised: Contributions.

Original: Summarizing, we: ( i ) Identify six desiderata that neuro-symbolic predictors forconstrained SOP tasks ought to satisfy, and show that state-of-the-art approaches fall short of oneor more of them (Table 1); ( ii ) Introduce SPL, a novel semantic probabilistic layer that leveragesprobabilistic circuits to satisfy all six desiderata, i.e., that can be plugged into neural networks toensure predictions to comply to given logical constraints, while retaining efﬁciency, expressivity,differentiability, and fully probabilistic semantics; ( iii ) We provide empirical evidence of the effectiveness of SPL in several challenging neuro-symbolic SOP tasks, such as HMLC and pathﬁnding,where it outperforms state-of-the-art neuro-symbolic approaches, often by a noticeable margin. 
Revised: Summarizing, we: ( i ) Identify six desiderata that neuro-symbolic predictors for constrained SOP tasks ought to satisfy, and show that state-of-the-art approaches fall short of one or more of them (Table 1); ( ii ) Introduce SPL, a novel semantic probabilistic layer that leverages probabilistic circuits to satisfy all six desiderata, i.e., that can be plugged into neural networks to ensure predictions to comply to given logical constraints, while retaining efﬁciency, expressivity, differentiability, and fully probabilistic semantics; ( iii ) We provide empirical evidence of the effectiveness of SPL in several challenging neuro-symbolic SOP tasks, such as HMLC and pathﬁnding, where it outperforms state-of-the-art neuro-symbolic approaches, often by a noticeable margin. We implemented SPLs in PyTorch [53], and our code is made publicly available on Github at github.com/KareemYousrii/SPL.

Original:  Designing a probabilistic layer for neuro-symbolic SOP
Revised:  Designing a probabilistic layer for neuro-symbolic SOP

Original: Notation.
Revised: Notation.

Original: In the following, we denote scalar constants x in lower case, random variables X inupper case, vectors of constants x in bold and vectors of random variables X in capital boldface.
Revised: In the following, we denote scalar constants x in lower case, random variables X in upper case, vectors of constants x in bold and vectors of random variables X in capital boldface.

Original: 57{ ϕ } denotes the indicator function that evaluates to 1 if the statement ϕ holds and to 0 otherwise.
Revised: { ϕ } denotes the indicator function that evaluates to 1 if the statement ϕ holds and to 0 otherwise.

Original: We denote by x | =
Revised: We denote by x | =

Original: K that the value assignment x to variables X satisﬁes a logical formula K .
Revised: K that the value assignment x to variables X satisﬁes a logical formula K .

Original: Neuro-symbolic SOP.
Revised: Neuro-symbolic SOP.

Original: We tackle SOP tasks in which a neural net classiﬁer must learn to associateinstances x ∈ R D to L interdependent labels, identiﬁed by the vector y ∈ { 0 , 1 } L .
Revised: We tackle SOP tasks in which a neural net classiﬁer must learn to associate instances x ∈ R D to L interdependent labels, identiﬁed by the vector y ∈ { 0 , 1 } L .

Original: We assume thatwe can abstract any neural classiﬁer into two components: a feature extractor f that maps inputs
Revised: We assume that we can abstract any neural classiﬁer into two components: a feature extractor f that maps inputs

Original: X to a M -dimensional embedding Z = f ( X ) and a predictive ﬁnal layer that outputs the labeldistribution p
Revised: X to a M -dimensional embedding Z = f ( X ) and a predictive ﬁnal layer that outputs the label distribution p

Original: ( Y | Z ) .
Revised: ( Y | Z ) .

Original: For example, the simplest, and yet widely adopted [49, 77, 31], predictivelayer in neural classiﬁers for SOP considers labels Y i to be conditionally independent from each othergiven Z , i.e., p ( Y | Z )
Revised: For example, the simplest, and yet widely adopted [50, 79, 31], predictive layer in neural classiﬁers for SOP considers labels Y i to be conditionally independent from each other given Z , i.e., p ( Y | Z )

Original: =
Revised: =

Original: (cid:81)
Revised: (cid:81)

Original: Li =1 p ( Y i | Z ) .
Revised: Li =1 p ( Y i | Z ) .

Original: We refer to this as fully independent layer (FIL).
Revised: We refer to this as fully independent layer (FIL).

Original: In a FIL,p ( Y i =
Revised: In a FIL, p ( Y i =

Original: y i | z ) is computed as σ ( w (cid:62) i z ) where w i ∈ R M is a vector of parameters and σ ( x ) is thesigmoid function 1 / (1 + e x ) . 
Revised: y i | z ) is computed as σ ( w (cid:62) i z ) where w i ∈ R M is a vector of parameters and σ ( x ) is the sigmoid function 1 / (1 + e x ) . [Table]

Original: We are interested in dependencies between labels that can occur both as correlations, as is the casein MLC [21], and as logical constraints encoded by logical formulas. For example, in a HMLCtask [31] one logical constraint can encode the fact that observing a label for the class cat and dog,implies observing the label for their superclass animal
Revised: We are interested in dependencies between labels that can occur both as correlations

Original: [Equation]
Revised: [Table]

Original: Speciﬁcally, we assume symbolic knowledge to be supplied in the form of constraints encoded as alogical formula K over the labels Y and optionally over a subset of the discrete input variables in X ,
Revised: as is the case in MLC [21]

Original: [Table]
Revised: [Table]

Original: if any (e [Table] g., in our experiments
Revised: and as logical constraints encoded by logical formulas. For example, in a HMLC task [31] one logical constraint can encode the fact that observing a label for the class cat and dog, implies observing the label for their superclass

Original: [Table]
Revised: [Equation]

Original:  the predicted simple path is constrained to lie within the subset ofedges appearing in the input graph, see Sec. 5).On the other hand [Table]
Revised: Speciﬁcally, we assume symbolic knowledge to be supplied in the form of constraints encoded as a logical formula K over the labels Y and optionally over a subset of the discrete input variables in X , if any (e.g., in our experiments, the predicted simple path is constrained to lie within the subset of edges appearing in the input graph, see Sec. 5).

Original:  we expect a model to learn thelabel dependencies from data.
Revised: On the other hand, we expect a model to learn the label dependencies from data.

Original: We call such tasks neuro-symbolic SOP tasks.
Revised: We call such tasks neuro-symbolic SOP tasks.

Original: Desiderata for neuro-symbolic SOP.
Revised: Desiderata for neuro-symbolic SOP.

Original: To tackle this setting, we seek an algorithmic strategy forreplacing the predictive layer in any neural network classiﬁer with little effort, with the aim of injectingcomplex symbolic knowledge and allowing for ﬂexible probabilistic reasoning.
Revised: To tackle this setting, we seek an algorithmic strategy for replacing the predictive layer in any neural network classiﬁer with little effort, with the aim of injecting complex symbolic knowledge and allowing for ﬂexible probabilistic reasoning.

Original: We formalize theseobservations into the following six desiderata for our predictive layer:
Revised: We formalize these observations into the following six desiderata for our predictive layer:

Original: D1.
Revised: D1.

Original: Probabilistic :
Revised: Probabilistic :

Original: The layer should enjoy sound probabilistic semantics, and deliver normalizedprobabilistic predictions to facilitate maximum-likelihood learning and sound decisionmaking by virtue of calibrated probabilistic predictionsD2.
Revised: The layer should enjoy sound probabilistic semantics, and deliver normalized probabilistic predictions to facilitate maximum-likelihood learning and sound decision making by virtue of calibrated probabilistic predictions D2.

Original: Expressive : It should be able to compactly encode intricate correlations between labels.
Revised: Expressive : It should be able to compactly encode intricate correlations between labels.

Original: D3.
Revised: D3.

Original: Consistent : It should always output predictions that are consistent with the prespeciﬁedsymbolic knowledge, i.e., for all x and y , if ( x , y ) (cid:54)| =
Revised: Consistent : It should always output predictions that are consistent with the prespeciﬁed symbolic knowledge, i.e., for all x and y , if ( x , y ) (cid:54)| =

Original: K then p ( y | x ) = 0 .D4.
Revised: K then p ( y | x ) = 0 . D4.

Original: General : It should support rich logical constraints over the labels expressed in some formallanguage, e.g., propositional logic.
Revised: General : It should support rich logical constraints over the labels expressed in some formal language, e.g., propositional logic.

Original: D5.
Revised: D5.

Original: Modular : It should be applicable to any off-the-shelf (and possibly pretrained) neuralnetwork in a modular fashion, enabling end-to-end learning and rapid prototyping.D6.
Revised: Modular : It should be applicable to any off-the-shelf (and possibly pretrained) neural network in a modular fashion, enabling end-to-end learning and rapid prototyping. D6.

Original: Efﬁcient : The time required by the predictor to compute a prediction should be linear in thesize of the predictor and of the hard constraint representation.
Revised: Efﬁcient : The time required by the predictor to compute a prediction should be linear in the size of the predictor and of the hard constraint representation.

Original: For example, FILs are clearly probabilistic (D1), modular (D5), and efﬁcient (D6), but at the costof being incapable of modeling intricate correlations and logical constraints and thus generatinginconsistent predictions (D2–D4) (see also Fig. 1).
Revised: For example, FILs are clearly probabilistic (D1), modular (D5), and efﬁcient (D6), but at the cost of being incapable of modeling intricate correlations and logical constraints and thus generating inconsistent predictions (D2–D4) (see also Fig. 1).

Original: Table 1 summarizes how the other popular andeffective approaches to neuro-symbolic SOP nowadays fall short of one of more desiderata as well.
Revised: Table 1 summarizes how the other popular and effective approaches to neuro-symbolic SOP nowadays fall short of one of more desiderata as well.

Original: We discuss this in detail in Sec. 4.
Revised: We discuss this in detail in Sec. 4.

Original: To the best of our knowledge, our proposed semantic probabilisticlayers (SPLs) are the ﬁrst algorithmic solution to satisfy all above desiderata.
Revised: To the best of our knowledge, our proposed semantic probabilistic layers (SPLs) are the ﬁrst algorithmic solution to satisfy all above desiderata.

Original: SPL.
Revised: SPL.

Original: At a high level, SPL realizes the above desiderata in a single layer that combines exact probabilistic inference with logical reasoning in a clean and modular way, learning complex distributionsand restricting their support to solutions of the constraint.
Revised: At a high level, SPL realizes the above desiderata in a single layer that combines exact probabilistic inference with logical reasoning in a clean and modular way, learning complex distributions and restricting their support to solutions of the constraint.

Original: Deﬁnition 2.1 (Semantic probabilistic layer (SPL)) .
Revised: Deﬁnition 2.1 (Semantic probabilistic layer (SPL)) .

Original: Given an input conﬁguration x , a SPL decomposes the computation of the probability of a label conﬁguration as:
Revised: Given an input conﬁguration x , a SPL decomposes the computation of the probability of a label conﬁguration as:

Original: [Equation]
Revised: [Equation]

Original: where 
Revised: where Z ( x ) = (cid:88) y q Θ ( y | x ) · c K ( x , y ) . Here, q Θ ( y | f ( x )) is a module to perform probabilistic reasoning by encoding an expressive distribution over the labels parameterized by Θ ; c K ( x , y ) is a module to ensure consistency of the

Original: [Equation]
Revised: [Figure]

Original: Here, q Θ ( y | f ( x )) is a module to perform probabilistic reasoning by encoding an expressivedistribution over the labels parameterized by Θ ; c K ( x , y ) is a module to ensure consistency of thepredictions by encoding logical constraints K and being non-zero only when K is satisﬁed, i.e.,c K ( x , y )
Revised: predictions by encoding logical constraints K and being non-zero only when K is satisﬁed, i.e., c K ( x , y )

Original: = 1 { ( x , y )
Revised: = 1 { ( x , y )

Original: | = K } ; and Z ( x ) is a renormalization term, also called the partition function.
Revised: | = K } ; and Z ( x ) is a renormalization term, also called the partition function.

Revised: It is worth noting that this amounts to taking a product of experts [36] which is, in general, hard.

Original: Fig.
Revised: Fig.

Original: 2 illustrates the computational graph of our SPL at training time.
Revised: 2 illustrates the computational graph of our SPL at training time.

Original: In order to satisfy all D1-D6, wewill realize both q Θ and c K as circuits [73, 12], constrained computational graphs that enable tractablecomputations.
Revised: In order to satisfy all D1-D6, we will realize both q Θ and c K as circuits [75, 12], constrained computational graphs that enable tractable computations.

Original: Differently from FILs, q Θ in SPLs can encode an expressive joint distributions overthe labels and therefore attain full expressiveness by scaling the number of parameters Θ (D2).
Revised: Differently from FILs, q Θ in SPLs can encode an expressive joint distributions over the labels and therefore attain full expressiveness by scaling the number of parameters Θ (D2).

Original: Consistency is guaranteed by the component c K : by multiplying it to the joint probability of a label[Figure] The by marginalizing product Θ , K , so conﬁguration the resulting product distribution r Θ , K ( y , x ) =
Revised: Consistency is guaranteed by the component c K : by multiplying it to the joint probability of a label conﬁguration the resulting product distribution r Θ , K ( y , x ) =

Original: q Θ ( y | f ( x )) ·
Revised: q Θ ( y | f ( x )) ·

Original: c K ( x , y ) will have itssupport effectively cut by K , and thus cannot allocate any probability mass to inconsistent predictions(D3).
Revised: c K ( x , y ) will have its support effectively cut by K , and thus cannot allocate any probability mass to inconsistent predictions (D3).

Original: Additionally, c K will allow to encode general propositional logical constraints in a compactcomputational graph (D4).
Revised: Additionally, c K will allow to encode general propositional logical constraints in a compact computational graph (D4).

Original: Lastly, the product r Θ , K ( x , y ) is fully differentiable and allows SPL to bean off-the-shelf replacement for other predictive layers (see Fig. 2) and enables end-to-end learning(D5).
Revised: Lastly, the product r Θ , K ( x , y ) is fully differentiable and allows SPL to be an off-the-shelf replacement for other predictive layers (see Fig. 2) and enables end-to-end learning (D5).

Original: By renormalizing r Θ , K ( x , y ) and outputting normalized probabilities, SPL enables the exactcomputation of gradients for Θ , which can therefore be trained by maximum likelihood.
Revised: By renormalizing r Θ , K ( x , y ) and outputting normalized probabilities, SPL enables the exact computation of gradients for Θ , which can therefore be trained by maximum likelihood.

Original: Thanks to recent advancements in the literature on circuits, we can compute the partition function
Revised: Thanks to recent advancements in the literature on circuits, we can compute the partition function

Original: Z ( x ) efﬁciently in time linear in the size of r Θ , K , thus preserving efﬁciency (D6) and not compromising on the other desiderata.
Revised: Z ( x ) efﬁciently in time linear in the size of r Θ , K , thus preserving efﬁciency (D6) and not compromising on the other desiderata.

Original: This will also yield correct (and consistent) predictions at testtime, when an SPL computes the MAP state y ∗ =
Revised: This will also yield correct (and consistent) predictions at test time, when an SPL computes the MAP state y ∗ =

Original: argmax y r Θ , K ( y , x ) / (cid:80) y r Θ , K ( y , x ) .
Revised: argmax y r Θ , K ( y , x ) / (cid:80) y r Θ , K ( y , x ) .

Original: The nextsection clariﬁes how to implement the modules of SPL as circuits while satisfying these desiderata.
Revised: The next section clariﬁes how to implement the modules of SPL as circuits while satisfying these desiderata.

Original:  Realizing SPLs with tractable circuit representations
Revised:  Realizing SPLs with tractable circuit representations

Original: The components of SPLs are circuits , a large class of computational graphs that can represent bothfunctions and distributions [12, 17].
Revised: The components of SPLs are circuits , a large class of computational graphs that can represent both functions and distributions [12, 17].

Original: Circuits subsume many tractable generative and discriminativeprobabilistic models—from Chow-Liu and latent tree models [13, 10], to hidden Markov models(HMMs) [59], sum-product networks (SPNs) [58], decision trees [39, 14], and deep regressors [38]—as well as many compact representations of logical formulas, such as (ordered) binary decisiondiagrams [3], sentential decision diagrams (SDDs) [16] and others [17].
Revised: Circuits subsume many tractable generative and discriminative probabilistic models—from Chow-Liu and latent tree models [13, 10], to hidden Markov models (HMMs) [61], sum-product networks (SPNs) [60], decision trees [40, 14], and deep regressors [39]— as well as many compact representations of logical formulas, such as (ordered) binary decision diagrams [3], sentential decision diagrams (SDDs) [16] and others [17].

Original: The key idea behind SPLs is to leverage this single formalism to represent both an expressive jointdistribution for q Θ ( y | f ( x )) and a compact encoding of the logical constraints for c K ( x , y ) , whileensuring the exact and efﬁcient evaluation of Eq.
Revised: The key idea behind SPLs is to leverage this single formalism to represent both an expressive joint distribution for q Θ ( y | f ( x )) and a compact encoding of the logical constraints for c K ( x , y ) , while ensuring the exact and efﬁcient evaluation of Eq.

Original: This can be achieved by ensuring that thesecomputational graphs abide certain structural properties: smoothness , decomposability , determinismand compatibility [17, 72].
Revised: This can be achieved by ensuring that these computational graphs abide certain structural properties: smoothness , decomposability , determinism and compatibility [17, 74].

Original: Next, we introduce probabilistic circuits for modeling q Θ (Sec. 3.1) andconstraint circuits for c K (Sec. 3.2), while in Sec.
Revised: Next, we introduce probabilistic circuits for modeling q Θ (Sec. 3.1) and constraint circuits for c K (Sec. 3.2), while in Sec.

Original: 3.4 we propose a more efﬁcient implementationof SPL utilizing a single circuit.
Revised: 3.4 we propose a more efﬁcient implementation of SPL utilizing a single circuit.

Original: 3.1 Representing expressive distributions with probabilistic circuits
Revised: 3.1 Representing expressive distributions with probabilistic circuits

Original: We start by introducing circuits for joint probability distributions, and then extend the discussion toconditional distributions, which we use to implement q Θ ( Y | f ( X )) in SPLs. 
Revised: We start by introducing circuits for joint probability distributions, and then extend the discussion to conditional distributions, which we use to implement q Θ ( Y | f ( X )) in SPLs. Y Y Y

Original:  Deﬁnition 3.1 (Circuits) .
Revised: Y× × Y Y× × q g 1 { Y 1 =0 } 1 { Y 1 =1 } 1 { Y 2 =0 } 1 { Y 2 =1 } × × × × 1 { Y 3 =1 } 1 { Y 3 =0 } × c Deﬁnition 3.1 (Circuits) .

Original: A circuit h over variables Y is a computational graph encoding a parameterized function h Θ ( Y ) by combining three kinds of computational units: input functionalunits, sum units, and product units.
Revised: A circuit h over variables Y is a computational graph encoding a parameterized function h Θ ( Y ) by combining three kinds of computational units: input functional units, sum units, and product units.

Original: An input functional n represents a base parametric functionh n ( sc ( n ); λ ) over some variables sc ( n ) ⊆
Revised: An input functional n represents a base parametric function h n ( sc ( n ); λ ) over some variables sc ( n ) ⊆

Original: Y , called its scope, and it is parameterized by λ .
Revised: Y , called its scope and it is parameterized by λ .

Original: Sum andproduct units n elaborate the output of other units, denoted in ( n ) .
Revised: Sum and product units n elaborate the output of other units, denoted in ( n ) .

Original: Sum units are parameterized by ωand compute the weighted sum of their inputs (cid:80) c ∈ in ( n ) ω
Revised: Sum units are parameterized by ω and compute the weighted sum of their inputs (cid:80) c ∈ in ( n ) ω

Original: c h c ( sc ( n )) , while product units compute148 (cid:81) c ∈ in ( n ) h c ( sc ( n )) .
Revised: c h c ( sc ( n )) , while product units compute (cid:81) c ∈ in ( n ) h c ( sc ( n )) .

Original: The parameters Θ of a circuit encompass the parameters of all input functionals( λ ) and sum units ( ω ).Y Y Y
Revised: The parameters Θ of a circuit encompass the parameters of all input functionals ( λ ) and sum units ( ω ).

Original: Y× × Y Y× × q g 1 { Y 1 =0 } 1 { Y 1 =1 } 1 { Y 2 =0 } 1 { Y 2 =1 } × × × × 1 { Y 3 =1 } 1 { Y 3 =0 } × c For any input y , the value of h Θ ( y ) can be evaluated by propagating the output of the input unitsthrough the computational graph and reading out the value of the last unit.
Revised: For any input y , the value of h Θ ( y ) can be evaluated by propagating the output of the input units through the computational graph and reading out the value of the last unit.

Original: The support of h is the set of all states y for which the output is non-zero, i.e., supp ( h ) =
Revised: The support of h is the set of all states y for which the output is non-zero, i.e., supp ( h ) =

Original: { y ∈ val ( Y )
Revised: { y ∈ val ( Y )

Original: | h ( y  (cid:54) = 0 } 
Revised: | h ( y ) (cid:54) = 0 } .

Original: Deﬁnition 3.2 (Probabilistic circuits (PCs)) .
Revised: Deﬁnition 3.2 (Probabilistic circuits (PCs)) .

Original: A circuit q is a PC if it encodes a (possibly unnormalized)probability distribution, i.e., q Θ ( y ) is non-negative for all conﬁgurations y of Y .
Revised: A circuit q is a PC if it encodes a (possibly unnormalized) probability distribution, i.e., q Θ ( y ) is non-negative for all conﬁgurations y of Y .

Original: From here on, we will assume PCs to have positive sum parameters ω and whose input units modelvalid distributions, e.g., Bernoullis, as these conditions are sufﬁcient for satisfying Def. 3.2.
Revised: From here on, we will assume PCs to have positive sum parameters ω and whose input units model valid distributions, e.g., Bernoullis, as these conditions are sufﬁcient for satisfying Def. 3.2.

Original: Moreover,w.l.o.g.
Revised: Moreover, w.l.o.g.

Original: we will assume the sum and product units to be organized into alternating layers, and thatevery product unit n receives only two inputs c 1 , c 2 , i.e., q n ( X ) =
Revised: we will assume the sum and product units to be organized into alternating layers, and that every product unit n receives only two inputs c 1 , c 2 , i.e., q n ( X ) =

Original: q c 1 ( Y ) · q c 2 ( Y ) .
Revised: q c 1 ( Y ) · q c 2 ( Y ) .

Original: These conditionscan easily be enforced in polynomial time [74, 75].
Revised: These conditions can easily be enforced in polynomial time [76, 77].

Original: We are speciﬁcally interested in smooth anddecomposable PCs, as they will be enabling efﬁcient inference in SPL (Sec. 3.3).
Revised: We are speciﬁcally interested in smooth and decomposable PCs, as they will be enabling efﬁcient inference in SPL (Sec. 3.3).

Original: Deﬁnition 3.3 (Smoothness & Decomposability) .
Revised: Deﬁnition 3.3 (Smoothness & Decomposability) .

Original: A circuit is smooth if for every sum unit n , its inputsdepend on the same variables: ∀ c 1 , c 2 ∈ in ( n ) , sc ( c 1 ) =
Revised: A circuit is smooth if for every sum unit n , its inputs depend on the same variables: ∀ c 1 , c 2 ∈ in ( n ) , sc ( c 1 ) =

Original: sc ( c 2 ) .
Revised: sc ( c 2 ) .

Original: It is decomposable if the inputs ofevery product unit n depend on disjoint sets of variables: in ( n ) = { c 1 , c 2 } , sc ( c 1 ) ∩ sc ( c 2 ) = ∅ .
Revised: It is decomposable if the inputs of every product unit n depend on disjoint sets of variables: in ( n ) = { c 1 , c 2 } , sc ( c 1 ) ∩ sc ( c 2 ) = ∅ .

Original: Smooth and decomposable PCs are both expressive and efﬁcient: they can encode distributions withhundred millions of parameters and be effectively learned by gradient ascent [55].
Revised: Smooth and decomposable PCs are both expressive and efﬁcient: they can encode distributions with hundred millions of parameters and be effectively learned by gradient ascent [57].

Original: The structure oftheir computational graph can be either speciﬁed manually [58, 55, 53] or acquired automaticallyfrom data [74, 61, 15], e.g., by ﬁrst learning a latent tree model and then compiling the latter into acircuit [44].
Revised: The structure of their computational graph can be either speciﬁed manually [60, 57, 55] or acquired automatically from data [76, 63, 15], e.g., by ﬁrst learning a latent tree model and then compiling the latter into a circuit [45].

Original: These circuits are competitive with intractable models such as variational autoencodersand normalizing ﬂows scores on several benchmarks [43].
Revised: These circuits are competitive with intractable models such as variational autoencoders and normalizing ﬂows scores on several benchmarks [44].

Original: As proposed by Shao et al.
Revised: As proposed by Shao et al.

Original: [65], any (smooth and decomposable) PC q Θ ( Y ) encoding a jointdistribution over the labels Y can be turned into a (smooth and decomposable) conditional circuit,conditioned by input variables X , by letting its parameters be a function of X .
Revised: [67], any (smooth and decomposable) PC q Θ ( Y ) encoding a joint distribution over the labels Y can be turned into a (smooth and decomposable) conditional circuit, conditioned by input variables X , by letting its parameters be a function of X .

Original: Deﬁnition 3.4 (Neural conditional circuits [64]) .
Revised: Deﬁnition 3.4 (Neural conditional circuits [66]) .

Original: A conditional circuit q ( Y ; Θ = g ( X )) models theconditional distribution p ( Y | X ) via a differentiable function g that maps every input conﬁgurationx to the set of parameters of Θ of p , also called the gating function .
Revised: A conditional circuit q ( Y ; Θ = g ( X )) models the conditional distribution p ( Y | X ) via a differentiable function g that maps every input conﬁguration x to the set of parameters of Θ of p , also called the gating function .

Original: An example of a smooth and decomposable conditional circuit is shown in Fig.
Revised: An example of a smooth and decomposable conditional circuit is shown in Fig.

Original: This designimmediately allows us to implement q Θ ( Y | f
Revised: This design immediately allows us to implement q Θ ( Y | f

Original: ( X )) in SPL as a conditional PC whose gating functionmaps the feature embedding space R K to the parameter space R | Θ | + , realizing q ( Y ; Θ = g ( f ( X ))) .
Revised: ( X )) in SPL as a conditional PC whose gating function maps the feature embedding space R K to the parameter space R | Θ | + , realizing q ( Y ; Θ = g ( f ( X ))) .

Original: As such, the gating function g creates a clean interface between any pre-trained feature extractorf and the PC q (Fig. 3).
Revised: As such, the gating function g creates a clean interface between any pre-trained feature extractor f and the PC q (Fig. 3).

Original: While one can devise g in several ways, we strive for simplicity in ourexperiments and adopt vanilla multi-layer perceptrons (MLPs) whose ﬁnal activations are eithersigmoids, if they have to predict the parameters λ of the Bernoulli input distributions of q , or softmax,if they output the sum unit parameters ω (Def. 3.1).
Revised: While one can devise g in several ways, we strive for simplicity in our experiments and adopt vanilla multi-layer perceptrons (MLPs) whose ﬁnal activations are either sigmoids, if they have to predict the parameters λ of the Bernoulli input distributions of q , or softmax, if they output the sum unit parameters ω (Def. 3.1).

Original: 3.2 Encoding logical formulas with constraint circuits
Revised: 3.2 Encoding logical formulas with constraint circuits

Original: The next step is to translate a logical constraint K into a smooth and decomposable circuit c K ( x , y ) .
Revised: The next step is to translate a logical constraint K into a smooth and decomposable circuit c K ( x , y ) .

Original: To this end, we employ a special type of PCs, deﬁned as follows.
Revised: To this end, we employ a special type of PCs, deﬁned as follows.

Original: Deﬁnition 3.5 (Constraint circuits) .
Revised: Deﬁnition 3.5 (Constraint circuits) .

Original: A PC c over variables X ∪ Y is a constraint circuit encodingprior knowledge K if it computes 1 { ( x , y ) |
Revised: A PC c over variables X ∪ Y is a constraint circuit encoding prior knowledge K if it computes 1 { ( x , y ) |

Original: = K } for every conﬁguration ( x , y ) .
Revised: = K } for every conﬁguration ( x , y ) .

Original: As a practical way to realize such a circuit, we will consider constraint circuits that have all sumunit parameters equal to 1 and input functionals that are indicator functions over their scope, e.g.,c n ( z ) = 1 { z | = ϕ ( n ) } where Z is the scope of the input and ϕ ( n ) a constraint over it.
Revised: As a practical way to realize such a circuit, we will consider constraint circuits that have all sum unit parameters equal to 1 and input functionals that are indicator functions over their scope, e.g., c n ( z ) = 1 { z | = ϕ ( n ) } where Z is the scope of the input and ϕ ( n ) a constraint over it.

Original: Furthermore,we require each sum unit in it to be deterministic .
Revised: Furthermore, we require each sum unit in it to be deterministic .

Original: Deﬁnition 3.6 (Determinism) .
Revised: Deﬁnition 3.6 (Determinism) .

Original: A sum unit n is deterministic if its inputs have disjoint supports, i.e.,∀ c 1 , c 2 ∈ in ( n ) , c 1 (cid:54)
Revised: A sum unit n is deterministic if its inputs have disjoint supports, i.e., ∀ c 1 , c 2 ∈ in ( n ) , c 1 (cid:54)

Original: =
Revised: =

Original: c 2 = ⇒ supp ( c 1 ) ∩ supp ( c 2 ) = ∅ .
Revised: c 2 = ⇒ supp ( c 1 ) ∩ supp ( c 2 ) = ∅ .

Original: Fig.
Revised: Fig.

Original: 3 shows an example of a deterministic constraint circuit.
Revised: 3 shows an example of a deterministic constraint circuit.

Original: Thanks to determinism, we can readilytranslate classical compact representations for logical formulas such as (ordered) binary decisiondiagrams [3, 8] and sentential decision diagrams (SDDs) [16] into constraint circuits as deﬁned above.
Revised: Thanks to determinism, we can readily translate classical compact representations for logical formulas such as (ordered) binary decision diagrams [3, 8] and sentential decision diagrams (SDDs) [16] into constraint circuits as deﬁned above.

Original: This becomes evident when they are written in the language of negation normal form [17] and theirand gates (resp. or gates) are replaced with product units (resp.
Revised: This becomes evident when they are written in the language of negation normal form [17] and their and gates (resp. or gates) are replaced with product units (resp.

Original: sum units) [12].
Revised: sum units) [12].

Original: A logic constraintcan therefore be represented as a constraint circuit for SPLs, by utilizing any of the many toolsavailable for OBDDs [69] or SDDs [9, 51].
Revised: A logic constraint can therefore be represented as a constraint circuit for SPLs, by utilizing any of the many tools available for OBDDs [71] or SDDs [9, 52].

Original: Sec.
Revised: Sec.

Original: B illustrates in detail how to compile the exampleconstraint of Eq.
Revised: B illustrates in detail how to compile the example constraint of Eq.

Original: (1) into the constraint circuit of Fig.
Revised: (1) into the constraint circuit of Fig.

Original: 3 in this way.
Revised: 3 in this way.

Revised: The worst-case size of the constraint circuit depends on a) the algorithm employed for compilation and, b) the local structure of the constraints, rather than the number of labels.

Revised: For example, in the

Original: {+Warcraft experiment (Sec. 5), we have a label conﬁguration space of 2 12 2 = 
+}
Revised: {+Warcraft experiment (Sec. 5), we have a label conﬁguration space of 2 12 2 = 
+}

Revised: ≈ 10 43 states but only 10 10 conﬁgurations are valid.

Original: 3.3 Efﬁcient inference in SPLs
Revised: If our compilation algorithm were to simply enumerate these conﬁgurations, putting them in a logical OR (as in MultiplexNet [37]), |c| would be 10 10 . In our experiments, we use the pySDD compiler, which generates circuits whose size is worst-case exponential in the treewidth of the CNF representation of the constraint, but typically much smaller. In Sec. F we provide the average timings for compilation—carried out once, and reused in all subsequent experiments, parameterization, MAP-state computation and cross-entropy (product + normalization).

Revised: All these timings, compilation excluded, are per batch.

Revised: We compare to the timings of semantic loss and neuro-symbolic entropy, where applicable.

Revised: We point out that the Warcraft experiment—by far the most time consuming—is a distribution over 10 10 possible paths, and yet we’re able to compute the likelihood in a mere 14 seconds per batch.

Revised: We also adopt 12 real-world datasets, from domains ranging from functional genomics to text categorization.

Revised: Some of these biological datasets have 499 labels, yielding a target space of 2 499 conﬁgurations and can be compiled in seconds into compact circuits of size ≈ 108 KB.

Original: As discussed above, PCs can be expressive (D1) and are modular (D5), while constraint circuitsensure consistency (D3) for general constraints (D4).
Revised: As discussed above, PCs can be expressive (D1) and are modular (D5), while constraint circuits ensure consistency (D3) for general constraints (D4).

Original: What remains to be shown to complete SPLs isthat the product supports efﬁcient normalization (D2) and inference (D6), speciﬁcally that it allowsfor the efﬁcient evaluation of the normalization constant of r Θ , K , and its MAP state.
Revised: What remains to be shown to complete SPLs is that the product supports efﬁcient normalization (D2) and inference (D6), speciﬁcally that it allows for the efﬁcient evaluation of the normalization constant of r Θ , K , and its MAP state.

Original: To this end, weneed to introduce the notion of compatibility between the two circuits [72].
Revised: To this end, we need to introduce the notion of compatibility between the two circuits [74].

Original: Deﬁnition 3.7 (Compatible circuits in SPLs) .
Revised: Deﬁnition 3.7 (Compatible circuits in SPLs) .

Original: A smooth and decomposable conditional PC q ( Y ; Θ )is compatible over variables Y with a smooth and decomposable constraint circuit c K ( Y , X ) ifany pair of product units n ∈ q and m ∈ c K with the same scope over Y can be rearrangedto be mutually compatible and decompose in the same way: ( sc ( n ) =
Revised: A smooth and decomposable conditional PC q ( Y ; Θ ) is compatible over variables Y with a smooth and decomposable constraint circuit c K ( Y , X ) if any pair of product units n ∈ q and m ∈ c K with the same scope over Y can be rearranged to be mutually compatible and decompose in the same way: ( sc ( n ) =

Original: sc ( m ))
Revised: sc ( m ))

Original: =
Revised: =

Original: ⇒ ( sc ( n i ) =
Revised: ⇒ ( sc ( n i ) =

Original: sc ( m i ) , n i and m i are compatible ) for some rearrangement of the inputs of n (resp.
Revised: sc ( m i ) , n i and m i are compatible ) for some rearrangement of the inputs of n (resp.

Original: m ) into n 1 , n 2(resp.
Revised: m ) into n 1 , n(resp.

Original: m 1 , m 2 ).
Revised: m 1 , m 2 ).

Original: Fig.
Revised: Fig.

Original: 3 shows two circuits q
Revised: 3 shows two circuits q

Original: and c that are compatible.
Revised: and c that are compatible.

Original:  Theorem 3.1 (Efﬁcient inference in SPLs) .
Revised: [Table] Theorem 3.1 (Efﬁcient inference in SPLs) .

Original: If q ( Y ; Θ ) and c K ( Y , X ) are two smooth, decomposableand compatible circuits, then computing Eq.
Revised: If q ( Y ; Θ ) and c K ( Y , X ) are two smooth, decomposable and compatible circuits, then computing Eq.

Original: (2) can be done in O ( | q || c | ) time.
Revised: (2) can be done in O ( | q || c | ) time.

Original: Furthermore, if they217 are also deterministic, then computing the MAP state can be done in
Revised: Furthermore, if they are also deterministic, then computing the MAP state can be done in

Original: O ( | q || c | ) time.
Revised: O ( | q || c | ) time.

Original: The proof can be found in Sec.
Revised: The proof can be found in Sec.

Original: A.
Revised: A.

Original: How do we come up with compatible circuits?
Revised: How do we come up with compatible circuits?

Original: One option is tohave a PC q that is compatible with every possible smooth and decomposable circuit c .
Revised: One option is to have a PC q that is compatible with every possible smooth and decomposable circuit c .

Original: To do so,we can represent q as a mixtures of M fully-independent models; i.e., (cid:80) Mi
Revised: To do so, we can represent q as a mixtures of M fully-independent models; i.e., (cid:80) Mi

Original: =1 ω i (cid:81) j q ( Y j ; Θ i ) .
Revised: =1 ω i (cid:81) j q ( Y j ; Θ i ) .

Original: Thisadditional sum unit can be enough to be more expressive than a FIL and already delivers more accuratepredictions than any competitor, as our experiments in pathﬁnding show (Sec. 5).
Revised: This additional sum unit can be enough to be more expressive than a FIL and already delivers more accurate predictions than any competitor, as our experiments in pathﬁnding show (Sec. 5).

Original: An example of sucha circuit is shown in Fig.
Revised: An example of such a circuit is shown in Fig.

Original: Another sufﬁcient condition for compatibility is that both q and c share theexact same hierarchical scope partitioning [72], sometimes called a vtree or variable ordering [12, 56].
Revised: Another sufﬁcient condition for compatibility is that both q and c share the exact same hierarchical scope partitioning [74], sometimes called a vtree or variable ordering [12, 58].

Original: This can be done easily if one ﬁrst compiles logical constraints into OBDDs or SDDs and then uses amechanized algorithm to build q as in [55] to create a compatible structure.
Revised: This can be done easily if one ﬁrst compiles logical constraints into OBDDs or SDDs and then uses a mechanized algorithm to build q as in [57] to create a compatible structure.

Original: Additionally, to ensure qis a deterministic PC, we could exploit the mechanized construction proposed in Shih and Ermon [66].
Revised: Additionally, to ensure q is a deterministic PC, we could exploit the mechanized construction proposed in Shih and Ermon [68].

Original: Computing the exact MAP state, however, is of less concern as approximate inference algorithms,e.g., beam search decoding [76] or iterative pruning [11], are nowadays a commodity in deep learningframeworks.
Revised: Computing the exact MAP state, however, is of less concern as approximate inference algorithms, e.g., beam search decoding [78] or iterative pruning [11], are nowadays a commodity in deep learning frameworks.

Original: For non-deterministic PCs, we compute the MAP state with a faster approximation byreplacing non-deterministic sum units with max units [52].
Revised: For non-deterministic PCs, we compute the MAP state with a faster approximation by replacing non-deterministic sum units with max units [54].

Original: This runs in time linear in the size of r ,and yet delivers state-of-the-art accuracies in our experiments Sec. 5.
Revised: This runs in time linear in the size of r , and yet delivers state-of-the-art accuracies in our experiments Sec. 5.

Original: 3.4 A single-circuit SPL
Revised: 3.4 A single-circuit SPL

Original: The two-circuit design we proposed for SPLs provides a clear and theoretically-backed interfacebetween neural networks and probabilistic and symbolic reasoning.
Revised: The two-circuit design we proposed for SPLs provides a clear and theoretically-backed interface between neural networks and probabilistic and symbolic reasoning.

Original: This setup, however, cansometimes be wasteful, as it requires to compute the product of two circuits and renormalize.
Revised: This setup, however, can sometimes be wasteful, as it requires to compute the product of two circuits and renormalize.

Original: Wecircumvent this issue by designing a single-circuit implementation of SPL.
Revised: We circumvent this issue by designing a single-circuit implementation of SPL.

Original: [Table]

Original: Deﬁnition 3.8 (Single-circuit SPL) .
Revised: Deﬁnition 3.8 (Single-circuit SPL) .

Original: Given an input conﬁguration x , a single-circuit SPL computesp ( y | x ) =
Revised: Given an input conﬁguration x , a single-circuit SPL computes p ( y | x ) =

Original: c K ( Y , X ; Ω = g ( f ( X )))
Revised: c K ( Y , X ; Ω = g ( f ( X )))

Original: where c K is a neural conditional constraint circuit whosesum-unit parameters Ω are non-unitary values parameterized via a gating function g .
Revised: where c K is a neural conditional constraint circuit whose sum-unit parameters Ω are non-unitary values parameterized via a gating function g .

Original: In a nutshell, we can directly realize SPL by compiling a complex logical constraints (D4) into adeterministic constraint circuit c K , as before, and then parameterizing it with a gating function ofthe network embeddings f ( X ) , i.e., allowing its sum units to be non-unitary and input dependent.
Revised: In a nutshell, we can directly realize SPL by compiling a complex logical constraints (D4) into a deterministic constraint circuit c K , as before, and then parameterizing it with a gating function of the network embeddings f ( X ) , i.e., allowing its sum units to be non-unitary and input dependent.

Original: Since the support of c K is already restricted to exactly match the constraint K (D3), parameterizing Ωinduces an expressive probability distribution over the label conﬁgurations that are consistent with
Revised: Since the support of c K is already restricted to exactly match the constraint K (D3), parameterizing Ω induces an expressive probability distribution over the label conﬁgurations that are consistent with

Original: K (D2).
Revised: K (D2).

Original: We can further guarantee that the circuit’s output are normalized probabilities (D1,D6) byenforcing the parameters ω of each sum unit to form a convex combination [54].
Revised: We can further guarantee that the circuit’s output are normalized probabilities (D1,D6) by enforcing the parameters ω of each sum unit to form a convex combination [56].

Original: This can be easilydone by utilizing a softmax activation function for g .
Revised: This can be easily done by utilizing a softmax activation function for g .

Original: One of the advantages of the two-circuit implementation of SPLs is that the size of the circuitq Θ can be easily increased to improve the capacity of the model (Sec. 3.1).
Revised: One of the advantages of the two-circuit implementation of SPLs is that the size of the circuit q Θ can be easily increased to improve the capacity of the model (Sec. 3.1).

Original: The single-circuitimplementation is not as ﬂexible, as normally the number of parameters is determined by thecomplexity of the constraint circuit, which depends entirely on the compilation step.
Revised: The single-circuit implementation is not as ﬂexible, as normally the number of parameters is determined by the complexity of the constraint circuit, which depends entirely on the compilation step.

Original: In this case,one option is to overparameterize the neural conditional circuit by introducing additional sum units,hence allowing it to capture more modes in the distribution.
Revised: In this case, one option is to overparameterize the neural conditional circuit by introducing additional sum units, hence allowing it to capture more modes in the distribution.

Original: We detail this process is Sec.
Revised: We detail this process is Sec.

Original: C. A sideeffect of overparameterization is that it relaxes determinism, meaning that MAP inference needs tobe approximated, as described in Sec. 3.3.
Revised: C. A side effect of overparameterization is that it relaxes determinism, meaning that MAP inference needs to be approximated, as described in Sec. 3.3.

Original: Additionally, training a gating function to map relativelysmall embeddings to large parameter vectors in overparameterized circuits, can slow down training.
Revised: Additionally, training a gating function to map relatively small embeddings to large parameter vectors in overparameterized circuits, can slow down training.

Original: In such cases, a two-circuit implementation of SPL is to be preferred .
Revised: In such cases, a two-circuit implementation of SPL is to be preferred.

Original:  Related works
Revised:  Related works

Original: In this section, we position SPLs against state-of-the-art approaches for enforcing constraints onneural network predictions.
Revised: In this section, we position SPLs against state-of-the-art approaches for enforcing constraints on neural network predictions.

Original: In-depth surveys on this topic can be found in [18] and [33].
Revised: In-depth surveys on this topic can be found in [18] and [33].

Original: Energy-based models.
Revised: Energy-based models.

Original: Deep energy-based models (EBMs) replace FILs with an unnormalizedfactor graph [40] that captures higher-order label dependencies [41] (D2) but at the cost of foregoingprobabilistic semantics (D1) and efﬁciency (D6).
Revised: Deep energy-based models (EBMs) replace FILs with an unnormalized factor graph [41] that captures higher-order label dependencies [42] (D2) but at the cost of foregoing probabilistic semantics (D1) and efﬁciency (D6).

Original: EBMs are typically unconcerned with hard constraints (D3).
Revised: EBMs are typically unconcerned with hard constraints (D3).

Original: Neural approaches for segmentation [45] and parsing [27, 79, 80] remedy to this byreplacing the factor graph with a full-ﬂedged intractable (discriminative) graphical model [40].
Revised: Neural approaches for segmentation [46] and parsing [27, 81, 82] remedy to this by replacing the factor graph with a full-ﬂedged intractable (discriminative) graphical model [41].

Original: Togain efﬁciency, one can restrict EBMs to simpler graphical models (e.g., chains, trees), compromisingexpressiveness (D2) and the ability to model non-trivial logical constraints (D3, D4).
Revised: To gain efﬁciency, one can restrict EBMs to simpler graphical models (e.g., chains, trees), compromising expressiveness (D2) and the ability to model non-trivial logical constraints (D3, D4).

Original: Loss-based methods.
Revised: Loss-based methods.

Original: A prominent strategy consists of penalizing the network for producing inconsistent predictions using an auxiliary loss [18, 33].
Revised: A prominent strategy consists of penalizing the network for producing inconsistent predictions using an auxiliary loss [18, 33].

Original: While popular, loss-based methods, howevercannot guarantee that the predictions will be consistent at test time.
Revised: While popular, loss-based methods, however cannot guarantee that the predictions will be consistent at test time.

Original: Common losses include translating logical constraints into a differentiable fuzzy logic [24, 25], as exempliﬁed by DL2 [28].
Revised: Common losses include translating logical constraints into a differentiable fuzzy logic [24, 25], as exempliﬁed by DL2 [28].

Original: Althoughefﬁcient (D6), this solution is not probabilistically sound (D1) and crucially is not syntax-invariant :different encodings of the same formula (e.g., conjunctive vs. disjunctive normal form) yield differentlosses [30, 23].
Revised: Although efﬁcient (D6), this solution is not probabilistically sound (D1) and crucially is not syntax-invariant : different encodings of the same formula (e.g., conjunctive vs. disjunctive normal form) yield different losses [30, 23].

Original: Closer to our SPL, the Semantic Loss (SL) [77] avoids this issue by penalizing thethe probability θ i associated to the i -th label by the neural network via the loss term
Revised: Closer to our SPL, the Semantic Loss (SL) [79] avoids this issue by penalizing the the probability θ i associated to the i -th label by the neural network via the loss term

Original: [Equation]
Revised: [Equation]

Original: When K is compiled into a constraint circuit c K one retrieves −Z ( x ) for a two-circuit version of SPLthat is as expressive as FIL as it assumes independent labels via a conditional PC (cid:81)
Revised: When K is compiled into a constraint circuit c K one retrieves −Z ( x ) for a two-circuit version of SPL that is as expressive as FIL as it assumes independent labels via a conditional PC (cid:81)

Original: i q ( Y i ; θ i ) .
Revised: i q ( Y i ; θ i ) .

Original: Theneuro-symbolic entropy (N E S Y E NT )
Revised: The neuro-symbolic entropy (N E S Y E NT )

Original: [2] extends L SL by an entropy term that improves (but still doesnot guarantee) consistency.
Revised: [2] extends L SL by an entropy term that improves (but still does not guarantee) consistency.

Original: It still makes the same independence assumptions over labels (D2).
Revised: It still makes the same independence assumptions over labels (D2).

Original: Predicted paths do not exactly match the ground truth are still valid paths and yield very close costs to the ground truth.

Original: Competitors’ predictions can have higher Hamming scores but be invalid.

Original: More examples in Sec.

Original: D.3.

Original: [Table]

Original: Consistency layers.
Revised: Consistency layers.

Original: Approaches ensuring consistency by embedding the constraints into the pre- dictive layer as in SPLs include MultiplexNet [36] and HMCCN [31].
Revised: Approaches ensuring consistency by embedding the constraints into the predictive layer as in SPLs include MultiplexNet [37] and HMCCN [31].

Original: MultiplexNet is able toencode only constraints in disjunctive normal form, which is problematic for generality (D4) andefﬁciency (D6) as neuro-symbolic SOP tasks involve an intractably large number of clauses – e.g. ourpathﬁnding experiments involves billions of clauses.
Revised: MultiplexNet is able to encode only constraints in disjunctive normal form, which is problematic for generality (D4) and efﬁciency (D6) as neuro-symbolic SOP tasks involve an intractably large number of clauses – e.g. our pathﬁnding experiments involves billions of clauses.

Original: HMCCN encodes label dependencies as fuzzyrelaxation and is the current state-of-the-art model for HMLC [31].
Revised: HMCCN encodes label dependencies as fuzzy relaxation and is the current state-of-the-art model for HMLC [31].

Original: Even its recent extension [32] isrestricted to a certain family of constraints (D4) that can be represented with fuzzy logic.
Revised: Even its recent extension [32] is restricted to a certain family of constraints (D4) that can be represented with fuzzy logic.

Original: Other approaches.
Revised: Other approaches.

Original: Other common approaches to neuro-symbolic SOP require to invoke a solver toeither obtain the MAP state or to compute (often only approximately) the gradient of the loss [22, 57,50 ].
Revised: Other common approaches to neuro-symbolic SOP require to invoke a solver to either obtain the MAP state or to compute (often only approximately) the gradient of the loss [22, 59, 51].

Original: SPLs have no such requirement.
Revised: SPLs have no such requirement.

Original: Some neuro-symbolic approaches [63] constrain the outputs ofneural networks within complex logical reasoning pipelines to solve tasks harder than neuro-symbolic
Revised: Some neuro-symbolic approaches [65] constrain the outputs of neural networks within complex logical reasoning pipelines to solve tasks harder than neuro-symbolic

Original: SOP.
Revised: SOP.

Original: For instance, DeepProblog [46] uses Prolog’s backward chaining algorithm for ﬁrst orderlogical rules whose probabilistic weights are predicted by the network.
Revised: For instance, DeepProblog [47] uses Prolog’s backward chaining algorithm for ﬁrst order logical rules whose probabilistic weights are predicted by the network.

Original: In modern implementations of
Revised: In modern implementations of

Original: Problog, grounding a ﬁrst order program and then compiling it into constraint circuits [26] producesa conditional circuit akin to those we use in SPLs, but in which ( i ) only input distributions areparameterized and ( ii ) increasing the parameter count is not considered a straightforward operation.
Revised: Problog, grounding a ﬁrst order program and then compiling it into constraint circuits [26] produces a conditional circuit akin to those we use in SPLs, but in which ( i ) only input distributions are parameterized and ( ii ) increasing the parameter count is not considered a straightforward operation.

Original: Scallop [37] provides a more scalable approach to deepproblog by considering only the top- k proofs.
Revised: Scallop [38] provides a more scalable approach to deepproblog by considering only the top- k proofs.

Original: We leave to future work how we could quickly compile only a speciﬁc query as DeepProblog/Scallopdo, to deal with ﬁrst-order representations efﬁciently
Revised: We leave to future work how we could quickly compile only a speciﬁc query as DeepProblog/Scallop do, to deal with ﬁrst-order representations efﬁciently

Original:  Experiments
Revised:  Experiments

Original: We evaluate SPLs on standard neuro-symbolic SOP benchmarks such as simple path prediction ,preference learning [77], shortest path ﬁnding in Warcraft [57] and HMLC [31].
Revised: We evaluate SPLs on standard neuro-symbolic SOP benchmarks such as simple path prediction , preference learning [79], shortest path ﬁnding in Warcraft [59] and HMLC [31].

Original: We compare SPLsagainst several state-of-the-art loss- and layer-based approaches (Sec. 4) by applying them to thesame base neural network architecture as feature extractor f .
Revised: We compare SPLs against several state-of-the-art loss- and layer-based approaches (Sec. 4) by applying them to the same base neural network architecture as feature extractor f .

Original: As we are interested in measuring howclose to the ground truth and how safe the predictions of all models are, we report the percentageof E XACT matches of the predicted labels, also called subset accuracy [71], and the percentage of
Revised: As we are interested in measuring how close to the ground truth and how safe the predictions of all models are, we report the percentage of E XACT matches of the predicted labels, also called subset accuracy [73], and the percentage of

Original: C ONSISTENT predictions, also called “Constraint” [77].
Revised: C ONSISTENT predictions, also called “Constraint” [79].

Original: Note that, like other consistency layers,
Revised: Note that, like other consistency layers,

Original: SPLs are guaranteed to always output 100% consistent predictions.
Revised: SPLs are guaranteed to always output 100% consistent predictions.

Original: Additionally, we report the 
Revised: Additionally, we report the ground valid and predictions can have higher Hamming scores but be invalid. More examples in

Original:  H AMMING score [71], mainly to maintain compatibility with previous experimental settings [77, 2].
Revised: [Table] [Table] H AMMING score [73], mainly to maintain compatibility with previous experimental settings [79, 2].

Original: This metric does not consider consistency of predictions and naturally favors competitors that assumelabel independence and thus can minimize the per-label cross-entropy [21] (Table 3).
Revised: This metric does not consider consistency of predictions and naturally favors competitors that assume label independence and thus can minimize the per-label cross-entropy [21] (Table 3).

Original: Sec.
Revised: Sec.

Original: D collectsall experimental details such as architectures and hyperparameters used for each experiment.
Revised: D collects all experimental details such as architectures and hyperparameters used for each experiment.

Original: Simple path prediction & preference learning.
Revised: Simple path prediction & preference learning.

Original: We start by comparing SPLs against loss-basedapproaches, reproducing the neuro-symbolic benchmarks of Xu et al.
Revised: We start by comparing SPLs against loss-based approaches, reproducing the neuro-symbolic benchmarks of Xu et al.

Original: [77] for simple path predictionand preference learning.
Revised: [79] for simple path prediction and preference learning.

Original: In the ﬁrst experiment, given a source and destination node in an unweightedgrid G =
Revised: In the ﬁrst experiment, given a source and destination node in an unweighted grid G =

Original: ( V, E ) , the neural net needs to ﬁnd the shortest unweighted path connecting them.
Revised: ( V, E ) , the neural net needs to ﬁnd the shortest unweighted path connecting them.

Original: Weconsider a 4 × 4 grid.
Revised: We consider a 4 × 4 grid.

Original: The input ( x , y ) is a binary vector of length | V | + | E | , with the ﬁrst | V |318 variables indicating the source and destination nodes, and the subsequent | E | variables indicating a319 subgraph G (cid:48)
Revised: The input ( x , y ) is a binary vector of length | V | + | E | , with the ﬁrst | V | variables indicating the source and destination nodes, and the subsequent | E | variables indicating a subgraph G (cid:48)

Original: ⊆ G .
Revised: ⊆ G .

Original: Each label is a binary vector of length | E | encoding the unique shortest path in
Revised: Each label is a binary vector of length | E | encoding the unique shortest path in

Original: G (cid:48) .
Revised: G (cid:48) .

Original: For each example, we obtain G (cid:48) by dropping one third of the edges in the graph G uniformly atrandom, ﬁlter out the connected components with fewer than 5 nodes, to reduce degenerate cases,and then sample a source and destination node uniformly at random from G (cid:48) .
Revised: For each example, we obtain G (cid:48) by dropping one third of the edges in the graph G uniformly at random, ﬁlter out the connected components with fewer than 5 nodes, to reduce degenerate cases, and then sample a source and destination node uniformly at random from G (cid:48) .

Original: The dataset consists of1600 such examples, with a 60 / 20 / 20 train/validation/test split.
Revised: The dataset consists of 1600 such examples, with a 60 / 20 / 20 train/validation/test split.

Original: In the preference learning task, given a user’s ranking over a subset of items, the network has topredict the user’s ranking over the remaining items.
Revised: In the preference learning task, given a user’s ranking over a subset of items, the network has to predict the user’s ranking over the remaining items.

Original: We encode an ordering over n items as a binarymatrix Y ij , where for each i, j ∈ 1 , . . .
Revised: We encode an ordering over n items as a binary matrix Y ij , where for each i, j ∈ 1 , . . .

Original: , n , Y ij indicates whether item i is the j th element in the 
Revised: , n , Y ij indicates whether item i is the j th element in the ordering.

Original: [Table] ordering. The input x consist of the user’s preference over 6 sushi types, and the model has to predictthe user’s preferences (a strict total order) over the remaining 4 .
Revised: The input x consist of the user’s preference over 6 sushi types, and the model has to predict the user’s preferences (a strict total order) over the remaining 4 .

Original: We use preference ranking data over10 types of sushi for 5 , 000 individuals, taken from [47], and a 60 / 20 / 20 split.
Revised: We use preference ranking data over 10 types of sushi for 5 , 000 individuals, taken from [48], and a 60 / 20 / 20 split.

Original: We employ a 5-layer and 3-layer MLP as a baseline for the simple path prediction, and preference learning, respectively, equipped with FIL layer and additionally with the Semantic Loss [77](MLP+ L SL ) or its entropic extension [2] (MLP+N E S Y E NT ).
Revised: We employ a 5-layer and 3-layer MLP as a baseline for the simple path prediction, and preference learning, respectively, equipped with FIL layer and additionally with the Semantic Loss [79] (MLP+ L SL ) or its entropic extension [2] (MLP+N E S Y E NT ).

Original: We compile the logical constraints intoan SDD [16] and then turn it into a the same constraint circuit c K that is used for L SL , N E S Y E NT(Sec. 4) and our 1-circuit implementation of SPLs.
Revised: We compile the logical constraints into an SDD [16] and then turn it into a the same constraint circuit c K that is used for L SL , N E S Y E NT (Sec. 4) and our 1-circuit implementation of SPLs.

Original: Table 2 clearly shows that the increased expressiveness of SPL, coming from overparameterizing c K , allows to outperform all competitors whileguaranteeing consistent predictions, as expected.
Revised: Table 2 clearly shows that the increased expressiveness of SPL, coming from overparameterizing c K , allows to outperform all competitors while guaranteeing consistent predictions, as expected.

Original: Warcraft Shortest Path.
Revised: Warcraft Shortest Path.

Original: Next, we evaluate SPL on the more challenging task of predicting theminimum cost path in a weighted 12 × 12 grid imposed over terrain maps of Warcraft II [57].
Revised: Next, we evaluate SPL on the more challenging task of predicting the minimum cost path in a weighted 12 × 12 grid imposed over terrain maps of Warcraft II [59].

Original: Eachvertex is assigned a cost corresponding to the type of the underlying terrain (e.g., earth has lower costthan water).
Revised: Each vertex is assigned a cost corresponding to the type of the underlying terrain (e.g., earth has lower cost than water).

Original: The minimum cost path between the top left and the bottom right vertices of the grid isencoded as an indicator matrix, and serves as a label.
Revised: The minimum cost path between the top left and the bottom right vertices of the grid is encoded as an indicator matrix, and serves as a label.

Original: As in [57] we use a ResNet18 [35] with FILoptionally with L SL as a baseline.
Revised: As in [59] we use a ResNet18 [35] with FIL optionally with L SL as a baseline.

Original: Given the largest size of the compiled constraint circuit c K in thiscase 10 10 , we use a two-circuit implementation of SPL.
Revised: Given the largest size of the compiled constraint circuit c K in this case 10 10 , we use a two-circuit implementation of SPL.

Original: Results in Fig.
Revised: Results in Fig.

Original: 1 and Table 3 are striking: notonly SPL outperforms competitors by a large margin – approx.
Revised: 1 and Table 3 are striking: not only SPL outperforms competitors by a large margin – approx.

Original: +23 % over FIL and +19 % over the
Revised: +23 % over FIL and +19 % over the

Original: SL – but also consistently delivers meaningful paths that are very close to the ground truth in termsof cost, even when they encode very different routes.
Revised: SL – but also consistently delivers meaningful paths that are very close to the ground truth in terms of cost, even when they encode very different routes.

Original: See Sec.
Revised: See Sec.

Original: D.3 for a gallery of these examples.
Revised: D.3 for a gallery of these examples.

Original: Hierarchical Multi-Label Classiﬁcation.
Revised: Hierarchical Multi-Label Classiﬁcation.

Original: Lastly, we follow the experimental setup of Giunchigliaand Lukasiewicz [31] and evaluate SPL on 12 real-world HMLC tasks spanning four different domains: 8 functional genomics, 2 medical images, 1 microalgea classiﬁcation, and 1 text categorization.
Revised: Lastly, we follow the experimental setup of Giunchiglia and Lukasiewicz [31] and evaluate SPL on 12 real-world HMLC tasks spanning four different domains: 8 functional genomics, 2 medical images, 1 microalgea classiﬁcation, and 1 text categorization.

Original: Fig.
Revised: Fig.

Original: 3 shows an example of a hierarchy of classes.
Revised: 3 shows an example of a hierarchy of classes.

Original: These tasks are especially challenging due to thelimited number of training samples, the large number of output classes, ranging from 56 to 499 , aswell as the sparsity of the output space.
Revised: These tasks are especially challenging due to the limited number of training samples, the large number of output classes, ranging from 56 to 499 , as well as the sparsity of the output space.

Original: For numeric features we replaced missing values by theirmean, and for categorical features by a vector of zeros, and standardized all features.
Revised: For numeric features we replaced missing values by their mean, and for categorical features by a vector of zeros, and standardized all features.

Original: We used thevalidation splits to determine the number of layers in the gating function as well as the overparameterization, keeping all other hyperparameters ﬁxed.
Revised: We used the validation splits to determine the number of layers in the gating function as well as the overparameterization, keeping all other hyperparameters ﬁxed.

Original: The ﬁnal models were obtained by trainingusing a batch size of 128 and early stopping on the validation set.
Revised: The ﬁnal models were obtained by training using a batch size of 128 and early stopping on the validation set.

Original: We compare our single-circuit SPLagainst HMCNN which was shown to outperform several other state-of-the-art HMLC approaches in
Revised: We compare our single-circuit SPL against HMCNN which was shown to outperform several other state-of-the-art HMLC approaches in

Original: Giunchiglia and Lukasiewicz [31].
Revised: Giunchiglia and Lukasiewicz [31].

Original: We study the effect of increasing the expressivenss of SPL viaoverparameterization in Sec.
Revised: We study the effect of increasing the expressivenss of SPL via overparameterization in Sec.

Original: D.4.
Revised: D.4.

Original: The results in Table 5 highlight that SPL signiﬁcantly outperforms
Revised: The results in Table 5 highlight that SPL signiﬁcantly outperforms

Original: HMCNN in terms of exact match on 11 data sets performing comparably on 1 ,
Revised: HMCNN in terms of exact match on 11 data sets performing comparably on 1 ,

Original:  Conclusion
Revised:  Conclusion

Original: SPLs offer the ﬁrst clear interface for integrating complex probabilitistic reasoning and logicalconstraints on top of any neural network classiﬁer while retaining efﬁcient inference and training.
Revised: SPLs offer the ﬁrst clear interface for integrating complex probabilitistic reasoning and logical constraints on top of any neural network classiﬁer while retaining efﬁcient inference and training.

Original: They improve by a noticeable margin the current state-of-the-art on challenging neuro-symbolic
Revised: They improve by a noticeable margin the current state-of-the-art on challenging neuro-symbolic

Original: SOP benchmarks such as pathﬁnding and HMLC.
Revised: SOP benchmarks such as pathﬁnding and HMLC.

Original: This opens up a number of interesting researchdirections.
Revised: This opens up a number of interesting research directions.

Original: First, SPLs can be extended to incorporate logical constraints over multiple networksand representable by ﬁrst-order formulas [46], which we plan to explore in future works, making thecircuit construction pipeline totally transparent to users [1] while possibly automatically learningconstraints from data [20, 48].
Revised: First, SPLs can be extended to incorporate logical constraints over multiple networks and representable by ﬁrst-order formulas [47], which we plan to explore in future works, making the circuit construction pipeline totally transparent to users [1] while possibly automatically learning constraints from data [20, 49].

Original: Second, we are interested in leveraging SPLs to inject scalable logicalconstraints into large language models [6] thus equipping them with probabilistic reasoning [29, 78].
Revised: Second, we are interested in leveraging SPLs to inject scalable logical constraints into large language models [6] thus equipping them with probabilistic reasoning [29, 80] Laguerrapiutotale . Acknowledgments and Disclosure of Funding

Original: Checklist
Revised: The authors would like to thank Arthur Choi for helpful discussions on compiling the constraints for the Warcraft Shortest Path task, and Andreas Grivas for proofreading a draft manuscript. The research of ST was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. This work was funded in part by the DARPA Perceptually-enabled Task Guidance (PTG) Program under contract number

Revised: HR00112220005„ NSF grants #IIS-1943641, #IIS-1956441, #CCF-1837129, Samsung, CISCO and a Sloan Fellowship.

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data are proprietary.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] Please see section 1 for a summary of our contributions (b) Did you describe the limitations of your work?
Revised: [Yes] Please see section 1 for a summary of our contributions (b) Did you describe the limitations of your work?

Original: [Yes] Please see Supplementary (c) Did you discuss any potential negative societal impacts of your work?[Yes] Please see Supplementary
Revised: [Yes] Please see Supplementary (c) Did you discuss any potential negative societal impacts of your work?

Original:  (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] Please see Supplementary (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?[Yes] Please see
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original:  supplementary material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] Please see supplementary material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Please see section 5 for details regarding the dataset splits.
Revised: [Yes] Please see section 5 for details regarding the dataset splits.

Original: All other details are included in the supplementary.
Revised: All other details are included in the supplementary.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We reported error bars and signiﬁcance testing for the hierarchical multi-label classiﬁcation experiment, please see
Revised: [Yes] We reported error bars and signiﬁcance testing for the hierarchical multi-label classiﬁcation experiment, please see(d)

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Please see supplementary material 
Revised: [Yes] Please see supplementary material 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?[N/A]
Revised: [N/A] (b) Did you mention the license of the assets?

Original:  (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review 
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: Board
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: c) Did you include the estimated hourly wage paid to participants and  the total amount
Revised: Theorem 3.1 (Efﬁcient inference in SPLs). If q ( Y ; Θ ) and c K ( Y , X ) are two smooth, decomposable and compatible circuits, then computing Eq. (2) can be done in O ( | q || c | ) time, where | · | denotes the circuit size.

Original: spent on participant compensation?
Revised: Furthermore, if they are also deterministic, then computing the MAP state can be done in O ( | q || c | ) time. . We prove the ﬁrst statement by ﬁrst showing that the partition function Z ( x ) in Eq. (2) can solved exactly in time O ( | q || c | ) .

Revised: It will then follow from it that computing Eq.

Revised: (2) can be done in O ( | q || c | + | q | + | c | )

Revised: ≈ O ( | q || c | ) where the last two additive factors derive from evaluating q and c for an input conﬁguration ( x , y ) .

Revised: To do so, we will exploit two ingredients: i) the product of q and c can be represented as a smooth and decomposable circuit in time

Revised: O ( | q || c | ) [74] and ii) any smooth and decomposable circuit guarantees tractable marginalization in time linear in its size [12].

Revised: The next two propositions formalize these statements.

Revised: Proposition A.1 (Tractable product of circuits) .

Revised: Let q ( Y ; Θ ) and c K ( Y , X ) be two smooth, decomposable circuits that are compatible over Y then computing their product as a circuit r Θ , K

Revised: ( X , Y )

Revised: =

Revised: q ( Y ; Θ ) · c K ( Y , X ) that is decomposable over Y can be done in O ( | q || c | ) .

Revised: If both q and c are also deterministic, then r is as well.

Revised: Proof.

Revised: The proof directly follows from Theorem 3.2 from Vergari et al.

Revised: Note that O ( | q || c | ) is a loose upperbound and the size of r is in practice smaller [74]. ??

Revised: shows as an example the circuit obtained by multiplying the PC and the constraint circuit in Fig.

Revised: Proposition A.2 (Tractable marginalization of circuits) .

Revised: Let r ( X , Y ) be a circuit that is smooth and decomposable over Y with input functions over Y that can be tractably marginalized out.

Revised: Then for any variables Y (cid:48) ⊆

Revised: Y and their assignment y (cid:48) , the marginalization (cid:80) y (cid:48) r ( y (cid:48) , y (cid:48)(cid:48) , x ) can be computed exactly in time linear in the size of r , where Y (cid:48)(cid:48) =

Revised: Y \ Y (cid:48) .

Revised: Proof.

Revised: The proof follows by considering that i) the input functionals in SPLs are simple distributions such as Bernoullis and indicators and can be easily marginalized in O (1) and ii) that for every conﬁguration x of variables X , r ( Y , x ) is a circuit only over Y and therefore Proposition 2.1 from

Revised: Vergari et al.

Revised: [74] can be directly applied.

Revised: Analogously, the second statement of Theorem 3.1 follows from Proposition A.1 and by recalling that the MAP state of a deterministic circuit can be computed in time linear in its size.

Revised: Proposition A.3 (Tractable MAP state of circuits (Choi et al.

Revised: Let r ( X , Y ) be a circuit that is smooth and decomposable and deterministic over Y then for a conﬁguration x its MAP state argmax y r ( x , y ) can be computed in time O

Revised: ( | r | ) .

Revised: B Compiling logical formulas into circuits

Revised: For our experiments we use standard compilation tools to obtain a constraint circuit starting from a propositional logical formula in conjunctive normal form.

Revised: Speciﬁcally, we use Graphillion 1 to compile the constraints in the Warcraft pathﬁnding experiment into an SDD.

Revised: For all other experiments, we use PySDD 2 a python SDD compiler [16, 9].

Revised: We now illustrate step-by-step one example of such a compilation for a simple logical formula.

Revised: Consider the constraint circuit c in Fig.

Revised: 3 encoding the constraint

Revised: [Equation]

Revised: Intuitively, our aim is to compile the above logical formula into a compact form representing all possible assignments to Y cat , Y dog , Y animal satisfying the above constraint.

Revised: We compile such a constraint by proceeding in a bottom up fashion, where bottom-up compilation can be seen as composing Boolean sub-functions whose domain is determined by a variable ordering, also called vtree (see Sec. 3.3).

Revised: In this example, we assume the function f

Revised: ( Y animal , Y cat , Y dog ) decomposes as f 1 ( Y animal ) · f 2 ( Y dog ) · f 3 ( Y cat )

Revised: We therefore start by compiling a constraint circuit that is a function of Y cat and Y dog , and compose it with a constraint circuit that is a function of Y animal

Revised: We ﬁrst introduce input functionals representing indicators associated with Y cat , Y dog , Y animal .

Revised: We will denote by Y i the indicator 1 { Y i = 1 } and by ¬ Y

Revised: i the indicator 1 { Y i = 0 } .

Revised: [Equation]

Revised: We start by disjoining the indicators Y cat with ¬ Y cat , and Y dog with ¬ Y dog .

Revised: This corresponds to introducing deterministic and smooth sum units in our circuits.

Revised: [Equation]

Revised: These units represent disjoint solutions to the logical formula, meaning there exists distinct assignments, characterized by the children, that satisfy the logical constraint e.g. Y cat , Y dog , Y animal and

Revised: Y cat , ¬ Y dog , Y animal are two distinct assignments that satisfy the logical constraint.

Revised: The compilation process proceeds by conjoining the constraint circuits for Y dog ∨ ¬ Y dog with Y cat ,

Revised: Y dog with Y cat ∨ ¬ Y cat , and ¬ Y dog with ¬ Y cat .

Revised: [Equation]

Revised: A decomposable product units composes functions over disjoint sets of variables.

Revised: The above three product nodes represent the Boolean functions ( Y dog ∨ ¬ Y dog )

Revised: ∧ Y cat , Y dog ∧ ( Y cat ∨ ¬ Y cat ) , and ¬ Y dog ∧ ¬ Y cat .

Revised: We again disjoin ( Y dog ∨ ¬ Y dog )

Revised: ∧ Y cat with Y dog ∧ ( Y cat ∨ ¬ Y cat ) , and ¬ Y dog ∧ ¬ Y cat with true , the logical multiplicative identity, guaranteeing alternating sum and product nodes, as mentioned in

Revised: Sec. 3.1.

Revised: So far, we have compiled constraint circuits for the logical formula

Revised: [Equation]

Revised: and the logical formula

Revised: [Equation]

Revised: What remains is to conjoin Eq.

Revised: (4) with Y animal , and Eq.

Revised: (5) with ¬ Y animal , and disjoin the resulting constraint circuits.

Revised: What we get is a a mixture distribution over the possible solutions of the constraint:

Revised: If we predict there is a dog or a cat, or both, in e.g., an image, we better predict that there’s an animal.

Revised: On the other hand, the absence of a dog and a cat from an image implies nothing as to the presence of an animal in the image.

Revised: 1 { Y 1 = 0 } 1 { Y 1 = 1 } 1 { Y 2 = 0 } 1 { Y 2 = 1 } × × × × 1 { Y 3 = 1 } 1 { Y 3 = 0 } ×

Revised: Compilation techniques like the one we illustrated do not, however, escape the hardness of the problem: the compiled circuit can be exponential in the size of the constraint, in the worst case .

Revised: In practice , nevertheless, we can obtain compact circuits because real-life logical constraints exhibit enough structure (e.g., they encode repeated sub-problems) that can be easily exploited by a compiler.

Revised: We refer to the literature of compilation for details on this [17].

Revised: C Overparameterizing the single-circuit SPL

Revised: As mentioned in Def. 3.8, SPLs can be realized as a single circuit by ﬁrst compiling a complex logical constraint into a deterministic constraint circuit, and then parameterizing it using a gating function of the network embeddings.

Revised: Intuitively, this parameterization induces a probability distribution over the possible solutions of a logical formula encoded in the constraint circuit.

Revised: The expressiveness of this distribution depends on the number of parameters of the constraint circuit, i.e., the number of weighted edges associated to sum units.

Revised: As we would like to endow our single-circuit SPL with the ability to induce complex distributions, we devise two strategies to introduce more parameters than what the constraint circuit alone can offer: replication and mixture multiplication .

Revised: Replication works by maintaining m copies of the circuit, and taking their weighted average, i.e., introducing a sum unit that mixes them [57].

Revised: Mixture multiplication, instead, substitutes a single local marginal distribution encoded by a sub-circuit rooted into a sum unit with k mixture models over the same scope.

Revised: In practice, we create k − 1 copies of each sum units and rewire them by computing a cross product of their inputs as in Peharz et al.

Revised: Algorithm 1 formalizes this process.

Revised: As mentioned in Def. 3.8, both strategies relax determinism.

Revised: However, note that they do not alter the support of the underlying distribution .

Revised: This guarantees that all the predictions will be consistent with the encoded constraint (D3)

Revised: (Sec. 2).

Revised: D Additional experimental details D.1 Simple path prediction and preference learning

Revised: In the simple path prediction task, given a source and destination node in an unweighted grid

Revised: G =

Revised: ( V, E ) , the neural net needs to ﬁnd the shortest unweighted path connecting them.

Revised: We consider a 4 × 4 grid.

Revised: The input ( x , y ) is a binary vector of length | V | + | E | , with the ﬁrst | V | variables indicating the source and destination nodes, and the subsequent | E | variables indicating a subgraph G (cid:48)

Revised: ⊆ G .

Revised: Each label is a binary vector of length | E | encoding the unique shortest path in G (cid:48) .

Revised: For each example, we obtain G (cid:48) by dropping one third of the edges in the graph G uniformly at random, ﬁltering out the connected components with fewer than 5 nodes, to reduce degenerate cases, and then sample a source and destination node uniformly at random from G (cid:48) .

Revised: The dataset consists ofsuch examples, with a 60 / 20 / 20 train/validation/test split.

Revised: In the preference learning task, given a user’s ranking over a subset of items, the network has to predict the user’s ranking over the remaining items.

Revised: We encode an ordering over n items as a binary matrix Y ij , where for each i, j ∈ 1 , . . .

Revised: , n , Y ij indicates whether item i is the j th element in the ordering.

Revised: The input x consist of the user’s preference over 6 sushi types, and the model has to predict

Revised: 6: nodes ←

Revised: 8: elements

Revised: 21: nodes nodes the user’s preferences (a strict total order) over the remaining 4 .

Revised: We use preference ranking data over 10 types of sushi for 5 , 000 individuals, taken from [48], and a 60 / 20 / 20 split.

Revised: We follow Xu et al.

Revised: [79] in employing a 5 -layer with 50 hidden units each and sigmoid activation functions, and 3-layer MLP with 50 hidden units each as a baseline for the simple path prediction, and preference learning, respectively.

Revised: We equip this baselines with a FIL and additionally with the

Revised: Semantic Loss [79] (MLP+ L SL ) or its entropic extension [2] (MLP+N E S Y E NT ).

Revised: We compile the logical constraints into an SDD [16] and then turn it into a constraint circuit c K that is used for L SL , N E S Y E NT (Sec. 4) and our 1-circuit implementation of SPLs.

Revised: To obtain the results for SPL in Table 2, we perform a grid search over the using the validation set for a maximum of 2000 iterations, similar to Xu et al.

Revised: We search over the learning rates in the range { 1 × 10 − 3 , 5 × 10 − 3 , 1 × 10 − 4 , 5 × 10 − 4 } , the overparameterization factor k in the range { 2 , 4 , 8 } , as well as the number of circuit mixtures m in the range { 2 , 4 , 8 } , evaluating the model with the best performance on the validation set.

Revised: D.2 Hierarchical Multi-Label Classiﬁcation

Revised: We follow the experimental setup of Giunchiglia and Lukasiewicz [31] and evaluate SPL onreal-world HMLC tasks spanning four different domains: 8 functional genomics, 2 medical images, 1 microalgea classiﬁcation, and 1 text categorization.

Revised: These tasks are especially challenging due to the limited number of training samples, the large number of output classes, ranging from 56 to 4130 , as well as the sparsity of the output space.

Revised: We used the same train-validation-test splits and experimental setup as [31].

Revised: For numeric features we replaced missing values by their mean, and for categorical features by a vector of zeros, and standardized all features.

Revised: We used the validation splits to determine the number of layers in the gating function in the range { 2 , 4 , 8 } , the overparameterization factor in the range { 2 , 4 , 8 } , and the number of mixtures in the range { 2 , 4 , 8 } , keeping all other hyperparameters ﬁxed.

Revised: The ﬁnal models were obtained by training using a batch size of 128 and early stopping with a patience of 20 on the validation set.

Original: [N/A]
Revised: [Table]

Revised: D.3 Warcraft pathﬁnding

Revised: We evaluate SPL on the more challenging task of predicting the minimum cost path in a weighted 12 × 12 grid imposed over terrain maps of Warcraft II [59].

Revised: Our setting differs from the one proposed by Pogan ˇ

Revised: ci

Revised: ´

Revised: c

Revised: et al.

Revised: [59] in two ways: i) a node only neighbors four nodes as instead of eight, excluding the diagonals; ii) the neural network predicts the edges in the path, as opposed to the vertices, resolving ambiguities in the previous task (note that a set of vertices can might ambiguously encode more than one path).

Revised: Each vertex is assigned a cost corresponding to the type of the underlying terrain (e.g., earth has lower cost than water).

Revised: The minimum cost path between the top left and the bottom right vertices of the grid is encoded as an indicator matrix, and serves as a label.

Revised: We use Graphillion 3 to compile the path constraint, limiting our constraint to the set of paths whose length is less than 29 , as determined on the training set.

Revised: As in [59] we use a ResNet18 [35] with FIL optionally with L SL as a baseline.

Revised: Given the largest size of the compiled constraint circuit c K in this case 10 10 , we use a two-circuit implementation of SPL.

Revised: We use the identity function as our gating function and do a grid search over only the number of mixtures in the range { 2 , 4 , 8 } in our model, keeping all other hyperparameters as proposed in [59].

Revised: D.4 A study on the effect of overparameterization in SPL

Revised: We now illustrate the effect that overparameterization has on the performance of the single-circuit

Revised: SPL.

Revised: To that end, we performed an ablation study, comparing single-circuit SPLs comprising a different number of circuit copies m for our replication strategy, a different number of layers in the gating function, denoted by Gates , and the overparameterization factor k as used in Algorithm 1 in our mixture multiplication strategy.

Revised: We report the exact match percentage of the predicted labels on the validation set of the 12 HMLC datasets in Table 5.

Revised: As a general trend, we can see that our overparameterization strategies pay off and in general more mixture nodes help ( k = 4 ) as well as using more replicas ( m ≥ 4 ).

Revised: The effect of employing a deeper gating function is less striking instead, with a two-layer gating function achieving highest performances on 9 datasets.

Revised: cost: 55 .

Revised: [Figure]

Revised: cost

Revised: [Figure]

Revised: cost: 55 .cost: 57 .cost

Revised: [Figure]

Revised: cost

Revised: [Figure]

Revised: cost: 58 .cost: 97 .cost

Revised: [Figure]

Revised: cost

Revised: [Figure]

Revised: cost: 98 .cost:

Revised: [Figure]

Revised: .cost

Revised: [Figure]

Revised: cost

Revised: [Figure]

Revised: cost:

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: E Ethical Considerations

Revised: SPLs are meant as a module to be added on top of neural networks, and as such it does not signiﬁcantly alter the ethical risk of the underlying model and target application.

Revised: One exception is if the symbolic constraint is wrong (because e.g. it was encoded by a non-expert), in which case enforcing consistency - as SPLs do - may lead to mistakes or bias in the model’s predictions.

Revised: F Timings

Revised: [Table]

Revised: P REFERENCE [79] 0 .0 .0 .0 .0 .

Revised: S IMPLE P ATH [79] 0 .0 .0 .0 .0 .