Original: Semantic Probabilistic Layers for Neuro-Symbolic Learning
Revised: Semantic Probabilistic Layers for Neuro-Symbolic Learning

Original: Abstract
Revised: Abstract

Original: We design a predictive layer for structured-output prediction (SOP) that can be plugged into any neural network guaranteeing its predictions are consistent with a set of predeﬁned symbolic constraints.
Revised: We design a predictive layer for structured-output prediction (SOP) that can be plugged into any neural network guaranteeing its predictions are consistent with a set of predeﬁned symbolic constraints.

Original: Our S emantic P robabilistic L ayer (SPL) can model intricate correlations, and hard constraints, over a structured output space all while being amenable to end-to-end learning via maximum likelihood.
Revised: Our S emantic P robabilistic L ayer (SPL) can model intricate correlations, and hard constraints, over a structured output space all while being amenable to end-to-end learning via maximum likelihood.

Original: SPLs combine exact probabilistic inference with logical reasoning in a clean and modular way, learning complex distributions and restricting their support to solutions of the constraint.
Revised: SPLs combine exact probabilistic inference with logical reasoning in a clean and modular way, learning complex distributions and restricting their support to solutions of the constraint.

Original: As such, they can faithfully, and efﬁciently, model complex SOP tasks beyond the reach of alternative neuro-symbolic approaches.
Revised: As such, they can faithfully, and efﬁciently, model complex SOP tasks beyond the reach of alternative neuro-symbolic approaches.

Original: We empirically demonstrate that SPLs outperform these competitors in terms of accuracy on challenging SOP tasks including hierarchical multi-label classiﬁcation, pathﬁnding and preference learning, while retaining perfect constraint satisfaction.
Revised: We empirically demonstrate that SPLs outperform these competitors in terms of accuracy on challenging SOP tasks including hierarchical multi-label classiﬁcation, pathﬁnding and preference learning, while retaining perfect constraint satisfaction.

Original: Our code is made publicly available on Github at github.com/KareemYousrii/SPL.
Revised: Our code is made publicly available on Github at github.com/KareemYousrii/SPL.

Original:  Introduction
Revised:  Introduction

Original: Modularity is among the major factors that propelled the Cambrian explosion of deep learning [35].
Revised: Modularity is among the major factors that propelled the Cambrian explosion of deep learning [35].

Original: By stacking multiple differentiable layers together, practitioners are able to train deep classiﬁers in an end-to-end fashion with little effort.
Revised: By stacking multiple differentiable layers together, practitioners are able to train deep classiﬁers in an end-to-end fashion with little effort.

Original: However, despite its ﬂexibility, this modular approach to learning does not guarantee that the predictions of these models conform to our expectations of what makes sense .
Revised: However, despite its ﬂexibility, this modular approach to learning does not guarantee that the predictions of these models conform to our expectations of what makes sense .

Original: On the contrary, unconstrained deep classiﬁers are notorious for leading to predictions that are inconsistent with the logical constraints governing an underlying domain.
Revised: On the contrary, unconstrained deep classiﬁers are notorious for leading to predictions that are inconsistent with the logical constraints governing an underlying domain.

Original: This is even more evident in, and crucial for, structured output prediction (SOP) tasks, where classiﬁers have to predict hundreds of mutually constrained labels [73, 8].
Revised: This is even more evident in, and crucial for, structured output prediction (SOP) tasks, where classiﬁers have to predict hundreds of mutually constrained labels [73, 8].

Original: Consider for example a classical SOP task such as multi-label classiﬁcation (MLC) [74].
Revised: Consider for example a classical SOP task such as multi-label classiﬁcation (MLC) [74].

Original: Learning a multi-label classiﬁer that disregards the correlations among labels, e.g., by considering them fully independent given the inputs, yields sub-optimal results [6].
Revised: Learning a multi-label classiﬁer that disregards the correlations among labels, e.g., by considering them fully independent given the inputs, yields sub-optimal results [6].

Original: In more challenging tasks such as hierarchical MLC (HMLC) [71] or pathﬁnding [60], leveraging the domain’s logical constraints (encoding, e.g., the label hierarchy or acyclicity and connectedness of a path) at training time can improve prediction accuracy [44], but it cannot guarantee that the predictions are always consistent with the constraints at inference time [32].
Revised: In more challenging tasks such as hierarchical MLC (HMLC) [71] or pathﬁnding [60], leveraging the domain’s logical constraints (encoding, e.g., the label hierarchy or acyclicity and connectedness of a path) at training time can improve prediction accuracy [44], but it cannot guarantee that the predictions are always consistent with the constraints at inference time [32].

Original: Fig.
Revised: Fig.

Original: 1 illustrates this problem in the context of pathﬁnding: constraint-unaware neural networks systematically fail to predict label conﬁgurations that form a valid path.
Revised: 1 illustrates this problem in the context of pathﬁnding: constraint-unaware neural networks systematically fail to predict label conﬁgurations that form a valid path.

Original: In many safety-critical 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Revised: In many safety-critical 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: [Figure]
Revised: [Figure]

Original: scenarios such as protein function [63] and interaction prediction [65], and drug discovery [20, 24], predicting inconsistent solutions can not only be harmful but also highly expensive [5, 34].
Revised: scenarios such as protein function [63] and interaction prediction [65], and drug discovery [20, 24], predicting inconsistent solutions can not only be harmful but also highly expensive [5, 34].

Original: Unsurprisingly, due to their discrete nature, injecting logical constraints into deep neural networks while retaining modularity and differentiability is extremely challenging, as demonstrated in the neurosymbolic learning literature [66].
Revised: Unsurprisingly, due to their discrete nature, injecting logical constraints into deep neural networks while retaining modularity and differentiability is extremely challenging, as demonstrated in the neurosymbolic learning literature [66].

Original: One such attempt has been to learn neural networks that satisfy the logical constraints by explicitly minimizing a differentiable loss term encoding the probability that the networks violates the constraint for a given prediction.
Revised: One such attempt has been to learn neural networks that satisfy the logical constraints by explicitly minimizing a differentiable loss term encoding the probability that the networks violates the constraint for a given prediction.

Original: And while successful, such approaches do not guarantee consistency of the predictions at test time.
Revised: And while successful, such approaches do not guarantee consistency of the predictions at test time.

Original: More recently, researchers have proposed predictive layers that do guarantee consistency, but these are restricted to speciﬁc kinds of symbolic knowledge [32, 70] or become intractable for even moderately complex logical constraints [38].
Revised: More recently, researchers have proposed predictive layers that do guarantee consistency, but these are restricted to speciﬁc kinds of symbolic knowledge [32, 70] or become intractable for even moderately complex logical constraints [38].

Original: Motivated by these observations, we introduce a novel S emantic P robabilistic L ayer (SPL) for modeling intricate correlations, and logical constraints on the labels of the output space in a modular and probabilistically sound manner.
Revised: Motivated by these observations, we introduce a novel S emantic P robabilistic L ayer (SPL) for modeling intricate correlations, and logical constraints on the labels of the output space in a modular and probabilistically sound manner.

Original: It does so by leveraging recent advancements in the literature on probabilistic circuits [76, 13].
Revised: It does so by leveraging recent advancements in the literature on probabilistic circuits [76, 13].

Original: The key features of SPL are that, on the one hand, it can be used as a drop-in replacement for common predictive layers of deep nets like sigmoid layers, and on the other, it guarantees the output’s consistency with any prespeciﬁed logical constraints.
Revised: The key features of SPL are that, on the one hand, it can be used as a drop-in replacement for common predictive layers of deep nets like sigmoid layers, and on the other, it guarantees the output’s consistency with any prespeciﬁed logical constraints.

Original: Importantly, SPL also supports efﬁcient inference and – perhaps surprisingly – does not complicate training.
Revised: Importantly, SPL also supports efﬁcient inference and – perhaps surprisingly – does not complicate training.

Original: Contributions.
Revised: Contributions.

Original: Summarizing, we: ( i ) Identify six desiderata that neuro-symbolic predictors ought to satisfy to ﬂexibly and reliably support real-world SOP tasks, and show that state-of-the-art approaches fall short of one or more of them (Table 1); ( ii ) Introduce SPL, a novel semantic probabilistic layer that leverages probabilistic circuits to satisfy all six desiderata, i.e., that can be plugged into neural networks to ensure predictions to comply to given logical constraints, while retaining efﬁciency, expressivity, differentiability, and fully probabilistic semantics; ( iii ) We provide empirical evidence of the effectiveness of SPL in several challenging neuro-symbolic SOP tasks, such as HMLC and pathﬁnding, where it outperforms state-of-the-art neuro-symbolic approaches, often by a noticeable margin.
Revised: Summarizing, we: ( i ) Identify six desiderata that neuro-symbolic predictors ought to satisfy to ﬂexibly and reliably support real-world SOP tasks, and show that state-of-the-art approaches fall short of one or more of them (Table 1); ( ii ) Introduce SPL, a novel semantic probabilistic layer that leverages probabilistic circuits to satisfy all six desiderata, i.e., that can be plugged into neural networks to ensure predictions to comply to given logical constraints, while retaining efﬁciency, expressivity, differentiability, and fully probabilistic semantics; ( iii ) We provide empirical evidence of the effectiveness of SPL in several challenging neuro-symbolic SOP tasks, such as HMLC and pathﬁnding, where it outperforms state-of-the-art neuro-symbolic approaches, often by a noticeable margin.

Original: We implemented SPLs in PyTorch [54], and our code is made publicly available on Github at github.com/KareemYousrii/SPL.
Revised: We implemented SPLs in PyTorch [54], and our code is made publicly available on Github at github.com/KareemYousrii/SPL.

Original:  Designing a probabilistic layer for neuro-symbolic SOP
Revised:  Designing a probabilistic layer for neuro-symbolic SOP

Original: Notation.
Revised: Notation.

Original: In the following, we denote scalar constants x in lower case, random variables X in upper case, vectors of constants x in bold and vectors of random variables X in capital boldface.
Revised: In the following, we denote scalar constants x in lower case, random variables X in upper case, vectors of constants x in bold and vectors of random variables X in capital boldface.

Original: { ϕ } denotes the indicator function that evaluates to 1 if the statement ϕ holds and to 0 otherwise.
Revised: { ϕ } denotes the indicator function that evaluates to 1 if the statement ϕ holds and to 0 otherwise.

Original: We denote by x | =
Revised: We denote by x | =

Original: K that the value assignment x to variables X satisﬁes a logical formula K .
Revised: K that the value assignment x to variables X satisﬁes a logical formula K .

Original: Neuro-symbolic SOP.
Revised: Neuro-symbolic SOP.

Original: We tackle SOP tasks in which a neural net classiﬁer must learn to associate instances x ∈ R D to L interdependent labels, identiﬁed by the vector y ∈ { 0 , 1 } L .
Revised: We tackle SOP tasks in which a neural net classiﬁer must learn to associate instances x ∈ R D to L interdependent labels, identiﬁed by the vector y ∈ { 0 , 1 } L .

Original: We assume that we can abstract any neural classiﬁer into two components: a feature extractor f that maps inputs
Revised: We assume that we can abstract any neural classiﬁer into two components: a feature extractor f that maps inputs

Original: X to a M -dimensional embedding Z = f ( X ) and a predictive ﬁnal layer that outputs the label distribution p
Revised: X to a M -dimensional embedding Z = f ( X ) and a predictive ﬁnal layer that outputs the label distribution p

Original: ( Y | Z ) .
Revised: ( Y | Z ) .

Original: For example, the simplest, and yet widely adopted [51, 80, 32], predictive layer in neural classiﬁers for SOP considers labels Y i to be conditionally independent from each other given Z , i.e., p ( Y | Z )
Revised: For example, the simplest, and yet widely adopted [51, 80, 32], predictive layer in neural classiﬁers for SOP considers labels Y i to be conditionally independent from each other given Z , i.e., p ( Y | Z )

Original: =
Revised: =

Original: (cid:81)
Revised: (cid:81)

Original: Li =1 p ( Y i | Z ) .
Revised: Li =1 p ( Y i | Z ) .

Original: We refer to this as fully independent layer (FIL).
Revised: We refer to this as fully independent layer (FIL).

Original: In a FIL,
Revised: In a FIL,

Original: [Table]
Revised: [Table]

Original: Y
Revised: Y

Original: = y i | z ) is computed as σ w (cid:62)
Revised: = y i | z ) is computed as σ w (cid:62)

Original: i z where w i ∈
Revised: i z where w i ∈

Original: R M is a vector of σ
Revised: R M is a vector of σ

Original: ) is logistic sigmoid function 1 / (1 + e − x )
Revised: ) is logistic sigmoid function 1 / (1 + e − x )

Original: [Table]
Revised: [Table]

Original: We are interested in dependencies between labels that can occur both as correlations , as is the case in MLC [22], and as logical constraints encoded by logical formulas.
Revised: We are interested in dependencies between labels that can occur both as correlations , as is the case in MLC [22], and as logical constraints encoded by logical formulas.

Original: For example, in a HMLC task [32] one logical constraint can encode the fact that observing a label for the class cat and dog, implies observing the label for their superclass animal
Revised: For example, in a HMLC task [32] one logical constraint can encode the fact that observing a label for the class cat and dog, implies observing the label for their superclass animal

Original: [Equation]
Revised: [Equation]

Original: Speciﬁcally, we assume symbolic knowledge to be supplied in the form of constraints encoded as a logical formula denoted as K and deﬁned over the labels Y and optionally over a subset of the discrete input variables in X , if any (e.g., in our experiments, the predicted simple path is constrained to lie within the subset of edges appearing in the input graph, see Sec. 5).
Revised: Speciﬁcally, we assume symbolic knowledge to be supplied in the form of constraints encoded as a logical formula denoted as K and deﬁned over the labels Y and optionally over a subset of the discrete input variables in X , if any (e.g., in our experiments, the predicted simple path is constrained to lie within the subset of edges appearing in the input graph, see Sec. 5).

Original: On the other hand, we expect a model to learn the label correlations from data.
Revised: On the other hand, we expect a model to learn the label correlations from data.

Original: We call such task neuro-symbolic SOP .
Revised: We call such task neuro-symbolic SOP .

Original: Desiderata for neuro-symbolic SOP.
Revised: Desiderata for neuro-symbolic SOP.

Original: To tackle this setting, we seek an algorithmic strategy for replacing the predictive layer in any neural network classiﬁer with little effort, with the aim of injecting complex symbolic knowledge and allowing for ﬂexible probabilistic reasoning.
Revised: To tackle this setting, we seek an algorithmic strategy for replacing the predictive layer in any neural network classiﬁer with little effort, with the aim of injecting complex symbolic knowledge and allowing for ﬂexible probabilistic reasoning.

Original: We formalize these observations into the following six desiderata for our predictive layer:
Revised: We formalize these observations into the following six desiderata for our predictive layer:

Original: D1.
Revised: D1.

Original: Probabilistic :
Revised: Probabilistic :

Original: The layer should enjoy sound probabilistic semantics, and deliver normalized probabilistic predictions to facilitate maximum-likelihood learning and sound decision making by virtue of calibrated probabilistic predictions D2.
Revised: The layer should enjoy sound probabilistic semantics, and deliver normalized probabilistic predictions to facilitate maximum-likelihood learning and sound decision making by virtue of calibrated probabilistic predictions D2.

Original: Expressive : It should be able to compactly encode intricate correlations between labels.
Revised: Expressive : It should be able to compactly encode intricate correlations between labels.

Original: D3.
Revised: D3.

Original: Consistent : It should always output predictions that are consistent with the prespeciﬁed symbolic knowledge,
Revised: Consistent : It should always output predictions that are consistent with the prespeciﬁed symbolic knowledge,

Original: i
Revised: i

Original: e., for all x and y , if ( x , y ) (cid:54)| =
Revised: e., for all x and y , if ( x , y ) (cid:54)| =

Original: K then p ( y | x ) = 0 .
Revised: K then p ( y | x ) = 0 .

Original: D4.
Revised: D4.

Original: General : It should support rich logical constraints over the labels expressed in some formal language, e.g., propositional logic.
Revised: General : It should support rich logical constraints over the labels expressed in some formal language, e.g., propositional logic.

Original: D5.
Revised: D5.

Original: Modular : It should be applicable to any off-the-shelf (and possibly pretrained) neural network in a modular fashion, enabling end-to-end learning and rapid prototyping.
Revised: Modular : It should be applicable to any off-the-shelf (and possibly pretrained) neural network in a modular fashion, enabling end-to-end learning and rapid prototyping.

Original: D6.
Revised: D6.

Original: Efﬁcient : The time required by the predictor to compute a prediction should be linear in the size of the predictor and of the hard constraint representation
Revised: Efﬁcient : The time required by the predictor to compute a prediction should be linear in the size of the predictor and of the hard constraint representation

Original: For example, FILs are clearly probabilistic (D1), modular (D5), and efﬁcient (D6), but at the cost of being incapable of modeling intricate correlations and logical constraints and thus generating inconsistent predictions (D2–D4) (see also Fig. 1).
Revised: For example, FILs are clearly probabilistic (D1), modular (D5), and efﬁcient (D6), but at the cost of being incapable of modeling intricate correlations and logical constraints and thus generating inconsistent predictions (D2–D4) (see also Fig. 1).

Original: Table 1 summarizes how the other popular and effective approaches to neuro-symbolic SOP nowadays fall short of one of more desiderata as well.
Revised: Table 1 summarizes how the other popular and effective approaches to neuro-symbolic SOP nowadays fall short of one of more desiderata as well.

Original: We discuss this in detail in Sec. 4.
Revised: We discuss this in detail in Sec. 4.

Original: To the best of our knowledge, our proposed semantic probabilistic layers (SPLs) are the ﬁrst algorithmic solution to satisfy all above desiderata.
Revised: To the best of our knowledge, our proposed semantic probabilistic layers (SPLs) are the ﬁrst algorithmic solution to satisfy all above desiderata.

Original: SPL.
Revised: SPL.

Original: At a high level, SPL realizes the above desiderata in a single layer that combines exact probabilistic inference with logical reasoning in a clean and modular way, learning complex distributions and restricting their support to solutions of the constraint.
Revised: At a high level, SPL realizes the above desiderata in a single layer that combines exact probabilistic inference with logical reasoning in a clean and modular way, learning complex distributions and restricting their support to solutions of the constraint.

Original: Deﬁnition 2.1 (Semantic probabilistic layer (SPL)) .
Revised: Deﬁnition 2.1 (Semantic probabilistic layer (SPL)) .

Original: Given an input conﬁguration x , a SPL decomposes the computation of the probability of a label conﬁguration as:
Revised: Given an input conﬁguration x , a SPL decomposes the computation of the probability of a label conﬁguration as:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: The result is normalized by marginalizing product circuit
Revised: The result is normalized by marginalizing product circuit

Original: Θ , K , so as by
Revised: Θ , K , so as by

Original: Here, q Θ ( y | f ( x )) is a module to perform probabilistic reasoning by encoding an expressive distribution over the labels parameterized by Θ ; c K ( x , y ) is a module to ensure consistency of the predictions by encoding logical constraints K and being non-zero only when K is satisﬁed, i.e., c K ( x , y )
Revised: Here, q Θ ( y | f ( x )) is a module to perform probabilistic reasoning by encoding an expressive distribution over the labels parameterized by Θ ; c K ( x , y ) is a module to ensure consistency of the predictions by encoding logical constraints K and being non-zero only when K is satisﬁed, i.e., c K ( x , y )

Original: = 1 { ( x , y )
Revised: = 1 { ( x , y )

Original: | = K } ; and Z ( x ) is a renormalization term, also called the partition function.
Revised: | = K } ; and Z ( x ) is a renormalization term, also called the partition function.

Original: It is worth noting that this amounts to taking a product of experts [37] which is, in general, hard.
Revised: It is worth noting that this amounts to taking a product of experts [37] which is, in general, hard.

Original: Fig.
Revised: Fig.

Original: 2 illustrates the computational graph of our SPL at training time.
Revised: 2 illustrates the computational graph of our SPL at training time.

Original: In order to satisfy all D1-D6, we will realize both q Θ and c K as circuits [76, 13], constrained computational graphs that enable tractable computations.
Revised: In order to satisfy all D1-D6, we will realize both q Θ and c K as circuits [76, 13], constrained computational graphs that enable tractable computations.

Original: Differently from FILs, q Θ in SPLs can encode an expressive joint distributions over the labels and therefore attain full expressiveness by scaling the number of parameters Θ (D2).
Revised: Differently from FILs, q Θ in SPLs can encode an expressive joint distributions over the labels and therefore attain full expressiveness by scaling the number of parameters Θ (D2).

Original: Consistency is guaranteed by the component c K : by multiplying it to the joint probability of a label conﬁguration the resulting product distribution r Θ , K ( y , x ) =
Revised: Consistency is guaranteed by the component c K : by multiplying it to the joint probability of a label conﬁguration the resulting product distribution r Θ , K ( y , x ) =

Original: q Θ ( y | f ( x )) ·
Revised: q Θ ( y | f ( x )) ·

Original: c K ( x , y ) will have its support effectively cut by K , and thus cannot allocate any probability mass to inconsistent predictions (D3).
Revised: c K ( x , y ) will have its support effectively cut by K , and thus cannot allocate any probability mass to inconsistent predictions (D3).

Original: Additionally, c K will allow to encode general propositional logical constraints in a compact computational graph (D4).
Revised: Additionally, c K will allow to encode general propositional logical constraints in a compact computational graph (D4).

Original: Lastly, the product r Θ , K ( x , y ) is fully differentiable and allows SPL to be an off-the-shelf replacement for other predictive layers (see Fig. 2) and enables end-to-end learning (D5).
Revised: Lastly, the product r Θ , K ( x , y ) is fully differentiable and allows SPL to be an off-the-shelf replacement for other predictive layers (see Fig. 2) and enables end-to-end learning (D5).

Original: By renormalizing r Θ , K ( x , y ) and outputting normalized probabilities, SPL enables the exact computation of gradients for Θ , which can therefore be trained by maximum likelihood (D1).
Revised: By renormalizing r Θ , K ( x , y ) and outputting normalized probabilities, SPL enables the exact computation of gradients for Θ , which can therefore be trained by maximum likelihood (D1).

Original: Thanks to recent advancements in the literature on circuits, we can compute the partition function
Revised: Thanks to recent advancements in the literature on circuits, we can compute the partition function

Original: Z ( x ) efﬁciently in time linear in the size of r Θ , K , thus preserving efﬁciency (D6) and not compromising on the other desiderata.
Revised: Z ( x ) efﬁciently in time linear in the size of r Θ , K , thus preserving efﬁciency (D6) and not compromising on the other desiderata.

Original: This will also yield correct (and consistent) predictions at test time, when an SPL computes the MAP state y ∗ =
Revised: This will also yield correct (and consistent) predictions at test time, when an SPL computes the MAP state y ∗ =

Original: argmax y r Θ , K ( y , x ) / (cid:80) y r Θ , K ( y , x ) .
Revised: argmax y r Θ , K ( y , x ) / (cid:80) y r Θ , K ( y , x ) .

Original: The next section clariﬁes how to implement the modules of SPL as circuits while satisfying these desiderata.
Revised: The next section clariﬁes how to implement the modules of SPL as circuits while satisfying these desiderata.

Original:  Realizing SPLs with tractable circuit representations
Revised:  Realizing SPLs with tractable circuit representations

Original: The components of SPLs are circuits , a large class of computational graphs that can represent both functions and distributions [13, 18].
Revised: The components of SPLs are circuits , a large class of computational graphs that can represent both functions and distributions [13, 18].

Original: Circuits subsume many tractable generative and discriminative probabilistic models—from Chow-Liu and latent tree models [14, 11], to hidden Markov models (HMMs) [62], sum-product networks (SPNs) [61], decision trees [41, 15], and deep regressors [40]— as well as many compact representations of logical formulas, such as (ordered) binary decision diagrams [4], sentential decision diagrams (SDDs) [17] and others [18].
Revised: Circuits subsume many tractable generative and discriminative probabilistic models—from Chow-Liu and latent tree models [14, 11], to hidden Markov models (HMMs) [62], sum-product networks (SPNs) [61], decision trees [41, 15], and deep regressors [40]— as well as many compact representations of logical formulas, such as (ordered) binary decision diagrams [4], sentential decision diagrams (SDDs) [17] and others [18].

Original: The key idea behind SPLs is to leverage this single formalism to represent both an expressive joint distribution for q Θ ( y | f ( x )) and a compact encoding of the logical constraints for c K ( x , y ) , while ensuring the exact and efﬁcient evaluation of Eq.
Revised: The key idea behind SPLs is to leverage this single formalism to represent both an expressive joint distribution for q Θ ( y | f ( x )) and a compact encoding of the logical constraints for c K ( x , y ) , while ensuring the exact and efﬁcient evaluation of Eq.

Original: This can be achieved by ensuring that these computational graphs abide certain structural properties: smoothness , decomposability , determinism and compatibility [18, 75].
Revised: This can be achieved by ensuring that these computational graphs abide certain structural properties: smoothness , decomposability , determinism and compatibility [18, 75].

Original: Next, we introduce probabilistic circuits for modeling q Θ (Sec. 3.1) and constraint circuits for c K (Sec. 3.2), while in Sec.
Revised: Next, we introduce probabilistic circuits for modeling q Θ (Sec. 3.1) and constraint circuits for c K (Sec. 3.2), while in Sec.

Original: 3.4 we propose a more efﬁcient implementation of
Revised: 3.4 we propose a more efﬁcient implementation of

Original: SPL utilizing a single circuit.
Revised: SPL utilizing a single circuit.

Original: Y Y
Revised: Y Y

Original: Y
Revised: Y

Original: Y× ×
Revised: Y× ×

Original: Y
Revised: Y

Original: Y× × q g 1 { Y 1 =0 } 1 { Y 1 =1 } 1 { Y 2 =0 } 1 { Y 2 =1 } × × × × 1 { Y 3 =1 }  1 { Y 3 =0 }× c
Revised: Y× × q g 1 { Y 1 =0 } 1 { Y 1 =1 } 1 { Y 2 =0 } 1 { Y 2 =1 } × × × × 1 { Y 3 =1 } × 1 { Y 3 =0 } c

Original: 3.1 Representing expressive distributions with probabilistic circuits
Revised: 3.1 Representing expressive distributions with probabilistic circuits

Original: We start by introducing circuits for joint probability distributions and then extend the discussion to conditional distributions, which we use to implement q Θ ( Y | f ( X )) in SPLs.
Revised: We start by introducing circuits for joint probability distributions and then extend the discussion to conditional distributions, which we use to implement q Θ ( Y | f ( X )) in SPLs.

Original: Deﬁnition 3.1 (Circuits) .
Revised: Deﬁnition 3.1 (Circuits) .

Original: A circuit h over variables Y is a computational graph encoding a parameterized function h Θ ( Y ) by combining three kinds of computational units: input functional units, sum units and product units.
Revised: A circuit h over variables Y is a computational graph encoding a parameterized function h Θ ( Y ) by combining three kinds of computational units: input functional units, sum units and product units.

Original: An input functional n represents a base parametric function h n ( sc ( n ); λ ) over some variables sc ( n ) ⊆
Revised: An input functional n represents a base parametric function h n ( sc ( n ); λ ) over some variables sc ( n ) ⊆

Original: Y , called its scope and it is parameterized by λ .
Revised: Y , called its scope and it is parameterized by λ .

Original: Sum and product units n elaborate the output of other units, denoted in ( n ) .
Revised: Sum and product units n elaborate the output of other units, denoted in ( n ) .

Original: Sum units are parameterized by ω and compute the weighted sum of their inputs (cid:80) c ∈ in ( n ) ω
Revised: Sum units are parameterized by ω and compute the weighted sum of their inputs (cid:80) c ∈ in ( n ) ω

Original: c h c ( sc ( n )) , while product units compute (cid:81) c ∈ in ( n ) h c ( sc ( n )) .
Revised: c h c ( sc ( n )) , while product units compute (cid:81) c ∈ in ( n ) h c ( sc ( n )) .

Original: The parameters Θ of a circuit encompass the parameters of all input functionals ( λ ) and the parameters of sum units ( ω ).
Revised: The parameters Θ of a circuit encompass the parameters of all input functionals ( λ ) and the parameters of sum units ( ω ).

Original: For any input y , the value of h Θ ( y ) can be evaluated by propagating the output of the input units through the computational graph and reading out the value of the last unit.
Revised: For any input y , the value of h Θ ( y ) can be evaluated by propagating the output of the input units through the computational graph and reading out the value of the last unit.

Original: The support of h is the set of all states y of Y for which the output is non-zero, i.e., supp ( h ) =
Revised: The support of h is the set of all states y of Y for which the output is non-zero, i.e., supp ( h ) =

Original: { y | h ( y ) (cid:54) = 0 } .
Revised: { y | h ( y ) (cid:54) = 0 } .

Original: Deﬁnition 3.2 (Probabilistic circuits (PCs)) .
Revised: Deﬁnition 3.2 (Probabilistic circuits (PCs)) .

Original: A circuit q is a PC if it encodes a (possibly unnormalized) probability distribution, i.e., q Θ ( y ) is non-negative for all conﬁgurations y of Y .
Revised: A circuit q is a PC if it encodes a (possibly unnormalized) probability distribution, i.e., q Θ ( y ) is non-negative for all conﬁgurations y of Y .

Original: From here on, we will assume PCs to have positive sum parameters ω and whose input units model valid distributions, e.g., Bernoullis, as these conditions are sufﬁcient for satisfying Def. 3.2.
Revised: From here on, we will assume PCs to have positive sum parameters ω and whose input units model valid distributions, e.g., Bernoullis, as these conditions are sufﬁcient for satisfying Def. 3.2.

Original: Moreover, w.l.o.g.
Revised: Moreover, w.l.o.g.

Original: we will assume the sum and product units to be organized into alternating layers, and that every product unit n receives only two inputs c 1 , c 2 , i.e., q n ( X ) =
Revised: we will assume the sum and product units to be organized into alternating layers, and that every product unit n receives only two inputs c 1 , c 2 , i.e., q n ( X ) =

Original: q c 1 ( Y ) · q c 2 ( Y ) .
Revised: q c 1 ( Y ) · q c 2 ( Y ) .

Original: These conditions can easily be enforced in polynomial time [77, 78].
Revised: These conditions can easily be enforced in polynomial time [77, 78].

Original: We are speciﬁcally interested in smooth and decomposable PCs, as they will be enabling efﬁcient inference in SPL (Sec. 3.3).
Revised: We are speciﬁcally interested in smooth and decomposable PCs, as they will be enabling efﬁcient inference in SPL (Sec. 3.3).

Original: Deﬁnition 3.3 (Smoothness & Decomposability) .
Revised: Deﬁnition 3.3 (Smoothness & Decomposability) .

Original: A circuit is smooth if for every sum unit n , its inputs depend on the same variables: ∀ c 1 , c 2 ∈ in ( n ) , sc ( c 1 ) =
Revised: A circuit is smooth if for every sum unit n , its inputs depend on the same variables: ∀ c 1 , c 2 ∈ in ( n ) , sc ( c 1 ) =

Original: sc ( c 2 ) .
Revised: sc ( c 2 ) .

Original: It is decomposable if the inputs of every product unit n depend on disjoint sets of variables: in ( n ) = { c 1 , c 2 } , sc ( c 1 ) ∩ sc ( c 2 ) = ∅ .
Revised: It is decomposable if the inputs of every product unit n depend on disjoint sets of variables: in ( n ) = { c 1 , c 2 } , sc ( c 1 ) ∩ sc ( c 2 ) = ∅ .

Original: Smooth and decomposable PCs are both expressive and efﬁcient: they can encode distributions with hundred millions of parameters and be effectively learned by gradient ascent [58].
Revised: Smooth and decomposable PCs are both expressive and efﬁcient: they can encode distributions with hundred millions of parameters and be effectively learned by gradient ascent [58].

Original: The structure of their computational graph can be either speciﬁed manually [61, 58, 56] or acquired automatically from data [77, 64, 16], e.g., by ﬁrst learning a latent tree model and then compiling the latter into a circuit [46].
Revised: The structure of their computational graph can be either speciﬁed manually [61, 58, 56] or acquired automatically from data [77, 64, 16], e.g., by ﬁrst learning a latent tree model and then compiling the latter into a circuit [46].

Original: These circuits are competitive with intractable models such as variational autoencoders and normalizing ﬂows scores on several benchmarks [45].
Revised: These circuits are competitive with intractable models such as variational autoencoders and normalizing ﬂows scores on several benchmarks [45].

Original: As proposed by Shao et al.
Revised: As proposed by Shao et al.

Original: [68], any (smooth and decomposable) PC q Θ ( Y ) encoding a joint distribution over the labels Y can be turned into a (smooth and decomposable) conditional circuit, conditioned by input variables X , by letting its parameters be a function of X .
Revised: [68], any (smooth and decomposable) PC q Θ ( Y ) encoding a joint distribution over the labels Y can be turned into a (smooth and decomposable) conditional circuit, conditioned by input variables X , by letting its parameters be a function of X .

Original: Deﬁnition 3.4 (Neural conditional circuits [67]) .
Revised: Deﬁnition 3.4 (Neural conditional circuits [67]) .

Original: A conditional circuit q ( Y ; Θ = g ( X )) models the conditional distribution p ( Y | X ) via a differentiable function g that maps every input conﬁguration x to the set of parameters of Θ of p , also called the gating function .
Revised: A conditional circuit q ( Y ; Θ = g ( X )) models the conditional distribution p ( Y | X ) via a differentiable function g that maps every input conﬁguration x to the set of parameters of Θ of p , also called the gating function .

Original: An example of a smooth and decomposable conditional circuit is shown in Fig.
Revised: An example of a smooth and decomposable conditional circuit is shown in Fig.

Original: This design immediately allows us to implement q Θ ( Y | f
Revised: This design immediately allows us to implement q Θ ( Y | f

Original: ( X )) in SPL as a conditional PC whose gating function maps the feature embedding space R K to the parameter space R | Θ | + , realizing q ( Y ; Θ = g ( f ( X ))) .
Revised: ( X )) in SPL as a conditional PC whose gating function maps the feature embedding space R K to the parameter space R | Θ | + , realizing q ( Y ; Θ = g ( f ( X ))) .

Original: As such, the gating function g creates a clean interface between any pre-trained feature extractor f and the PC q (Fig. 3).
Revised: As such, the gating function g creates a clean interface between any pre-trained feature extractor f and the PC q (Fig. 3).

Original: While one can devise g in several ways, we strive for simplicity in our experiments and adopt vanilla multi-layer perceptrons (MLPs) whose ﬁnal activations are either sigmoids, if they have to predict the parameters λ of the Bernoulli input distributions of q , or softmax, if they output the sum unit parameters ω (Def. 3.1).
Revised: While one can devise g in several ways, we strive for simplicity in our experiments and adopt vanilla multi-layer perceptrons (MLPs) whose ﬁnal activations are either sigmoids, if they have to predict the parameters λ of the Bernoulli input distributions of q , or softmax, if they output the sum unit parameters ω (Def. 3.1).

Original: 3.2 Encoding logical formulas with constraint circuits
Revised: 3.2 Encoding logical formulas with constraint circuits

Original: The next step is to translate a logical constraint K into a smooth and decomposable circuit c K ( x , y ) .
Revised: The next step is to translate a logical constraint K into a smooth and decomposable circuit c K ( x , y ) .

Original: To this end, we employ a special type of PCs, deﬁned as follows.
Revised: To this end, we employ a special type of PCs, deﬁned as follows.

Original: Deﬁnition 3.5 (Constraint circuits) .
Revised: Deﬁnition 3.5 (Constraint circuits) .

Original: A PC c over variables X ∪ Y is a constraint circuit encoding prior knowledge K if it computes 1 { ( x , y ) |
Revised: A PC c over variables X ∪ Y is a constraint circuit encoding prior knowledge K if it computes 1 { ( x , y ) |

Original: = K } for every conﬁguration ( x , y ) .
Revised: = K } for every conﬁguration ( x , y ) .

Original: As a practical way to realize such a circuit, we will consider constraint circuits that have all sum unit parameters equal to 1 and input functionals that are indicator functions over their scope, e.g., c n ( z ) = 1 { z | = ϕ ( n ) } where Z is the scope of the input and ϕ ( n ) a constraint over it.
Revised: As a practical way to realize such a circuit, we will consider constraint circuits that have all sum unit parameters equal to 1 and input functionals that are indicator functions over their scope, e.g., c n ( z ) = 1 { z | = ϕ ( n ) } where Z is the scope of the input and ϕ ( n ) a constraint over it.

Original: Furthermore, we require each sum unit in it to be deterministic .
Revised: Furthermore, we require each sum unit in it to be deterministic .

Original: Deﬁnition 3.6 (Determinism) .
Revised: Deﬁnition 3.6 (Determinism) .

Original: A sum unit n is deterministic if its inputs have disjoint supports, i.e., ∀ c 1 , c 2 ∈ in ( n ) , c 1 (cid:54)
Revised: A sum unit n is deterministic if its inputs have disjoint supports, i.e., ∀ c 1 , c 2 ∈ in ( n ) , c 1 (cid:54)

Original: =
Revised: =

Original: c 2 = ⇒ supp ( c 1 ) ∩ supp ( c 2 ) = ∅ .
Revised: c 2 = ⇒ supp ( c 1 ) ∩ supp ( c 2 ) = ∅ .

Original: Fig.
Revised: Fig.

Original: 3 shows an example of a deterministic constraint circuit.
Revised: 3 shows an example of a deterministic constraint circuit.

Original: Thanks to determinism, we can readily translate classical compact representations for logical formulas such as (ordered) binary decision diagrams [4, 9] and sentential decision diagrams (SDDs)
Revised: Thanks to determinism, we can readily translate classical compact representations for logical formulas such as (ordered) binary decision diagrams [4, 9] and sentential decision diagrams (SDDs)

Original: [17] into constraint circuits as deﬁned above.
Revised: [17] into constraint circuits as deﬁned above.

Original: This becomes evident when they are written in the language of negation normal form [18] and their and gates (resp. or gates) are replaced with product units (resp.
Revised: This becomes evident when they are written in the language of negation normal form [18] and their and gates (resp. or gates) are replaced with product units (resp.

Original: sum units) [13].
Revised: sum units) [13].

Original: A logic constraint can therefore be represented as a constraint circuit for SPLs, by utilizing any of the many tools available for OBDDs [72] or SDDs [10, 53].
Revised: A logic constraint can therefore be represented as a constraint circuit for SPLs, by utilizing any of the many tools available for OBDDs [72] or SDDs [10, 53].

Original: Sec.
Revised: Sec.

Original: B illustrates in detail how to compile the example constraint of Eq.
Revised: B illustrates in detail how to compile the example constraint of Eq.

Original: (1) into the constraint circuit of Fig.
Revised: (1) into the constraint circuit of Fig.

Original: 3 in this way.
Revised: 3 in this way.

Original: The worst-case size of the constraint circuit depends on a) the algorithm employed for compilation and, b) the local structure of the constraints, rather than the number of labels.
Revised: The worst-case size of the constraint circuit depends on a) the algorithm employed for compilation and, b) the local structure of the constraints, rather than the number of labels.

Original: For example, in our
Revised: For example, in our

Original: Warcraft experiment (see Sec. 5), we have a label conﬁguration space over edges in a 12 × 12 grid, yielding 2 12 2 = 2 144 ≈ 10 43 states.
Revised: Warcraft experiment (see Sec. 5), we have a label conﬁguration space over edges in a 12 × 12 grid, yielding 2 12 2 = 2 144 ≈ 10 43 states.

Original: However, only 10 10 conﬁgurations satisfy the constraint that these edge labels form a valid path in the grid.
Revised: However, only 10 10 conﬁgurations satisfy the constraint that these edge labels form a valid path in the grid.

Original: If our compilation algorithm were to simply enumerate these conﬁgurations, putting them in a logical OR (as done in some neuro-symbolic learners such as MultiplexNet [38]), the size of the constraint circuit, denoted as | c | , would be 10 10 .
Revised: If our compilation algorithm were to simply enumerate these conﬁgurations, putting them in a logical OR (as done in some neuro-symbolic learners such as MultiplexNet [38]), the size of the constraint circuit, denoted as | c | , would be 10 10 .

Original: However, by using recent advancements in compiling logical formulas into constraints circuits, we can can greatly reduce the circuit size.
Revised: However, by using recent advancements in compiling logical formulas into constraints circuits, we can can greatly reduce the circuit size.

Original: For example, compiler we use [1] generates circuits whose size is worst-case exponential in the treewidth of the CNF representation of the logical formula, but typically much smaller.
Revised: For example, compiler we use [1] generates circuits whose size is worst-case exponential in the treewidth of the CNF representation of the logical formula, but typically much smaller.

Original: See Sec. 5 and Sec.
Revised: See Sec. 5 and Sec.

Original: F for details.
Revised: F for details.

Original: 3.3 Efﬁcient inference in SPLs
Revised: 3.3 Efﬁcient inference in SPLs

Original: As discussed above, PCs can be expressive (D2) and are modular (D5), while constraint circuits ensure consistency (D3) for general constraints (D4).
Revised: As discussed above, PCs can be expressive (D2) and are modular (D5), while constraint circuits ensure consistency (D3) for general constraints (D4).

Original: What remains to be shown to complete SPLs is that the product supports efﬁcient normalization (D1) and inference (D6), speciﬁcally that it allows for the efﬁcient evaluation of the normalization constant of r Θ , K , and its MAP state.
Revised: What remains to be shown to complete SPLs is that the product supports efﬁcient normalization (D1) and inference (D6), speciﬁcally that it allows for the efﬁcient evaluation of the normalization constant of r Θ , K , and its MAP state.

Original: To this end, we need to introduce the notion of compatibility between the two circuits [75].
Revised: To this end, we need to introduce the notion of compatibility between the two circuits [75].

Original: Deﬁnition 3.7 (Compatible circuits in SPLs) .
Revised: Deﬁnition 3.7 (Compatible circuits in SPLs) .

Original: A smooth and decomposable conditional PC q ( Y ; Θ ) is compatible over variables Y with a smooth and decomposable constraint circuit c K ( Y , X ) if any pair of product units n ∈ q and m
Revised: A smooth and decomposable conditional PC q ( Y ; Θ ) is compatible over variables Y with a smooth and decomposable constraint circuit c K ( Y , X ) if any pair of product units n ∈ q and m

Original: ∈ c K with the same scope over Y can be rearranged to be mutually compatible and decompose in the same way: ( sc ( n ) =
Revised: ∈ c K with the same scope over Y can be rearranged to be mutually compatible and decompose in the same way: ( sc ( n ) =

Original: sc ( m ))
Revised: sc ( m ))

Original: =
Revised: =

Original: ⇒ ( sc ( n i ) =
Revised: ⇒ ( sc ( n i ) =

Original: sc ( m i ) , n i and m i are compatible ) for some rearrangement of the inputs of n (resp.
Revised: sc ( m i ) , n i and m i are compatible ) for some rearrangement of the inputs of n (resp.

Original: m ) into n 1 , n(resp.
Revised: m ) into n 1 , n(resp.

Original: m 1 , m 2 ).
Revised: m 1 , m 2 ).

Original: The two circuits q and c shown in Fig.
Revised: The two circuits q and c shown in Fig.

Original: 3 are compatible.
Revised: 3 are compatible.

Original: [Table]
Revised: [Table]

Original: Theorem 3.1 (Efﬁcient inference in SPLs) .
Revised: Theorem 3.1 (Efﬁcient inference in SPLs) .

Original: If q ( Y ; Θ ) and c K ( Y , X ) are two smooth, decomposable and compatible circuits, then computing Eq.
Revised: If q ( Y ; Θ ) and c K ( Y , X ) are two smooth, decomposable and compatible circuits, then computing Eq.

Original: (2) can be done in O ( | q || c | ) time.
Revised: (2) can be done in O ( | q || c | ) time.

Original: Furthermore, if they are also deterministic, then computing the MAP state can be done in
Revised: Furthermore, if they are also deterministic, then computing the MAP state can be done in

Original: O ( | q || c | ) time.
Revised: O ( | q || c | ) time.

Original: The proof can be found in Sec.
Revised: The proof can be found in Sec.

Original: A.
Revised: A.

Original: How do we come up with compatible circuits?
Revised: How do we come up with compatible circuits?

Original: One option is to have a PC q that is compatible with every possible smooth and decomposable circuit c .
Revised: One option is to have a PC q that is compatible with every possible smooth and decomposable circuit c .

Original: To do so, we can represent q as a mixtures of M fully-independent models; i.e., (cid:80) Mi
Revised: To do so, we can represent q as a mixtures of M fully-independent models; i.e., (cid:80) Mi

Original: =1 ω i (cid:81) j q ( Y j ; Θ i ) .
Revised: =1 ω i (cid:81) j q ( Y j ; Θ i ) .

Original: This additional sum unit can be enough to be more expressive than a FIL and already delivers more accurate predictions than any competitor, as our experiments in pathﬁnding show (Sec. 5).
Revised: This additional sum unit can be enough to be more expressive than a FIL and already delivers more accurate predictions than any competitor, as our experiments in pathﬁnding show (Sec. 5).

Original: An example of such a circuit is shown in Fig.
Revised: An example of such a circuit is shown in Fig.

Original: Another sufﬁcient condition for compatibility is that both q and c share the exact same hierarchical scope partitioning [75], sometimes called a vtree or variable ordering [13, 59].
Revised: Another sufﬁcient condition for compatibility is that both q and c share the exact same hierarchical scope partitioning [75], sometimes called a vtree or variable ordering [13, 59].

Original: This can be done easily if one ﬁrst compiles logical constraints into OBDDs or SDDs and then uses a mechanized algorithm to build q as in [58] to create a compatible structure.
Revised: This can be done easily if one ﬁrst compiles logical constraints into OBDDs or SDDs and then uses a mechanized algorithm to build q as in [58] to create a compatible structure.

Original: Additionally, to ensure q is a deterministic PC, we could exploit the mechanized construction proposed in Shih and Ermon [69].
Revised: Additionally, to ensure q is a deterministic PC, we could exploit the mechanized construction proposed in Shih and Ermon [69].

Original: Computing the exact MAP state, however, is of less concern as approximate inference algorithms, e.g., beam search decoding [79] or iterative pruning [12], are nowadays a commodity in deep learning frameworks.
Revised: Computing the exact MAP state, however, is of less concern as approximate inference algorithms, e.g., beam search decoding [79] or iterative pruning [12], are nowadays a commodity in deep learning frameworks.

Original: For non-deterministic PCs, we compute the MAP state with a faster approximation by replacing non-deterministic sum units with max units [55].
Revised: For non-deterministic PCs, we compute the MAP state with a faster approximation by replacing non-deterministic sum units with max units [55].

Original: This runs in time linear in the size of r , and yet delivers state-of-the-art accuracies in our experiments Sec. 5.
Revised: This runs in time linear in the size of r , and yet delivers state-of-the-art accuracies in our experiments Sec. 5.

Original: 3.4 A single-circuit SPL
Revised: 3.4 A single-circuit SPL

Original: The two-circuit design we proposed for SPLs provides a clear and theoretically-backed interface between neural networks and probabilistic and symbolic reasoning.
Revised: The two-circuit design we proposed for SPLs provides a clear and theoretically-backed interface between neural networks and probabilistic and symbolic reasoning.

Original: This setup, however, can sometimes be wasteful, as it requires to compute the product of two circuits and renormalize.
Revised: This setup, however, can sometimes be wasteful, as it requires to compute the product of two circuits and renormalize.

Original: We circumvent this issue by designing a single-circuit implementation of SPL.
Revised: We circumvent this issue by designing a single-circuit implementation of SPL.

Original: Deﬁnition 3.8 (Single-circuit SPL) .
Revised: Deﬁnition 3.8 (Single-circuit SPL) .

Original: Given an input conﬁguration x , a single-circuit SPL computes p ( y | x ) =
Revised: Given an input conﬁguration x , a single-circuit SPL computes p ( y | x ) =

Original: c K ( Y , X ; Ω = g ( f ( X )))
Revised: c K ( Y , X ; Ω = g ( f ( X )))

Original: where c K is a neural conditional constraint circuit whose sum-unit parameters
Revised: where c K is a neural conditional constraint circuit whose sum-unit parameters

Original: Ω are non-unitary values parameterized via a gating function g .
Revised: Ω are non-unitary values parameterized via a gating function g .

Original: In a nutshell, we can directly realize SPL by compiling a complex logical constraints (D4) into a deterministic constraint circuit c K , as before, and then parameterizing it with a gating function of the network embeddings f ( X ) , i.e., allowing its sum units to be non-unitary and input dependent.
Revised: In a nutshell, we can directly realize SPL by compiling a complex logical constraints (D4) into a deterministic constraint circuit c K , as before, and then parameterizing it with a gating function of the network embeddings f ( X ) , i.e., allowing its sum units to be non-unitary and input dependent.

Original: Since the support of c K is already restricted to exactly match the constraint K (D3), parameterizing Ω induces an expressive probability distribution over the label conﬁgurations that are consistent with
Revised: Since the support of c K is already restricted to exactly match the constraint K (D3), parameterizing Ω induces an expressive probability distribution over the label conﬁgurations that are consistent with

Original: K (D2).
Revised: K (D2).

Original: We can further guarantee that the circuit’s output are normalized probabilities (D1, D6) by enforcing the parameters ω of each sum unit to form a convex combination [57].
Revised: We can further guarantee that the circuit’s output are normalized probabilities (D1, D6) by enforcing the parameters ω of each sum unit to form a convex combination [57].

Original: This can be easily done by utilizing a softmax activation function for g .
Revised: This can be easily done by utilizing a softmax activation function for g .

Original: One of the advantages of the two-circuit implementation of SPLs is that the size of the circuit q Θ can be easily increased to improve the capacity of the model (Sec. 3.1).
Revised: One of the advantages of the two-circuit implementation of SPLs is that the size of the circuit q Θ can be easily increased to improve the capacity of the model (Sec. 3.1).

Original: The single-circuit implementation is not as ﬂexible, as normally the number of parameters is determined by the complexity of the constraint circuit, which depends entirely on the compilation step.
Revised: The single-circuit implementation is not as ﬂexible, as normally the number of parameters is determined by the complexity of the constraint circuit, which depends entirely on the compilation step.

Original: In this case, one option is to overparameterize the neural conditional circuit by introducing additional sum units, hence allowing it to capture more modes in the distribution.
Revised: In this case, one option is to overparameterize the neural conditional circuit by introducing additional sum units, hence allowing it to capture more modes in the distribution.

Original: We detail this process is Sec.
Revised: We detail this process is Sec.

Original: C. A side effect of overparameterization is that it relaxes determinism, meaning that MAP inference needs to be approximated, as described in Sec. 3.3.
Revised: C. A side effect of overparameterization is that it relaxes determinism, meaning that MAP inference needs to be approximated, as described in Sec. 3.3.

Original: Additionally, training a gating function to map relatively small embeddings to large parameter vectors in overparameterized circuits, can slow down training.
Revised: Additionally, training a gating function to map relatively small embeddings to large parameter vectors in overparameterized circuits, can slow down training.

Original: In such cases, a two-circuit implementation of SPL is to be preferred.
Revised: In such cases, a two-circuit implementation of SPL is to be preferred.

Original:  Related works
Revised:  Related works

Original: In this section, we position SPLs against state-of-the-art approaches for enforcing constraints on neural network predictions.
Revised: In this section, we position SPLs against state-of-the-art approaches for enforcing constraints on neural network predictions.

Original: In-depth surveys on this topic can be found in [19] and [34].
Revised: In-depth surveys on this topic can be found in [19] and [34].

Original: Energy-based models.
Revised: Energy-based models.

Original: Deep energy-based models (EBMs) replace FILs with an unnormalized factor graph [42] that captures higher-order label dependencies [43] (D2) but at the cost of foregoing probabilistic semantics (D1) and efﬁciency (D6).
Revised: Deep energy-based models (EBMs) replace FILs with an unnormalized factor graph [42] that captures higher-order label dependencies [43] (D2) but at the cost of foregoing probabilistic semantics (D1) and efﬁciency (D6).

Original: EBMs are typically unconcerned with hard constraints (D3).
Revised: EBMs are typically unconcerned with hard constraints (D3).

Original: Neural approaches for segmentation [47] and parsing [28, 82, 83] remedy to this by replacing the factor graph with a full-ﬂedged intractable (discriminative) graphical model [42].
Revised: Neural approaches for segmentation [47] and parsing [28, 82, 83] remedy to this by replacing the factor graph with a full-ﬂedged intractable (discriminative) graphical model [42].

Original: To gain efﬁciency, one can restrict EBMs to simpler graphical models (e.g., chains, trees), compromising expressiveness (D2) and the ability to model non-trivial logical constraints (D3, D4).
Revised: To gain efﬁciency, one can restrict EBMs to simpler graphical models (e.g., chains, trees), compromising expressiveness (D2) and the ability to model non-trivial logical constraints (D3, D4).

Original: Loss-based methods.
Revised: Loss-based methods.

Original: A prominent strategy consists of penalizing the network for producing inconsistent predictions using an auxiliary loss [19, 34].
Revised: A prominent strategy consists of penalizing the network for producing inconsistent predictions using an auxiliary loss [19, 34].

Original: While popular, loss-based methods, however cannot guarantee that the predictions will be consistent at test time.
Revised: While popular, loss-based methods, however cannot guarantee that the predictions will be consistent at test time.

Original: Common losses include translating logical constraints into a differentiable fuzzy logic [25, 26], as exempliﬁed by DL2 [29].
Revised: Common losses include translating logical constraints into a differentiable fuzzy logic [25, 26], as exempliﬁed by DL2 [29].

Original: Although efﬁcient (D6), this solution is not probabilistically sound (D1) and crucially is not syntax-invariant : different encodings of the same formula (e.g., conjunctive vs. disjunctive normal form) yield different losses [31, 24].
Revised: Although efﬁcient (D6), this solution is not probabilistically sound (D1) and crucially is not syntax-invariant : different encodings of the same formula (e.g., conjunctive vs. disjunctive normal form) yield different losses [31, 24].

Original: Closer to our SPL, the Semantic Loss (SL) [80] avoids this issue by penalizing the the probability θ i associated to the i -th label by the neural network via the loss term
Revised: Closer to our SPL, the Semantic Loss (SL) [80] avoids this issue by penalizing the the probability θ i associated to the i -th label by the neural network via the loss term

Original: [Equation]
Revised: [Equation]

Original: When K is compiled into a constraint circuit c K one retrieves −Z ( x ) for a two-circuit version of SPL that is as expressive as FIL as it assumes independent labels via a conditional PC (cid:81)
Revised: When K is compiled into a constraint circuit c K one retrieves −Z ( x ) for a two-circuit version of SPL that is as expressive as FIL as it assumes independent labels via a conditional PC (cid:81)

Original: i q ( Y i ; θ i ) .
Revised: i q ( Y i ; θ i ) .

Original: The neuro-symbolic entropy (N E S Y E NT )
Revised: The neuro-symbolic entropy (N E S Y E NT )

Original: [3] extends L SL by an entropy term that improves (but still does not guarantee) consistency.
Revised: [3] extends L SL by an entropy term that improves (but still does not guarantee) consistency.

Original: It still makes the same independence assumptions over labels (D2).
Revised: It still makes the same independence assumptions over labels (D2).

Original: Consistency layers.
Revised: Consistency layers.

Original: Approaches ensuring consistency by embedding the constraints into the predictive layer as in SPLs include MultiplexNet [38] and HMCCN [32].
Revised: Approaches ensuring consistency by embedding the constraints into the predictive layer as in SPLs include MultiplexNet [38] and HMCCN [32].

Original: MultiplexNet is able to encode only constraints in disjunctive normal form, which is problematic for generality (D4) and efﬁciency (D6) as neuro-symbolic SOP tasks involve an intractably large number of clauses – e.g. our pathﬁnding experiments involves billions of clauses.
Revised: MultiplexNet is able to encode only constraints in disjunctive normal form, which is problematic for generality (D4) and efﬁciency (D6) as neuro-symbolic SOP tasks involve an intractably large number of clauses – e.g. our pathﬁnding experiments involves billions of clauses.

Original: HMCCN encodes label dependencies as fuzzy relaxation and is the current state-of-the-art model for HMLC [32].
Revised: HMCCN encodes label dependencies as fuzzy relaxation and is the current state-of-the-art model for HMLC [32].

Original: HMCCN and even its recent extension [33] are restricted to only certain constraints that can be exactly encoded with fuzzy logic easily.
Revised: HMCCN and even its recent extension [33] are restricted to only certain constraints that can be exactly encoded with fuzzy logic easily.

Original: SPLs instead can express constraints encoded as arbitrary propositional logical formulas (D4).
Revised: SPLs instead can express constraints encoded as arbitrary propositional logical formulas (D4).

Original: Other approaches.
Revised: Other approaches.

Original: Other common approaches to neuro-symbolic SOP require to invoke a solver to either obtain the MAP state or to compute (often only approximately) the gradient of the loss [23, 60, 52].
Revised: Other common approaches to neuro-symbolic SOP require to invoke a solver to either obtain the MAP state or to compute (often only approximately) the gradient of the loss [23, 60, 52].

Original: SPLs have no such requirement.
Revised: SPLs have no such requirement.

Original: Some neuro-symbolic approaches [66] constrain the outputs of neural networks within complex logical reasoning pipelines to solve tasks harder than neuro-symbolic
Revised: Some neuro-symbolic approaches [66] constrain the outputs of neural networks within complex logical reasoning pipelines to solve tasks harder than neuro-symbolic

Original: SOP.
Revised: SOP.

Original: For instance, DeepProblog [48] uses Prolog’s backward chaining algorithm for ﬁrst order logical rules whose probabilistic weights are predicted by the network.
Revised: For instance, DeepProblog [48] uses Prolog’s backward chaining algorithm for ﬁrst order logical rules whose probabilistic weights are predicted by the network.

Original: In modern implementations of
Revised: In modern implementations of

Original: Problog, grounding a ﬁrst order program and then compiling it into constraint circuits [27] produces a conditional circuit akin to those we use in SPLs, but in which ( i ) only input distributions are parameterized and ( ii ) increasing the parameter count is not considered a straightforward operation.
Revised: Problog, grounding a ﬁrst order program and then compiling it into constraint circuits [27] produces a conditional circuit akin to those we use in SPLs, but in which ( i ) only input distributions are parameterized and ( ii ) increasing the parameter count is not considered a straightforward operation.

Original: Scallop [39] provides a more scalable approach to deepproblog by considering only the top- k proofs.
Revised: Scallop [39] provides a more scalable approach to deepproblog by considering only the top- k proofs.

Original: We leave to future work how we could quickly compile only a speciﬁc query as DeepProblog/Scallop do, to deal with ﬁrst-order representations efﬁciently.
Revised: We leave to future work how we could quickly compile only a speciﬁc query as DeepProblog/Scallop do, to deal with ﬁrst-order representations efﬁciently.

Original:  Experiments
Revised:  Experiments

Original: We evaluate SPLs on standard neuro-symbolic SOP benchmarks such as simple path prediction , preference learning [80], shortest path ﬁnding in Warcraft [60] and HMLC [32].
Revised: We evaluate SPLs on standard neuro-symbolic SOP benchmarks such as simple path prediction , preference learning [80], shortest path ﬁnding in Warcraft [60] and HMLC [32].

Original: We compare SPLs against several state-of-the-art loss- and layer-based approaches (Sec. 4) by applying them to the same base neural network architecture as feature extractor f .
Revised: We compare SPLs against several state-of-the-art loss- and layer-based approaches (Sec. 4) by applying them to the same base neural network architecture as feature extractor f .

Original: As we are interested in measuring how close to the ground truth and how safe the predictions of all models are, we report the percentage of E XACT matches of the predicted labels, also called subset accuracy [74], and the percentage of
Revised: As we are interested in measuring how close to the ground truth and how safe the predictions of all models are, we report the percentage of E XACT matches of the predicted labels, also called subset accuracy [74], and the percentage of

Original: C ONSISTENT predictions, also called “Constraint” [80].
Revised: C ONSISTENT predictions, also called “Constraint” [80].

Original: Note that, like other consistency layers,
Revised: Note that, like other consistency layers,

Original: [Table]
Revised: [Table]

Original: SPLs are guaranteed to always output 100% consistent predictions.
Revised: SPLs are guaranteed to always output 100% consistent predictions.

Original: Additionally, we report the
Revised: Additionally, we report the

Original: H AMMING score [74], mainly to maintain compatibility with previous experimental settings [80, 3].
Revised: H AMMING score [74], mainly to maintain compatibility with previous experimental settings [80, 3].

Original: This metric does not consider consistency of predictions and naturally favors competitors that assume label independence and thus can minimize the per-label cross-entropy [22] (Table 3).
Revised: This metric does not consider consistency of predictions and naturally favors competitors that assume label independence and thus can minimize the per-label cross-entropy [22] (Table 3).

Original: Sec.
Revised: Sec.

Original: D collects all experimental details such as architectures and hyperparameters used for each experiment.
Revised: D collects all experimental details such as architectures and hyperparameters used for each experiment.

Original: In Sec.
Revised: In Sec.

Original: F we provide the average timings for compiling logical formulas into circuits—carried out once, and reused in all subsequent experiments, for parameterizing the conditional circuits, computing the MAP-state of SPL and the loss function at training time (including the cost of computing the product circuit r and its normalization).
Revised: F we provide the average timings for compiling logical formulas into circuits—carried out once, and reused in all subsequent experiments, for parameterizing the conditional circuits, computing the MAP-state of SPL and the loss function at training time (including the cost of computing the product circuit r and its normalization).

Original: All these timings, compilation excluded, are reported per batch.
Revised: All these timings, compilation excluded, are reported per batch.

Original: We compare to the timings of baselines such as semantic loss and neuro-symbolic entropy, where applicable, to which SPL is highly competitive.
Revised: We compare to the timings of baselines such as semantic loss and neuro-symbolic entropy, where applicable, to which SPL is highly competitive.

Original: Simple path prediction & preference learning.
Revised: Simple path prediction & preference learning.

Original: We start by comparing SPLs against loss-based approaches, reproducing the neuro-symbolic benchmarks of Xu et al.
Revised: We start by comparing SPLs against loss-based approaches, reproducing the neuro-symbolic benchmarks of Xu et al.

Original: [80] for simple path prediction and preference learning.
Revised: [80] for simple path prediction and preference learning.

Original: In the ﬁrst experiment, given a source and destination node in an unweighted grid G =
Revised: In the ﬁrst experiment, given a source and destination node in an unweighted grid G =

Original: ( V, E ) , the neural net needs to ﬁnd the shortest unweighted path connecting them.
Revised: ( V, E ) , the neural net needs to ﬁnd the shortest unweighted path connecting them.

Original: We consider a 4 × 4 grid.
Revised: We consider a 4 × 4 grid.

Original: The input ( x , y ) is a binary vector of length | V | + | E | , with the ﬁrst | V | variables indicating the source and destination nodes, and the subsequent | E | variables indicating a subgraph G (cid:48)
Revised: The input ( x , y ) is a binary vector of length | V | + | E | , with the ﬁrst | V | variables indicating the source and destination nodes, and the subsequent | E | variables indicating a subgraph G (cid:48)

Original: ⊆ G .
Revised: ⊆ G .

Original: Each label is a binary vector of length | E | encoding the unique shortest path in
Revised: Each label is a binary vector of length | E | encoding the unique shortest path in

Original: G (cid:48) .
Revised: G (cid:48) .

Original: For each example, we obtain G (cid:48) by dropping one third of the edges in the graph G uniformly at random, ﬁlter out the connected components with fewer than 5 nodes, to reduce degenerate cases, and then sample a source and destination node uniformly at random from G (cid:48) .
Revised: For each example, we obtain G (cid:48) by dropping one third of the edges in the graph G uniformly at random, ﬁlter out the connected components with fewer than 5 nodes, to reduce degenerate cases, and then sample a source and destination node uniformly at random from G (cid:48) .

Original: The dataset consists of 1600 such examples, with a 60 / 20 / 20 train/validation/test split.
Revised: The dataset consists of 1600 such examples, with a 60 / 20 / 20 train/validation/test split.

Original: In the preference learning task, given a user’s ranking over a subset of items, the network has to predict the user’s ranking over the remaining items.
Revised: In the preference learning task, given a user’s ranking over a subset of items, the network has to predict the user’s ranking over the remaining items.

Original: We encode an ordering over n items as a binary matrix Y ij , where for each i, j ∈ 1 , . . .
Revised: We encode an ordering over n items as a binary matrix Y ij , where for each i, j ∈ 1 , . . .

Original: , n , Y ij indicates whether item i is the j th element in the ordering.
Revised: , n , Y ij indicates whether item i is the j th element in the ordering.

Original: The input x consist of the user’s preference over 6 sushi types, and the model has to predict the user’s preferences (a strict total order) over the remaining 4 .
Revised: The input x consist of the user’s preference over 6 sushi types, and the model has to predict the user’s preferences (a strict total order) over the remaining 4 .

Original: We use preference ranking data over 10 types of sushi for 5 , 000 individuals, taken from [49], and a 60 / 20 / 20 split.
Revised: We use preference ranking data over 10 types of sushi for 5 , 000 individuals, taken from [49], and a 60 / 20 / 20 split.

Original: We employ a 5-layer and 3-layer MLP as a baseline for the simple path prediction, and preference learning, respectively, equipped with FIL layer and additionally with the Semantic Loss [80] (MLP+ L SL ) or its entropic extension [3] (MLP+N E S Y E NT ).
Revised: We employ a 5-layer and 3-layer MLP as a baseline for the simple path prediction, and preference learning, respectively, equipped with FIL layer and additionally with the Semantic Loss [80] (MLP+ L SL ) or its entropic extension [3] (MLP+N E S Y E NT ).

Original: We compile the logical constraints into an SDD [17] and then turn it into a the same constraint circuit c K that is used for L SL , N E S Y E NT (Sec. 4) and our 1-circuit implementation of SPLs.
Revised: We compile the logical constraints into an SDD [17] and then turn it into a the same constraint circuit c K that is used for L SL , N E S Y E NT (Sec. 4) and our 1-circuit implementation of SPLs.

Original: Table 2 clearly shows that the increased expressiveness of SPL, coming from overparameterizing c K , allows to outperform all competitors while guaranteeing consistent predictions, as expected.
Revised: Table 2 clearly shows that the increased expressiveness of SPL, coming from overparameterizing c K , allows to outperform all competitors while guaranteeing consistent predictions, as expected.

Original: Warcraft Shortest Path.
Revised: Warcraft Shortest Path.

Original: Next, we evaluate SPL on the more challenging task of predicting the minimum cost path in a weighted 12 × 12 grid imposed over terrain maps of Warcraft II [60].
Revised: Next, we evaluate SPL on the more challenging task of predicting the minimum cost path in a weighted 12 × 12 grid imposed over terrain maps of Warcraft II [60].

Original: Each vertex is assigned a cost corresponding to the type of the underlying terrain (e.g., earth has lower cost than water).
Revised: Each vertex is assigned a cost corresponding to the type of the underlying terrain (e.g., earth has lower cost than water).

Original: The minimum cost path between the top left and the bottom right vertices of the grid is encoded as an indicator matrix, and serves as a label.
Revised: The minimum cost path between the top left and the bottom right vertices of the grid is encoded as an indicator matrix, and serves as a label.

Original: As in [60] we use a ResNet18 [36] with FIL optionally with L SL as a baseline.
Revised: As in [60] we use a ResNet18 [36] with FIL optionally with L SL as a baseline.

Original: Given the largest size of the compiled constraint circuit c K in this case 10 10 , we use a two-circuit implementation of SPL.
Revised: Given the largest size of the compiled constraint circuit c K in this case 10 10 , we use a two-circuit implementation of SPL.

Original: Results in Fig.
Revised: Results in Fig.

Original: 1 and Table 3 are striking: not only SPL outperforms competitors by a large margin – approx.
Revised: 1 and Table 3 are striking: not only SPL outperforms competitors by a large margin – approx.

Original: +23 % over FIL and +19 % over the
Revised: +23 % over FIL and +19 % over the

Original: SL – but also consistently delivers meaningful paths that are very close to the ground truth in terms of cost, even when they encode very different routes.
Revised: SL – but also consistently delivers meaningful paths that are very close to the ground truth in terms of cost, even when they encode very different routes.

Original: See Sec.
Revised: See Sec.

Original: D.3 for a gallery of these examples.
Revised: D.3 for a gallery of these examples.

Original: Concerning times, SPLs are able to compute the likelihood in a mere 14 seconds per batch even on a 10 10 valid conﬁguration space (Sec. F).
Revised: Concerning times, SPLs are able to compute the likelihood in a mere 14 seconds per batch even on a 10 10 valid conﬁguration space (Sec. F).

Original: [Table]
Revised: [Table]

Original: Hierarchical Multi-Label Classiﬁcation.
Revised: Hierarchical Multi-Label Classiﬁcation.

Original: Lastly, we follow the experimental setup of Giunchiglia and Lukasiewicz [32] and evaluate SPL on 12 real-world HMLC tasks spanning four different domains: 8 functional genomics, 2 medical images, 1 microalgea classiﬁcation, and 1 text categorization.
Revised: Lastly, we follow the experimental setup of Giunchiglia and Lukasiewicz [32] and evaluate SPL on 12 real-world HMLC tasks spanning four different domains: 8 functional genomics, 2 medical images, 1 microalgea classiﬁcation, and 1 text categorization.

Original: Fig.
Revised: Fig.

Original: 3 shows an example of a hierarchy of classes.
Revised: 3 shows an example of a hierarchy of classes.

Original: These tasks are especially challenging due to the limited number of training samples, the large number of output classes, ranging from 56 to 499 , as well as the sparsity of the output space.
Revised: These tasks are especially challenging due to the limited number of training samples, the large number of output classes, ranging from 56 to 499 , as well as the sparsity of the output space.

Original: The larger datasets yield a label space of 2 499 conﬁgurations, but we can compile them in seconds into compact constraints circuits of size ≈ 108 KB (Sec. F).
Revised: The larger datasets yield a label space of 2 499 conﬁgurations, but we can compile them in seconds into compact constraints circuits of size ≈ 108 KB (Sec. F).

Original: For numeric features we replaced missing values by their mean, and for categorical features by a vector of zeros, and standardized all features.
Revised: For numeric features we replaced missing values by their mean, and for categorical features by a vector of zeros, and standardized all features.

Original: We used the validation splits to determine the number of layers in the gating function as well as the overparameterization, keeping all other hyperparameters ﬁxed.
Revised: We used the validation splits to determine the number of layers in the gating function as well as the overparameterization, keeping all other hyperparameters ﬁxed.

Original: The ﬁnal models were obtained by training using a batch size of 128 and early stopping on the validation set.
Revised: The ﬁnal models were obtained by training using a batch size of 128 and early stopping on the validation set.

Original: We compare our single-circuit SPL against HMCNN which was shown to outperform several other state-of-the-art HMLC approaches in Giunchiglia and Lukasiewicz [32].
Revised: We compare our single-circuit SPL against HMCNN which was shown to outperform several other state-of-the-art HMLC approaches in Giunchiglia and Lukasiewicz [32].

Original: We study the effect of increasing the expressivenss of SPL via overparameterization in Sec.
Revised: We study the effect of increasing the expressivenss of SPL via overparameterization in Sec.

Original: D.4.
Revised: D.4.

Original: The results in
Revised: The results in

Original: Table 5 highlight that SPL signiﬁcantly outperforms HMCNN in terms of exact match on 11 data sets performing comparably on 1 ,
Revised: Table 5 highlight that SPL signiﬁcantly outperforms HMCNN in terms of exact match on 11 data sets performing comparably on 1 ,

Original:  Conclusion
Revised:  Conclusion

Original: SPLs offer the ﬁrst clear interface for integrating complex probabilitistic reasoning and logical constraints on top of any neural network classiﬁer while retaining efﬁcient inference and training.
Revised: SPLs offer the ﬁrst clear interface for integrating complex probabilitistic reasoning and logical constraints on top of any neural network classiﬁer while retaining efﬁcient inference and training.

Original: They improve by a noticeable margin the current state-of-the-art on challenging neuro-symbolic
Revised: They improve by a noticeable margin the current state-of-the-art on challenging neuro-symbolic

Original: SOP benchmarks such as pathﬁnding and HMLC.
Revised: SOP benchmarks such as pathﬁnding and HMLC.

Original: This opens up a number of interesting research directions.
Revised: This opens up a number of interesting research directions.

Original: First, SPLs can be extended to incorporate logical constraints over multiple networks and representable by ﬁrst-order formulas [48], which we plan to explore in future works, making the circuit construction pipeline totally transparent to users [2] while possibly automatically learning constraints from data [21, 50].
Revised: First, SPLs can be extended to incorporate logical constraints over multiple networks and representable by ﬁrst-order formulas [48], which we plan to explore in future works, making the circuit construction pipeline totally transparent to users [2] while possibly automatically learning constraints from data [21, 50].

Original: Second, we are interested in leveraging SPLs to inject scalable logical constraints into large language models [7] thus equipping them with probabilistic reasoning [30, 81].
Revised: Second, we are interested in leveraging SPLs to inject scalable logical constraints into large language models [7] thus equipping them with probabilistic reasoning [30, 81].

Original: Laguerrapiutotale
Revised: Laguerrapiutotale

Original: . Acknowledgments and Disclosure of Funding
Revised: . Acknowledgments and Disclosure of Funding

Original: The authors would like to thank Arthur Choi for helpful discussions on compiling the constraints for the Warcraft Shortest Path task, and Andreas Grivas for proofreading a draft manuscript.
Revised: The authors would like to thank Arthur Choi for helpful discussions on compiling the constraints for the Warcraft Shortest Path task, and Andreas Grivas for proofreading a draft manuscript.

Original: The research of ST was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215.
Revised: The research of ST was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215.

Original: This work was funded in part by the DARPA
Revised: This work was funded in part by the DARPA

Original: Perceptually-enabled Task Guidance (PTG) Program under contract number
Revised: Perceptually-enabled Task Guidance (PTG) Program under contract number

Original: HR00112220005„ NSF grants #IIS-1943641, #IIS-1956441, #CCF-1837129, Samsung, CISCO, and a Sloan Fellowship.
Revised: HR00112220005„ NSF grants #IIS-1943641, #IIS-1956441, #CCF-1837129, Samsung, CISCO, and a Sloan Fellowship.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] . You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] See Section ?? . • Did you include the license to the code and datasets? [No] The code and the data are proprietary. • Did you include the license to the code and datasets?
Revised: [Yes] Please see section 1 for a summary of our contributions (b) Did you describe the limitations of your work?

Original: [N/A] Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit.
Revised: [Yes] Please see Supplementary (c) Did you discuss any potential negative societal impacts of your work?

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] Please see section 1 for a summary of our contributions (b) Did you describe the limitations of your work? [Yes] Please see Supplementary (c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see Supplementary (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] Please see Supplementary (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Please see supplementary material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] Please see supplementary material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Please see section 5 for details regarding the dataset splits.
Revised: [Yes] Please see section 5 for details regarding the dataset splits.

Original: All other details are included in the supplementary.
Revised: All other details are included in the supplementary.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We reported error bars and signiﬁcance testing for the hierarchical multi-label classiﬁcation experiment, please see(d)
Revised: [Yes] We reported error bars and signiﬁcance testing for the hierarchical multi-label classiﬁcation experiment, please see(d)

Original: Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Please see supplementary material 4.
Revised: [Yes] Please see supplementary material 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Proofs

Original: Theorem 3.1 (Efﬁcient inference in SPLs).

Original: If q ( Y ; Θ ) and c K ( Y , X ) are two smooth, decomposable and compatible circuits, then computing Eq.

Original: (2) can be done in O ( | q || c | ) time, where | · | denotes the circuit size.

Original: Furthermore, if they are also deterministic, then computing the MAP state can be done in

Original: O ( | q || c | ) time. .

Original: We prove the ﬁrst statement by ﬁrst showing that the partition function Z ( x ) in Eq.

Original: (2) can solved exactly in time O ( | q || c | ) .

Original: It will then follow from it that computing Eq.

Original: (2) can be done in O ( | q || c | + | q | + | c | )

Original: ≈ O ( | q || c | ) where the last two additive factors derive from evaluating q and c for an input conﬁguration ( x , y ) .

Original: To do so, we will exploit two ingredients: i) the product of q and c can be represented as a smooth and decomposable circuit in time

Original: O ( | q || c | ) [75] and ii) any smooth and decomposable circuit guarantees tractable marginalization in time linear in its size [13].

Original: The next two propositions formalize these statements.

Original: Proposition A.1 (Tractable product of circuits) .

Original: Let q ( Y ; Θ ) and c K ( Y , X ) be two smooth, decomposable circuits that are compatible over Y then computing their product as a circuit r Θ , K

Original: ( X , Y ) = q ( Y ; Θ ) · c K ( Y , X ) that is decomposable over Y can be done in O ( | q || c | ) . If both q and c are also deterministic, then r is as well.
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: Proof.

Original: The proof directly follows from Theorem 3.2 from Vergari et al.

Original: Note that O ( | q || c | ) is a loose upperbound and the size of r is in practice smaller [75].

Original: Proposition A.2 (Tractable marginalization of circuits) .

Original: Let r ( X , Y ) be a circuit that is smooth and decomposable over Y with input functions over Y that can be tractably marginalized out.

Original: Then for any variables Y (cid:48) ⊆

Original: Y and their assignment y (cid:48) , the marginalization (cid:80) y (cid:48) r ( y (cid:48) , y (cid:48)(cid:48) , x ) can be computed exactly in time linear in the size of r , where Y (cid:48)(cid:48) =

Original: Y \ Y (cid:48) .

Original: Proof.

Original: The proof follows by considering that i) the input functionals in SPLs are simple distributions such as Bernoullis and indicators and can be easily marginalized in O (1) and ii) that for every conﬁguration x of variables X , r ( Y , x ) is a circuit only over Y and therefore Proposition 2.1 from

Original: Vergari et al.

Original: [75] can be directly applied.

Original: Analogously, the second statement of Theorem 3.1 follows from Proposition A.1 and by recalling that the MAP state of a deterministic circuit can be computed in time linear in its size.

Original: Proposition A.3 (Tractable MAP state of circuits (Choi et al.

Original: Let r ( X , Y ) be a circuit that is smooth and decomposable and deterministic over Y then for a conﬁguration x its MAP state argmax y r ( x , y ) can be computed in time O

Original: ( | r | ) .

Original: B Compiling logical formulas into circuits

Original: For our experiments we use standard compilation tools to obtain a constraint circuit starting from a propositional logical formula in conjunctive normal form.

Original: Speciﬁcally, we use Graphillion 1 to compile the constraints in the Warcraft pathﬁnding experiment into an SDD.

Original: For all other experiments, we use PySDD 2 [1] a python SDD compiler [17, 10].

Original: We now illustrate step-by-step one example of such a compilation for a simple logical formula.

Original: Consider the constraint circuit c in Fig.

Original: 3 encoding the constraint

Original: [Equation]

Original: Intuitively, our aim is to compile the above logical formula into a compact form representing all possible assignments to Y cat , Y dog , Y animal satisfying the above constraint.

Original: We compile such a constraint by proceeding in a bottom up fashion, where bottom-up compilation can be seen as composing Boolean sub-functions whose domain is determined by a variable ordering, also called vtree (see Sec. 3.3).

Original: In this example, we assume the function f

Original: ( Y animal , Y cat , Y dog ) decomposes as f 1 ( Y animal ) · f 2 ( Y dog ) · f 3 ( Y cat )

Original: We therefore start by compiling a constraint circuit that is a function of Y cat and Y dog , and compose it with a constraint circuit that is a function of Y animal

Original: We ﬁrst introduce input functionals representing indicators associated with Y cat , Y dog , Y animal .

Original: We will denote by Y i the indicator 1 { Y i = 1 } and by ¬ Y

Original: i the indicator 1 { Y i = 0 } .

Original: [Equation]

Original: { Y 1 =

Original: [Equation]

Original: We start by disjoining the indicators Y cat with ¬ Y cat , and Y dog with ¬ Y dog .

Original: This corresponds to introducing deterministic and smooth sum units in our circuits.

Original: [Equation]

Original: These units represent disjoint solutions to the logical formula, meaning there exists distinct assignments, characterized by the children, that satisfy the logical constraint e.g. Y cat , Y dog , Y animal and

Original: Y cat , ¬ Y dog , Y animal are two distinct assignments that satisfy the logical constraint.

Original: The compilation process proceeds by conjoining the constraint circuits for Y dog ∨ ¬ Y dog with Y cat ,

Original: Y dog with Y cat ∨ ¬ Y cat , and ¬ Y dog with ¬ Y cat .

Original: [Equation]

Original: A decomposable product units composes functions over disjoint sets of variables.

Original: The above three product nodes represent the Boolean functions ( Y dog ∨ ¬ Y dog )

Original: ∧ Y cat , Y dog ∧ ( Y cat ∨ ¬ Y cat ) , and ¬ Y dog ∧ ¬ Y cat .

Original: We again disjoin ( Y dog ∨ ¬ Y dog )

Original: ∧ Y cat with Y dog ∧ ( Y cat ∨ ¬ Y cat ) , and ¬ Y dog ∧ ¬ Y cat with true , the logical multiplicative identity, guaranteeing alternating sum and product nodes, as mentioned in

Original: Sec. 3.1.

Original: So far, we have compiled constraint circuits for the logical formula

Original: [Equation]
Revised: [N/A] 5.

Original: and the logical formula

Original: [Equation]

Original: What remains is to conjoin Eq.

Original: (4) with Y animal , and Eq.

Original: (5) with ¬ Y animal , and disjoin the resulting constraint circuits.

Original: What we get is a a mixture distribution over the possible solutions of the constraint:

Original: If we predict there is a dog or a cat, or both, in e.g., an image, we better predict that there’s an animal.

Original: On the other hand, the absence of a dog and a cat from an image implies nothing as to the presence of an animal in the image.

Original: [Equation]

Original: Compilation techniques like the one we illustrated do not, however, escape the hardness of the problem: the compiled circuit can be exponential in the size of the constraint, in the worst case .

Original: In practice , nevertheless, we can obtain compact circuits because real-life logical constraints exhibit enough structure (e.g., they encode repeated sub-problems) that can be easily exploited by a compiler.

Original: We refer to the literature of compilation for details on this [18].

Original: C Overparameterizing the single-circuit SPL

Original: As mentioned in Def. 3.8, SPLs can be realized as a single circuit by ﬁrst compiling a complex logical constraint into a deterministic constraint circuit, and then parameterizing it using a gating function of the network embeddings.

Original: Intuitively, this parameterization induces a probability distribution over the possible solutions of a logical formula encoded in the constraint circuit.

Original: The expressiveness of this distribution depends on the number of parameters of the constraint circuit, i.e., the number of weighted edges associated to sum units.

Original: As we would like to endow our single-circuit SPL with the ability to induce complex distributions, we devise two strategies to introduce more parameters than what the constraint circuit alone can offer: replication and mixture multiplication .

Original: Replication works by maintaining m copies of the circuit, and taking their weighted average, i.e., introducing a sum unit that mixes them [58].

Original: Mixture multiplication, instead, substitutes a single local marginal distribution encoded by a sub-circuit rooted into a sum unit with k mixture models over the same scope.

Original: In practice, we create k − 1 copies of each sum units and rewire them by computing a cross product of their inputs as in Peharz et al.

Original: Algorithm 1 formalizes this process.

Original: As mentioned in Def. 3.8, both strategies relax determinism.

Original: However, note that they do not alter the support of the underlying distribution .

Original: This guarantees that all the predictions will be consistent with the encoded constraint (D3) (Sec. 2). D Additional experimental details D.1 Simple path prediction and preference learning In the simple path prediction task, given a source and destination node in an unweighted grid
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: G = ( V, E ) , the neural net needs to ﬁnd the shortest unweighted path connecting them. We consider a 4 × 4 grid. The input ( x , y ) is a binary vector of length | V | + | E | , with the ﬁrst | V | variables indicating the source and destination nodes, and the subsequent | E | variables indicating a subgraph G (cid:48)
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: ⊆ G .

Original: Each label is a binary vector of length | E | encoding the unique shortest path in G (cid:48) .

Original: For each example, we obtain G (cid:48) by dropping one third of the edges in the graph G uniformly at random, ﬁltering out the connected components with fewer than 5 nodes, to reduce degenerate cases, and then sample a source and destination node uniformly at random from G (cid:48) .

Original: The dataset consists ofsuch examples, with a 60 / 20 / 20 train/validation/test split.

Original: In the preference learning task, given a user’s ranking over a subset of items, the network has to predict the user’s ranking over the remaining items.

Original: We encode an ordering over n items as a binary matrix Y ij , where for each i, j ∈ 1 , . . .

Original: , n , Y ij indicates whether item i is the j th element in the ordering. The input x consist of the user’s preference over 6 sushi types, and the model has to predict theuser’s preferences (a strict total order) over the remaining 4 . We use preference ranking data over 10 types of sushi for 5 , 000 individuals, taken from [49], and a 60 / 20 / 20 split. nodes
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: We follow Xu et al.

Original: [80] in employing a 5 -layer with 50 hidden units each and sigmoid activation functions, and 3-layer MLP with 50 hidden units each as a baseline for the simple path prediction, and preference learning, respectively.

Original: We equip this baselines with a FIL and additionally with the

Original: Semantic Loss [80] (MLP+ L SL ) or its entropic extension [3] (MLP+N E S Y E NT ).

Original: We compile the logical constraints into an SDD [17] and then turn it into a constraint circuit c K that is used for L SL , N E S Y E NT (Sec. 4) and our 1-circuit implementation of SPLs.

Original: To obtain the results for SPL in Table 2, we perform a grid search over the using the validation set for a maximum of 2000 iterations, similar to Xu et al.

Original: We search over the learning rates in the range { 1 × 10 − 3 , 5 × 10 − 3 , 1 × 10 − 4 , 5 × 10 − 4 } , the overparameterization factor k in the range { 2 , 4 , 8 } , as well as the number of circuit mixtures m in the range { 2 , 4 , 8 } , evaluating the model with the best performance on the validation set.

Original: D.2 Hierarchical Multi-Label Classiﬁcation

Original: We follow the experimental setup of Giunchiglia and Lukasiewicz [32] and evaluate SPL onreal-world HMLC tasks spanning four different domains: 8 functional genomics, 2 medical images, 1 microalgea classiﬁcation, and 1 text categorization.

Original: These tasks are especially challenging due to the limited number of training samples, the large number of output classes, ranging from 56 to 4130 , as well as the sparsity of the output space.

Original: We used the same train-validation-test splits and experimental setup as [32].

Original: For numeric features we replaced missing values by their mean, and for categorical features by a vector of zeros, and standardized all features.

Original: We used the validation splits to determine the number of layers in the gating function in the range { 2 , 4 , 8 } , the overparameterization factor in the range { 2 , 4 , 8 } , and the number of mixtures in the range { 2 , 4 , 8 } , keeping all other hyperparameters ﬁxed.

Original: The ﬁnal models were obtained by training using a batch size of 128 and early stopping with a patience of 20 on the validation set.

Original: D.3 Warcraft pathﬁnding

Original: We evaluate SPL on the more challenging task of predicting the minimum cost path in a weighted 12 × 12 grid imposed over terrain maps of Warcraft II [60].

Original: Our setting differs from the one proposed by Poganˇci´c et al.

Original: [60] in two ways: i) a node only neighbors four nodes as instead of eight, excluding

Original: [Table]
Revised: [N/A]

Original: the diagonals; ii) the neural network predicts the edges in the path, as opposed to the vertices, resolving ambiguities in the previous task (note that a set of vertices can might ambiguously encode more than one path).

Original: Each vertex is assigned a cost corresponding to the type of the underlying terrain (e.g., earth has lower cost than water).

Original: The minimum cost path between the top left and the bottom right vertices of the grid is encoded as an indicator matrix, and serves as a label.

Original: We use Graphillion 3 to compile the path constraint, limiting our constraint to the set of paths whose length is less than 29 , as determined on the training set.

Original: As in [60] we use a ResNet18 [36] with FIL optionally with L SL as a baseline.

Original: Given the largest size of the compiled constraint circuit c K in this case 10 10 , we use a two-circuit implementation of SPL.

Original: We use the identity function as our gating function and do a grid search over only the number of mixtures in the range { 2 , 4 , 8 } in our model, keeping all other hyperparameters as proposed in [60].

Original: D.4 A study on the effect of overparameterization in SPL

Original: We now illustrate the effect that overparameterization has on the performance of the single-circuit

Original: SPL.

Original: To that end, we performed an ablation study, comparing single-circuit SPLs comprising a different number of circuit copies m for our replication strategy, a different number of layers in the gating function, denoted by Gates , and the overparameterization factor k as used in Algorithm 1 in our mixture multiplication strategy.

Original: We report the exact match percentage of the predicted labels on the validation set of the 12 HMLC datasets in Table 5.

Original: As a general trend, we can see that our overparameterization strategies pay off and in general more mixture nodes help ( k = 4 ) as well as using more replicas ( m ≥ 4 ).

Original: The effect of employing a deeper gating function is less striking instead, with a two-layer gating function achieving highest performances on 9 datasets.

Original: E Ethical Considerations

Original: SPLs are meant as a module to be added on top of neural networks, and as such it does not signiﬁcantly alter the ethical risk of the underlying model and target application.

Original: One exception is if the symbolic constraint is wrong (because e.g. it was encoded by a non-expert), in which case enforcing consistency - as SPLs do - may lead to mistakes or bias in the model’s predictions.

Original: (ours) cost: 55 .cost: ∞ cost: ∞ cost: 55 .cost: 57 .cost: ∞ cost: ∞ cost: 58 .cost: 97 .cost: ∞ cost: ∞ cost: 98 .cost: 30 .cost: ∞ cost: ∞ cost: 30 .cost: 39 .cost:

Original: [Figure]

Original: cost

Original: [Figure]

Original: cost: 45 .

Original: F Timings

Original: D ATASET

Original: [Table]