Original: SLASH: E MBRACING P ROBABILISTIC C IRCUITS INTO N EURAL A NSWER S ET P ROGRAMMING
Revised: SLASH: E MBRACING P ROBABILISTIC C IRCUITS INTO N EURAL A NSWER S ET P ROGRAMMING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The goal of combining the robustness of neural networks and the expressivity of symbolic methods has rekindled the interest in Neuro-Symbolic AI.
Revised: The goal of combining the robustness of neural networks and the expressivity of symbolic methods has rekindled the interest in Neuro-Symbolic AI.

Original: Recent advancements in Neuro-Symbolic AI often consider speciﬁcally-tailored architectures consisting of disjoint neural and symbolic components, and thus do not exhibit desired gains that can be achieved by integrating them into a unifying framework.
Revised: Recent advancements in Neuro-Symbolic AI often consider speciﬁcally-tailored architectures consisting of disjoint neural and symbolic components, and thus do not exhibit desired gains that can be achieved by integrating them into a unifying framework.

Original: We introduce SLASH – a novel deep probabilistic programming language (DPPL).
Revised: We introduce SLASH – a novel deep probabilistic programming language (DPPL).

Original: At its core, SLASH consists of Neural-Probabilistic Predicates (NPPs) and logical programs which are united via answer set programming.
Revised: At its core, SLASH consists of Neural-Probabilistic Predicates (NPPs) and logical programs which are united via answer set programming.

Original: The probability estimates resulting from NPPs act as the binding element between the logical program and raw input data, thereby allowing SLASH to answer task-dependent logical queries.
Revised: The probability estimates resulting from NPPs act as the binding element between the logical program and raw input data, thereby allowing SLASH to answer task-dependent logical queries.

Original: This allows SLASH to elegantly integrate the symbolic and neural components in a uniﬁed framework.
Revised: This allows SLASH to elegantly integrate the symbolic and neural components in a uniﬁed framework.

Original: We evaluate SLASH on the benchmark data of MNIST addition as well as novel tasks for DPPLs such as missing data prediction and set prediction with state-of-the-art performance, thereby showing the effectiveness and generality of our method.
Revised: We evaluate SLASH on the benchmark data of MNIST addition as well as novel tasks for DPPLs such as missing data prediction and set prediction with state-of-the-art performance, thereby showing the effectiveness and generality of our method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, Neuro-Symbolic AI approaches to learning (Hudson & Manning, 2019;
Revised: In recent years, Neuro-Symbolic AI approaches to learning (Hudson & Manning, 2019;

Original: d’Avila
Revised: d’Avila

Original: Garcez et al., 2019; Jiang & Ahn, 2020; d’Avila Garcez & Lamb, 2020), which integrates low-level perception with high-level reasoning by combining data-driven neural modules with logic-based symbolic modules, has gained traction.
Revised: Garcez et al., 2019; Jiang & Ahn, 2020; d’Avila Garcez & Lamb, 2020), which integrates low-level perception with high-level reasoning by combining data-driven neural modules with logic-based symbolic modules, has gained traction.

Original: This combination of sub-symbolic and symbolic systems has been shown to have several advantages for various tasks such as visual question answering and reasoning (Yi et al., 2018), concept learning (Mao et al., 2019) and improved properties for explainable and revisable models (Ciravegna et al., 2020; Stammer et al., 2021).
Revised: This combination of sub-symbolic and symbolic systems has been shown to have several advantages for various tasks such as visual question answering and reasoning (Yi et al., 2018), concept learning (Mao et al., 2019) and improved properties for explainable and revisable models (Ciravegna et al., 2020; Stammer et al., 2021).

Original: Rather than designing speciﬁcally tailored Neuro-Symbolic architectures, where often the neural and symbolic modules are disjoint and trained independently (Yi et al., 2018; Mao et al., 2019; Stammer et al., 2021), deep probabilistic programming languages (DPPLs) provide an exciting alternative (Bingham et al., 2019; Tran et al., 2017; Manhaeve et al., 2018;
Revised: Rather than designing speciﬁcally tailored Neuro-Symbolic architectures, where often the neural and symbolic modules are disjoint and trained independently (Yi et al., 2018; Mao et al., 2019; Stammer et al., 2021), deep probabilistic programming languages (DPPLs) provide an exciting alternative (Bingham et al., 2019; Tran et al., 2017; Manhaeve et al., 2018;

Original: Yang et al., 2020).
Revised: Yang et al., 2020).

Original: Speciﬁcally, DPPLs integrate neural and symbolic modules via a unifying programming framework with probability estimates acting as the “glue” between separate modules allowing for reasoning over noisy, uncertain data and, importantly, joint training of the modules.
Revised: Speciﬁcally, DPPLs integrate neural and symbolic modules via a unifying programming framework with probability estimates acting as the “glue” between separate modules allowing for reasoning over noisy, uncertain data and, importantly, joint training of the modules.

Original: Additionally, prior knowledge and biases in the form of logical rules can easily be added with DPPLs, rather than creating implicit architectural biases, thereby integrating neural networks into downstream logical reasoning tasks.
Revised: Additionally, prior knowledge and biases in the form of logical rules can easily be added with DPPLs, rather than creating implicit architectural biases, thereby integrating neural networks into downstream logical reasoning tasks.

Original: Object-centric deep learning has recently brought forth several exciting avenues of research by introducing inductive biases to neural networks to extract objects from visual scenes in an unsupervised manner (Zhang et al., 2019; Burgess et al., 2019; Engelcke et al., 2020; Greff et al., 2019; Lin et al., 2020;
Revised: Object-centric deep learning has recently brought forth several exciting avenues of research by introducing inductive biases to neural networks to extract objects from visual scenes in an unsupervised manner (Zhang et al., 2019; Burgess et al., 2019; Engelcke et al., 2020; Greff et al., 2019; Lin et al., 2020;

Original: Locatello et al., 2020; Jiang & Ahn, 2020).
Revised: Locatello et al., 2020; Jiang & Ahn, 2020).

Original: We refer to Greff et al.
Revised: We refer to Greff et al.

Original: (2020) for a detailed overview.
Revised: (2020) for a detailed overview.

Original: A motivation for this speciﬁc line of investigation, which notably has been around for a longer period of time (Fodor & Pylyshyn, 1988; Marcus, 2019), is that objects occur as natural building blocks in human perception and possess advantageous properties for many cognitive tasks, such as scene understanding and reasoning.
Revised: A motivation for this speciﬁc line of investigation, which notably has been around for a longer period of time (Fodor & Pylyshyn, 1988; Marcus, 2019), is that objects occur as natural building blocks in human perception and possess advantageous properties for many cognitive tasks, such as scene understanding and reasoning.

Original: With a DPPL, these advancements can be improved by integrating the previously mentioned components into the DPPL’s programming framework and further adding constraints about objects and their properties in form of logical statements e.g. about color singularity, rather than implicitly enforcing this via one hot encodings.
Revised: With a DPPL, these advancements can be improved by integrating the previously mentioned components into the DPPL’s programming framework and further adding constraints about objects and their properties in form of logical statements e.g. about color singularity, rather than implicitly enforcing this via one hot encodings.

Original: The slot encoder is shared over all NPPs.
Revised: The slot encoder is shared over all NPPs.

Original: Each triangle in the ﬁgure represents a single EiNet that gives us a joint distribution at the root node.
Revised: Each triangle in the ﬁgure represents a single EiNet that gives us a joint distribution at the root node.

Original: Thus, each PC learns the joint distribution over slot encodings, z i , and object attributes, C , of a speciﬁc category, e.g. color attributes.
Revised: Thus, each PC learns the joint distribution over slot encodings, z i , and object attributes, C , of a speciﬁc category, e.g. color attributes.

Original: Via targeted queries to the NPPs, one can obtain task-related probabilities, e.g. conditional probabilities for the task of set prediction.
Revised: Via targeted queries to the NPPs, one can obtain task-related probabilities, e.g. conditional probabilities for the task of set prediction.

Original: Given the probability estimates from the NPP(s) and a SLASH program, containing a set of facts and logical statements about the world, the probability of the truth value of a task-related query are computed via answer set programming.
Revised: Given the probability estimates from the NPP(s) and a SLASH program, containing a set of facts and logical statements about the world, the probability of the truth value of a task-related query are computed via answer set programming.

Original: The entire system, including the neural and probabilistic modules, are ﬁnally trained end-to-end via a single loss function.
Revised: The entire system, including the neural and probabilistic modules, are ﬁnally trained end-to-end via a single loss function.

Original: We propose SLASH – a novel DPPL that, similar to the punctuation symbol, can be used to efﬁciently combine several paradigms into one.
Revised: We propose SLASH – a novel DPPL that, similar to the punctuation symbol, can be used to efﬁciently combine several paradigms into one.

Original: Speciﬁcally, SLASH represents a scalable programming language that seamlessly integrates probabilistic logical programming with neural representations and tractable probabilistic estimations.
Revised: Speciﬁcally, SLASH represents a scalable programming language that seamlessly integrates probabilistic logical programming with neural representations and tractable probabilistic estimations.

Original: Fig.
Revised: Fig.

Original: 1 shows an example instantiation of SLASH, termed
Revised: 1 shows an example instantiation of SLASH, termed

Original: SLASH Attention, for object-centric set prediction.
Revised: SLASH Attention, for object-centric set prediction.

Original: SLASH consists of several key building blocks.
Revised: SLASH consists of several key building blocks.

Original: Firstly, it makes use of Neural-Probabilistic Predicates (NPPs) for probability estimation.
Revised: Firstly, it makes use of Neural-Probabilistic Predicates (NPPs) for probability estimation.

Original: NPPs consist of neural and/or probabilistic circuit (PC) modules and act as a unifying term, encompassing the neural predicates of DeepProbLog and NeurASP, as well as purely probabilistic predicates.
Revised: NPPs consist of neural and/or probabilistic circuit (PC) modules and act as a unifying term, encompassing the neural predicates of DeepProbLog and NeurASP, as well as purely probabilistic predicates.

Original: In this work, we introduce a much more powerful “ﬂavor” of NPPs that consist jointly of neural and PC modules, taking advantage of the power of neural computations together with true density estimation of PCs.
Revised: In this work, we introduce a much more powerful “ﬂavor” of NPPs that consist jointly of neural and PC modules, taking advantage of the power of neural computations together with true density estimation of PCs.

Original: Depending on the underlying task one can thus ask a range of queries to the NPP, e.g. sample an unknown, desired variable, but also query for conditional class probabilities.
Revised: Depending on the underlying task one can thus ask a range of queries to the NPP, e.g. sample an unknown, desired variable, but also query for conditional class probabilities.

Original: Example
Revised: Example

Original: NPPs consisting of a slot attention encoder and several PCs are depicted in Fig.
Revised: NPPs consisting of a slot attention encoder and several PCs are depicted in Fig.

Original: 1 for the task of set prediction.
Revised: 1 for the task of set prediction.

Original: The slot encoder is shared across all NPPs, whereas the PC of each NPP models a separate category of attributes.
Revised: The slot encoder is shared across all NPPs, whereas the PC of each NPP models a separate category of attributes.

Original: In this way, each NPP models the joint distribution over slot encodings and object attribute values, such as the color of an object.
Revised: In this way, each NPP models the joint distribution over slot encodings and object attribute values, such as the color of an object.

Original: By querying the NPP, one can obtain task-related probability estimations, such as the conditional attribute probability.
Revised: By querying the NPP, one can obtain task-related probability estimations, such as the conditional attribute probability.

Original: The second component of SLASH is the logical program, which consists of a set of facts and logical statements deﬁning the state of the world of the underlying task.
Revised: The second component of SLASH is the logical program, which consists of a set of facts and logical statements deﬁning the state of the world of the underlying task.

Original: For example, one can deﬁne the rules for when an object possesses a speciﬁc set of attributes ( cf.
Revised: For example, one can deﬁne the rules for when an object possesses a speciﬁc set of attributes ( cf.

Original: Fig. 1).
Revised: Fig. 1).

Original: Thirdly, an ASP module is used to combine the ﬁrst two components.
Revised: Thirdly, an ASP module is used to combine the ﬁrst two components.

Original: Given a logical query about the input data, the logical program and the probability estimates obtained from the NPP(s), the ASP module produces a probability estimate about the truth value of the query, stating, e.g., how likely it is for a speciﬁc object in an image to be a large, dark red triangle.
Revised: Given a logical query about the input data, the logical program and the probability estimates obtained from the NPP(s), the ASP module produces a probability estimate about the truth value of the query, stating, e.g., how likely it is for a speciﬁc object in an image to be a large, dark red triangle.

Original: In contrast to query evaluation in Prolog (Colmerauer & Roussel, 1993;
Revised: In contrast to query evaluation in Prolog (Colmerauer & Roussel, 1993;

Original: Clocksin & Mellish, 1981) which may lead to an inﬁnite loop, many modern answer set solvers use
Revised: Clocksin & Mellish, 1981) which may lead to an inﬁnite loop, many modern answer set solvers use

Original: Conﬂict-Driven-Clause-Learning (CDPL) which, in principle, always terminates.
Revised: Conﬂict-Driven-Clause-Learning (CDPL) which, in principle, always terminates.

Original: Training in SLASH is performed efﬁciently in a batch-wise and end-to-end fashion, by integrating the parameters of all modules, neural and probabilistic, into a single loss term.
Revised: Training in SLASH is performed efﬁciently in a batch-wise and end-to-end fashion, by integrating the parameters of all modules, neural and probabilistic, into a single loss term.

Original: SLASH thus allows a simple, fast and effective integration of sub-symbolic and symbolic computations.
Revised: SLASH thus allows a simple, fast and effective integration of sub-symbolic and symbolic computations.

Original: In our experiments, we investigate the advantages of SLASH in comparison to SOTA DPPLs on the benchmark task of
Revised: In our experiments, we investigate the advantages of SLASH in comparison to SOTA DPPLs on the benchmark task of

Original: MNIST-Addition (Manhaeve et al., 2018).
Revised: MNIST-Addition (Manhaeve et al., 2018).

Original: We hereby show SLASH’s increased scalability regarding computation time, as well as SLASH’s ability to handle incomplete data via true probabilistic density modelling.
Revised: We hereby show SLASH’s increased scalability regarding computation time, as well as SLASH’s ability to handle incomplete data via true probabilistic density modelling.

Original: Next, we show that SLASH Attention provides superior results for set prediction in terms of accuracy and generalization abilities compared to a baseline slot attention encoder.
Revised: Next, we show that SLASH Attention provides superior results for set prediction in terms of accuracy and generalization abilities compared to a baseline slot attention encoder.

Original: With our experiments, we thus show that SLASH is a realization of “one system – two approaches” (Bengio, 2019), that can successfully be used for performing various tasks and on a variety of data types.
Revised: With our experiments, we thus show that SLASH is a realization of “one system – two approaches” (Bengio, 2019), that can successfully be used for performing various tasks and on a variety of data types.

Original: We make the following contributions: (1) We introduce neural-probabilistic predicates, efﬁciently integrating answer set programming with probabilistic inference via our novel DPPL, SLASH.
Revised: We make the following contributions: (1) We introduce neural-probabilistic predicates, efﬁciently integrating answer set programming with probabilistic inference via our novel DPPL, SLASH.

Original: (2) We successfully train neural, probabilistic and logic modules within SLASH for complex data structures end-to-end via a simple, single loss term.
Revised: (2) We successfully train neural, probabilistic and logic modules within SLASH for complex data structures end-to-end via a simple, single loss term.

Original: (3) We show that SLASH provides various advantages across a variety of tasks and data sets compared to state-of-the-art DPPLs and neural models.
Revised: (3) We show that SLASH provides various advantages across a variety of tasks and data sets compared to state-of-the-art DPPLs and neural models.

Original:  N EURO -S YMBOLIC L OGIC P ROGRAMMING
Revised:  N EURO -S YMBOLIC L OGIC P ROGRAMMING

Original: Neuro-Symbolic AI can be divided into two lines of research, depending on the starting point.
Revised: Neuro-Symbolic AI can be divided into two lines of research, depending on the starting point.

Original: Both, however, have the same ﬁnal goal: to combine low-level perception with logical constraints and reasoning.
Revised: Both, however, have the same ﬁnal goal: to combine low-level perception with logical constraints and reasoning.

Original: A key motivation of Neuro-Symbolic AI (d’Avila Garcez et al., 2009; Mao et al., 2019; Hudson &
Revised: A key motivation of Neuro-Symbolic AI (d’Avila Garcez et al., 2009; Mao et al., 2019; Hudson &

Original: Manning, 2019; d’Avila Garcez et al., 2019; Jiang & Ahn, 2020; d’Avila Garcez & Lamb, 2020) is to combine the advantages of symbolic and neural representations into a joint system.
Revised: Manning, 2019; d’Avila Garcez et al., 2019; Jiang & Ahn, 2020; d’Avila Garcez & Lamb, 2020) is to combine the advantages of symbolic and neural representations into a joint system.

Original: This is often done in a hybrid approach where a neural network acts as a perception module that interfaces with a symbolic reasoning system, e.g. (Mao et al., 2019;
Revised: This is often done in a hybrid approach where a neural network acts as a perception module that interfaces with a symbolic reasoning system, e.g. (Mao et al., 2019;

Original: Yi et al., 2018).
Revised: Yi et al., 2018).

Original: The goal of such an approach is to mitigate the issues of one type of representation by the other, e.g. using the power of symbolic reasoning systems to handle the generalizability issues of neural networks and on the other hand handle the difﬁculty of noisy data for symbolic systems via neural networks.
Revised: The goal of such an approach is to mitigate the issues of one type of representation by the other, e.g. using the power of symbolic reasoning systems to handle the generalizability issues of neural networks and on the other hand handle the difﬁculty of noisy data for symbolic systems via neural networks.

Original: Recent work has also shown the advantage of Neuro-Symbolic approaches for explaining and revising incorrect decisions (Ciravegna et al., 2020;
Revised: Recent work has also shown the advantage of Neuro-Symbolic approaches for explaining and revising incorrect decisions (Ciravegna et al., 2020;

Original: Stammer et al., 2021).
Revised: Stammer et al., 2021).

Original: Many of these previous works, however, train the sub-symbolic and symbolic modules separately.
Revised: Many of these previous works, however, train the sub-symbolic and symbolic modules separately.

Original: Deep Probabilistic Programming Languages (DPPLs) are programming languages that combine deep neural networks with probabilistic models and allow a user to express a probabilistic model via a logical program.
Revised: Deep Probabilistic Programming Languages (DPPLs) are programming languages that combine deep neural networks with probabilistic models and allow a user to express a probabilistic model via a logical program.

Original: Similar to Neuro-Symbolic architectures, DPPLs thereby unite the advantages of different paradigms.
Revised: Similar to Neuro-Symbolic architectures, DPPLs thereby unite the advantages of different paradigms.

Original: DPPLs are related to earlier works such as Markov Logic Networks (MLNs) (Richardson & Domingos, 2006).
Revised: DPPLs are related to earlier works such as Markov Logic Networks (MLNs) (Richardson & Domingos, 2006).

Original: Thereby, the binding link is the Weighted Model Counting (WMC) introduced in LP MLN (Lee & Wang, 2016).
Revised: Thereby, the binding link is the Weighted Model Counting (WMC) introduced in LP MLN (Lee & Wang, 2016).

Original: Several DPPLs have been proposed by now, among which are Pyro (Bingham et al., 2019), Edward (Tran et al., 2017), DeepProbLog (Manhaeve et al., 2018), and NeurASP (Yang et al., 2020).
Revised: Several DPPLs have been proposed by now, among which are Pyro (Bingham et al., 2019), Edward (Tran et al., 2017), DeepProbLog (Manhaeve et al., 2018), and NeurASP (Yang et al., 2020).

Original: To resolve the scalability issues of DeepProbLog, which use Sentential Decision Diagrams (SDDs) (Darwiche, 2011) as the underlying data structure to evaluate queries, NeurASP (Yang et al., 2020), offers a solution by utilizing Answer Set Programming (ASP)
Revised: To resolve the scalability issues of DeepProbLog, which use Sentential Decision Diagrams (SDDs) (Darwiche, 2011) as the underlying data structure to evaluate queries, NeurASP (Yang et al., 2020), offers a solution by utilizing Answer Set Programming (ASP)

Original: (Dimopoulos et al., 1997; Soininen &
Revised: (Dimopoulos et al., 1997; Soininen &

Original: Niemelä, 1999; Marek & Truszczynski, 1999; Calimeri et al., 2020).
Revised: Niemelä, 1999; Marek & Truszczynski, 1999; Calimeri et al., 2020).

Original: In this way, NeurASP changes the paradigm from query evaluation to model generation, i.e. instead of constructing an SDD or similar knowledge representation system, NeurASP generates a set of all possible solutions (one model per solution) and estimates the probability for the truth value of each of these solutions.
Revised: In this way, NeurASP changes the paradigm from query evaluation to model generation, i.e. instead of constructing an SDD or similar knowledge representation system, NeurASP generates a set of all possible solutions (one model per solution) and estimates the probability for the truth value of each of these solutions.

Original: Of those DPPLs that handle learning in a relational, probabilistic setting and in an end-to-end fashion, all of these are limited to estimating only conditional class probabilities.
Revised: Of those DPPLs that handle learning in a relational, probabilistic setting and in an end-to-end fashion, all of these are limited to estimating only conditional class probabilities.

Original:  T HE SLASH F RAMEWORK
Revised:  T HE SLASH F RAMEWORK

Original: In this section, we introduce our novel DPPL, SLASH.
Revised: In this section, we introduce our novel DPPL, SLASH.

Original: Before we dive into the details of this, it is necessary to ﬁrst introduce Neural-Probabilistic Predicates, for which we require an understanding of
Revised: Before we dive into the details of this, it is necessary to ﬁrst introduce Neural-Probabilistic Predicates, for which we require an understanding of

Original: Probabilistic Circuits.
Revised: Probabilistic Circuits.

Original: Finally, we will present the learning paradigm of SLASH.
Revised: Finally, we will present the learning paradigm of SLASH.

Original: The term probabilistic circuit (PC) (Choi et al., 2020) represents a unifying framework that encompasses all computational graphs which encode probability distributions and guarantee tractable probabilistic modelling.
Revised: The term probabilistic circuit (PC) (Choi et al., 2020) represents a unifying framework that encompasses all computational graphs which encode probability distributions and guarantee tractable probabilistic modelling.

Original: These include Sum-Product Networks (SPNs) (Poon & Domingos, 2011) which are deep mixture models represented via a rooted directed acyclic graphs with a recursively deﬁned structure.
Revised: These include Sum-Product Networks (SPNs) (Poon & Domingos, 2011) which are deep mixture models represented via a rooted directed acyclic graphs with a recursively deﬁned structure.

Original: [Figure]
Revised: [Figure]

Original: slash symbol)
Revised: slash symbol)

Original: only one
Revised: only one

Original: of these two
Revised: of these two

Original: (b) A minimal SLASH program and query for the set prediction task, here only showing the NPP that models the color category per object.
Revised: (b) A minimal SLASH program and query for the set prediction task, here only showing the NPP that models the color category per object.

Original: For the full program, we
Revised: For the full program, we

Original: refer
Revised: refer

Original: to
Revised: to

Original: the Appendix.
Revised: the Appendix.

Original: 3. N EURAL -P ROBABILISTIC P REDICATES
Revised: 3. N EURAL -P ROBABILISTIC P REDICATES

Original: Previous DPPLs, DeepProbLog (Manhaeve et al., 2018) and NeurASP (Yang et al., 2020), introduced the Neural Predicate as an annotated-disjunction or as a propositional atom, respectively, to acquire conditional class probabilities, P ( C | X ) , via the softmax function at the output of an arbitrary
Revised: Previous DPPLs, DeepProbLog (Manhaeve et al., 2018) and NeurASP (Yang et al., 2020), introduced the Neural Predicate as an annotated-disjunction or as a propositional atom, respectively, to acquire conditional class probabilities, P ( C | X ) , via the softmax function at the output of an arbitrary

Original: DNN.
Revised: DNN.

Original: As mentioned in the introduction, this approach has certain limitations concerning inference capabilities.
Revised: As mentioned in the introduction, this approach has certain limitations concerning inference capabilities.

Original: To resolve this issue, we introduce Neural-Probabilisitic Predicates (NPPs).
Revised: To resolve this issue, we introduce Neural-Probabilisitic Predicates (NPPs).

Original: Formally, we denote with
Revised: Formally, we denote with

Original: [Equation]
Revised: [Equation]

Original: a Neural-Probabilistic Predicate h .
Revised: a Neural-Probabilistic Predicate h .

Original: Thereby, (i) npp is a reserved word to label an NPP, (ii) h a symbolic name of either a PC, NN or a joint of a PC and NN ( cf.
Revised: Thereby, (i) npp is a reserved word to label an NPP, (ii) h a symbolic name of either a PC, NN or a joint of a PC and NN ( cf.

Original: Fig.
Revised: Fig.

Original: 2a), e.g., color_attr is the name of an NPP of Fig. 2b.
Revised: 2a), e.g., color_attr is the name of an NPP of Fig. 2b.

Original: Additionally, (iii) x denotes a “term” and (iv) v 1 , . .
Revised: Additionally, (iii) x denotes a “term” and (iv) v 1 , . .

Original: . , v n are placeholders for each of the n possible outcomes of h .
Revised: . , v n are placeholders for each of the n possible outcomes of h .

Original: For example, the placeholders for color_attr are the color attributes of an object ( Red, Blue, Green, etc. ).
Revised: For example, the placeholders for color_attr are the color attributes of an object ( Red, Blue, Green, etc. ).

Original: An NPP abbreviates an arithmetic literal of the form c = v with c ∈ { h ( x ) } and v ∈ { v 1 , . .
Revised: An NPP abbreviates an arithmetic literal of the form c = v with c ∈ { h ( x ) } and v ∈ { v 1 , . .

Original: . , v n } .
Revised: . , v n } .

Original: Furthermore, we denote with Π npp a set of NPPs of the form stated in (Eq. 1) and r npp the set of all rules c = v of one NPP, which denotes the possible outcomes, obtained from an NPP in Π npp , e.g. r color _ attr =
Revised: Furthermore, we denote with Π npp a set of NPPs of the form stated in (Eq. 1) and r npp the set of all rules c = v of one NPP, which denotes the possible outcomes, obtained from an NPP in Π npp , e.g. r color _ attr =

Original: { c = Red, c
Revised: { c = Red, c

Original: = Blue, c = Green, ... } for the example depicted in Fig. 2b.
Revised: = Blue, c = Green, ... } for the example depicted in Fig. 2b.

Original: Rules of the form npp ( h ( x ) , [ v 1 , . .
Revised: Rules of the form npp ( h ( x ) , [ v 1 , . .

Original: . , v n ]) ← Body are used as an abbreviation for application to multiple entities, e.g. multiple slots for the task of set prediction ( cf.
Revised: . , v n ]) ← Body are used as an abbreviation for application to multiple entities, e.g. multiple slots for the task of set prediction ( cf.

Original: Fig.
Revised: Fig.

Original: 2b).
Revised: 2b).

Original: Hereby, Body of the rule is identiﬁed by (cid:62) (tautology, true) or ⊥ (contradiction, false) during grounding.
Revised: Hereby, Body of the rule is identiﬁed by (cid:62) (tautology, true) or ⊥ (contradiction, false) during grounding.

Original: Rules of the form Head ← Body with r npp appearing in Head are prohibited for Π npp .
Revised: Rules of the form Head ← Body with r npp appearing in Head are prohibited for Π npp .

Original: In this work, we largely make use of NPPs that contain probabilistic circuits (speciﬁcally SPNs) which allow for tractable density estimation and modelling of joint probabilities.
Revised: In this work, we largely make use of NPPs that contain probabilistic circuits (speciﬁcally SPNs) which allow for tractable density estimation and modelling of joint probabilities.

Original: In this way, it is possible to answer a much richer set of probabilistic queries, i.e. P ( X, C ) , P ( X | C ) and P ( C | X ) .
Revised: In this way, it is possible to answer a much richer set of probabilistic queries, i.e. P ( X, C ) , P ( X | C ) and P ( C | X ) .

Original: In addition to this, we introduce the arguably more interesting type of NPP that combines a neural module with a PC.
Revised: In addition to this, we introduce the arguably more interesting type of NPP that combines a neural module with a PC.

Original: Hereby, the neural module learns to map the raw input data into an optimal latent representation, e.g. object-based slot representations.
Revised: Hereby, the neural module learns to map the raw input data into an optimal latent representation, e.g. object-based slot representations.

Original: The PC, in turn, learns to model the joint distribution of these latent variables and produces the ﬁnal probability estimates.
Revised: The PC, in turn, learns to model the joint distribution of these latent variables and produces the ﬁnal probability estimates.

Original: This type of
Revised: This type of

Original: NPP nicely combines the representational power of neural networks with the advantages of PCs in probability estimation and query ﬂexibility.
Revised: NPP nicely combines the representational power of neural networks with the advantages of PCs in probability estimation and query ﬂexibility.

Original: For making the different probabilistic queries distinguishable in a SLASH program, we introduce the following notation.
Revised: For making the different probabilistic queries distinguishable in a SLASH program, we introduce the following notation.

Original: We denote a given variable with + and the query variable with − .
Revised: We denote a given variable with + and the query variable with − .

Original: E.g., within the running example of set prediction ( cf. Fig.
Revised: E.g., within the running example of set prediction ( cf. Fig.

Original: 1 and 2b), with the query color _ attr (+ X, − C ) one is asking for P ( C | X ) .
Revised: 1 and 2b), with the query color _ attr (+ X, − C ) one is asking for P ( C | X ) .

Original: Similarly, with color _ attr ( − X, + C ) one is asking for P ( X | C ) and, ﬁnally, with color _
Revised: Similarly, with color _ attr ( − X, + C ) one is asking for P ( X | C ) and, ﬁnally, with color _

Original: attr ( − X, − C ) for P ( X, C ) .
Revised: attr ( − X, − C ) for P ( X, C ) .

Original: To summarize, an NPP can consist of neural and/or probabilistic modules and produces querydependent probability estimates.
Revised: To summarize, an NPP can consist of neural and/or probabilistic modules and produces querydependent probability estimates.

Original: Due to the ﬂexibility of its deﬁnition, the term NPP contains the predicates of previous works (Manhaeve et al., 2018;
Revised: Due to the ﬂexibility of its deﬁnition, the term NPP contains the predicates of previous works (Manhaeve et al., 2018;

Original: Yang et al., 2020), but also more interesting predicates discussed above.
Revised: Yang et al., 2020), but also more interesting predicates discussed above.

Original: The speciﬁc “ﬂavor” of an NPP should be chosen depending on what type of probability estimation is required ( cf.
Revised: The speciﬁc “ﬂavor” of an NPP should be chosen depending on what type of probability estimation is required ( cf.

Original: Fig 2a).
Revised: Fig 2a).

Original: Lastly, NPPs have the uniﬁed loss function of the negative log-likelihood:
Revised: Lastly, NPPs have the uniﬁed loss function of the negative log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: whereby we are assuming the data to be i.i.d
Revised: whereby we are assuming the data to be i.i.d

Original: ., ground truth x i to be the all-ones vector, ξ to be the parameters of the NPP and P ( X,C ) ξ are the predictions ˆ x i obtained from the PC encoded in the NPP.
Revised: ., ground truth x i to be the all-ones vector, ξ to be the parameters of the NPP and P ( X,C ) ξ are the predictions ˆ x i obtained from the PC encoded in the NPP.

Original: 3. T HE SLASH L ANGUAGE AND S EMANTICS
Revised: 3. T HE SLASH L ANGUAGE AND S EMANTICS

Original: Fig.
Revised: Fig.

Original: 1 presents an illustration of SLASH, exempliﬁed for the task of set prediction, with all of its key components.
Revised: 1 presents an illustration of SLASH, exempliﬁed for the task of set prediction, with all of its key components.

Original: Having introduced the NPPs previously, which produce probability estimates, we now continue in the pipeline on how to use these probability estimates for answering logical queries.
Revised: Having introduced the NPPs previously, which produce probability estimates, we now continue in the pipeline on how to use these probability estimates for answering logical queries.

Original: We begin by formally deﬁning a SLASH program.
Revised: We begin by formally deﬁning a SLASH program.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A SLASH program Π is the union of Π asp , Π npp .
Revised: A SLASH program Π is the union of Π asp , Π npp .

Original: Therewith, Π asp is the set of propositional rules (standard rules from ASP-Core-2 (Calimeri et al., 2020)), and Π npp is a set of
Revised: Therewith, Π asp is the set of propositional rules (standard rules from ASP-Core-2 (Calimeri et al., 2020)), and Π npp is a set of

Original: Neural-Probabilistic Predicates of the form stated in Eq.
Revised: Neural-Probabilistic Predicates of the form stated in Eq.

Original: Fig.
Revised: Fig.

Original: 2b depicts a minimal SLASH program for the task of set prediction, exemplifying a set of propositional rules and neural predicates.
Revised: 2b depicts a minimal SLASH program for the task of set prediction, exemplifying a set of propositional rules and neural predicates.

Original: Similar to NeurASP, SLASH requires ASP and as such adopts its syntax to most part.
Revised: Similar to NeurASP, SLASH requires ASP and as such adopts its syntax to most part.

Original: We therefore now address integrating our NPPs into an ASP compatible form to obtain the success probability for the logical query given all possible solutions.
Revised: We therefore now address integrating our NPPs into an ASP compatible form to obtain the success probability for the logical query given all possible solutions.

Original: Thus, we deﬁne SLASH’s semantics .
Revised: Thus, we deﬁne SLASH’s semantics .

Original: For SLASH to translate the program Π to the ASP-solver’s compatible form, the rules (Eq. 1) will be rewritten to the set of rules:
Revised: For SLASH to translate the program Π to the ASP-solver’s compatible form, the rules (Eq. 1) will be rewritten to the set of rules:

Original: [Equation]
Revised: [Equation]

Original: The ASP-solver should understand this as “Pick exactly one rule from the set”.
Revised: The ASP-solver should understand this as “Pick exactly one rule from the set”.

Original: After the translation is done, we can ask an ASP-solver for the solutions for Π .
Revised: After the translation is done, we can ask an ASP-solver for the solutions for Π .

Original: We denote a set of ASP constraints in the form ← Body , as queries Q (annotation).
Revised: We denote a set of ASP constraints in the form ← Body , as queries Q (annotation).

Original: and each of the solutions with respect to Q as a potential solution , I , (referred to as stable model in ASP).
Revised: and each of the solutions with respect to Q as a potential solution , I , (referred to as stable model in ASP).

Original: With I | r
Revised: With I | r

Original: npp we denote the projection of the I onto r npp , Num ( I | r npp , Π) – the number of the possible solutions of the program Π agreeing with I | r npp on r npp .
Revised: npp we denote the projection of the I onto r npp , Num ( I | r npp , Π) – the number of the possible solutions of the program Π agreeing with I | r npp on r npp .

Original: Because we aim to calculate the success probability of the query Q , we formalize the probability of a potential solution I beforehand.
Revised: Because we aim to calculate the success probability of the query Q , we formalize the probability of a potential solution I beforehand.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: We specify the probability of the potential solution, I , for the program Π as the product of the probabilities of all atoms c = v in I | r npp divided by the number of potential solutions of Π agreeing with I | r npp on r npp :
Revised: We specify the probability of the potential solution, I , for the program Π as the product of the probabilities of all atoms c = v in I | r npp divided by the number of potential solutions of Π agreeing with I | r npp on r npp :

Original: [Equation]
Revised: [Equation]

Original: Therefore, the probability of a query can be deﬁned as follows.
Revised: Therefore, the probability of a query can be deﬁned as follows.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: The probability of the query Q given the set of possible solutions I is deﬁned as
Revised: The probability of the query Q given the set of possible solutions I is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Thereby, I | =
Revised: Thereby, I | =

Original: Q reads as “ I satisﬁes Q ”.
Revised: Q reads as “ I satisﬁes Q ”.

Original: The probability of the set of queries Q = { Q 1 , . .
Revised: The probability of the set of queries Q = { Q 1 , . .

Original: . , Q l } is deﬁned as the product of the probability of each.
Revised: . , Q l } is deﬁned as the product of the probability of each.

Original: I.e.
Revised: I.e.

Original: [Equation]
Revised: [Equation]

Original: 3. P ARAMETER L EARNING IN SLASH
Revised: 3. P ARAMETER L EARNING IN SLASH

Original: We denote with Π( θ ) the SLASH program under consideration, thereby θ is the set of the parameters associated with Π .
Revised: We denote with Π( θ ) the SLASH program under consideration, thereby θ is the set of the parameters associated with Π .

Original: Further, making the i.i.d.
Revised: Further, making the i.i.d.

Original: assumption of the query set Q , we follow Manhaeve et al.
Revised: assumption of the query set Q , we follow Manhaeve et al.

Original: (2018) and Skryagin et al.
Revised: (2018) and Skryagin et al.

Original: (2020), and use the learning from entailment setting.
Revised: (2020), and use the learning from entailment setting.

Original: That is, the training examples are logical queries that are known to be true in the SLASH program
Revised: That is, the training examples are logical queries that are known to be true in the SLASH program

Original: Π( θ ) .
Revised: Π( θ ) .

Original: The goal is now to learn the parameters θ of the SLASH program Π( θ ) so that the observed queries are most likely.
Revised: The goal is now to learn the parameters θ of the SLASH program Π( θ ) so that the observed queries are most likely.

Original: To this end, we employ the negative log-likelihood and the cross-entropy of the observed queries
Revised: To this end, we employ the negative log-likelihood and the cross-entropy of the observed queries

Original: P Π( θ ) ( Q i ) and their predicted probability value P ( X Q ,C ) ( x Q i ) , assuming the NPPs are ﬁxed:
Revised: P Π( θ ) ( Q i ) and their predicted probability value P ( X Q ,C ) ( x Q i ) , assuming the NPPs are ﬁxed:

Original: L ENT :=
Revised: L ENT :=

Original: [Equation]
Revised: [Equation]

Original: This loss function aims at maximizing the estimated success probability.
Revised: This loss function aims at maximizing the estimated success probability.

Original: We remark that the deﬁned loss function is true regardless of the NPP’s form (NN with Softmax, PC or PC jointly with NN).
Revised: We remark that the deﬁned loss function is true regardless of the NPP’s form (NN with Softmax, PC or PC jointly with NN).

Original: The only difference will be the second term, i.e. P ( C | X Q ) ( x Q ) or P ( X Q | C ) ( x Q )) depending on the NPP and task.
Revised: The only difference will be the second term, i.e. P ( C | X Q ) ( x Q ) or P ( X Q | C ) ( x Q )) depending on the NPP and task.

Original: Furthermore, we assume that for the set of queries Q holds
Revised: Furthermore, we assume that for the set of queries Q holds

Original: [Equation]
Revised: [Equation]

Original: In accordance with the semantics, we seek to reward the right solutions v = c and penalize wrong ones v (cid:54)
Revised: In accordance with the semantics, we seek to reward the right solutions v = c and penalize wrong ones v (cid:54)

Original: =
Revised: =

Original: c .
Revised: c .

Original: Referring to the probabilities in r npp (the set of logical rules denoting NPPs, see Def. 2) as p , one can compute their gradients w.r.t.
Revised: Referring to the probabilities in r npp (the set of logical rules denoting NPPs, see Def. 2) as p , one can compute their gradients w.r.t.

Original: θ via backpropagation as
Revised: θ via backpropagation as

Original: [Equation]
Revised: [Equation]

Original: The term ∂ p ∂ θ can now be computed as usual via backward propagation through the NPPs (see Eq.in the appendix for details).
Revised: The term ∂ p ∂ θ can now be computed as usual via backward propagation through the NPPs (see Eq.in the appendix for details).

Original: By letting p to be the label of the probability of an atom c = v in r npp and denoting P Π( θ ) ( c = v ) , the term ∂ log ( P Π( θ ) ( Q ) )
Revised: By letting p to be the label of the probability of an atom c = v in r npp and denoting P Π( θ ) ( c = v ) , the term ∂ log ( P Π( θ ) ( Q ) )

Original: ∂ p follows from NeurASP (Yang et al., 2020) as
Revised: ∂ p follows from NeurASP (Yang et al., 2020) as

Original: [Equation]
Revised: [Equation]

Original: This is sensible.
Revised: This is sensible.

Original: For instance, if a query to be true is not likely to be entailed, the gradient is positive.
Revised: For instance, if a query to be true is not likely to be entailed, the gradient is positive.

Original: Putting everything together, the ﬁnal loss function is
Revised: Putting everything together, the ﬁnal loss function is

Original: [Equation]
Revised: [Equation]

Original: and we perform training using coordinate descent, i.e., we train the NPPs, the train the program with ﬁxed NPPs, train the NPPs with the program ﬁxed, and so on.
Revised: and we perform training using coordinate descent, i.e., we train the NPPs, the train the program with ﬁxed NPPs, train the NPPs with the program ﬁxed, and so on.

Original: In hindsight, rather than requiring a novel loss function for each individual task and data set, with
Revised: In hindsight, rather than requiring a novel loss function for each individual task and data set, with

Original: SLASH, it is possible to simply incorporate the speciﬁc requirements into the logic program.
Revised: SLASH, it is possible to simply incorporate the speciﬁc requirements into the logic program.

Original: The training loss, however, remains the same.
Revised: The training loss, however, remains the same.

Original: We refer to the Appendix A for further details, including the derivation of the total loss gradient.
Revised: We refer to the Appendix A for further details, including the derivation of the total loss gradient.

Original:  E MPIRICAL R ESULTS
Revised:  E MPIRICAL R ESULTS

Original: The advantage of SLASH lies in the efﬁcient integration of neural, probabilistic and symbolic computations.
Revised: The advantage of SLASH lies in the efﬁcient integration of neural, probabilistic and symbolic computations.

Original: To emphasize this, we conduct a variety of experimental evaluations.
Revised: To emphasize this, we conduct a variety of experimental evaluations.

Original: Experimental Details.
Revised: Experimental Details.

Original: We use two benchmark data sets, namely MNIST (LeCun et al., 1998b) for the task of MNIST-Addition and a variant of the ShapeWorld data set (Kuhnle & Copestake, 2017) for
Revised: We use two benchmark data sets, namely MNIST (LeCun et al., 1998b) for the task of MNIST-Addition and a variant of the ShapeWorld data set (Kuhnle & Copestake, 2017) for

Original: [Table]
Revised: [Table]

Original: object-centric set prediction.
Revised: object-centric set prediction.

Original: For all experiments we present the average and the standard deviation over ﬁve runs with different random seeds for parameter initialization.
Revised: For all experiments we present the average and the standard deviation over ﬁve runs with different random seeds for parameter initialization.

Original: For ShapeWorld experiments
Revised: For ShapeWorld experiments

Original: [Table]
Revised: [Table]

Original: we generate a data set we refer to as ShapeWorld4.
Revised: we generate a data set we refer to as ShapeWorld4.

Original: Images of
Revised: Images of

Original: ShapeWorld4 contain between one and four objects, with each object consisting of four attributes: a color (red, blue, green, gray, brown, magenta, cyan or yellow), a shade (bright, or dark), a shape (circle, triangle or square) and a size (small or big).
Revised: ShapeWorld4 contain between one and four objects, with each object consisting of four attributes: a color (red, blue, green, gray, brown, magenta, cyan or yellow), a shade (bright, or dark), a shape (circle, triangle or square) and a size (small or big).

Original: Thus, each object can be created from 84 different combinations of attributes.
Revised: Thus, each object can be created from 84 different combinations of attributes.

Original: Fig.
Revised: Fig.

Original: 1 depicts an example image.
Revised: 1 depicts an example image.

Original: We measure performance via classiﬁcation accuracies in the MNIST-Addition task.
Revised: We measure performance via classiﬁcation accuracies in the MNIST-Addition task.

Original: In our ShapeWorld4 experiments, we present the average precision.
Revised: In our ShapeWorld4 experiments, we present the average precision.

Original: We refer to appendix B for the SLASH programs and queries of each experiment, and appendix C for a detailed description of hyperparameters and further details.
Revised: We refer to appendix B for the SLASH programs and queries of each experiment, and appendix C for a detailed description of hyperparameters and further details.

Original: Evaluation 1: SLASH outperforms SOTA DPPLs in MNIST-Addition.
Revised: Evaluation 1: SLASH outperforms SOTA DPPLs in MNIST-Addition.

Original: The task of MNISTAddition (Manhaeve et al., 2018) is to predict the sum of two MNIST digits, presented only as raw images.
Revised: The task of MNISTAddition (Manhaeve et al., 2018) is to predict the sum of two MNIST digits, presented only as raw images.

Original: During test time, however, a model should classify the images directly.
Revised: During test time, however, a model should classify the images directly.

Original: Thus, although a model does not receive explicit information about the depicted digits, it must learn to identify digits via indirect feedback on the sum prediction.
Revised: Thus, although a model does not receive explicit information about the depicted digits, it must learn to identify digits via indirect feedback on the sum prediction.

Original: We compare the test accuracy after convergence between the three DPPLs: DeepProbLog (Manhaeve et al., 2018), NeurASP (Yang et al., 2020) and SLASH, using a probabilistic circuit (PC) or a deep neural network (DNN) as NPP.
Revised: We compare the test accuracy after convergence between the three DPPLs: DeepProbLog (Manhaeve et al., 2018), NeurASP (Yang et al., 2020) and SLASH, using a probabilistic circuit (PC) or a deep neural network (DNN) as NPP.

Original: Notably, the DNN used in SLASH (DNN) is the LeNet5 model (LeCun et al., 1998a) of DeepProbLog and NeurASP.
Revised: Notably, the DNN used in SLASH (DNN) is the LeNet5 model (LeCun et al., 1998a) of DeepProbLog and NeurASP.

Original: We note that when using the PC as NPP, we have also extracted conditional class probabilities P ( C | X ) , by marginalizing the class variables C to acquire the normalization constant P ( X ) from the joint P ( X, C ) , and calculating P ( X | C ) .
Revised: We note that when using the PC as NPP, we have also extracted conditional class probabilities P ( C | X ) , by marginalizing the class variables C to acquire the normalization constant P ( X ) from the joint P ( X, C ) , and calculating P ( X | C ) .

Original: The results can be seen in Tab. 1a.
Revised: The results can be seen in Tab. 1a.

Original: We observe that training SLASH with a DNN NPP produces
Revised: We observe that training SLASH with a DNN NPP produces

Original: SOTA accuracies compared to DeepProbLog and NeurASP, conﬁrming that SLASH’s batch-wise loss computation leads to improved performances.
Revised: SOTA accuracies compared to DeepProbLog and NeurASP, conﬁrming that SLASH’s batch-wise loss computation leads to improved performances.

Original: We further observe that the test accuracy of
Revised: We further observe that the test accuracy of

Original: SLASH with a PC NPP is slightly below the other DPPLs, however we argue that this may be since a PC, in comparison to a DNN, is learning a true mixture density rather than just conditional probabilities.
Revised: SLASH with a PC NPP is slightly below the other DPPLs, however we argue that this may be since a PC, in comparison to a DNN, is learning a true mixture density rather than just conditional probabilities.

Original: The advantages of doing so will be investigated in the next experiments.
Revised: The advantages of doing so will be investigated in the next experiments.

Original: Note that, optimal architecture search for PCs, e.g. for computer vision, is an open research question.
Revised: Note that, optimal architecture search for PCs, e.g. for computer vision, is an open research question.

Original: These evaluations show SLASH’s advantages on the benchmark MNIST-Addition task.
Revised: These evaluations show SLASH’s advantages on the benchmark MNIST-Addition task.

Original: Additional beneﬁts will be made clear in the following experiments.
Revised: Additional beneﬁts will be made clear in the following experiments.

Original: Evaluation 2: Handling Missing Data with SLASH.
Revised: Evaluation 2: Handling Missing Data with SLASH.

Original: SLASH offers the advantage of its ﬂexibility to use various kinds of NPPs.
Revised: SLASH offers the advantage of its ﬂexibility to use various kinds of NPPs.

Original: Thus, in comparison to previous DPPLs, one can easily integrate
Revised: Thus, in comparison to previous DPPLs, one can easily integrate

Original: NPPs into SLASH that perform joint probability estimation.
Revised: NPPs into SLASH that perform joint probability estimation.

Original: For this evaluation, we consider the task of MNIST-Addition with missing data.
Revised: For this evaluation, we consider the task of MNIST-Addition with missing data.

Original: We trained SLASH (PC) and DeepProbLog with the
Revised: We trained SLASH (PC) and DeepProbLog with the

Original: MNIST-Addition task with images in which a percentage of pixels per image has been removed.
Revised: MNIST-Addition task with images in which a percentage of pixels per image has been removed.

Original: It is important to mention here that whereas DeepProbLog handles the missing data simply as background pixels, SLASH (PC) speciﬁcally models the missing data as uncertain data by marginalizing the denoted pixels at inference time.
Revised: It is important to mention here that whereas DeepProbLog handles the missing data simply as background pixels, SLASH (PC) speciﬁcally models the missing data as uncertain data by marginalizing the denoted pixels at inference time.

Original: We use DeepProbLog here representative of DPPLs without true density estimation.
Revised: We use DeepProbLog here representative of DPPLs without true density estimation.

Original: The results can be seen in Tab.
Revised: The results can be seen in Tab.

Original: 1b for 50% , 80% , 90% and 97% missing pixels per image.
Revised: 1b for 50% , 80% , 90% and 97% missing pixels per image.

Original: We observe that at 50% , DeepProbLog and SLASH produce almost equal accuracies.
Revised: We observe that at 50% , DeepProbLog and SLASH produce almost equal accuracies.

Original: With 80% percent missing pixels, there is a substantial difference in the ability of the two DPPLs to correctly classify images, with SLASH being very stable.
Revised: With 80% percent missing pixels, there is a substantial difference in the ability of the two DPPLs to correctly classify images, with SLASH being very stable.

Original: By further increasing the percentage of missing pixels, this difference becomes even more substantial with SLASH still reaching a 82% test accuracy even when 97% of the pixels per image are missing, whereas DeepProbLog degrades to an average of 32% test accuracy.
Revised: By further increasing the percentage of missing pixels, this difference becomes even more substantial with SLASH still reaching a 82% test accuracy even when 97% of the pixels per image are missing, whereas DeepProbLog degrades to an average of 32% test accuracy.

Original: We further note that SLASH, in comparison to DeepProbLog, produces largely reduced standard deviations over runs.
Revised: We further note that SLASH, in comparison to DeepProbLog, produces largely reduced standard deviations over runs.

Original: Thus, by utilizing the power of true density estimation SLASH, with an appropriate NPP, can produce more robust results in comparison to other DPPLs.
Revised: Thus, by utilizing the power of true density estimation SLASH, with an appropriate NPP, can produce more robust results in comparison to other DPPLs.

Original: Further, we refer to Appendix D, which contains results of additional experiments where training is performed with the full MNIST data set whereas only the test set entails different rates of missing pixels.
Revised: Further, we refer to Appendix D, which contains results of additional experiments where training is performed with the full MNIST data set whereas only the test set entails different rates of missing pixels.

Original: Evaluation 3:
Revised: Evaluation 3:

Original: Improved Concept Learning via SLASH.
Revised: Improved Concept Learning via SLASH.

Original: We show that SLASH can be very effective for the complex task of set prediction, which previous DPPLs have not tackled.
Revised: We show that SLASH can be very effective for the complex task of set prediction, which previous DPPLs have not tackled.

Original: We revert to the ShapeWorld4 data set for this setting.
Revised: We revert to the ShapeWorld4 data set for this setting.

Original: For set prediction, a model is trained to predict the discrete attributes of a set of objects in an image ( cf.
Revised: For set prediction, a model is trained to predict the discrete attributes of a set of objects in an image ( cf.

Original: Fig.
Revised: Fig.

Original: 1 for an example ShapeWorld4 image).
Revised: 1 for an example ShapeWorld4 image).

Original: The difﬁculty for the model lies therein that it must match an unordered set of corresponding attributes (with varying number of entities over samples) with its internal representations of the image.
Revised: The difﬁculty for the model lies therein that it must match an unordered set of corresponding attributes (with varying number of entities over samples) with its internal representations of the image.

Original: The slot attention module introduced by Locatello et al.
Revised: The slot attention module introduced by Locatello et al.

Original: (2020) allows for an attractive object-centric approach to this task.
Revised: (2020) allows for an attractive object-centric approach to this task.

Original: Speciﬁcally, this module represents a pluggable, differentiable module that can be easily added to any architecture and, through a competitive softmax-based attention mechanism, can enforce the binding of speciﬁc parts of a latent representation into permutation-invariant, taskspeciﬁc vectors, called slots.
Revised: Speciﬁcally, this module represents a pluggable, differentiable module that can be easily added to any architecture and, through a competitive softmax-based attention mechanism, can enforce the binding of speciﬁc parts of a latent representation into permutation-invariant, taskspeciﬁc vectors, called slots.

Original: In our experiments, we wish to show that by adding logical constraints to the training setting, one can improve the overall performances and generalization properties of such a model.
Revised: In our experiments, we wish to show that by adding logical constraints to the training setting, one can improve the overall performances and generalization properties of such a model.

Original: For this, we train SLASH with NPPs as depicted in Fig.
Revised: For this, we train SLASH with NPPs as depicted in Fig.

Original: 1 consisting of a shared slot encoder and separate PCs, each modelling the mixture of latent slot variables and the attributes of one category, e.g. color.
Revised: 1 consisting of a shared slot encoder and separate PCs, each modelling the mixture of latent slot variables and the attributes of one category, e.g. color.

Original: For ShapeWorld4, we thereby have altogether four NPPs.
Revised: For ShapeWorld4, we thereby have altogether four NPPs.

Original: SLASH is trained via queries of the kind exempliﬁed in Fig.
Revised: SLASH is trained via queries of the kind exempliﬁed in Fig.

Original: 7 in the Appendix.
Revised: 7 in the Appendix.

Original: We refer to this conﬁguration as SLASH Attention .
Revised: We refer to this conﬁguration as SLASH Attention .

Original: We compare SLASH Attention to a baseline slot attention encoder using an MLP and Hungarian loss for predicting the object properties from the slot encodings as in Locatello et al.
Revised: We compare SLASH Attention to a baseline slot attention encoder using an MLP and Hungarian loss for predicting the object properties from the slot encodings as in Locatello et al.

Original: The results of these experiments can be found in Fig.
Revised: The results of these experiments can be found in Fig.

Original: 3a (top).
Revised: 3a (top).

Original: We observe that the average precision after convergence on the held-out test set with SLASH Attention is greatly improved to that of the baseline model.
Revised: We observe that the average precision after convergence on the held-out test set with SLASH Attention is greatly improved to that of the baseline model.

Original: Additionally, in Fig.
Revised: Additionally, in Fig.

Original: 3b we observe that SLASH Attention reaches the average precision value of the baseline model in much fewer number of epochs.
Revised: 3b we observe that SLASH Attention reaches the average precision value of the baseline model in much fewer number of epochs.

Original: Thus, we can summarize that adding logical knowledge in the training procedure via SLASH can greatly improve the capabilities of a neural module for set prediction.
Revised: Thus, we can summarize that adding logical knowledge in the training procedure via SLASH can greatly improve the capabilities of a neural module for set prediction.

Original: Evaluation 4:
Revised: Evaluation 4:

Original: Improved Compositional Generalization with SLASH.
Revised: Improved Compositional Generalization with SLASH.

Original: To test the hypothesis that
Revised: To test the hypothesis that

Original: SLASH Attention possesses improved generalization properties in comparison to the baseline model, we ran experiments on a variant of ShapeWorld4 similar to the CLEVR Compositional Generalization
Revised: SLASH Attention possesses improved generalization properties in comparison to the baseline model, we ran experiments on a variant of ShapeWorld4 similar to the CLEVR Compositional Generalization

Original: Test (CoGenT) (Johnson et al., 2017).
Revised: Test (CoGenT) (Johnson et al., 2017).

Original: The goal of CoGenT is to investigate a model’s ability to handle novel combinations of attributes that were not seen during training.
Revised: The goal of CoGenT is to investigate a model’s ability to handle novel combinations of attributes that were not seen during training.

Original: For this purpose, we established two conditions within a ShapeWorld4 CoGenT data set: Condition (A) – the training and test data set contains squares with the colors gray , blue , brown , or yellow , triangles with the colors red , green , magenta , or cyan and circles of all colors.
Revised: For this purpose, we established two conditions within a ShapeWorld4 CoGenT data set: Condition (A) – the training and test data set contains squares with the colors gray , blue , brown , or yellow , triangles with the colors red , green , magenta , or cyan and circles of all colors.

Original: Condition (B) – the training set is as in Condition (A).
Revised: Condition (B) – the training set is as in Condition (A).

Original: However, the test set contains squares with the colors red , green , magenta , or cyan , triangles with the colors gray , blue , brown , or yellow and circles of all colors.
Revised: However, the test set contains squares with the colors red , green , magenta , or cyan , triangles with the colors gray , blue , brown , or yellow and circles of all colors.

Original: The goal is to investigate how well a model can generalize that, e.g., also squares can have the color red, although never having seen evidence for this during training.
Revised: The goal is to investigate how well a model can generalize that, e.g., also squares can have the color red, although never having seen evidence for this during training.

Original: The resulting average precision test scores are presented in Fig.
Revised: The resulting average precision test scores are presented in Fig.

Original: 3a (bottom).
Revised: 3a (bottom).

Original: We observe that, even though the SLASH Program used for this experiment was not explicitly written to handle composition generalization, SLASH Attention shows greatly improved generalization capabilities.
Revised: We observe that, even though the SLASH Program used for this experiment was not explicitly written to handle composition generalization, SLASH Attention shows greatly improved generalization capabilities.

Original: This can be seen in the approx.
Revised: This can be seen in the approx.

Original: 13% higher average precision scores on the Condition (B) test set in comparison to the baseline model.
Revised: 13% higher average precision scores on the Condition (B) test set in comparison to the baseline model.

Original: Importantly, this trend still holds even when subtracting the higher precision scores observed in Condition (A).
Revised: Importantly, this trend still holds even when subtracting the higher precision scores observed in Condition (A).

Original: To summarize our ﬁndings from the experiments on set prediction: we observe that adding prior knowledge in the form of logical constraints via SLASH can greatly improve a neural module in terms of performance and generalizability.
Revised: To summarize our ﬁndings from the experiments on set prediction: we observe that adding prior knowledge in the form of logical constraints via SLASH can greatly improve a neural module in terms of performance and generalizability.

Original: On a side note: training neural networks for novel tasks, often involves deﬁning explicit loss functions, e.g. Hungarian loss for set prediction.
Revised: On a side note: training neural networks for novel tasks, often involves deﬁning explicit loss functions, e.g. Hungarian loss for set prediction.

Original: In contrast with SLASH, no matter the choice of NPP and underlying task, the training loss remains the same.
Revised: In contrast with SLASH, no matter the choice of NPP and underlying task, the training loss remains the same.

Original: Task-related requirements simply need to be added as lines of code to the SLASH program.
Revised: Task-related requirements simply need to be added as lines of code to the SLASH program.

Original: This additionally highlights SLASH’s versatility and ﬂexibility.
Revised: This additionally highlights SLASH’s versatility and ﬂexibility.

Original: Summary of all Empirical Results.
Revised: Summary of all Empirical Results.

Original: All empirical results together demonstrate that the ﬂexibility of
Revised: All empirical results together demonstrate that the ﬂexibility of

Original: SLASH is highly beneﬁcial and can easily outperform state-of-the-art: one can freely combine what is required to solve the underlying task — (deep) neural networks, PCs, and logic.
Revised: SLASH is highly beneﬁcial and can easily outperform state-of-the-art: one can freely combine what is required to solve the underlying task — (deep) neural networks, PCs, and logic.

Original: Particularly, the results indicate the potential of integrating PCs via SLASH into DPPLs.
Revised: Particularly, the results indicate the potential of integrating PCs via SLASH into DPPLs.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We introduce SLASH, a novel DPPL that integrates neural computations with tractable probability estimates and logical statements.
Revised: We introduce SLASH, a novel DPPL that integrates neural computations with tractable probability estimates and logical statements.

Original: The key ingredient of SLASH to achieve this are Neural-Probabilistic
Revised: The key ingredient of SLASH to achieve this are Neural-Probabilistic

Original: Predicates (NPPs) that can be ﬂexibly constructed out of neural and/or probabilistic circuit modules based on the data and underlying task.
Revised: Predicates (NPPs) that can be ﬂexibly constructed out of neural and/or probabilistic circuit modules based on the data and underlying task.

Original: With these NPPs, one can produce task-speciﬁc probability estimates.
Revised: With these NPPs, one can produce task-speciﬁc probability estimates.

Original: The details and additional prior knowledge of a task are neatly encompassed within a
Revised: The details and additional prior knowledge of a task are neatly encompassed within a

Original: SLASH program with only few lines of code.
Revised: SLASH program with only few lines of code.

Original: Finally, via Answer Set Programming and Weighted
Revised: Finally, via Answer Set Programming and Weighted

Original: Model Counting, the logical SLASH program and probability estimates from the NPPs are combined to estimate the truth value of a task-speciﬁc query.
Revised: Model Counting, the logical SLASH program and probability estimates from the NPPs are combined to estimate the truth value of a task-speciﬁc query.

Original: Our experiments show the power and efﬁciency of SLASH, improving upon previous DPPLs in the benchmark MNIST-Addition task in terms of performance, efﬁciency and robustness.
Revised: Our experiments show the power and efﬁciency of SLASH, improving upon previous DPPLs in the benchmark MNIST-Addition task in terms of performance, efﬁciency and robustness.

Original: Importantly, by integrating a SOTA slot attention encoder into NPPs and adding few logical constraints, SLASH demonstrates improved performances and generalizability in comparison to the pure slot encoder for the task of object-centric set prediction; a setting no DPPL has tackled yet.
Revised: Importantly, by integrating a SOTA slot attention encoder into NPPs and adding few logical constraints, SLASH demonstrates improved performances and generalizability in comparison to the pure slot encoder for the task of object-centric set prediction; a setting no DPPL has tackled yet.

Original: This shows the great potential of DPPLs to elegantly combine logical reasoning with neural computations and uncertainty estimates.
Revised: This shows the great potential of DPPLs to elegantly combine logical reasoning with neural computations and uncertainty estimates.

Original: Interesting avenues for future work include benchmarking SLASH on additional data types and tasks.
Revised: Interesting avenues for future work include benchmarking SLASH on additional data types and tasks.

Original: One should explore unsupervised and weakly supervised learning using logic with SLASH and investigate how far logical constraints can help unsupervised object discovery.
Revised: One should explore unsupervised and weakly supervised learning using logic with SLASH and investigate how far logical constraints can help unsupervised object discovery.

Original: In direct alignment with our work, one should also investigate image generation via the beneﬁcial feature of PCs to generate random samples.
Revised: In direct alignment with our work, one should also investigate image generation via the beneﬁcial feature of PCs to generate random samples.

Original: Actually, it should be possible to generate images that encapsulate logical knowledge bases.
Revised: Actually, it should be possible to generate images that encapsulate logical knowledge bases.

Original: This is important to move from data-rich to knowledge-rich AI.
Revised: This is important to move from data-rich to knowledge-rich AI.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: With our work we have shown that one can add prior knowledge and logical constraints to the training of learning systems.
Revised: With our work we have shown that one can add prior knowledge and logical constraints to the training of learning systems.

Original: We postulate that SLASH can therefore additionally be used to identify and remove biases or undesirable behavior, by adding constraints within the SLASH program.
Revised: We postulate that SLASH can therefore additionally be used to identify and remove biases or undesirable behavior, by adding constraints within the SLASH program.

Original: We observe that this feature, however, also has the potential danger to be used in the opposite way, e g. explicitly adding bias and discriminatory factors to a system.
Revised: We observe that this feature, however, also has the potential danger to be used in the opposite way, e g. explicitly adding bias and discriminatory factors to a system.

Original: To the best of our knowledge, our study does not raise any ethical, privacy or conﬂict of interest concerns.
Revised: To the best of our knowledge, our study does not raise any ethical, privacy or conﬂict of interest concerns.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: An ofﬁcial, curated GitHub repository will be made public with the ﬁnal version, containing the code of SLASH, as well as scripts to reproduce the experiments and generate data sets.
Revised: An ofﬁcial, curated GitHub repository will be made public with the ﬁnal version, containing the code of SLASH, as well as scripts to reproduce the experiments and generate data sets.

Original: In addition to this, architectural details and hyperparameters are included in the appendix.
Revised: In addition to this, architectural details and hyperparameters are included in the appendix.

Original: Preliminary code will be uploaded upon submission.
Revised: Preliminary code will be uploaded upon submission.

Original: Lastly, details on the evaluation metrics and relevant data sets are given in the main text as well as appendix.
Revised: Lastly, details on the evaluation metrics and relevant data sets are given in the main text as well as appendix.

Original: A A PPENDIX A – D ETAILS ON P ARAMETER L EARNING
Revised: A A PPENDIX A – D ETAILS ON P ARAMETER L EARNING

Original: In the Appendix, we want to discuss details on parameter learning in SLASH.
Revised: In the Appendix, we want to discuss details on parameter learning in SLASH.

Original: Since we use coordinate descent for training SLASH we present the derivative of each component of the loss function deﬁned in equation 10 since while optimization, one component has to be kept ﬁxed.
Revised: Since we use coordinate descent for training SLASH we present the derivative of each component of the loss function deﬁned in equation 10 since while optimization, one component has to be kept ﬁxed.

Original: We start with the gradient of the NPP loss function L NPP i.e. the negative log-likelihood, deﬁned in equation
Revised: We start with the gradient of the NPP loss function L NPP i.e. the negative log-likelihood, deﬁned in equation

Original: [Equation]
Revised: [Equation]

Original: Here, we remark that ∂x Qi ∂ξ will be carried out by back-propagation and the expression after it is the initial gradient.
Revised: Here, we remark that ∂x Qi ∂ξ will be carried out by back-propagation and the expression after it is the initial gradient.

Original: Next, we derive the gradient of the logical entailment loss function L ENT , as deﬁned in equation 7 .
Revised: Next, we derive the gradient of the logical entailment loss function L ENT , as deﬁned in equation 7 .

Original: One estimates the gradient as follows
Revised: One estimates the gradient as follows

Original: [Equation]
Revised: [Equation]

Original: whereby
Revised: whereby

Original: • X Q is the set of random variables associated with the set of the queries Q , • x Q i is a training sample, a realization of the set of random variables X Q associated with the particular query Q i , • P ( X Q ,C ) ( x Q i ) is the probability of the realization x Q i estimated by the NPP modelling the joint over the set X Q and C – the set of classes (the domain of the NPP), • log( P Π( θ ) ( Q i ))
Revised: • X Q is the set of random variables associated with the set of the queries Q , • x Q i is a training sample, a realization of the set of random variables X Q associated with the particular query Q i , • P ( X Q ,C ) ( x Q i ) is the probability of the realization x Q i estimated by the NPP modelling the joint over the set X Q and C – the set of classes (the domain of the NPP), • log( P Π( θ ) ( Q i ))

Original: – the probability of the query Q i under the program Π( θ ) calculated by SLASH (for the reference see the equation (5) ), • and ∂ log( P Π( θ ) ( Q i ))
Revised: – the probability of the query Q i under the program Π( θ ) calculated by SLASH (for the reference see the equation (5) ), • and ∂ log( P Π( θ ) ( Q i ))

Original: ∂p is the gradient as deﬁned in Eq.9.
Revised: ∂p is the gradient as deﬁned in Eq.9.

Original: We begin with the deﬁnition of the cross-entropy for two vectors y i and ˆ y i :
Revised: We begin with the deﬁnition of the cross-entropy for two vectors y i and ˆ y i :

Original: H ( y i , ˆ y i ) :
Revised: H ( y i , ˆ y i ) :

Original: = m (cid:88)
Revised: = m (cid:88)

Original: j = y ij · log
Revised: j = y ij · log

Original: [Equation]
Revised: [Equation]

Original: Hereafter we substitute y i =
Revised: Hereafter we substitute y i =

Original: log
Revised: log

Original: ( P Π( θ ) ( Q i )) and ˆ y i =
Revised: ( P Π( θ ) ( Q i )) and ˆ y i =

Original: P ( X Q ,C ) ( x Q i ) and obtain
Revised: P ( X Q ,C ) ( x Q i ) and obtain

Original: H ( y i , ˆ y i ) =
Revised: H ( y i , ˆ y i ) =

Original: H (cid:16) log( P Π( θ ) ( Q i )) , P ( X Q ,C ) ( x Q i ) (cid:17) =
Revised: H (cid:16) log( P Π( θ ) ( Q i )) , P ( X Q ,C ) ( x Q i ) (cid:17) =

Original: − m (cid:88)
Revised: − m (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ij
Revised: ij

Original: [Equation]
Revised: [Equation]

Original: (cid:16) P ( X Q
Revised: (cid:16) P ( X Q

Original: [Equation]
Revised: [Equation]

Original: C ) ( x Q ij ) (cid:17) .
Revised: C ) ( x Q ij ) (cid:17) .

Original: We remark that m represent the number of classes deﬁned in the domain of an NPP.
Revised: We remark that m represent the number of classes deﬁned in the domain of an NPP.

Original: Now, we differentiate the equation (11) with the respect to p depicted as in Eq.
Revised: Now, we differentiate the equation (11) with the respect to p depicted as in Eq.

Original: 9 to be the label of the probability of an atom c = v in r npp , denoting P Π( θ ) ( c = v ) .
Revised: 9 to be the label of the probability of an atom c = v in r npp , denoting P Π( θ ) ( c = v ) .

Original: Since differentiation is linear, the product rule is applicable directly:
Revised: Since differentiation is linear, the product rule is applicable directly:

Original: [Equation]
Revised: [Equation]

Original: We do not wish to consider the latter term of log( P Π
Revised: We do not wish to consider the latter term of log( P Π

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: because it represents the rescaling and to keep the ﬁrst since SLASH procure ∂ log( P Π( θ ) ( Q i ))
Revised: because it represents the rescaling and to keep the ﬁrst since SLASH procure ∂ log( P Π( θ ) ( Q i ))

Original: ∂p following Eq. 9 .
Revised: ∂p following Eq. 9 .

Original: To achieve this, we estimate equation from above downwards as
Revised: To achieve this, we estimate equation from above downwards as

Original: [Equation]
Revised: [Equation]

Original: Furthermore, let us recall that under i.i.d assumption we obtain from the deﬁnition of likelihood
Revised: Furthermore, let us recall that under i.i.d assumption we obtain from the deﬁnition of likelihood

Original: [Equation]
Revised: [Equation]

Original: and following the negative likelihood coupled with the knowledge that the log-likelihood of y i is the log of a particular entry of ˆ y
Revised: and following the negative likelihood coupled with the knowledge that the log-likelihood of y i is the log of a particular entry of ˆ y

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Finally, we obtain the following estimate applying inequality (12)
Revised: Finally, we obtain the following estimate applying inequality (12)

Original: [Equation]
Revised: [Equation]

Original: Also, we note that the mathematical transformations listed above hold for any type of NPP and the task dependent queries (NN with Softmax, PC or PC jointly with NN).
Revised: Also, we note that the mathematical transformations listed above hold for any type of NPP and the task dependent queries (NN with Softmax, PC or PC jointly with NN).

Original: The only difference will be the second term, i.e., log( P ( C | X Q ) ( x Q ij )) or log( P ( X Q | C ) ( x Q ij )) depending on the NPP and task.
Revised: The only difference will be the second term, i.e., log( P ( C | X Q ) ( x Q ij )) or log( P ( X Q | C ) ( x Q ij )) depending on the NPP and task.

Original: The NPP in a form of a single PC modeling the joint over X Q and C was depicted to be the example.
Revised: The NPP in a form of a single PC modeling the joint over X Q and C was depicted to be the example.

Original: With that, the derivation of gradients for both loss functions 2 and 7 is complete, and the training is carried out by coordinated descent.
Revised: With that, the derivation of gradients for both loss functions 2 and 7 is complete, and the training is carried out by coordinated descent.

Original: Backpropagation for joint NN and PC NPPs: If within the SLASH program, Π( θ ) , the NPP forwards the data tensor through a NN ﬁrst, i.e., the NPP models a joint over the NN’s output variables by a PC, then we rewrite (8) to
Revised: Backpropagation for joint NN and PC NPPs: If within the SLASH program, Π( θ ) , the NPP forwards the data tensor through a NN ﬁrst, i.e., the NPP models a joint over the NN’s output variables by a PC, then we rewrite (8) to

Original: [Equation]
Revised: [Equation]

Original: Thereby, γ is the set of the NN’s parameters and ∂ θ ∂ γ is computed by the backward propagation through the NN.
Revised: Thereby, γ is the set of the NN’s parameters and ∂ θ ∂ γ is computed by the backward propagation through the NN.

Original: B A PPENDIX B – SLASH P ROGRAMS
Revised: B A PPENDIX B – SLASH P ROGRAMS

Original: Here, the interested reader will ﬁnd the SLASH programs which we compiled for our experiments.
Revised: Here, the interested reader will ﬁnd the SLASH programs which we compiled for our experiments.

Original: Figure 4 presents the one for the MNIST Addition task, Figure 6 – for the set prediction task with slot attention encoder and the subsequent CoGenT test.
Revised: Figure 4 presents the one for the MNIST Addition task, Figure 6 – for the set prediction task with slot attention encoder and the subsequent CoGenT test.

Original: Note the use of the “+” and “-” notation for indicating whether a random variable is given or being queried for.
Revised: Note the use of the “+” and “-” notation for indicating whether a random variable is given or being queried for.

Original: 1 # Define images 2 img(i1).
Revised: 1 # Define images 2 img(i1).

Original: img(i2).
Revised: img(i2).

Original: 3 # Define Neural-Probabilistic Predicate 4 npp(digit(X), [0,1,2,3,4,5,6,7,8,9]) :- img(X).
Revised: 3 # Define Neural-Probabilistic Predicate 4 npp(digit(X), [0,1,2,3,4,5,6,7,8,9]) :- img(X).

Original: 5 # Define the addition of digits given two images and the resulting sum
Revised: 5 # Define the addition of digits given two images and the resulting sum

Original: [Figure]
Revised: [Figure]

Original: 1 # Is 7 the sum of the digits in img1 and img2?
Revised: 1 # Is 7 the sum of the digits in img1 and img2?

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: slot(s1).
Revised: slot(s1).

Original: slot(s2).
Revised: slot(s2).

Original: slot(s3).
Revised: slot(s3).

Original: slot(s4).
Revised: slot(s4).

Original: 3 # Define identifiers for the objects in the image 4 # (there are up to four objects in one image).
Revised: 3 # Define identifiers for the objects in the image 4 # (there are up to four objects in one image).

Original: 5 obj(o1).
Revised: 5 obj(o1).

Original: obj(o2).
Revised: obj(o2).

Original: obj(o3).
Revised: obj(o3).

Original: obj(o4).
Revised: obj(o4).

Original: 6 # Assign each slot to an object identifier 7 {assign_one_slot_to_one_object(X, O): slot(X)}=1 :- obj(O).
Revised: 6 # Assign each slot to an object identifier 7 {assign_one_slot_to_one_object(X, O): slot(X)}=1 :- obj(O).

Original: 8 # Make sure the matching is one-to-one between slots 9 # and objects identifiers.
Revised: 8 # Make sure the matching is one-to-one between slots 9 # and objects identifiers.

Original: 10 :- assign_one_slot_to_one_object(X1, O1), assign_one_
Revised: 10 :- assign_one_slot_to_one_object(X1, O1), assign_one_

Original: _to_one
Revised: _to_one

Original: X2,
Revised: X2,

Original: X1==X2, O1!=O2.
Revised: X1==X2, O1!=O2.

Original: 13 # Define all Neural-Probabilistic Predicates 14 npp color_attr
Revised: 13 # Define all Neural-Probabilistic Predicates 14 npp color_attr

Original: X) red, blue, green, grey, brown, magenta, cyan, yellow, bg]) : slot
Revised: X) red, blue, green, grey, brown, magenta, cyan, yellow, bg]) : slot

Original: X).
Revised: X).

Original: 16 npp attr
Revised: 16 npp attr

Original: X triangle bg]) :-
Revised: X triangle bg]) :-

Original: X
Revised: X

Original: 17 npp shade_attr(X
Revised: 17 npp shade_attr(X

Original: , [bright, dark, bg]) :
Revised: , [bright, dark, bg]) :

Original: 18 npp bg]
Revised: 18 npp bg]

Original: 20 has_attributes
Revised: 20 has_attributes

Original: , assign_one
Revised: , assign_one

Original: _to_one
Revised: _to_one

Original: , color
Revised: , color

Original: +X, -C
Revised: +X, -C

Original: , shape
Revised: , shape

Original: +X
Revised: +X

Original: S
Revised: S

Original: , shade(+X, -H), size(+X, -Z
Revised: , shade(+X, -H), size(+X, -Z

Original: [Figure]
Revised: [Figure]

Original: Does object o1 have the attributes red, circle, bright, small?
Revised: Does object o1 have the attributes red, circle, bright, small?

Original: [Figure]
Revised: [Figure]

Original: has_attributes(o1,
Revised: has_attributes(o1,

Original: [Figure]
Revised: [Figure]

Original: , circle, bright, small)
Revised: , circle, bright, small)

Original: C A PPENDIX C – E XPERIMENTAL D ETAILS C. S HAPE W ORLD 4 G ENERATION
Revised: C A PPENDIX C – E XPERIMENTAL D ETAILS C. S HAPE W ORLD 4 G ENERATION

Original: The ShapeWorld4 and ShapeWorld4 CoGenT data sets were generated using the original scripts of (Kuhnle & Copestake, 2017) ( https://github.com/AlexKuhnle/ShapeWorld ).
Revised: The ShapeWorld4 and ShapeWorld4 CoGenT data sets were generated using the original scripts of (Kuhnle & Copestake, 2017) ( https://github.com/AlexKuhnle/ShapeWorld ).

Original: The exact scripts will be added together with the SLASH source code.
Revised: The exact scripts will be added together with the SLASH source code.

Original: C. A VERAGE P RECISION COMPUTATION (S HAPE W ORLD 4)
Revised: C. A VERAGE P RECISION COMPUTATION (S HAPE W ORLD 4)

Original: For the baseline slot encoder experiments on ShapeWorld4 we measured the average precision score as in Locatello et al.
Revised: For the baseline slot encoder experiments on ShapeWorld4 we measured the average precision score as in Locatello et al.

Original: In comparison to the baseline slot encoder, when applying SLASH
Revised: In comparison to the baseline slot encoder, when applying SLASH

Original: Attention, however, we handled the case of a slot not containing an object, e.g. only background variables, differently.
Revised: Attention, however, we handled the case of a slot not containing an object, e.g. only background variables, differently.

Original: Whereas Locatello et al.
Revised: Whereas Locatello et al.

Original: (2020) add an additional binary identiﬁer to the multi-label ground truth vectors, we have added a background (bg) attribute to each category ( cf.
Revised: (2020) add an additional binary identiﬁer to the multi-label ground truth vectors, we have added a background (bg) attribute to each category ( cf.

Original: Fig. 6).
Revised: Fig. 6).

Original: A slot is thus considered to be empty (i.e. not containing an object) if each NPP returns the maximal conditional probability for the bg attribute.
Revised: A slot is thus considered to be empty (i.e. not containing an object) if each NPP returns the maximal conditional probability for the bg attribute.

Original: As the ShapeWorld4 prediction task only included discrete object properties both for Slot Attention as well as for SLASH Attention the distance threshold for the average precision computation was inﬁnity (thus corresponding to no threshold).
Revised: As the ShapeWorld4 prediction task only included discrete object properties both for Slot Attention as well as for SLASH Attention the distance threshold for the average precision computation was inﬁnity (thus corresponding to no threshold).

Original: C. M ODEL D ETAILS
Revised: C. M ODEL D ETAILS

Original: For those experiments using NPPs with PC we have used Einsum Networks (EiNets) for implementing the probabilistic circuits.
Revised: For those experiments using NPPs with PC we have used Einsum Networks (EiNets) for implementing the probabilistic circuits.

Original: EiNets are a novel implementation design for SPNs introduced by Peharz et al.
Revised: EiNets are a novel implementation design for SPNs introduced by Peharz et al.

Original: (2020) that minimize the issue of computational costs that initial SPNs had suffered.
Revised: (2020) that minimize the issue of computational costs that initial SPNs had suffered.

Original: This is accomplished by combining several arithmetic operations via a single monolithic einsum-operation.
Revised: This is accomplished by combining several arithmetic operations via a single monolithic einsum-operation.

Original: For all experiments, the ADAM optimizer (Kingma & Ba, 2015) with β 1 = 0 .
Revised: For all experiments, the ADAM optimizer (Kingma & Ba, 2015) with β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 , (cid:15) = 1 e − 8 and no weight decay was used.
Revised: 999 , (cid:15) = 1 e − 8 and no weight decay was used.

Original: MNIST-Addition Experiments For the MNIST-Addition experiments
Revised: MNIST-Addition Experiments For the MNIST-Addition experiments

Original: , we ran the DeepProbLog and NeurASP programs with their original conﬁgurations, as stated in (Manhaeve et al., 2018) and (Yang et al., 2020), respectively.
Revised: , we ran the DeepProbLog and NeurASP programs with their original conﬁgurations, as stated in (Manhaeve et al., 2018) and (Yang et al., 2020), respectively.

Original: For the SLASH MNIST-Addition experiments, we have used the same neural module as in DeepProbLog and NeurASP, when training SLASH with the neural NPP (SLASH (DNN)) represented in Tab. 2.
Revised: For the SLASH MNIST-Addition experiments, we have used the same neural module as in DeepProbLog and NeurASP, when training SLASH with the neural NPP (SLASH (DNN)) represented in Tab. 2.

Original: When using a PC NPP (SLASH (PC))
Revised: When using a PC NPP (SLASH (PC))

Original: we have used an EiNet with the Poon-Domingos (PD) structure (Poon & Domingos, 2011) and normal distribution for the leafs.
Revised: we have used an EiNet with the Poon-Domingos (PD) structure (Poon & Domingos, 2011) and normal distribution for the leafs.

Original: The formal hyperparameters for the EiNet are depicted in Tab. 3.
Revised: The formal hyperparameters for the EiNet are depicted in Tab. 3.

Original: The learning rate and batch size for the DNN were 0.005 and 100, for DeepProbLog, NeurASP and
Revised: The learning rate and batch size for the DNN were 0.005 and 100, for DeepProbLog, NeurASP and

Original: SLASH (DNN).
Revised: SLASH (DNN).

Original: For the EiNet, these were 0.01 and 100.
Revised: For the EiNet, these were 0.01 and 100.

Original: [Table]
Revised: [Table]

Original: ShapeWorld4 Experiments For the baseline slot attention experiments with the ShapeWorld4 data set we have used the architecture presented in Tab. 4.
Revised: ShapeWorld4 Experiments For the baseline slot attention experiments with the ShapeWorld4 data set we have used the architecture presented in Tab. 4.

Original: For further details on this, we refer to the original work of Locatello et al.
Revised: For further details on this, we refer to the original work of Locatello et al.

Original: The slot encoder had a number of 4 slots and 3 attention iterations over all experiments.
Revised: The slot encoder had a number of 4 slots and 3 attention iterations over all experiments.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: For the SLASH Attention experiments with ShapeWorld4 we have used the same slot encoder as in
Revised: For the SLASH Attention experiments with ShapeWorld4 we have used the same slot encoder as in

Original: Tab.
Revised: Tab.

Original: [Table]
Revised: [Table]

Original: however
Revised: however

Original: [Table]
Revised: [Table]

Original: we replaced the ﬁnal MLPs with 4 individual EiNets with Poon-Domingos structure (Poon & Domingos, 2011)
Revised: we replaced the ﬁnal MLPs with 4 individual EiNets with Poon-Domingos structure (Poon & Domingos, 2011)

Original: [Table]
Revised: [Table]

Original: Their hyperparameters are represented in Tab. 5.
Revised: Their hyperparameters are represented in Tab. 5.

Original: [Table]
Revised: [Table]

Original: The learning rate and batch size for SLASH Attention were 0.01 and 512, for ShapeWorld4 and
Revised: The learning rate and batch size for SLASH Attention were 0.01 and 512, for ShapeWorld4 and

Original: ShapeWorld4 CoGenT.
Revised: ShapeWorld4 CoGenT.

Original: The learning rate for the baseline slot encoder were 0.0004 and 512.
Revised: The learning rate for the baseline slot encoder were 0.0004 and 512.

Original: D A PPENDIX D - A DDITIONAL R ESULTS ON MNIST A DDITION
Revised: D A PPENDIX D - A DDITIONAL R ESULTS ON MNIST A DDITION

Original: [Table]
Revised: [Table]

Original: In addition to the setting considered in Evaluation 2, we adjusted the settings for the MNIST Addition task with missing data into the following way.
Revised: In addition to the setting considered in Evaluation 2, we adjusted the settings for the MNIST Addition task with missing data into the following way.

Original: Training was performed with the full MNIST data set, however the test data set contained different rates of missing pixels.
Revised: Training was performed with the full MNIST data set, however the test data set contained different rates of missing pixels.

Original: Whereas using an NPP with a PC allows, among other things, to compute marginalization “out of the box” without requiring an update to the architecture or a retraining, this is not so trivial for purely neural-based predicates as in DeepProbLog.
Revised: Whereas using an NPP with a PC allows, among other things, to compute marginalization “out of the box” without requiring an update to the architecture or a retraining, this is not so trivial for purely neural-based predicates as in DeepProbLog.

Original: Thus, we allowed SLASH (PC) to marginalize over the missing pixels, where this was not directly possible for DeepProbLog.
Revised: Thus, we allowed SLASH (PC) to marginalize over the missing pixels, where this was not directly possible for DeepProbLog.

Original: The results can be seen in Tab. 6 for  80% , 90% and 97% of missing pixels per image in the test set.
Revised: The results can be seen in Tab. 6 for 50% , 80% , 90% and 97% of missing pixels per image in the test set.

Original:  For all  rates, we observe that SLASH (PC) reaches signiﬁcantly higher test accuracies than DeepProbLog.
Revised: We observe that at 50% , DeepProbLog outperforms SLASH by a small margin. For all other rates, we observe that SLASH (PC) reaches signiﬁcantly higher test accuracies than DeepProbLog.

Original: However, we remark that in this setting, SLASH produces larger standard deviations in comparison to DeepProbLog.
Revised: However, we remark that in this setting, SLASH produces larger standard deviations in comparison to DeepProbLog.

Original: These results indicate that the conclusions, drawn in the main part of our work, remain true also in this setting of handling missing data.
Revised: These results indicate that the conclusions, drawn in the main part of our work, remain true also in this setting of handling missing data.