Paper: LogicMP: A Neuro-symbolic Approach for Encoding First-order Logic Constraints (ICLR-2024)

Reviewer comment (Theory):
Although LogicMP focuses on encoding FOLs into neural networks, it cannot handle existential quantifiers, which significantly limits its applicability.

Author response:
## Q2: The Incapability of Handling Existential Quantifier
The inability to use existential quantifiers stems from MLN limitations, which typically consider only universal quantifiers. This issue is common across MLN-based methods. We are exploring attention-based methods to model Skolem forms to address existential quantifiers.

Reviewer comment (Evaluation):
The evaluation appears somewhat limited. Firstly, it only compares 3 neuro-symbolic baselines (SL, its variant SLrelax, SPL). These methods compile the constraint into a probabilistic circuit and then perform exact inference. Given that LogicMP performs approximate inference, comparing it with methods using exact inference seems somewhat unfair.

Author response:
## Q1: Compared Methods in the Evaluation
1. Among the propositional logic methods, we compare against SL and SPL in Sec. 5.1, which use ACs to compute WMC in parallel. To the best of our knowledge, they are more scalable than other methods without ACs and are the most scalable propositional methods, even though they fail to complete the task.
2. Among the first-order logic methods, we compare against other MLN-based methods in Sec. 5.2, including lifted MLN, HL-MRF, and ExpressGNN. They have demonstrated strong performance and scalability.

Our experiments cover three general tasks over image, graph, and text, demonstrating LogicMP's effectiveness and versatility.

Reviewer comment (Theory):
What is the expressiveness of LogicMP? Can it encode any quantifier-free first-order logic formula? Additionally, the notation in the first paragraph of Section 2 is somewhat confusing. For instance, is a specific structure required for \( f \)? In other words, should \( f \) be represented in the form of a disjunction of logical atoms? Moreover, is the logical atom \(\mathtt{C}(e_1, e_2)\) a general form to represent any relation between \( e_1 \) and \( e_2 \)?

Author response:
## Q3: The Expressiveness of LogicMP
In LogicMP, $f$ can be any FOL in conjunctive normal form (CNF) with universal quantifiers (the disjunction of literals, i.e., clauses, are the items in CNF). $\mathtt{C}(e_1,e_2)$ denotes the "coexist" predicate for entities $e_1$ and $e_2$ in our CV task (Sec. 5.1). Besides this, LogicMP supports predicates with arbitrary numbers of arguments. Table 11 demonstrates several FOL for the task in Sec. 5.2, which also contains unary predicates and ternary predicates, e.g., "taughtBy(c, p, q)" (p is taught by q in the course c).

Reviewer comment (Theory):
Why not directly solve problem Eq.1? In problem Eq.1, we can perform the weighted counting in a parallel manner, rather than using sequential generation as required when solving problems Eq.2 and Eq.3. Moreover, quite a few techniques in fuzzy logic can efficiently handle the discrete potential function \(\phi_f(\cdot)\), such as translating the disjunction into the product or the minimum.

Author response:
## Q4: Eq.1 is Intractable even with WMC
As "Contribution of Our Paper" mentions, directly solving Eq.1 is theoretically intractable. Specifically, for any FOL that is not liftable, performing FOL reasoning is #P-complete. For such problems, WMC methods, such as SDD, generate a tree with an extremely large treewidth, making computation infeasible. An example is given in the App. J.3, where SDD is used to compile the transitivity rule with various entities. When the number of entities exceeds 8, the compilation becomes invalid.

## Q5: Using Fuzzy Logic is Insufficient
Using fuzzy logic mitigates the inference difficulty. However, previous propositional methods that fail to leverage the FOL structural symmetries may face difficulties dealing with massive groundings, e.g., 134M+ groundings in Sec. 5.1 and 600B+ groundings in Sec. 5.2. DL2 [1] proposes a method to implement individual propositional logic rules via neural networks, but how to handle such many groundings remains challenging. LogicMP does not require instantiating the groundings; all are automatically aggregated in Einsum operations that leverage the FOL structural symmetries, where the basic computation unit is the grouped ground atoms of each predicate.

Reviewer comment (Novelty):
The paper claims to be the first method capable of encoding FOLC (pag. 2, “Contributions”). This is not true. The authors themselves cite ExpressGNN. However, there are many other papers attempting this, including: Deep Logic Models, Marra et al, ECML 2019; Relational Neural Machines, Marra et al, ECAI 2020; NeuPSL: Neural Probabilistic Soft Logic, Pryor et al, 2023; DeepPSL: End-to-End Perception and Reasoning, Dasaratha et al, IJCAI 2023; Backpropagating Through MLNs, Betz et al, IJCLR 2021. Many of these systems have CV and citation networks experiments.

Author response:
Please see "Contribution of Our Paper". We acknowledge these important methods in the related work section in the new version. We rephrase the sentence to be more appropriate -- "first fully differentiable neuro-symbolic approach capable of encoding FOLCs for arbitrary neural networks."

Reviewer comment (Writing):
The paper mentions FOLC but it never defines them. All the examples, though, are definite clauses. Are non-definite clauses supported? If yes, are you employing them in your experiments?

Author response:
The FOLC was defined in the first line of the second paragraph of the "Introduction" section with an example describing it. 

LogicMP supports conjunctive normal form (CNF) with universal quantifiers, including definite and non-definite clauses. In the collective classification experiment, we used several non-definite clauses shown in Table 11, such as "¬taughtBy(c, p, q) ∨ ¬courseLevel(c, Level_500) ∨ ¬ta(c, s, q) ∨ advisedBy(s, p) ∨ tempAdvisedBy(s, p)".

Reviewer comment (Novelty):
In terms of significance, there has been a long history of work in lifted inference with the same underlying principle of using symmetries to scale-up inference in MLNs. One of the key takeaways from such work (e.g. Broeck & Darwiche 2013) is that evidence can destroy symmetries in which case lifted inference reduces to ground inference (if guarantees on the inference results are required). Here, while the approach is scalable, would the same problem be encountered.

Author response:
Yes, we are very excited about this property of LogicMP: it can perform efficient MLN inference **with distinctive evidence**. Note that our formalization in Eq. 1 contains the distinctive evidence, i.e., the first "neural semantics" item ($\sum_i \phi_u(v_i)$). Performing inference over Eq. 1 means performing MLN inference with distinctive evidence. In our first FUNSD experiment, there are 262K ground atoms with distinctive evidence (distinctive neural predictions). Using a modern GPU, LogicMP can perform MLN inference with 262K variables and 134M groundings in just 0.03 seconds.

Let's explain how it is achieved. Since the exact inference for general FOL reasoning (WFOMC) is proved to be #P-complete, our LogicMP achieves efficient inference at the cost of variational approximation. LogicMP breaks down the dependency among the ground atoms by variational inference and uses the mean-field algorithm to iteratively approach the variational approximation of MLN distribution. Even though the mean-field update is theoretically possible, vanilla sequential implementation is impractical -- instantiating and computing 134M groundings is not feasible in practice. The efficiency of LogicMP is essentially achieved by leveraging the structural symmetries in the groundings of FOL and formalizing the message aggregation into Einsum tensor operations, which is principally different from previous lifted methods for WFOMC.

Reviewer comment (Writing):
In the related work section, it is mentioned that for earlier methods, “The latter consists of symmetric lifted algorithms which become inefficient with distinctive evidence”. Does this mean that LogicMP does not have this issue?

Author response:
LogicMP is designed to cope with the distinctive evidence from the neural networks, so it will **not** affect the computational efficiency of Einsum. LogicMP first computes a marginal distribution $\mathbf{Q}$ for each ground atom using the distinctive evidence (when no evidence, i.e., no neural prediction is provided, zero is used as the evidence). Then, the Einsum operation takes the marginal probability $\mathbf{Q}$ and calculates the grounding messages at the level of FOL in parallel. In this way, the computation remains the same with or without distinctive evidence. The distinctive evidence will destroy the symmetries in the lifted inference but not the structural symmetries among the FOL groundings. Instead of relying on the identity among the groundings, LogicMP leverages the structural symmetries among the FOL groundings, which persists when distinctive evidence is given.

Reviewer comment (Theory):
How does evidence affect Einsum computations? Does it break symmetries making it harder to parallelize?

Author response:
LogicMP is designed to cope with the distinctive evidence from the neural networks, so it will **not** affect the computational efficiency of Einsum. LogicMP first computes a marginal distribution $\mathbf{Q}$ for each ground atom using the distinctive evidence (when no evidence, i.e., no neural prediction is provided, zero is used as the evidence). Then, the Einsum operation takes the marginal probability $\mathbf{Q}$ and calculates the grounding messages at the level of FOL in parallel. In this way, the computation remains the same with or without distinctive evidence. The distinctive evidence will destroy the symmetries in the lifted inference but not the structural symmetries among the FOL groundings. Instead of relying on the identity among the groundings, LogicMP leverages the structural symmetries among the FOL groundings, which persists when distinctive evidence is given.

Reviewer comment (Novelty):
There has been studies in databases regarding width of a FOL (Vardi) (e.g. in a chain formula, the width is small). This has also been used to scale-up inference using CSPs (Venugopal et al. AAAI 2015, Sarkhel et al. IJCAI 2016). Would this be related to Einsum optimization?

Author response:
The relationship between the CSP and Einsum formalization was discussed in the App. H. Concretely, CSP is proposed for grounding counting (GC), while Einsum is for grounding message aggregation (GMA). GC is similar to GMA as both can be formalized as the summation of the product. CSP optimizes GC by reducing the problem into sub-problems, and it is related to Einsum optimization -- both Einsum optimization and CSP are equivalent to the classical ML problem of variable elimination in the message passing.

Even though CSP has several differences from our approach:

1. Unlike GC, the Einsum handles GMA derived from the mean-field algorithm.
2. More importantly, we propose Einsum as a **parallel tensor operation** that brings computational efficiency and enables LogicMP to be an effective neural module, which is not emphasized in CSP. Using Einsum, MLN inference can be expressed as an efficient parallel NN.

Reviewer comment (Experiments):
In the experiments were the weights for the MLN formulas encoded by LogicMP learned (it is mentioned in one case that the weights was set to 1). How do these weights impact performance?

Author response:
The weights are learned via back-propagation in the supervised tasks (CV FUNSD and NLP sequence labeling tasks). We found that the weights were not sensitive to the initialization during the experiments. The learned weight converges to ~0.1 for various initializations in the FUNSD task. This is reasonable as the balance between the FOLC and neural network is automatically achieved during the training. For the collective classification task in Sec. 5.2, we adopt a fashion of semi-supervised learning where the rule is used to enhance the neural networks and set the weight to 1. In this setup, the amount of weight is a coefficient of the learning rate in the training. In our experiments, we found it robust.

Reviewer comment (Writing):
While the authors included an overview of Markic Logic Networks there are still some concepts that look a bit obscure. What does the weight associated with each formula represent? Is it a way of representing the importance assigned to the formula? Why do the authors need the open-world assumption? When explaining the MLNs, can you please add an example with: (i) an ML problem, (ii) what would the FOLC be in the problem, and (iii) what would be the observed and unobserved variables in the problem.

Author response:
Markov Logic Networks merge classical Markov networks with a potential function quantifying how well a state adheres to given rules. The key is in weighting these rules: each rule is assigned a weight that signifies its importance. The higher the weight, the more significant its impact on the potential function, suggesting greater importance. In extreme cases, a high weight effectively turns a rule into a hard constraint.

In the general tasks where logic is integrated, making assumptions about the states of unobserved ground atoms is problematic. Since each atom could be either True or False, relying on the closed-world assumption is impractical. For example, in our first case study (Sec. 5.1), all instances of $\mathtt{C}(e_i,e_j)$ are unobserved. However, each instance has a non-negligible probability of being True, rendering the closed-world assumption unsuitable.

Task Overview: Consider an image document containing both image pixels and text tokens (e.g., an image with the text "Date: February 25, 1998 ... CLIENT TO: L8557.002 ..."). The objective here is to segment these tokens into blocks (such as grouping "February 25, 1998" into a single block).

   - ML Problem Formalization: We are presented with a dataset $\mathbf{D}$ comprising various samples. Each sample includes a tuple $(\mathbf{e}, \mathbf{m}, \mathbf{V})$, where $\mathbf{m}$ denotes the input image, $\mathbf{e} = \{e_1, ..., e_N\}$ represents $N$ input token IDs, and $\mathbf{V} = \{\mathtt{C}(e_i, e_j)\}_{i, j}$ is an $N\times N$ matrix indicating whether tokens $e_i$ and $e_j$ coexist in the same block. The goal is to develop a function $f: (\mathbf{e}, \mathbf{m}) \rightarrow \mathbf{V}$ that maps this input to a label space $\mathbf{V}$. In our experiments, this function is implemented by LayoutLM+LogicMP and trained on $\mathbf{D}$ to optimize the likelihood.
   - Observations: These include the input elements, namely the image $\mathbf{m}$ and the token IDs $\mathbf{e}$.
   - Latent Variables: These are the binary variables we aim to predict, represented as $\mathbf{V}=[v_{\mathtt{C}(1, 1)}, ...., v_{\mathtt{C}(1, N)}, ..., v_{\mathtt{C}(N, 1)}, ..., v_{\mathtt{C}(N, N)}]$.
   - FOLC of 'Transitivity': If tokens $i$ and $j$ are in the same block and tokens $j$ and $k$ are also together, then tokens $i$ and $k$ should be in the same block. This is formally stated as "$\forall i, j, k; \mathtt{C}(i, j) \wedge \mathtt{C}(j, k) \rightarrow \mathtt{C}(i, k)$" where $\mathtt{C}$ denotes the $coexist$ predicate.

Reviewer comment (Writing):
What does it mean that \( Z_i \) is the partition function? Over what?

Author response:
The Partition Function $Z_i$

The partition function $Z_i$ is the summation of potentials for two states of $v_i$: $v_i=0$ and $v_i=1$. Specifically, it's calculated as 
$Z_i = \exp(\phi_u(v_i=0) + \sum_{f\in F} w_f \sum_{g\in G_f(i)} \hat{Q}\_{i, g}(v_i=0)) + \exp(\phi_u(v_i=1) + \sum_{f\in F} w_f \sum_{g\in G_f(i)} \hat{Q}_{i, g}(v_i=1))$.
This approach, in contrast to Eq.1 sums over $2^{|V|}$ items, simplifies the computation by only considering two scenarios, greatly easing the computational burden.

Reviewer comment (Presentation):
I am not sure how to read Table 1. Same applies for Figure 3.

Author response:
Interpreting Table 1 and Figure 3

Table 1 demonstrates the influence of the grounding $\neg \mathtt{C}(e_1,e_2) \vee \neg \mathtt{C}(e_2,e_3) \vee \mathtt{C}(e_1,e_3)$ on $\mathtt{C}(e_2, e_3)$. It corresponds to the implication: $\mathtt{C}(e_1,e_2) \wedge \mathtt{C}(e_2,e_3) \implies \mathtt{C}(e_1,e_3)$. The table shows that when the premise $\mathtt{C}(e_1,e_2) \wedge \mathtt{C}(e_2,e_3)$ is False, i.e., $(\mathtt{C}(e_1,e_2), \mathtt{C}(e_2,e_3)) = (0, 0), (0, 1)$ or $(1, 0)$, the rule holds regardless of the hypothesis ($\mathtt{C}(e_2,e_3)$) being True or False, implying that in such states, the grounding does not influence the hypothesis, allowing us to skip certain computations.

Figure 3 visualizes the parallel message passing for the transitivity rule. It comprises three parallel operations based on different implication statements:
   1. The first row in the update (grey area) corresponds to $\forall a, b, c: \mathtt{C}(a, b) \wedge \mathtt{C}(b,c) \implies \mathtt{C}(a, c)$, calculated using $\mathtt{einsum}(\text{``}ab,bc\to ac\text{''}, {\mathbf{Q}}\_{\mathtt{C}}(\mathbf{1}), {\mathbf{Q}}_{\mathtt{C}}(\mathbf{1}))$.
   2. The second row corresponds to $\forall a, b, c: \mathtt{C}(a, b) \wedge \neg \mathtt{C}(a,c) \implies \neg \mathtt{C}(a, c)$, calculated using $\mathtt{einsum}(\text{``}ab,ac\to bc\text{''}, {\mathbf{Q}}\_{\mathtt{C}}(\mathbf{1}),{\mathbf{Q}}_{\mathtt{C}}(\mathbf{0}))$.
   3. The third row corresponds to $\forall a, b, c: \mathtt{C}(b, c) \wedge \neg \mathtt{C}(a,c) \implies \neg \mathtt{C}(a, b)$, calculated using $\mathtt{einsum}(\text{``}bc,ac\to ac\text{''}, {\mathbf{Q}}\_{\mathtt{C}}(\mathbf{1}),{\mathbf{Q}}_{\mathtt{C}}(\mathbf{0}))$.

The red frame highlights the relation between a grounding message's calculation in the sequential process and the parallel computation.

Reviewer comment (Theory):
How is it possible that the complexity does not depend on the number of formulas \( |F| \)?

Author response:
Complexity Notation

The most intricate rule typically dominates complexity in our LogicMP. Therefore, we represent overall complexity w.r.t. the most complex rule, which provides a practical measure of the system's computational demands. The complexity improvement remains the same when including $|F|$.