Paper: A neuro-symbolic framework for answering conjunctive queries (ICLR-2024)

Reviewer comment (Novelty):
The claimed contribution is rather limited. Specifically, Contribution (C1) is not directly related to the field of Machine Learning; the results it claims are strictly within the domain of database theory. Unravelling techniques are conventional in database theory and knowledge representation. Furthermore, the observation that the tree unraveling (to any depth) of a non-tree-shaped conjunctive query results in an over-approximation of the original query might be considered common knowledge within the community. As a researcher with a main background in database theory, logic, and knowledge representation, the primary findings presented in Section 4.1 appear to be straightforward observations that might not carry sufficient significance for publication.

Author response:
Please note that we are not claiming the notion of tree-like approximation itself as a contribution, and we mention it has been studied before in the second paragraph of Section 2. We agree with the reviewer that unravelings are common in the literature and that the completeness claim is straightforward. The optimality claim is less straightforward, but it is not difficult. We are not aware of any optimality result of this kind (to the best of our knowledge, similar previous results deal with treewidth-1 queries, as mentioned in Section 2). If the reviewer has a concrete reference showing a similar result, we are happy to add it to the paper. Also, we are happy to improve the wording so it is clear to the reader that we are not claiming the notion of tree-like approximation as a contribution. 

The contribution, as explained in the paper, is the use of tree-like approximations of conjunctive queries in the context of Machine Learning, to obtain neuro-symbolic models that can handle cyclic conjunctive queries. We stress that handling cyclic queries has been identified as an open problem in the literature (Ren et al 2023).

Reviewer comment (Novelty):
The significance of Contribution (C1) in the context of the current machine learning-based query answering state-of-the-art is not evident. Existing approaches are unable to handle arbitrary monadic tree-shaped conjunctive queries (CQs) without the requirement of query anchoring, and consequently most of these approaches cannot directly leverage the proposed approximation.

Author response:
We are not claiming (C1) as a contribution, see our previous answer. As far as we can see, it is not clear how to define a notion of tree-like approximations that can be handled directly by current methods and that enjoys strong theoretical guarantees (e.g. completeness, optimality). On the other hand, we believe that the fact that current methods need to be adapted to use tree-like approximations, does not make our contribution less significant.

Reviewer comment (Theory):
Contribution (C2), in which the authors introduce an expansion of Zhu et al.'s method to encompass unanchored queries, is not sufficiently elaborated. It is unclear in what manner it enables the technique by Zhu et al. to “support” arbitrary tree-shaped CQs. Additionally, it is uncertain whether this extension can be applied to other methods, especially those dependent on the existence of embeddings for the anchors; this does not seem straightforward.

Author response:
We describe this in the 3rd paragraph of Section 4.3. The idea is that GNN-QE (Zhu et al) is a method that allows one to encompass existential quantification in the latent space as a full unitary vector (in other more complex methods one would probably have to devise a more clever encoding scheme to allow for unanchored queries).  We would appreciate any concrete suggestions for improving the presentation.

Reviewer comment (Experiments):
The experimental results are somewhat perplexing. In the comparison between GNN-QE and its extension, ∃GNN-QE, applied to anchored queries, both systems are trained on distinct datasets, making the significance of the reported results unclear.

Author response:
Both GNN-QE and \exists GNN-QE are trained on the same knowledge graphs, but different set of queries: GNN-QE is trained on tree-like anchored queries and \exists GNN-QE is trained on tree-like queries (with and without anchors). In other words, the training\validation set of GNN-QE is a proper subset of the training\validation set of \exists GNN-QE. Experiments show that training on unanchored tree-like queries improves the performance of the model in such queries without compromising its original objective (anchored tree-like queries): \exists GNN-QE achieves competitive results in both anchored and unanchored tree-like queries.

Reviewer comment (Experiments):
There is confusion in the experiments because they appear to include results for queries that may not strictly adhere to conjunctive queries (CQs) and may incorporate other first-order constructs, including negation. To the best of current knowledge, the results presented in the core technical sections are confined to CQs, and the seminal result by Chandra and Merlin only applies to CQs.

Author response:
Regarding negation, our approximation scheme is indeed devised only for CQs, but GNN-QE and \exists GNN-QE are architectures capable of answering arbitrary anchored and arbitrary tree-like queries, which may include union and negation. Hence we include two different type of tests: we compare GNN-QE and \exists GNN-QE in terms of their ability to answer tree-like queries, and then test how  \exists GNN-QE handles (cyclic) CQs via our approximation scheme.

Reviewer comment (Theory):
Please clarify the applicability of your results to queries involving disjunction and negation.

Author response:
Regarding negation, our approximation scheme is indeed devised only for CQs, but GNN-QE and \exists GNN-QE are architectures capable of answering arbitrary anchored and arbitrary tree-like queries, which may include union and negation. Hence we include two different type of tests: we compare GNN-QE and \exists GNN-QE in terms of their ability to answer tree-like queries, and then test how  \exists GNN-QE handles (cyclic) CQs via our approximation scheme.

Reviewer comment (Novelty):
Please clarify whether methods other than that by Zhu et al. can be easily extended to support CQs.

Author response:
We agree with the reviewer. It is not clear how to adapt other methods to handle existential variables. This is why in this paper we focus on GNN-QE which naturally admits this extension. We believe that trying to adapt other methods (or propose new methods) that handle existential variables is an interesting question for future research.

Reviewer comment (Presentation):
The presentation of the paper is poor, which prevents the understanding of the content.

Author response:
We regret that the reviewer finds the presentation poor and we will do our utmost best to improve the presentation. Any specific suggestions by the reviewer as to where things can be improved would be appreciated.

Reviewer comment (Writing):
The motivation and the research question are not clear.

Author response:
**Motivation:** Current state of the art only supports tree-like queries, how to use this to deal with more complex queries? **Answer:** tree-like approximations. **Research question:** What are such approximations and how do these work in practice? We answer both questions in the paper.

Reviewer comment (Evaluation):
The experiment results are not always better than the baseline.

Author response:
Unfortunately we were not able to fully reproduce results from the original GNN-QE paper due to hardware limitations.

In the case of anchored tree-like queries, on our implementation and training, results are closer, while still being slightly better in the original GNN-QE. We believe that this might be a result of the way we created unanchored queries. 

In the case of unanchored tree-like queries, the results are almost always better for \exists GNN-QE, and in the cases where GNN-QE is better the difference is very small.

Reviewer comment (Experiments):
Only experiments on benchmark datasets FB15K and FB15k-237 are not sufficient to support the authors' second and third contribution claims.

Author response:
We also report on NELL in the supplementary material as mentioned in the paper. This is precisely the same experimental setup presented in the original GNN-QE paper and other previous models such as Q2B, betaE and conE.

Reviewer comment (Novelty):
Why shall we be interested in the research of answering arbitrary conjunctive queries over incomplete knowledge graphs?

Author response:
Knowledge bases, and graph databases in general, are one of the most important trends in data management, with dozens of engines available, large public graphs such as Wikidata, and interest fro big companies such as Apache, Google or Amazon. 

Answering conjunctive queries in knowledge graphs is important because they serve as the building block of most knowledge graph query languages (see e.g. Hogan, et al. "Knowledge graphs." ACM Computing Surveys 2021). Conjunctive queries are also important outside knowledge graphs, as they serve as a formalization of SELECT FROM WHERE statements in SQL. 

Finally, since knowledge graphs are commonly used to capture information, one can rarely assume knowledge graphs are complete. The most prominent example is Wikidata, serving as the knowledge graph of Wikipedia: since the aim of Wikipedia is to capture all information known to the human world, in practice we will never have a complete version of it. Hence, link prediction is important in knowledge graphs, and so is being able to answer queries (specially conjunctive queries) in the presence of missing links.

Reviewer comment (Experiments):
Would this method also work for complete knowledge graphs?

Author response:
The model does not incorporate a way of letting it know the input graph is complete, so even if the graph is complete, it will continue trying to predict the most likely links, and using them to answer queries. However, in training we will see all relations present in the graph, so the error should be minimal. 
Although knowledge graphs in practice, as we mention, are usually incomplete, if, theoretically speaking, we find a complete knowledge graph, it will be more efficient to use the standard database machinery developed for complete knowledge graphs, as it is specifically designed for this scenario.

Reviewer comment (Theory):
What is the intuition behind the idea of "approximating a cyclic query by an infinite family of tree-like queries"?

Author response:
The idea is to have tree-like queries that approximate the original query in the sense that their answer sets are close, independently of the underlying knowledge graph. As we take better approximations in the family we expect their answer sets to be closer to the answer set of the original query.

Reviewer comment (Theory):
What if a relation is self-reflective?

Author response:
Our approach is oblivious to the fact that a knowledge graph relation may be self-reflective. If you mean the presence of self-loops in the conjunctive queries, these are considered as cycles. Our approximation approach still applies in the presence of self-loops.

Reviewer comment (Writing):
What do you mean by "neuro-symbolic framework"?

Author response:
We mean methods that work with neural architectures, but in which we introduce a specific bias, obliging that the query processing algorithm uses the symbolic information in how the query is written in a formal query language.

Reviewer comment (Experiments):
One apparent weakness seems to be the addition of yet another hyper-parameter $d$ which determines the depth of the tree to which the cyclic logical query is unraveled.

Author response:
We agree, and this is the reason we did an empirical fitting of the depth. Experiments show that a good depth is when the root of the unraveling appears again in the query.

Reviewer comment (Evaluation):
The proposed approach seems to achieve a lower performance compared to the baseline when evaluated on anchored tree-like queries.

Author response:
Our proposed approach does indeed perform worse on anchored tree-like queries if we look at the MRR. Unfortunately we were not able to fully reproduce results from the original GNN-QE paper due to hardware limitations. In our trained GNN-QE the results are closer, while still being slightly better than the /exists GNN-QE. 

We believe that this might be due to the way we created unanchored queries: we took the anchored ones and randomly removed anchors, this might have added some unwanted noise in the training and validation dataset and worsen the performance on anchored queries. 

We must emphasize: the benefit we obtain for paying this slight price in performance for anchored queries is noticeable: we can now answer unanchored queries with a similar performance, which, in turns, allows for approximating cyclic queries. We feel it is a small price to pay for all we obtain.

Reviewer comment (Evaluation):
Do you have any intuition as to why the proposed approach seems to perform worse, on average, compared to the baseline on anchored tree-like queries?

Author response:
Our proposed approach does indeed perform worse on anchored tree-like queries if we look at the MRR. Unfortunately we were not able to fully reproduce results from the original GNN-QE paper due to hardware limitations. In our trained GNN-QE the results are closer, while still being slightly better than the /exists GNN-QE. 

We believe that this might be due to the way we created unanchored queries: we took the anchored ones and randomly removed anchors, this might have added some unwanted noise in the training and validation dataset and worsen the performance on anchored queries. 

We must emphasize: the benefit we obtain for paying this slight price in performance for anchored queries is noticeable: we can now answer unanchored queries with a similar performance, which, in turns, allows for approximating cyclic queries. We feel it is a small price to pay for all we obtain.

Reviewer comment (Experiments):
In the experimental setup you mentioned that you "additionally provide a new set of training, validation and test queries...". Is this in addition to the unanchored set originally in the dataset? I was under the impression that your method could only handle unanchored queries?

Author response:
Our method can handle both anchored or unanchored queries. 
The original datasets don't include unanchored queries, because previous approaches cannot handle them. We build a set of unanchored queries by randomly removing anchors from the original query sets.