Paper: ${\rm EFO}_k$-CQA: Towards Knowledge Graph Complex Query Answering beyond Set Operation (ICLR-2024)

Reviewer comment (Presentation):
The presentation of the paper needs to be improved.

Author response:
Thank you for your valuable feedback. We highly value your expertise in logic and databases, and will improve our paper, including more rigorous definitions, correcting typos in presentation, and citing related previous research according to your suggestions. However, there are also some misunderstandings we would like to clarify:

Reviewer comment (Novelty):
The treatment of parameters upon which the difficulty of conjunctive query evaluation depends is somewhat superficial. Much research on this topic has been conducted in databases, but the authors do not take this into consideration.

Author response:
We really appreciate your suggestion. It is really important and necessary to include more parameters like treewidth and is also a common practice in the discussion of traditional database/CSP communities. However, we want to point out that we have made the assumption in Section 5, the footnote on the 8th page that the number of edge of query graph is no more than the number of nodes, therefore, the treewidth is either 2 or 1, making it pointless for discussion. We will leave this to future work to include even more complicated query graph.

Reviewer comment (Presentation):
The presentation of the paper needs to be improved. In particular, some definitions need to be clarified.

Author response:
Thank you for your valuable feedback. We highly value your expertise in logic and databases, and will improve our paper, including more rigorous definitions, correcting typos in presentation, and citing related previous research according to your suggestions. However, there are also some misunderstandings we would like to clarify:

Reviewer comment (Theory):
Definition 4: Consider the formula \(P(x) \wedge \exists x R(x,y)\). In this case, both \(x\) and \(y\) are free variables, although only \(y\) is considered free according to the definition in the paper. This may seem like a minor point, but the correct definition of this notion is necessary when considering a logic with a bounded number of variables, which is a standard way to construct fragments of first-order logic with tractable query evaluation.

Author response:
We appreciate your suggestions. We presume that all variables are named differently as it is the standard way in first order logic without clear explanation. We have corrected this point to make our discussion more clear.

Reviewer comment (Theory):
Paragraph after Definition 9: The authors claim the following: "... the inference of existential formulas is easier than solving CSP instances since the existential variables do not need to be kept track of". This claim is unclear, as the complexity of CSP solving and conjunctive query evaluation is the same (both problems are NP-complete).

Author response:
Your understanding is right and both solving  CSP and conjunctive query evaluation are NP-complete. However, our claim is about the polynomial time reduction itself. Moreover, this polynomial reduction is very powerful in a smaller n (<10) like in our cases. In practice, our inference techniques are 100 times faster than CSP solvers. Certainly, both problems are NP-complete and it doesn’t conflict.

Reviewer comment (Theory):
Definition 7: What does it mean that “\(\phi(a_1, ..., a_k)\) is True”? Where is this query evaluated? Do you assume a fixed knowledge graph over which all queries are evaluated?

Author response:
Thank you for your suggestions, since our semantics is standard and it is a common practice[1] to state in a intuitive way, we didn’t give a rigorous definition of the semantics of the query. Regarding the evaluation and OWA, the underlying knowledge graph KG is taken as ground truth to evaluate the query. Regarding the negation, any unobserved triple is considered negated. We will include this part of definition in the preliminary.

Reviewer comment (Theory):
Definition 7: How is the semantics of negation defined under OWA? Do you consider a certain answer semantics over some possible worlds (otherwise no negative atoms can be inferred)? The authors use the term “\(\phi(a_1, ..., a_k)\) is True” without defining it.

Author response:
Thank you for your suggestions, since our semantics is standard and it is a common practice[1] to state in a intuitive way, we didn’t give a rigorous definition of the semantics of the query. Regarding the evaluation and OWA, the underlying knowledge graph KG is taken as ground truth to evaluate the query. Regarding the negation, any unobserved triple is considered negated. We will include this part of definition in the preliminary.

Reviewer comment (Theory):
The treatment of parameters upon which the difficulty of conjunctive query evaluation depends is somewhat superficial. The authors have contributed by lifting the restriction that conjunctive queries must be tree-shaped and by considering an approach to generate general graph-shaped queries. However, between trees and general graphs, there exists a large number of structures defined in terms of parameters studied to analyze the complexity of query evaluation. Notably, the authors could have considered the notions of treewidth and hypertree width, as explained in the following references:

Jörg Flum, Martin Grohe: Parameterized Complexity Theory. Texts in Theoretical Computer Science. An EATCS Series, Springer 2006.

Georg Gottlob, Gianluigi Greco, Nicola Leone, Francesco Scarcello: Hypertree Decompositions: Questions and Answers. PODS 2016: 57-74

Additionally, the following book provides a detailed view of fast conjunctive query evaluation:

https://github.com/pdm-book/community

Author response:
We really appreciate your suggestion. It is really important and necessary to include more parameters like treewidth and is also a common practice in the discussion of traditional database/CSP communities. However, we want to point out that we have made the assumption in Section 5, the footnote on the 8th page that the number of edge of query graph is no more than the number of nodes, therefore, the treewidth is either 2 or 1, making it pointless for discussion. We will leave this to future work to include even more complicated query graph.

Reviewer comment (Novelty):
Why do you need to develop your own algorithm to compute the answers to an existential formula, as described in Section 4.3? Why can't you leverage the substantial body of work and existing implementations for answering first-order queries?

Author response:
Thank you for your suggestions. Firstly, our implementation is consistent with existing implementations and we have already double-checked its correctness. Additionally, we will cite more existing work for clarification and we reckon that our own algorithm in Section 4.3 was not a theoretical innovation but rather a convenient implementation that fits into our framework well.

Reviewer comment (Experiments):
The authors discussed some theoretical properties of EFO(k) queries, but did not provide enough insights on how to design a CQA model for the new types of queries that satisfy these properties. It would significantly increase the impact of the paper if the authors could explicitly include such a section.

Author response:
We would like to express our gratitude for your helpful reminder regarding the details of extending previous models to answer queries beyond tree structures. In fact, we believe this is a general direction in the database community, and we have noted very recent research trying to filling this gap[1], we will provide this discussion in our paper. Actually, our work provides some general extension of the previous model like ConE which can only deal with tree-form query initially. And we show that more general methods like LMPNN suffers from the biased selection of training data severely. However, we have not designed any new methods since this is a dataset and benchmark paper, and we strive to test the generalizability of existing methods, providing corresponding discussion. We will leave the more sophisticated design of CQA models in the future but we will surely include more discussions of it thanks to your advice.

Reviewer comment (Theory):
It is not clear how the previous model like BetaE/LogicE/ConE etc. are extended to handle cyclic and multigraph queries. These models are known to be able to only handle tree-form queries, as far as I know.

Author response:
Thank you for your correct observation, surely these models are known to be able to only handle tree-form queries. In fact, this extension is done by us while ensuring the backward compatibility, this is explained in the Section 4.4 and detailed in Appendix F, and we consider this as one of our contribution of our paper.

Reviewer comment (Experiments):
Dataset of query graphs with Answer Sets built from Query graph sampling - It is not validated whether these produce meaningful query graphs when grounded to real-concepts, and how representative is this dataset for Complex QA on a KG with natural questions.

Author response:
> Dataset of query graphs with Answer Sets built from Query graph sampling - It is not validated whether these produce meaningful query graphs when grounded to real-concepts….

> There seem to be several key limitations - How does this help with improving Complex questions answering of natural question datasets e.g. WebQSP, Complex WebQSP..…

We would like to answer these two questions together, as they seem to make the same presumption that query in formal language (first order logic) is only practical when it can be found in natural language, and the task of complex query answering (CQA) is therefore subordinate to natural language task like Knowledge Graph Question Answering (KGQA). However, this is not the case. We would like to refer you to the first question in our general feedback for the discussion where we refute this presumption and we do not reiterate the context here.

In short, the introduction of complicated queries, such as multi-variable queries, even though hard to find in natural language, is crucial for advancing the development of CQA in the sense of strengthening/benchmarking the combinatorial generalizability, which greatly differs from KGQA which is conducted in natural language.

Reviewer comment (Experiments):
There seem to be several key limitations - How does this help with improving Complex questions answering of natural question datasets e.g. WebQSP, Complex WebQSP. First, the approach here does not examine how natural questions from such datasets can be formulated into query graphs with any automatic methods (that are not manual, which would be critical to use this approach).

Author response:
> Dataset of query graphs with Answer Sets built from Query graph sampling - It is not validated whether these produce meaningful query graphs when grounded to real-concepts….

> There seem to be several key limitations - How does this help with improving Complex questions answering of natural question datasets e.g. WebQSP, Complex WebQSP..…

We would like to answer these two questions together, as they seem to make the same presumption that query in formal language (first order logic) is only practical when it can be found in natural language, and the task of complex query answering (CQA) is therefore subordinate to natural language task like Knowledge Graph Question Answering (KGQA). However, this is not the case. We would like to refer you to the first question in our general feedback for the discussion where we refute this presumption and we do not reiterate the context here.

In short, the introduction of complicated queries, such as multi-variable queries, even though hard to find in natural language, is crucial for advancing the development of CQA in the sense of strengthening/benchmarking the combinatorial generalizability, which greatly differs from KGQA which is conducted in natural language.

Reviewer comment (Novelty):
I am not clear about how the learning-based methods are relevant here. The dataset is formed by solving some version of CSP (details lacking entirely of their "own algorithm" in Sec 4.3), and then what is the relevance of learning based methods to embed the query graph and try to infer the answer? And this still does not connect to the complex natural question KGQA methods to produce answers for natural questions from the KG.

Author response:
> I am not clear about how the learning-based methods are relevant here…And this still does not connect to the complex natural question KGQA methods to produce answers for natural questions from the KG….

We thank you for your questions. In fact, there have been many previous learning-base methods that focus on complex query answering, and one of our core contributions of this paper, is to benchmark the generalizability, how well they can handle query with different graph features, like different topology properties that may have not been considered during their model design. Regarding the connection to KGQA, we believe this question are addressed above.

Reviewer comment (Writing):
Section 4.3 "we develop our own algorithm following the standard solving technique of CSP, which ensures consistency conditions in the first step, and do the backtracking to get the final answers in the second step".

Author response:
> My other question is as follows: The CSP approximation solver (since its NP hard otherwise) is used to find the answer set of free variables in that query graph. When sampling, isn't it already known what entities/concepts from the KG fill those free variables? then why is CSP needed? And if I understand correctly, the learning based approach is just used to learn the CSP solution- because prior works have used learning based solutions? But how is it necessary when CSP solver can just be used instead?

We thank you for your concerns. However, as stated in the introduction, we have open world assumption (OWA) about the incompleteness of knowledge graph in our problem setting, which is reiterated in the first paragraph in Section 2.1 that the **observed knowledge graph** is only part of the **real knowledge graph**. Therefore, CQA models are always evaluated by how well they can infer those “hard answers” that can not be found in the observed knowledge graph, which is explained in Section 4.5, we welcome you to check that. On the other hand, a CSP solver can not infer any of those “hard answers”,  and thus CSP solver is only useful when preparing the data when we can use the underlying real knowledge graph rather than just the observed part. 

To summarize, we need learning-based models because we need their generalizability to infer those missing information in the knowledge graph.

Reviewer comment (Presentation):
Approach presentation and clarity - The whole Framework section (4.1-4.4) in the main paper is limited, and does not at all deliver what the approach is (and even if we read through the Appendix, its not presented well enough to grasp the key points of the approach at high-level but with sufficient detail). The presentation is poor with unnecessary assumptions and elaborations (unrelated to the proposed contributions) listed in the main paper, and much of the methodology in the Appendix.

Author response:
> Approach presentation and clarity - The whole Framework section (4.1-4.4) in the main paper is limited,..not presented well enough to grasp the key points …
Regarding the issue of presentation, we would like to hear actionable suggestions about the context, both in the main paper and the appendix. Our goal is to explain the intuitive and brief idea in the main paper and provide enough details in the appendix. For example, in the main paper, we provide Figure 3, where each component in the framework section  (4.1-4.4) is clearly illustrated, which can grasp the key point of our framework. In the appendix, we offer additional illustrations in Figure 5, together with the writing in Appendix D.1 and Section 4.1, clearly explains how to enumerate all possible abstract query graphs. Similarly, Appendix F is for Section 4.4 where we provide with the pseudo-code here. We will certainly strive to improve the presentation of our paper, but we would like to hear more actionable suggestions.

Reviewer comment (Novelty):
This paper is derivative - It extends to only 2 free variables in practice (and the extension to multiple free variables, over their prior work Yin et al 2023 seems trivial?).

Author response:
> This paper is derivative…..

Regarding this issue, we would like to refer you to the second question in our general rebuttal. Here, we add some brief additional comments.

>> It extends to only 2 free variables in practice (and the extension to multiple free variables, over their prior work Yin et al 2023 seems trivial?).

As discussed in the general rebuttal, the extension from 1 free variable to 2 is fundramental, while 2 to k is derivative instead.

Reviewer comment (Theory):
The two assumptions they define are extraneous as they themselves suggest this cannot be checked in practice (so the contribution claim that "Our assumption is more systematic than previous ones as shown by the example in Figure 2." is not useful).

Author response:
>> The two assumptions they define are extraneous as they themselves suggest this cannot be checked in practice.

This is a very ** clear misunderstandings**. As we state in Section 3.1, Assumption 13 and 14 apply to **abstract query graph** because putting these assumptions to **grounded query graph** should be avoided in practice. Therefore, we strictly abide by these two assumptions we put forward in practice.

Reviewer comment (Experiments):
The heuristics to sample and ground the dataset are not obvious, clear, or validated, or effectively presented.

Author response:
>> The heuristics to sample and ground the dataset are not obvious, clear, or validated, or effectively presented.

If your concern is about the natural language,  it has been addressed above. If you concern is about our presentation, we would like to hear more detailed suggestions as we have also discussed above.

Reviewer comment (Experiments):
benchmark does not include ways of generating new data

Author response:
Thank you for your questions. However, we have produced many new queries with novel query types including both single variable and multi variables. To the best of our knowledge, we surpass all previous work regarding the query types we investigated and the process of sampling data is totally novel.

Reviewer comment (Experiments):
no concern over which queries are more practical than others, results may be altered by queries that would be hard to find in practice.

Author response:
We thank you for your concerns, however, it seems that you presume query in formal language (first order logic) is only practical when it can be found in natural language. However, this is not the case. We refer you to the first question in our general feedback for the discussion where we refute this presumption. In short, the input of CQA models is in formal language rather than neutral language, and the CQA models aim to be trained on selected query types and combinatorially generalize to unseen complex patterns in first-order logic[1]. In this way, the study of combinatorial generalizability of the CQA models plays a pivotal role in the task of representation learning[2]. Therefore, the introduction of those complicated queries that can hardly be found in natural language has its own merit: benchmarking the combinatorial generalizability of the CQA models.

Reviewer comment (Novelty):
limited impact: I consider this as an add on to complement work by Ren et al.

Author response:
We thank you for your concerns, however, our dataset is a huge extension to the work of Ren[1]. We would like to refer you to the second question in our general feedback. In short, we already explained in the introduction that existing dataset in complex query answering is severely inadequate, because their construction is biased and has not discussed queries with some patterns entirely: multigraph, cyclic graph, multiple free variables, resulting in lack of both combinatorial answers and structural hardness. This huge extension was also mentioned in Section 3.2 and detailed in Appendix D.3, where we provide Figure 6, clearly illustrating how huge our extension is. We welcome you to check that.

Reviewer comment (Theory):
Please clarify the rationale behind the idea of answering queries with more than 1 free variable by decomposing that queri into several unary pieces. Queries with more than one free variable have a standard interpretation, which is answering tuples, and it is not clear to me that any of the measures would be actually important in practice.

Author response:
If you are referring to the construction of our dataset, we abide by Assumption 14, which ensures that a query with more than 1 free variable can not be structurally decomposed into several unary pieces, which ensures the combinatorial hardness we mention in the introduction that each answer should be a tuple and the answer to each free variable are thus correlated. Therefore, the decomposition is strictly avoided when constructing our EFOk-CQA dataset.

If you are referring to our implementation and evaluation of the previous CQA models, your understanding is correct and we all agree that the ultimate goal is to answer tuples. However, all previous methods **are designed to infer queries with only 1 free variable**, and they fundamentally lack the ability to retrieve combinatorial answers. Therefore, in our implementation, we have to let those models infer the answer of each free variable individually, as explained in Section 4.4 and Appendix F. In our evaluation, **only the joint metric** proposed in Section 4.5 serves the ultimate goal of answering tuples/combinatorial answers, while others are designed as weaker metrics, as a **compromise** when we currently have no method that can retrieve the combinatorial answers. The benchmark result in Table 2 where all methods have very low joint HIT@10 score supports our discussion above and thus needs other easier metrics for the discussion. We believe the marginal and multiply metrics all only serve as milestones towards the ultimate goal, the joint metric.