Paper: Understanding Inter-Session Intentions via Complex Logical Reasoning (ICLR-2024)

Reviewer comment (Novelty):
Though the proposed task is novel, the proposed technical solution LSGT relies on existing hypergraph structures and transformer architeactures. Such designs have limited differences compared to existing sequential models and graph models. This lower the technical contribution of this paper.

Author response:
Re W1

As part of our technical contribution, LSGT proposes a new strategy of using tokens to represent sessions and logical structures. This approach demonstrates its novelty and advantage by achieving an average improvement of 2.32 on complex structures including negations, such as 2ina, 2ins, 3in, inp, and pin, over three datasets compared to a model that also uses a Transformer and backbone (SQE).

Furthermore, we conducted additional experiments on four previously unseen query types (3iA, 3ip, 3inA, and 3inp) and showed that our method achieves an average improvement of 2.44 in MRR over three datasets compared to a previous method that also uses a Transformer encoder (SQE+Transformer). The details of this experiment are provided in Re W2.

Reviewer comment (Experiments):
The evaluation part could be enhanced with more diverse experiments to conduct a more comprehensive empirical study, such as ablation study, hyperparameter study, case study on the generated queries, and an investigations on the benefits of LSGT brought to downstream tasks like session-based recommendation.

Author response:
Re W2

Following your suggestions, we included two ablation studies on the hypergraph and logical structures. In the first study, we removed the tokens representing the logical structures. In the second study, we removed the order information in the hypergraph by removing the positional encoding features of item tokens in each session.

Here are the results:

| Dataset    | Encoder             | Average |   1p  |   2p  |  2ia  |  2is  |   3i  |   pi  |   ip  |   2u  |   up  | 2ina  | 2ins  | 3in   | inp   | pin   |
|------------|---------------------|---------|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|-------|-------|-------|-------|-------|
| Amazon     | LSGT                |  31.99  | 17.73 |  9.10 | 56.73 | 84.26 | 69.39 | 19.39 | 19.47 | 15.40 |  7.86 | 20.98 | 22.00 | 60.70 | 35.95 |  8.84 |
|            | w/o Logic Structure |  15.98  |  5.41 |  2.31 | 30.31 | 50.21 | 45.21 |  3.75 |  5.49 |  4.88 |  2.56 | 16.32 | 15.77 | 38.19 |  2.13 |  1.11 |
|            | w/o Session Order   |   8.45  |  6.29 |  2.59 | 17.22 | 13.85 | 19.34 | 14.07 |  3.23 |  3.49 |  1.73 |  5.50 |  4.75 | 17.54 |  4.92 |  3.73 |
| Diginetica | LSGT                |  40.59  | 32.00 | 15.27 | 83.34 | 90.61 | 86.05 | 15.62 | 33.80 | 26.34 | 14.45 | 24.15 | 28.69 | 83.04 | 19.21 | 15.62 |
|            | w/o Logic Structure |  27.17  | 18.61 |  3.84 | 68.40 | 62.80 | 64.87 | 10.13 | 20.22 | 16.08 |  8.49 | 17.38 | 14.17 | 60.37 |  9.21 |  5.74 |
|            | w/o Session Order   |  17.07  |  5.08 |  9.71 | 45.49 | 34.42 | 43.23 |  9.69 | 21.71 |  3.66 |  7.92 |  4.39 |  2.56 | 35.80 |  9.98 |  5.36 |
| Dressipi   | LSGT                |  70.22  | 31.12 | 96.16 | 64.26 | 76.85 | 78.66 | 98.02 | 96.98 | 28.83 | 96.04 | 25.58 | 30.66 | 65.93 | 97.74 | 96.30 |
|            | w/o Logic Structure |  25.13  | 14.87 |  2.45 | 42.03 | 59.63 | 67.62 |  9.27 | 17.71 | 18.05 |  7.64 | 19.62 | 24.67 | 59.01 |  1.95 |  7.29 |
|            | w/o Session Order   |  39.78  |  9.21 | 42.80 | 21.57 | 19.57 | 23.28 | 88.31 | 61.47 |  6.31 | 68.27 |  7.41 |  6.87 | 15.96 | 96.53 | 89.35 |

Removing the logical structure information drastically dropped the model's performance, especially for queries involving negations and multi-hop reasoning, such as ip, pi, inp, and pin. Without the logical structure, the model could only use co-occurrence information like "bag of sessions" and "bag of items" to rank candidate answers. While this information may be useful for simple structured queries, it is not very useful for complex structured queries.

Similarly, removing the order information within each session also drastically dropped the overall performance. This demonstrates two things: First, the item orders in each session are critical in this task. Second, the LSGT model is effectively able to utilize the order information for this task.

In addition, we included a new evaluation on the compositional generalization of LSGT. The newly included query types are 3iA, 3ip, 3inA, and 3inp. We selected them because they are all complex types involving three anchors, cover both EPFO queries and queries involving negations, and have both 1-hop and 2-hop relational projections in the reasoning process. These query types were not trained during the training process but were evaluated in a zero-shot manner.

Here are their performances against the baselines:

## Rebuttal 2 (ID: zMAtflrN8D)
| Dataset | Query Encoder | 3iA | 3ip | 3inA | 3inp | Average OOD |
|---|---|:---:|:---:|:---:|:---:|:---:|
| Amazon | FuzzQE + Attn-Mixer | 66.72 | 29.67 | 54.33 | 48.76 | 49.87 |
|  | Q2P + Attn-Mixer | 33.51 | 11.42 | 51.47 | 41.46 | 34.47 |
|  | NQE | 61.72 | 1.98 | 46.47 | 34,04 | 36.72 |
|  | SQE + Transformers | 66.03 | 28.41 | 55.61 | 51.28 | 50.33 |
|  | LSGT (Ours) | **68.44** | **34.22** | **58.50** | **51.49** | **53.16 (+2.83)** |
| Diginetica | FuzzQE + Attn-Mixer | 88.30 | 32.88 | 82.75 | 34.50 | 59.61 |
|  | Q2P + Attn-Mixer | 40.28 | **43.93** | 54.31 | **48.20** | 46.68 |
|  | NQE | 86.25 | 20.79 | 64.74 | 20.93 | 48.18 |
|  | SQE + Transformers | 88.05 | 31.33 | 81.77 | 35.83 | 59.25 |
|  | LSGT (Ours) | **91.71** | 35.24 | **83.30** | 41.05 | **62.83 (+3.22)** |
| Dressipi | FuzzQE + Attn-Mixer | 65.43 | 95.64 | 53.36 | 97.75 | 78.05 |
|  | Q2P + Attn-Mixer | 60.64 | 96.78 | 52.22 | 97.28 | 76.73 |
|  | NQE | 31.96 | 96.18 | 9.89 | 97.80 | 58.96 |
|  | SQE + Transformers | 72.61 | 97.12 | 55.20 | 98.14 | 80.77 |
|  | LSGT (Ours) | **74.34** | **97.30** | **58.30** | **98.23** | **82.04 (+1.28)** |

Our proposed method demonstrates stronger compositional generalization on these previously unseen query types and achieves MRR improvements ranging from 1.28 to 3.22 on three datasets.

Reviewer comment (Novelty):
The formulated task of Logical Session Query Answering is essentially query answering over hypergraphs. Sessions are n-ary edges, and other relations form 2-ary edges, so the hypergraph has edges of different arity. The temporal aspect of items in session hyperedges (that items follow each other in one session) seems to be of little use as the best-performing models are not using this information anyway. The reviewer recommends the authors to focus the contribution on extending complex query answering to hypergraphs as there is not that much work in that subfield (StarQE is for hyper-relational graphs, and NQE supports both hyper-relational and hypergraphs).

Author response:
Regarding the task formulation, we have taken your advice and extended the problem definition of CQA to include sessions as ordered hypergraphs of varying arity, namely LS-CQA. We also want to emphasize that the order of items in the session is crucial for recommending relevant items, and top-performing models such as SQE, LSGT, and session encoders all take this into account.

Furthermore, we want to highlight the novelty of our problem formulation, as we are the first to bridge the CQA problem with session-based recommendations. CQA methods and knowledge graph reasoning are essential for session understanding, and our approach is unique in that it addresses user anonymity, which is a common feature of session-based recommendation tasks. By incorporating logical graph reasoning methods, we believe our approach is not only novel but also useful for improving session-based recommendation systems.

Reviewer comment (Novelty):
The proposed logical session graph transformer (LSGT) is just one of the many query linearization strategies, e.g., BiQE, kgTransformer, or SQE, that convert the query graph into a sequence with some positional information to be sent jointly into a Transformer. Architecture-wise, LSGT is TokenGT but with a slightly different input format that sends tokens of logical operators. Experimentally, LSGT is very close to SQE (the gap is often <1 MRR point) so it is hard to claim any novelty or effectiveness in this linearization strategy or in a slightly different transformer encoder.

Author response:
We have taken your advice and demonstrated that transformer-based methods have comparable performance despite using different linearization strategies. However, we want to highlight that the linearization strategy employed in LSGT outperforms others in queries with negations.

Furthermore, we have conducted additional experiments to investigate the compositional generalization of different linearization strategies. Our results show that the linearization strategy used in LSGT exhibits better compositional generalization than other existing strategies. Below are the detailed performance results: 


| Dataset | Query Encoder | 3iA | 3ip | 3inA | 3inp | Average OOD |
|---|---|:---:|:---:|:---:|:---:|:---:|
| Amazon | FuzzQE + Attn-Mixer | 66.72 | 29.67 | 54.33 | 48.76 | 49.87 |
|  | Q2P + Attn-Mixer | 33.51 | 11.42 | 51.47 | 41.46 | 34.47 |
|  | NQE | 61.72 | 1.98 | 46.47 | 34,04 | 36.72 |
|  | SQE + Transformers | 66.03 | 28.41 | 55.61 | 51.28 | 50.33 |
|  | LSGT (Ours) | **68.44** | **34.22** | **58.50** | **51.49** | **53.16 (+2.83)** |
| Diginetica | FuzzQE + Attn-Mixer | 88.30 | 32.88 | 82.75 | 34.50 | 59.61 |
|  | Q2P + Attn-Mixer | 40.28 | **43.93** | 54.31 | **48.20** | 46.68 |
|  | NQE | 86.25 | 20.79 | 64.74 | 20.93 | 48.18 |
|  | SQE + Transformers | 88.05 | 31.33 | 81.77 | 35.83 | 59.25 |
|  | LSGT (Ours) | **91.71** | 35.24 | **83.30** | 41.05 | **62.83 (+3.22)** |
| Dressipi | FuzzQE + Attn-Mixer | 65.43 | 95.64 | 53.36 | 97.75 | 78.05 |
|  | Q2P + Attn-Mixer | 60.64 | 96.78 | 52.22 | 97.28 | 76.73 |
|  | NQE | 31.96 | 96.18 | 9.89 | 97.80 | 58.96 |
|  | SQE + Transformers | 72.61 | 97.12 | 55.20 | 98.14 | 80.77 |
|  | LSGT (Ours) | **74.34** | **97.30** | **58.30** | **98.23** | **82.04 (+1.28)** |

Reviewer comment (Theory):
The theoretical study in Section 4.5 is derived from TokenGT and seems to be hardly applicable to the case of logical query answering. TokenGT’s theory of WL expressiveness assumes the graphs are non-relational whereas all logical query graphs studied in this work are relational, i.e., they have labeled edge types. There is a different line of work studying expressiveness of GNNs over relational graphs and the reviewer recommends starting from them in order to derive any expressiveness claims. Permutation invariance proofs are rather trivial because the Transformer architecture itself is permutation equivariant.

Author response:
We have read and analyzed the reference papers [1,2], and we developed a new expressive claim based on the relational expressiveness of LSGT. We proved that the LSGT is at least as powerful as 1-RWL, namely the expressiveness of R-GCN and CompGCN. We include the proof in the appendix and mark it in blue. 


[1] Barcelo et al. Weisfeiler and Leman Go Relational. LOG 2022.
[2] Huang et al. A theory of link prediction via relational Weisfeiler-Leman. NeurIPS 2023.

We really appreciate your constructive feedback. If you have any other concerns please do not hesitate to tell us.

Reviewer comment (Writing):
There are some obvious typos. Authors should scrutinize the writing of the paper.

Author response:
Thank you so much, we have fixed the typos you mentioned and marked them in blue in the revised paper. We have also conducted a thorough check on the grammatical issues and typos and also fixed them at the same time.

Reviewer comment (Presentation):
The paper lacks detailed description for figures especially Figure 3, which is hard to understand for readers.

Author response:
We've made some updates to our paper. Specifically, we added a detailed description for Figure 3 to better explain our problem setting and relevant query encoding methods. We marked the newly added explanations in blue for your convenience.

Reviewer comment (Experiments):
It would be better to evaluate the model’s generalization ability of unseen query structures like [1,2,3].

Author response:
We have followed your suggestions and added new experiment results on compositional generalization. The newly included query types are 3iA, 3ip, 3inA, and 3inp. We selected these query types because they are complex, involve three anchors, cover both EPFO queries and queries involving negations, and have both 1-hop and 2-hop relational projections in the reasoning process. These query types were not trained during the training process but were evaluated in a zero-shot manner.


| Dataset | Query Encoder | 3iA | 3ip | 3inA | 3inp | Average OOD |
|---|---|:---:|:---:|:---:|:---:|:---:|
| Amazon | FuzzQE + Attn-Mixer | 66.72 | 29.67 | 54.33 | 48.76 | 49.87 |
|  | Q2P + Attn-Mixer | 33.51 | 11.42 | 51.47 | 41.46 | 34.47 |
|  | NQE | 61.72 | 1.98 | 46.47 | 34,04 | 36.72 |
|  | SQE + Transformers | 66.03 | 28.41 | 55.61 | 51.28 | 50.33 |
|  | LSGT (Ours) | **68.44** | **34.22** | **58.50** | **51.49** | **53.16 (+2.83)** |
| Diginetica | FuzzQE + Attn-Mixer | 88.30 | 32.88 | 82.75 | 34.50 | 59.61 |
|  | Q2P + Attn-Mixer | 40.28 | **43.93** | 54.31 | **48.20** | 46.68 |
|  | NQE | 86.25 | 20.79 | 64.74 | 20.93 | 48.18 |
|  | SQE + Transformers | 88.05 | 31.33 | 81.77 | 35.83 | 59.25 |
|  | LSGT (Ours) | **91.71** | 35.24 | **83.30** | 41.05 | **62.83 (+3.22)** |
| Dressipi | FuzzQE + Attn-Mixer | 65.43 | 95.64 | 53.36 | 97.75 | 78.05 |
|  | Q2P + Attn-Mixer | 60.64 | 96.78 | 52.22 | 97.28 | 76.73 |
|  | NQE | 31.96 | 96.18 | 9.89 | 97.80 | 58.96 |
|  | SQE + Transformers | 72.61 | 97.12 | 55.20 | 98.14 | 80.77 |
|  | LSGT (Ours) | **74.34** | **97.30** | **58.30** | **98.23** | **82.04 (+1.28)** |

We compared their performance against the baselines and found that our proposed method showed stronger compositional generalization on these unseen query types. It achieved MRR improvement ranging from 1.28 to 3.22 on three datasets.

Reviewer comment (Experiments):
Why do authors not evaluate the model’s generalization ability of unseen query structures like existing works?

Author response:
We have added the evaluation of compositional generalization, as in previous work, and included the results in the original paper. Initially, we did not include compositional generalization because we believed it was not very important in the real setting of session reasoning. We thought that if we wanted to improve performance on some unseen query types, a better approach would be to sample more data with that query type and conduct supervised training on it. 

However, we now realize the importance of compositional generalization in the general problem of complex query answering. Therefore, we further included the experiment in Re W4 to demonstrate the effectiveness of our method on compositional generalization.

Reviewer comment (Experiments):
Is there an explanation for the author's choice of 14 query structures? Can some other query structures like 2i, and pni be incorporated?

Author response:
We selected and derived fourteen query types from the original fourteen types of queries in [1,2]. The 2iS and 2iA are both derived from 2i queries. They involve 1 session 1 item and 2 sessions as their anchors, respectively.

Regarding pni, we found it difficult to interpret its practical meaning in a real problem setting. It can be interpreted as "Predict the attribute value of a given product that is not desired by a given session." However, we believe that predicting the attribute value for a single item is not meaningful in session understanding. Therefore, we did not include this type in our training and evaluation so that we could focus on other important query types.

Reviewer comment (Experiments):
Is it possible to make an ablation study for hypergraph and logical reasoning?

Author response:
We followed your suggestions and included two ablation studies on the hypergraph and logical structures, respectively. In the first ablation study, we removed the tokens representing the logical structures. In the second ablation study, we removed the order information in the hypergraph by removing the positional encoding features of item tokens in each session.

The results are as follows:

| Dataset    | Encoder             | Average |   1p  |   2p  |  2ia  |  2is  |   3i  |   pi  |   ip  |   2u  |   up  | 2ina  | 2ins  | 3in   | inp   | pin   |
|------------|---------------------|---------|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|-------|-------|-------|-------|-------|
| Amazon     | LSGT                |  31.99  | 17.73 |  9.10 | 56.73 | 84.26 | 69.39 | 19.39 | 19.47 | 15.40 |  7.86 | 20.98 | 22.00 | 60.70 | 35.95 |  8.84 |
|            | w/o Logic Structure |  15.98  |  5.41 |  2.31 | 30.31 | 50.21 | 45.21 |  3.75 |  5.49 |  4.88 |  2.56 | 16.32 | 15.77 | 38.19 |  2.13 |  1.11 |
|            | w/o Session Order   |   8.45  |  6.29 |  2.59 | 17.22 | 13.85 | 19.34 | 14.07 |  3.23 |  3.49 |  1.73 |  5.50 |  4.75 | 17.54 |  4.92 |  3.73 |
| Diginetica | LSGT                |  40.59  | 32.00 | 15.27 | 83.34 | 90.61 | 86.05 | 15.62 | 33.80 | 26.34 | 14.45 | 24.15 | 28.69 | 83.04 | 19.21 | 15.62 |
|            | w/o Logic Structure |  27.17  | 18.61 |  3.84 | 68.40 | 62.80 | 64.87 | 10.13 | 20.22 | 16.08 |  8.49 | 17.38 | 14.17 | 60.37 |  9.21 |  5.74 |
|            | w/o Session Order   |  17.07  |  5.08 |  9.71 | 45.49 | 34.42 | 43.23 |  9.69 | 21.71 |  3.66 |  7.92 |  4.39 |  2.56 | 35.80 |  9.98 |  5.36 |
| Dressipi   | LSGT                |  70.22  | 31.12 | 96.16 | 64.26 | 76.85 | 78.66 | 98.02 | 96.98 | 28.83 | 96.04 | 25.58 | 30.66 | 65.93 | 97.74 | 96.30 |
|            | w/o Logic Structure |  25.13  | 14.87 |  2.45 | 42.03 | 59.63 | 67.62 |  9.27 | 17.71 | 18.05 |  7.64 | 19.62 | 24.67 | 59.01 |  1.95 |  7.29 |
|            | w/o Session Order   |  39.78  |  9.21 | 42.80 | 21.57 | 19.57 | 23.28 | 88.31 | 61.47 |  6.31 | 68.27 |  7.41 |  6.87 | 15.96 | 96.53 | 89.35 |

When we removed the logical structure information, the model's performance drastically dropped, especially for queries involving negations and multi-hop reasoning, such as ip, pi, inp, and pin. Without the logical structure, the model could only use co-occurrence information like "bag of sessions" and "bag of items" to rank candidate answers. While this information may be useful for simple structured queries, it is not very useful for complex structured queries.

Similarly, when we removed the order information within each session, the overall performance also drastically dropped. This demonstrates two things: First, the item orders in each session are critical in this task. Second, the LSGT model is effectively able to utilize the order information for this task.