Paper: JoinGym: An Efficient Query Optimization Environment for Reinforcement Learning (ICLR-2024)

Reviewer comment (Evaluation):
The paper will become more valuable by additionally discussing the pros and cons of the compared reinforcement learning (RL) methods and clearly outlining future research challenges.

Author response:
**W1.** We've outlined three concrete future research directions as 1) long-tailed returns, 2) generalization in combinatorial optimization and 3) partial observability. Admittedly, our discussion section was cut short due to space constraints. Hence, we will add the following paragraph discussing the pros and cons of compared RL methods:

To recap, we compared four RL algorithms spanning three categories: off-policy Q-learning (DQN), off-policy actor-critic (TD3 and SAC), and on-policy actor-critic (PPO). Although PPO does not always obtain the best test-set performance, we can see in Tables 5,6 that it reliably obtains decent quantile performance in all settings of JoinGym. Hence, PPO has the advantage of being the most stable and the disadvantage of often plateauing at a sub-optimal solution. On the other hand, off-policy actor-critic methods (TD3 and SAC) perform the best for mean and $99\%$ quantile, although PPO's CCM is usually within an order of magnitude. Hence, off-policy actor-critic methods are less stable but can sometimes converge to a better solution. DQN and DDQN tend to be uniformly worse off, which suggests that actor critic outperforms Q-learning for this problem. With this said, all of these algorithms are optimizing for the mean of a highly varied, long-tailed distribution of returns; thus, designing better algorithms that can handle long-tailed rewards will likely improve performance across all the algorithms.

Reviewer comment (Writing):
Most parts of the paper heavily rely on knowledge of Markov Decision Process (MDP), which makes it difficult to read. I suggest the authors provide a preliminary overview of MDP to improve readability.

Author response:
**W3.** Please see Section 4.1 for the MDP setup. We kept it short due to space constraints and can certainly point the reader to a longer exposition, e.g. Sutton and Barto, 2018.

Reviewer comment (Presentation):
The abstract seems somewhat inconsistent with the main content of the paper. For example, the authors state "key advantages of JOINGYM are usability and significantly higher throughput," but there is no description in the main content that pertains to usability and throughput.

Author response:
**W4.** Please see our discussion on usability and higher throughput in "Related works: Environments for Query Optimization" and a longer discussion in Appendix C. We acknowledge that this could have been highlighted better and we will amend our paper with the following paragraph:

JoinGym improves over prior query optimization simulators in two main ways: easier to use and higher throughput. First, JoinGym is easier to use because we do not require the user to set up a DBMS or cost model while prior simulators do. In other words, starting up JoinGym is as simple as starting up CartPole or Mountain Car in OpenAI Gym. Second, JoinGym can achieve a much higher throughput because each simulator step is simply a table lookup from our novel dataset. Prior simulators either execute the query (which can take hours to days for large queries) or estimate the query cost with a cost model (which can still take seconds), both of which are slower and require more infrastructure setup. Indeed, JoinGym can reliably simulate thousands of queries per second on a standard laptop. Moreover, the cost model can have estimation errors, while JoinGym always returns the true cardinality.

Reviewer comment (Reproducibility):
How do you estimate c_h (the cardinality of the IR incurred at time h)?

Author response:
**W2 and Q1:** We never estimate cardinalities since JoinGym can simulate *true* cardinalities derived from our novel dataset. Thus, it does not make sense to compare against existing cost models since JoinGym does not require a cost model. Again, this is a key advantage of JoinGym.

Reviewer comment (Experiments):
Significance: The conclusion of compared reinforcement learning approaches could be clearer in terms of which kind of predictions it enables. A non-learning baseline would also help to root the results.

Author response:
**W1 and Q1.** Regarding what can be learned about RL approaches in the benchmark, we mentioned the most salient points in the discussion part of Section 5. To add more details, we'll amend it with the following paragraph, comparing pros and cons of compared RL methods for the query optimization task:

To recap, we compared four RL algorithms spanning three categories: off-policy Q-learning (DQN), off-policy actor-critic (TD3 and SAC), and on-policy actor-critic (PPO). Although PPO does not always obtain the best test-set performance, we can see in Tables 5,6 that it reliably obtains decent quantile performance in all settings of JoinGym. Hence, PPO has the advantage of being the most stable and the disadvantage of often plateauing at a sub-optimal solution. On the other hand, off-policy actor-critic methods (TD3 and SAC) perform the best for mean and 
 quantile, although PPO's CCM is usually within an order of magnitude. Hence, off-policy actor-critic methods are less stable but can sometimes converge to a better solution. DQN and DDQN tend to be uniformly worse off, which suggests that actor critic outperforms Q-learning for this problem. With this said, all of these algorithms are optimizing for the mean of a highly varied, long-tailed distribution of returns; thus, designing better algorithms that can handle long-tailed rewards will likely improve performance across all the algorithms.
 
Regarding comparison with a non-learning approach: we have new results on how well Postgres, a strong non-learning baseline, performs on the JoinGym query suite. We ran the Postgres query optimizer on all 3300 queries of JoinGym and computed the join plans' cumulative cost multiples (CCMs). The results are shown below:
| Query Set | Mean | p90 | p95 | p99 |
| --------- | ---- | --- | --- | --- |
| Train     | 1.72e+06 | 2.60e+03 | 4.92e+04 | 4.52e+06
| Val       | 5.81e+05 | 4.46e+03 | 9.97e+04 | 7.10e+06
| Test      | 6.09e+04 | 5.00e+03 | 3.41e+04 | 2.31e+06

By examining Tables 5 and 6 of the paper, we can see that RL algorithms in the left-deep disable CP JoinGym often yields better results than Postgres. For example, PPO is consistently better for train, validation and test queries. This suggests that these RL algorithms are indeed proposing high-quality joins that are competitive with Postgres.

Reviewer comment (Novelty):
Originality: The work does not seem to (explicitly) propose any novel approach.

Author response:
**W2 and Q2.** Our contribution is a novel RL environment JoinGym and extensive benchmarking of existing RL algorithms. As a submission for the "dataset and benchmarks" primary area, our goal is to provide a RL environment to accelerate and facilitate future algorithm research.

Reviewer comment (Novelty):
Relevance: The focus of the paper seems to be less on reinforcement learning and more about a particular application of reinforcement learning in database management systems.

Author response:
**W3 and Q3.** We believe our work will be valuable to the RL community for the following reasons. First, as mentioned in the previous point, our new efficient environment will enable anyone with a laptop to rapidly prototype new RL ideas for the query optimization problem, without needing expensive and complex setup with real DB systems. Second, in Section 5 of the paper, we have identified three clear research directions for RL: (1) dealing with long-tailed returns, (2) generalization in discrete optimization, and (3) partial observability. Please also see our global response regarding this point.

Reviewer comment (Experiments):
What can be learned about the performance of different reinforcement learning approaches in this benchmark and how does it compare to the performance of a non-learning approach (as used in modern commercial systems)?

Author response:
**W1 and Q1.** Regarding what can be learned about RL approaches in the benchmark, we mentioned the most salient points in the discussion part of Section 5. To add more details, we'll amend it with the following paragraph, comparing pros and cons of compared RL methods for the query optimization task:

To recap, we compared four RL algorithms spanning three categories: off-policy Q-learning (DQN), off-policy actor-critic (TD3 and SAC), and on-policy actor-critic (PPO). Although PPO does not always obtain the best test-set performance, we can see in Tables 5,6 that it reliably obtains decent quantile performance in all settings of JoinGym. Hence, PPO has the advantage of being the most stable and the disadvantage of often plateauing at a sub-optimal solution. On the other hand, off-policy actor-critic methods (TD3 and SAC) perform the best for mean and 
 quantile, although PPO's CCM is usually within an order of magnitude. Hence, off-policy actor-critic methods are less stable but can sometimes converge to a better solution. DQN and DDQN tend to be uniformly worse off, which suggests that actor critic outperforms Q-learning for this problem. With this said, all of these algorithms are optimizing for the mean of a highly varied, long-tailed distribution of returns; thus, designing better algorithms that can handle long-tailed rewards will likely improve performance across all the algorithms.
 
Regarding comparison with a non-learning approach: we have new results on how well Postgres, a strong non-learning baseline, performs on the JoinGym query suite. We ran the Postgres query optimizer on all 3300 queries of JoinGym and computed the join plans' cumulative cost multiples (CCMs). The results are shown below:
| Query Set | Mean | p90 | p95 | p99 |
| --------- | ---- | --- | --- | --- |
| Train     | 1.72e+06 | 2.60e+03 | 4.92e+04 | 4.52e+06
| Val       | 5.81e+05 | 4.46e+03 | 9.97e+04 | 7.10e+06
| Test      | 6.09e+04 | 5.00e+03 | 3.41e+04 | 2.31e+06

By examining Tables 5 and 6 of the paper, we can see that RL algorithms in the left-deep disable CP JoinGym often yields better results than Postgres. For example, PPO is consistently better for train, validation and test queries. This suggests that these RL algorithms are indeed proposing high-quality joins that are competitive with Postgres.

Reviewer comment (Novelty):
Do the any of the results conclude any new approach or variation of an approach that has not been previously considered?

Author response:
**W2 and Q2.** Our contribution is a novel RL environment JoinGym and extensive benchmarking of existing RL algorithms. As a submission for the "dataset and benchmarks" primary area, our goal is to provide a RL environment to accelerate and facilitate future algorithm research.

Reviewer comment (Novelty):
What kind of progress does this work make in terms of reinforcement learning approaches (e.g., via trial and error) beyond measuring their empirical performance?

Author response:
**W3 and Q3.** We believe our work will be valuable to the RL community for the following reasons. First, as mentioned in the previous point, our new efficient environment will enable anyone with a laptop to rapidly prototype new RL ideas for the query optimization problem, without needing expensive and complex setup with real DB systems. Second, in Section 5 of the paper, we have identified three clear research directions for RL: (1) dealing with long-tailed returns, (2) generalization in discrete optimization, and (3) partial observability. Please also see our global response regarding this point.

Reviewer comment (Writing):
The title of the article is very strange and difficult to understand. The ‘query optimization’ is the background of this paper. The title may be “An efficient reinforcement learning environment for query optimization” instead of “An efficient query optimization environment for reinforcement learning”?

Author response:
**Q1.** Our title highlights that JoinGym is designed for the RL community, rather than to improve query optimization directly. Up until now, query optimization has not been easy for RL researchers to benchmark due to large overhead in setting up a real DBMS. Our work removes this barrier.  With JoinGym, we hope to motivate research from diverse ML/RL communities to create better algorithms for query optimization. Please also see our global response regarding this point.

Reviewer comment (Experiments):
Do the ‘simulator’ and ‘environment’ refer to the same thing? Authors mention that ‘our aim is to provide a lightweight yet realistic simulator’ and ‘looking up the size of intermediate tables from join sequences’. The size or called cardinality estimation is a popular research field in query optimization. They should compare their method to the existing query size simulators as follows.  
[1] Sun, J., & Li, G. (n.d.). An End-to-End Learning-based Cost Estimator. VLDB, 2020. Kipf [2] A, Kipf T, Radke B, et al. Learned cardinalities: Estimating correlated joins with deep learning. CIDR, 2019.  
[3] Yang, Z., Kamsetty, A., Luan, S., Liang, E., Duan, Y., Chen, X., & Stoica, I. (2020). Neurocard: One cardinality estimator for all tables. Proceedings of the VLDB Endowment, 14(1), 61–73, 2020.  
[4] Ziniu Wu, Parimarjan Negi, Mohammad Alizadeh, Tim Kraska, Samuel Madden. FactorJoin: A New Cardinality Estimation Framework for Join Queries. SIGMOD, 2023

Author response:
**Q2.** Yes, "simulator" is the RL environment. Thanks for bringing up prior works on cardinality estimation, which we will cite and discuss -- our work is orthogonal to [1-4] because we are using *true cardinalities* instead of estimates from cost models (which can have errors). Moreover, [1-4] do not provide a RL environment for query optimization, which is our focus and main contribution.