Paper: RL Simplex: Bringing Computational Efficiency in Linear Programming via Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
Unclear motivation. Generally, incorporating ML models to the Simplex task is intractable, as simplex in modern solvers is extremely fast (usually faster than $1ms$ for one iteration), while ML models usually require $10$x or even $100$x more time. Simplex iteration usually execute for thousands of times or even more in real-world applications, making the additional cost unacceptable. Previous research [1] based on MCTS claims that their approach can provide the best pivot labels for all kinds of supervised learning methods, but what is the motivation for this paper?

Author response:
W1: Thanks for the comment. To further clarify our motivation, we have included an additional paragraph in both the introduction and Section 2.2. Please review

Reviewer comment (Experiments):
Toy applications. LP simplex is widely used in modern solvers for general LP and MILP problems. However, this paper only focus on the TSP problem (with very small size), making this study impractical for real-world applications.

Author response:
W2: This is a valid point. However, we tried to follow the strategy used in the relevant literature, e.g. [3] that uses TSP as test problems. Our primary goal in the computational part was to compare our RL simplex method with the mainstream commercial solvers based on the simplex method, such as Gurobi and SciPy. This type of comparison is itself something that not many studies in the existing literature have done. Within the scope of 50 cities, solving instances within the specified iteration limit has become exceedingly challenging. 

In the latest version of our paper, we have added a section for article reviews. We have also compared our algorithm with several other recent articles on machine learning and the simplex method published in recent years. We found that the test set we considered far exceeds the size of the largest test set used in other articles (over 10x), including those mentioned in this comment. Overall, we believe that the RLSimplex algorithm can address problems that can be solved by the simplex method. Most importantly, research in this field is still in its early stages, and we plan to delve deeper into this direction in the future.

Reviewer comment (Experiments):
Lack of comparative baselines. Both [2] and [3] propose similar approaches in this task, what is the comparison between this approach and them? If time is not taken into consideration, then maybe the non-data-driven "strong branching" policy proposed in [2] can outperform some data-driven policies.

Author response:
W3: Thanks for the comment. In the new version of the paper, we have added some related articles and conducted a comparative analysis of our algorithm with others that employ simplex and machine learning approaches. It is worth noting that due to the unavailability of their source code, our comparisons are based solely on the results presented in the respective papers.
Regarding the strong branching issue, in general, this technique is primarily used to tackle MIP problems and has never been considered in the context of linear relaxation problems before. Furthermore, for large-scale problems, strong branching tends to impose significant computational demands. Additionally, it cannot rely on a rule similar to our algorithm's largest pivot rule to narrow down the selection space during the exploration process. This could be an interesting direction, but it requires further testing for comparison in the future.

Reviewer comment (Theory):
Reward design is too empirical. The reward design seems to be totally empirical. However, in RL, designing rewards in this way can sometimes result in unexpected agent behaviors. Maybe a reward that completely proportional to the number of iterations is more proper.

Author response:
W4: During the learning process, we expect RL Simplex to explore a wide range of possibilities (i.e., different actions).  Therefore, we have incorporated UCB techniques to ensure a balance between exploration and the pursuit of high rewards, avoiding a singular focus on high rewards at the expense of neglecting other potential actions.  Additionally, if introducing a proportion into the rewards, we believe it is necessary to pre-calculate or estimate the required number of iterations for instances within that range.  However, this poses a significant cost, especially for large-scale problems.  In practical terms, such estimations are also challenging.  Taking our tested 50-city TSP problem as an example, the choice of the same numerical value but different variables as pivots can result in a difference of over 4850+ iterations. Additionally, experiments in paper [1] have also demonstrated the success of this straightforward reward design approach.

Reviewer comment (Experiments):
Unfair comparison to modern solvers. The pricing rules in most modern LP solvers are designed to take iteration as fast as possible. Generally, rules like the steepest pivot rule are not even the one-step greedy rule. Thus, comparing the number of iterations with them is not so fair.

Author response:
W5: Based on the two articles provided by the reviewer, both studies [1][3] compare the steepest rule with other rules. From the results, it is evident that the steepest rule requires fewer iterations compared to other rules, such as our largest pivot rule. Furthermore, while we acknowledge that solver runtime is a crucial consideration, it is not explicitly stated here that solvers completely disregard iteration counts in their design, solely pursuing faster runtimes. Additionally, what we want to emphasize is that in other relevant literature[1-4], the comparison of solvers is typically based solely on the number of iterations. We understand that developers implement various computational improvements and different matrix decompositions based on the structure to reduce solution time and the number of iterations. We are not attempting to replace existing solvers in our paper; our focus is solely on the algorithmic aspect.

Reviewer comment (Experiments):
Missing experiments on dual simplex and on OOD data. Generally, dual simplex is more preferred by LP solvers as the default LP approach. Thus, experiment on dual simplex is also critical. Experiments on OOD data is also critical to test the generalization ability.

Author response:
W6: This is indeed an important direction. Due to time constraints, we plan to focus on the study of the Dual Simplex method in more detail in a different research. We had considered testing on some classic linear programming instance datasets, but our perspective aligns with that of [4], acknowledging that the existing data scope and scale are insufficient to meet our testing expectations compared to our standards.

Reviewer comment (Reproducibility):
The RL approach section (Section 3.1) is not well organized and expressed. The methods are mostly descriptive, lacking rigorous mathematical statements. This makes it somehow hard to follow every detail, especially when the reader wants to reproduce the method for future research.

Author response:
Thank you for your suggestion. We have provided more detailed descriptions for some definitions in Section 3.1, and made revisions in other sections to enhance the overall coherence of the article.

Reviewer comment (Experiments):
The method is only tested on the Euclidean TSP problem. It would be more convincing if more experiments on other LP problems can be conducted.

Author response:
Overall, we followed the approach used in the ML or RL-based simplex [1-4] and tested our method using standard TSP. We will continue to pursue research in this field and aim to expand the scope of the problem to other types of LP problems.

Reviewer comment (Reproducibility):
Regarding the action space and reward function design, does that means whenever there exists a positive non-basic variable in $s_{t+1}$, then the reward received is $-kt$ *regardless* of the specific action $a_t$ chosen from possible largest non-basic coefficients?

Author response:
Yes, because the simplex method requires verifying whether all non-basic variables are negative before proceeding to the next iteration. Because the algorithm has not yet satisfied the termination condition, the reward generated by the new iteration is -kt. One similar reward setting can be found in reference [1].

Reviewer comment (Novelty):
There is recently a large body of literature on RL-based (mixed) integer linear programming algorithms, which is also extensively cited in this paper. However, it seems that only a little is discussed about the literature on solving standard LP problems assisted with ML methods, which is the focus of this work. Can you provide more about this line of research? Also, there is no comparison with existing methods for ML-based LP algorithms. How is the RL simplex method compare with other ML-based algorithms?

Author response:
Thanks for the comment. We have included the relevant literature on machine learning and simplex algorithm-related research in our latest edition. We intended to compare our work with other relevant approaches;  however, we could not get access to the open-source code of these algorithms, making it challenging to provide detailed comparisons in the experimental results. We have attempted to contact the authors of the relevant papers, but unfortunately, we have not received any response.

Nevertheless, we can still conduct some preliminary comparisons by referencing dataset sizes or the use of similar modern solvers in other literature. For example, in the Simplex MCTS algorithm [1], the authors conducted a comparison with the SciPy solver as well. From Table 3 in the literature [1], it is evident that all problem ranges can be solved within 900 iterations. However, in our 50-city instances, utilizing the same SciPy Linprog Solver failed to resolve the problem within 5100 iterations. Therefore, we cannot ascertain the stability of Simplex MCTS on large-scale datasets.

Another comparison is with the DeepRL Simplex, sharing a similar problem context, namely the TSP. From the test results [3] for the 5-city TSP problem, we both used Dantzig (Largest Private Rule) as a benchmark. Although the results suggest that DeepSimplex performs slightly better than ours in the 5-city TSP problem, it is essential to emphasize that DeepSimplex overlooks the impact of different city intervals on the simplex method, particularly the influence of Dantzig Largest Private Rule. This conclusion can be drawn from our Table 1 - City Range Analysis. We noticed an enhancement in the performance of the Dantzig Largest Private Rule with larger city intervals. However, it's important to note that the results presented in Table 2 are derived from a range of 1-5.

Reviewer comment (Presentation):
All font sizes in Figure 1 are not uniform. Besides, what is the meaning of A, B and C in the Q-table.

Author response:
Thank you for your note. The updated Figure 1 has been incorporated into the new version of the paper. Regarding the elements A, b, c, these definitions are also explained in Section 2.1.

Reviewer comment (Theory):
There may be some mistakes in the acting phase part. As the definition says, the parameter cij denotes the distance or cost between each pair of cities on the tour. But the definition of at is a set of largest cij corresponding to the variables of the objective function, which are not actions.

Author response:
We have improved the explanation of actions in Section 3.1 of the article. We removed the corresponding expression but we provided clarifying descriptions.

Reviewer comment (Writing):
The paper contains some abbreviations without full names, such as LP problem.

Author response:
It has been corrected in the new version of the paper.

Reviewer comment (Writing):
The paper could be easier to follow, especially the description part of the key.

Author response:
We have provided further clarification on the key in Section 4.1, Learning and Acting Phase. We also streamlined the exposition to better reflect the description part of the key.

Reviewer comment (Presentation):
The content marked in red in the experimental results part of the paper needs to be correct. For instance, in the row of 50 cities in Table 2, the solution result of HiGHS solver is better than that of RL Simplex, but the solution result of RL Simplex is incorrectly marked in red. Besides, Table 1 does not draw the optimal results.

Author response:
We have now marked both the best-performing method and the second-best-performing method in Table 2.

Reviewer comment (Presentation):
Some formulas are missing numbers, such as those in the Action Space part, UCB part, and Q-Value update part.

Author response:
It has been corrected in the new version of the paper.

Reviewer comment (Experiments):
RL Simplex is particularly effective in improving the efficiency of solving linear programming problems in datasets with small sizes. However, a question arises regarding its performance when applied to larger datasets. Can you provide instances where traditional simplex methods fail to solve while RL Simplex successfully finds a solution? Such instances would serve as compelling evidence of RL Simplex's capabilities.

Author response:
From our current results and literature review, it appears that the TSP with 50 cities already poses significant computational complexity on solvers. In testing for large instances, we set a limit of 5100 iterations, far exceeding the tolerance for iteration counts observed by peers[1-4]. However, without such limits, existing solvers, including the SciPy solver and Gurobi, can reach iteration counts exceeding 8000 without convergence in some instances. It is crucial to emphasize that the HiGHS solver, unlike our other testing methods, does not solely rely on the simplex method to solve LP problems.  Hence, it exhibits advantages in handling large datasets.  We chose to include the HiGHS solver in the comparison to understand the extent of the difference between our algorithm and a solver specifically designed for large-scale data. This comparative approach is being applied for the first time in related research articles with a similar focus.

Reviewer comment (Reproducibility):
As in the paper, if the solver meets a key not in the Q-table, it will always follow the largest coefficient rule. But as I understand it, most of the keys should be previously unencountered, especially in the process of doing different instances. Please give further instructions on how to apply the Q-table obtained on the training set to the test set? Besides, is it necessary to train a Q-table for different city-size instances? If it is needed, what is the generalization of this solver?

Author response:
We suggest training on different city scales. This is because the number of variables and constraints vary significantly with different city sizes in the TSPs, as indicated in the leftmost column of Table 2. We can train TSPs with different city sizes together and place them in a trained Q-table. However, this would increase the time required for key matching during testing. 
Additionally, regarding how to apply the trained Q-table to new instances, traditional Q-learning learns from past experiences based on state. Still, in our algorithm, the definition of state is slightly different—please refer to Section 3.1. Therefore, we need to extract the Cij part of each iteration as a key to help us find past learning experiences from the trained Q-table.
It's worth noting that the simplex method is not a one-shot algorithm; we can consider each iteration as solving a new LP problem. Therefore, new instances can also gain experience from the solving process of previous instances. While training for large-scale TSP problems may require a significant number of iterations (exploring different actions), it proves beneficial for testing new instances.