Paper: RL4CO: a Unified Reinforcement Learning for Combinatorial Optimization Library (ICLR-2024)

Reviewer comment (Presentation):
The title of this paper is RL4CO, but it only addresses routing-related problems. Would it be more appropriate to name it RL4Routing?

Author response:
RL4CO is not confined to routing problems only. In fact, Appendix B is about the decap placement problems from electronic design automation (EDA), which we did not include in the main text mainly because of space - given the practicality of the problem needed more background, we felt it would be better for EDA to be in a separate section of the paper. 

Our goal is to include a wide variety of problems with the help of our community. For instance, we recently introduced the library scheduling problems including the Single Machine Total Weighted Tardiness Problem (SMTWTP) and the Flexible Flow Shop Problem (FFSP). Moreover, RL4CO is easily extensible to other problems; for instance, a recent paper [1r] has extended our implementation to the max cover problem and facility location, which we plan to integrate in RL4CO.

Reviewer comment (Novelty):
While this paper is confined to the scope of RL4Routing, it implements various RL methods for these problems. However, at their core, these methods belong to the same category (and share very similar algorithm procedures), and there is a lack of insights to organize this category effectively. If this article could provide some useful conclusions or derive new methods based on these benchmarks, it would become novel.

Author response:
Thanks for your insight! We suppose by the same category you mean autoregressive constructive methods for combinatorial optimization, i.e., that directly construct a feasible solution given a problem instance node-by-node. In terms of software organization, our fundamental insight is to divide and conquer the problems (in terms of data, environment, model, training) into more manageable subproblems. For instance, while several practical implementations in the NCO community separate different problems into separate schemes - one can think about different folders each with its own problem to solve but with mostly redundant code, that make management and progressive development hard, especially for new researchers - we decided to rely on unified structures. In the case of data, this was e.g. thanks to TensorDict, that allows for batched operations on named tensors, greatly simplifying passing around instances. In the case of environment, thanks to TorchRL we created a blueprint for vectorized CO environments that can run on GPU thus greatly alleviating the typical CPU simulation bottleneck typical of RL. As for the models, given common structures in the encoding-decoding architecture, we managed to modularize the policies as described in Figure 3.2, with the main idea being to abstract away problem-specific modules from common structures as encoder and decoder layers. For the training, we use PyTorch Lightning to reorganize the flow from input to output (in training, from data to loss function to optimize the parameter). Once again, many of these procedures are common among different RL algorithms (e.g. in A2C and REINFORCE with different baselines, this means modularizing the baselines). In terms of research, we have identified several pitfalls that can be encountered when training CO solvers, such as out-of-distribution generalization, that could be alleviated by introducing several inductive biases or exploring different directions as highlited in the paragraph “Future Directions in RL4CO” in Section 4.3. Finally, we believe the first steps in deriving a new methods is to recognize the problems as we highlighted in our benchmarking sections, and importantly, having a solid software base from which to start experimenting.

Reviewer comment (Experiments):
In the case of TSP problems, many non-autoregressive (or non-RL) methods [1,2,3] have recently achieved better performance than RL-based methods, with advantages such as faster inference time, lower optimal gap, and the ability to run on larger instances (TSP1000 and TSP10000). Therefore, as a benchmark for routing problems (TSP, CVRP, etc.), the comparison of methods is not comprehensive enough.

Author response:
We agree with you. Non-autoregressive methods as the supervised DIFUSCO [1], the RL-based DIMES [2], and new sampling techniques as iSCO [3] can scale well to thousands of nodes.

However, we should note that non-autoregressive methods often suffer to poor generalizability to more constrained and practical problems - in many cases, these methods can only extend to NP-complete problems, which are relatively easier than NP-hard problems such as the vehicle routing problems. DIFUSCO [1] specifically restricts to NP-complete problems as TSP. In DIMES [2], it is also unclear whether it can be extended to problems with more complex constraints; in fact, the authors only mention possible extensions to other NP-complete problems. As for the sampling-based iSCO [3], quoting the authors, “The current EBM formulation relies on the penalty form of the original problem. When it becomes nontrivial to find even just a feasible solution, the generic formulation of iSCO would probably fail.”.

For the above reasons, we believe autoregressive approaches, such as the ones in the nowadays ubiquitous large language models, are more general and can deal with a wider variety of problems, that include, in combinatorial optimization, tasks such as routing, electronic design automation, scheduling, and ultimately more complex practical problems.

Reviewer comment (Novelty):
Though the authors claim that they aim to propose a unified framework, the methods considered in their paper are mainly based on AM and POMO, in other words, the auto-regressive methods. As far as I know, there are also other methods other than auto-regressive, such as Local-rewrite[1]. Instead of constructing the solution from scratch, methods like Local-rewrite aim to improve an existing solution. Are these kinds of methods able to integrate into the proposed RL4CO framework?

Author response:
You are correct. Indeed, RL algorithms for CO can be generally divided into 1) constructive and 2) improvement. 

Constructive methods usually generate the solution from scratch, which we focus on in the paper. They are generally faster, but the quality of solutions can be lower than that of improvement methods. In construction methods, inference techniques (such as greedy multistart, sampling and augmentation) focus on getting the best out of several generated solutions, while search methods such as EAS focus on improving the quality of generated solutions efficiently by updating the model at test time.

On the other hand, improvement methods (such as the papers [1,2]) learn to improve existing solutions. Usually obtain a better solution, but are slower and may be less general (i.e. specialized to a class of problems such as routing). The reason why in RL4CO we focus first on construction methods is that they are more generally applicable to a wide variety of problems, such as routing and electronic design automation, while most learned improvement approaches are ad-hoc for a single class of problems, such as integer programming [2r] or routing problems [1r].

Reviewer comment (Experiments):
The proposed unified framework mainly focuses on TSP/VRP and the graph-based CO problems. However, there are other kinds of CO such as bin-packing, job scheduling, and mixed integer programming. I wonder if the proposed RL4CO covers these CO problems.

Author response:
We agree with your opinion. We created RL4CO with not just routing problems in mind - for instance, as shown in Appendix B, we implemented electronic design automation. Our goal is to include a wide variety of problems with the help of our community. For instance, we recently introduced in the library scheduling problems including the Single Machine Total Weighted Tardiness Problem (SMTWTP) and the Flexible Flow Shop Problem (FFSP). Moreover, RL4CO is easily extensible to other problems; for instance, a recent paper [4r] has extended our implementation to the max cover problem and facility location, which we plan to integrate in RL4CO.

Reviewer comment (Experiments):
In the experiments, only three methods are considered as baselines. I think the authors should test more RL4CO methods as they claimed they propose a unified library.

Author response:
Thanks for your comment. Ideally, we would like to include as many baselines and problems as possible, and given resource constraints, we found ourselves with an optimization problem for which we decided to prioritize user experience, flexibility and easyness of future expansion. As we are constantly developing the library in parallel with other research projects, we expect it to grow to include much more in the near future - also thanks to our active community.

Reviewer comment (Experiments):
Still in the experiments, the size of TSP/VRP is only 50. I consider this size to be relatively small.

Author response:
We partially agree, in the sense that most papers in the NCO area pre-train models on 100 nodes. While pre-training on several hundred nodes and even more is possible, especially thanks to advancements we have included as FlashAttention, this would result in burdensome computational costs that would be prohibitive with our limited resources - for instance, Sym-NCO took the authors around 2 weeks to train on a single problem of 100 nodes.  We believe that pre-training on small instances and generalizing to larger ones is an important topic in NCO (for instance, some recent supervised approaches can effectively scale to several thousands of nodes such as [3r] - as such, it would be an interesting avenue of research to make RL scale better as well). Another approach is adaptation, as we did with AS and EAS (up to 1000 nodes in our experiments from the pre-trained models on 50 nodes).

Reviewer comment (Novelty):
This paper quite narrows the idea of RL algorithms for CO. There could be different ways to formulate the CO as an RL problem, not limited to the one outlined in Equation (1) - (3). For example, neighborhood search first constructs an initial feasible solution and optimizes the current solution at each step [1,2], DIMES predicts a solution directly and optimizes the model with REINFORCE (one-step RL) [3].

Author response:
You are correct. Indeed, RL algorithms for CO can be generally divided into 1) constructive and 2) improvement. 

Constructive methods usually generate the solution from scratch, which we focus on in the paper. They are generally faster but the quality of solutions can be lower than improvement methods. In construction methods, inference techniques (such as greedy multistart, sampling and augmentation) focus on getting the best out of several generated solutions, while search methods such as EAS focus on improving the quality of generated solutions efficiently by updating the model at test time.

On the other hand, improvement methods (such as the papers [1,2]) learn to improve existing solutions. Usually obtain a better solution, but are slower and may be less general (i.e. specialized to a class of problems such as routing). The reason why in RL4CO we focus first on construction methods is that they are more generally applicable to various CO problems, such as routing and electronic design automation, while most learned improvement approaches are ad-hoc for a single class of problems, such as integer programming [1] or routing problems [2r].

Reviewer comment (Experiments):
Most CO problems presented in the paper are routing problems, other classical CO problems like set covering, and maximum independent set are not covered.

Author response:
We agree with your insight, it would be useful to have such problems covered as well in the future and we believe it will not be hard thanks to our design choices. Our goal is to include a wide variety of problems with the help of our community. For instance, we recently introduced the library scheduling problems including the Single Machine Total Weighted Tardiness Problem (SMTWTP) and the Flexible Flow Shop Problem (FFSP). Moreover, RL4CO is easily extensible to other problems thanks to its flexibility; for instance, a recent paper [3r] has extended our implementation to the max cover problem and facility location, which we plan to integrate in our library.

Reviewer comment (Presentation):
I note that DIMES is listed as a supervised approach (footnote on page 1), which I think is incorrect. I'm wondering if the authors have any specific criterion for the method classification or it is just a mistake (relevant to the first point in weaknesses).

Author response:
About the mistake: yes, you are right, this is a typo on our side; thanks for spotting it! We have removed it from supervised approaches in the revised PDF.

About benchmarking DIMES: we agree with you; we would like to do this! However, we would like to remind that our method is more general regarding solvable problems, i.e., we can deal with more complex constraints. In DIMES, it unclear whether the approach can be extended to problems with more complex constraints; in fact, authors only mention possible extension to other NP-complete problems (i.e., quoting the authors, “extendable to NP-complete problems [1r]”), but not to NP-hard problems such as vehicle routing problems in general. This limits several practical applications of NCO, which involve several more complex constraints than the ones found MIS and TSP.