Paper: DER-Solomon: A Large Number of CVRPTW Instances Generated Based on the Solomon Benchmark Distribution (ICLR-2024)

Reviewer comment (Experiments):
The practicality is not clear. Is there any evidence to demonstrate that, with only 56 instances, the original Solomon benchmark is complex enough to test *all aspects* of (traditional or DL-based) algorithms? If a company trains a DL-based model on DER-Solomon, could we guarantee its reliability in practice?

Author response:
A1. Thank you for your insightful comments. It’s challenging to directly apply the model trained on the Solomon benchmark to real-world problems. We need to further collect real-world data and use our method to expand the collected data as training instances. Since most traditional algorithms demonstrate their performance on the Solomon benchmark, our work in this paper mainly constructs a dataset for training so that DRL algorithms can also show their inherent performance on the Solomon benchmark. This allows for a fair comparison between traditional algorithms and DRL algorithms, accelerating algorithm research in this field.

Reviewer comment (Experiments):
Besides Solomon, is the proposed method generalizable to approximate other datasets?

Author response:
A2. Thank you for your helpful comments. We have included additional expanded instances, validated through a comparison experiment on the Homberger benchmark, in the appendix of the updated manuscript. In the final version of the paper, we will add approximations for most of the public test benchmark in the VRP field. Due to time limits, experiments on them will be gradually released in the open-source code.

Reviewer comment (Experiments):
For the traditional algorithms, it would be better to add HGS. For the DL-based method, some recent studies [1, 2] demonstrate superior performance on CVRPTW, it would be better to benchmark them as well.

Author response:
A4. Thank you for your thoughtful comments. Both papers used HGS, thus, we believe it should be a very useful solver. However, we noticed that HGS only has CVRP code implementation, and we are unsure whether it can be applied to CVRPTW. Article [1] is a great work, and the key is that there is a source code implementation. The README file is written in great detail, and the author have put a lot of effort into it. Thank you very much for recommending this article. We are very willing to use this algorithm in our experiment. Article [2] is also a great work, but we do not know whether it has source code published. If no source code is published and the author has not published the experimental results of the algorithm on the Solomon benchmark, it is challenging to conduct an experiment on it.

Reviewer comment (Presentation):
The writing and presentation of this paper should be improved: All figures should use PDF format; the current version is blurry when zooming in.

Author response:
A6. Thank you for your helpful comments. In update manuscript, Figures 1 and 3 have been converted to PDF format in the updated manuscript. Figure 2 may be clear enough. The original Figure 4 has been deleted (as it duplicates the content of Table 1, and there is a 9-page limit). The new Figures 4 and 5 both have used high-resolution images (300dpi).

Reviewer comment (Presentation):
The writing and presentation of this paper should be improved: Better to provide some visualizations of the generated instances.

Author response:
A7. Thank you for your helpful comments. To better provide some visualizations of the generated instances, we have added a distribution function diagram in the appendix. Please let us know if there are any remaining questions!

Reviewer comment (Presentation):
The writing and presentation of this paper should be improved: The best result (e.g., in all tables) should be in bold for a better view.

Author response:
A8. Thank you for your helpful comments. In the updated manuscript, in Table 1, we have bolded the maximum Gap between DER-Solomon and Solomon. In Table 2, we have marked the shortest computation time for the DRL algorithm and the traditional algorithm, respectively. In Appendix Table 5, we have marked the optimal values for the DRL algorithm and the traditional algorithm, respectively.

Reviewer comment (Experiments):
Overall, this paper only focuses on *one dataset of a single problem (i.e., CVRPTW)*, and therefore the contribution may not be enough for ICLR.

Author response:
A9. Thank you for your encouraging comments and great efforts in helping improve our work. Indeed, the entire VRP field needs such work. If you look at the articles on DRL methods in the VRP field, you will find that they basically (to the best of our knowledge) test the performance of the algorithm with the author's custom dataset. In contrast, non-learning-based traditional methods (those algorithms that do not need to use other data for training) all use public test data to demonstrate algorithm performance. This has led to a lack of comparison between the two types of algorithms and a gap between the two fields. 
If we could expand the entire VRP field dataset, it would be very meaningful. 

However, many validation experiments need to be done, such as proving the similarity between the expanded data and the original data through the similarity of the solution results of the traditional algorithm on the original data and the expanded data. For example, use the expanded data to train the DRL method, and then compare it with the traditional method on the public test data, and show that using this method can indeed make the DRL method have a good performance on the public test data, so as to attract future DRL method authors to use this method to show the performance of their algorithm on the public test data, because I believe that if this method cannot make the DRL method perform well on the public test data, I am afraid that no author is willing to put their carefully developed algorithm in a not objective and fair scenario to deliberately show poor performance.

The workload of so many experiments on various VRP benchmark data is very huge, and we will proceed with the following work step by step: 

First, expand the Homberger benchmark dataset and conduct corresponding validation experiments on the expanded dataset. 

Then, reverse deduce the public test dataset of various VRP variants, expand it, and publish the expanded dataset.

Finally, conduct experimental verification on the expanded dataset.

Reviewer comment (Reproducibility):
Will you release the source code and datasets?

Author response:
A10. We will share our source code and datasets in the final version of our paper.

Reviewer comment (Writing):
The paper lacks background descriptions of the CVRP and CVRPTW problems and also probably some more detailed introductions on the existing traditional algorithm approaches. Given there is plenty of space for the paper, such background information could be very beneficial to the audience. VRP may be well-known in the community, but the variants are probably not.

Author response:
A1. Thank you for your helpful comments. To make the background descriptions clearer, we have added more background information on the CVRP and CVRPTW, as well as detailed introductions to existing traditional algorithm approaches in the updated manuscript.

Reviewer comment (Reproducibility):
A lot of details are not presented in the paper. E.g., given the estimated distributions of the parameters, how are the new instances sampled? Are all parameters considered independently? Why generate 1280k instances? Moreover, there are no given details on how the DRL model is trained using DER-Solomon.

Author response:
A2.  Thank you for your insightful comments. We sincerely apologize for any lack of detail in certain areas. Given the complexity of the subject matter, it can be challenging to cover every aspect in depth. To address this, we plan to release the source code in the final version of the paper. This will allow for a more comprehensive understanding as many details can be directly gleaned from the code.

- New instances are generated by random sampling from the estimated distributions.
- All parameters are considered independently. However, in Solomon's public test instance and another public test instance of CVRPTW, Homberger changes between instances, except for the time window variable, while other variables remain fixed. So, we also kept other variables fixed in the new instance, and only one variable changed. Whether other variables are independent of each other is irrelevant.
- The reason for generating 1280k instances is that most of the DRL algorithms solving VRP after 2019 are based on the transformer network and attention mechanism. The first author [1] to use this method set the training dataset to 1280k instances in his open-source code, so most subsequent researchers have followed suit.
- We didn’t mentioned in this paper how the DRL model is trained using DER-Solomon because the cited DRL algorithms, AM, MDAM, POMO, and MARDAM, all have source code released, and their articles also detailed their network structure and training methods. As for the training process, in short, the coordinates, demand, and time window data of DER-Solomon are input into the DRL network, and then the DRL network will plan a path that meets the time window and load constraints for each instance. The negative value of the average path length is used as the reward value, and the DRL network is trained using the gradient optimization algorithm.


[1] Kool, W, Van Hoof H, and Welling M. "Attention, Learn to Solve Routing Problems!." International Conference on Learning Representations. 2018.

Reviewer comment (Theory):
Curretly I think parameters of the dataset are assumed to be independently samples (correct me if that is wrong). Is there a reason to make such an assumption? Would it be beneficial to consider a more complex distribution of the parameters?

Author response:
A4. Thank you for your suggestion, we indeed did not consider this. In the Solomon public test instances, except for the time window variable is changing, other variables (e.g. coordinates, demand, etc.) are fixed, so we didn't think about whether the parameters were independently sampled. But considering that we will extend this method to the entire VRP field next, we can try to analyze the relationship between parameters, and then use the joint probability distribution to constrain non-independent parameters.

Reviewer comment (Experiments):
The studied problem has a relative restricted scope. Could such techniques explored in the paper get applied to solving other classic optimization problems as well? Or what special properties of the Solomon dataset makes the approach most effective?

Author response:
A5. Thank you for your insightful comments. Our method can be applied to optimization problems in various fields, not just the Solomon dataset, even when limited by a small number of public test instances. We have included additional expanded instances, validated through a comparison experiment on the Homberger benchmark, in the appendix of the updated manuscript. In the final version of the paper, we will add approximations for most of the public test data in the VRP field. Due to time limits, experiments on them will be gradually released in the open-source code.

Reviewer comment (Presentation):
Page 3: "its frequency histogram is shown in Figure 2(a)" but there is no index of (a) or (b) in Figure 2.

Author response:
A6. Thank you for your insightful comments. The details of the figures have been updated in the updated manuscript.

Reviewer comment (Experiments):
How many instances are included in the DER-Solomon benchmark?

Author response:
As briefly mentioned in Appendix A.3, we used 1,280,000 DER-Solomon instances as the training set. In addition, during testing, 1024 instances were used as the test set in Tables 1 and 2. However, it's important to note that the DER-Solomon benchmark is not a fixed set of instances. It is more of an instance generation method that provides a distribution function of instances, generating a large number of training instances through random sampling to help the DRL method test its performance on Solomon.

Reviewer comment (Experiments):
How many DER-Solomon instances are used to train the learning-based method?

Author response:
As briefly mentioned in Appendix A.3, we used 1,280,000 DER-Solomon instances as the training set.

Reviewer comment (Evaluation):
How to calculate the std gaps reported in Table 1? They seem to be unequal to the relative gaps between the std values of two benchmarks.

Author response:
We always set the denominator to be the mean. The calculation method of the gap is: 

std gap: abs (std on Solomon – std on DER-Solomon ) / mean on Solomon. 

mean gap: abs (mean on Solomon – mean on DER-Solomon ) / mean on Solomon.

And we have explained it in Table 1 in the updated manuscript.

Reviewer comment (Experiments):
The comparison with neural solvers is missed in Table 1.

Author response:
A1. The purpose of Table 1 is to compare the differences between DER-Solomon and Solomon, not the differences between different algorithms. To clarify the manuscript, we have added vertical lines between algorithms and have made annotations at the bottom of Table 1. In addition, the comparison between algorithms are recorded in appendix Table 5.

Reviewer comment (Presentation):
The figures are not well presented.

Author response:
A2. Thank you for your thoughtful comments. To make the figures present clear, we have described more content in legends of the figures and have made some essential explanations in the titles of the figures. We have utilized the PDF format for some Figures while rendering others in higher resolution for improved visual quality.