Paper: DIFUSCO-LNS: Diffusion-Guided Large Neighbourhood Search for Integer Linear Programming (ICLR-2024)

Reviewer comment (Evaluation):
The problem instances also seem to be effectively solved quite quickly with the primal gap quickly reaching below 10-3. It would help to explain what level of primal gap is reasonable for these instances to be considered solved.

Author response:
We compute the primal gap based on the optimal primal bound among all methods in our experiment, therefore, reaching $10^{-3}$ does not really mean the model reaches the optimal value. Also please see our general responses on this.

Reviewer comment (Theory):
The authors should consider some rephrasing to better situate their work within the context of optimization. In the 1st paragraph, last sentence ("Our work … belongs to the category of approximate solvers"), it seems that this work doesn’t give approximation guarantees and also doesn’t give indication of how close to optimality a solution is so instead should be considered a primal solver.

Author response:
Thanks for pointing this out. We have changed the phrase from approximate solvers to primal solvers.

Reviewer comment (Theory):
In the 3rd paragraph, it is unclear why the referenced LNS approaches require domain-expert knowledge. It seems that they all readily take the ILP formulation as input without much tuning (other than selecting the neighborhood size which is needed in this work as well). In the same paragraph, it is not obvious why a data-driven approach is necessarily a better alternative as it assumes access to a distribution of problem instances and requires offline training. Here it might be helpful to give a high-level explanation of why learning-based methods should work well.

Author response:
Thanks for the insightful comment. We have added a paragraph explaining that.

Reviewer comment (Theory):
It would be helpful to explain why local branching is something we desire to learn, does it give performance guarantees for LNS? Does it empirically work well but is just too slow?

Author response:
LB (local branching) can give a guaranteed optimal neighborhood for LNS if it can be solved optimally, but it is NP-complete and shares the same time complexity as the original problem. Therefore, it is usually not used when the primal heuristics are needed within a short time.

Reviewer comment (Evaluation):
Average rank seems to be computed using the summary statistics. However, it might be informative to include a metric measuring the average rank that averages over problem instances. This would help give an idea of whether a given algorithm was generally solving problems faster overall.

Author response:
This is a really helpful suggestion. We have made the corresponding change in Table 2 and 3. From the current average rank, CL-LNS looks stronger which is also consistent with the conclusion in the original paper.

Reviewer comment (Experiments):
In my opinion, the paper is lacking additional ablation studies or experiments that evaluate the influence of the different hyperparameters. The authors only conduct one experiment that analyzes the effect of the number of diffusion steps. They report results for the values 5, 10, 20, 50 and find that 50 steps lead to the best results. This raises the question if the performance of the approach can be further improved by selecting an even higher number of steps. Overall, the relationship between number of diffusion step, prediction time, and prediction quality could be explored further. Fig. 4, which shows the results for the existing experiments, is also a bit difficult to read. Furthermore, the influence of other hyperparameters (e.g., destroy size) could be evaluated further.

Author response:
The LNS-related hyperparameters are set to the same as used in the previous work (CL-LNS, Huang et al., 2023). For diffusion related parameters, please see Figure 5. Our main finding is that different types of problems have different best diffusion steps and diffusion schedules. For example, a linear schedule with 10 steps works best for MIS problems, while a cosine schedule with 2 steps works best for CA problems.

Reviewer comment (Presentation):
The quality of Fig. 1 and Fig. 2 could be improved. For example, by using the same font for all Figures. Both figures are also not mentioned or explained in the text. For Fig. 2 it is not clear what elements are added and why only x_1 and x_n are considered on the right hand side (and not x_2, x_3, …).

Author response:
Thanks for pointing this out! We have improved the quality of the figures.

Reviewer comment (Evaluation):
It would be interesting to see the comparison prediction accuracy / per-iteration improvement to confirm that difusco-LNS is indeed making better predictions.

Author response:
We verify that DIFUSCO-LNS learns a better destroy policy than CL-LNS so it brings a larger per-iteration improvement in the primal bound on average. We visualize the comparison in Figure 7. We plot the 10-iteration improvement on MVC-S, CA-S, and SC-S and the 100-iteration improvement on MIS-S since the LNS typically has a much larger number of iterations on MIS-S (greater than 500) than that on other datasets (less than 100). We can also clearly observe that given the same number of iterations, DIFUSCO-LNS achieves a better primal bound, which verifies our assumption that DIFUSCO-LNS learns a better destroy policy.

Reviewer comment (Reproducibility):
You mentioned that you were not able to reproduce results for some baselines due to differences in hardware/compute resources. Can you elaborate more on this? For LB-RELAX there seems to be quite a huge difference. From my own experience, sometimes it is due to different software versions (like SCIP or Gurobi). The other time it might be due to hardware differences: a slower machine computes different heuristics at the BnB nodes and thus produces different results if the wall-clock time budget is fixed.

Author response:
In our research, we diligently followed the methodologies outlined in the publicly available code from the previous study (found at https://github.com/facebookresearch/CL-LNS). However, during our replication efforts on our own hardware, we observed variations in the results. Specifically, the trend of the performance curve differed from the original findings. This discrepancy could be attributed to differences in hardware and computing resources between our setup and that of the original study.

Reviewer comment (Experiments):
Difusco-LNS also takes advantage of multiple good solutions following previous work. I wonder if a contrastive learning component can be built into the diffusion model so that you can leverage bad solutions from LB?

Author response:
Good question.  Although we currently have focused on improving the  generative probability of good solutions with diffusion, we do believe that contrastive learning [1], especially contrastive generative modeling [2], is an important future direction.  Thanks for this suggestion.

Reviewer comment (Experiments):
From the ablation studies, it seems that DIFUSCO-LNS is sensitive to a couple of hyperparameters. I wish to understand whether you need to fine-tune them for different benchmarks.

Author response:
The LNS-related hyperparameters are set to the same as used in the previous work (CL-LNS, Huang et al., 2023). For diffusion related parameters, please see Figure 5.