Paper: Effective Generation of Feasible Solutions for Integer Programming via Guided Diffusion (ICLR-2024)

Reviewer comment (Experiments):
“For SCIP, we adopt the first solution obtained through non-trivial heuristic algorithms during the solving phase.” I don’t know whether this comparison is fair. Did you try, for example, using the solutions they get within a fixed window of time?

Author response:
**Fair comparison and Gurobi:** The main goal of this paper is to generate good and complete initial feasible solutions end-to-end using neural network approaches, which is different from both SCIP and Gurobi solves: they aim for producing optimal solutions using initial feasible solutions as a starting point by using the branch and bound algorithm within a fixed time window. Hence, we compare the heuristic solutions obtained from SCIP and solutions after completing from Neural Diving. To further ensure the comparison comprehensive, in the paper, we have added additional results of the best heuristic solutions from Gurobi, as well as the best heuristic solutions from the PS+Gurobi algorithm, as suggested by Reviewer wgcj. The results show that the complete solutions generated by our methods have comparable quality to the best heuristic solutions from Gurobi in the SC dataset and better objectives in the CF and IS datasets. The partial solutions produced from our methods, combined with the CompleteSol heuristic, further improve the quality of solutions beyond all baseline methods. For more details, please refer to table 1 and table 2 in Section 5 in the updated manuscript (or check Table 1 and Table 2 in our global responses).

Reviewer comment (Experiments):
Why do you compare your algorithm mostly with SCIP instead of Gurobi which is possibly a much better solver.

Author response:
**Fair comparison and Gurobi:** The main goal of this paper is to generate good and complete initial feasible solutions end-to-end using neural network approaches, which is different from both SCIP and Gurobi solves: they aim for producing optimal solutions using initial feasible solutions as a starting point by using the branch and bound algorithm within a fixed time window. Hence, we compare the heuristic solutions obtained from SCIP and solutions after completing from Neural Diving. To further ensure the comparison comprehensive, in the paper, we have added additional results of the best heuristic solutions from Gurobi, as well as the best heuristic solutions from the PS+Gurobi algorithm, as suggested by Reviewer wgcj. The results show that the complete solutions generated by our methods have comparable quality to the best heuristic solutions from Gurobi in the SC dataset and better objectives in the CF and IS datasets. The partial solutions produced from our methods, combined with the CompleteSol heuristic, further improve the quality of solutions beyond all baseline methods. For more details, please refer to table 1 and table 2 in Section 5 in the updated manuscript (or check Table 1 and Table 2 in our global responses).

Reviewer comment (Evaluation):
How does the objective value that you sampled compare to the optimal solution? How close are they? If they are far from each other, having a high feasible ratio does not mean anything. The feasible region increases exponentially, so there could be a large number of feasible solutions that are far from the optimal solution.

Author response:
Moreover, we have included qualitative analysis of the generatived solutions: we sampled 1000 solutions from a single instance of the SC and IS datasets and plotted the distribution of corresponding objectives, see Section 5.3 in the updated manuscript (or see [here](https://anonymous.4open.science/r/Guided_diffusion_for_IP-B4C0/SC_instance.pdf) for the distribution of SC instance and [here](https://anonymous.4open.science/r/Guided_diffusion_for_IP-B4C0/IS_instance.pdf) for the IS instance ). The majority of solutions (over 95%) from our methods have better quality than the Gurobi heuristic. Furthermore, when combining our methods with the CompleteSol heuristic,the distributions of solutions are even closer to the optimal values.

Reviewer comment (Novelty):
In “Related work”, you mentioned “our method aims to learn the latent structure …, without any reliance on the IP solver.”, but you still need to complete partial solutions using the Completesol heuristic from SCIP.

Author response:
* [Partial solution] No, our approach does not need CompleteSol to generate complete feasible solutions.

Reviewer comment (Experiments):
The authors may want to add [1] as a baseline.

Author response:
**More experiments:** We added more experimental results comparing our method with Gurobi and the baseline suggested by the reviewer. Please check the results in the Section 5 of the manuscript (or check Table 1 and Table 2 in our global responses). Results show that our methods consistently produce higher quality solutions than these two baselines.

Reviewer comment (Theory):
The prediction loss defined in Eq. (3) empirically performs better than that from general diffusion models. It would be better to provide some intuitive interpretation. Moreover, the authors may want to provide the inference algorithm of the modified diffusion model.

Author response:
**Q2: loss and inference algorithm**

The loss in Eq.(3) utilizes noisy embeddings to reconstruct the original embeddings as the objective. Intuitively, this helps enhance denoising capability in the neural networks, and facilitates the simultaneous training of the solution decoder via estimating $\mathbf{z} _{\mathbf{x}}$. Regarding the inference algorithm, as mentioned in Section 2, in the reverse process, the mean of $\mathbf{z} _{\mathbf{x}}^{(t-1)}$ can be approximated by adding $\mathbf{z} _{\mathbf{x}}^{(0)}$ as a condition
$$
  \mathbf{\mu} _{\theta}(\mathbf{z} _{\mathbf{x}}^{(t)},t) = \frac{\sqrt{\alpha _t}(1-\bar{\alpha} _{t-1})}{1-\bar{\alpha} _t}\mathbf{z} _{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha} _{t-1}}\beta _t}{1-\bar{\alpha} _t}\mathbf{z} _{\mathbf{x}}^{(0)} \tag{1}
$$
In paper [1], they use the following formula (2) from the forward process 
$$
    \mathbf{z} _{\mathbf{x}}^{(t)} = \sqrt{\bar{\alpha} _t} \mathbf{z} _{\mathbf{x}}^{(0)}+\sqrt{1-\bar{\alpha} _t} \mathbf{\epsilon} \tag{2}
$$
to replace $\mathbf{z} _{\mathbf{x}}^{(0)}$.  It implies that
$$ 
            \mathbf{\mu} _{\theta}(\mathbf{z} _{\mathbf{x}}^{(t)},t) = \frac{\sqrt{\alpha _t}(1-\bar{\alpha} _{t-1})}{1-\bar{\alpha} _t}\mathbf{z} _{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha} _{t-1}}\beta_t}{1-\bar{\alpha} _t}\mathbf{z} _{\mathbf{x}}^{(0)} 
            = \frac{\sqrt{\alpha _t}(1-\bar{\alpha} _{t-1})}{1-\bar{\alpha} _t}\mathbf{z} _{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha} _{t-1}}\beta _t}{(1-\bar{\alpha} _t)\sqrt{\bar{\alpha} _t}}(\mathbf{z} _{\mathbf{x}}^{(t)}- \sqrt{1-\bar{\alpha} _t} \mathbf{\epsilon}) 
            = \frac{1}{\sqrt{\alpha _t}} (\mathbf{z} _{\mathbf{x}}^{(t)}- \frac{1-\alpha _t}{\sqrt{1-\bar{\alpha} _t}} \mathbf{\epsilon)}. 
$$
The last equation holds because $\alpha _t:=1-\beta _t$ and $\bar{\alpha} := \prod _{s=1}^t \alpha _s$.  Therefore, we obtain the original sampling method of DDPM (Ho et al., 2020).  In our work, since we use the neural network to predict $\mathbf{z} _{\mathbf{x}}^{(0)}$,  we can directly use Eq.(1) to estimate the mean of $\mathbf{z} _{\mathbf{x}}^{(t-1)}$ in sampling phase as shown in Appendix A.4 in the updated manuscript. In the experiments, we use the same variance estimation as DDPM (see Eq.(7) in Ho et al. (2020)). Based on Eq.(2), we also produce the estimation of $\mathbf{\epsilon}$ by using $\mathbf{z} _{\mathbf{x}}^{(t)}$ and predicted $\mathbf{z} _{\mathbf{x}}^{(0)}$ and use it in the inference phase for DDIM.

[1]. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.

Reviewer comment (Evaluation):
As diffusion generative models may suffer from inefficiency in both training and inference, the authors may want to report the training and inference time.

Author response:
We also report toal training time and total sampling time for 3000 solutions on a workstation equipped with two Intel(R) Xeon(R) Platinum 8163 CPUs @ 2.50GHz, 176GB RAM, and two Nvidia V100 GPUs. During the inference phase, IP Guided DDIM exhibits faster performance than IP Guided DDPM, with average time of 0.46s-1.68s for sampling each solution (see Appendix A.8 or Table 3 in global responses for more detail).

Reviewer comment (Novelty):
Is this work the first one to generate complete solutions?

Author response:
**Q1**:
Yes, to the best of our knowledge, our approach is the first that generates complete and feasible solutions using pure neural techniques, without relying on any solvers.

Reviewer comment (Theory):
See Weakness 2. The training loss defined in Eq. (3) is different from general diffusion models. Does it cause a different inference algorithm?

Author response:
**Q2: loss and inference algorithm**

The loss in Eq.(3) utilizes noisy embeddings to reconstruct the original embeddings as the objective. Intuitively, this helps enhance denoising capability in the neural networks, and facilitates the simultaneous training of the solution decoder via estimating $\mathbf{z} _{\mathbf{x}}$. Regarding the inference algorithm, as mentioned in Section 2, in the reverse process, the mean of $\mathbf{z} _{\mathbf{x}}^{(t-1)}$ can be approximated by adding $\mathbf{z} _{\mathbf{x}}^{(0)}$ as a condition
$$
  \mathbf{\mu} _{\theta}(\mathbf{z} _{\mathbf{x}}^{(t)},t) = \frac{\sqrt{\alpha _t}(1-\bar{\alpha} _{t-1})}{1-\bar{\alpha} _t}\mathbf{z} _{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha} _{t-1}}\beta _t}{1-\bar{\alpha} _t}\mathbf{z} _{\mathbf{x}}^{(0)} \tag{1}
$$
In paper [1], they use the following formula (2) from the forward process 
$$
    \mathbf{z} _{\mathbf{x}}^{(t)} = \sqrt{\bar{\alpha} _t} \mathbf{z} _{\mathbf{x}}^{(0)}+\sqrt{1-\bar{\alpha} _t} \mathbf{\epsilon} \tag{2}
$$
to replace $\mathbf{z} _{\mathbf{x}}^{(0)}$.  It implies that
$$ 
            \mathbf{\mu} _{\theta}(\mathbf{z} _{\mathbf{x}}^{(t)},t) = \frac{\sqrt{\alpha _t}(1-\bar{\alpha} _{t-1})}{1-\bar{\alpha} _t}\mathbf{z} _{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha} _{t-1}}\beta_t}{1-\bar{\alpha} _t}\mathbf{z} _{\mathbf{x}}^{(0)} 
            = \frac{\sqrt{\alpha _t}(1-\bar{\alpha} _{t-1})}{1-\bar{\alpha} _t}\mathbf{z} _{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha} _{t-1}}\beta _t}{(1-\bar{\alpha} _t)\sqrt{\bar{\alpha} _t}}(\mathbf{z} _{\mathbf{x}}^{(t)}- \sqrt{1-\bar{\alpha} _t} \mathbf{\epsilon}) 
            = \frac{1}{\sqrt{\alpha _t}} (\mathbf{z} _{\mathbf{x}}^{(t)}- \frac{1-\alpha _t}{\sqrt{1-\bar{\alpha} _t}} \mathbf{\epsilon)}. 
$$
The last equation holds because $\alpha _t:=1-\beta _t$ and $\bar{\alpha} := \prod _{s=1}^t \alpha _s$.  Therefore, we obtain the original sampling method of DDPM (Ho et al., 2020).  In our work, since we use the neural network to predict $\mathbf{z} _{\mathbf{x}}^{(0)}$,  we can directly use Eq.(1) to estimate the mean of $\mathbf{z} _{\mathbf{x}}^{(t-1)}$ in sampling phase as shown in Appendix A.4 in the updated manuscript. In the experiments, we use the same variance estimation as DDPM (see Eq.(7) in Ho et al. (2020)). Based on Eq.(2), we also produce the estimation of $\mathbf{\epsilon}$ by using $\mathbf{z} _{\mathbf{x}}^{(t)}$ and predicted $\mathbf{z} _{\mathbf{x}}^{(0)}$ and use it in the inference phase for DDIM.

[1]. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.

Reviewer comment (Experiments):
My first concern is the insufficient comparison in experiments. As described in related work, considerable literature attempted to improve the diving method in solvers. Except Neural Diving (Nair et al., 2020), many follow-up works continue similar research topics. More recent methods should be compared. Even by only comparing Neural Diving, the results are not enough.

Author response:
**First concern and Q1: More empirical comparisons.** We added more experimental results comparing our method with Gurobi and a more recent baseline method (Han et al., 2023) suggested by Reviewer wgcj, which shows that our methods consistently produce higher quality solutions than these two baselines. Please check Table 1 and Table 2 in Section 5 for more detail (or check Table 1 and Table 2 in our global response).

Reviewer comment (Reproducibility):
The training time and resource usage are not clear, which is important to show practicality and efficiency of applying multiple deep neural networks in the proposed method.

Author response:
**Other concerns and Q4: ablations and training time.** Further, we ablated on the contrastive learning, i.e., with and without contrastive learning for the embeddings. Specifically, we include an ablation experiment in which we train IP and solution embeddings directly via the training produce of diffusion and decoder (Algorithm 2 in appendix A.3 in the updated manuscript) without using CISP, the results were updated in the Appendix A.9 of the updated manuscript (or check Table 4 in the global responses ).

The results show that CISP plays a crucial role in ensuring that the solutions produced by our methods are more feasible. We found that the advantage of contrastive learning is its ability to extract meaningful representations for IP instances and solutions, which is achieved by the assumption that instances should stay close to their feasible solutions and away from their infeasible ones. This can further integrate features from different forms, as the instance is represented using a bipartite graph and the solution is represented in a vector space. In contrast, this cannot be simply achieved by supervised learning.

As for the training time and computation usage, all our evaluations were conducted on a workstation equipped with two Intel(R) Xeon(R) Platinum 8163 CPUs @ 2.50GHz, 176GB RAM, and two Nvidia V100 GPUs. We have provided the total training time and the total inference time for generating 3000 solutions below (also see appendix A.8 in the updated manuscript). During the inference phase, IP Guided DDIM exhibits faster performance than IP Guided DDPM, with average time of 0.46s-1.68s for sampling each solution (see Appendix A.8 or Table 3 in global responses for more detail).

Reviewer comment (Experiments):
The functions of contrastive model and generative model are not showcased by ablation study.

Author response:
**Other concerns and Q4: ablations and training time.** Further, we ablated on the contrastive learning, i.e., with and without contrastive learning for the embeddings. Specifically, we include an ablation experiment in which we train IP and solution embeddings directly via the training produce of diffusion and decoder (Algorithm 2 in appendix A.3 in the updated manuscript) without using CISP, the results were updated in the Appendix A.9 of the updated manuscript (or check Table 4 in the global responses ).

The results show that CISP plays a crucial role in ensuring that the solutions produced by our methods are more feasible. We found that the advantage of contrastive learning is its ability to extract meaningful representations for IP instances and solutions, which is achieved by the assumption that instances should stay close to their feasible solutions and away from their infeasible ones. This can further integrate features from different forms, as the instance is represented using a bipartite graph and the solution is represented in a vector space. In contrast, this cannot be simply achieved by supervised learning.

Reviewer comment (Novelty):
Many works apply deep learning methods to solve integer programming problems with totally feasible solutions. To name a few, "A general large neighborhood search framework for solving integer linear programs", "Learning large neighborhood search policy for integer programming", "Mip-gnn: A data-driven framework for guiding combinatorial solvers". The advantage of this research over this line of works is not clear.

Author response:
**Other related works.**
We thank the reviewer for sharing many papers. However, it appears that there may have been some misunderstanding regarding the relationship between our work and other papers. Therefore, we would like to provide further context in order to clarify our position and help the reviewer better understand our contribution.

It is true that numerous works have been undertaken to apply deep learning methods to solve Integer Programming problems. But the underlying ideas are quite different to each other. Particularly, Bengio et al. (2021) categorize existing methods into three main groups (see Appendix A.12 in our update manuscript for a detailed discussion):

* Group 1 End-to-end learning, which refers to the training of a machine learning model to directly generate solutions based on input instances. In the context of solving integer programming (IP) problems, this involves learning to construct solutions, as demonstrated by methods like Neural Diving (Nair et al.,2020; Taehyun Yoon, 2022) and Predict-Search framework (Han et al., 2023). However, **these methods still need to rely on a solver to generate complete solution**. Another line in end-to-end learning focuses on learning to improve solutions, i.e., neighborhood search techniques (Sonnerat et al., 2021; Wu et al., 2021). **It is noteworthy that these methods typically require a solver to acquire an initial solution**. 

* Group 2 Complex optimization algorithms usually have a set of hyper-parameters left constant during optimization. This area use machine learning to select the values of hyper-parameters.

* Group 3 Learning alongside optimization:  This field focuses on developing existed CO algorithms, typically the branch-and-bound framework, that continuously utilize a machine learning model throughout their execution, including techniques such as learning to branch (Gasse et al., 2019) and learning to node selection (Khalil et al., 2022). These works aim to generate high-quality solutions by combining ML method with the branch-and-bound framework in solvers. 

Our approach falls into learning to construct solutions of Group 1. Importantly, to the best of our knowledge, our approach is the first that generates complete and feasible solutions using pure neural techniques, without relying on any solvers. 

The papers mentioned by the review fall into other groups and study different research questions. Specifically, the papers (Sonnerat et al., 2021; Wu et al., 2021) falls into learning to improve solutions of Group 1 since they focus on learning methods for enhancing solution quality using neighborhood search techniques, not directly related to solution generation. The papers Khalil et al., (2022) falls more into Group 3 and primarily focuses on node selection and also proposes the policy of producing a partial solution through a prescribed rounding threshold, which is then completed using SCIP, similar to Nair et al. (2020) and Taehyun Yoon (2022).

Reviewer comment (Evaluation):
GCN is described by "It does not explicitly incorporate objective and constraint information during sampling, often resulting in infeasible complete solutions." In Gasse et al. (2019), GCN always gains feasible solutions.

Author response:
**Q2.**
We think there is misunderstanding of some parts of our work possibly due to some misleading discussions in our paper. We agree with the reviewer that Gasse et al. (2019) proposed using a bipartite graph structure to model an IP instance and applied GCN to extract variable representations. But their primary focus was on learning the branching policy, i.e., selecting the variable for partitioning the node's search space. Follow-up studies, such as Nair et al. (2020), Yoon Taehyun (2022) and Han et al. (2023) confirm that directly using the bipartite graph and GCN to learn solutions for an IP instance may result in infeasible solutions, and thus extended this method by using partial assignments and CompleteSol heuristic to obtain complete feasible solutions. We clarified this in the revised manuscript in the paragraph 2 of Section 1.

Reviewer comment (Experiments):
Any integer programming problem can be converted into a 0-1 programming. But the conversion increases the number of constraints a lot. How large integer programming can the method solve?

Author response:
**Q3.**
We agree with the review that the conversion increases the number of constraints. But the point we tried to make is that our approach is generic enough to solve more general integer programming problems. In fact, with the aforementioned computation space, our approach can solve sizable IP problems: the largest instance (CF dataset) solved by our approach has about 5000 variables and 5000 constraints. These sizes can be further improved with more computation resources and more pre-training on typical small-size IP instances.

Reviewer comment (Experiments):
What is the advantage of contrastive learning compared to supervised learning? Additional experiment should be provided to see the effect of contrastive learning without labeled solutions.

Author response:
**Other concerns and Q4: ablations and training time.** Further, we ablated on the contrastive learning, i.e., with and without contrastive learning for the embeddings. Specifically, we include an ablation experiment in which we train IP and solution embeddings directly via the training produce of diffusion and decoder (Algorithm 2 in appendix A.3 in the updated manuscript) without using CISP, the results were updated in the Appendix A.9 of the updated manuscript (or check Table 4 in the global responses ).

The results show that CISP plays a crucial role in ensuring that the solutions produced by our methods are more feasible. We found that the advantage of contrastive learning is its ability to extract meaningful representations for IP instances and solutions, which is achieved by the assumption that instances should stay close to their feasible solutions and away from their infeasible ones. This can further integrate features from different forms, as the instance is represented using a bipartite graph and the solution is represented in a vector space. In contrast, this cannot be simply achieved by supervised learning.

Reviewer comment (Evaluation):
To better validate the quality of the proposed solver, comparison between the found optimal objective value and the ground-truth (global optimum) would be more convincing, as the paper only provided relative comparison between the proposed solver and two baseline approaches.

Author response:
Thank you for your suggestion! We have employed Gurobi to solve each dataset instance for 100 seconds, aiming to obtain the best possible solutions as optimal values. In the updated manuscript, we have included the average optimal values for each dataset in Table 1 and Table 2 in Section 5 (we also reported these results in Table 1 and Table 2 in the global responses). The experimental results demonstrate that our feasible solutions achieve a gap of 7% to 34% compared to the optimal values. This performance is superior to the heuristic methods from Gurobi and SCIP.