Paper: Towards Imitation Learning to Branch for MIP: A Hybrid Reinforcement Learning based Sample Augmentation Approach (ICLR-2024)

Reviewer comment (Novelty):
Concerns about the motivation. Personally, I am not convinced about the motivation to introducing RL in the branching task. As reported in some previous work [1,2], the number of expanding nodes of FSB is significantly less than that of SOTA ML approaches, indicating that *FSB is a good enough expert policy to imitate*. Instead, the bottleneck in this topic may mainly lie in the low IL accuracy. The lower accuracy may due to the unsatisfactory model structure or the insufficient information in the widely-used bipartite graph states (observations, more precisely). However, currently I found no definitive answer about that in recent researches. The ablation study in a recent research [3] gives clues that the historical information is effective as the process is a POMDP. Thus, in my personal opinion, the latter reason seems closer to the truth. Thus, based on these results, I think all researches that proposing more complex online RL framework is somewhat incremental, as improving the IL accuracy seems to be the key.

Author response:
>***Q1:*** Concerns about the motivation. I think all research that proposes a more complex online RL framework is somewhat incremental, as improving the IL accuracy seems to be the key.

Although Strong Branching is often used for imitation learning and the number of expanding nodes of FSB is significantly less than that of SOTA ML approaches. But it is a rule-based method, which means it tests which of the fractional candidates gives the best progress in the dual bound before actually branching on any of them, which can be viewed as finding the locally best variable to branch on [1]. This means the quality of variable selection in Strong branching can be further improved. So, our method aims to further improve the quality of training samples for imitation learning and jump out from the locally best variable selection strategy of expert rules. As you mentioned, improving the IL accuracy means the exploitation is much higher, but exploration is also important for imitation learning, so how to balance exploitation and exploration also can improve the performance of imitation learning, because in the previous research [2] it has been proven that adding past good experiences can add exploration for imitation learning and may find a potentially better policy than only imitating the expert policy.

Reviewer comment (Experiments):
Concerns about the unnecessary complexity for introducing RL. Though usually higher asymptotic performance, many RL approaches are usually sensitive to the hyperparameters, making their application requiring much manual tunings. I am doing research in both RL and CO, empirically, I found their combinations can be fragile. In this paper, the online RL is mainly used for data collection. However, as far as I know, RL approaches are usually sensitive to the data distribution due to the deadly triad. For example, TD3 [4] may fail in MuJoCo tasks when the initial 20k random steps of data collection is turned off; BCQ [5] may fail when the offline data are collected with hybrid policies. Thus, in real-world CO tasks, I prefer use simple GNN+IL approach rather than other complex but fragile approaches, even if they claim higher performance in the four synthetic benchmarks. For this paper, I am concerned about the complexity for introducing RL.

Author response:
>***Q2:*** Concerns about the unnecessary complexity of introducing RL and the online updating of the policy during data collection.

To further enhance the efficiency of imitation learning, our approach focuses on the decision of whether to include past good experiences generated by the previously learned policy in the training samples. To achieve this, we employ an online RL agent in each variable selection to make this decision. However, as the past-learned policy iteratively improves, it is crucial to periodically update the online RL agent to ensure more accurate decision-making. Alongside the online RL agent, we also train an offline RL agent as part of our methodology. The offline RL agent serves two purposes: improving the quality of training samples and reducing training time for imitation learning. By leveraging the offline RL agent, we can filter and select higher-quality training samples, leading to improved learning performance.

In terms of experimental results, our method consistently outperforms state-of-the-art learning-based models in synthetic benchmarks, as well as in three ML4CO datasets, as demonstrated in Table 6. Moreover, our approach significantly reduces the training time compared to ML4CO-KIDA, as shown in Table 7 and Table 8. Overall, our approach combines the use of an online RL agent for decision-making and an offline RL agent for sample filtering. Through these enhancements, our method achieves superior branching decisions. Additionally, our approach effectively reduces training time compared to the ML4CO-KIDA method.

Reviewer comment (Evaluation):
More explanations is required. In this paper, the RL based approach achieves lower IL accuracy while higher e2e performance. Thus, I am interested in what kind of multi-step information it learns from the input states. I believe this is more meaningful than simply reporting performance improvements, as it can guide us in designing better input features as mentioned in Point 1. If only the approach is given but the explanations is missed, then this paper is more like just a application of existing RL approaches to a new "Atari".

Author response:
>***Q3:*** More explanations about the RL-based approach achieving lower IL accuracy while higher e2e performance is required.

As motivation mentioned in Q1, our method focuses on balancing exploitation and exploration to improve imitation learning. The biggest difference between our method and GCNN, and ML4CO-KIDA methods is the training samples, which are combined with input features and labels. For the input feature, all the methods use the bipartite graph feature. For the label, our method's use of the online RL decides a better label in each variable selection, so, our method can generate high-quality training samples rather than only collecting samples from experts. Besides, the offline RL agent also filters higher cumulative reward samples generated from the online RL phase, which not only improves the quality of training samples but also reduces training time for imitation learning. with the iteration, the training sample and learned branching policy will be further optimized. Interestingly, Table 4, Table 5, and Table 6 show that the accuracy of the different methods, namely our method, GCNN, and ML4CO-KIDA, does not correlate with their overall performance.

Reviewer comment (Novelty):
Limited improvement compared to ML4CO-KIDA. The performance of HRL-Aug reported in Table 7 seems to be marginal to ML4CO-KIDA.

Author response:
>***Q4:*** Limited improvement compared to ML4CO-KIDA.

As shown in Table 5, as the difficulty of the problem increased, in the easy and medium difficulty, the improvement over ML4CO-KIDA was marginal. But in the hard difficulty, especially in the Maximum Independent Set problem, the improvement becomes more obvious. So, we further evaluate our method in the ML4CO datasets in Table 6, our method's reward dominates the ML4CO-KIDA in all three datasets, especially in anonymous problems, which are inspired by real-world applications of large-scale problems.

Reviewer comment (Presentation):
The primary weakness in the current draft is the presentation. The ordering of the Tables in the Experiments section is confusing — tables are introduced and described in a non-sequential order, and there are typos and odd sentences in the text that make the descriptions hard to follow at times. For example, Table 3 and 4 show the Ablation results of comparing HRL-Aug to ML4CO-KIDA while the main results are presented later in Table 7 and 8. Just re-ordering the Table sequences would make it much easier to read.

Author response:
>***Q1:*** The ordering of the Tables in the Experiments section is confusing and there are typos and odd sentences in the text.

To make the readers follow the descriptions effectively, we have addressed these issues in the updated PDF file. We assure you that we will provide clearer explanations in the final version of the paper.

Reviewer comment (Writing):
I find the writing on how the proposed method works is a bit confusing, this might be partly due to the complexity of the problem. For example, what exactly actor critic method did you use in the online setting (for example, A type of SAC? DQN?)? And how exactly does the online RL agent decide whether to use the expert or to use the learned policy? (if for example, you say the action space is discrete, with 2 actions, one to choose the expert and the other to choose the learned policy for the online RL agent and it is a Q-learning type of agent, then it becomes much clearer) Currently I don't fully follow what is happening here. (also see questions)

Author response:
>***Q1:*** The explanation of how the online agent works and how exactly does the online agent decide whether to use the learned policy or expert policy?

We really appreciate your valuable suggestion. Our methods use the advantage actor-critic (A2C) to train the online RL agent, which combines the Actor-Critic method and the concept of advantage function. The Actor is responsible for selecting an action (the action space is discrete, with 2 actions, one to choose the expert and the other to choose the learned policy) based on the current state. At the same time, the Critic evaluates the value of this action. The advantage function represents the advantage of an action relative to other actions. It can be used to evaluate the value of an action. Then use the cumulative reward to measure the quality of the agent's behavior in a given state. So, this algorithm can better estimate the value of actions and update parameters based on these values. When the online RL agent is trained, as shown in Figure 1, in each variable selection, input the state $s_t^O$ to the online RL agent, which will output an action 0 or 1, representing collect the sample generated from past-learned policy or expert policy. It is expected to make better decision-making in each variable selection, generate higher-quality training samples, and achieve effective exploration for imitation learning. This way, the sample generation process can be balanced between exploration and exploitation.

Reviewer comment (Novelty):
Another concern is the proposed method seems to only apply to only the MIP problems, and although the empirical results are interesting, it is a little unclear to me how much technical novelty is in the design of this framework and whether the contributions in this paper is significant enough.

Author response:
>***Q2:*** It is a little unclear to me how much technical novelty is in the design of this framework.

In Appendix A.4, we further discuss the Novelty of Our Data Augmentation Framework for Imitation Learning. Specifically, our framework not only proposed a new approach to enhancing the effectiveness of imitation learning, which is to better balance the exploration and exploitation of imitation learning but also improved the training efficiency of imitation learning, overall, this is a novel data augmentation framework and is technically nontrivial.

We also have undertaken comprehensive tests of our proposed framework. Extensive experiments on different MIP problems show the effectiveness of our method compared with open-source solver SCIP and other competing baselines, even achieving superior performance over the leading commercial solver in some cases. In addition, our method shows superior generalization ability.

Reviewer comment (Evaluation):
How time consuming is strong branching compared to your method? Can you provide a wall-clock time comparison?

Author response:
>***Q3:*** How time-consuming is strong branching compared to your method?

The strong branch is often used for imitation learning because it is a globally effective yet computationally expensive rule. At the beginning of our experiment, we also compared with the strong branch rule, as shown in the table below, we tested the result in the combinatorial auction problem with easy difficulty. However, as the experiment progressed, we found the strong branch was too slow, and it seriously affected the efficiency of testing and its solving time was significantly more than other methods. So we removed the strong branch in our experiment.

| Method | Time             | 
| ------ | ---------------- | 
| SCIP   | 2.42 $\pm$ 11.25% |
| GCNN       | 1.75 $\pm$ 9.73%     |
| ML4CO-KIDA | 1.70 $\pm$ 9.60%     |
| HRL-Aug    | **1.68** $\pm$ 9.44% |
| Strong Branching | 17.11 $\pm$ 20.34% |

Reviewer comment (Reproducibility):
It is a bit unclear to me how exactly does the online agent decide whether to use the learned policy or expert policy?

Author response:
>***Q1:*** The explanation of how the online agent works and how exactly does the online agent decide whether to use the learned policy or expert policy?

We really appreciate your valuable suggestion. Our methods use the advantage actor-critic (A2C) to train the online RL agent, which combines the Actor-Critic method and the concept of advantage function. The Actor is responsible for selecting an action (the action space is discrete, with 2 actions, one to choose the expert and the other to choose the learned policy) based on the current state. At the same time, the Critic evaluates the value of this action. The advantage function represents the advantage of an action relative to other actions. It can be used to evaluate the value of an action. Then use the cumulative reward to measure the quality of the agent's behavior in a given state. So, this algorithm can better estimate the value of actions and update parameters based on these values. When the online RL agent is trained, as shown in Figure 1, in each variable selection, input the state $s_t^O$ to the online RL agent, which will output an action 0 or 1, representing collect the sample generated from past-learned policy or expert policy. It is expected to make better decision-making in each variable selection, generate higher-quality training samples, and achieve effective exploration for imitation learning. This way, the sample generation process can be balanced between exploration and exploitation.

Reviewer comment (Writing):
Page 5, section 3.3, "batch DRL" is a general concept and is essentially the same as "offline DRL", which is initially discussed in some earlier papers (e.g. "Off-policy deep reinforcement learning without exploration" by Fujimoto et al.). The method in Chen et al., 2020 (Best Action Imitation Learning) is one of the imitation learning methods to tackle the batch/offline DRL problem. You might want to change the writing here to be more accurate.

Author response:
>***Q4:*** "Offline DRL", which was initially discussed in some earlier papers, You might want to change the writing here to be more accurate.

We really appreciate your valuable suggestion. The "offline DRL" is initially discussed in Fujimoto et al. (2018) [1], we will clarify this in the final version.

Reviewer comment (Reproducibility):
What is the "ActorCritic" algorithm you are using?

Author response:
>***Q1:*** The explanation of how the online agent works and how exactly does the online agent decide whether to use the learned policy or expert policy?

We really appreciate your valuable suggestion. Our methods use the advantage actor-critic (A2C) to train the online RL agent, which combines the Actor-Critic method and the concept of advantage function. The Actor is responsible for selecting an action (the action space is discrete, with 2 actions, one to choose the expert and the other to choose the learned policy) based on the current state. At the same time, the Critic evaluates the value of this action. The advantage function represents the advantage of an action relative to other actions. It can be used to evaluate the value of an action. Then use the cumulative reward to measure the quality of the agent's behavior in a given state. So, this algorithm can better estimate the value of actions and update parameters based on these values. When the online RL agent is trained, as shown in Figure 1, in each variable selection, input the state $s_t^O$ to the online RL agent, which will output an action 0 or 1, representing collect the sample generated from past-learned policy or expert policy. It is expected to make better decision-making in each variable selection, generate higher-quality training samples, and achieve effective exploration for imitation learning. This way, the sample generation process can be balanced between exploration and exploitation.

Reviewer comment (Presentation):
Table 6 why there is no highlight on the best performing method for each task? Or the values are not related to performance?

Author response:
>***Q5:*** Why there is no highlight on the best-accuracy method for each task?

Upon reviewing the findings presented in Table 4, Table 5, and Table 6, it becomes evident that the accuracy of the different methods, namely our method, GCNN, and ML4CO-KIDA, does not correlate with their overall performance. Therefore, it is inappropriate to single out any specific method as the best-performing one.

Reviewer comment (Evaluation):
Typically RL agents can take time to train, why is it the case that the proposed method, despite its complexity and a multi-stage/agent setup, can reduce training time compared to other methods?

Author response:
>***Q6:*** Why is it the case that the proposed method can reduce training time compared to other methods?

As shown in Table 7 and Table 8, For the training time, the offline RL plays a significant role in reducing the training samples, the training sample size of our proposed HRL-Aug reduced by around 30% at each iteration compared with ML4CO-KIDA, which is the reason why our method can reduce training time compared to ML4CO-KIDA. Moreover, our online RL agents operate within a relatively small state-action space, involving only two actions. Hence, despite the general notion that RL agents can be time-consuming to train, our method effectively mitigates this concern by leveraging offline RL and operating within a compact state-action space, resulting in a negligible increase in training time compared to the overall training process.