Paper: Dantzig-Wolfe Decomposition and Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The methods of Danzig-Wolfe Reformulation and Column Generation are existing approaches. The authors did not contribute significantly by improving these methods or specifically addressing the bin packing problem, rendering the contribution somewhat lacking.

Author response:
We have extended the final paragraph of the introduction to better highlight the novel contributions of the paper. We agree that Dantzig-Wolfe and Column Generation are existing approaches, this is the first time they have been combined with an RL agent in the sub-problem. Existing RL solutions to the offline bin packing problem have thus far only tackled simpler instances that only pack 1 bin with a reduced number of constraints. Such RL solutions struggle to handle offline bin-packing instances where more than 1 bin needs to be packed in the optimal solution. Our paper is the first example we know of in the literature that tests existing RL solutions on such instances. We show that RL without CG struggles to obtain good solutions, whereas the CG algorithm helps the RL obtain better results.

Reviewer comment (Presentation):
Section 2.1, the core of the authors' method, is hard to comprehend. It is necessary to incorporate some illustrations and motivation to enhance readability.

Author response:
We have improved our explanation in 2.1 by adding more intuitive explanations to the maths as illustrated matrices as a visual aid. We reference appendix E where the algorithm is demonstrated on a trivial example for clarity.

Reviewer comment (Theory):
There is an issue of local optimal solutions when employing the RL algorithm to solve subproblems. Therefore, after the author decomposes the full problem into a set partition, how can they ensure that RL will inevitably find a solution?

Author response:
Indeed, the sub-problem is a 3D knapsack problem, when solved using anything but exact solvers, we cannot guarantee the optimality of the solution (as explained in the introduction of the paper).  However, since we initialize with a feasible (but suboptimal) solution (the sequential RL solution) we can always guarantee a final solution that is feasible. In the worst case scenario, the sequential RL solution is the result we’ll get at the end of the CG algorithm. We see that this is the case for some simpler instances, but not for the more challenging ones. The final paragraph of the abstract mentions lower and upper bounds more clearly to reduce confusion around feasible solutions.

Reviewer comment (Novelty):
Although the combination is novel, the novelty in the machine learning part seems to be limited.

Author response:
The final paragraph of the introduction now details all the novel contributions of the paper more clearly. You rightly point out that our main contribution is combining RL with column generation. However we do have novel contributions to the RL part of the work too. A “value” feature was added to the items and used in the reward function. Previous literature optimized for volume utilization and not value packed. New constraints not fully taken into account by previous RL implementations of the knapsack problem were added. RL algorithms have not previously been tested on off-line test cases that require multiple bins to pack all items. This novel contribution shows the need for either increasing the action space or combining the 3D knapsack RL with the column generation algorithm as we have done in this paper.

Reviewer comment (Writing):
The meaning of "normal patterns" is missing from the presentation.

Author response:
We added a brief description of “normal patterns” in appendix E: 
> Normal patterns are a discretization scheme that is used to discretize the three axes of the container that consider all the possible points of placing any item along one axis of the container. They were first used by Junqueira et al. (2012) to solve the problem of 3D knapsack with cargo stability and load-bearing constraints. Kurpel et al. (2020) then extended these patterns to include all possible rotations of items. A full mathematical explanation of these patterns can be found in either reference.

Reviewer comment (Presentation):
The architecture of the deep RL agent is missing from the presentation.

Author response:
Section 4.1 gives a brief summary of the deep RL algorithm used and full details of the architecture are given in Appendix C.

Reviewer comment (Reproducibility):
The training methodology of the RL agent is missing from the presentation.

Author response:
We used the same architecture as suggested in the Jumanji github for the BinPack environment, the details are in appendix C. The advantage actor-critic algorithm (A2C) was used to train the deep RL agent; more details on this can now also be found in appendix C.

Reviewer comment (Presentation):
The formatting in the paper has issues, for example "(1-5) and (6-10)" should be better indented.

Author response:
The formatting issues you pointed out should now be fixed. The indentation of equations are now all aligned along the (in)equality. “Jumanji binpack” has been replaced by Jumanji BinPack to be consistent with their github.

Reviewer comment (Presentation):
The formatting in the paper has issues, for example "Jumanji bin pack" should be "Jumanji binpack".

Author response:
The formatting issues you pointed out should now be fixed. The indentation of equations are now all aligned along the (in)equality. “Jumanji binpack” has been replaced by Jumanji BinPack to be consistent with their github.

Reviewer comment (Writing):
The term "liquid volume" should be clarified.

Author response:
“Liquid volume” is a term often used as a concise way of saying the sum of the volume of all items in an instance. Your comment has made us realize that this may cause confusion, therefore we have replaced “liquid volume” in section 3.2 by “sum of the volume of all items” .