Paper: Dynamic Neighborhood Construction for Structured Large Discrete Action Spaces (ICLR-2024)

Reviewer comment (Experiments):
My one issue is with the first set of experiments in the maze. The results do not really demonstrate that DNC actually provides any advantage over the other baselines.

Author response:
Thank you for providing highly useful feedback and valuing our paper’s originality, the quality of our
proposed approach, the clarity of our write-up, and our paper’s significance.

**Weakness:** We agree that the maze environment is not ideal to benchmark our DNC method, as also mentioned by the other reviewers. However, we included the maze environment as it was used like this in prior works on large discrete action spaces [cf. 1, 2]. Accordingly, we decided to study the same environment to contextualize our results. Its purpose is to show that our approach does not fall behind the quality of other approaches in established environments. 

The motivation for DNC are contextual multi-stage stochastic optimization problems that arise in real-world applications. We followed your suggestion and added another environment from those problem domains, specifically a dynamic job-shop scheduling problem [cf. 3, 4]. In this  extensively studied problem a set of $G$ jobs needs to be allocated to $N$ machines that can serve up to $L$ jobs each, yielding an action space of size $|\mathcal{A}|=L^N$. We consider a load-balancing variant of this problem, for which jobs need to be balanced over heterogeneous machines. The action space in our experimental design amounts to $10^{10}$ actions, hence only DNC, DNC w/o SA, and MinMax can learn policies at all. The results show that DNC significantly outperforms DNC w/o SA and MinMax. For a more thorough results discussion as well as the updated convergence graphs in Figure 3, we refer to the updated paper. To keep the results section concise, we replaced the maze environment with 28 actuators by the job-shop scheduling results.

Moreover, we improved the manuscript according to your suggestions and now discuss DNC's memory advantage. DNC exhibits a consistent memory usage even as the action space size increases, highlighting its practical benefits. Our results, presented in Table 3 in Appendix H (see also pasted below), show that while the memory requirements for kNN, LAR, and VAC rise substantially when scaling from a maze of size $2^{12}$ to a job-shop environment with $10^{10}$ actions, the memory consumption for DNC remains stable at 68 to 73 MB. The memory consumption in the case of kNN, LAR, and VAC is driven by the need to setup a matrix containing all discrete actions. For example, in the case of the job-shop scheduling problem this would be a matrix with 5 rows and $10^{10}$ columns. We obtained the consumed memory using the Python profiler guppy. The stability of DNC is vital for applications with limited resources or when scaling for larger datasets without a significant increase in resource demands. Its constant memory footprint allows it to handle larger action spaces efficiently without performance loss due to memory constraints. This scalability is critical in real-world applications with finite computational resources. DNC's ability to maintain consistent memory usage, regardless of the action space size, makes it an essential tool in machine learning, particularly when resources are limited.

| **Algorithm** | Maze ($2^{12}$) | Maze ($2^{20}$) | Job-shop ($10^{10}$) |
|--------------------|-------------------|------------------|----------------------|
| DNC               | 68 MB           | 68 MB           |               73 MB |
| DNC w/o SA   | 68 MB           | 68 MB           |               73 MB |
| MinMax          | 68 MB           | 68 MB           |               73 MB |
| kNN                | 68 MB           | 252 MB          |     373,000 MB |
| LAR                | 68 MB           | 252 MB          |     373,000 MB |
| VAC                | 68 MB           | 252 MB          |     373,000 MB |

Reviewer comment (Experiments):
While the second set of experiments do demonstrate advantages, adding another domain may support performance claims by the authors.

Author response:
Thank you for providing highly useful feedback and valuing our paper’s originality, the quality of our
proposed approach, the clarity of our write-up, and our paper’s significance.

**Weakness:** We agree that the maze environment is not ideal to benchmark our DNC method, as also mentioned by the other reviewers. However, we included the maze environment as it was used like this in prior works on large discrete action spaces [cf. 1, 2]. Accordingly, we decided to study the same environment to contextualize our results. Its purpose is to show that our approach does not fall behind the quality of other approaches in established environments. 

The motivation for DNC are contextual multi-stage stochastic optimization problems that arise in real-world applications. We followed your suggestion and added another environment from those problem domains, specifically a dynamic job-shop scheduling problem [cf. 3, 4]. In this  extensively studied problem a set of $G$ jobs needs to be allocated to $N$ machines that can serve up to $L$ jobs each, yielding an action space of size $|\mathcal{A}|=L^N$. We consider a load-balancing variant of this problem, for which jobs need to be balanced over heterogeneous machines. The action space in our experimental design amounts to $10^{10}$ actions, hence only DNC, DNC w/o SA, and MinMax can learn policies at all. The results show that DNC significantly outperforms DNC w/o SA and MinMax. For a more thorough results discussion as well as the updated convergence graphs in Figure 3, we refer to the updated paper. To keep the results section concise, we replaced the maze environment with 28 actuators by the job-shop scheduling results.

Moreover, we improved the manuscript according to your suggestions and now discuss DNC's memory advantage. DNC exhibits a consistent memory usage even as the action space size increases, highlighting its practical benefits. Our results, presented in Table 3 in Appendix H (see also pasted below), show that while the memory requirements for kNN, LAR, and VAC rise substantially when scaling from a maze of size $2^{12}$ to a job-shop environment with $10^{10}$ actions, the memory consumption for DNC remains stable at 68 to 73 MB. The memory consumption in the case of kNN, LAR, and VAC is driven by the need to setup a matrix containing all discrete actions. For example, in the case of the job-shop scheduling problem this would be a matrix with 5 rows and $10^{10}$ columns. We obtained the consumed memory using the Python profiler guppy. The stability of DNC is vital for applications with limited resources or when scaling for larger datasets without a significant increase in resource demands. Its constant memory footprint allows it to handle larger action spaces efficiently without performance loss due to memory constraints. This scalability is critical in real-world applications with finite computational resources. DNC's ability to maintain consistent memory usage, regardless of the action space size, makes it an essential tool in machine learning, particularly when resources are limited.

| **Algorithm** | Maze ($2^{12}$) | Maze ($2^{20}$) | Job-shop ($10^{10}$) |
|--------------------|-------------------|------------------|----------------------|
| DNC               | 68 MB           | 68 MB           |               73 MB |
| DNC w/o SA   | 68 MB           | 68 MB           |               73 MB |
| MinMax          | 68 MB           | 68 MB           |               73 MB |
| kNN                | 68 MB           | 252 MB          |     373,000 MB |
| LAR                | 68 MB           | 252 MB          |     373,000 MB |
| VAC                | 68 MB           | 252 MB          |     373,000 MB |

Reviewer comment (Evaluation):
Alternatively, if the argument is that the performance of DNC is competitive to other SOTA, what other benefits may it provide (better wall-clock time, memory usage, etc.)?

Author response:
Thank you for providing highly useful feedback and valuing our paper’s originality, the quality of our
proposed approach, the clarity of our write-up, and our paper’s significance.

**Weakness:** We agree that the maze environment is not ideal to benchmark our DNC method, as also mentioned by the other reviewers. However, we included the maze environment as it was used like this in prior works on large discrete action spaces [cf. 1, 2]. Accordingly, we decided to study the same environment to contextualize our results. Its purpose is to show that our approach does not fall behind the quality of other approaches in established environments. 

The motivation for DNC are contextual multi-stage stochastic optimization problems that arise in real-world applications. We followed your suggestion and added another environment from those problem domains, specifically a dynamic job-shop scheduling problem [cf. 3, 4]. In this  extensively studied problem a set of $G$ jobs needs to be allocated to $N$ machines that can serve up to $L$ jobs each, yielding an action space of size $|\mathcal{A}|=L^N$. We consider a load-balancing variant of this problem, for which jobs need to be balanced over heterogeneous machines. The action space in our experimental design amounts to $10^{10}$ actions, hence only DNC, DNC w/o SA, and MinMax can learn policies at all. The results show that DNC significantly outperforms DNC w/o SA and MinMax. For a more thorough results discussion as well as the updated convergence graphs in Figure 3, we refer to the updated paper. To keep the results section concise, we replaced the maze environment with 28 actuators by the job-shop scheduling results.

Moreover, we improved the manuscript according to your suggestions and now discuss DNC's memory advantage. DNC exhibits a consistent memory usage even as the action space size increases, highlighting its practical benefits. Our results, presented in Table 3 in Appendix H (see also pasted below), show that while the memory requirements for kNN, LAR, and VAC rise substantially when scaling from a maze of size $2^{12}$ to a job-shop environment with $10^{10}$ actions, the memory consumption for DNC remains stable at 68 to 73 MB. The memory consumption in the case of kNN, LAR, and VAC is driven by the need to setup a matrix containing all discrete actions. For example, in the case of the job-shop scheduling problem this would be a matrix with 5 rows and $10^{10}$ columns. We obtained the consumed memory using the Python profiler guppy. The stability of DNC is vital for applications with limited resources or when scaling for larger datasets without a significant increase in resource demands. Its constant memory footprint allows it to handle larger action spaces efficiently without performance loss due to memory constraints. This scalability is critical in real-world applications with finite computational resources. DNC's ability to maintain consistent memory usage, regardless of the action space size, makes it an essential tool in machine learning, particularly when resources are limited.

| **Algorithm** | Maze ($2^{12}$) | Maze ($2^{20}$) | Job-shop ($10^{10}$) |
|--------------------|-------------------|------------------|----------------------|
| DNC               | 68 MB           | 68 MB           |               73 MB |
| DNC w/o SA   | 68 MB           | 68 MB           |               73 MB |
| MinMax          | 68 MB           | 68 MB           |               73 MB |
| kNN                | 68 MB           | 252 MB          |     373,000 MB |
| LAR                | 68 MB           | 252 MB          |     373,000 MB |
| VAC                | 68 MB           | 252 MB          |     373,000 MB |

Reviewer comment (Writing):
Tiny nitpick: The second sentence in Problem Definition — it would read better to describe the states and actions in the order you introduce them.

Author response:
**Questions:** Thank you for the suggestion. We have altered the order in this sentence.

Reviewer comment (Experiments):
The maze domain seems a bit too trivial to be useful, it is informative for a demonstration of the method, but unlike the inventory management environment, I'm not sure what the real-world justification might be. I would prefer to have at least one more grounded and complex environment (perhaps a video game with many different possible actions, perhaps a multi-agent setting? I'm not the most familiar with the domains in this subfield).

Author response:
We appreciate your feedback regarding the choice of the maze domain and the suggestion for more complex environments. We included the maze environment as it was used like this in prior works on large discrete action spaces [1,2]. Accordingly, we decided to study the same environment to contextualize our results. Its purpose is to show that our approach does not fall behind the quality of other approaches in established environments.

The motivation for DNC are contextual multi-stage stochastic optimization problems that arise in real-world applications. We followed your suggestion and added another environment from those problem domains, specifically a dynamic job-shop scheduling problem [cf. 3,4]. In this  extensively studied problem a set of $G$ jobs needs to be allocated to $N$ machines, that can serve up to $L$ jobs each, yielding an action space of size $|\mathcal{A}|=L^N$. We consider a load-balancing variant of this problem, for which jobs need to be balanced over heterogeneous machines. The action space in our experimental design amounts to $10^{10}$ actions, hence only DNC, DNC w/o SA, and MinMax can learn policies at all. The results show that DNC significantly outperforms DNC w/o SA and MinMax. For a more thorough results discussion as well as the updated convergence graphs in Figure 3, we refer to the updated paper. To keep the results section concise, we replaced the maze environment with 28 actuators by the job-shop scheduling results.

Reviewer comment (Writing):
Perhaps this is my unfamiliarity with the domain, but sometimes the text is quite hard to follow. I think part of this is the reliance on lots of acronyms (LDAS, SLDAS, MILP, SA), but also the claim in Section 2 "we formulate the task of finding a Q-value maximizing a as a mixed-integer linear program" was a bit confusing on first read, as really the authors are proposing an approximate solution, so saying this upfront might be a bit more clear.

Author response:
Thank you for your suggestions. We understand that the abbreviations, while aimed at maintaining conciseness to adhere to the page limit, may have impacted the readability to some extent. Because of the page limit, we decided to keep the abbreviations in the main document. We made several textual changes to enhance the clarity of the paper. 

We have improved the sentence related to the MILP, highlighting that we only use the MILP to formalize the problem but aim for an efficient algorithm to solve it.

Reviewer comment (Presentation):
Would be nice to see MILP results in Figure 5 against the proposed method and other baselines in Figure 3, and the discussion at the end of Appendix B is informative and it may be nice to have a note in the text.

Author response:
Thank you for pointing this out. We added the remaining results in Figure 5 and added a note referring to Appendix B in the main text.

Reviewer comment (Experiments):
My primary criticism is that the considered environments and scenarios where the technique applies are somewhat contrived. If one is faced with such a huge decision space, we would typically expect to spend the time in the modelling stage of the problem to design an action space that is substantially more manageable. For example, in the maze environment, we could design a discrete set of actions (move left, etc.), with the actual execution being handled by lower-level control primitives. Indeed, this needs to be performed for each individual problem (as the authors do mention).

Author response:
**W1:** We agree that the huge action space in the maze could be avoided by refined modeling. We included the maze as it was used like this in prior works on large discrete action spaces [1,2]. Accordingly, we decided to study the same environment to contextualize our results. Its purpose is to show that our approach does not fall behind the quality of other approaches in established environments.

We believe using DNC and a refined modeling effort are not necessarily mutually exclusive. For example, one can use DNC to create benchmarks to verify the performance of the modeling approach. The main motivation of our work does not relate to problems handled by refined modeling, but stems from contextual stochastic multi-stage optimization settings grounded in real-world applications. Such settings, e.g., the inventory and job-shop environments, often lack room for action space simplification, as it would infer a significant loss in solution quality.

Reviewer comment (Evaluation):
For the maze environments, the variability of the techniques is such that the confidence intervals overlap substantially, and it is not possible to draw reliable conclusions from these figures alone. Would this still happen with substantially more seeds (e.g. 50+)? Alternatively, would another procedure (e.g., running many episodes with randomly initialised positions) yield more reliable (i.e., with non-overlapping CIs)?

Author response:
**W2.1:** We re-ran the maze with 12 actuators and with teleporting tiles (50 seeds). For statistical significance and consistency we also ran 50 seeds for the other environments. To reduce the computational burden and save space in the main text, we excluded the results on the maze with 28 actuators. We report the results in Fig 3.

* For the maze with 12 actuators, CIs are narrower than in the original submission. However, CIs still overlap and the avg. performances of the different approaches do not differ significantly from the results in the original paper.

* For the maze with teleporting tiles, we reconfirm previous observations. The CI of DNC overlaps with MinMax's CI only slightly, showing DNC's advantage over MinMax in this environment. DNC is therefore able to provide high quality policies in more complex environments, comparable to kNN. This result is noteworthy, as MinMax is the only SOTA method able to scale to very large SLDAS such as the inventory environment. To disclose the CI overlap of DNC and MinMax, we included Fig 8 in Ap. H and summarized the findings in the main paper.

In addition, we added boxplots in the appendix, displaying the avg. performance across the maze in the final training episode (see Fig 9 in Ap. H) and summarized the findings in the main paper. The results suggest that convergence in the maze is much more stable than perceived (cf. Fig 3). The wide CIs can be attributed to the inherent reward variance of the environment. We refer to Ap. H, Fig 10 for a more thorough discussion.

We addressed your alternative suggestion to run several episodes with random initial positions. The resulting outcomes and discussion are detailed in Fig 11 in Ap. H.

Reviewer comment (Evaluation):
The simulated annealing search procedure must introduce a computational overhead compared to picking the best action greedily (e.g., as in DNC w/o SA). This is not mentioned, and the impact on runtimes is not analyzed or discussed. This needs to be addressed, accompanied by measurements.

Author response:
**W2.2:** We measured the avg. duration of obtaining a decision from the policy and applying it to the environment, i.e., step time, within an episode during testing. We apply DNC during both testing and training, so the reported step times also apply during training.  We have added a discussion and Fig 7 in Ap. H, wherein we compare avg. step times (50 seeds) for the benchmarks. We added a reference in the main paper.  Our findings suggest that while SA increases training time, its benefit in terms of search quality may outweigh these costs, especially if the action space is so large that other methods fail to learn at all due to memory restrictions (kNN) or learn less performant policies (MinMax, DNC w/o SA). During training, we consider longer computational times to be acceptable as the focus is on model accuracy, not speed. In practical implementation, DNC offers step times in the range of milliseconds, remaining practical for real-time decision-making.

Reviewer comment (Writing):
The paper refers in several places to approaches for "unstructured" LDAS (abstract, intro, figure 1, etc.). This is somewhat of a misnomer since these problems *do* exhibit structure (i.e., the action space is such that actions that are close in embedding space yield similar Q-values and lead to similar outcomes when actuated in the environment). I think a better term would be *irregularly* structured. Similarly, the proposed method applies to *regularly* structured action spaces.

Author response:
**C1:** We added the proposed terms and made a clarification in the beginning of paragraph 2. In the rest of the paper we kept the original submission's "structured" and "unstructured" for conciseness.

Reviewer comment (Theory):
Related to the second point of W2 above, the paper mentions the use of "SA to efficiently search across different and potentially better neighborhoods" (Discussion on p6). While this statement potentially applies for the memory efficiency, it certainly doesn't in terms of time, and should be qualified. SA in general is not computationally efficient.

Author response:
**C2:** We have adjusted wording on page 6 to more accurately reflect the characteristics of SA. SA's time impact within DNC remains controllable, see Fig 7 in Ap. H. The max. no. SA iterations is limited by DNC's hyperparameters. The max. no. SA search iterations needed to achieve good performance is limited to 10 SA search iterations in the inventory environment (cf. Table 2, Ap. F). During a single SA iteration, DNC evaluates multiple neighbors in parallel through our dynamic neighborhood construction procedure, see Section 3. Hence, evaluating a larger neighborhood does not necessarily require more SA search iterations.

Reviewer comment (Writing):
Small nitpick: "Constraints 2" -> "Constraint 2", etc. in the text at the end of Section 2.

Author response:
**C3:** Thank you for the comment. We use the plural to emphasize that we refer to a set of constraints. For example, Equation (4) has to hold for all neurons $d\in \mathcal{D}_2$.