Paper: DNCs require more planning steps (ICLR-2024)

Reviewer comment (Presentation):
Some of the important topics are glanced over whereas more relevant parts could have been explored in more detail. For the readers unfamiliar with DNCs, this paper would be hard to grasp. I think more time should be spent on Section 2, and including an illustration would greatly help. This can be done by reducing Section 3.1 to a single paragraph describing the general inspiration.

Author response:
2. We agree that it’s difficult to faithfully present the DNC in such a short paragraph as we have. We took inspiration from the way other papers in the field presented the DNC. An illustration of the DNC isn’t simple, and we felt that by adding one we’ll have to spend much more time explaining it than we’d like. Following your suggestions, as we wanted to incorporate the new experiments into the revised paper, we had to move this introduction to an appendix. This will allow us to elaborate further on the DNC model in the final submission if needed.

Reviewer comment (Evaluation):
The paper is sometimes too loose when it comes to the results interpretation and some phrasings. I will put more details on this in the 'questions' subsection of the review, but here are some examples:  
- Section 3.2 is called 'adaptive planning budget improves generalisation'. Figure 1 shows that training with budget = n is better than training with budget = 10. However, we don't know what will happen when we train with n = 20 which is fixed, but more than the original n = 10.  
- Section 3.2 proposes an explanation that is not scrutinised in the experimental section: "Since the model was only allowed a constant planning budget during training, it was forced to find an average-case algorithm that can be efficiently run, and whose internal representation of the data can be efficiently written to and read out of memory." To clarify things, I find this hypothesis extremely interesting and worth exploring. But it has to be tested in some way.  
- I find phrases like 'learning a more general algorithm' to be quite vague and misleading. Furthermore, in my opinion they make it much harder to read the paper as it would be if the reasoning was just about generalisation properties of the model. For example, instead of 'learning a more general algorithm' one can say 'generalise to a wider range of input sizes' or something similar. I believe reasoning about the effect of a particular experiment from the perspective of out-of-distribution generalisation would be easier to interpret and reason about.

Author response:
3. Loose phrasing:  Given the additional and improved evidence we supply in our revised version, we hope it gives better support to our usage of general language. \
Importantly, we also improved our terminology, specifically around the notion of “more general usage of memory” - please see Section 3.3.
    1. This is a very good point. The original DNC paper as well as following research used 10 planning steps or none at all, which is why we chose this value. To the best of our knowledge, no other DNC paper tried to increase this value. Our motivation was that any constant budget inherently poses a limit to the DNC model. We added Section 5.2, where we further discuss this as well as present results of new experiments with p=75 and p=200. 
    2. We agree that this statement is merely an intuitive explanation and not empirically demonstrated. For the purpose of our paper it doesn’t really matter what algorithm was learned, we simply gave a potential explanation as an example for one possibility. An experiment demonstrating it could be very interesting and we consider doing it for future work.
    3. We agree here too. We still didn’t find the best terminology for what we want to say, however we feel that the current terminology is better as we explained in point 1 above.

Reviewer comment (Experiments):
The total number of domains used in the experiments is limited. The authors run experiments on only two problems, and the performance differential is significant on only one of the problems (Shortest Path). Could you please motivate the choice of using a single performant domain for the analysis?

Author response:
We agree that showing experiments for more problem domains is important. Unfortunately, compute resources and runtimes were the main limiting factors in this work. Especially for a more complex problem like MinCut, training with large planning budgets becomes prohibitive very quickly. A lot of the time was spent on optimizing the training code to reduce this overhead, but we still couldn’t completely remove this problem.  \
We have updated Section 3.1 to better explain our motives in choosing these specific problems. We believe most problems previously used to evaluate DNCs are too simple and therefore not relevant, as they don’t fall in the category of algorithms that would benefit from an adaptive planning budget.  \
We also wish to state that most DNC papers do a rather “shallow” exploration of the problems they evaluate on. We tried to do a deeper dive into the mechanism of the model as exhibited in our chosen problems. For example, choosing the correct data distribution to prevent the model from overfitting was a laborious task. This prevented us from examining more problems in time for this submission.

Reviewer comment (Evaluation):
There is no analysis provided about the resources expended for training with the adaptive versus the fixed scheme. Including simple plots or data showing the training times versus performance trade-off would have enhanced the clarity of the paper. Do you have any data pertaining to the training time required for the adaptive case?

Author response:
This is a very good question. We ran some further experiments including suggestions from other reviewers, which address this question directly. It is now explained in Section 5.2, including a figure comparing the generalization performance of models with different training budgets vs. the compute used during training. Note that “compute” here is a bit nuanced, please refer to the updated section for details. Specifically, Figure 6(b) shows that training resources used to train the adaptive model match the constant baseline of p=10. Furthermore, to reach the same performance as the adaptive model with a large constant budget, it required quadruple the amount of computation resources during training.

Reviewer comment (Experiments):
Currently, there is no way to automatically determine the adaptive planning budget, and it relies on human-expert input such as "the total number of edges". Do you have any general intuition as to how to better select the adaptive planning budget as opposed to an empirical approach? (I recognize that this is a hard problem in itself, but I would like to know if an easier training scheme exists.)

Author response:
This is indeed a very hard problem to solve generally. We can say two things here: first, future work could tackle this question directly; second, if we estimate the complexity of a problem we could use the asymptotic complexity as the function for the size of the planning budget. For example, the Shortest Path problem is known to have linear complexity, hence our choice for it makes sense. For MinCut, the best known algorithm is |V|  |E|, which motivates us to use that value. We will be adding this run to the final submission.

Reviewer comment (Evaluation):
In Figure 3b, the purple (edges=40) line drops starkly in performance when given more planning steps; i.e., the adaptive scheme was not able to maintain steady state performance (as the authors put it in Sec 4.2). Do you have any intuition as to why this happened for the purple case?

Author response:
We don’t have a good intuition for this effect yet. This question is interesting, and points at potential nuance in the behavior of the DNC model. Answering it might lead to further improvements. However, note that the rate of degradation is approximately linear with the number of extra planning steps, in contrast with the almost instantaneous reduction that occurs for the constant model. Even though the adaptive model’s steady-state is not as stable for small inputs as it is for large inputs, it’s still much more stable than the constant model.

Reviewer comment (Experiments):
The DNC with constant planning budget of 10 steps has much less computation available than the DNC with flexible planning budget, which uses up to 75 steps during training. At test time, the difference becomes even larger as we evaluate the DNCs on graphs with up to 200 edges. It does not seem fair to compare models with such different compute budgets.

Author response:
First, we agree with your observation that comparing our adaptive method with a constant planning budget of size p=10 may be unfair due to the difference in computation available to the models. All current work on DNC, to the best of our knowledge, uses a constant planning budget of p=10 or none at all, which is the justification for that choice for this specific constant for the baseline. This is a detail which we have accidentally removed from the first version of the paper. We made sure to clearly state this in our revised version, in multiple locations.

Reviewer comment (Experiments):
It may be that the fixed planning budget of 10 steps is simply too small to learn general algorithms. To prove the point that flexible planning budgets are required for learning generalizable algorithms, the authors should show that a baseline with a fixed planning budget equivalent to the longest graphs the model is evaluated on (i.e. 200 steps in the experiments) still fails to learn generalizable algorithms.

Author response:
Throughout the paper we describe how this choice hinders performance significantly. So, we ran a few experiments in the past week, where we trained two models with larger constant planning steps, one with p=75,which is equal to the size of the largest input seen during training, and another with p=200, the size of the largest input seen during evaluation.

Reviewer comment (Experiments):
For the latter point, one could argue that it is impractical or wasteful to train with the max planning length even for small inputs. But still, the fixed planning budget should be set in a way that it matches the compute flops used for the flexible planning model during training. It is unclear how the current value of 10 was chosen and it seems to result in a baseline that has overall less compute and thus an unfair comparison.

Author response:
The results from this experiment affirm that a larger constant planning budget helps the model’s performance, which aligns with our basic premise, which is also the title of the paper - DNCs require more planning steps. However, the results also point out the natural limitations of such models. To make this clearer, we modified the explanation in Section 3.1, and we analyzed the results and compared them to the adaptive budget in the newly added Section 5.2. Notably, the results also demonstrate that the resources required for training the adaptive linear model match those required for the baseline p=10 model. Moreover, to match the performance of the linear model a constant of at least p=200 is required, which comes at the cost of quadrupling the training resources.

Reviewer comment (Experiments):
The same argument about fairness in compute budget also holds true for the "increased memory utilization" experiments.

Author response:
As for the memory utilization – the reviewer had a concern that a constant model with a bigger memory is missing, but it is not feasible, resource-wise.

Reviewer comment (Evaluation):
A separate concern is the stability of the proposed approach. The results in appendix C suggest that the models' generalization capability heavily varies between different seeds. This is concerning since the model presented in section C does not exhibit the central generalization capabilities that the paper claims.

Author response:
Regarding the instability with respect to the random seed as presented in Appendix C, we also observe this phenomenon with some of the constant budgets. Hence this property does not affect our comparison, it is simply a property of training a DNC.

Reviewer comment (Experiments):
The paper uses the simple heuristic of setting the number of planning steps to be equivalent to the length of the input. This is an arbitrary choice that may not generalize well to other tasks and may require hand-tuning on any new task family.

Author response:
Regarding the heuristic choice of adaptive budget, we agree that searching for an optimal adaptive budget scheme (rather than linear) might be required per task, and could be an interesting direction for future work. We will be adding an adaptive p=|V||E| model to the MinCut in the final submission to showcase other adaptive schemes.