Paper: Combinatorial Optimization via Memory Metropolis: Template Networks for Proposal Distributions in Simulated Annealing applied to Nanophotonic Inverse Design (ICLR-2024)

Reviewer comment (Theory):
The technical contribution is not strong. The proposal is generally mired in complexity which may make it inaccessible for readers not deeply familiar with all the integrated disciplines. Intentionally violating the Markov property without substantial justification is concerning. Further evidence or theoretical underpinnings are needed to support this decision.

Author response:
2. We agree, that the markov property should not be violated if one seeks to directly sample from the underlying distribution (here i.e., the Boltzmann distribution of fitness values).
  However, we are only interest in finding a final state of high fitness while clustered structures are preferred.
  Thus, violating the Markov property is not of big concern in our case which we state now more clearly in our text.\
  Furthermore, we added a detailed analysis of the consequences of the violation of the Markov property in Sec. A.15.
  We there conclude, that the violation actually results in a new Markov process, which generates samples from the product of the Boltzmann distribution of fitness values and the bias term $B = \exp{2Q/T_S}$ where Q is the learned template after the template matching operation (TMO).
  We think this theoretical analysis substantially strengthen our motivation for violating the Markov property and hope this answers the question on the implications.

Reviewer comment (Writing):
Rewriting for clarity can make the paper more accessible to a broader audience.

Author response:
Thanks for your feedback.
We updated the manuscript regarding abstract, methods and conclusion and by this made it more clear and accessible.
By this we incorporated your suggestions listed under _Weaknesses_, except (1) which we are answering as part of your questions:

Reviewer comment (Writing):
Abstract is too lengthy.

Author response:
Thanks for your feedback.
We updated the manuscript regarding abstract, methods and conclusion and by this made it more clear and accessible.
By this we incorporated your suggestions listed under _Weaknesses_, except (1) which we are answering as part of your questions:

Reviewer comment (Presentation):
The conclusion should reiterate the major findings, their implications, and potential future work in a more detailed manner.

Author response:
Thanks for your feedback.
We updated the manuscript regarding abstract, methods and conclusion and by this made it more clear and accessible.
By this we incorporated your suggestions listed under _Weaknesses_, except (1) which we are answering as part of your questions:

Reviewer comment (Experiments):
A detailed side-by-side comparison with existing SA and regularized SA methodologies is required.

Author response:
1. Since we do not only seek to maximize our devices fitness $f$, but also minimize the granularity $g$ we can not formulate a single objective which can be compared easily e.g. in a table, unfortunately.
  Instead, we chose to depict both objectives in Fig. 4 for a direct side-by-side comparison.
  In Fig. 4A SA as well as a pure policy gradient RL approach are given by the border cases of $T_S=\infty$ and $T_M=\infty$, respectively.
  As indicated, regularized SA is a suitable alternative approach to MeMe, we thus include a comparison in Fig. 4B and further show, that MeMe outperforms regularized SA for any regularization factor $\lambda$ in Sec. A.3.

Reviewer comment (Theory):
Delving deeper into the reasons for violating the Markov property and the potential implications can make the proposal more convincing.

Author response:
2. We agree, that the markov property should not be violated if one seeks to directly sample from the underlying distribution (here i.e., the Boltzmann distribution of fitness values).
  However, we are only interest in finding a final state of high fitness while clustered structures are preferred.
  Thus, violating the Markov property is not of big concern in our case which we state now more clearly in our text.\
  Furthermore, we added a detailed analysis of the consequences of the violation of the Markov property in Sec. A.15.
  We there conclude, that the violation actually results in a new Markov process, which generates samples from the product of the Boltzmann distribution of fitness values and the bias term $B = \exp{2Q/T_S}$ where Q is the learned template after the template matching operation (TMO).
  We think this theoretical analysis substantially strengthen our motivation for violating the Markov property and hope this answers the question on the implications.

Reviewer comment (Evaluation):
Authors are suggested to discuss the broader applicability of the MeMe algorithm, beyond the specific case study presented.

Author response:
3. / 4. While we focused on an in-depth analysis of the problem of nanophotonic inverse design here, many other applications exist where similar structures are sought.
  This is especially true for engineering where optimization of structures for various objectives and under specific constraints exist.
  However, while many of these applications exist, they are ofentimes rather specific and require a detailed problem introduction as we did for the nanophotonic inverse design.

Reviewer comment (Experiments):
The paper doesn't clarify whether the approach is generalizable outside of the specific domain it was applied to.

Author response:
3. / 4. While we focused on an in-depth analysis of the problem of nanophotonic inverse design here, many other applications exist where similar structures are sought.
  This is especially true for engineering where optimization of structures for various objectives and under specific constraints exist.
  However, while many of these applications exist, they are ofentimes rather specific and require a detailed problem introduction as we did for the nanophotonic inverse design.

Reviewer comment (Novelty):
A major weakness in my opinion is that it is unclear how much of the method is generally applicable to other problem settings. It appears that the design of the template networks requires quite a bit careful engineering and domain knowledge and can be potentially challenging on other tasks. The paper's narrow focus on a specific application also makes it somewhat poorly positioned for the audience at ICLR, even though the domain is introduced appropriately. I encourage the authors to consider alternative venues where the particular application is a focus.

Author response:
We agree that our application is focused on nanophotonic inverse design.
  However, while other applications exist, they are oftentimes rather specific and require a detailed problem introduction as we did for the nanophotonic inverse design in our work.\
  Furthermore, we think our proposed algorithm might be of interest to the ICLR audience from a conceptional perspective, which can only hardly be adresses in a more application focused venue.
  The concept of input-less template networks is, to the best of our knowledge, new to deep RL in general and offers new perspectives especially for optimization applications.

Reviewer comment (Experiments):
Another major shortcoming is the lack of baselines - the authors only compare the apporach to simulated annealing but it would be good to have other baselines for instance some standard RL methods like PPO.

Author response:
We agree that comparisons to additional algorithms would be interesting, however, to the best of our knowledge there are no promising alternative algorithms for the problem we studied in our work.
  When analyzing the effect of the sampling temperature $T_S$ and the metropolis temperature $T_M$ in detail, multiple baseline algorithms emerge in their borderline cases.
  Next to SA and greedy-exploration, this also includes a direct policy gradient apporach if $T_M=\infty$.
  Furthermore, we also use loss clipping, which is a key part of PPO.
  As training our template networks differs from classical deep RL in many aspects, we many components which where developed for other RL agents like PPO, are missing motivation for our proposed algorithm.
  This includes value functions, lower discounts, rollouts of multiple samples, multiple policy update steps (as in PPO objective maximization).
  Instead, we seek to update the learned template as direct and fast as possible with newly encountered feedback from the environment.
  We thus included features of e.g. PPO which are transferable to our application while think that more sophisticated RL methods are not well motivated as a baseline for our application.

Reviewer comment (Evaluation):
Can you provide more details on the computational cost of MeMe? How does it scale with the problem size?

Author response:
The computational cost of our algorithm is dominated by the FDFD-simulations.
  So an empirical evaluation would only observe the scaling of these (which scales quadratic with the system size).
  E.g. for our experiments in Fig. 4 computations took $\approx 20$h on 26 Intel Xeon Gold 6140 2.30 GHz CPU cores per optimization.\
  Since updates of the template network are only propagated to a limited local field of view, training as well as updates of the proposal distribution are scaling linear with the number of pixels (instead of quadratic as expected for e.g. FC layers).

Reviewer comment (Experiments):
There is a potential issue of overfitting in the training of the template network? If so, how is it addressed?

Author response:
Indeed, we intend to fully fit our networks to single optimization runs as a key concept of template networks.
  This distinguises template networks from most other architectures used in deep RL.
  In this regard out networks share a lot with e.g. NeRFs which are also fitted to single scenes.
  Thus overfitting is not of concern.