Paper: Hessian-Aware Bayesian Optimization for Decision Making Systems (ICLR-2024)

Reviewer comment (Writing):
The paper is not easy to follow for many readers, particularly for those unfamiliar with reinforcement learning with multi-agent systems. Even the basic problem setting is difficult to fully understand due to unclear descriptions.

Author response:
- Basic problem setting
- - The basic problem setting is that of multiple agents interacting cooperatively to accomplish a common goal. This is also in a memory constrained setting (such as on IoT devices or drone delivery tasks). Within this problem setting, we propose a variant of high dimensional Bayesian optimization and show how it can optimize a compact policy space (memory constrained setting). Specifically we show in the case of malformed reward scenarios or complex multi-agent interactions, globally optimizing a small policy using our approach outperforms related work.

Reviewer comment (Writing):
The meanings of several key words (such as abstraction, immutable, compile, and so on) are not clarified in the text, making the technical description difficult to understand for readers who are not familiar with the topic.

Author response:
- Definitions of key words
- - Abstraction: We solve several problems in our work by using abstraction. Abstraction, is separating a complex system into many components which interact in a well defined manner. In our work we use abstraction to separate decision making into role assignment, and role interaction,
- - Immutable: Immutable here means something that does not change. As we are optimizing over a parameter space, the immutable part of the policy (the instructions Alg. 1,2,3) are immutable as they are not searched over or optimized.
- - Compile: Here compilation means the immutable and mutable (parameters) part of the policy are joined together to create the actual policy.

Reviewer comment (Writing):
What is the definition of the role affinity function \(\Lambda^\theta_{r,i}\)? What does it indicate?

Author response:
- What is the definition of the role affinity function \Lambda^\theta_r,i ? What does it indicate?
- - The role affinity function is trying to quantify the affinity for an agent to take on a specific role. It indicates the affinity for that agent to take on that specific role.

Reviewer comment (Writing):
What is \(h\) without a subscript in line 4 of Algorithm 4?

Author response:
- What is h without subscript in line 4 of Algorithm 4?
- - h is a tensor (a stack of matrices) formed by collecting many Hessian matrices from line 3 in Algorithm 4.

Reviewer comment (Theory):
What is a surrogate of the Hessian? How is it constructed?

Author response:
- What is a surrogate of Hessian? How is it constructed ?
- - The surrogate Hessian is a surrogate quantity used in the paper as access to the true Hessian is either not available or too expensive to approximate. We discuss this further in Appendix H. The construction of the surrogate is as follows (found in Appendix A)
- - To estimate the Hessian, we used the Hessian-Vector product approximation. We relaxed the discrete portions of our metamodel policy into differentiable continuous approximation for this phase using the Sinkhorn-Knopp algorithm for the Role Assignment phase. For role interaction network connectivity, we used a sigmoid to create differentiable “soft” edges between each role.

Reviewer comment (Theory):
The authors claim the cumulative regret of the proposed method scales with \(O(\log D)\). What assumption makes this possible? The implication behind assumption 1 is difficult to identify. Does it assume some low-dimensional essential dependency in the underlying true function?

Author response:
- The authors claim the cumulative regret of the proposed method scale with O(log D). What assumption makes it possible? The implication behind assumption 1 is difficult to find for me. Does it assume some low dimensional essential dependency in the underlying true function?
- - The approximate high level sketch of Assumption 1 is that the dependency graph drawn from the Erdős-Rényi model is of low degree/connectivity. The base of the logarithm is related to the quantity p_g we discuss in Assumption 1. This is elaborated on in the proof of Theorem 1.

Reviewer comment (Presentation):
The main weakness of the paper (and it is understandable) is the amount of clutter in the paper. Most figures and tables are impossible to read without zoom. The amount of information provided should be more appropriate to a journal article, although I can see the motivation behind a submission to ICLR instead. However, the paper is clear enough.

Author response:
Dear reviewer,

We are looking into making changes to the draft to aid in readability. Here are immediate answers to your questions.

Reviewer comment (Theory):
Despite reading appendix G, regarding the method presented, it is still unclear to me if H_pi can always be used as a reliable surrogate for H_v, as g might have some 0 (or small) components. In fact, for the optimal policy, shouldn’t it be 0?

Author response:
- H_pi can always be used as a reliable surrogate for H_v?
- - This is a good question. In all of our validation scenarios, H_pi always served well in learning the dependency structure. We believe that this is due to the "constructed nature" of the reinforcement learning benchmarks. The tasks themselves are constructed in such a way that learning is possible. We're thinking about whether it is possible to make an argument along this direction in the Appendix.

Reviewer comment (Novelty):
Why do you need JIT-compilation? Isn’t it just instantiation with certain parameter values?

Author response:
- Need for JIT-compilation?
- - The JIT-compilation was a design decision in order to have a compact policy space. We are looking for citations and references which show that this approach is a reasonable way to simplify a parameter space.

Reviewer comment (Reproducibility):
In algorithm 4, do you initialise the parameters with a uniform distribution? In BO literature, it is common to use low discrepancy sequences or sampling procedures.

Author response:
- Algorithm 4 initialization procedure
- - We use Sobol sequences in order to quasi-randomly sample the parameter space.

Reviewer comment (Presentation):
The presentation of the paper is poor.

Author response:
We are looking into improving the readability with another draft.

Reviewer comment (Writing):
More background knowledge should be introduced to further clarify the motivations, like the necessity of the JIT compilation.

Author response:
This is an interesting point, our approach with JIT compilation was to save on resources as we were considering the specific use case of memory constrained devices. We will see if we can find other related work which does the same to justify this design choice.

Reviewer comment (Writing):
The notation is extensively abused and many algorithms and notations have no formal definitions (See questions for some of them).

Author response:
We attempted our best to not abuse notation. Unfortunately, to make contributions that would be interesting and valuable to the decision making systems community, as well as have theoretical results, some of our notation was slightly abused.

Reviewer comment (Theory):
The motivation of one of the main contributions, using Hessian in Bayesian optimization, is not very clear. As Bayesian optimization is a black-box function, directly querying Hessian might be very hard.

Author response:
We will immediately reach out to the PC as this should not be the case. We have uploaded many revisions for the paper.
- We elaborate on the costs and benefits of the Hessian several times within the paper, and show that due to the expensive nature of the Hessian, we focus our specific use case to only decision making systems where a useful and valuable surrogate Hessian can be extracted from observing the policy generation process.
- - Introduction: In our work, we overcome this shortcoming by observing the GEN process of the HOM. In particular, we can measure a surrogate Hessian during the GEN process which significantly simplifies the task of learning the additive structure. We term this approach Hessian-Aware GP-UCB (HA-GP-UCB) and visualize our approach in Fig. 1. 
- - Section 4.4: In practice, observing the hessian $H_v$ is not possible due to v being a black box function. However, during the GEN process we can observe a surrogate Hessian, $H_\pi$. This surrogate Hessian is closely related to the $H_v$ as $v(\theta)$ is determined through interaction of the policy with an unknown environment. Because the value of a policy is a function of the policy; it follows by the chain rule $H_\pi$ is an important sub-component of $H_v$. We utilize the surrogate Hessian in our work and demonstrate its strong empirical performance in validation.
- - Appendix H: In Section 4.5 we remarked that although we cannot observe $H_v$, we can observe a surrogate hessian, $H_\pi$ which is related to $H_v$ by the chain rule. We justify our choice here with showing how $H_\pi$ is an important sub-component of $H_v$ (Skorski, 2019).
- - Appendix H: A possible avenue of overcoming this limitation is considering Hessian estimation through zero’th order queries. Several works along this direction have recently appeared using Finite Differences (Cheng et al., 2021), as well as Gaussian Processes (Müller et al., 2021). We consider removing this dependency on the surrogate Hessian for future work.  

We want to especially highlight, our approach is not a general purpose approach to Bayesian Optimization. Our approach is an approach to solving difficult problems in decision making systems *using* Bayesian Optimization. General purpose HA-GP-UCB is outside the scope of this work.

Reviewer comment (Theory):
There are no proofs or lemmas for the theorems.

Author response:
We are not sure about what the reviewer means? Our proofs and lemmas were included in the supplementary materials. Could it be the case that this was not uploaded in OpenReview? If this is the case then we will reach out to the PCs in order to make sure that this will be fixed.

Reviewer comment (Novelty):
Related works on the following topics should be added: learning the dependency graph; modeling dependency graph as a MPNN; Bayesian optimization to learn the graph structure.

Author response:
RW on Learning the dependency graph;
- - We are not aware of other works that learn the dependency graph other than Gibbs sampling or evolutionary algorithms, which we have included.

RW on Modeling dependency graph as a MPNN;
- - We are not aware of any additional work which models the dependency graph in Bayesian optimization as a MPNN. If the reviewer knows of such a work, we are happy to add it. The closest such related work is that of Rolland et. al.

RW on Bayesian optimization to learn the graph structure;
- - We do not use Bayesian optimization to learn the graph structure. We use a random sampling strategy over the domain. This is described in Algorithm 4.

Reviewer comment (Reproducibility):
How to observe the surrogate Hessian of the black-box function?

Author response:
We discuss this in Appendix A: To estimate the Hessian, we used the Hessian-Vector product approximation. We relaxed the discrete portions of our metamodel policy into differentiable continuous approximation for this phase using the Sinkhorn-Knopp algorithm for the Role Assignment phase. For role interaction network connectivity, we used a sigmoid to create differentiable “soft” edges between each role.

Reviewer comment (Theory):
What is the motivation of using Hessian in BO? If you already observed Hessian, why not use gradient descent?

Author response:
The motivation for using the Hessian in BO is to generalize the theoretical arguments of Srinivas et. al., Duvenaud et. al., Kandasamy et. al. and Rolland et. al. We believed that by using the Hessian a tighter regret bound in the high-dimensional Bayesian optimization could be obtained. We're not sure what the second part of the question means. Could the reviewer clarify?

Reviewer comment (Writing):
What do you mean by JIT compilations? The meaning is not clear in the context of multi-agent policy structure.

Author response:
JIT compilation is taking the immutable (unchanging) part of the metamodel policy (Alg. 1,2,3) and combining it with the mutable (changing) part of the metamodel policy (the parameters) and constructing the actual model which takes decisions.

Reviewer comment (Experiments):
Why do you use MPNN rather than graph convolution, graph attention, or other transformer variants? These models should be more expressive.

Author response:
As we were interested in making contributions to the decision making systems field, we were not interested in using these techniques from the deep learning literature. It could certainly be the case that using these architectures could be more expressive. We are not able to comment on this as we are not in the deep learning field.

Reviewer comment (Writing):
What do the subscripts of $\theta$ mean? For example, sometimes you use $\theta_{r,i}$ and sometimes $\theta_{g,e}$ without any explanations on what $r,g,i,e$ are.

Author response:
We defined these terms implicitly when we used them. Unfortunately due to the space constraints of the conference format, this was the only way we could submit this paper to a conference.
- - These terms are defined, implicitly, in Section 4.1, 4.2, as well as in the graphical illustration of Fig. 2.
- - We also invite the reviewer to read the code included in the supplementary materials, which we have tried to make very understandable and readable.

Reviewer comment (Writing):
What does the superscript ${}^(i)$ mean in section 4.4?

Author response:
Superscript $(i)$ in Section 4.4
- - Superscript $(i)$ represents the maximal clique within the additive decomposition. Here a complex high dimensional function space being search over is simplified by assuming it is constructed through addition of functional subspaces. This allows us to use the nice properties of linearity of addition. We touch upon this further in the proof of Proposition 1 in Appendix E.

Reviewer comment (Theory):
What is the mathematical expression of regret $r(t)$ of querying Hessian?

Author response:
What is the mathematical expression of regret $r(t)$ of querying Hessian?
- - The regret mathematical expression is related to the model of computation we are analyzing in. For example, often time or space complexity is analyzed in order to understand and evaluate the quality of algorithms. Communication complexity is another complexity term that is used in distributed computing environments. In decision making algorithms, typically the key complexity is the regret, which compares the decision made to the best possible decision that could have been made with perfect information.

Reviewer comment (Writing):
What is $\mathcal U$ in Algorithm 4?

Author response:
What is $\mathcal{U}$ in Algorithm 4?
- - $\mathcal{U}$ represents the uniform random distribution that we sample from in order to learn the dependency structure.

Reviewer comment (Writing):
What is the Max-Cliques operation in Algorithm 4?

Author response:
What is the Max-Cliques operation in Algorithm 4?
- - The Max-Cliques operation in Algorithm 4 computes the maximal cliques that was discussed earlier in our response. This is related to finding the Max-Cliques that allow us to simplify the functional space which we discuss further in the proof of Proposition 1. We note that typically, this specific operation is NP-hard in time complexity, however, this is typically thought not to be a problem within the model of computation we are analyzing in (decision making).
- - This specific operation was approximated quickly using the NetworKit library.

Reviewer comment (Writing):
Writing and presentation should be improved. For example, the Introduction section is started by the following sentences: "Decision Making Systems choose sequences of actions to accomplish a goal. Multi-Agent Decision Making Systems choose actions for multiple actors working together towards a shared goal. Multi-Agent Reinforcement Learning (MARL) has emerged as a competitive approach for optimizing Decision Making Systems in the multi-agent settings" These are three independent sentences. This part should be re-written. In addition, a period is missing. Also, this sentence "We propose the usage of Bayesian Optimization (BO)" is somewhat unnatural. Please revise it. There are other cases, but I would not enumerate all of them. Please revise an article carefully.

Author response:
We are looking into how best to edit this part of the paper. We do want to highlight that the purpose of these three sentences is to highlight and show that, our specific contribution is to the field of multi-agent decision making systems, not reinforcement learning, or Bayesian optimization. As this is the case, we needed to frame and introduce all of these areas and how they're related within the introduction and how our work fits in the context of these areas.

Reviewer comment (Presentation):
Figures are too small and legends are overlapped with graphs; please see Figures 4 and 5.

Author response:
We can include larger figures which are the exact same as Figure 4 and 5 in the Appendix. As our contribution requires significant validation, we had to make the figures a bit smaller to present all of the important validation in the main text.

Reviewer comment (Novelty):
Theoretical results heavily rely on the previous work, in particular (Srinivas et al., 2010).

Author response:
We respectfully disagree. We believe that our work builds on top of Srinivas et. al., as well as Duvenaud et. al., Kandasamy et. al. and Rolland et. al. We do want to highlight that our theoretical results are novel as they provide a regret bound that scales with O(log D) with the dimension, which is significantly better than other bounds in high dimensional Bayesian optimization.

Reviewer comment (Evaluation):
I would like to ask about the results in Figure 5. First off, why do BO results only show current maxima until a particular iteration, i.e., monotonically increasing? Why are the other results fluctuated? Also, why do the results at initial iterations differ across algorithms? It seems unfair to the baseline methods. Moreover, some baseline methods are better than the proposed methods. These results seem reasonable. Why do the authors use Bayesian optimization instead of reinforcement learning in these problems?

Author response:
This is intimately related to how (in the area of decision making systems) different approaches are evaluated. In reinforcement learning based decision making systems, the policy that is evaluated is the singular policy that is being trained using reinforcement. In Bayesian optimization, the policy that is evaluated is the best found policy so far. Hence the difference between policy search (Bayesian optimization) and policy reinforcement (reinforcement learning). Our validation and our results show in resource constrained environments, with malformed or sparse reward, our approach of policy search using Bayesian optimization outperforms policy reinforcement methods such as various reinforcement learning methods we validated against.

Although it is highly unusual, and we have not seen results in reinforcement learning papers displayed in such a manner. We have created a new Appendix, Appendix K, where we replot the results from reinforcement learning approaches using the "best policy found so far" approach similar to BO. We want to point out that in practice, this is not how reinforcement learning works. However, in the spirit of giving reinforcement learning the best benefit of the doubt, we have created Appendix K with these results.

In this Appendix, we see that all our conclusions from the paper still remain and stand.

In addition, we repeat the same procedure for Table 5, and Table 6 which investigate RL and MARL under sparse or malformed reward. This is presented in Appendix L. In Appendix L, we see that all our conclusions from the paper still remain and stand as shown by Table 7 and Table 8.

We have also expanded footnote 10 to refer to Appendix J, Appendix K, and Appendix L showing these alternate presentations of data.

We hope that this helps the reviewer compare the two approaches of policy search and policy reinforcement. We also note that in all these cases, the data presented is overly favorable to RL and MARL, yet still our conclusions and observations hold.

We are also happy to help clarify any other concerns and engage with the reviewer further for this paper.

Reviewer comment (Evaluation):
Does Table 1 show the superior performance of your algorithm? To my understanding, HA-GP-UCB does not outperform some algorithms.

Author response:
In Table 1 we show the conditions under which HA-GP-UCB outperforms related work. This is related to the motivation and problem statement of the paper where we are interested in sparse or malformed rewards in cooperative multi-agent decision making scenarios.