Paper: Black-Box Gradient Matching for Reliable Offline Black-Box Optimization (ICLR-2024)

Reviewer comment (Theory):
The paper frequently refers to "the gradient fields of the Oracle (i.e., the true objective function)." It's essential to note that a true objective function doesn't inherently possess a gradient. Continual mention of the oracle's gradient is foundational to this paper, and this assumption should be explicitly clarified by the author. I think this is the key drawback. If the author can clarify this point, I will increase my score.

Author response:
**1. Assumption that the oracle has gradient.** 

We agree with the reviewer that not all oracle functions possess a gradient. 

**However, we want to point out that the implicit premise of the entire offline optimization field is that the developed techniques are meant for oracle functions which are continuous and differentiable. This premise is set the moment we set out to optimize the oracle using a differentiable surrogate such as the neural network.** 

This is the common theme in most (if not all) prior work. 

**This is also not an unreasonable assumption because attempting to optimize an oracle function with discontinuities is otherwise an ill-posed problem. Indeed, without the assumed differentiability, the oracle can be discontinued at any unseen data points at which its output behavior can be arbitrary, giving us no solid ground to relate between the training and (unseen) test output.** 

With this, we hope the reviewer would agree with us that we had in fact not used any assumptions prior work had not used, and consider re-evaluating the rating of our work.

Reviewer comment (Experiments):
The paper seems to overlook crucial baselines. It would be beneficial to reference and juxtapose the presented work against established benchmarks such as NEMO (https://arxiv.org/abs/2102.07970), CMA-ES, BO-qEI, BDI (https://arxiv.org/abs/2209.07507), and IOM (https://openreview.net/forum?id=gKe_A-DxzkH). These baselines are pivotal in this domain and warrant inclusion for a comprehensive analysis.

Author response:
**2. Discussing and providing empirical comparisons with suggested baselines.**

We would like to point out that we did compare with CMA-ES. We will include in our revision the following positioning with BO-qEI, BDI, NEMO, and IOM.

**(1)** BDI uses forward and backward mappings to distill knowledge from the offline dataset to the design. BDI employs infinite width (neural tangent kernels) neural nets as surrogate models which might be limited in representation power for some applications. 

**(2)** IOM uses domain adaptation methods to enforce representation invariance between the training dataset and the distribution of optimized designs. Figuring out the right notion of distribution over optimized designs is an open challenge and the one used by IOM (points found during gradient ascent with a given surrogate) might not be the most effective one.

**(3)** BO-qEI is standard Bayesian optimization (BO) run with a surrogate model as the objective function. 

**(4)** NEMO is an uncertainty quantification approach that uses normalized maximum-likelihood
(NML) estimator which requires quantizing the output scores y. This quantization might be sub-optimal.

Furthermore, we have also compared our method with BO-qEI & BDI and reported the results below.

**100-th percentile**

|           | Ant   | Dkitty | Hopper | Superconductor | tfbind-8 | tfbind-10 |
|-----------|-------|--------|--------|----------------|----------|-----------|
| BO-qEI    | 0.812 | 0.896  | 0.528  | 0.576          | 0.607    | 0.864     |
| BDI       | **0.967** | 0.940  | **1.706**  | **0.735**          | 0.973    | OOM       |
| MATCH-OPT | 0.931 | **0.957**  | 1.572  | 0.732          | **0.977**    | **0.924**     |

**50-th percentile**

|           | Ant   | Dkitty | Hopper | Superconductor | tfbind-8 | tfbind-10 |
|-----------|-------|--------|--------|----------------|----------|-----------|
| BO-qEI    | 0.568 | 0.883  | 0.360  | **0.490**          | 0.439    | 0.557     |
| BDI       | 0.583 | 0.870  | **0.400**  | 0.480          | **0.595**    | OOM       |
| MATCH-OPT | **0.611** | **0.887**  | 0.393  | 0.439          | 0.594    | **0.720**     |

Note: OOM means out of memory.

The results show that MATCH-OPT performs the best in 6 out of 12 cases (across both the 100-th and 50-th percentile settings) while BO-qEI only performs best in 1 out of 12 cases. BDI performs best in 5 out of 12 cases, runs out of memory in 2 out of 12 cases. Overall, MATCH-OPT appears to perform more stable than BDI and is marginally better than BDI. It is also more memory-efficient than BDI as it does run successfully in all cases, while BDI runs out of memory in 2 cases. MATCH-OPT also outperforms BO-qEI significantly in 11 out of 12 cases. These new results will be incorporated into our main text.

In addition, we would also be happy to run comparative analysis with NEMO and IOM but the code for those baselines has not been released. We hope the reviewer would sympathize with us on this point since offline optimization is a fast-growing field and providing an exhaustive comparison with all prior work, including those with no release code is too difficult. Nonetheless, we have tried our best to compare with most prior work that provides released code.

We hope the reviewer would take into account such practical difficulties and re-consider the rating of our work.
We are looking forward to hearing back from you & will be happy to continue the discussion if you still have other questions for us.

Reviewer comment (Presentation):
For better structuring, consider relocating the in-depth experimental results pertaining to the second question from the introduction to the exp section. This would make the introduction more concise and allow readers to delve into the specifics at the appropriate juncture.

Author response:
**3. Restructuring the experiment to make the introduction more concise.** We agree with the reviewer and will adjust the content accordingly in our revision.

Thank you again for the detailed review & suggestions.

Reviewer comment (Theory):
$\textbf{The optimum of the objective in Eq.(11) can not guarantee the learned gradient is close to the true gradient }$. Minimizing the term $(\Delta z - \Delta x ^\top \int _0^1\nabla g _\phi (tx + (1-t)x') dt)^2$ in Eq.(10) and Eq.(11) can not guarantee the gradient is close. To be clear, we have:  
\begin{equation}
(\Delta z - \Delta x ^\top \int _0^1\nabla g _\phi (tx + (1-t)x') dt)^2 = (\Delta x ^\top \int _0^1\nabla g  (tx + (1-t)x') dt - \Delta x ^\top \int _0^1\nabla g _\phi (tx + (1-t)x') dt)^2  
\end{equation}  
\begin{equation}
= ( \Delta x ^\top  \int _0^1 (\nabla g  (tx + (1-t)x') -  \nabla g _\phi (tx + (1-t)x') )dt   )^2
\end{equation}  
The optimum is $\Delta x ^\top  \int _0^1 (\nabla g  (tx + (1-t)x') -  \nabla g _\phi (tx + (1-t)x') )dt = 0$ not necessarily $ \nabla g (\cdot)  = \nabla g _\phi (\cdot)  $. Thus, when $\Delta x$ is orthogonal to $\int _0^1 (\nabla g  (tx + (1-t)x') -  \nabla g _\phi (tx + (1-t)x') )dt$, we get a trivial solution. The difference between $  \nabla g (\cdot)   $ and  $  \nabla g _\phi (\cdot)$ at the trivial solution can be arbitrarily large. The true objective used in the manuscript is Eq.(13) instead of the gradient matching objective (11). The objective (13) contains a standard regression objective term. According to the above issue, the reviewer guesses that the regression objective term in Eq.(13) is still a key effective component.

Author response:
**1. Optimizing Eq. (11) might not guarantee gradient match.** Thank you for the question. We will prove below that **optimizing Eq. (11) does necessarily guarantee gradient match** to address this concern. 

First, we understand the reviewer's key concern here is that there might exist a parameter $\phi$ such that the integrated vector

$\mathbf{F}_{\phi}(\mathbf{x},\mathbf{x}') \ = \ \int_0^1 \nabla g(t\mathbf{x} + (1-t)\mathbf{x}')\mathrm{d}t$ 

$- \int_0^1 \nabla g_\phi(t\mathbf{x} + (1-t)\mathbf{x}')\mathrm{d}t$

is non-zero but instead orthogonal to $\Delta\mathbf{x} = \mathbf{x} - \mathbf{x}'$. We can show that **such a parameter does not exist** in what follows.

First, following your argument, for such a parameter phi, we have 

$\mathbf{F}_{\phi}(\mathbf{x}, \mathbf{x}’)^\top(\mathbf{x} - \mathbf{x}’) = 0$ for all $(\mathbf{x}, \mathbf{x}’)$

Next, by the line integration theorem, we have:

**(1)** $g(\mathbf{x}) - g(\mathbf{x}’) = (\mathbf{x} - \mathbf{x}’)^\top \int_0^1 \nabla g(t\mathbf{x} + (1-t)\mathbf{x}’)\mathrm{d}t$

**(2)** $g_\phi(\mathbf{x}) - g_\phi(\mathbf{x}’) = (\mathbf{x} - \mathbf{x}’)^\top \int_0^1 \nabla g_\phi(t\mathbf{x} + (1-t)\mathbf{x}’)\mathrm{d}t$

for all $(\mathbf{x}, \mathbf{x}’)$. As such, combining **(1)** and **(2)** leads to

$\mathbf{F}_\phi(\mathbf{x}, \mathbf{x}’)^\top(\mathbf{x} - \mathbf{x}’) = $ 

$g(\mathbf{x}) - g(\mathbf{x}’) - g_\phi(\mathbf{x}) + g_\phi(\mathbf{x}’)$

Or equivalently, $g(\mathbf{x}) - g(\mathbf{x}’) - g_\phi(\mathbf{x}) + g_\phi(\mathbf{x}’) =  0$ since $\mathbf{F}_\phi(\mathbf{x}, \mathbf{x}’)^\top(\mathbf{x} - \mathbf{x}’) = 0$ for all $(\mathbf{x}, \mathbf{x}’)$.

We can thus rewrite the above as $g(\mathbf{x}) - g_\phi(\mathbf{x}) = g(\mathbf{x}’) - g_\phi(\mathbf{x}’)$ for all $(\mathbf{x}, \mathbf{x}’)$, which means there must exist a constant $c$ such that $g(\mathbf{x}) - g_\phi(\mathbf{x}) = c$ for all $\mathbf{x}$.

Taking gradient with respect to $\mathbf{x}$ on both sides leads to

$\nabla g(\mathbf{x}) - \nabla g_\phi(\mathbf{x}) = 0$ or equivalently, $\nabla g(\mathbf{x}) = \nabla g_\phi(\mathbf{x})$

This means for any parameter $\phi$ such that 

$\mathbf{F}_\phi(\mathbf{x}, \mathbf{x}’)^\top(\mathbf{x} - \mathbf{x}’) = 0$ 

for all $(\mathbf{x}, \mathbf{x}')$, the gradients of $g(\mathbf{x})$ and $g_\phi(\mathbf{x})$ are guaranteed to match. 

Thus, **the case that the reviewer is concerned about does not exist**. Please let us know if this has addressed your concern.

--

Furthermore, please also refer to Appendix C where we show an ablation study between the  training objective with and without the regression term. This addresses the concern of the reviewer that the performance might come solely from the regression term. This is not true according to our experiment in Appendix C: including the regression term improves the performance as expected but even without it, the performance is still very competitive, asserting the impact of the gradient match term, which matches with the implication of Theorem 1.

Reviewer comment (Theory):
$\textbf{The bound in Theorem 1 is very loose}$. The bound in Theorem 1 is trivial and loose, which exponentially grows w.r.t. the number of update steps $m$. In addition, the term $\max_{x}|| \nabla g(x) - \nabla g_\phi (x)  ||$ in Theorem 1 can grow to infinity.

Author response:
**2. The bound in Theorem 1 seems to grow exponentially in $m$ and hence, might be loose.** Thank you for another critical question. We understand the concern here is that the bound seems to grow exponentially in the no. of iterations $m$, and might be loose as a result. To address this concern, **we want to assure the reviewer that the bound will NOT grow exponentially in the no. of iterations with a standard choice of the learning rate  $\lambda$**. Here is how.

Let us choose $\lambda$ such that it is no more than $1/m$, which is the case in all our experiments where $m = 200$ and $\lambda = 10^-4$, as stated in the appendix. With this choice, we have:

$(1 + \lambda \cdot \mu)^{(m - 1)} \leq (1 + \mu / m)^{(m-1)} < (1 + \mu/m)^m$ 

which will approach $e^mu$ in the limit of $m$. Here, we use the known fact that $\mathrm{lim}_{m\rightarrow\infty}(1 + \mu/m)^m = e^\mu$ with $\mu > 0$.

As such, when $m$ is sufficiently large the bound in Theorem 1 is upper-bounded with

$m \cdot\lambda \cdot \ell \cdot (1 + \lambda \cdot \mu)^{(m - 1)} \cdot \text{gradient-gap} \simeq \ell \cdot e^\mu \cdot\text{gradient-gap}$

which asserts that the worst-case performance gap of our offline optimizer is approaching (in the limit of $m$) $\ell \cdot e^\mu \cdot \text{gradien-gap} = \mathbf{O}(\text{gradient-gap})$ which is not dependent on the number of gradient steps.

Thus, the bound is not growing exponential in $m$ and hence, it is not loose and trivial. Please let us know if this has addressed the reviewer’s concern.

Reviewer comment (Evaluation):
$\textbf{The empirical improvement is not significant}$. The empirical results are not convincing enough to demonstrate the claimed advantage of the proposed method. In Table 1 and Table 2, the proposed method does not consistently outperform other baseline methods.

Author response:
**3. The empirical improvement is not significant.** We respectfully disagree with the statement that the empirical improvement is not significant because our method does not outperform all other methods in all individual cases. 

We believe it is not a fair statement given that none of the existing methods (including DDOM [1]) has superior empirical improvement over all other methods in all individual cases. For example, DDOM outperforms all other methods in 2/6 tasks but argues that its average mean rank is better. It is thus only fair for MATCH-OPT to claim empirical significance using the exact same argument: Its average mean rank is better than all others in both Tables 1 and 2.

In addition, we note that both Reviewer **eLsu** and Reviewer **7Vbx** also agrees on the significance our empirical improvement:

**eLsu**: "The overall ranking performance of the method is commendable, underscoring its efficacy"

**7Vbx**: "The algorithm performs well over baselines both on Mean Normalized Rank and Mean normalized score metrics for both 50 percentile and 100 percentile."

Furthermore, Reviewer **ehny** also gave us a high overall score, which (we believe) also indicates agreement with our substantial empirical improvement. 

We hope the reviewer would reconsider the original assessment of our empirical work given the above.

Reviewer comment (Theory):
Could you share more insight on why organizing training data into monotonically increasing trajectories is able to mimic optimization paths? In particular, could you shed more light on the equation (13)?

Author response:
To answer your question, organizing training data into monotonically increasing trajectories encourages the model to learn the behavior of a gradient-based optimization algorithm, which in fact produces (mostly) monotonically increasing trajectories. This allows the gradient matching algorithm to focus more on strategic input pairs that are more relevant for gradient estimation.

Furthermore, Eq. (13) combines both the gradient match loss and the regression loss. The regression loss helps as a regularizer that amplifies the importance of accurate gradient matching along the sampled trajectories which encodes our bias that the optimal trajectories would be sufficiently close to one of those sampled trajectories.

Reviewer comment (Writing):
Typo: Page 9, missing reference.

Author response:
**6. Other suggestions on including missing reference & improving readability of Evaluation Methodology, Results and Discussion sections** Thank you for the suggestions. We will revise our paper according to your feedback.

Reviewer comment (Writing):
There is some repetition in the section Evaluation Methodology and section on Results and Discussion, the readability of those sections can be improved.

Author response:
**6. Other suggestions on including missing reference & improving readability of Evaluation Methodology, Results and Discussion sections** Thank you for the suggestions. We will revise our paper according to your feedback.

Reviewer comment (Experiments):
The paper does not compare the memory and time complexity of the algorithm with baselines. This will also depend on the hyperparameters of the optimization algorithm like the discretization parameter.

Author response:
**1. Compare running time with other baselines.** Thank you again for the suggestion. We will quote the memory and time complexity of the baselines in the revised version. Some of these are not made explicitly in their corresponding paper but we can look into their formulation to deduce their complexities. 

We do, however, want to remark that such complexity comparison is only tangential to our main contribution. Our main focus is on building optimizer with better and more stable performance overall, even at an affordable increase of running time. Furthermore, we want to point out that as some of the baselines (such as BONET) use an entirely different model which has a different number of parameters than ours, the complexity comparison might not be apple-to-apple. 

But regardless, we want to report the averaged running time of the baselines below.

|      | OUR  | BO-qEI | CMA-ES | ROMA | MINS | CBAS | BONET | GA | ENS-MEAN | ENS-MIN | DDOM |
|------|------|--------|--------|------|------|------|-------|----|----------|---------|------|
| Time | 4785 | 111    | 3804   | 489  | 359  | 189  | 614   | 45 | 179      | 179     | 2658 |

All reported running times are in seconds. Our algorithm incurs more time than other baselines but its total running time is still affordable in the offline setting: 4785s = 1.32hr.

Once again, **we want to emphasize that (1) it is not our claim contribution to be faster than the baseline; and (2) as we are optimizing expensive-to-evaluate objective functions (e.g., evaluating candidate materials through physical experiments), providing more stable performance (i.e., MNR) in exchange for an affordable increase (around 1hr) of offline computational time is still preferred by the practitioners.**

Reviewer comment (Evaluation):
The paper does not explicitly state its limitations compared to baselines especially since no baseline and proposed algorithm consistently outperforms the other methods on all tasks. Can the end-user make a judgment?

Author response:
**2. Limitations compared to baselines.** One potential limitation of our approach in comparison to other baselines is that our gradient match algorithm learns from pairs of data points. Thus, the total number of training pairs it needs to consume grows quadratically in the number of offline data points. For example, an offline dataset with N examples will result in a set of O(N^2) training pairs for our algorithm, which increases the training time quadratically. However, an intuition here is that training pairs are not equally informative and, in our experiments, it suffices to get competitive performance by just focusing on pairs of data along the sampled trajectories with monotonically increasing objective function values. This allows us to keep training cost linearly with respect to N. 

On another note, while it is true that none of the existing baselines (including our algorithm) outperform others on all tasks, we believe that at least on these benchmark datasets, our algorithm tends to perform most stably across all tasks, as measured by the mean averaged rank reported in each of our performance tables. This is a single metric that is computed based on the performance of all baselines across all tasks. The end-user can make a judgment based on such metrics. In practice, by looking at how existing baselines perform overall on a set of benchmark tasks that are similar to a target task, one can decide empirically which baseline is most likely to be best for the target task.

Reviewer comment (Experiments):
How does the discretization parameter affect the performance of the algorithm?

Author response:
**3.How does the discretization parameter affect the performance of the algorithm?** Our empirical inspections suggest that a discretization $\kappa = 5$ in Eq. 12 is sufficient to get good performance. Further increasing $\kappa$ only results in negligible changes in performance. In addition, increasing kappa will increase the time complexity linearly, as detailed in our Complexity Analysis paragraph following Eq. (13).

Reviewer comment (Experiments):
How do the optimization hyperparameters affect the performance, also how do you choose or tune $\alpha$, the term balancing the two loss terms?

Author response:
**4. How do you choose or tune $alpha$, the term balancing the two loss terms?** We set $\alpha = 1$ in all our experiments since the regression and gradient match terms have the same unit scale. Both can be re-interpreted in terms of output differences. For the gradient match term, the line integration theorem implies:

(1) $g(\mathbf{x}) - g(\mathbf{x}’) = (\mathbf{x} - \mathbf{x}’)^\top \int_0^1 \nabla g(t\mathbf{x} + (1-t)\mathbf{x}’)\mathrm{d}t$

(2) $g_\phi(x) - g_\phi(\mathbf{x}’) = (\mathbf{x} - \mathbf{x}’)^\top \int_0^1 \nabla g_\phi(t\mathbf{x} + (1-t)\mathbf{x}’)\mathrm{d}t$

for all $(\mathbf{x}, \mathbf{x}’)$. As such, combining (1) and (2) allows us to re-interpret the gradient loss in the summation form of $(g(\mathbf{x}) - g(\mathbf{x}’) - g_\phi(\mathbf{x}) + g_{\phi}(\mathbf{x}’))^2$ over multiple pairs of $(\mathbf{x}, \mathbf{x}’)$. This is clearly in the same unit scale as the regression loss (regardless of the task).

Reviewer comment (Experiments):
Right now, the objective contains a sum of two terms: value matching loss and gradient matching loss as shown in Fig. 2, what effect does each term have and are they of the same scale or their scales vary a lot on these tasks?

Author response:
**5. What effect do value matching loss and gradient matching loss have and are they of the same scale?** The two terms in our objective function are indeed on the same scale, regardless of the matching loss. This is due to the fact that by the line integration theorem, the gradient match loss can be re-interpreted as the difference between corresponding function outputs, which is incorporated directly into the regression loss. We have shown this in the answer to your previous question.

The effect of the terms are as follows: (1) the gradient match loss  helps in minimizing the upper-bound of the offline optimization quality as  shown by Theorem 1; and (2) the regression loss on the other hand acts as a regularizer that amplifies the importance of accurate gradient matching along the sampled trajectories which encodes our bias that the optimal trajectories would be sufficiently close to one of those sampled trajectories.