Paper: A Region-Shrinking-Based Acceleration for Classification-Based Derivative-Free Optimization (ICLR-2024)

Reviewer comment (Experiments):
There was no clear discussion of the computational complexity of the algorithms, only an observation that CMA-ES is more expensive than SRACOS or RACE-CARS. In addition, the LLM experiment only includes part of the baselines.

Author response:
For computational complexity upper bounds on function evaluations, you can refer to Theorem 3.1 and 3.2, for SRACOS and RACE-CARS respectively. As for numerical computation, neither SRACOS nor RACE-CARS involves matrix computation, the training and replacing sub-procedures consist of assignment operations and uniform sampling. You can refer to (Liu et al., 2017; 2019b.) for more running time comparisons or try code package in supplementary material if you interest. The only different operation of RACE-CARS is region shrinking, which consist of $4n$ boolean and assignment operations, the additional computational complexity can even be neglected.

Reviewer comment (Experiments):
How expensive is RACE-CARS compared to SRACOS (and other algorithms)? Some discussion and empirical study of the computational complexity would be helpful?

Author response:
For computational complexity upper bounds on function evaluations, you can refer to Theorem 3.1 and 3.2, for SRACOS and RACE-CARS respectively. As for numerical computation, neither SRACOS nor RACE-CARS involves matrix computation, the training and replacing sub-procedures consist of assignment operations and uniform sampling. You can refer to (Liu et al., 2017; 2019b.) for more running time comparisons or try code package in supplementary material if you interest. The only different operation of RACE-CARS is region shrinking, which consist of $4n$ boolean and assignment operations, the additional computational complexity can even be neglected.

Reviewer comment (Experiments):
Did you try a BO algorithm as a baseline in the experiments? In addition, why was DE and ZO-Adam not included in the LLM experiment?

Author response:
Generally, BO is a good choice for functions with less oscillation in low dimension, but it fails in handerds dimensions (Bickel  Levina, 2004; Hall et al., 2005; Fan  Fan, 2008). In addition, randomly embedding BO was selected as a baseline in (Qian et al., 2016a;b; Hu et al., 2017), for RACOS and SRACOS respectively, and has been proven worse than baseline in our experiments. These are the reasons why we didn't try BO. We mentioned at footnote 2 in page 8, that DE and ZO-Adam are poor performed and we excluded these baselines in LLM for concision and explicit comparisons.

Reviewer comment (Theory):
$\textbf{Exponentially Growing Complexity Bound}$

The query complexity in Theorem 3.1 and 3.2 exponentially grows w.r.t. the problem dimension $n$. For example, consider $\Omega =[-1,1]^n$ and $f(\boldsymbol{x}) = ||\boldsymbol{x}||_1$, where $||\cdot||_1$ denote the $l_1$-norm, then $|\Omega_{\epsilon}| = \mathbb{P}(\Omega_{\epsilon}) \le \epsilon^n$. As a result, the term $\frac{1}{|\Omega_{\epsilon}|}$ in the complexity bound equals to $\frac{1}{|\Omega_{\epsilon}|} = (\frac{1}{\epsilon})^n$, which grows exponentially for $\epsilon < 1$.

Author response:
In order to prove universal acceleration, upper bounds in Theorem 3.1 and Theorem 3.2 are established only on the non-zero-measure assumption for objective $f.$ Consequently, the established bounds are inevitable to compromise for certain extreme objective functions. This compromise is quite common in convergence rate analysis. For instance, when we are talking about first order optimization algorithms for L-smooth objectives, the best convergence rate for convex functions is sublinear while for strongly convex functions is linear. As far as the authors' knowledge, for black-box objectives without additional assumption, curse of dimensionality is unavoidable. Fortunately, as Corollary 1 and 2 in Yu et al. (2016) show, the complexity upper bound of RACOS is polynomial about dimensionality $n$ (which we mentioned in introduction) when the objective is locally Holder continuous (the objective is not even point-wise continuous according to the definition), let alone the acceleration to SRACOS. In addition, in terms of the 4 Holder continuous synthetic functions, empirical results in appendix show the convergence complexity increases linearly w.r.t. dimensionality.

Reviewer comment (Theory):
$\textbf{Impractical Assumptions}$

The Hypothesis-Target $\eta$-Shattering assumption in Definition 3.1 together with the $\gamma$-Shrinking assumption in Definition 2.3 are too strong to be practical. The Hypothesis-Target $\eta$-Shattering assumption requires the classifier prediction to have at least a $\eta$ overlap with the true $\epsilon$-solution domain $\Omega_{\epsilon}$. In the above example, the size of $\Omega_{\epsilon}$ decays exponentially w.r.t. the dimension $n$, which requires the classifier to be very accurate or to have high positive prediction percentage ($h(x) = 1$).

Author response:
+ Firstly, it is true that $\gamma$-shrinking assumption is to some extent irrational. But it is used in the former studies and is no longer needed in this paper. Rethinks of its plausibility is one of the reasons why we propose to shrink the region proactively.

+ Secondly, hypothesis-target $\eta$-shattering assumption coincides with error-target $\theta$-dependence. It can be proven that $\eta>0$ as long as $\theta<1-\mathbb{P}(R_t)$, meaning that our new assumption is at least not stronger than former studies. Moreover, the mechanism of training sub-procedure, i.e., RACOS in former studies, has shown the ability to maintain error-target $\theta$-dependence for small $\theta,$ in other words $\eta$-overlap with $\Omega_\epsilon.$ In fact, the trained classifier at each iteration entails to encompass as much proportion of true solution as possible, i.e.,  being more precise rather than being strong or accurate. RACOS itself is a radical training method, therefore satisfying the assumption is not bothering.

+ Lastly, the proposed DFO algorithm requires application-dependent hyper-parameters tuning. Introduction of supplementary hyper-parameters, which is reasonable for black-box exploration given the "no free lunch" theorem, indicates a concrete direction towards applicable DFO algorithms. In the present study, we address hyper-parameters selection from a practical perspective (see Discussion 5.2). Consolidation of the proposed theoretical analysis is considered as an important direction for our future works.

Reviewer comment (Theory):
$\textbf{Impractical Assumptions - $\gamma$-Shrinking}$

The $\gamma$-Shrinking assumption requires the region of the positive prediction to have a $\gamma$-shrinking for all $t$. As a result, the region of the positive prediction exponentially decays w.r.t. $t$, i.e., $\gamma^t$. However, the Hypothesis-Target $\eta$-Shattering assumption requires to have at least a $\eta$ overlap with the true $\epsilon$-solution domain $\Omega_{\epsilon}$, which requires a very strong classifier for high-dimensional problems. However, how to achieve a strong classifier with limited training samples for high-dimensional problems is challenging.

Author response:
+ Firstly, it is true that $\gamma$-shrinking assumption is to some extent irrational. But it is used in the former studies and is no longer needed in this paper. Rethinks of its plausibility is one of the reasons why we propose to shrink the region proactively.

+ Secondly, hypothesis-target $\eta$-shattering assumption coincides with error-target $\theta$-dependence. It can be proven that $\eta>0$ as long as $\theta<1-\mathbb{P}(R_t)$, meaning that our new assumption is at least not stronger than former studies. Moreover, the mechanism of training sub-procedure, i.e., RACOS in former studies, has shown the ability to maintain error-target $\theta$-dependence for small $\theta,$ in other words $\eta$-overlap with $\Omega_\epsilon.$ In fact, the trained classifier at each iteration entails to encompass as much proportion of true solution as possible, i.e.,  being more precise rather than being strong or accurate. RACOS itself is a radical training method, therefore satisfying the assumption is not bothering.

+ Lastly, the proposed DFO algorithm requires application-dependent hyper-parameters tuning. Introduction of supplementary hyper-parameters, which is reasonable for black-box exploration given the "no free lunch" theorem, indicates a concrete direction towards applicable DFO algorithms. In the present study, we address hyper-parameters selection from a practical perspective (see Discussion 5.2). Consolidation of the proposed theoretical analysis is considered as an important direction for our future works.

Reviewer comment (Theory):
$\textbf{No Convergence Guarantee}$

The complexity analysis can not guarantee convergence as $t$ tends to infinity. In Theorem 3.3, the left-hand side of the inequality decays exponentially fast w.r.t $t$. However, the RHS of the inequality can not guarantee an exponential decay.

Author response:
To be frank, Theorem 3.3 plays as a sufficient but not necessary condition for Theorem 3.2, introducing a theoretic concept shrinking factor $\gamma^{n\rho}$ for empirical hyper-parameters tuning, rather than a convergence conclusion of the proposed algorithm. It is not trivial to establish a sufficient and necessary condition of Theorem 3.2, given the complexity of stochastic process coupling $Training$ and $replacing$ sub-procedures. However, according to our empirical analysis, region shrinking does not jeopardize the convergence of algorithm as long as hyper-parameters are well selected (see Discussion 5.2). Moreover, even an extremely conservative selection of hyper-parameters, i.e., $\gamma^\rho$ is extremely close to 1, achieves acceleration to SRACOS.

Reviewer comment (Experiments):
In the black-box tuning task, only the SST2 dataset is employed. The evaluation on a single dataset is not convincing enough to support the claim.

Author response:
Experiments on other datasets have been updated. We will release the new manuscript soon. Please wait patiently.

Reviewer comment (Experiments):
Could the authors please compare other datasets employed in (Sun et al. 2022) besides SST2 for better evaluation?

Author response:
Experiments on other datasets have been updated. We will release the new manuscript soon. Please wait patiently.