Paper: Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles (ICLR-2024)

Reviewer comment (Evaluation):
While the experiments are diverse, their comprehensiveness could be improved. The image generation experiment is very limited. While the three examples presented in the main paper exhibit some improvements over the baseline model, looking at additional examples in the appendix the improvements are harder to assess. Typically for image generation, some kind of human evaluation is necessary to provide a more objective evaluation.

Author response:
We acknowledge that it is indeed better to include an objective evaluation of the generated images. Regrettably, since the generation process for a single image requires a real human to provide 10-20 rounds of ranking feedback,  it is neither affordable for us to generate a sufficient amount of images using ZO-RankSGD for a rigorous evaluation nor to hire a real human for evaluation. Therefore, we wish to remain the main scope of this work within providing the first theoretical and principled framework for optimization with ranking oracles, and to inspire future research on incorporating this framework with more engineering consideration.

Reviewer comment (Experiments):
Towards the end of section 3.1 on how to choose $\mu$ in Algorithm 1, can the authors expand on how to choose $\mu$ in practice, for example in the stable diffusion experiment?

Author response:
As explained in section 3.1, the principle is to choose the $\mu$ as small as possible while human is still able to perceive the difference between generated images. For the stable diffusion experiment, we just start from a large value of $\mu$, and then gradually decrease it until it becomes a bit hard to tell the difference of generated image.

Reviewer comment (Theory):
Section 4.1 on investigating the impact of $m$ and $k$: Equation (10) seems to be dominated by $M_1(f, \mu)$ when $k$ and $m$ are of relatively large values. That means that the second variance is actually not the dominant quantity. Please let me know if my understanding is correct.

Author response:
Yes, your understanding is correct. When $k$ and $m$ are sufficiently large, the only remaining variance term is $M_1(f,\mu)$.

Reviewer comment (Experiments):
In section 4.2, the authors should compare with recent preference-based RL algorithms mentioned above. Additionally, it would be useful to include achieved rewards for standard RL algorithms with scalar rewards to show the performance gap (if any) using preference-based rewards.

Author response:
As discussed in our last response, it is tricky to compare to model-based RL algorithms. However, comparing rank-based algorithms to value-based algorithms is indeed a valuable suggestion. Therefore, similar to our experiment in section 4.1, we also include the value-based zero-order algorithm for the RL experiment in the revised manuscript. The results are interesting, ZO-RankSGD exhibits performance on par with ZO-SGD, reinforcing our findings from the experiment illustrated in Figure 3 and underscoring the effectiveness of the ranking oracle in providing substantial optimization-relevant information.

Reviewer comment (Theory):
My main concern about this paper is that the oracles are deterministic. However, in practical settings, even human labelers have a lot of stochasticity/inconsistency when generating feedback (see e.g. [1]). Hence there is a gap between the theory and practice even though the algorithm performs well in the diffusion model application. Since most of the prior works assume a stochastic oracle, this setting needs to be further justified.

Author response:
We agree with you that the extension to noisy ranking oracles is indeed important, and this is why we mention that it is an important future direction in the previous manuscript. It is essential to note that, unlike the comparison oracle introduced by Cai et al. (2022), which employs flipping probabilities to represent errors in noisy comparison feedback, formulating the errors in noisy ranking feedback is not straightforward. This is the key reason why we did not consider including the results on noisy ranking feedback.

As a response to this concern, we have added some preliminary experiments for investigating the performance of ZO-RankSGD on handling noisy ranking feedback, see Appendix C.3 of the revised manuscript for more details.

Here we provide a summary for these new results:

To simulate noisy ranking oracles for our preliminary experiments, we consider the scenario of directly adding Gaussian noise to the ground-truth function value, then construct the corresponding noisy ranking feedback based on the perturbed values.

In these new experiments, we found that ZO-RankSGD demonstrates resilience to additive noise across different levels of variance, consistently maintaining performance comparable to ZO-SGD. Notably, for the Rosenbrock function, ZO-RankSGD outperforms ZO-SGD, indicating superior robustness to additive noise. We speculate that this advantage stems from ZO-RankSGD relying solely on rank information for optimization, which may exhibit less variability under mild additive noise.

As one of the most important future directions, we hope to establish a well-defined formulation for the noisy ranking oracles and extend the theoretical analysis of ZO-RankSGD within this context.

Reviewer comment (Evaluation):
There is no rigorous quantitative results in real-world applications (i.e., the diffusion model). In the paper, there are only a few examples of images generated using ZO-RankSGD and human feedback.

Author response:
We acknowledge that it is indeed better to include a quantitative evaluation of the generated images. Unfortunately, we were unable to do that for two reasons. Firstly, it is unaffordable for us to generate a sufficient amount of images using ZO-RankSGD for a rigorous evaluation, because the generation process for a single image requires a real human to provide 10-20 rounds of ranking feedback. Secondly,  there is still no widely acknowledged metric to access the images generated from Stable Diffusion. A common way in existing literature on diffusion models is to report metrics like FID for toy datasets, and then show a few examples for Stable Diffusion. In our work, FID is not a good metric as it does not reflect human preference.

In all, the main scope of this work is only to provide the first theoretical and principled framework for optimization with ranking oracles, and to inspire future research on incorporating this framework with more engineering consideration.

Reviewer comment (Theory):
Is the convergence rate in Corollary 1 optimal in this setting?

Author response:
Yes, our convergence rate in Corollary 1 is optimal, according to the conclusion in "Optimal rates for zero-order convex optimization: the power of two function evaluations" Duchi et al.

Reviewer comment (Experiments):
Limited experments and unsupported claims: the evaluation of the algorithm's performance on noisy and uncertain ranking feedback is very limited. Also, the authors suggest furture direction for extending the algorithm to handle the aforementioned scenarios, however, they don't have any empirical results to support the claim. I would suggest the authors to provide at least some preliminary results for coroborate the argument.

Author response:
Thank you for inputting valuable feedback for this manuscript. Below is our response to your major concern.

### 1. The first concern is "the evaluation of the algorithm's performance on noisy and uncertain ranking feedback is very limited ..., they don't have any empirical results to support the claim. I would suggest the authors to provide at least some preliminary results for coroborate the argument."


   As a response to this concern, we have added some preliminary experiments for investigating the performance of ZO-RankSGD on handling noisy ranking feedback, see Appendix C.3 of the revised manuscript for more details.

   Here we provide a summary for these new results: 

   First of all, It is essential to note that, unlike the comparison oracle introduced by Cai et al. (2022), which employs flipping probabilities to represent errors in noisy comparison feedback, formulating the errors in noisy ranking feedback is not straightforward. This is the key reason why we did not consider including the results on noisy ranking feedback. 

   To simulate noisy ranking oracles for our preliminary experiments, we consider the scenario of directly adding Gaussian noise to the ground-truth function value, then construct the corresponding noisy ranking feedback based on the perturbed values. 

   In these new experiments, we found that ZO-RankSGD demonstrates resilience to additive noise across different levels of variance, consistently maintaining performance comparable to ZO-SGD. Notably, for the Rosenbrock function, ZO-RankSGD outperforms ZO-SGD, indicating superior robustness to additive noise. We speculate that this advantage stems from ZO-RankSGD relying solely on rank information for optimization, which may exhibit less variability under mild additive noise. 

   As one of the most important future directions, we hope to establish a well-defined formulation for the noisy ranking oracles and extend the theoretical analysis of ZO-RankSGD within this context. 

  ---

Reviewer comment (Experiments):
Narrow focus: the authors only focused on image generation with human feedback, which is also a bit limited. For human feedbacks, it's more commonly used in tuning language models, though there are recent works start to explore how can we incoporate human feedbacks into improving diffusion models. The method itself is interesting, and it would be a great add if the authors can show that the method is also effective for tuning language models.

Author response:
### 2. The second concern is "Narrow focus: the authors only focused on image generation with human feedback, which is also a bit limited ... it would be a great add if the authors can show that the method is also effective for tuning language models."

   We agree with you that testing our algorithm for language models is certainly an interesting extension, and we did think about it for once. Unfortunately, since language models generally require much more computing power than diffusion models, and our computing resources are very limited, we were unable to perform such an experiment. 

   While saying that, we wish to argue that not including the experiment on LLM should not be overly criticized. On one hand, from our perspective, incorporating human feedback into improving diffusion models is more urgent for now as this area is still in its infancy, while there have been extensive works on language models. On the other hand, the main scope of this work is only to provide the first theoretical and principled framework for optimization with ranking oracles, and to inspire future research on incorporating this framework with more engineering consideration.

   We sincerely hope the reviewer could understand our situation and take this explanation into the evaluation of the revised manuscript.