Paper: Learning Differentially Private Rewards from Human Feedback (ICLR-2024)

Reviewer comment (Novelty):
My main concern is about the technical difficulty. Given the algorithm in [1], Algorithm 1 is a standard application of objective perturbation, Algorithm 2 is a straightforward application of Gaussian mechanism, and Algorithm 3 is derived from replacing the estimations by their private counterparts. It would be better if the authors could highlight their technical contributions.  

[1] Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons.

Author response:
The main technical contribution is that standard objective perturbation (as used in Algorithm 1) requires strong-convexity of the original loss in order to achieve the best possible rates for DP. In our case, the loss is not in fact strongly convex in the standard sense (i.e. with respect to the $\ell_2$-norm, but rather is strongly convex with respect to a data-dependent norm. This makes it completely unclear if there are instantiations of these DP algorithms, that when combined will achieve DP with the same statistical rates of the non-private algorithms. Our main contribution is in showing that this is indeed possible, which requires a somewhat delicate analysis of how these different DP methods interact with strong-convexity with respect to a dataset-dependent norm.

Reviewer comment (Theory):
The sub-optimality bound has dependence $\tilde{O}(1/\sqrt{n\epsilon})$, where the dependence is not optimal and the second term could dominate if $\epsilon$ goes to 0. As is shown in [2], the additional cost due to DP could be of $\tilde{O}(1/n\epsilon)$ for empirical risk minimization. In addition, [3] shows that the additional cost due to DP could be of $\tilde{O}(1/n\epsilon)$ for offline RL tasks. What is the challenge to derive such dependence?  

[2] Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression.  

[3] Dan Qiao and Yu-Xiang Wang. Offline reinforcement learning with differential privacy.

Author response:
The additional cost of DP from [3] in the linear function approximation case includes an explicit dependence on $\frac{1}{\kappa}$, where $\kappa$ is the minimum eigenvalue of the data covariance matrix (analogous to our $\Sigma_D$). In contrast our work, precisely as in Zhu et al. (2023), requires no assumption on the eigenvalues of $\Sigma_D$, and indeed still holds when some eigenvalues are zero. This is the main source of the difference in bounds achieved in our case versus the offline RL setting of [3].

Reviewer comment (Theory):
For the MDP setting, the reward for a whole trajectory is still a linear function of $\theta$. Given that the occupancy measure $\rho$ is an input of Algorithm 3, is this setting identical to the contextual bandit setting?

Author response:
The settings are similar but not quite identical. In particular, the loss on line 1 of Algorithm 3 needs to be modified to be:

$$ \hat{J}(\pi) = \min_{\theta\in\Theta(\tilde{\theta},\lambda)} \mathbb E_{s\sim\rho_{\pi}}[\langle\theta,\phi(s,\pi(s))-v\rangle]$$
where $\rho_{\pi}$ is the occupancy measure induced by the policy $\pi$. That is, the expectation defining the loss is now over a policy-dependent distribution, rather than a fixed distribution as in the bandit case.

Reviewer comment (Theory):
In the first equation of proof of Theorem 4.2, the lambda is should be \sqrt{\lambda}.

Author response:
Thank you for your careful reading in identifying typos in the proofs. We have fixed the problems you point out and updated the paper.

Reviewer comment (Theory):
The equation right after eq. 14 is not correct in terms of \lambda.

Author response:
Thank you for your careful reading in identifying typos in the proofs. We have fixed the problems you point out and updated the paper.

Reviewer comment (Theory):
There must be some condition of the accuracy parameter \beta with respect to the privacy parameter and smoothness of the loss. This will in turn limit the value of the privacy parameter.

Author response:
> " there must be some condition of the accuracy parameter \beta with respect to the privacy parameter and smoothness of the loss."

Note that in Algorithm 1 $\beta < \frac{1}{n}$ is required for our results. Then the regularization parameter $\alpha$ is set based on the strong convexity parameter $\gamma$. Note that this setting for $\alpha$ only achieves the desired accuracy and privacy goals when $\beta < \frac{1}{n}$. The noise added to the output for privacy further depends on the relationship between $\beta$ and $\alpha$.

Note also that by Lemma A.3 the smoothness of the loss is bounded by $2L$, and we have assumed that $L$ is a fixed constant independent of $n$. Thus, extra factors depending on the smoothness are absorbed in the $O(\cdot)$ in our main theorems.

Reviewer comment (Experiments):
My major concern is the lack of experimental results. I know that this is a theoretical paper, but I do think experimental results are necessary here, for the following reasons: The purpose of introducing differential privacy to RLHF is to protect the privacy of the human data providers __in practice__. This paper's theoretical results provide upper bounds on privacy loss and the learning performance loss due to privacy guarantee, _under an idealized model_ ([Zhu et al, 2023]'s model). Humans' ranking behavior may not follow that idealized model, and the RLHF algorithms (for both the reward training and the downstream RL) used in practice are not necessarily the algorithms analyzed in this paper. Whether the theoretical results in this paper can be applied is questionable. Given that the theoretical contribution of this paper is only marginal (see below), I think an empirical contribution (trying DP on real datasets with real RLHF algorithms) is needed here.

Author response:
> "(1) How does [Zhu et al, 2023]'s idealized model capture real-world RLHF algorithms?"

One main way in which [Zhu et al, 2023]'s model captures real-world issues is in the necessity of using pessimistic policy optimization in the RLHF setting. That is, they show that one gets asymptotically worse convergence rates using standard policy optimization. Real-world RLHF algorithms generally include a KL-divergence regularizer designed to ensure that the learned policy does not deviate too far from the original LLM, which can be seen as a form of pessimism in policy optimization.

> "(2) How does adding DP to their model really inform RLHF in practice?"

The high-level takeaway of our DP results for practical RLHF is that one can maintain privacy without asymptotically affecting the amount of data that needs to be collected for RLHF to succeed. This sets a target for practical research, by demonstrating that in the theoretical setting where provable statistical gaurantees are possible, privacy for RLHF comes at no additional cost.

Reviewer comment (Novelty):
In the related work section, it appears that the author overlooked several works that also aim to provide differential privacy guarantees [1,2]. It would be valuable for the authors to provide comments on the relationship or differences between their approach and these existing works.

Author response:
> Related work: [1] Locally differentially private reinforcement learning for linear mixture markov decision processes, [2] Differentially private reinforcement learning with linear function approximation.

 Thank you for pointing out this related work, we will gladly add it to the related work section of the paper. To briefly highlight the differences with our work, note that in the RLHF setting the goal is to learn a reward function, and then train a policy using the learned reward. In contrast, the papers [1] and [2] both achieve differential privacy in standard reinforcement learning settings, where a reward function is already given as part of the interaction with the MDP. The difference in settings is significant even without differential privacy, and different algorithmic choices are required (e.g. optimistic algorithms in standard MDPs, vs pessimistic policy optimization needed for RLHF).