Paper: On Stationary Point Convergence of PPO-Clip (ICLR-2024)

Reviewer comment (Novelty):
I struggle to understand how this work connects with the wider research community.

Author response:
We thank the reviewer for the feedback. We are glad to learn that the reviewer acknowledges our novelty and the challenges involved in the analysis. Indeed, there are few analysis that directly tackles the vanilla version of the PPO with the clip surrogate objective. We are pleased to bring up the convergence characterizations of PPO-Clip, under a minimum set of assumptions.

With this level of popularity of PPO in particular (e.g. 5,500 search results on GitHub), we believe it is relevant to overcome the technical challenges, and bring the understanding of PPO up to the community. Such knowledge could result in a more long-term implication that invokes more new algorithms to be proposed, which could be more valuable than proposing a single variant alone.

Reviewer comment (Evaluation):
What is the implication of this work?

Author response:
We fully agree with the reviewer that it is hard to imagine the immediate use cases of the work. The primary cause is that our analysis is on the original version of PPO-Clip and does not introduce new algorithms. Nevertheless, this does not mean a lack of implication. One example is our analysis provides an understanding toward the clip operator, which is not typically used in optimization. Our analysis on the events $B_{n,k}$ and $C_{n,k}$ characterizes the effectiveness of the clip and also how often the clip is expected to happen in different stages of the optimization. Another example is our setting of $T$, which denotes the number of steps that the same old reference policy is used. If $T$ is too large, then the clip happens too often, and the gradient becomes zero too often which makes the update less efficient. If $T$ is small, then the samples are less effectively utilized, and the algorithm will take more time interacting with the environment. There will be a tradeoff between these two effects.

Reviewer comment (Theory):
Even though the aim of this paper is to tackle theoretical analysis of practical implementation of PPO, still there is some gap. Using a inner and outer loop style update seems to be far from practical implementation.

Author response:
Fortunately, our results hold for both the single-timescale version and the two-timescale version of PPO. We believe this is a strength rather than a weakness.

Reviewer comment (Theory):
Assumption 3.4. restricts the generality of $T$ and the learning rate. Providing simple examples on the learning rate, and $T$ would be helpful to understand the conditions about Assumption 3.4. For example, can we use $\frac{1}{k}$ or $\frac{1}{\sqrt{k}}$ as the step-size?

Author response:
We first wanted to explain that $T$ is a small value, something between $1$ to $10$ in practice. It is the number of off-policy PPO updates on the same reference old policy. Our notation of calling it "T" might results in some confusion that it sounds like something large.

Assumption 3.4 is only related to $k$, which approaches to infinity, and is not related to $T$, which is a small constant. The assumption is actually quite weak, which only requires the learning rate to not decay too quickly. Examples are $\epsilon_{n,k}=1/\sqrt{n}$, $\epsilon_{n,k}=1/n$, and $\epsilon_{n,k}=1/n\ln n$. Examples that do not satisfy our assumption are $\epsilon_{n,k}=1/n^2$, which decreases too fast and causes the value function update to stop before reaching a stationary point.

Reviewer comment (Theory):
Above the paragraph of Step 1, what does it mean to have similar recursive inequality like policy gradient? Please provide more details.

Author response:
For the PG algorithm, i.e., $$\theta_{n+1}=\theta_{n}+\epsilon_{n}\hat{\nabla} V_{H}(\theta_{n}),$$ we can obtain the following recursive inequality 
$$E(V^*-V(\theta_{n+1})|F_n)\le V^*-V(\theta_n)-\epsilon_n\hat{\nabla}(\theta_{n})^{\top}\nabla V(\theta_{n})+\frac{{L}\epsilon_n^2}{2}\\|\hat{\nabla}V(\theta_{n})\\|^{2}.$$
This inequality is crucial for the convergence of the PG algorithm. Therefore, a very natural approach to prove the convergence of the PPO algorithm is to construct a similar inequality to the one mentioned above, and thereby establish the convergence of PPO.

Reviewer comment (Theory):
In Step 1, how does the estimated clipped PPO loss have gradient? Should it be sub-gradient due to the non-smoothness of the clipping operator? Please provide more clarifications.

Author response:
The gradient of the clipped variable is defined as normal gradient when it is unclipped, and zero when it is clipped. This is the same as what the auto-gradient will implement in practice. The only caveat thing in analysis is the "boundary" points, where the ratio is exactly $1+\delta_0$ or $1-\delta_0$. We set the gradient of these points as zero. In fact, we can set the gradient to any value in the subgradient and the analysis still holds, because those boundary points set has a zero measure.

For the events, $X_{n,k}$ and $Y_{n,k}$ represent the errors of the object with the clip operation and the object without the clip operation, respectively. Since the remaining part after removing the clip operation (the first term on the right side of the inequality in Step 1) is a standard policy gradient, by bounding these two error terms we could obtain the final result.

Reviewer comment (Theory):
In Step 2, what is the meaning of bound of $\mathbb{E}[||X_k|| \mathcal{F} ]$? Depending on $\frac{1}{\sqrt{\pi_{\theta_{n-1,1}}(s,a)}}$ seems to be problematic. How is this term compensated? In deriving (14) where did $\frac{1}{\sqrt{\pi_{\theta_{n-1,1}(a\mid s)}}}$ go?

Author response:
We thank the reviewer for pointing this out. We did made a typo in this term in the original version of the appendix. In the revised version (updated pdf), we have corrected this typo. We additionally added the detailed derivation to clarify how the term is obtained. Please refer to lines 17-27 on page 14 of the revised appendix for the updated information.