Group Adaptive Clipping Policy Optimization

Group-relative policy optimization methods for reinforcement learning with verifiable rewards (RLVR) typically use a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation of this design: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. In particular, rollouts with low group success, few correct solutions within a rollout group, tend to exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by uniform clipping.

To address this issue, we propose \textit{Group Adaptive Clipping Policy Optimization (GAPO)}, a simple plug-in modification to group-relative policy optimization that adapts the IS clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. Importantly, GAPO requires no reward shaping or objective modification, preserving the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama base models, GAPO consistently improves both pass@1 and pass@k over fixed clipping and advantage-shaping baselines on mathematical reasoning benchmarks.