kabiriitb/ppo_advantage_estimator

PPO Advantage Bias Under UAV Turbulence

This repository studies whether PPO performance degradation under UAV turbulence follows the same quadratic scaling law previously observed for Generalized Advantage Estimation (GAE) bias.

The original NumPy/SciPy experiment is still available in run_experiment.py. It uses a linear UAV model, an LQR controller, the exact Riccati value function, and Monte Carlo simulations to show that advantage-estimation error scales as C sigma^2. The repository now also contains a complete Stable-Baselines3 PPO training and analysis pipeline for testing the same hypothesis with learned policies.

Research Question

Does PPO performance degradation under turbulence follow the same quadratic law as GAE bias?

The framework trains independent PPO policies for:

sigma = [0.00, 0.05, 0.10, 0.20, 0.30, 0.50, 0.80, 1.00]

For each turbulence level it measures:

• Average episode return
• Success rate
• Policy loss
• Value loss
• Approximate KL divergence
• Manual GAE advantage bias

It then fits:

• Bias = C sigma^2
• Performance Drop = K sigma^p

and reports C, R^2, K, and the performance exponent p.

Environment

The Gymnasium environment is envs/uav_env.py::UAVTurbulenceEnv.

State:

[x, vx, y, vy]

Action:

[ax, ay]

Dynamics:

s_{t+1} = A s_t + B a_t + E w_t
w_t ~ N(0, sigma^2)

The matrices match the original double-integrator experiment:

A = [[1, dt, 0,  0 ],
     [0, 1,  0,  0 ],
     [0, 0,  1,  dt],
     [0, 0,  0,  1 ]]

B = [[0,  0 ],
     [dt, 0 ],
     [0,  0 ],
     [0,  dt]]

E = [0, 0, 0, 1]

Reward:

r = -(s^T Q s + a^T R a)
Q = I
R = 0.1 I

Episode length is 200 steps.

Repository Structure

.
|-- envs/
|   |-- __init__.py
|   `-- uav_env.py
|-- training/
|   |-- __init__.py
|   `-- train_ppo.py
|-- analysis/
|   |-- __init__.py
|   |-- compute_gae_bias.py
|   |-- evaluate_models.py
|   `-- fit_models.py
|-- figures/
|-- models/
|-- results/
|-- config.py
|-- run_training.py
|-- run_analysis.py
|-- run_experiment.py
|-- make_plots.py
|-- README.md
`-- requirements.txt

Installation

Use Python 3.10 or newer.

pip install -r requirements.txt

The implementation is CPU-compatible. Stable-Baselines3 will use device="cpu" by default.

Train PPO Models

Run the full research training sweep:

python run_training.py

Default PPO settings are in config.py:

• Timesteps per sigma: 1,000,000
• Policy network: 64 -> 64
• Value network: 64 -> 64
• Activation: Tanh
• gamma = 0.99
• gae_lambda = 0.95
• Seed: 0

Models are saved automatically under models/.

For a quick smoke test:

python run_training.py --sigmas 0.0 0.1 0.2 --total-timesteps 5000 --n-steps 512 --n-epochs 2

The smaller rollout length and epoch count make the smoke test much faster than the full research configuration. The training runner also stops at the requested timestep budget instead of letting Stable-Baselines3 round up to the next full rollout.

Run Analysis

After training, run:

python run_analysis.py

This writes:

• results/ppo_evaluation.csv
• results/gae_bias.csv
• results/fit_statistics.csv
• results/model_residuals.csv

For faster analysis while debugging:

python run_analysis.py --sigmas 0.0 0.1 0.2 --eval-episodes 3 --gae-episodes 5

Use at least two nonzero sigma values if you want the log-log bias plot to be generated during a smoke test.

Generate Figures

run_analysis.py generates all plots automatically. You can regenerate figures from existing result CSVs with:

python make_plots.py

Figures are saved in figures/ as both PNG and PDF:

• fig_advantage_bias_vs_sigma
• fig_ppo_return_vs_sigma
• fig_advantage_bias_vs_return
• fig_loglog_bias
• fig_residual_analysis
• fig_training_curves

Original Analytical Reproduction

The original paper-style numerical experiment is preserved:

python run_experiment.py

The analytical script still writes its original text and CSV outputs:

• results_table.csv
• regression_stats.txt
• critical_threshold.txt

The pre-rendered analytical figures from the earlier repository remain in figures/. The updated make_plots.py is now dedicated to the PPO analysis CSVs produced by run_analysis.py.

Citation

If you use this code, please cite:

@article{patil2026quadratic,
  title   = {A Quadratic Bound on PPO Advantage Bias Under UAV Turbulence},
  author  = {Sahil, Nasiruddin},
  journal = {International Journal of Engineering Research \& Technology (IJERT)},
  volume  = {15},
  number  = {05},
  year    = {2026}
}