Repository: ray-project/ray
File: rllib/examples/ray_tune/appo_hyperparameter_tune.py (Python)

Code before:
This script uses Ray Tune's HyperOpt search algorithm to optimize APPO
hyperparameters for CartPole-v1 (though is applicable to any RLlib algorithm).
HyperOpt uses Tree-structured Parzen Estimators (TPE) for efficient Bayesian
optimization of the hyperparameter search space.

The script runs 4 parallel trials by default, with HyperOpt suggesting new
hyperparameter configurations based on results from completed trials.
For each trial, it defaults to using a 1 GPU per learner, meaning that
you need to be running on a cluster with 4 GPUs available.
Otherwise, we recommend users change the `num_gpus_per_learner` to zero
or the `max_concurrent_trials` to one (if only single GPU is available).

Key hyperparameters being tuned:
- lr: Learning rate
- entropy_coeff: Entropy coefficient for exploration
- vf_loss_coeff: Value function loss coefficient
- train_batch_size_per_learner: Batch size per learner
- circular_buffer_num_batches: Number of batches in circular buffer
- circular_buffer_iterations_per_batch: Replay iterations per batch
- target_network_update_freq: Target network update frequency
- broadcast_interval: Weight synchronization interval

Note on storage for multi-node clusters
---------------------------------------
Ray Tune requires centralized storage accessible by all nodes in a multi-node cluster.
This can be an S3 bucket or local storage assessable to all nodes.
If running on an Anyscale job, it has an internal S3 bucket defined by the
ANYSCALE_ARTIFACT_STORAGE environment variable.
See https://docs.ray.io/en/latest/train/user-guides/persistent-storage.html for more details.

How to run this script
----------------------
Run with 4 parallel trials (default):
`python appo_hyperparameter_tune.py`

Run with custom number of parallel trials (max-concurrent-trials) and
the total number of trials (num_samples):
`python appo_hyperparameter_tune.py --max-concurrent-trials=2 --num_samples=20`

Run on a cluster with cloud or local filesystem storage:
`python appo_hyperparameter_tune.py --storage-path=s3://my-bucket/appo-hyperopt`
`python appo_hyperparameter_tune.py --storage-path=/mnt/nfs/appo-hyperopt`

Run locally with only a single GPU
`python appo_hyperparameter_tune.py --max-concurrent-trials=1 --num_samples=5 --storage-path=/mnt/nfs/appo-hyperopt`

Results to expect
-----------------
HyperOpt will explore the hyperparameter space and converge toward configurations
that achieve reward of 475+ on CartPole within 2 million timesteps. The best
trial's hyperparameters will be logged at the end of training.

Review comment (suggestion):
```suggestion
This can be an S3 bucket or local storage accessible to all nodes.
```

Code after:
This script uses Ray Tune's BasicVariantGenerator to perform grid/random search
over APPO hyperparameters for CartPole-v1 (though is applicable to any RLlib algorithm).

BasicVariantGenerator is Tune's default search algorithm that generates trial
configurations from the search space without using historical trial results.
It supports grid search (tune.grid_search), random sampling (tune.uniform, etc.),
and combinations thereof.

Alternative Search Algorithms
-----------------------------
Ray Tune supports many search algorithms that can leverage results from previous
trials to guide the search more efficiently:

- HyperOptSearch: Bayesian optimization using Tree-structured Parzen Estimators (TPE)
- OptunaSearch: Bayesian optimization with pruning support via Optuna
- BayesOptSearch: Gaussian process-based Bayesian optimization
- AxSearch: Adaptive experimentation platform from Meta
- BlendSearch/CFO: Cost-aware optimization algorithms from Microsoft FLAML
- BOHB: Bayesian Optimization and HyperBand
- Nevergrad: Derivative-free optimization
- ZOOpt: Zeroth-order optimization

See the full list and usage examples at:
https://docs.ray.io/en/latest/tune/api/suggestion.html

Note: When using these advanced search algorithms, wrap them with ConcurrencyLimiter
to control parallelism (e.g., `ConcurrencyLimiter(HyperOptSearch(), max_concurrent=4)`).
BasicVariantGenerator has built-in concurrency control via its `max_concurrent` parameter.

The script runs 4 parallel trials by default.
For each trial, it defaults to using 1 GPU per learner, meaning that
you need to be running on a cluster with 4 GPUs available.
Otherwise, we recommend users change `num_gpus_per_learner` to zero
or `max_concurrent_trials` to one (if only single GPU is available).

Key hyperparameters being tuned:
- lr: Learning rate
- entropy_coeff: Entropy coefficient for exploration
- vf_loss_coeff: Value function loss coefficient
- train_batch_size_per_learner: Batch size per learner
- circular_buffer_num_batches: Number of batches in circular buffer
- circular_buffer_iterations_per_batch: Replay iterations per batch
- target_network_update_freq: Target network update frequency
- broadcast_interval: Weight synchronization interval

Note on storage for multi-node clusters
---------------------------------------
Ray Tune requires centralized storage accessible by all nodes in a multi-node cluster.
This can be an S3 bucket or local storage accessible to all nodes.
If running on an Anyscale job, it has an internal S3 bucket defined by the
ANYSCALE_ARTIFACT_STORAGE environment variable.