Gray-Box Gaussian Processes for Automated Reinforcement Learning
Abstract
Despite having achieved spectacular milestones in an array of important realworld applications, most Reinforcement Learning (RL) methods are very brittle concerning their hyperparameters.
Notwithstanding the crucial importance of setting the hyperparameters in training state-of-the-art agents, the task of hyperparameter optimization (HPO) in RL is understudied.
In this paper, we propose a novel gray-box Bayesian Optimization technique for HPO in RL, that enriches
Gaussian Processes with reward curve estimations based on generalized logistic functions.
We thus not only reason about the performance of learning algorithms, transferring information across configurations but also about epochs of the learning algorithm.
In a very large-scale experimental protocol, comprising 5 popular RL methods (DDPG, A2C, PPO, SAC, TD3), 22 environments (OpenAI Gym:
Mujoco, Atari, Classic Control), and 7 HPO baselines, we demonstrate that our method significantly outperforms current HPO practices in RL.
Introduction
While Reinforcement Learning (RL) has celebrated amazing successes in many applications [1– 5] it remains very brittle [6, 7].
The successes of RL are achieved by leading experts in the field with many years of expertise in the “art” of RL but the field does not yet provide a technology that broadly yields successes off the shelf.
A crucial hindrance for both broader impact and faster progress in research is that an RL algorithm that has been well-tuned for one problem does not necessarily work for another one; especially, optimal hyperparameters are environment-specific and must be carefully tuned in order to yield strong performance.
Despite the crucial importance of strong hyperparameter settings in RL [6, 8–10] the field of hyperparameter optimization (HPO) for RL is understudied.
The field is largely dominated by manual tuning, computationally expensive hyperparameter sweeps, or population-based training which trains many agents in parallel that exchange hyperparameters and states [11] 1 .
While these methods are feasible for large industrial research labs, they are costly, substantially increase the CO2 footprint of artificial intelligence research [12], and make it very hard for smaller industrial and academic labs to partake in RL research.
In this paper, we address this gap, developing a computationally efficient yet robust HPO method for RL.
The method we propose exploits the fact that reward curves tend to have similar shapes.
As a result, future rewards an agent collects with a given hyperparameter setting can be predicted quite well based on initial rewards, providing a computationally cheap mechanism to compare hyperparameter settings against each other.
We combine this insight in a novel gray-box Bayesian optimization method that includes a parametric reward curve extrapolation layer in a neural network for computing a Gaussian process kernel.
In a large-scale empirical evaluation using 5 popular RL methods (DDPG, A2C, PPO, SAC, TD3), 22 environments (OpenAI Gym: Mujoco, Atari, Classic Control), and 7 HPO baselines, we demonstrate that our resulting method, the Reward-Curve Gaussian Process (RCGP) , yields state-of-the-art performance across the board.
In summary, our contributions are as follows:
• We introduce a novel method for extrapolating initial reward curves of RL agents with given hyperparameters based on partial learning curves with different hyperparameters.
• We introduce RCGP, a novel Bayesian optimization method that exploits such predictions to allocate more budget to the most promising hyperparameter settings.
• We carry out the most comprehensive experimental analysis of HPO for RL we are aware of to date (including 5 popular RL agents, 22 environments and 8 methods), concluding that RGCP sets a new state of the art for optimizing RL hyperparameters in low compute budgets.
To ensure reproducibility (another issue in modern RL) and broad use of RGCP, all our code is open-sourced at https://github.com/releaunifreiburg/RCGP.git .
Method Preliminaries
Hyperparameter Optimization (HPO) focuses on discovering the best hyperparameter configuration λ ∈ Λ of a Machine Learning method.
Gray-box HPO refers to the concept of cheaper and approximate evaluations of the performance of hyperparameter configurations.
For example, we can approximately evaluate the final reward of the configurations of a deep RL method system after every epoch of stochastic gradient descent (gray-box evaluations)
[13, 14], instead of waiting for the full convergence (black-box evaluations).
The reward after a budget b (i.e. after b epochs) is defined as R ( λ, b ) : Λ × N → R + .
To address the noisieness of reward curves of RL algorithms we smooth the curves using a best-so-far transformation.
We average rewards based on windows of h training steps, and select the highest reward at any past window, as:
[Equation]
From now on, we refer to the smoothed R ( max ) as the reward R .
In addition, let us define the cost (e.g., wall-clock time) of evaluating a configuration for a specific budget as C ( λ, b ) : Λ × N → R + .
We define the history of N evaluated configurations and the respective budget as H ( K ) =
{ ( λ 1 , b 1 , R ( λ 1 , b 1 )) , . . .
, ( λ K , b K , R ( λ K , b K )) } .
A gray-box algorithm A is a policy that recommends the next configuration to evaluate and its budget as ( λ K +1 , b K +1 ) :
=
A (cid:0) H ( K ) (cid:1) where A : (Λ × N × R + ) K
→ Λ × N .
Gray-box HPO formally focuses on policies
A that are sequentially executed for as many steps (denoted K ) as needed to reach a total budget Ω .
The best policy discovers the configuration with the largest reward at any budget, as arg
[Equation]
Bayesian optimization (BO) is a very popular HPO policy that sequentially recommends hyperparameters to evaluate.
BO operates in sequences of two steps: by (i) fitting a probabilistic regression model to approximate the observed performances R ( λ, b ) of the evaluated configurations and budgets in H ; and (ii) applying an acquisition to select the next configuration to evaluate.
In the first step, we train Gaussian Processes [15] to approximate the observed performances (i.e.
R ( λ, b ) ≈ GP ( λ, b ; θ ) ) by finding the optimal GP parameters θ ∗ via MLE:
[Equation]
At the second step, we use an acquisition function α : Λ × N + → R + that scores how ”well” a previously unevaluated configuration might perform at a future budget, based on the estimation of the GP fitted above.
Typical acquisition functions, such as the Expected Improvement [15] recommend configurations with a high predicted performance (high GP posterior mean), but also explore configurations where the GP is uncertain on their performance (high posterior variance).
A naive gray-box BO can be formalized as a special HPO policy, based on a fitted GP with parameters θ ∗ from a history of i evaluations H ( i ) , that recommends the ( i + 1) -th configuration as:
[Equation]
Proposed Method
A multi-fidelity Gaussian Process can be modeled as a standard GP with an augmented feature vector z :=
[ λ, b ] ∈ Λ × N [16–18].
We present the preliminaries of our method in Appendix 2. From the history of evaluations H ( K ) we define the training features z i =
[ λ i , b i ] and their respective targets y i =
R ( λ i , b i ) for i ∈ { 1 , . .
. , K } .
A kernel function measures the similarity of features as k ( z i , z j ) : (Λ × N ) 2 → R + .
The aim of the GP is to estimate the posterior distribution of the unknown target of a new observed test instance z ∗ =
[ λ ∗ , b ∗ ] .
The covariance matrix between training features’ pairs is defined as K ( z, z ) =
[ k ( z i , z j )] ∀ i,j .
Similarly, the covariance between test-totraining features is K ( z ∗ , z ) =
[ k ( z ∗ , z i )] ∀ i , and the test-to-test one as K ( z ∗ , z ∗ ) =
k ( z ∗ , z ∗ ) .
Ultimately, the posterior prediction of the unknown test target is:
[Equation]
It was recently pointed out that a sigmoidal relationship exists between the reward curve of Reinforcement Learning methods and the optimization budget [16].
In this paper, we model the reward curve R ( λ, b ) of configuration λ at budget b as a generalized logistic function (Richard’s curve) with five coefficients [19].
Furthermore, we do not naively fit one sigmoid function on each reward curve for each hyperparameter configuration.
Instead, we propose to condition the sigmoid coefficients on the hyperparameter configurations within a multi-layer perceptron g : Λ → R 5 with weights w (where g ( λ, b ; w ) i represents the i -th output neuron) as:
[Equation]
In this paper, we propose a novel GP that exploits the pattern of the reward curve of the RL algorithm, by introducing the sigmoidal reward curve of (Equation 6).
We augment the feature space with the estimation of the reward curve as [ λ, b ] → [ λ, b, ˆ R ( λ, b ; w )] .
Therefore, the kernel becomes:
[Equation]
We train the parameters w using the established machinery of kernel learning for GPs [20] and then use this GP for gray-box HPO [16, 17].
It is worth pointing out that the acquisition function of Bayesian optimization (BO) queries the GP’s estimation of the final performance of an unevaluated configuration.
For brevity, we omit the basics of BO here, however, we refer the interested reader to Snoek et al.
In this context, the reward curve model of Equation 6 offers crucial information in estimating the full ( b max ) performance of an unknown configuration (i.e., [ λ, b max ] → [ λ, b max , ˆ R ( λ, b max ; w )] ), and enables the acquisition function of the BO algorithm (expected improvement at convergence) to discover performant hyper-parameter configurations.
Our novel gray-box HPO method is summarized by the pseudocode of Algorithm 1 in Appendix B.
Experimental Protocol
We focus on evaluating the performance of our proposed method, RCGP (Reward-Curve GP), for optimizing the hyperparameters of five popular model-free RL algorithms: PPO [21], A2C [22], [Figure]
DDPG [23], SAC [24], and TD3 [25].
In total, we consider 22 distinct Gym [26] environments, grouped into the Atari [27], Classic Control, and Mujoco [28] categories.
We denote the search spaces for the hyperparameters of each RL algorithm, and the full list of environments and their respective action space types in Appendix C.
We evaluated static hyperparameter optimization (HPO) methods by querying AutoRL-Bench 2 , which is a tabular benchmark for AutoRL that contains reward curves for three different random seeds belonging to runs of RL algorithms with every possible combination of hyperparameter values from the search spaces shown in Table 1 in Appendix C. For the dynamic HPO methods (PBT [11] and PB2 [29], details in Appendix C.1), we ran our own evaluations of the RL pipelines.
In every environment, we set the budget for all baselines to the run-time equivalent of 10 full training procedures, based on the expected run-time figures of AutoRL-Bench.
Each training procedure consists of 10 6 steps on the training environment.
All methods are evaluated for three seeds in each environment and RL algorithm.
The plots show the mean and standard deviations of the relative ranks of all methods, with the training timesteps in the x-axis.
Furthermore, we included the code for evaluating the performance of all the HPO methods in our GitHub repo https://anonymous.4open.science/r/RCGP-65CC .
Research Hypotheses and Experimental Results
Hypothesis 1 : Using the reward curve information helps in discovering efficient hyperparameters for model-free RL algorithms in the low budget regime.
We compare the performance of RCGP to RS and GP, as standard HPO baselines which do not utilize learning curve information.
We evaluate RS and GP for 10 full RL algorithm runs.
The leftmost plot in Figure 1 shows the performance comparison in terms of rank per timesteps in the training environment for the PPO search space in the Atari environments.
Initially, RS, GP, and
RCGP start the search with the same 4 hyperparameter configurations sampled uniformly at random.
RCGP queries the learning curve of evaluation returns of these initial configurations for the smallest budget of 10 5 steps on the training environment.
RS, and GP, being black-box optimization methods, query AutoRL-Bench for final evaluation returns after 10 6 training steps, for both the initial and subsequently suggested configurations.
Figures 4 to 6 (Appendix D) show the results on all the (search space, environment) combinations.
In all the cases, RCGP outperforms RS and GP within the wall-clock time budget of our experimental protocol.
We conclude that gray-box HPO is more efficient than black-box HPO in RL.
Hypothesis 2 : RCGP outperforms state-of-the-art multi-fidelity HPO methods in optimizing the hyperparameters of model-free RL algorithms.
We compare the performance of RCGP to BOHB, SMAC and DEHB, as state-of-the-art multifidelity HPO baselines.
We evaluate each method for the equivalent time of 10 full RL algorithm runs.
Initially, all four methods start the search with the same 4 hyperparameter configurations sampled uniformly at random.
The middle plot in Figure 1 shows the performance comparison on the PPO search space for the Atari environments.
Figures 7 to 9 (Appendix D) include the experiments on the PPO, A2C, DDPG, SAC, and TD3 search spaces for the Atari, Classic Control, and Mujoco environments.
In all experiments, our method RCGP on average outperforms BOHB,
SMAC, and DEHB in the low budget regime of up to 10 full function evaluations.
We therefore conclude that RCGP sets the state-of-the-art in gray-box HPO for RL.
Hypothesis 3 : Our method outperforms PBT and PB2, the state-of-the-art HPO in RL.
In this experiment, we compare our method RCGP to PBT and PB2, to assess the efficiency of our gray-box HPO technique against state-of-the-art dynamic HPO methods.
For ensuring a fair comparison, we evaluated PB2 and PBT using the recommended population size of 4 [29], leading to a budget equivalence of 4 full training routines.
In addition, all three optimization methods use an initial design consisting of the same 4 hyperparameter configurations sampled uniformly at random.
The rightmost plot in Figure 1 shows the performance comparison on the PPO search space for the
Atari environments.
The associated experiments of Figures 10 to 12 (Appendix D), include experiments on all the (environment, algorithm) combinations.
The plots show RCGP clearly outperforms
PBT and PB2 in the low budget regime of up to 4 full function evaluations.
Although PBT and PB are able to dynamically configure the hyperparameters of an RL algorithm, they require extensive parallel resources, and thus perform sub-optimally on the low budget regime.
Conclusion
Reinforcement Learning (RL) is one of the premier research sub-areas of Machine Learning, due to the impressive achievements of modern RL methods.
Unfortunately, the performance of trained RL agents depends heavily on the choice of the methods’ hyperparameters.
In this paper we introduced a novel gray-box HPO method that fits Gaussian Processes (GP) to partially-observed reward curves.
Our GP variant fuses hyperparameter configurations, budget information and reward curve models based on generalized logistic functions.
In a large-scale experimental protocol we demonstrated that our proposed method significantly advances the state-of-the-art for HPO in RL.
Especially, we largely outperform evolutionary search HPO methods in RL (PBT and PB2), as well as existing gray-box HPO techniques Acknowledgements
We acknowledge funding by European Research Council (ERC) Consolidator Grant “Deep Learning 2.0” (grant no. 101045765). Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them.
The authors acknowledge funding by The Carl Zeiss Foundation through the research network ”Responsive and
Scalable Learning for Robots
Assisting Humans” (ReScaLe) of the University of Freiburg.
A Related Work
RL training pipelines are complex and often brittle [6, 7, 10].
This makes RL difficult to use for novel applications.
To mitigate this, automated reinforcement learning [AutoRL; 31] aims to alleviate a human practitioner from the tedious and error prone task of manually setting up the RL pipeline.
While there exist different approaches to automate the choice of algorithm, architecture [32] or even environment components [33], in this work we focus on hyperparameter optimization [HPO; 34, 35] for RL.
There exist various approaches in the literature of HPO for RL [see, e.g., 36, 8, 37, 38, for a detailed survey we refer to Parker-Holder et al.
Due to the non-stationarity of RL training, in recent years, most applications of hyperparameter optimization for RL have focused on dynamically adapting hyperparameters throughout the run.
For example, population based training [PBT; 11] and variants thereof [see, e.g., 39, 29] have found more wide-spread use in the community.
This style of HPO uses a population of agents to optimize their hyperparameters while training.
Parts of the population are used to explore different hyperparameter settings while the rest are kept to exploit the so far best performing configurations.
While this has proven a successful HPO method, a drawback of population based methods is that they come with an increased compute cost due to needing to maintain a population of parallel agents.
Thus, most extensions of PBT, such as PB2 [29], aim at reducing the required population size.
Still, to guarantee sufficient exploration, larger populations might be required which makes such methods hard to use with small compute budgets.
In the field of automated machine learning [AutoML; 30], multi-fidelity optimization has gained popularity to reduce the cost of the optimization procedure.
Such methods [see, e.g., 40–45] leverage lower fidelities, such as dataset subsets, lower number of epochs or low numbers of repetitions, to quickly explore the configuration space.
For the special case of number of epochs as a fidelity, there also exists a rich literature on learning curve prediction [13, 46–49, 14].
Multi-fidelity optimization typically evaluates the most promising configurations on higher fidelities, including the full budget.
This style of optimization has proven a cost-efficient way of doing HPO for many applications.
Still, multi-fidelity optimization has been explored only little in the context of RL.
We are only aware of three such works: Runge et al.
[50] used a multi-fidelity optimizer to tune the hyperparameters of a PPO agent [21] that was tasked with learning to design RNA, allowing the so-tuned agent to substantially improve over the state of the art.
Nguyen et al.
[16] also modelled the training curves, providing a signal to guide the search.
In the realm of model-based RL, it was shown that dynamic tuning methods such as PBT can produce well-performing policies but often fail to generate robust results whereas static multi-fidelity approaches produced much more stable configurations that might not result in as high final rewards [9].
Crucially, however, these previous studies did not evaluate how multi-fidelity and PBT style methods compare in the low budget regime, a setting that is more realistic for most research groups.
B RCGP Pseudocode
We show the pseudocode in Algorithm 1.
We stress that we are concurrently training one agent for each hyperparameter configuration, but we advance only one training procedure at a time, using an intelligent selection mechanism based on Bayesian optimization.
In the first stage, we fit a GP (line 3) using the aforementioned novel kernel function that combines hyperparameter configurations, budgets, and estimated rewards (Equations 4-7).
Afterward, we select the next configuration with the highest estimated acquisition at the end of the convergence (line 4).
Then we train the RL agent corresponding to the selected configuration for one more budget increment (e.g. continue training for ∆ b = 10 b more training steps) and measure the observed reward at the end of the next budget (lines 5-6).
Note that line 5 defines the next budget for both new configurations ( ∄ λ next : ( λ next , · , · ) ∈ H ) as well as existing ones.
We add the evaluation to the history (line 7) and continue the BO procedure until no budget is left (line 3).
Input : Search space Λ , initial design H (0) , budget increment ∆ b , max budget per agent b max Output: Best hyperparameter configuration λ ∗ 1 Evaluate initial configurations and budgets H :
= H (0) ; 2 while still budget do Fit a GP on H using Equations 4-7, for estimating µ ( λ, b ) , σ 2 ( λ, b ) ; Run acquisition a ( µ, σ ) to select λ next := arg max λ ∈ Λ a (cid:0) µ ( λ, b max ) , σ 2 ( λ, b max ) (cid:1) ; Define the next budget until which to train the selected agent λ next : b next := min (cid:32) b max , (cid:40) ∆ b ∄ λ next : ( λ
next , · , · ) ∈ H ∆ b + max ( λ next ,b, · ) ∈ H b, otherwise (cid:33) ; Resume training agent with λ next until b next and measure reward R ( λ next , b next ) ; Append to history H ← H ∪ { ( λ next , b next , R ( λ next , b next )) } ; 8 end 9 return Best configuration λ ∗ with highest reward [Table]
[Table]
C Experimental Setup C.1 Baselines
We focus on comparing the performance of RCGP to existing HPO approaches within a given time budget.
We compare against three types of baselines.
The first type includes standard baselines that do not utilize fidelity information during optimization, namely:
• Random Search (RS) [51] is a simple and common HPO baseline.
It optimizes hyperparameters by selecting configurations uniformly at random.
• Bayesian optimization with Gaussian Proccesses (GP) [15] is another standard HPO baseline, using GPs as the surrogate model in standard blackbox Bayesian optimization.
We used a GPytorch [52] implementation with a Matern 5 / 2 kernel.
The second type of baselines consists of multi-fidelity baselines which exploit
intermediate learning (a.k.a. reward) curve information.
Concretely, we compare against the following multi-fidelity HPO techniques:
[Table] • BOHB [43] is a multi-fidelity HPO baseline that combines Bayesian optimization and Hyperband [41].
It uses tree-based Parzen estimators (TPE) [53] as a surrogate model for Bayesian optimization.
We used the source code provided by the authors.
• SMAC [54] is a recent variant of BOHB that uses Random Forests (RF) as a surrogate model.
Here again, we used the implementation provided by the authors.
• DEHB [45] is a state-of-the-art multi-fidelity HPO baseline that combines Differential Evolution and Hyperband.
We used the source code released by the authors.
The third type of baselines includes online HPO techniques (which apply different hyperparameter configurations within a single RL agent training procedure).
We compare against two state-of-the-art online HPO methods in RL, that are based on evolutionary search:
• Population-Based Training (PBT) [11] is an evolutionary HPO method that dynamically optimizes the hyperparameters during the run of the algorithm (i
e., RL agent training).
It discards the worst-performing members of the population after a number of steps and replaces them with new hyperparameter configurations that are generated by perturbing the best-performing configuration.
We used the PBT implementation in the Ray Tune library [55].
To facilitate a fair comparison on a small compute budget we follow the protocol of [29] and use a population of 4 individuals.
• Population-based bandits (PB2) [29] is a PBT-like dynamic HPO method.
It replaces the random perturbation with a time-varying GP, as a mechanism to identify well-performing regions of the search space.
Again, we used the implementation of PB2 in the Ray Tune library [55] with a population of 4 individuals.
D Additional Experimental Results D.1 Ablating our design choices
Throughout the paper we based our technical novelty on the hypothesis that reward curves have predictable shapes, and as a result, we can model them accurately with generalized logistic functions (Equation 6).
In this section, we ablate the effect of enriching the feature-space of our surrogate with the reward curve estimations, i [Figure] e. [ λ, b ] → [ λ, b, ˆ R ( λ, b ; w )] .
The ablations of Figure 2demonstrate that using our novel reward curve modeling offers a major boost on the quality of the optimization.
In addition, we ablate the effect of the max-smoothing transformation of the reward curves (Equation 1).
The empirical results further demonstrate that smoothing the noisy reward curves improves the performance of RCGP in the low-budget regime, especially as the early segments of reward curves are very noisy
[Figure]
[Figure]
[Figure]
information, and (iii) with max-smoothing of the reward curve curve, for the A2C search space in the Atari, Classic Control, and Mujoco environments.
D.2 Comparison to Baselines
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]