Original: G RAY -B OX G AUSSIAN P ROCESSES FOR A UTOMATED R EINFORCEMENT L EARNING
Revised: G RAY -B OX G AUSSIAN P ROCESSES FOR A UTOMATED R EINFORCEMENT L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite having achieved spectacular milestones in an array of important realworld applications, most Reinforcement Learning (RL) methods are very brittle concerning their hyperparameters.
Revised: Despite having achieved spectacular milestones in an array of important realworld applications, most Reinforcement Learning (RL) methods are very brittle concerning their hyperparameters.

Original: Notwithstanding the crucial importance of setting the hyperparameters in training state-of-the-art agents, the task of hyperparameter optimization (HPO) in RL is understudied.
Revised: Notwithstanding the crucial importance of setting the hyperparameters in training state-of-the-art agents, the task of hyperparameter optimization (HPO) in RL is understudied.

Original: In this paper, we propose a novel gray-box Bayesian Optimization technique for HPO in RL, that enriches
Revised: In this paper, we propose a novel gray-box Bayesian Optimization technique for HPO in RL, that enriches

Original: Gaussian Processes with reward curve estimations based on generalized logistic functions.
Revised: Gaussian Processes with reward curve estimations based on generalized logistic functions.

Original: In a very large-scale experimental protocol, comprising 5 popular
Revised: In a very large-scale experimental protocol, comprising 5 popular

Original: RL methods (DDPG, A2C, PPO, SAC, TD3), 22 environments (OpenAI Gym:
Revised: RL methods (DDPG, A2C, PPO, SAC, TD3), 22 environments (OpenAI Gym:

Original: Mujoco, Atari, Classic Control), and 7 HPO baselines, we demonstrate that our method significantly outperforms current HPO practices in RL.
Revised: Mujoco, Atari, Classic Control), and 7 HPO baselines, we demonstrate that our method significantly outperforms current HPO practices in RL.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While Reinforcement Learning (RL) has celebrated amazing successes in many applications (Mnih et al., 2015; Silver et al., 2016; OpenAI, 2018; Andrychowicz et al., 2020; Degrave et al., 2022), it remains very brittle (Henderson et al., 2018; Engstrom et al., 2020).
Revised: While Reinforcement Learning (RL) has celebrated amazing successes in many applications (Mnih et al., 2015; Silver et al., 2016; OpenAI, 2018; Andrychowicz et al., 2020; Degrave et al., 2022), it remains very brittle (Henderson et al., 2018; Engstrom et al., 2020).

Original: The successes of RL are achieved by leading experts in the field with many years of expertise in the “art” of RL, but the field does not yet provide a technology that broadly yields successes off the shelf.
Revised: The successes of RL are achieved by leading experts in the field with many years of expertise in the “art” of RL, but the field does not yet provide a technology that broadly yields successes off the shelf.

Original: A crucial hindrance for both broader impact and faster progress in research is that an RL algorithm that has been well-tuned for one problem does not necessarily work for another one; especially, optimal hyperparameters are environment-specific and must be carefully tuned in order to yield strong performance.
Revised: A crucial hindrance for both broader impact and faster progress in research is that an RL algorithm that has been well-tuned for one problem does not necessarily work for another one; especially, optimal hyperparameters are environment-specific and must be carefully tuned in order to yield strong performance.

Original: Despite the crucial importance of strong hyperparameter settings in RL (Henderson et al., 2018;
Revised: Despite the crucial importance of strong hyperparameter settings in RL (Henderson et al., 2018;

Original: Chen et al., 2018; Zhang et al., 2021; Andrychowicz et al., 2021), the field of hyperparameter optimization (HPO) for RL is understudied.
Revised: Chen et al., 2018; Zhang et al., 2021; Andrychowicz et al., 2021), the field of hyperparameter optimization (HPO) for RL is understudied.

Original: The field is largely dominated by manual tuning, computationally expensive hyperparameter sweeps, or population-based training which trains many agents in parallel that exchange hyperparameters and states (Jaderberg et al., 2017).
Revised: The field is largely dominated by manual tuning, computationally expensive hyperparameter sweeps, or population-based training which trains many agents in parallel that exchange hyperparameters and states (Jaderberg et al., 2017).

Original: While these methods are feasible for large industrial research labs, they are costly, substantially increase the CO footprint of artificial intelligence research (Dhar, 2020), and make it very hard for smaller industrial and academic labs to partake in RL research.
Revised: While these methods are feasible for large industrial research labs, they are costly, substantially increase the CO footprint of artificial intelligence research (Dhar, 2020), and make it very hard for smaller industrial and academic labs to partake in RL research.

Original: In this paper, we address this gap, developing a computationally efficient yet robust HPO method for RL.
Revised: In this paper, we address this gap, developing a computationally efficient yet robust HPO method for RL.

Original: The method we propose exploits the fact that reward curves tend to have similar shapes.
Revised: The method we propose exploits the fact that reward curves tend to have similar shapes.

Original: As a result, future rewards an agent collects with a given hyperparameter setting can be predicted quite well based on initial rewards, providing a computationally cheap mechanism to compare hyperparameter settings against each other.
Revised: As a result, future rewards an agent collects with a given hyperparameter setting can be predicted quite well based on initial rewards, providing a computationally cheap mechanism to compare hyperparameter settings against each other.

Original: We combine this insight in a novel gray-boy Bayesian optimization method that includes a parametric reward curve extrapolation layer in a neural network for computing a Gaussian process kernel.
Revised: We combine this insight in a novel gray-boy Bayesian optimization method that includes a parametric reward curve extrapolation layer in a neural network for computing a Gaussian process kernel.

Original: In a large-scale empirical evaluation using 5 popular RL methods (DDPG, A2C, PPO, SAC, TD3), 22 environments (OpenAI Gym: Mujoco, Atari, Classic Control), and 7 HPO baselines, we demonstrate that our resulting method, the Reward-Curve Gaussian Process (RCGP) , yields state-of-the-art performance across the board.
Revised: In a large-scale empirical evaluation using 5 popular RL methods (DDPG, A2C, PPO, SAC, TD3), 22 environments (OpenAI Gym: Mujoco, Atari, Classic Control), and 7 HPO baselines, we demonstrate that our resulting method, the Reward-Curve Gaussian Process (RCGP) , yields state-of-the-art performance across the board.

Original: In summary, our contributions are as follows:
Revised: In summary, our contributions are as follows:

Original: • We introduce a novel method for extrapolating initial reward curves of RL agents with given hyperparameters based on partial learning curves with different hyperparameters.
Revised: • We introduce a novel method for extrapolating initial reward curves of RL agents with given hyperparameters based on partial learning curves with different hyperparameters.

Original: • We introduce RCGP, a novel Bayesian optimization method that exploits such predictions to allocate more budget to the most promising hyperparameter settings.
Revised: • We introduce RCGP, a novel Bayesian optimization method that exploits such predictions to allocate more budget to the most promising hyperparameter settings.

Original: • We carry out the most comprehensive experimental analysis of HPO for RL we are aware of to date (including 5 popular RL agents, 22 environments and 8 methods), concluding that RGCP sets a new state of the art for optimizing RL hyperparameters in low compute budgets.
Revised: • We carry out the most comprehensive experimental analysis of HPO for RL we are aware of to date (including 5 popular RL agents, 22 environments and 8 methods), concluding that RGCP sets a new state of the art for optimizing RL hyperparameters in low compute budgets.

Original: To ensure reproducibility (another issue in modern RL) and broad use of RGCP, all our code is open-sourced at https://anonymous.4open.science/r/RCGP-65CC .
Revised: To ensure reproducibility (another issue in modern RL) and broad use of RGCP, all our code is open-sourced at https://github.com/releaunifreiburg/RCGP .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: RL training pipelines are complex and often brittle (Henderson et al., 2018; Engstrom et al., 2020;
Revised: RL training pipelines are complex and often brittle (Henderson et al., 2018; Engstrom et al., 2020;

Original: Andrychowicz et al., 2021).
Revised: Andrychowicz et al., 2021).

Original: This makes RL difficult to use for novel applications.
Revised: This makes RL difficult to use for novel applications.

Original: To mitigate this, automated reinforcement learning (AutoRL; Parker-Holder et al., 2022) aims to alleviate a human practitioner from the tedious and error prone task of manually setting up the RL pipeline.
Revised: To mitigate this, automated reinforcement learning (AutoRL; Parker-Holder et al., 2022) aims to alleviate a human practitioner from the tedious and error prone task of manually setting up the RL pipeline.

Original: While there exists different approaches to automate the choice of algorithm, architecture (Miao et al., 2022) or even environment components (Gleave et al., 2021), in this work we focus on hyperparameter optimization (HPO; Feurer & Hutter, 2019; Bischl et al., 2021) for RL.
Revised: While there exists different approaches to automate the choice of algorithm, architecture (Miao et al., 2022) or even environment components (Gleave et al., 2021), in this work we focus on hyperparameter optimization (HPO; Feurer & Hutter, 2019; Bischl et al., 2021) for RL.

Original: There exist various approaches in the literature of HPO for RL (see, e.g., Eriksson et al., 2003; Chen et al., 2018; Hertel et al., 2020; Ashraf et al., 2021, for a detailed survey we refer to Parker-Holder et al.
Revised: There exist various approaches in the literature of HPO for RL (see, e.g., Eriksson et al., 2003; Chen et al., 2018; Hertel et al., 2020; Ashraf et al., 2021, for a detailed survey we refer to Parker-Holder et al.

Original: Due to the non-stationarity of RL training, in recent years, most applications of hyperparameter optimization for RL have focused on dynamically adapting hyperparameters throughout the run.
Revised: Due to the non-stationarity of RL training, in recent years, most applications of hyperparameter optimization for RL have focused on dynamically adapting hyperparameters throughout the run.

Original: For example, population based training (PBT; Jaderberg et al., 2017) and variants thereof (see, e.g., Franke et al., 2021; Parker-Holder et al., 2020) have found more wide-spread use in the community.
Revised: For example, population based training (PBT; Jaderberg et al., 2017) and variants thereof (see, e.g., Franke et al., 2021; Parker-Holder et al., 2020) have found more wide-spread use in the community.

Original: This style of HPO uses a population of agents to optimize their hyperparameters while training.
Revised: This style of HPO uses a population of agents to optimize their hyperparameters while training.

Original: Parts of the population are used to explore different hyperparameter settings while the rest are kept to exploit the so far best performing configurations.
Revised: Parts of the population are used to explore different hyperparameter settings while the rest are kept to exploit the so far best performing configurations.

Original: While this has proven a successful
Revised: While this has proven a successful

Original: HPO method, a drawback of population based methods is that they come with an increased compute cost due to needing to maintain a population of parallel agents.
Revised: HPO method, a drawback of population based methods is that they come with an increased compute cost due to needing to maintain a population of parallel agents.

Original: Thus, most extensions of PBT, such as PB2 (Parker-Holder et al., 2020), aim at reducing the required population size.
Revised: Thus, most extensions of PBT, such as PB2 (Parker-Holder et al., 2020), aim at reducing the required population size.

Original: Still, to guarantee sufficient exploration, larger populations might be required which makes such methods hard to use with small compute budgets.
Revised: Still, to guarantee sufficient exploration, larger populations might be required which makes such methods hard to use with small compute budgets.

Original: In the field of automated machine learning (AutoML; Hutter et al., 2019), multi-fidelity optimization has gained popularity to reduce the cost of the optimization procedure.
Revised: In the field of automated machine learning (AutoML; Hutter et al., 2019), multi-fidelity optimization has gained popularity to reduce the cost of the optimization procedure.

Original: Such methods (see, e.g.,
Revised: Such methods (see, e.g.,

Original: Kandasamy et al., 2017; Li et al., 2017; Klein et al., 2017a;
Revised: Kandasamy et al., 2017; Li et al., 2017; Klein et al., 2017a;

Original: Falkner et al., 2018; Li et al., 2020;
Revised: Falkner et al., 2018; Li et al., 2020;

Original: Awad et al., 2021) leverage lower fidelities, such as dataset subsets, lower number of epochs or low numbers of repetitions, to quickly explore the configuration space.
Revised: Awad et al., 2021) leverage lower fidelities, such as dataset subsets, lower number of epochs or low numbers of repetitions, to quickly explore the configuration space.

Original: For the special case of number of epochs as a fidelity, there also exists a rich literature on learning curve prediction (Swersky et al., 2014;
Revised: For the special case of number of epochs as a fidelity, there also exists a rich literature on learning curve prediction (Swersky et al., 2014;

Original: Domhan et al., 2015; Baker et al., 2017; Chandrashekaran & Lane, 2017; Klein et al., 2017b; Wistuba et al., 2022).
Revised: Domhan et al., 2015; Baker et al., 2017; Chandrashekaran & Lane, 2017; Klein et al., 2017b; Wistuba et al., 2022).

Original: Multi-fidelity optimization typically evaluates the most promising configurations on higher fidelities, including the full budget.
Revised: Multi-fidelity optimization typically evaluates the most promising configurations on higher fidelities, including the full budget.

Original: This style of optimization has proven a cost-efficient way of doing HPO for many applications.
Revised: This style of optimization has proven a cost-efficient way of doing HPO for many applications.

Original: Still, multi-fidelity optimization has been explored only little in the context of RL.
Revised: Still, multi-fidelity optimization has been explored only little in the context of RL.

Original: We are only aware of three such works: Runge et al.
Revised: We are only aware of three such works: Runge et al.

Original: (2019) used a multi-fidelity optimizer to tune the hyperparameters of a PPO agent (Schulman et al., 2017) that was tasked with learning to design RNA, allowing the so-tuned agent to substantially improve over the state of the art.
Revised: (2019) used a multi-fidelity optimizer to tune the hyperparameters of a PPO agent (Schulman et al., 2017) that was tasked with learning to design RNA, allowing the so-tuned agent to substantially improve over the state of the art.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: (2020) also modelled the training curves, providing a signal to guide the search.
Revised: (2020) also modelled the training curves, providing a signal to guide the search.

Original: In the realm of model-based RL, it was shown that dynamic tuning methods such as PBT can produce well-performing policies but often fail to generate robust results whereas static multi-fidelity approaches produced much more stable configurations that might not result in as high final rewards (Zhang et al., 2021).
Revised: In the realm of model-based RL, it was shown that dynamic tuning methods such as PBT can produce well-performing policies but often fail to generate robust results whereas static multi-fidelity approaches produced much more stable configurations that might not result in as high final rewards (Zhang et al., 2021).

Original: Crucially, however, these previous studies did not evaluate how multi-fidelity and PBT style methods compare in the low budget regime, a setting that is more realistic for most research groups.
Revised: Crucially, however, these previous studies did not evaluate how multi-fidelity and PBT style methods compare in the low budget regime, a setting that is more realistic for most research groups.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: Hyperparameter Optimization (HPO) focuses on discovering the best hyperparameter configuration λ ∈ Λ of a Machine Learning method.
Revised: Hyperparameter Optimization (HPO) focuses on discovering the best hyperparameter configuration λ ∈ Λ of a Machine Learning method.

Original: Gray-box HPO refers to the concept of cheaper and approximate evaluations of the performance of hyperparameter configurations.
Revised: Gray-box HPO refers to the concept of cheaper and approximate evaluations of the performance of hyperparameter configurations.

Original: For example, we can approximately evaluate the final reward of the configurations of a deep RL method system after every epoch of stochastic gradient descent (gray-box evaluations)
Revised: For example, we can approximately evaluate the final reward of the configurations of a deep RL method system after every epoch of stochastic gradient descent (gray-box evaluations)

Original: (Swersky et al., 2014; Wistuba et al., 2022), instead of waiting for the full convergence (black-box evaluations).
Revised: (Swersky et al., 2014; Wistuba et al., 2022), instead of waiting for the full convergence (black-box evaluations).

Original: The reward after a budget b (i.e. after b epochs) is defined as R ( λ, b ) : Λ × N → R + .
Revised: The reward after a budget b (i.e. after b epochs) is defined as R ( λ, b ) : Λ × N → R + .

Original: To address the noisieness of reward curves of RL algorithms we smooth the curves using a best-so-far transformation.
Revised: To address the noisieness of reward curves of RL algorithms we smooth the curves using a best-so-far transformation.

Original: We average rewards based on windows of h training steps, and select the highest reward at any past window, as:
Revised: We average rewards based on windows of h training steps, and select the highest reward at any past window, as:

Original: [Equation]
Revised: [Equation]

Original: From now on, we refer to the smoothed R ( max ) as the reward R .
Revised: From now on, we refer to the smoothed R ( max ) as the reward R .

Original: In addition, let us define the cost (e.g., wall-clock time) of evaluating a configuration for a specific budget as C ( λ, b ) : Λ × N → R + .
Revised: In addition, let us define the cost (e.g., wall-clock time) of evaluating a configuration for a specific budget as C ( λ, b ) : Λ × N → R + .

Original: We define the history of N evaluated configurations and the respective budget as H ( K ) =
Revised: We define the history of N evaluated configurations and the respective budget as H ( K ) =

Original: { ( λ 1 , b 1 , R ( λ 1 , b 1 )) , . . .
Revised: { ( λ 1 , b 1 , R ( λ 1 , b 1 )) , . . .

Original: , ( λ K , b K , R ( λ K , b K )) } .
Revised: , ( λ K , b K , R ( λ K , b K )) } .

Original: A gray-box algorithm A is a policy that recommends the next configuration to evaluate and its budget as ( λ K +1 , b K +1 ) :
Revised: A gray-box algorithm A is a policy that recommends the next configuration to evaluate and its budget as ( λ K +1 , b K +1 ) :

Original: =
Revised: =

Original: A (cid:0) H ( K ) (cid:1) where A : (Λ × N × R + ) K
Revised: A (cid:0) H ( K ) (cid:1) where A : (Λ × N × R + ) K

Original: → Λ × N .
Revised: → Λ × N .

Original: Gray-box HPO formally focuses on policies
Revised: Gray-box HPO formally focuses on policies

Original: A that are sequentially executed for as many steps (denoted K ) as needed to reach a total budget Ω .
Revised: A that are sequentially executed for as many steps (denoted K ) as needed to reach a total budget Ω .

Original: The best policy discovers the configuration with the largest reward at any budget, as arg max
Revised: The best policy discovers the configuration with the largest reward at any budget, as arg max

Original: [Equation]
Revised: [Equation]

Original: [Equation] (cid:0) ( λ i +1 ,
Revised: i =

Original: [Equation]
Revised: [Equation]

Original:  i +1 ) : [Equation] (cid:0)
Revised: ,...,K R (cid:0) ( λ i +1 , b i +1 ) :=

Original: [Equation]
Revised: [Equation]

Original:  ( i ) (cid:1)(cid:1) s.t.
Revised: (cid:0) H ( i ) (cid:1)(cid:1) s.t.

Original: K = [Equation] j ∈ N + Ω > (cid:80) ji =1 C ( λ i , b i ) .
Revised: K = max j ∈ N + Ω > (cid:80) ji =1 C ( λ i , b i ) .

Original: Bayesian optimization (BO) is a very popular HPO policy that sequentially recommends hyperparameters to evaluate.
Revised: Bayesian optimization (BO) is a very popular HPO policy that sequentially recommends hyperparameters to evaluate.

Original: BO operates in sequences of two steps: by (i) fitting a probabilistic regression model to approximate the observed performances R ( λ, b ) of the evaluated configurations and budgets in H ; and (ii) applying an acquisition to select the next configuration to evaluate.
Revised: BO operates in sequences of two steps: by (i) fitting a probabilistic regression model to approximate the observed performances R ( λ, b ) of the evaluated configurations and budgets in H ; and (ii) applying an acquisition to select the next configuration to evaluate.

Original: In the first step, we train Gaussian Processes (Snoek et al., 2012) to approximate the observed performances (i.e. R ( λ, b ) ≈ GP ( λ, b ; θ ) ) by finding the optimal GP parameters θ ∗ via MLE:
Revised: In the first step, we train Gaussian Processes (Snoek et al., 2012) to approximate the observed performances (i.e. R ( λ, b ) ≈ GP ( λ, b ; θ ) ) by finding the optimal GP parameters θ ∗ via MLE:

Original: [Equation]
Revised: [Equation]

Original: At the second step, we use an acquisition function α : Λ × N + → R +  that scores how ”well” a previously unevaluated configuration might perform at a future budget, based on the estimation of the GP fitted above.
Revised: At the second step, we use an acquisition function, e.g. Expected Improvement (Snoek et al., 2012) α : Λ × N + → R + . that scores how ”well” a previously unevaluated configuration might perform at a future budget, based on the estimation of the GP fitted above.

Original: Typical acquisition functions, such as the Expected Improvement (Snoek et al., 2012) recommend configurations with a high predicted performance (high GP posterior mean), but also explore configurations where the GP is uncertain on their performance (high posterior variance).

Original: A naive gray-box BO can be formalized as a special HPO policy, based on a fitted GP with parameters θ ∗ from a history of i evaluations H ( i ) , that recommends the ( i + 1) -th configuration as:
Revised: A naive gray-box BO can be formalized as a special HPO policy, based on a fitted GP with parameters θ ∗ from a history of i evaluations H ( i ) , that recommends the ( i + 1) -th configuration as:

Original: [Equation]
Revised: [Equation]

Original: 3. M ULTI - FIDELITY G AUSSIAN P ROCESSES WITH R EWARD C URVES
Revised: 3. M ULTI - FIDELITY G AUSSIAN P ROCESSES WITH R EWARD C URVES

Original: A multi-fidelity Gaussian Process can be modeled as a standard GP with an augmented feature vector z :=
Revised: A multi-fidelity Gaussian Process can be modeled as a standard GP with an augmented feature vector z :=

Original: [ λ, b ] ∈ Λ × N (Nguyen et al., 2020; Song et al., 2019; Swersky et al., 2013).
Revised: [ λ, b ] ∈ Λ × N (Nguyen et al., 2020; Song et al., 2019; Swersky et al., 2013).

Original: From the history of evaluations H ( K ) we define the training features z i =
Revised: From the history of evaluations H ( K ) we define the training features z i =

Original: [ λ i , b i ] and their respective targets y i =
Revised: [ λ i , b i ] and their respective targets y i =

Original: R ( λ i , b i ) for i ∈ { 1 , . . .
Revised: R ( λ i , b i ) for i ∈ { 1 , . . .

Original: , K } .
Revised: , K } .

Original: A kernel function measures the similarity of features as k ( z i , z j ) : (Λ × N ) 2 → R + .
Revised: A kernel function measures the similarity of features as k ( z i , z j ) : (Λ × N ) 2 → R + .

Original: The aim of the GP is to estimate the posterior distribution of the unknown target of a new observed test instance z ∗ =
Revised: The aim of the GP is to estimate the posterior distribution of the unknown target of a new observed test instance z ∗ =

Original: [ λ ∗ , b ∗ ] .
Revised: [ λ ∗ , b ∗ ] .

Original: The covariance matrix between training features’ pairs is defined as K ( z, z ) =
Revised: The covariance matrix between training features’ pairs is defined as K ( z, z ) =

Original: [ k ( z i , z j )] ∀ i,j .
Revised: [ k ( z i , z j )] ∀ i,j .

Original: Similarly, the covariance between test-to-training features 
Revised: Similarly, the covariance between test-to-training features is K ( z ∗ , z ) =

Original: Input : Search
Revised: [ k ( z ∗ , z i )] ∀ i , and the test-to-test one as K ( z ∗ , z ∗ ) =

Original: Λ , initial design H (0) , budget increment ∆ b , max budget per agent b max Output hyperparameter configuration λ ∗ do Fit estimating µ 
Revised: k ( z ∗ , z ∗ ) . Ultimately, the posterior prediction of the unknown test target is: λ ∈ Λ a µ ( λ, b max ) ,

Original: , b, b
Revised: , b b

Original: Run acquisition a to select λ next :=arg max λ ∈ Λ a (cid:0) µ ( λ, b max ) , σ 2 ( λ, b max ) (cid:1) ;
Revised: :=

Original: Define the next until which to train the selected agent λ next : b next := min (cid:32) b max , (cid:40)∆ b ∄ λ next : ( λ next , · , · ) ∈ H
Revised: (cid:32) b max , (cid:40)

Original: [Equation]
Revised: [Equation]

Original: otherwise (cid:33) ; measure 8 end
Revised: ∄ λ next : ( λ next , · , · ) ∈ H

Original: [Equation]
Revised: [Equation]

Original: is K ( z ∗ , z ) = [ k ( z ∗ , z i )] ∀ i , and the test-to-test one as K ( z ∗ , z ∗ ) = k ( z ∗ , z ∗ ) . Ultimately, the posterior prediction of the unknown test target is:
Revised: otherwise (cid:33) ;

Original: [Equation]
Revised: [Equation]

Original: It was recently pointed out that a sigmoidal relationship exists between the reward curve of Reinforcement Learning methods and the optimization budget (Nguyen et al., 2020).
Revised: It was recently pointed out that a sigmoidal relationship exists between the reward curve of Reinforcement Learning methods and the optimization budget (Nguyen et al., 2020).

Original: In this paper, we model the reward curve R ( λ, b ) of configuration λ at budget b as a generalized logistic function (Richard’s curve) with five coefficients (Richards, 1959).
Revised: In this paper, we model the reward curve R ( λ, b ) of configuration λ at budget b as a generalized logistic function (Richard’s curve) with five coefficients (Richards, 1959).

Original: Furthermore, we do not naively fit one sigmoid function on each reward curve for each hyperparameter configuration.
Revised: Furthermore, we do not naively fit one sigmoid function on each reward curve for each hyperparameter configuration.

Original: Instead, we propose to condition the sigmoid coefficients on the hyperparameter configurations within a multi-layer perceptron g : Λ → R 5 with weights w (where g ( λ, b ; w ) i represents the i -th output neuron) as:
Revised: Instead, we propose to condition the sigmoid coefficients on the hyperparameter configurations within a multi-layer perceptron g : Λ → R 5 with weights w (where g ( λ, b ; w ) i represents the i -th output neuron) as:

Original: [Equation]
Revised: [Equation]

Original: In this paper, we propose a novel GP that exploits the pattern of the reward curve of the RL algorithm, by introducing the sigmoidal reward curve of (Equation 6).
Revised: In this paper, we propose a novel GP that exploits the pattern of the reward curve of the RL algorithm, by introducing the sigmoidal reward curve of (Equation 6).

Original: We augment the feature space with the estimation of the reward curve as [ λ, b ] → [ λ, b, ˆ R ( λ, b ; w )] .
Revised: We augment the feature space with the estimation of the reward curve as [ λ, b ] → [ λ, b, ˆ R ( λ, b ; w )] .

Original: Therefore, the kernel becomes:
Revised: Therefore, the kernel becomes:

Original: [Equation]
Revised: [Equation]

Original: We train the parameters w using the established machinery of kernel learning for GPs (Wilson et al., 2016) and then use this GP for gray-box HPO (Nguyen et al., 2020; Song et al., 2019).
Revised: We train the parameters w using the established machinery of kernel learning for GPs (Wilson et al., 2016) and then use this GP for gray-box HPO (Nguyen et al., 2020; Song et al., 2019).

Original: It is worth pointing out that the acquisition function of Bayesian optimization (BO) queries the GP’s estimation of the final performance of an unevaluated configuration.
Revised: We use Expected Improvement (Snoek et al., 2012) at the highest budget b max as an acquisition function.

Original: For brevity, we omit the basics of BO here, however, we refer the interested reader to Snoek et al.
Revised: For brevity, we omit the basics of BO here, however, we refer the interested reader to Snoek et al.

Original: In this context, the reward curve model of Equation 6 offers crucial information in estimating the full ( b max ) performance of an unknown configuration (i.e., [ λ, b max ] → [ λ, b max , ˆ R ( λ, b max ; w )] ), and enables the acquisition function of the BO algorithm to discover performant hyper-parameter configurations. 
Revised: In this context, the reward curve model of Equation 6 offers crucial information in estimating the full ( b max ) performance of an unknown configuration (i.e., [ λ, b max ] → [ λ, b max , ˆ R ( λ, b max ; w )] ), and enables the acquisition function of the BO algorithm to discover performant hyper-parameter configurations. We conducted an analysis on the predictive accuracy of our surrogate in Appendix B.6.

Original: Our novel gray-box HPO method is summarized by the pseudocode of Algorithm 1.
Revised: Our novel gray-box HPO method is summarized by the pseudocode of Algorithm 1.

Original: We stress that we are concurrently training one agent for each hyperparameter configuration, but we advance only one training procedure at a time, using an intelligent selection mechanism based on Bayesian optimization.
Revised: We stress that we are concurrently training one agent for each hyperparameter configuration, but we advance only one training procedure at a time, using an intelligent selection mechanism based on Bayesian optimization.

Original: In the first stage, we fit a GP (line 3) using the aforementioned novel kernel function that combines hyperparameter configurations, budgets, and estimated rewards (Equations 4-7).
Revised: In the first stage, we fit a GP (line 3) using the aforementioned novel kernel function that combines hyperparameter configurations, budgets, and estimated rewards (Equations 4-7).

Original: Afterward, we select the next configuration with the highest estimated acquisition at the end of the convergence (line 4).
Revised: Afterward, we select the next configuration with the highest estimated acquisition at the end of the convergence (line 4).

Original: Then we train the RL agent corresponding to the selected configuration for one more budget increment (e.g. continue training for ∆ b = 10 b more training steps) and measure the observed reward at the end of the next budget (lines 5-6).
Revised: Then we train the RL agent corresponding to the selected configuration for one more budget increment (e.g. continue training for ∆ b = 10 b more training steps) and measure the observed reward at the end of the next budget (lines 5-6).

Original: Note that line 5 defines the next budget for both new configurations ( ∄ λ next : ( λ next , · , · ) ∈ H ) as well as existing ones.
Revised: Note that line 5 defines the next budget for both new configurations ( ∄ λ next : ( λ next , · , · ) ∈ H ) as well as existing ones.

Original: We add the evaluation to the history (line 7) and continue the BO procedure until no budget is left (line 3).
Revised: We add the evaluation to the history (line 7) and continue the BO procedure until no budget is left (line 3).

Original:  E XPERIMENTAL P ROTOCOL 4. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL P ROTOCOL 4. E XPERIMENTAL S ETUP

Original: We focus on evaluating the performance of our proposed method, RCGP (Reward-Curve GP), for optimizing the hyperparameters of five popular model-free RL algorithms: PPO (Schulman et al., 2017), A2C Mnih et al.
Revised: We focus on evaluating the performance of our proposed method, RCGP (Reward-Curve GP), for optimizing the hyperparameters of five popular model-free RL algorithms: PPO (Schulman et al., 2017), A2C Mnih et al.

Original: , DDPG (Lillicrap et al., 2016), SAC (Haarnoja et al., 2018), and
Revised: , DDPG (Lillicrap et al., 2016), SAC (Haarnoja et al., 2018), and

Original: TD3 (Fujimoto et al., 2018).
Revised: TD3 (Fujimoto et al., 2018).

Original: In total, we consider 22 distinct Gym (Brockman et al., 2016) environments, grouped into the Atari (Bellemare et al., 2013), Classic Control, and Mujoco (Todorov et al., 2012) categories.
Revised: In total, we consider 22 distinct Gym (Brockman et al., 2016) environments, grouped into the Atari (Bellemare et al., 2013), Classic Control, and Mujoco (Todorov et al., 2012) categories.

Original: We denote the full list of environments and their respective action space types in
Revised: We denote the full list of environments and their respective action space types in

Original: Appendix A, and we list the search spaces for the hyperparameters of each RL algorithm in Table 1.
Revised: Appendix A, and we list the search spaces for the hyperparameters of each RL algorithm in Table 1.

Original: [Table]
Revised: [Table]

Original: We evaluated static hyperparameter optimization (HPO) methods by querying AutoRL-Bench 1 , which is a tabular benchmark for AutoRL that contains reward curves for three different random seeds belonging to runs of RL algorithms with every possible combination of hyperparameter values from the search spaces shown in Table 1.
Revised: We evaluated static hyperparameter optimization (HPO) methods by querying AutoRL-Bench 1 , which is a tabular benchmark for AutoRL that contains reward curves for three different random seeds belonging to runs of RL algorithms with every possible combination of hyperparameter values from the search spaces shown in Table 1.

Original: For the dynamic HPO methods (PBT (Jaderberg et al., 2017) and PB2 (Parker-Holder et al., 2020), details in Section 4.2), we ran our own evaluations of the RL pipelines.
Revised: For the dynamic HPO methods (PBT (Jaderberg et al., 2017) and PB2 (Parker-Holder et al., 2020), details in Section 4.2) [Table] we ran our own evaluations of the RL pipelines.

Original: In every environment, we set the budget for all baselines to the run-time equivalent of 10 full training procedures, based on the expected run-time figures of AutoRL-Bench.
Revised: In every environment, we set the budget for all baselines to the run-time equivalent of 10 full training procedures, based on the expected run-time figures of AutoRL-Bench.

Original: Each training procedure consists of 10 6 steps on the training environment.
Revised: Each training procedure consists of 10 6 steps on the training environment.

Original: All methods are evaluated for ten seeds in each environment and RL algorithm.
Revised: All methods are evaluated for ten seeds in each environment and RL algorithm.

Original: The plots show the mean and standard deviations of the relative ranks of all methods, with the training timesteps in the x-axis.
Revised: The plots show the mean and standard deviations of the relative ranks of all methods, with the training timesteps in the x-axis.

Original:  Furthermore, we included the code for evaluating the performance of all the HPO methods in ourGitHub repo https://anonymous.4open.science/r/RCGP-65CC .
Revised: We detail the evaluation protocol in Appendix C and the procedure we use to generate the plots in Appendix D. Furthermore, we included the code for evaluating the performance of all the HPO methods in our

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We focus on comparing the performance of RCGP to existing HPO approaches within a given time budget.
Revised: We focus on comparing the performance of RCGP to existing HPO approaches within a given time budget.

Original: We compare against three types of baselines.
Revised: We compare against three types of baselines.

Original: The first type includes standard baselines that do not utilize fidelity information during optimization, namely:
Revised: The first type includes standard baselines that do not utilize fidelity information during optimization, namely:

Original: • Random Search (RS) (Bergstra & Bengio, 2012) is a simple and common HPO baseline.
Revised: • Random Search (RS) (Bergstra & Bengio, 2012) is a simple and common HPO baseline.

Original: It optimizes hyperparameters by selecting configurations uniformly at random.
Revised: It optimizes hyperparameters by selecting configurations uniformly at random.

Original: • Bayesian optimization with Gaussian Proccesses (GP) (Snoek et al., 2012) is another standard HPO baseline, using GPs as the surrogate model in standard blackbox Bayesian optimization.
Revised: • Bayesian optimization with Gaussian Proccesses (GP) (Snoek et al., 2012) is another standard HPO baseline, using GPs as the surrogate model in standard blackbox Bayesian optimization.

Original: We used a GPytorch (Gardner et al., 2018) implementation with a Matern 5 / 2 kernel.
Revised: We used a GPytorch (Gardner et al., 2018) implementation with a Matern 5 / 2 kernel.

Original: The second type of baselines consists of multi-fidelity baselines which exploit
Revised: The second type of baselines consists of multi-fidelity baselines which exploit

Original: intermediate learning (a.k.a. reward) curve information.
Revised: intermediate learning (a.k.a. reward) curve information.

Original: Concretely, we compare against the following multi-fidelity HPO techniques:
Revised: Concretely, we compare against the following multi-fidelity HPO techniques:

Original: • BOHB
Revised: • BOHB

Original: (Falkner et al., 2018) is a multi-fidelity HPO baseline that combines Bayesian optimization and Hyperband (Li et al., 2017).
Revised: (Falkner et al., 2018) is a multi-fidelity HPO baseline that combines Bayesian optimization and Hyperband (Li et al., 2017).

Original: It uses tree-based Parzen estimators (TPE) (Bergstra et al., 2011) as a surrogate model for Bayesian optimization.
Revised: It uses tree-based Parzen estimators (TPE) (Bergstra et al., 2011) as a surrogate model for Bayesian optimization.

Original: We used the source code provided by the authors.
Revised: We used the source code provided by the authors.

Original: • SMAC (Lindauer et al., 2022) is a recent variant of BOHB that uses Random Forests (RF) as a surrogate model.
Revised: • SMAC (Lindauer et al., 2022) is a recent variant of BOHB that uses Random Forests (RF) as a surrogate model.

Original: Here again, we used the implementation provided by the authors.
Revised: Here again, we used the implementation provided by the authors.

Original: • DEHB (Awad et al., 2021) is a state-of-the-art multi-fidelity HPO baseline that combines Differential Evolution and Hyperband.
Revised: • DEHB (Awad et al., 2021) is a state-of-the-art multi-fidelity HPO baseline that combines Differential Evolution and Hyperband.

Original: We used the source code released by the authors.
Revised: We used the source code released by the authors.

Original: The third type of baselines includes online HPO techniques (which apply different hyperparameter configurations within a single RL agent training procedure).
Revised: The third type of baselines includes online HPO techniques (which apply different hyperparameter configurations within a single RL agent training procedure).

Original: We compare against two state-of-the-art online HPO methods in RL, that are based on evolutionary search:
Revised: We compare against two state-of-the-art online HPO methods in RL, that are based on evolutionary search:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: • Population-Based Training (PBT) (Jaderberg et al., 2017) is an evolutionary HPO method that dynamically optimizes the hyperparameters during the run of the algorithm (i
Revised: • Population-Based Training (PBT) (Jaderberg et al., 2017) is an evolutionary HPO method that dynamically optimizes the hyperparameters during the run of the algorithm (i

Original: [Figure]
Revised: [Figure]

Original: e., RL agent training).
Revised: e., RL agent training).

Original: It discards the worst-performing members of the population after a number of steps and replaces them with new hyperparameter configurations that are generated by perturbing the best-performing configuration.
Revised: It discards the worst-performing members of the population after a number of steps and replaces them with new hyperparameter configurations that are generated by perturbing the best-performing configuration.

Original: We used the PBT implementation in the Ray Tune library (Liaw et al., 2018).
Revised: We used the PBT implementation in the Ray Tune library (Liaw et al., 2018).

Original: To facilitate a fair comparison on a small compute budget we follow the protocol of Parker-Holder et al.
Revised: To facilitate a fair comparison on a small compute budget we follow the protocol of Parker-Holder et al.

Original: and use a population of 4 individuals.
Revised: and use a population of 4 individuals.

Original: • Population-based bandits (PB2) (Parker-
Revised: • Population-based bandits (PB2) (Parker-Holder

Original: Holder et al., 2020) is a PBT-like dynamic HPO method.
Revised: et al., 2020) is a PBT-like dynamic HPO method.

Original: It replaces the random perturbation with a time-varying GP, as a mechanism to identify well-performing regions of the search space.
Revised: It replaces the random perturbation with a time-varying GP, as a mechanism to identify well-performing regions of the search space.

Original: Again, we used the implementation of PB2 in the Ray Tune library (Liaw et al., 2018) with a population of 4 individuals.
Revised: Again, we used the implementation of PB2 in the Ray Tune library (Liaw et al., 2018) with a population of 4 individuals.

Original:  R ESEARCH H YPOTHESES AND E XPERIMENTAL R ESULTS
Revised:  R ESEARCH H YPOTHESES AND E XPERIMENTAL R ESULTS

Original: Hypothesis 1 : Using the reward curve information helps in discovering efficient hyperparameters for model-free RL algorithms in the low budget regime.
Revised: Hypothesis 1 : Using the reward curve information helps in discovering efficient hyperparameters for model-free RL algorithms in the low budget regime.

Original: We compare the performance of RCGP to RS and GP, as standard HPO baselines which do not utilize learning curve information.
Revised: We compare the performance of RCGP to RS and GP, as standard HPO baselines which do not utilize learning curve information.

Original: We evaluate RS and GP for 10 full RL algorithm runs.
Revised: We evaluate RS and GP for 10 full RL algorithm runs.

Original: Initially, RS,
Revised: Initially, RS,

Original: GP, and RCGP start the search with the same 4 hyperparameter configurations sampled uniformly at random.
Revised: GP, and RCGP start the search with the same 4 hyperparameter configurations sampled uniformly at random.

Original: RCGP queries the learning curve of evaluation returns of these initial configurations for the smallest budget of 10 5 steps on the training environment.
Revised: RCGP queries the learning curve of evaluation returns of these initial configurations for the smallest budget of 10 5 steps on the training environment.

Original: RS, and GP, being black-box optimization methods, query AutoRL-Bench for final evaluation returns after 10 6 training steps, for both the initial and subsequently suggested configurations.
Revised: RS, and GP, being black-box optimization methods, query AutoRL-Bench for final evaluation returns after 10 6 training steps, for both the initial and subsequently suggested configurations.

Original: Figure 1 shows the comparison on the PPO search space for the Atari, Classic Control, and Mujoco classes of environments.
Revised: Figure 1 shows the comparison on the PPO search space for the Atari, Classic Control, and Mujoco classes of environments.

Original: Similarly,Figure 2 presents the
Revised: Similarly,

Original:  A2C results, while Figure 9 (Appendix B) the results on the DDPG, SAC and TD3 search spaces.
Revised: Figure 2 presents the A2C results, while Figure 9 (Appendix B) the results on the DDPG, SAC and TD3 search spaces.

Original: In all the cases, RCGP outperforms RS and GP within the wall-clock time budget of our experimental protocol.
Revised: In all the cases, RCGP outperforms RS and GP within the wall-clock time budget of our experimental protocol.

Revised: In Appendix B.2 we also compare to enhanced variants of the black-box HPO methods, which evaluate configurations at smaller budgets.

Original: We conclude that gray-box HPO is more efficient than black-box HPO in RL.
Revised: We conclude that gray-box HPO is more efficient than black-box HPO in RL.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Hypothesis 2 : RCGP outperforms state-of-the-art multi-fidelity HPO methods in optimizing the hyperparameters of model-free RL algorithms.
Revised: Hypothesis 2 : RCGP outperforms state-of-the-art multi-fidelity HPO methods in optimizing the hyperparameters of model-free RL algorithms.

Original: We compare the performance of RCGP to BOHB, SMAC and DEHB, as state-of-the-art multifidelity HPO baselines.
Revised: We compare the performance of RCGP to BOHB, SMAC and DEHB, as state-of-the-art multifidelity HPO baselines.

Original: We evaluate each method for the equivalent time of 10 full RL algorithm runs.
Revised: We evaluate each method for the equivalent time of 10 full RL algorithm runs.

Original: Initially, all four methods start the search with the same 4 hyperparameter configurations sampled uniformly at random.
Revised: Initially, all four methods start the search with the same 4 hyperparameter configurations sampled uniformly at random.

Original: Figure 3 shows the performance comparison on the PPO search space for the Atari, Classic Control, and Mujoco classes of environments.
Revised: Figure 3 shows the performance comparison on the PPO search space for the Atari, Classic Control, and Mujoco classes of environments.

Original: On the other hand Figure 4 and 10 (Appendix B) include the experiments on the A2C, DDPG, SAC [Figure] and TD3 search spaces.
Revised: On the other hand Figure 4 and 10 (Appendix B) include the experiments on the A2C, DDPG, SAC, and TD3 search spaces.

Original: In all experiments, our method RCGP on average outperforms BOHB, SMAC [Figure] and DEHB in the low budget regime of up to 10 full function evaluations.
Revised: In all experiments, our method RCGP on average outperforms BOHB, SMAC, and DEHB in the low budget regime of up to 10 full function evaluations.

Original: We therefore conclude that RCGP sets the state-of-the-art in gray-box HPO for RL.
Revised: We therefore conclude that RCGP sets the stateof-the-art in gray-box HPO for RL.

Revised: We also compared to variants of the multi-fidelity HPO baselines that use max-smoothed reward curves, as detailed in Appendix B.4.

Revised: In addition, we conducted an analysis of the gain of running the HPO procedure for longer budgets, as detailed in Appendix B.5.

Original: Hypothesis 3 : Our method outperforms PBT and PB2, the state-of-the-art HPO in RL.
Revised: Hypothesis 3 : Our method outperforms PBT and PB2, the state-of-the-art HPO in RL.

Original: In this experiment, we compare our method RCGP to PBT and PB2, to assess the efficiency of our gray-box HPO technique against state-of-the-art dynamic HPO methods.
Revised: In this experiment, we compare our method RCGP to PBT and PB2, to assess the efficiency of our gray-box HPO technique against state-of-the-art dynamic HPO methods.

Original: For ensuring a fair comparison, we evaluated PB2 and PBT using the recommended population size of 4 (Parker-Holder et al., 2020), leading to a budget equivalence of 4 full training routines.
Revised: For ensuring a fair comparison, we evaluated PB2 and PBT using the recommended population size of 4 (Parker-Holder et al., 2020), leading to a budget equivalence of 4 full training routines.

Original: In addition, all three optimization methods use an initial design consisting of the same 4 hyperparameter configurations sampled uniformly at random.
Revised: In addition, all three optimization methods use an initial design consisting of the same 4 hyperparameter configurations sampled uniformly at random.

Original: Figure 5 shows the performance comparison on the PPO search space for the Atari, Classic Control, and Mujoco environments.
Revised: Figure 5 shows the performance comparison on the PPO search space for the Atari, Classic Control, and Mujoco environments.

Original: The associated experiments of Figure 6 and 11 (Appendix B), include experiments on A2C, DDPG, SAC, and TD3.
Revised: The associated experiments of Figure 6 and 11 (Appendix B), include experiments on A2C, DDPG, SAC, and TD3.

Original: The plots show RCGP clearly outperforms PBT and PB2 in the low budget regime of up to 4 full function evaluations.
Revised: The plots show RCGP clearly outperforms PBT and PB2 in the low budget regime of up to 4 full function evaluations.

Original: Although PBT and PB2 are able to dynamically configure the hyperparameters of an RL algorithm, they require extensive parallel resources, and thus perform sub-optimally on the low budget regime.
Revised: Although PBT and PB2 are able to dynamically configure the hyperparameters of an RL algorithm, they require extensive parallel resources, and thus perform sub-optimally on the low budget regime.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 5. A BLATING OUR DESIGN CHOICES
Revised: 5. A BLATING OUR DESIGN CHOICES

Original: Throughout the paper we based our technical novelty on the hypothesis that reward curves have predictable shapes
Revised: Throughout the paper we based our technical novelty on the hypothesis that reward curves have predictable shapes

Original: [Figure]
Revised: [Figure]

Original: and as a result, we can model them accurately with generalized logistic functions (Equation 6).
Revised: and as a result, we can model them accurately with generalized logistic functions (Equation 6).

Original: In this section, we ablate the effect of enriching the feature-space of our surrogate with the reward curve estimations, i.e. [ λ, b ] → [ λ, b, ˆ R ( λ, b ; w )] .
Revised: In this section, we ablate the effect of enriching the feature-space of our surrogate with the reward curve estimations, i.e. [ λ, b ] → [ λ, b, ˆ R ( λ, b ; w )] .

Original: The ablations of Figure 7demonstrate that using our novel reward curve modeling offers a major boost on the quality of the optimization.
Revised: The ablations of Figure 7demonstrate that using our novel reward curve modeling offers a major boost on the quality of the optimization.

Original: In addition, we ablate the effect of the max-smoothing transformation of the reward curves (Equation 1).
Revised: In addition, we ablate the effect of the max-smoothing transformation of the reward curves (Equation 1).

Original: The empirical results further demonstrate that smoothing the noisy reward curves improves the performance of RCGP in the low-budget regime, especially as the early segments of reward curves are very noisy.
Revised: The empirical results further demonstrate that smoothing the noisy reward curves improves the performance of RCGP in the low-budget regime, especially as the early segments of reward curves are very noisy.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Reinforcement Learning (RL) is one of the premier research sub-areas of Machine Learning, due to the impressive achievements of modern RL methods.
Revised: Reinforcement Learning (RL) is one of the premier research sub-areas of Machine Learning, due to the impressive achievements of modern RL methods.

Original: Unfortunately, the performance of trained RL agents depends heavily on the choice of the methods’ hyperparameters.
Revised: Unfortunately, the performance of trained RL agents depends heavily on the choice of the methods’ hyperparameters.

Original: In this paper we introduced a novel gray-box HPO method that fits Gaussian Processes (GP) to partially-observed reward curves.
Revised: In this paper we introduced a novel gray-box HPO method that fits Gaussian Processes (GP) to partially-observed reward curves.

Original: Our GP variant fuses hyperparameter configurations, budget information and reward curve models based on generalized logistic functions.
Revised: Our GP variant fuses hyperparameter configurations, budget information and reward curve models based on generalized logistic functions.

Original: In a large-scale experimental protocol we demonstrated that our proposed method significantly advances the state-of-the-art for HPO in RL.
Revised: In a large-scale experimental protocol we demonstrated that our proposed method significantly advances the state-of-the-art for HPO in RL.

Original: Especially, we largely outperform evolutionary search HPO methods in RL (PBT and PB2), as well as existing gray-box HPO techniques. 
Revised: Especially, we largely outperform evolutionary search HPO methods in RL (PBT and PB2), as well as existing gray-box HPO techniques. A CKNOWLEDGEMENTS

Revised: We would like to acknowledge the grant awarded by the Eva-Mayr-Stihl Stiftung.

Revised: In addition, this research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828.

Revised: In addition, Josif Grabocka acknowledges the support of the

Revised: BrainLinks-BrainTools center of excellence.

Original: Reproducibility Statement.
Revised: We acknowledge funding by European Research Council (ERC) Consolidator Grant “Deep Learning 2.0” (grant no. 101045765). Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC.

Revised: Neither the European Union nor the ERC can be held responsible for them.

Revised: The authors acknowledge funding by The Carl Zeiss Foundation through the research network ”Responsive and Scalable Learning for Robots

Revised: Assisting Humans” (ReScaLe) of the University of Freiburg.

Original:  To ensure reproducibility, we give detailed descriptions of the experimental setup including the search spaces, environments, and seeds in Section 4 and
Revised: Reproducibility Statement. To ensure reproducibility, we give detailed descriptions of the experimental setup including the search spaces, environments and seeds in Section 4 and

Original: Appendix A.
Revised: Appendix A.

Original: We also make available the code we used to generate the results that we present in this paper.
Revised: We also make available the code we used to generate the results that we present in this paper.

Original: The benchmark we use is publicly available, and we reference it correspondingly.
Revised: The benchmark we use is publicly available, and we reference it correspondingly

Original: A RL E NVIRONMENTS
Revised: A RL E NVIRONMENTS

Original: [Table]
Revised: [Table]

Original: B A DDITIONAL E XPERIMENTAL R ESULTS
Revised: B A DDITIONAL E XPERIMENTAL R ESULTS

Revised: In this Section we present additional experimental results, including other search spaces (DDPG,

Original: [Figure]
Revised: SAC [Table]

Revised: and TD3) in B.1, comparisons to the black-box HPO baselines modified to run suggested configurations for a smaller budget than the budget of a full RL algorithm training procedure in

Revised: B.2, comparison to PBT running on a continuous version of the PPO search space in B.

Revised: [Table]

Revised: as well as

Revised: average episodic reward plots comparing PBT, PB2, SMAC, BOHB,

Revised: DEHB

Revised: [Table]

Revised: and RCGP in B.4.

Revised: We additionally show average normalized regret plots comparing SMAC, BOHB, DEHB

Revised: [Table]

Revised: and RCGP in

Revised: B.5.

Revised: In B.6, we evaluate the predictive performance of RCGP with the max-smoothing transformation of the learning curves, and the version trained using non-transformed learning curves.

Revised: B. R ESULTS ON OTHER SEARCH SPACES

Revised: In Figures

Revised: 9- 11 we show the average rank for the baselines and RCGP across 10 seeds and all

Revised: MuJoCo environments for the DDPG, SAC

Revised: [Table]

Revised: and TD3 search spaces

Revised: . B. C OMPARISON TO B LACK - BOX B ASELINES

Revised: In Figures 12- 20 we show the average rank for the black-box HPO baselines with a modified search protocol and RCGP.

Revised: To allow the black-box baselines to search for more configurations within the maximum search budget, during the search the suggested configurations are evaluated for 18 , 14 and2 of a full RL algorithm training procedure.

Revised: During plotting, we still adhere to the same plotting protocol as indicated in Algorithm 3.

Revised: B. S ENSIBILITY OF S EARCH S PACE D ISCRETIZATION

Original: Figure 12 shows a comparison of RCGP, PBT optimizing on the discrete search space of PPO (as specified in Table 1), as well as PBT optimizing on a continuous version of this search spaces, with  the same bounds as the discrete version.
Revised: Figure 21 shows a comparison of RCGP, PBT optimizing on the discrete search space of PPO (as specified in Table 1), as well as PBT optimizing on a continuous version of this search space, with [Figure] [Figure] the same bounds as the discrete version.

Original: We label the latter as PBT-cont .
Revised: We label the latter as PBT-cont .

Original: The performance of PBT is visibly similar compared to PBT-cont on the MUJOCO environments, thus showing the sensibility of the discretization. 
Revised: The performance of PBT is visibly similar compared to PBT-cont on the MUJOCO environments, thus showing the sensibility of the discretization. B. A VERAGE E PISODIC R EWARD P LOTS In Figures

Revised: [Figure]

Revised: we show the average episodic reward across the search budget for PBT, PB

Revised: [Figure]

Original: C A UTO RL-B ENCH R EWARD C URVES
Revised: and RCGP in each environment for each search space. Figures 27- 31, show the average episodic reward across the search budget for SMAC, BOHB, DEHB

Revised: [Figure]

Revised: and RCGP in each environment for each search space.

Revised: All these methods use non-transformed episodic reward curves.

Revised: We show the comparison in terms of average episodic reward of the grey-box baselines and RCGP using max-smoothed episodic reward curves in Figures 32- 36.

Revised: B. N ORMALIZED R EGRET P LOTS

Revised: A common metric in the HPO community for evaluating the distance of the discovered configuration (a.k.a. incumbent) to the optimal configuration (a.k.a. oracle) is the regret defined as (oracle - incumbent).

Revised: If we let the HPO run longer we are going to discover the oracle in the best-case scenario.

Revised: In our tabular benchmark, the oracle is known (we can argmax the final returns of all possible hyperparameter configurations), therefore we can directly assess the potential gain of running longer HPO by computing the regret.

Revised: Furthermore, a percentual difference variant is (oracle - incumbent)/oracle, which shows the potential gain in percentage.

Revised: We show the performance of SMAC, BOHB, DEHB, and RCGP in terms of average percentual regret for each environment in all search spaces in Figures 37- 41.

Revised: All four methods use maxsmoothed episodic reward curves.

Revised: As can be observed the percentual regret is between 10% and 1% .

Revised: [Figure]

Revised: [Figure]

Revised: Therefore

Revised: [Figure]

Revised: letting our HPO run till infinity will improve the results between 1% to 10% in the bestcase scenario..

Revised: B. P REDICTIVE P ERFORMANCE OF RCGP

Revised: In Figure 42 we show the predictive performance, and the rank correlation of RCGP.

Revised: The plots show as y-axis (i) the predictive error (square error) and (ii) the Pearson correlation.

Revised: The x-axis indicates the length of the reward curves which we train RCGP on, in order to estimate the final return.

Revised: The ground-truth final return is the final return of the non-max-smoothed reward curve.

Revised: In terms of predictive accuracy, the plots show that the error decreases as a longer segment of the reward curve is observed.

Revised: The squared error of RCGP using max-smoothed curves for training is higher than the version using the original reward curve to train.

Revised: However, in terms of the correlation of ranks of the predicted vs. the ground truth values of the final return, RCGP with max-smoothing has a significantly larger correlation between estimated and final return values.

Revised: As the correlation is more essential for HPO than the forecasting loss, these plots provide a strong analysis explaining the superiority of our method in terms of HPO performance.

Revised: C E VALUATION P ROTOCOL

Revised: We detail the evaluation protocol we use for all baselines and RCGP in Algorithm 2.

Revised: D P LOTTING P ROTOCOL

Revised: The detailed procedure that we use to generate the plots showing the rank of the methods is shown in Algorithm 3.

Revised: [Figure]

Revised: [Figure]

Original:  To further motivate our design choice for modeling the reward curves using a generalised logistic function, we have plotted the reward curves in the AutoRL-Bench tabular benchmark.
Revised: Algorithm seed SEEDS do budgets B end end E A UTO RL-B ENCH R EWARD C URVES To further motivate our design choice for modeling the reward curves using a generalised logistic function, we have plotted the reward curves in the AutoRL-Bench tabular benchmark.

Original: Figures 
Revised: Figures 43show

Revised: the reward curves for the PPO, A2C, DDPG, SAC, and TD3 search spaces, respectively.

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: 1 for seed in SEEDS do for different budgets B in the x

Original: [Figure]
Revised: -axis do Compute the y-axis by following steps (i) and (ii): (i) for each environment in a benchmark do for each method do Get final return (step 2 in Algorithm 2). end

Original: [Figure]
Revised: Compute the rank of each method for that environment based on the returns. end (ii) Compute the mean of the ranks across the environments in step (i) for

Revised: each method.end 12 end

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: of the the MuJoCo envi-

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: the the MuJoCo envi-

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Figure]

Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: n

Original: [Figure] 
Revised: [Figure] [Figure]

Original: Pong-v
Revised: Pong-v

Original: Alien-v
Revised: Alien-v

Original: BankHeist-v
Revised: BankHeist-v

Original: BeamRider-v
Revised: BeamRider-v

Original: [Figure]
Revised: [Figure]

Original: w
Revised: w

Original: [Figure]
Revised: [Figure]

Original: Breakout-v
Revised: Breakout-v

Original: Enduro-v
Revised: Enduro-v

Original: Phoenix-v
Revised: Phoenix-v

Original: Seaquest-v
Revised: Seaquest-v

Revised: [Figure]

Revised: w

Original: [Figure]
Revised: [Figure]

Original: SpaceInvaders-v
Revised: SpaceInvaders-v

Original: Riverraid-v
Revised: Riverraid-v

Original: Tennis-v
Revised: Tennis-v

Original: Skiing-v
Revised: Skiing-v

Original: [Figure]
Revised: [Figure]

Original: Boxing-v
Revised: Boxing-v

Original: Bowling-v
Revised: Bowling-v

Original: Asteroids-v
Revised: Asteroids-v

Original: CartPole-v
Revised: CartPole-v

Original: [Figure]
Revised: [Figure]

Original: MountainCar-v
Revised: MountainCar-v

Original: Acrobot-v
Revised: Acrobot-v

Original: Timesteps
Revised: Timesteps

Original: Pendulum-v
Revised: Pendulum-v

Original: Timesteps
Revised: Timesteps

Original: Ant-v
Revised: Ant-v

Original: Timesteps
Revised: Timesteps

Original: [Figure]
Revised: [Figure]

Original: Hopper-v
Revised: Hopper-v

Original: Timesteps
Revised: Timesteps

Original: Humanoid-v
Revised: Humanoid-v

Original: [Figure]
Revised: [Figure]

Original: Pong-v
Revised: Pong-v

Original: Alien-v
Revised: Alien-v

Original: BankHeist-v
Revised: BankHeist-v

Original: BeamRider-v
Revised: BeamRider-v

Original: [Figure]
Revised: [Figure]

Original: w
Revised: w

Original: [Figure]
Revised: [Figure]

Original: Breakout-v
Revised: Breakout-v

Original: Enduro-v
Revised: Enduro-v

Original: Phoenix-v
Revised: Phoenix-v

Original: Seaquest-v
Revised: Seaquest-v

Revised: [Figure]

Revised: w

Original: [Figure]
Revised: [Figure]

Original: SpaceInvaders-v
Revised: SpaceInvaders-v

Original: Riverraid-v
Revised: Riverraid-v

Original: Tennis-v
Revised: Tennis-v

Original: Skiing-v
Revised: Skiing-v

Revised: [Figure]

Revised: w

Original: [Figure]
Revised: [Figure]

Original: Boxing-v
Revised: Boxing-v

Original: Bowling-v
Revised: Bowling-v

Original: Asteroids-v
Revised: Asteroids-v

Original: CartPole-v
Revised: CartPole-v

Original: [Figure]
Revised: [Figure]

Revised: w

Revised: [Figure]

Original:  MountainCar-v
Revised: d MountainCar-v

Original: Acrobot-v
Revised: Acrobot-v

Original: Timesteps
Revised: Timesteps

Original: Pendulum-v
Revised: Pendulum-v

Original: Timesteps
Revised: Timesteps

Original: Ant-v
Revised: Ant-v

Original: Timesteps
Revised: Timesteps

Original: [Figure]
Revised: [Figure]

Original: Hopper-v
Revised: Hopper-v

Original: Timesteps
Revised: Timesteps

Original: Humanoid-v
Revised: Humanoid-v

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure]