L EARNABLE B EHAVIOR C ONTROL : B REAKING A TARI H UMAN W ORLD R ECORDS VIA S AMPLE -E FFICIENT B EHAVIOR S ELECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
The exploration problem is one of the main challenges in deep reinforcement learning (RL).
Recent promising works tried to handle the problem with populationbased methods, which collect samples with diverse behaviors derived from a population of different exploratory policies.
Goal-directed policy selection has been adopted for behavior control.
However, the behavior selection space is largely limited by the predefined policy population, which further limits behavior diversity.
In this paper, we propose a general framework called L earnable B ehavioral C ontrol (LBC) to address the limitation, which a) enables a significantly enlarged behavior selection space via formulating a hybrid behavior mapping from all policies; b) constructs a unified goal-directed learnable process for behavior selection.
We introduce LBC into distributed off-policy actor-critic methods and achieve behavior control via optimizing the selection of the behavior mappings with bandit-based meta-controllers.
Our agents have achieved 10077.52% mean human normalized score and surpassed 24 human world records within 1B training frames in the
Arcade Learning Environment, which demonstrates our significant state-of-the-art (SOTA) performance without degrading the sample efficiency.
[Figure] [Figure]
I NTRODUCTION
Reinforcement learning (RL) has led to tremendous progress in a variety of domains ranging from video games (Mnih et al., 2015) to robotics (Schulman et al., 2015; 2017).
However, efficient exploration remains one of the significant challenges.
Recent prominent works tried to address the problem with population-based training (Jaderberg et al., 2017, PBT) wherein a population of policies with different degrees of exploration is jointly trained to keep both the long-term and shortterm exploration capabilities throughout the learning process.
A set of actors is created to acquire diverse behaviors derived from the policy population (Badia et al., 2020b;a).
Despite the significant improvement in the performance, these methods suffer from the aggravated high sample complexity due to the joint training on the whole population while keeping the diversity property.
To acquire diverse behaviors, NGU uniformly selects policies in the population regardless of their contribution to the learning progress (Badia et al., 2020b).
As an improvement, Agent57 adopts an adaptive policy selection mechanism that each behavior used for sampling is periodically selected from the 𝑎 𝑡
Env Actor L 𝑠 𝑡 ,𝑟 𝑡 ,𝑠 𝑡+1 𝑎 𝑡
Env Actor𝑠 𝑡 ,𝑟 𝑡 ,𝑠 𝑡+1 Φ(𝑠,𝑎) ⋅⋅⋅
Hybrid Behavior Mapping ℱ 𝝍𝝍 1 ∼𝒫 Ψ∗
Hybrid Behavior 𝜇 1 (𝑠,𝑎)
Policy N 𝒉 𝑁 ,𝜽 𝑁 𝜋 𝑁 (𝑠,𝑎)
Policy𝒉 2 ,𝜽𝜋 2 (𝑠,𝑎) PolicyΦ 𝜽 1 ,𝐡𝜋 1 (𝑠,𝑎) ⋅⋅⋅
Policy Population 𝜋 𝑖 (𝑠,𝑎) ⋅⋅⋅ 𝝍 1 ∼𝒫 Ψ∗
𝝍 1 ∼𝒫 Ψ∗
Hybrid Behavior Mapping ℱ 𝝍
Meta-controller 𝒫 Ψ∗
=argmax 𝒫 Ψ ℒ
𝒫 𝝍 1 ∼𝒫 Ψ∗
Hybrid Behavior ⋅⋅⋅ ⋅⋅⋅
L 𝐺 𝝍𝜇 𝝍 1 (𝑠,𝑎)
Shared Replay Buffer
Policy Training
Data Collection 𝑎 𝑡
Env Actor𝑟 𝑡 ,𝑠 𝑡+ population according to a goal-directed meta-controller (Badia et al., 2020a).
Although Agent achieved significantly better results on the Arcade Learning Environment (ALE) benchmark, it costs tens of billions of environment interactions as much as NGU.
Conclusively, prior PBT-based methods boost behavior diversity with expanding policy population, which is particularly data-consuming since the policies with heterogeneous training objectives require more data to converge.
In this paper, we try to improve the sample efficiency of population-based reinforcement learning methods by taking a step towards a more challenging setting to control behaviors with significantly enlarged behavior space with a small size of the policy population.
We formulate the process of getting sampling behaviors from all policies as hybrid behavior mapping , and the behavior control problem is directly transformed into selecting appropriate mapping functions.
By combining all policies, the behavior selection space increases exponentially along with the population size.
As a special case that population size degrades to one , diverse behaviors can also be obtained by choosing different behavior mappings.
This two-fold mechanism enables tremendous larger space for behavior selection.
By properly parameterizing the mapping functions, our method enables a unified goal-directed learnable process, and we call this general framework L earnable B ehavior C ontrol.
We use the Arcade Learning Environment (ALE) to evaluate the performance of the proposed methods, which is an important testing ground that requires a broad set of skills such as perception, exploration, and control (Badia et al., 2020a).
Previous works use the normalized human score to summarize the performance on ALE and claim superhuman performance (Bellemare et al., 2013).
However, the human baseline is far from representative of the best human player, which greatly underestimates the ability of humanity.
In this paper, we introduce a more challenging baseline, i .
e ., the human world records baseline (see Toromanoff et al.
; Hafner et al.
(2021) for more information on Atari human world records).
We summarize the number of games that agents can outperform the human world records ( i .
e ., HWRB, see Figs. 1) to claim a real superhuman performance in these games, inducing a more challenging and fair comparison with human intelligence.
Experimental results show that the sample efficiency of our method also outperforms the concurrent work MEME Kapturowski et al.
(2022), which is 200x faster than Agent57.
In summary, our contributions are as follows:
A data-efficient RL framework named LBC .
We propose a general framework called L earnable B ehavior C ontrol (LBC), which enables a significantly enlarged behavior selection space without increasing the policy population size via formulating a hybrid behavior mapping from all policies, and constructs a unified goal-directed learnable process for behavior selection.
A family of LBC-based RL algorithms .
We provide a family of LBC-based algorithms by combining LBC with existing distributed off-policy RL algorithms, which shows the generality and scalability of the proposed method.
The state-of-the-art performance with superior sample efficiency .
From Figs.
1, our method has achieved 10077.52% mean human normalized score (HNS) and surpassedhuman world records within 1B training frames in the Arcade Learning Environment (ALE), which demonstrates our state-of-the-art
B ACKGROUND 2. R EINFORCEMENT L EARNING
RL can be formulated as a Markov Decision Process (Howard, 1960, MDP) defined by ( S , A , p, r, γ, ρ 0 ) .
Considering a discounted episodic MDP, the initial state s 0 is sampled from the initial distribution ρ 0 ( s ) :
S → P ( S ) , where we use P to represent the probability distribution.
At each time t , the agent chooses an action a t ∈ A according to the policy π ( a t | s t ) :
S → P ( A ) at state s t ∈ S .
The environment receives a t , produces the reward r t ∼ r
( s, a ) :
S ×
A → R and transfers to the next state s t +1 according to the transition distribution p ( s ′ | s, a ) : S ×
A → P ( S ) .
The process continues until the agent reaches a terminal state or a maximum time step.
Define the discounted state visitation distribution as d πρ 0 ( s ) =
(1 − γ )
E s
[Equation]
0 [ (cid:80) ∞ t =0 γ t P ( s t = s | s 0 )] .
Define return G t = (cid:80) ∞
k
=0 γ k r t + k wherein γ ∈ (0 , 1) is the discount factor.
The goal of reinforcement learning is to find the optimal policy π ∗ that maximizes the expected sum of discounted rewards G t :
[Equation]
[Equation]
2. B EHAVIOR C ONTROL F OR P OPULATION -B ASED R EINFORCEMENT L EARNING
In value-based RL algorithms, a behavior policy can be derived from a state-action value function
Q π θ , h ( s, a ) or a advantage function A π θ , h ( s, a ) via ϵ -greedy.
In policy-based methods, a behavior policy can be derived from the policy logits Φ θ , h (Li et al., 2018) via Boltzmann operator.
For convenience, we define that a behavior policy can be derived from the policy model Φ θ , h via behavior mapping, which maps a single policy model to a behavior, e .
g .
, ϵ -greedy (Φ θ , h ) parameterized by ϵ .
Population-based training has been introduced into population-based reinforcement learning (PBRL) recently to handle the exploration problem by jointly training a population of policies to obtain diverse behaviors.
In population-based methods, there would be a set of policy models { Φ θ 1 , h 1 , ..., Φ θ N , h N } , each of which is parameterized by θ i and trained under its own hyper-parameters
h i , wherein θ
i ∈ Θ = { θ 1 , ..., θ N } and h i ∈ H
= { h 1 , ..., h N } .
The behavior control in population-based methods is normally achieved in two steps: i) select a policy model Φ θ , h from the population.
ii) applying a behavior mapping to the selected policy model.
When the behavior mapping is rule-based for each actor ( e .
g ., ϵ -greedy with rule-based ϵ ), the behavior control can be transformed into the policy model selection (See Proposition 1).
Therefore, the optimization of the selection of the policy models becomes one of the critical problems in achieving effective behavior control.
Following the literature on PBRL, NGU adopts a uniform selection, which is unoptimized and inefficient.
Built upon NGU, Agent57 adopts a goal-directed meta-controller to adaptively selected a policy model from the population to generate the behavior for each actor, which is implemented by a non-stationary multi-arm bandit algorithm.
However, the policy model selection requires maintaining a large number of different policy models, which is particularly data-consuming since each policy model in the population holds heterogeneous training objectives.
To handle this problem, we try to achieve behavior control via optimizing the behavior mapping (See Proposition 2), which may enable a larger behavior space with fewer policy models.
Moreover, recent notable works such as GDI (Fan & Xiao, 2022) has also noticed the effectiveness of optimizing the behavior mapping and proposed a hybrid behavior mapping that mappings from two policy models, which has achieved great success.
However, it still lacks a general study of why the behavior mapping optimization is better than policy model selection (See Sec. 3.3) and how to design the behavior space with a hybrid behavior mapping from policy model population of any size (See Sec. 4.1).
L EARNABLE B EHAVIOR C ONTROL
In this section, we first formulate the behavior control problem and decouple it into two sub-problems: behavior space construction and behavior selection.
Then, we discuss how to construct the behavior space and select behaviors based on the formulation.
By integrating both, we can obtain a general framework to achieve behavior control in RL, called learnable behavior control (LBC).
Moreover, we provide a novel perspective of notable RL methods based on LBC in the App.
G.
3. B EHAVIOR C ONTROL F ORMULATION
Behavior Mapping Define behavior mapping F as a mapping from some policy model(s) to a behavior.
In previous works, a behavior policy is typically obtained using a single policy model.
In this paper, as a generalization, we define two kinds of F according to how many policy models they take as input to get a behavior.
The first one, individual behavior mapping , is defined as a mapping from a single model to a behavior that is widely used in prior works, e .
g ., ϵ -greedy and
Boltzmann Strategy for discrete action space and Gaussian Strategy for continuous action space; And the second one, hybrid behavior mapping , is defined to map all policy models to a single behavior, i .
e ., F (Φ θ 1 , h 1 , ..., Φ θ N , h N ) .
The hybrid behavior mapping enables us to get a hybrid behavior by combining all policies together, which provides a greater degree of freedom to acquire a larger behavior space.
For any behavior mapping F ψ parameterized by ψ , there exists a family of behavior mappings F Ψ
=
{F ψ | ψ ∈ Ψ } that hold the same parametrization form with F ψ , where Ψ ⊆
R k is a parameter set that contains all possible parameter ψ .
Behavior Formulation As described above, in our work, a behavior can be acquired by applying a behavior mapping F ψ to some policy model(s).
For the individual behavior mapping case, a behavior can be formulated as µ θ , h , ψ = F ψ ( Φ θ , h ) , which is also the most used case in previous works.
As for the hybrid behavior mapping case, a behavior is formulated as µ Θ , H , ψ = F ψ ( Φ Θ , H ) , wherein Φ Θ ,
H = { Φ θ 1 , h 1 , ..., Φ θ N , h N } is a policy model set containing all policy models.
Behavior Control Formulation Behavior control can be decoupled into two sub-problems: 1) which behaviors can be selected for each actor at each training time, namely the behavior space construction .
2) how to select proper behaviors, namely the behavior selection .
Based on the behavior formulation, we can formulate these sub-problems:
Definition 3.1 (Behavior Space Construction) .
Considering the RL problem that behaviors µ are generated from some policy model(s).
We can acquire a family of realizable behaviors by applying a family of behavior mappings F Ψ to these policy model(s).
Define the set that contains all of these realizable behaviors as the behavior space, which can be formulated as:
[Equation]
Definition 3.2 (Behavior Selection) .
Behavior selection can be formulated as finding a optimal selection distribution P ∗ M Θ , H , Ψ to select the behaviors µ from behavior space M Θ , H , Ψ and maximizing some optimization target L P , wherein L P is the optimization target of behavior selection:
[Equation]
3. B EHAVIOR S PACE C ONSTRUCTION
In this section, we further simplify the equation 2, and discuss how to construct the behavior space.
Assumption 1.
Assume all policy models share the same network structure, and do not consider the impact of random initialization and other randomization operations during training (such as stochastic batch sampling), the parameters θ of a learned model Φ θ , h is only conditioned on its hyper-parameters h .
Thus Φ θ , h can be abbreviated as Φ h .
Unless otherwise specified, in this paper, we assume Assumption 1 holds.
Under Assumption 1, the behavior space defined in equation 2 can be simplified as,
[Equation]
for individual behavior mapping
[Equation]
According to equation 4, four core factors need to be considered when constructing a behavior space: the network structure Φ , the form of behavior mapping F , the hyper-parameter set H and the parameter set Ψ .
Many notable representation learning approaches have explored how to design the network structure (Chen et al., 2021;
Irie et al., 2021), but it is not the focus of our work.
In this paper, we do not make any assumptions about the model structure, which means it can be applied to any model structure.
Hence, there remains three factors, which will be discussed below.
For cases that behavior space is constructed with individual behavior mappings , there are two things to be considered if one want to select a specific behavior from the behavior space: the policy model Φ h and behavior mapping F ψ .
Prior methods have tried to realize behavior control via selecting a policy model Φ h i from the population { Φ h 1 , ..., Φ h N } (See Proposition 1).
The main drawback of this approach is that only one policy model is considered to generate behavior, leaving other policy models in the population unused.
In this paper, we argue that we can tackle this problem via hybrid behavior mapping , wherein the hybrid behavior is generated based on all policy models.
In this paper, we only consider the case that all of the N policy models are used for behavior generating, i .
e ., µ H , ψ = F ψ ( Φ H ) .
Now there is only one thing to be considered , i .
e ., the behavior mapping function F ψ , and the behavior control problem will be transformed into the optimization of the behavior mapping (See Proposition 2).
We also do not make any assumptions about the form of the mapping.
As an example, one could acquire a hybrid behavior from all policy models via network distillation, parameter fusion, mixture models, etc.
3. B EHAVIOR S ELECTION
According to equation 4, each behavior can be indexed by h and ψ for individual behavior mapping cases, and when the ψ is not learned for each actor, the behavior selection can be cast to the selection of h (see Proposition 1).
As for the hybrid behavior mapping cases, since each behavior can be indexed by ψ , the behavior selection can be cast into the selection of ψ (see Proposition 2).
Moreover, according to equation 3, there are two keys in behavior selection: 1) Optimization Target L P . 2)
The optimization algorithm to learn the selection distribution P M H , Ψ and maximize L P .
In this section, we will discuss them sequentially.
Optimization Target Two core factors have to be considered for the optimization target: the diversity-based measurement V TD µ (Eysenbach et al., 2019) and the value-based measurement
V TV µ (Parker-Holder et al., 2020).
By integrating both, the optimization target can be formulated as:
[Equation]
wherein, R µ ∼P MH , Ψ and D µ ∼P MH , Ψ is the expectation of value and diversity of behavior µ over the selection distribution P M H , Ψ .
When F ψ is unlearned and deterministic for each actor, behavior selection for each actor can be simplified into the selection of the policy model:
Proposition 1 (Policy Model Selection) .
When F ψ is a deterministic and individual behavior mapping for each actor at each training step (wall-clock), e.g., Agent57 , the behavior for each actor can be uniquely indexed by h , so equation 5 can be simplified into
[Equation]
where P H is a selection distribution of h ∈ H
= { h 1 , ..., h N } .
For each actor, the behavior is generated from a selected policy model Φ h i with a pre-defined behavior mapping F ψ .
In Proposition 1, the behavior space size is controlled by the policy model population size ( i .
e ., | H | ).
However, maintaining a large population of different policy models is data-consuming.
Hence, we try to control behaviors via optimizing the selection of behavior mappings:
Proposition 2 (Behavior Mapping Optimization) .
When all the policy models are used to generate each behavior, e.g., µ ψ = F ψ (Φ θ , h ) for single policy model cases or µ ψ = F ψ (Φ θ 1 , h 1 , ..., Φ θ N , h N ) for N policy models cases, each behavior can be uniquely indexed by F ψ , and equation 5 can be simplified into
[Equation]
(7) where P Ψ is a selection distribution of ψ ∈ Ψ .
In Proposition 2, the behavior space is majorly controlled by | Ψ | , which could be a continuous parameter space.
Hence, a larger behavior space can be enabled.
Selection Distribution Optimization Given the optimization target L P , we seek to find the optimal behavior selection distribution P ∗ µ that maximizes L P :
[Equation]
where (1) and (2) hold because we have Proposition 1 and 2, respectively.
This optimization problem can be solved with existing optimizers, e .
g ., evolutionary algorithm (Jaderberg et al., 2017), multi-arm bandits (MAB) (Badia et al., 2020a), etc.
LBC-BM: A B OLTZMANN M IXTURE BASED I MPLEMENTATION FOR LBC
In this section, we provide an example of improving the behavior control of distributed off-policy actor-critic methods (Espeholt et al., 2018) via optimizing the behavior mappings as Proposition 2.
We provide a novel design of hybrid behavior mapping, inducing an implementation of LBC, which we call B oltzmann M ixture based LBC , namely LBC- BM .
By choosing different H and Ψ , we can obtain a family of implementations of LBC- BM with different behavior spaces (see Sec. 5.4).
4. B OLTZMANN M IXTURE B ASED B EHAVIOR S PACE C ONSTRUCTION
In this section, we implement a novel hybrid behavior mapping in three sub-processes:
Generalized Policy Selection In Agent57, behavior control is achieved by selecting a single policy from the policy population at each iteration.
Following this idea, we generalize the method to the case where multiple policies can be selected.
More specifically, we introduce a importance weights vector ω to describe how much each policy will contribute to the generated behavior, ω
=
[ ω 1 , ..., ω N ] , ω i ≥ 0 , (cid:80) N i =1 ω i
= 1 , where ω i represents the importance of i th policy in the population ( i .
e ., Φ h i ).
In particular, if ω is a one-hot vector, i .
e ., ∃ i ∈ { 1 , 2 , ..., N } , ω i = 1; ∀ j ∈ { 1 , 2 , ..., N } ̸
= i, ω j = 0 , then the policy selection becomes a single policy selection as Proposition 1.
Therefore, it can be seen as a generalization of single policy selection, and we call this process generalized policy selection .
Policy-Wise Entropy Control
In our work, we propose to use entropy control (which is typically rule-based controlled in previous works) to make a better trade-off between exploration and exploitation.
For a policy model Φ h
i from the population, we will apply a entropy control function f
τ i ( · ) , i .
e .,
π h i ,τ i =
f
τ i (Φ h i ) , where π h i ,τ
i is the new policy after entropy control, and f τ i ( · ) is parameterized by τ i .
Here we should note that the entropy of all the policies from the population is controlled in a policy-wise manner.
Thus there would be a set of entropy control functions to be considered, which is parameterized by τ =
[ τ 1 , ..., τ N ] .
Behavior Distillation from Multiple Policies Different from previous methods where only one policy is used to generate the behavior, in our approach, we combine N policies [ π h 1 ,τ 1 , ..., π h N ,τ N ] , together with their importance weights ω
=
[ ω 1 , ..., ω N ] . Specially, in order to make full use of these policies according to their importance, we introduce a behavior distillation function g which takes both the policies and importance weights as input, i . e ., µ H , τ , ω = g
( π h 1 ,τ 1 , ..., π h N ,τ N , ω ) .
The distillation function g ( · , ω ) can be implemented in different ways, e .
g .
, knowledge distillation (supervised learning), parameters fusion, etc.
In conclusion, the behavior space can be constructed as,
[Equation]
wherein Ψ = { ψ = ( τ 1 , ω 1 , ..., τ N , ω N ) } , H = { h 1 , ..., h N } .
Note that this is a general approach which can be applied to different tasks and algorithms by simply selecting different entropy control function f
τ i ( · ) and behavior distillation function g ( · , ω ) .
As an example, for Atari task, we model the policy as a Boltzmann distribution, i .
e .,
π h i ,
τ i ( a | s ) =
e τ i
Φ h
i ( a | s ) (cid:80) a ′
e τ i
Φ h
i ( a ′ | s ) , where τ i ∈ (0 , ∞ ) .
The entropy can thus be controlled by controlling the temperature ( e .
g .
, τ i = 0 for full exploitation, τ i = ∞ for full exploration).
As for the behavior distillation function, we implement it in a more direct way, i .
e ., combining the policies using a mixture model , i .
e ., µ H , τ , ω ( a | s )
= (cid:80) N i =1 ω i
π h
i ,τ i ( a | s ) , where the importance weights play the role of mixture weights of the mixture model.
Then, the behavior space becomes,
[Equation]
4. MAB B ASED B EHAVIOR S ELECTION
According to Proposition 2, the behavior selection over behavior space 10 can be simplified to the selection of ψ .
In this paper, we use MAB-based meta-controller to seek ψ ∗ ∈ Ψ .
Since Ψ is a continuous multidimensional space, we discretize Ψ into K regions { Ψ 1 , ..., Ψ K } , and each region corresponds to an arm of MAB.
At the beginning of a trajectory i , l -th actor will use MAB to sample a region Ψ k indexed by arm k according to P Ψ =
softmax(Score Ψ k ) =
[Equation]
We adopt UCB score as Score Ψ k
= V Ψ k + c ·
[Equation]
k to tackle the reward-diversity trade-off problem in equation 7 (Garivier & Moulines, 2011).
N Ψ k means the number of the visit of Ψ k indexed by arm k .
V Ψ k is calculated by the expectation of the undiscounted episodic returns to measure the value of each Ψ k , and the UCB item
[Equation]
k is used to avoid selecting the same arm repeatedly and ensure sufficient diverse behavior mappings can be selected to boost the behavior diversity.
After an Ψ k is sampled, a ψ will be uniformly sampled from Ψ k , corresponding to a behavior mapping F ψ .
With F ψ , we can obtain a behavior µ ψ according to equation 10.
Then, the l -th actor acts µ ψ to obtain a trajectory τ
i and the undiscounted episodic return G i , then G i is used to update the reward model V Ψ k of region Ψ k indexed by arm k .
For more details, see App.
E.
E XPERIMENT
In this section, we design our experiment to answer the following questions:
• Whether our methods can outperform prior SOTA RL algorithms in both sample efficiency and final performance (See Sec.
5.2 and Figs. 3)?
• Can our methods adaptively adjust the exploration-exploration trade-off (See Figs. 4)?
• How to enlarge or narrow down the behavior space?
What is the performance of methods with different behavior spaces (See Sec. 5.4)?
• How much performance will be degraded without proper behavior selection (See Figs. 5)?
5. E XPERIMENTAL S ETUP 5.1. E XPERIMENTAL D ETAILS
We conduct our experiments in ALE (Bellemare et al., 2013).
The standard pre-processing settings of Atari are identical to those of Agent57 (Badia et al., 2020a), and related parameters have been concluded in App.
I. We employ a separate evaluation process to record scores continuously.
We record the undiscounted episodic returns averaged over five seeds using a windowed mean overepisodes.
To avoid any issues that aggregated metrics may have, App.
J provides full learning curves for all games and detailed comparison tables of raw and normalized scores.
Apart from the mean and median HNS, we also report how many human worlds records our agents have broken to emphasize the superhuman performance of our methods.
For more experimental details, see App. H.
5.1. I MPLEMENTATION D ETAILS
We jointly train three models, and each model can be indexed by the hyper-parameters
h i =
( γ i , RS i ) , wherein RS i is a reward shaping method (Badia et al., 2020a), and γ i is the discounted [Figure] [Figure]
Each policy model Φ h i adopts the dueling network structure (Wang et al., 2016), where Φ h i
=
A h i
=
Q h
i − V h
i .
More details of the network structure can be found in App.
L.
To correct for harmful discrepancy of off-policy learning, we adopt V-Trace (Espeholt et al., 2018) and ReTrace (Munos et al., 2016) to learn V h i and Q h i , respectively.
The policy is learned by policy gradient (Schulman et al., 2017).
Based on equation 10, we could build a behavior space with a hybrid mapping as M H , Ψ
= { µ H , ψ = (cid:80)
i =1 ω i softmax τ i (Φ h i ) } , wherein H = { h 1 , h 2 , h 3 } , Ψ = { ψ = ( τ 1 , ω 1 , τ 2 , ω 2 , τ 3 , ω 3 ) | τ i ∈ (0 , τ + ) , (cid:80) 3 j =1 ω j = 1 } .
The behavior selection is achieved by MAB described in Sec. 4.2, and more details can see App.
E. Finally, we could obtain an implementation of LBC- BM , which is our main algorithm .
5. S UMMARY OF R ESULTS
Results on Atari Benchmark
The aggregated results across games are reported in Figs.
From
Figs.
3, our agents achieve the best mean HNS and surpass the most human world records acrossgames of the Atari benchmark with minimal training frames, leading to the best learning efficiency.
Noting that Agent57 reported the maximum scores across training as the final score, and if we report our performance in the same manner, our median is 1934%, which is higher than Agent57 and demonstrates our SOTA performance.
Discussion of Overall Results With LBC, we can understand why Agent57 could achieve high performance but relatively low learning efficiency.
This is due to threefold reasons: i) Agent57 has a high-capacity behavior selection space (i.e., | H | = 32 policy models) and a meta-controller to select which model to prioritize to generate behavior ii) to boost the exploration and behavior diversity,
Agent57 introduced the intrinsic rewards into the training objective, but iii) multi-objective training typically requires more data to converge.
Compared to Agent57, our success comes from threefold reasons: i) significantly enlarging the behavior space (approaching infinite size), ii) a novel hybrid behavior mapping to automatically tackle the exploration and exploitation trade-off of all policy models, iii) a meta-controller to adjust the exploration-exploitation trade-off across training process.
Each of the proposed improvements will be succinctly studied in our ablation studies.
5. C ASE S TUDY : B EHAVIOR C ONTROL
To further explore the mechanisms underlying the success of our methods, we adopt a case study to showcase our control process of behaviors.
As shown in Figs.
4, in most tasks, our agents prefer exploratory behaviors first (i.e., high stochasticity policies with high entropy), and, as training
[Figure]
progresses, the agents shift into producing experience from more exploitative behaviors.
On the verge of peaking, the entropy of the behaviors could be maintained at a certain level (task-wise) instead of collapsing swiftly to zero to avoid converging prematurely to sub-optimal policies.
5. A BLATION S TUDY
In this section, we investigate several properties of our method.
For more details, see App.
K.
Behavior Space Decomposition To explore the effect of different behavior spaces, we decompose the behavior space of our main algorithm via reducing H and Ψ : 1) Reducing H .
When we set all the policy models of our main algorithm the same, the behavior space transforms from F (Φ h 1 , Φ h 2 , Φ h 3 ) into F (Φ h 1 , Φ h 1 , Φ h 1 ) .
H degenerates from { h 1 , h 2 , h 3 } into { h 1 } .
We can obtain a control group with a smaller behavior space by reducing H .
2)Reducing H and Ψ .
Based on the control group reducing H , we can further reduce Ψ to further narrow down the behavior space.
Specially, we can directly adopt a individual behavior mapping to build the behavior space as M H , Ψ
= { µ ψ = softmax τ (Φ h 1 ) } , where Ψ degenerates from { ω 1 , ω 2 , ω 3 , τ 1 , τ 2 , τ 3 } to { τ } and H = { h 1 } .
Then, we can obtain a control group with the smallest behavior space by reducing H and Ψ .
The performance of these methods is illustrated in Figs. 5, and from left to right, the behavior space of the first three algorithms decreases in turn (According to Corollary 4 in App. C).
It is evident that narrowing the behavior space via reducing H or Ψ will degrade the performance.
On the contrary, the performance can be boosted by enlarging the behavior space, which could be a promising way to improve the performance of existing methods.
Behavior Selection To highlight the importance of an appropriate behavior selection, we replace the meta-controller of our main algorithm with a random selection.
The ablation results are illustrated in Figs.
5, from which it is evident that, with the same behavior space, not learning an appropriate selection distribution of behaviors will significantly degrade the performance.
We conduct a t-SNE analysis in App.
K.3 to demonstrate that our methods can acquire more diverse behaviors than the control group with pre-defined behavior mapping.
Another ablation study that removed the UCB item has been conducted in App.
K.3 to demonstrate the behavior diversity may be boosted by the
UCB item, which can encourage the agents to select more different behavior mappings.
C ONCLUSION
We present the first deep reinforcement learning agent to break 24 human world records in Atari using only 1B training frames.
To achieve this, we propose a general framework called LBC, which enables a significantly enlarged behavior selection space via formulating a hybrid behavior mapping from all policies, and constructs a unified goal-directed learnable process for behavior selection.
We introduced LBC into off-policy actor-critic methods and obtained a family of implementations.
A large number of experiments on Atari have been conducted to demonstrate the effectiveness of our methods empirically.
Apart from the full results, we do detailed ablation studies to examine the effectiveness of the proposed components.
While there are many improvements and extensions to be explored going forward, we believe that the ability of LBC to enhance the optimization process of behaviors results in a powerful platform to propel future research.
R