N OVELTY S EARCH IN REPRESENTATIONAL SPACE FOR SAMPLE EFFICIENT EXPLORATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a new approach for efﬁcient exploration which leverages a lowdimensional encoding of the environment learned with a combination of model based and model-free objectives.
Our approach uses intrinsic rewards that are based on a weighted distance of nearest neighbors in the low dimensional representational space to gauge novelty.
We then leverage these intrinsic rewards for sample-efﬁcient exploration with planning routines in representational space.
One key element of our approach is that we perform more gradient steps in-between every environment step in order to ensure the model accuracy.
We test our approach on a number of maze tasks, as well as a control problem and show that our exploration approach is more sample-efﬁcient compared to strong baselines
I NTRODUCTION
In order to solve a task efﬁciently in Reinforcement Learning (RL), one of the main challenges is to gather informative experiences thanks to an efﬁcient exploration of the state space.
A common approach to exploration is intrinsic rewards correlated with some novelty heuristics (Schmidhuber, 2010; Stadie et al., 2015; Houthooft et al., 2016).
With intrinsic rewards, an agent can be incentivized to efﬁciently explore its state space.
A direct approach to calculating these novelty heuristics is to derive a reward based on the observations, such as a count-based reward (Bellemare et al., 2016;
Ostrovski et al., 2017) or a prediction-error based reward (Burda et al., 2018b).
However, an issue occurs when measuring novelty directly from the raw observations, as some information in the pixel space (such as randomness) might be irrelevant.
In this case, if an agent wants to efﬁciently explore its state space it should only focus on meaningful and novel information.
In this work, we propose a method of sample-efﬁcient exploration by leveraging novelty heuristics in a meaningful abstract state space.
We leverage a low-dimensional abstract representation of states, which is learned by ﬁtting both model-based and model-free components through a joint representation.
This provides a meaningful abstract representation where states that are close temporally in dynamics are brought close together in low-dimensional representation space.
We also add additional constraints to ensure that a measure of distance between states is meaningful.
With this distance in representational space, we form a novelty heuristic inspired by the Novelty Search algorithm (Lehman and Stanley, 2011) to generate intrinsic rewards that we use for efﬁcient exploration.
We show that with a good low-dimensional representation of states, a policy based on planning with our novelty heuristic is able to explore with high sample-efﬁciency.
In our experiments, we measure the effectiveness of our exploration methods by the number of samples required to explore the state space.
One key element of our approach is that we perform more gradient steps in-between every environment step in order to ensure the model accuracy is high (and hence ensure an accurate novelty heuristic).
Through this training scheme, our agent is also able to learn a meaningful representation of its state space in an extremely sample-efﬁcient manner.
P ROBLEM FORMULATION
Our agent interacts with its environment over discrete time steps, modeled as a Markov Decision
Process (MDP), deﬁned by the 6-tuple ( S , S 0 , A , τ, R , G ) (Puterman, 1994).
In this setting, S is the state space, S 0 is the initial state distribution, A is the discrete action space, τ : S ×
A → S is the transition function that is assumed deterministic (with the possibility of extension to stochastic environments with generative methods), R : S ×
A → R is the reward function ( R = [ − 1 , 1] ),
G : S ×
A → [0 , 1) is the per time-step discount factor.
At time step t in state s t ∈ S , the agent chooses an action a t ∈ A based on policy π : S ×
A → [0 , 1] , such that a t ∼ π ( s t , · ) .
After taking a t , the agent is in state s t
+1 =
τ ( s t , a t ) and receives reward r t ∼ R ( s t , a t ) and a discount factor γ t ∼ G ( s t , a t ) .
Over n environment steps, we deﬁne the history of visited states as
H s =
( s 1 , . . .
, s n ) , where s i ∈ S ∀ i ∈ N .
In RL, the usual objective is to maximize the sum of expected future rewards
V π ( s ) =
E (cid:104) r t + (cid:80) ∞ i =1 (cid:16)(cid:81)
i − 1 j =
γ t + j (cid:17) r t
+ i | s = s t , π (cid:105) .
To learn a policy π that maximizes the expected return, an RL agent has to efﬁciently explore its environment.
In this paper, we consider exploration tasks with sparse rewards, and are interested in exploration strategies that require as few steps as possible to explore the state space.
A BSTRACT STATE REPRESENTATIONS
In this work, we focus on learning a lower-dimensional representation of state when our state (or observations in the partially observable (Kaelbling et al., 1998) case) is high-dimensional (Dayan, 1993;
Tamar et al., 2016; Silver et al., 2016;
Oh et al., 2017; de Bruin et al., 2018; Ha and Schmidhuber, 2018; Franc¸ois-Lavet et al., 2018; Hafner et al., 2018; Gelada et al., 2019).
We learn our lower-dimensional abstract representation by encoding the high-dimensional state with an encoder ˆ e : S → X parameterized by θ ˆ e , where X ∈ R n X .
The dynamics are learned via the following functions: a transition function ˆ τ : X ×
A → X parameterized by θ ˆ τ , a reward function ˆ r :
X ×
A → [0 , R max ] parameterized by θ ˆ r , and a per timestep discount factor function ˆ γ : X ×
A → [0 , 1) parameterized by θ ˆ γ .
In order to leverage all past experiences, we use an off-policy learning algorithm that sample transition tuples ( s, a, r, γ, s (cid:48) ) from a replay buffer.
We ﬁrst encode our current and next states:
[Equation]
where x, x (cid:48) ∈ X and θ ˆ e are our encoder parameters.
The Q-function is learned using the DDQN (van Hasselt et al., 2015) algorithm, which uses the target
[Equation]
where θ Q − and θ ˆ e − are parameters of an earlier buffered Q-function (or our target Q-function) and encoder respectively.
The agent then minimizes the following loss:
[Equation]
We learn the dynamics of our environment through the following losses:
[Equation]
By jointly learning the weights of the encoder and the different components, the abstract representation is shaped in a meaningful way according to the dynamics of the environment.
For instance, states that are temporally close through the transition function are brought close in the lowerdimensional space–a phenomenon reported in Franc¸ois-Lavet et al.
To ensure that learning the transition function between abstract states doesn’t tend to collapse the abstract representations to a constant, we add the following soft constraints on the entropy:
[Equation]
where s, s (cid:48) are two randomly sampled states from the agent’s history and C d is a hyperparameter.
We also use a soft constraint between consecutive states that tends to enforce that two consecutive abstract representations are at a distance of at least ω :
[Equation]
where ω is a hyperparameter (a discussion of this loss is provided in Appendix A).
The hyperparameter ω can be used to estimate the accuracy of our transition loss, hence of our novelty estimates.
In order to gauge when our representation is accurate enough to use our novelty heuristic, we use a function of this hyperparameter and transition loss to set a cutoff point for accuracy to know when to take the next environment step.
If ω is the minimum distance between successive states, then when
L τ ≤ (cid:0) ωδ (cid:1) 2 , the transitions are on average within a ball of radius ωδ of the target state.
Here δ >is a hyperparameter that we call the slack ratio.
Before taking a new step in the environment, we keep training all the parameters with all these losses until this threshold is reached and our novelty heuristic becomes useful.
Details on the slack ratios used in the experiments are given in Appendix E.
We minimize the sum of all the aforementioned losses through gradient descent with learning rate α
[Equation]
Through these losses, the agent learns a low-dimensional representation of the environment.
These losses are well-suited for our novelty metric because of the meaningfulness the losses ensure in terms of the L 2 norm in our representation space (see Appendix C).
Planning techniques that combine the knowledge of the model and the value function can then be used to select actions that will maximize intrinsic rewards.
C OMBINING M ODEL -F REE AND M ODEL -B ASED COMPONENTS FOR EXPLORATION POLICIES
Similarly to previous works (e.g. Oh et al., 2017), we use a combination of model-based planning with model-free Q-learning to obtain a good policy.
For our approach, we calculate rollout estimates of next states based on our transition model ˆ τ and sum up the corresponding rewards (which we denote as r : X × A → [0 , R max ] and can be a combination of both intrinsic and extrinsic rewards). We calculate expected returns based on the discounted rewards of our d -depth rollouts:
[Equation]
Note that we simulate only b -best options at each expansion step based on Q ( x, a ; θ Q ) , where b ≤ |A| .
We then use a simple sum of the Q-values obtained with planning up to a depth D :
[Equation]
The estimated optimal action is given by argmax a ∈A
Q Dplan ( x, a ) .
The actual action chosen at each step follows an (cid:15) -greedy strategy ( (cid:15) ∈ [0 , 1] ), where the agent follows the estimated optimal action with probability 1 − (cid:15) and a random action with probability (cid:15) .
N OVELTY S EARCH IN ABSTRACT REPRESENTATIONAL SPACE
Our approach uses intrinsic motivation (Schmidhuber, 1990; Chentanez et al., 2005; Achiam and
Sastry, 2017) where an agent rewards itself based purely on the fact that it gathers interesting experiences.
The speciﬁc formulation of the rewards is inspired by the Novelty Search algorithm (Lehman and Stanley, 2011).
Since the concept of proximity loses meaning in high-dimensional spaces (Aggarwal et al., 2002), the nearest neighbor problem is ill-deﬁned.
So instead of calculating a measure of novelty in pixel space, we leverage the low-dimensional abstract representations.
To get a novelty metric for our abstract representations, we can apply a weighted version of the novelty measure in abstract representational space to our current state s with our encoder applied ˆ x ← ˆ e ( s ; θ ˆ e ) , and a history of encoded states.
We deﬁne the following novelty measure over abstract representational space:
[Equation]
i dist [Equation] [Equation] where s i is the i th closest state to s in the abstract representations and dist is a distance measure in representational space.
In this work, we use (cid:96) 2 norm as our measure of distance.
We motivate this choice in Appendix C
In the heuristic above, we weight the distances between the k nearest neighbors and s by their ranking based on distances from the corresponding abstract states.
We do this in order to compensate for biases towards state-action pairs that have fewer direct neighbors as opposed to purely novel states.
We clarify this point with a toy example in Appendix B and we compare this formulation with an unweighted average in Appendix F. We also analyze the limiting behaviour of our novelty reward in Appendix D.
Incorporating all of the above, our algorithm is described in Algorithm 1.
E XPERIMENTS
We conduct experiments on environments of varying difﬁculty.
All experiments use a training scheme where parameters converge on an accurate representation before taking an environment step.
We optimize the losses (over multiple training iterations) given in Section 3 before each environment step to train the agent’s state representation.
We discuss all environment-speciﬁc hyperparameters in Appendix E.
6. L ABYRINTH EXPLORATION
We consider two 21 × 21 versions of the grid-world environment, as per Figure 1.
The ﬁrst is an open labyrinth grid-world, with no walls except for bordering walls. The second is a similar sized grid-world split into four connected rooms.
In these environments the actions A are the four cardinal directions.
These environments have no rewards or terminal states and the goal is to explore, agnostic of the task.
We use two metrics to gauge exploration for this environment: the ﬁrst is the ratio of states visited only once, the second is the proportion of total states visited.
6.1. O PEN LABYRINTH
In the open labyrinth experiments (Figure 4a), we compare a number of variations of the above policy with a random baseline and a count-based baseline (Bellemare et al., 2016)
(as we can count states in this tabular setting).
Variations of the policy include an argmax over state values ( d = 0 ) and planning depths of d = { 1 , 5 } .
Our method is able to leverage a good learnt representation in order to explore in a sample-efﬁcient way.
All variations of our method outperform the two baselines in this task, with a slight increase [Figure] in performance as planning depth d increases.
In the open labyrinth, our agent is able to reach 100% of possible states (a total of 19 × 19 = 361 unique states) in approximately 700 steps, and 80% of possible states ( ≈ 290 states) in less than 400 steps.
These counts also include the n init number of random steps taken preceding training.
Our agent is also able to learn highly interpretable abstract representations in very few environment steps (as shown in Figure 2a) as it explores its state space.
[Figure]
6.1. 4- ROOM LABYRINTH
While the learned dynamics model and abstract representation are able to create an interpretable representation of the open labyrinth environment (Figure 2a) the 4-room labyrinth environment is more challenging.
Indeed, our encoder ˆ e takes a high-dimensional input and compresses it to a lowdimensional representation.
In the case of the labyrinth environment, the representation incorporates knowledge related to the position of the agent in 2-dimensions that we call primary features .
It also learns other information such as agent surroundings (walls, open space) etc., but it does so only via the transition function learned through experience.
We call this extraneous but necessary information secondary features .
As most of these secondary features are encoded only in the dynamics model ˆ τ , our agent has to experience a transition in order to accurately represent both primary and secondary features. [Figure]
In this environment speciﬁcally, our dynamics model tends to over-generalize for walls between rooms and can thus also fail to try out transitions in the passageways between rooms.
Our agent then tends to visit uniformly all the states that are reachable within the known rooms.
With an (cid:15) -greedy policy, our approach still ensures that the agent explores passageways even if it has over-generalized to the surrounding walls.
We run the same experiments as we do on the open labyrinth and report results in Figure 4b.
Our approach still outperforms the random and count-based baselines.
After visiting most unseen states in its environment, our agent tends to uniformly explore its state space due to the nature of our novelty heuristic.
We can see an example of this for the open labyrinth in Figure 3.
The bright spot that begins after 200 counts is the agent learning the dynamics of labyrinth walls.
6. C ONTROL AND SUB - GOAL EXPLORATION
In order to test the efﬁcacy of our method on environments with harder dynamics, we conduct experiments on the control-based environment Acrobot (Brockman et al., 2016) and a multi-step maze environment.
Our method (with planning depth d = 5 ) is compared to strong exploration baselines with different archetypes:
Prediction error incentivized exploration (Stadie et al., 2015) 2.
Hash count-based exploration (Tang et al., 2016) 3.
Bootstrap DQN (Osband et al., 2016)
In order to maintain consistency in our results, we use the same model architectures and model-free methods throughout.
Since we experiment in the deterministic domain, we exclude baselines that require some form of stochasticity or density estimation as baselines (for example, Shyam et al.
and Osband et al.
Normally, for most approaches that include a model-free component, a step in the environment is taken with every update step.
In our experiments, we use orders of magnitude less samples as compared to most model-free RL algorithms (all within the same episode).
To ensure a fair comparison between our approach and baselines, we run multiple training iterations in between each environment step for all experiments.
6.2. A CROBOT
We now test our approach on Acrobot (Brockman et al., 2016), which has a continuous state space unlike the labyrinth environment.
We speciﬁcally choose this control task because the nature of this environment makes exploration inherently difﬁcult.
The agent only has control of the actuator for the inner joint and has to transfer enough energy into the second joint in order to swing it to its goal state.
We modify this environment so that each episode is at most 3000 environment steps.
We use the default reward setup in Acrobot, where the agent is rewarded with − 1 at every environment step until it reaches its goal state where it receives a reward of 0.
To measure the performance of our exploration approach, we measure the average number of steps that the agent takes to move its second joint above a given line as per Figure 5a in a single episode.
To demonstrate the ability of our method to learn a low dimensional abstract representation from pixel inputs, we use 4 consecutive pixel frames as input instead of the 6 -dimensional full state vector.
We use a 4 -dimensional abstract representation of our state and results from experiments are shown in Table 6.2.1.
Our method reaches the goal state more efﬁciently
Acrobot
[Table]
Prediction error incentivized exploration
[Table]
Novelty Search with abstract representations
[Table]
6.2. M ULTI - STEP GOAL MAZE
We also test our method on an environment with a temporal aspect - a maze with the sub-goal of picking up a key to open doors before the main goal of reaching the reward is available.
We build our environment with the Pycolab game engine (Stepleton, 2017).
The environment can be seen in
Figure 5b.
Each episode is at most 4000 environment steps.
While this environment does admit an extrinsic reward (1 for picking up the key, 10 for reaching the ﬁnal state), we ignore these rewards and only focus on intrinsic rewards.
Similarly to the labyrinth environment, the input to our agent is a top-down view of the environment.
We also employ an (cid:15) -greedy policy.
From our experiments, we show that our agent is able to learn an interpretable representation of the environment in a sample-efﬁcient manner.
It is able to encode this temporal aspect of the environment (whether the key was collected) as shown in Figure 2b.
With our intrinsic reward generated with this representation, our agent is able to more efﬁciently explore its state space and reach the end goal in fewer steps as compared to our baselines.
R ELATED W ORK
The proposed exploration strategy falls under the category of directed exploration (Thrun, 1992) that makes use of the past interactions with the environment to guide the discovery of new states.
This work is inspired by the Novelty Search algorithm (Lehman and Stanley, 2011) that leverages a nearest-neighbor scoring function, but does so in behavior space.
Exploration strategies have been investigated with both model-free and model-based approaches.
In Bellemare et al.
(2016) and
Ostrovski et al.
(2017), a model-free algorithm provides the notion of novelty through a pseudocount from an arbitrary density model that provides an estimate of how many times an action has been taken in similar states.
Several exploration strategies have also used a model of the environment along with planning.
Hester and Stone (2012) employ a two-part strategy to calculate intrinsic rewards, combining model uncertainty (from a random-forest based model) and a novelty reward based on L 1 distance in feature space.
A strategy investigated in Salge et al.
; Mohamed and Rezende (2015); Gregor et al.
; Chiappa et al.
(2017) is to have the agent choose a sequence of actions by planning that leads to a representation of state as different as possible to the current state.
In Pathak et al.
; Haber et al.
, the agent optimizes both a model of its environment and a separate model that predicts the error/uncertainty of its own model.
Burda et al.
(2018a) similarly uses an intrinsic reward based on the uncertainty of its dynamics model.
The agent can thus seek actions that adversarially challenge its knowledge of the environment (Savinov et al., 2018).
In Shyam et al.
(2018), multiple model forward models of the environment are also employed to plan to observe novel states by using a measure of novelty derived from disagreement between future states.
D ISCUSSION
In this paper, we show that with an interpretable abstract representation of states, our novelty metric is able to serve as an intrinsic reward that enables efﬁcient exploration.
By using this novelty metric with a combination of model-based and model-free approaches for planning, we demonstrate the efﬁciency of our method in multiple environments.
As with most methods, our approach also has limitations.
While the problem of distance metrics in high-dimensional space is partially solved in our method with the dimensionality reduction of observations by our encoder, the (cid:96) 2 -norm still requires a low dimension to be useful (Aggarwal et al., 2002). This implies that our novelty metric may lose its effectiveness as we increase the dimension of our abstract representation.
In addition, our exploration strategy beneﬁts greatly from the meaningful abstractions and internal model.
In some cases, the model can over-generalize with the consequence that the low-dimensional representation loses information that is crucial for the exploration of the entire state space.
An interesting direction for future work would be ﬁnd ways of incorporating the secondary features mentioned in Section 6.1.2.
A D ISCUSSION ON THE ENTROPY CONSTRAINT
As for our soft constraints on representation magnitude, we use a local constraint instead of a global constraint on magnitude such that it is more suited for our novelty metric.
If we are to calculate some form of intrinsic reward based on distance between neighboring states, then this distance needs to be non-zero and ideally consistent as the number of states in our history increases.
In the global constraint case, if the intrinsic rewards decreases with an increase in number of states in the agent’s history, then the agent will fail to be motivated to explore further.
Even though the entropy maximization losses ensures the maximization of distances between random states, if we have | H s | number of states in the history of the agent, then a global constraint on representation magnitude might lead to
[Equation]
B M OTIVATION FOR RANKED WEIGHTING OF K -NN SCORES
Here H s is a list of all possible states in S in any order, with each possible state appearing only once.
If we let k = 4 , we have that:
[Equation] [Equation]
While it may seem redundant to include the 1 st nearest neighbor distance in this metric (which would be itself if we’ve visited the state), the 1 st nearest neighbor is non-zero when we calculate the novelty of a predicted state using our learned transition function ˆ τ .
From this example, we can see that there is a bias towards states with fewer direct neighbors due to the nature of our novelty metric.
This poses an issue - if our goal is for sample-efﬁcient exploration of our state space, then there is no reason to favor states with less direct neighbors.
C M OTIVATION FOR USING L 2 NORM AS A DISTANCE MEASURE IN ABSTRACT REPRESENTATION SPACE
We consider the entropy maximization loss L d 1 and consecutive distance loss L consec . On the one hand, the entropy maximization loss ensures maximal (cid:96) 2 distance between two random states due to the inverse relationship of the negative exponential. On the other hand, minimization of our consecutive distance loss ensures that the distance between two consecutive states is ≤ ω .
We consider the minima of the combination of these two losses for a consecutive states s and s (cid:48) :
[Equation]
The minimum of the combination of the losses is obtained when the distance between s and s (cid:48) is ω .
These losses shape abstract state space so that (cid:96) 2 norm as a distance measure encodes a notion of closeness in state space that we leverage in our novelty heuristic.
D L IMITING BEHAVIOR FOR NOVELTY HEURISTIC
In the limiting case, our novelty heuristic is 0 for all states.
Indded, if we consider the case where each state has been visited k times, the k nearest neighbors to state s are all identical and equal to s in the deterministic case.
Based on equation 8, we have the following novelty heuristic score for each state:
[Equation]
As for stochastic environments, we let s (cid:48)(cid:48) be a stochastic observation of the state s , and have that
[Equation]
If E [ dist (ˆ e ( s ; θ ˆ e ) , ˆ e ( s (cid:48)(cid:48) ; θ ˆ e ))]
→ 0 as the encoder reaches optimality, then our novelty heuristic will return 0 in expectation in the limiting case.
E E XPERIMENTAL SETUP AND HYPERPARAMETERS
For all of our experiments, we use a batch size of 64 and take 64 random steps transitions before beginning training.
We also use the same discount factor for all experiments ( γ = 0 . 8 ) and the same freeze interval for target parameters 1000.
The reason behind our low discount factor is due to the high density of non-stationary intrinsic rewards in our state.
For all model-based abstract representation training, the following hyperparameters were all kept constant: minimum distance between consecutive states ω = 0 .
5 , slack ratio δ = 6 and transition model dropout of 0 . 1 .
For all experiments run with our novelty metric, we use k = 5 for our k-NN calculations.
E. N EURAL N ETWORK A RCHITECTURES
For reference, ’Dense’ implies a full-connected layer. ’
Conv2D’ refers to a 2D convolutional layer with stride 1. ’
MaxPooling2D’ refers to a max pooling operation.
All networks were trained with the RMSProp optimizer.
Throughout all experiments, we use the following neural network architectures:
[Equation]
E NCODER
For all our non-control task inputs, we ﬂatten our input and use the following feed-forward neural network architecture for ˆ e :
• Dense(200, activation=’tanh’)
• Dense(100, activation=’tanh’)
• Dense(50, activation=’tanh’)
• Dense(10, activation=’tanh’)
• Dense(abstract representation dimension).
For our control task, we use a convolution-based encoder:
• Conv2D(channels=8, kernel=(3,3), activation=’tanh’)
• Conv2D(channels=16, kernel=(3,3), activation=’tanh’)
• MaxPool2D(pool size=(4,4))
• Conv2D(channels=32, kernel=(3,3), activation=’tanh’)
• MaxPool2D(pool size=(3,3))
• Dense(abstract state representation dimension).
E.1. T RANSITION MODEL
The input to our transition model is a concatenation of an abstract representation and an action.
We use the following architecture
• Dense(10, activation=’tanh’, dropout=0.1) • Dense(30, activation=’tanh’, dropout=0.1) • Dense(30, activation=’tanh’, dropout=0.1) •
Dense(10, activation=’tanh’, dropout=0.1) • Dense(abstract representation dimension)
and add the output of this to the input abstract representation.
E.1. R EWARD AND DISCOUNT FACTOR MODELS
For both reward and discount factor estimators, we use the following architecture:
• Dense(10, activation=’tanh’)
• Dense(50, activation=’tanh’) • Dense(20, activation=’tanh’)
• Dense(1).
E.1. Q FUNCTION APPROXIMATOR
We use two different architecture based on the type of input.
If we use the concatenation of abstract representation and action, we use the following architecture:
• Dense(20, activation=’relu’)
• Dense(50, activation=’relu’) • Dense(20, activation=’relu’)
• Dense( n actions )
For the pixel frame inputs for our control environments, we use:
• Conv2D(channels=8, kernel=(3,3), activation=’tanh’)
• Conv2D(channels=16, kernel=(3,3), activation=’tanh’) • MaxPool2D(pool size=(4,4))
• Conv2D(channels=32, kernel=(3,3), activation=’tanh’)
• MaxPool2D(pool size=(3,3))
• Dense( n actions ).
Finally, for our (purely model free) gridworld environments we use: • Dense(500, activation=’tanh’) • Dense(200, activation=’tanh’) • Dense(50, activation=’tanh’)
• Dense(10, activation=’tanh’)
• Dense( n actions )
As for our Bootstrap DQN implementation, we use the same architecture as above, except we replace the ﬁnal Dense layer with 10 separate heads (each a Dense layer with n actions nodes).
E. L ABYRINTH ENVIRONMENTS
Both environments used the same hyperparameters except for two: we add an (cid:15) -greedy ( (cid:15)
= 0 . 2 ) policy for the 4-room maze, and increased n freq from 1 to 3 in the 4-room case due to unnecessary over-training.
We have the following hyperparameters for our two labyrinth environments:
• n iters
=• α = 0 .
E. C ONTROL ENVIRONMENT
In our Acrobot environment, the input to our agent is 4 stacked consecutive pixel frames, where we reduce each frame down to a 32 × 32 pixel frame.
Our abstract representation dimension is 4 .
We use a learning rate of α = 0 .
00025 for all experiments.
We train for n iters =
30000 for both model-free experiments and n iters = 50000 for both experiments incorporating model-based components - this discrepancy is due to the need for more iterations for the model-based portion to converge.
E. M ULTI - STEP MAZE ENVIRONMENT
In our multistep maze environment, the input to our agent is a single 15 × 15 frame of an overview of the environment.
Our abstract representation dimension is 3 .
We use an (cid:15) -greedy ( (cid:15)
= 0 . 1 ) policy for this environment.
We use α = 0 .
00025 , n iters =
30000 for our model-free algorithms and α = 0 .
000025 , n iters = 50000 for experiments that include a model-base component.
This is again due to the need for our model-based component to converge.
F A BLATION STUDY
[Figure]
We perform an ablation study between unweighted