Original: U NCERTAINTY - SENSITIVE L EARNING AND P LANNING WITH E NSEMBLES
Revised: Uncertainty-sensitive Learning and Planning with Ensembles

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: Abstract

Original: We propose a reinforcement learning framework for discrete environments in which an agent optimizes its behavior on two timescales.
Revised: We propose a reinforcement learning framework for discrete environments in which an agent makes both strategic and tactical decisions.

Original: For the short one, it uses tree search methods to perform tactical decisions. The long strategic level is handled with an ensemble of value functions learned using TD -like backups.
Revised: The former manifests itself through the use of value function, while the latter is powered by a tree search planner.

Original: Combining these two techniques brings synergies.
Revised: These tools complement each other.

Original: The planning module performs  what-if analysis allowing to avoid short-term pitfalls and boost backups of the value function.
Revised: The planning module performs a local what-if analysis, which allows to avoid tactical pitfalls and boost backups of the value function.

Original: Notably, our method performs well in environments with sparse rewards where standard TD (1) backups fail.

Original: On the other hand, the value functions compensate for inherent short-sightedness of planning.
Revised: The value function, being global in nature, compensates for inherent locality of the planner.

Original: Importantly, we use ensembles to measure theepistemic uncertainty of value functions. This serves two purposes: a) it stabilizes planning, b) it guides exploration.
Revised: In order to further solidify this synergy, we introduce an exploration mechanism with two distinctive components: uncertainty modelling and risk measurement. To model the uncertainty we use value function ensembles, and to reﬂect risk we use propose several functionals that summarize the implied by the ensemble.

Original: We evaluate our methods on discrete environments with sparse rewards: the Deep sea chain environment, toy Montezuma’s Revenge, and Sokoban.
Revised: We show that our method performs well on hard exploration environments: Deep-sea, toy Montezuma’s Revenge, and Sokoban.

Original: In all the cases, we obtain speed-up of learning and boost to the ﬁnal performance.
Revised: In all the cases, we obtain speed-up in learning and boost in performance.

Original:  I NTRODUCTION
Revised:  Introduction

Original: The model-free and model-based approaches to reinforcement learning  have a complementary set of strengths and weaknesses.
Revised: The model-free and model-based approaches to reinforcement learning (RL) have a complementary set of strengths and weaknesses.

Original: While the former offers good asymptotic performance, it suffers from poor sample complexity.
Revised: While the former oﬀers good asymptotic performance, it suﬀers from poor sample complexity.

Original: In contrast, the latter usually needs signiﬁcantly less training samples, but often fails to achieve state-of-the-art results on complex tasks (primarily attributed to models’ imperfection).
Revised: In contrast, the latter usually needs significantly less training samples, but often fails to achieve state-of-the-art results on complex tasks (which is primarily attributed to models’ imperfections). The interplay of model-based and model-free approaches in RL has received a lot of research attention.

Original: In environments with sparse rewards the value estimation has large variance, which may hinder the learning process, especially in its early stage.
Revised: This led, for example, to strong AI systems like Silver et al. (2017; 2018) or more recently to Lowrey et al. (2018), which is closely related to our work. When dealing with challenging new RL domains it is helpful to develop tools for addressing strategic and tactical decision-making.

Original: In this work we propose a new approach blending model-based, model-free methods and utilizing risk-sensitivity information.
Revised: These two perspectives complement each other: strategic perspective is global, static, and often imprecise, while tactical perspective is local, dynamic and exact. We argue that value function can be considered as an example of the former, while a planner as an example of the latter. Indeed, value function approximators provide noisy estimates (imprecise) of values (static) to every state (global). Conversely, planning provides optimal control, which starting from a given state (local) generates actions (dynamic) that are temporally coherent and result in a better-executed trajectories (exact).

Original: It enhances explorations and mitigates theproblems of sparsity in two ways, a) theplanning module effectively shortens time horizon allowing to capture a denser learning signal, b) both value estimation and its uncertainty is used for bootstrapping and guiding exploration.
Revised: It is only reasonable to combine the two into one system. This is achieved by plugging the value function into the planner, to provide guided heuristics in the local search, shorten the search horizon, and make the search computationally eﬃcient. For complex problems, the above setup has to be supplemented by an exploration strategy. Such a strategy might be based on modeling uncertainty of the value function approximation (e.g., using dropout sampling, variational inference, distributional RL, bootstrap ensemble, etc.).

Original: We deal with environments where the optimal trajectory does not visit the same state twice, and we assume access to the perfect model.
Revised: The uncertainty is quantiﬁed by a risk measure, which is then utilized by a planner to guide exploration. Consider a situation of an agent with limited memory and computational resources, being dropped into a complex and diverse environment. This setup excludes any hope for brute force strategy and implies that a perfect planner is out of reach. In this paper, we consider Sokoban, a classic logical puzzle known for its combinatorial complexity and a recent benchmark for RL, ChainEnv, a seemingly impossible task also known as a ’hay in a needle- stack’ problem (see Osband et al.

Original: Our planning module is a Monte-Carlo Tree Search (MCTS)- like algorithm with several improvements utilizing this access to the  model.
Revised: (2018)), and Toy Montezuma’s Revenge, environment notoriously known for its exploration diﬃculty (see Guo et al. Importantly, we focus on sparse reward variants of the aforementioned environments (see Section 3). Since of our method involves planning, an agent has to have an access to the world model.

Original: Although they are not crucial to our method, we think they might be of general interests.
Revised: In this paper we assume that it is the perfect model. The reason is that challenges presented by the selected environments are the dominating factor driving diﬃculty of the task (in fact, answering the question of whether a Sokoban level is solvable is NP-hard, see e.g. Dor & Zwick (1999)).

Original: Using MCTS is not necessary, in principle virtually any planning method could be used (e.g., Levin tree search, Orseau et al.
Revised: It is also in part due to this fact that a signiﬁcant body of work dealing with diﬃcult environments makes a similar assumption (see e.g. Silver et al. Orseau et al.

Original: (2018b)).
Revised: (2018b)).

Original: Handling and using the uncertainty of value functions is an important topic. We explore risksensitivity measures based on moments and relative majority vote.
Revised: The main contribution of this work is showing how recent progress in AI can be brought together to improve planning, value function learning, and exploration, in a way that together they form robust algorithms for solving challenging reinforcement learning environments.

Original: Secondly, we test ideas inspired by the Thompson sampling and Osband et al. 
Revised: In particular: For uncertainty modeling, we assume the point of view Osband et al. (2018), which uses ensembles to approximate posterior distribution. In the spirit of Lowrey et al.

Original: Finally, in some experiments, we use MonteCarlo tree search methods, with their notions of exploration. 
Revised: (2018), we incorporate risk measures to guide exploration. For the planner, we base on AlphaZero Monte-Carlo Tree Search (see Silver et al.

Original: We found the interaction of these methods quite interesting and important.
Revised: (2017)), which we enhance with the possibility of exploiting the graph structure of environments and the properties of the optimal solutions. In the value function training protocol, we introduce several improvements, including a version of self imitation learning mechanism (Oh et al. (2018a)) and hindsight (Andrychowicz et al.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: In the next subsection we provide an overview of related topics.
Revised: In the next subsection we provide an overview of related work.

Original: In Section 2 we present and discuss our method.
Revised: In Section 2 we present and discuss our method in details.

Original: This is followed by a section with experiments and passing to conclusions.
Revised: This is followed by a section with experiments and passing to conclusions.

Original: We provide code to our work https://github.com/ learningandplanningICLR/learningandplanning and a dedicated website https: //sites.google.com/view/learn-and-plan-with-ensembles with more details and movies.
Revised: We provide code to our work https://github.com/learningandplanningICLR/learningandplanning and a dedicated website https://sites.google.com/view/learn-and-plan-with-ensembles with more details and movies.

Original: 1. R ELATED WORK
Revised: 1. Related work

Original: The ideas of mixing model-based and model-free learning were perhaps ﬁrst stated explicitly in Sutton (1990).
Revised: The ideas of mixing model-based and model-free learning were perhaps ﬁrst stated explicitly in Sutton (1990).

Original: Many approaches followed.
Revised: Many approaches followed.

Original: For example in the groundbreaking series of papers Silver et al.
Revised: More recently, in the groundbreaking series of papers Silver et al.

Original: (2017), Silver et al.
Revised: (2017), Silver et al.

Original: (2018) culminating in AlphaZero the authors have developed an elaborated system that plans and performs model-free training to master the game of Go (and others).
Revised: (2018) culminating in AlphaZero, the authors have developed an elaborate system that plans and performs model-free training to master the game of Go (and others).

Original: Similar ideas werestudied also  in Anthony et al.
Revised: Similar ideas were also studied in Anthony et al.

Revised: Perhaps, the work which is closest to ours is Lowrey et al.

Original: Constructing neural network models which would incorporate uncertainty in a principled Bayesian way has proven to be challenging and still unresolved.
Revised: It is argued in the paper that an agent with limited computational resources in a complex environment needs both to plan and learn from the incoming stream of experience. Importantly, the value function in Lowrey et al. (2018) is modeled by an ensemble of value functions.

Revised: The risk measure used to combine them is given by the ’log-sum-exp formula’ (Lowrey et al., 2018, 6).

Revised: The authors show experimentally that this approach leads to improvements in various continuous tasks, including humanoid training.

Revised: In our work, we deal with a discrete setting, which enforces a diﬀerent planning module (here MCTS-based).

Revised: Moreover, we treat more diverse class of risk measures (see Section 2).

Revised: Constructing neural network models that would incorporate uncertainty in a principled

Revised: Bayesian way has proven to be challenging and remains and open problem.

Original: A promising new resultsusing ensembles include Osbandet al.
Revised: A promising new results ensembles include Osband

Original: (2018; 2017), Lakshminarayanan et al..
Revised: ), Lakshminarayanan et al.

Original: Ensembles of models were also successfully used  improve model-based RL training, see Kurutach et al.
Revised: Ensembles of models were also successfully used to improve model-based RL training, see Kurutach et al.

Original: (2018), Chua et al.
Revised: (2018), Chua et al.

Original: (2018), and the references therein.
Revised: (2018), and the references therein.

Original: Perhaps, the work which is closest to ours is Lowrey et al. They argue, that an agent with limited computational resources in complex environment, needs both to plan and learn from the incoming stream of experience.
Revised: We note that unprincipled ensemble methods are willingly used by RL practitioners. For example, in a recent competition Kidzinski et al.

Original: They propose a system POLO, which plans using MPC Camacho & Bordons (2007) method and learns value function. Importantly, their value function is modeled by an ensemble of value functions which are aggregated using ’log-sum-exp formula’ (Lowrey et al., 2018, 6), which roughly speaking corresponds a weighted sum of their mean and variance. They show experimentally that this approach lead to improvements in various tasks, notably in training of humanoid.
Revised: (2019) aimed to train an agent able to use a prosthetic leg, four top-ten solutions used some sort of ensembles.

Original: Another work similar to ours is Guo et al.
Revised: Another work similar to ours is Guo et al.

Original: (2014), in which the authors use MCTS in the role of ‘an expert’ from which a neural policy is learn using the dagger algorithm.
Revised: (2014), in which the authors use MCTS in the role of an ‘expert’ from which a neural policy is learnt using the dagger algorithm, Ross & Bagnell (2014).

Original: The basic difference is that Guo et al.
Revised: The basic diﬀerence is that Guo et al.

Original: (2014) uses a classical MCTS without value function nor ensembles.
Revised: (2014) uses a classical MCTS without value function nor ensembles.

Original: There are a number of works, which blend planning and learning into a end-to-end architectures, including Value Prediction Networks (Oh et al.
Revised: Many works aim to build planning and learning into neural network architectures, see e.g. Oh et al.

Original: (2017)) or TreeQN (Farquhar et al.
Revised: (2017) or Farquhar et al.

Original: A recent work on model based Atari Kaiser et al.
Revised: Kaiser et al.

Original: (2019) has showed possibility of sample efﬁcient reinforcement learning with an explicit visual model.
Revised: (2019), a recent work on model-based Atari, has shown the possibility of sample eﬃcient reinforcement learning with an explicit visual model.

Original: Gu et al.
Revised: Gu et al.

Original: (2016) uses model-based at the beginning phase of training and model-free methods in ‘ﬁne-tuning’, that is exactly where they are expected to excel.
Revised: (2016) uses model-based at the beginning phase of training and model-free methods in ‘ﬁne-tuning’.

Original: There is a line of work which attempts to learn  planning module, some of them including ’imagination’ modules Pascanu et al.
Revised: Furthermore, there is a body of work that attempts to learn the planning module, see Pascanu et al.

Original: (2017), Racani`ere et al.
Revised: (2017), Racani`ere et al.

Original: or mimicking a general scheme of MCTS-like algorithm into neural architecture Guez et al.
Revised: (2017), and Guez et al.

Original: Our paper isalso related to body ofwork related to exploration.
Revised: Finally, our paper is related to research focusing on study of exploration.

Original: Fundamental results in this area concern the multi-arm bandits problem, see Lattimore & Szepesv ´ ari (2018).
Revised: Fundamental results in this area concern the multi-arm bandits problem, see Lattimore & Szepesv´ari (2018) and the references therein.

Original: Methods developed in this area have been successfully applied in planning algorithms, see Kocsis et al.
Revised: Methods developed in this area have been successfully applied in planning algorithms, see Kocsis et al.

Original: (2006) and Silver et al.
Revised: (2006) and Silver et al.

Original:  Another set of methods have been developed in an attempt to solve notoriously hard Montezuma’s Revenge, see for example Go-Explore Ecoffet et al.
Revised: Furthermore, a measure with a loading on variance (deﬁned in Section 2) is related to UCB-V algorithm developed in Audibert et al. Another set of methods have been developed in an attempt to solve notoriously hard Montezuma’s Revenge, see for example Ecoﬀet et al.

Original: (2019) or 
Revised: (2019) or Guo et al.

Original: selfimitation learning techniques Guo et al. Another particularly interesting way of dealing with exploration is Hindsight method, see Andrychowicz et al. 
Revised: In our work we use also hindsight, see Andrychowicz et al. Its primarily motivation is to enrich the learning signal and train generalized value function. However, in certain situation it can be considered as an exploration algorithm.

Original:  M ETHOD DESCRIPTION
Revised:  Method description

Original: From a high-level perspective, our method alternates between data collection, which uses a planner, and learning, which optimizes a given architecture, see Algorithm 1.
Revised: In this section, we describe our method: the planning and exploration components, as well as the training protocol. Algorithm 1 shows how the components are brought together in the training loop. The pseudo-code for planner.run episode is listed in Algorithm 2, and the remainder of pseudo-code can be found in Appendix A. Require: Environment env , Model model episode , solved ← planner run episode s

Original: In this section we describe the aforementioned key components: planner, learning and architecture.
Revised: , V see Algorithm6: values ← evaluate episode ( episode ) .

Original: The method does not rely on a particular form of a planner.
Revised: see Algorithm7: Optionally calculate a mask . see Appendix C 8: replay buffer . add episode , values , solved , mask ) .

Original: Having said that, in this work we focused on a version of MCTS similar to AlphaZero (Silver et al.
Revised: see Algorithm9: B ← replay_buffer.batch() . B = { ( s b , v b , m b } , see Algorithm10:

Revised: Update V by one step of gradient descent .

Revised: e.g. RMSProp

Revised: [Equation]

Revised: 

Revised: [Equation]

Revised: for i ∈ { 1 , . . .

Original: The method is novel in the sense that it introduces a uncertainty-sensitive policy (for tree traversal and action choice), as well as guided search which exploits the form of an optimal strategy (an idea which lies at the very heart of dynamic programming).
Revised: [Equation] K } 11: until convergence For the planer component we develop an MCTS-inspired algorithm. 1 It is novel in two aspects: it utilizes a risk-sensitive policy (for tree traversal and action choice) intended to guide exploration, and it includes methods exploiting the graph structure and enables avoiding cycles.

Revised: The loop avoidance is achieved in two ways: by backpropagation of some ﬁxed negative value through the in-tree path ending with a leaf having no unvisited neightbours (see the pseudo-code in Appendix A related to dead ends ), and during tree traversal, when the agent encouraged to avoid actions leading to previously visited states on the path (see pseudo-code related to penalty p in Appendix A).

Revised: To reinforce this eﬀect, the agent is also encouraged to avoid actions leading to previously visited states on the episode level (see Algorithm 2, where penalty e ).

Revised: These enhancements combined, make it possible to learn even in sparse rewards scenarios, which is experimentally demonstrated in Section 3 (all used environments have sparse rewards).

Revised: We note that Algorithm 1 is not MCTS speciﬁc, other planners could be used as well.

Revised: The logic of our MCTS is laid out in Algorithm 2.

Original: 1: Input: initial parameters of  value function ensemble θ i , i = 1 , .  .
Revised: We assume that each node of the search tree, say n , stores a visit count n.count , accumulated value n.value , accumulated reward n.reward ( a ), and its children are denoted by n.child ( a ) for each action a ∈ A . We deﬁne b Q θ ( n , a ) = n . reward ( a ) + γ n . child ( a ) .

Revised: value , where γ > 0 is a discount rate.

Revised: Algorithm

Original: , K , empty replay buffer D 2: repeat 3:
Revised: Require: max_episode_len mcts_passes penalty e . Episode penalty V θ . Value function

Original: Reset environment and initialize starting state  s  .
Revised: Input: s . Starting state 1: episode ← ∅ 2: for step = 1 to max episode len do 3: root ← s 4: root . value ← root . value − penalty e 5: for 1 to num mcts passes do 6: path , leaf ← traversal ( root ) 7: value ← expand leaf ( leaf ; V θ ) 8: backpropagate(value,path) 9: a ← choose action ( root , { root } ) 10: s , r , done ← model .

Original: 4: while s is not terminal do (cid:46) Planning stage 5:
Revised: step ( a ) 11: episode . append (( root , a , r )) 12: if done then break 13: return episode , done

Original: Initialize root (cid:46)
Revised: Require: avoid loops . Bool

Original: Possibly soft-penalize loop 6: while within computational budget do (cid:46) dubbed MCTS steps, e.g.7: Start from the root and move down the tree (cid:46) Tree-policy 8:
Revised: Input: n , seen 1: if avoid loops then 2: A ← { a ∈ A ( n ): n . child ( a ) / ∈ seen } 3: if A = ∅ then .

Original: Expand leaf (cid:46)
Revised: Terminal or dead-end 4: return None 5: else 6:

Original: Using ensemble value network 9:
Revised: A ← A ( n ) 7: Choose action according to equation

Original: Backpropagate (cid:46) Possibly penalize a dead-end 10: end while 11: Choose next action and next state s (cid:46) Final policy, possibly penalize 1-cycles 12: end while 13: Evaluate episode (cid:46) Use e.g. factual or bootstrapped rewards 14: Add the episode to replay buffer (cid:46)
Revised: [Equation]

Original: Possibly assigning masks 15: for however many updates do (cid:46) Learning stage 16: Randomly sample batch B of states and values (cid:46) Possibly also masks 17: Use B to update ensemble value function with gradient descent (e.g. RMSProp) 18: end for 19: until convergence
Revised: 8: return a ∗

Original: The approach is inspired by Osband et al. and Lowrey et al. The former introduces a way of modeling uncertainty via ensembles it was shown how ensembles can be viewed as samples from approximate posterior distribution (given the data gathered so far).
Revised: Our proposed risk-sensitive exploration method is implemented at the level of MCTS traversal tree, see step 7 in Algorithm 3.

Original: The latter introduced a uncertainty-sensitive view into the planning. Consequently, we will consider the following strategies:
Revised: Its key deﬁning elements are uncertainty modeling and risk measurement. A risk-sensitive tree traversal policy is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: (1) where A is the action space, φ a : R |A| → R is a uncertainty measure, and (cid:98) Q θ is an estimator of  Q -function. 
Revised: where A is the action space, φ a : R |A| → R is a risk measure, and b Q θ is an estimator of the Q -function. The posterior distribution Θ models uncertainty in the value estimation.

Original: We take expectation over an ensemble of estimators using  posterior distribution Θ .
Revised: We implement the posterior Θ using ensembles, namely E θ ∼ Θ [ f ( θ )] is approximated by 1 K P Ki = 1 f ( θ i ), where K is the size of an ensemble. This approximation stems from Osband et al. (2018, Lemma 3), which shows that for Bayesian regression in a Gaussian linear model, generating samples from posterior distribution is equivalent to solving a suitable optimization problem, see Appendix C.

Original:  Lowrey et al.
Revised: In our case we use an ensemble of value functions. In some cases, we ”sub-sample” from Θ, which is inspired by Osband et al. and the classical Thomson sampling (see Appendix C for details). Using risk measure is inspired by Lowrey et al.

Original: (2018) considered φ a ( x ) =
Revised: (2018), who used φ a ( x ) =

Original: e κx a for x ∈ R |A| and κ > 0 .
Revised: e κx a for x ∈ R |A| and κ > 0.

Original: In this paper we consider the following  φ a :
Revised: In this paper we consider the following choices of φ a :

Original: •  φ a ( x ) =
Revised: • A measure with a loading on variance, φ a ( x ) =

Original: x a + κx 2 a , κ > 0 .
Revised: x a + κx 2 a , κ > 0.

Original: This includes second moments. This can be easily generalized to include variance, standard deviation and exploration bonuses. • we propose also
Revised: This includes second moments and can be easily generalized to include variance, standard deviation and exploration bonuses. • A relative majority vote (also known as plurality vote) measure,

Original: [Equation]
Revised: [Equation]

Original: Contrary to the other cases this φ a  depends not only on marginal values of its input, but the whole input (i.e. the estimator vector (cid:98) Q θ ).
Revised: Contrary to the other cases, φ a deﬁned in equation 2 depends not only on marginal values of its input, but the whole input (i.e. the estimator vector b Q θ ).

Original: It leads to a rule resembling optimal Bayes classiﬁer form, 
Revised: It leads to a rule resembling optimal Bayes classiﬁer form, i.e. the one which chooses a minimizing P (

Original: i
Revised: a ∗ θ

Original: e. the one which chooses a minimizing P ( a ∗ θ ( s ) (cid:54) = a  ) .
Revised: ( s ) 6 = a ). The intuitions behind the aforementioned choices of φ a ’s are as follows. One can think of φ a ( x ) =

Original: Such a rule is also known as  the relative majority measure.
Revised: e κx a used in Lowrey et al. (2018) as a measure capturing all moments of value ensemble. For small values of κ ≈ 0, it behaves like the measure with a loading on variance (via the Tylor expansion). The mean approximates the value of a given action, while the variance quantiﬁes the epistemic uncertainty.

Original: Apart for the above mechanisms in some experiment we also use the Thomson sampling in a similar fashion to Osband et al. In the formulation of equation 1 this can be understood as ”subsampling” from Θ .
Revised: Taking weighted sum of these terms (as we do in ”mean+var” experiments, see Table 3) has the aim of balancing exploitation and exploration. It is also related to UCB-V algorithm, see Audibert et al. Voting, on the other hand, is a well established approach when combining ensembles, see e.g. Breiman (1996) or Rokach (2010).

Revised: A relative majority vote, in particular, is simple and it can lead to good performance, see e.g. Osband et al.

Original: In our MCTS planner we introduced model-based mechanism which enhances learning. Namely, while planning for the next action we unroll several rollouts, during this process we avoid entering perviously visited vertices.
Revised: In the context of planning and RL, it has several interesting properties. In particular, the distribution of votes across ensembles encodes the uncertainty related to the optimal action in a given state. High uncertainty may result in stochastic movement (caused by a uniform tie breaking), and consequently lead to higher exploration.

Original: This might lead to a situation when no-vertex is available, which we dubbed as dead-end and penalize.
Revised: On the other hand, low uncertainty may result in an exploitative behaviour of an agent and, as a result, lower exploration. Additionally, voting may improve decision making of an agent in the states, where some action can be dangerous (e.g. lead to irreversible states). We conclude this section by describing the part of Algorithm 1 related to value functions update. The episodes are collected and stored in a replay buﬀer.

Original: Another similar mechanism avoids entering the previously visited states on the whole episode, which we call 1 -loops. We found this mechanism  to be beneﬁcial and is able to learn even in sparse rewards scenarios (see ”Learning solution for single boards” paragraph in Section 3.3).
Revised: The replay buﬀer performs some addition bookkeeping by storing, for each transition, information whether it comes from a solved episode or not (in the case of environments that provide such information, like Sokoban). This mechanism allows sampling batches with a ﬁxed ratio of solved to unsolved transitions (see Algorithm 8 in Appendix A). Such a method resembles self-imitation techniques, see e.g. Oh et al.

Original: During the learning phase, a batch is drawn from a replay buffer and the ensemble are optimized to minimize l 2 distance from  sampled targets.
Revised: (2018b). The value functions are trained to minimize l 2 distance from target values sampled from the buﬀer.

Original: In this paper we consider two ways of calculating targets: a) values accumulated in tree nodes during planning phase (called bootstrap ), b) (discounted) values of rewardscollected in an episode (called factual ).
Revised: The target values can be computed in two modes: bootstrap , which utilizes the values accumulated during the MCTS phase, and factual , corresponding to discounted rewards in an episode (see Algorithm 8 in Appendix

Original: Our replay buffer also records for episodes, whether they are solved (i.e. those, which ended with positive ﬁnal reward) or not. In our typical experiment batches are composed to contained a ﬁxed ratio of transition from solved and unsolved episodes.
Revised: A).

Original: In some experimentsin training we use mask similar tothe on in Osband et al.
Revised: Additionally, some experiments use masks which, analogously to Osband et al.

Original: In this work wetypically implement ensembles as a set of neural networks (we also experiment with mult-headed architectures sharing some lower layers).
Revised: (2018), form a mechanism of assigning a transition to a value function (see Appendix C). Finally, we implement ensembles as a set of neural networks (we also experiment with multi-headed architectures sharing some lower layers).

Original: We use both architectures with and without random priors Osband et al.
Revised: We use both architectures with or without random priors, for details see Appendix B and Osband et al.

Original: (2018), for details see Appendix A.

Original:  E XPERIMENTS
Revised:  Experiments

Original: In this section we provided experimental evidence to show that using risks measures are useful.
Revised: In this section, we provide experimental evidence to show that using ensembles and risk measures is useful.

Original: We chose three environments, Deep-sea, toy Montezuma’s Revenge and Sokoban.
Revised: We chose three environments, Deep-sea, toy Montezuma’s Revenge, and Sokoban.

Original: In each case we  use an MCTS planner with the number of passes equal to 10 (see line 6 of Algorithm 1).
Revised: In all the cases we work with single reward versions of the these environments i.e. the agent’s is rewarded only upon success completion of its task. We use an MCTS planner with the number of passes equal to 10 (see line 4 of Algorithm 2).

Original: We consider this number to bea rather small for MCTS-like planning methods, still we observed that  is is big enough to improve ensemble value functions.
Revised: We consider this number to be rather small for MCTS-like planning methods. Interestingly, we observed that such a relatively weak planner is suﬃcient to obtain a well-preforming algorithm.

Original: We utilize various neural network architectures, see Appendix A. In most cases we measure uncertainty using either variance or standard deviation with an exception of Sokoban wih randomly generated board, where voting was used, see equation 2. 
Revised: We utilize various neural network architectures, see Appendix B. We measure uncertainty using either variance or standard deviation except for the case of Sokoban with randomlygenerated boards, where voting was used, see equation 2. Conﬁguration of the experiments is summarized in Table 3 in Appendix C.

Original: 3. D EEP - SEA
Revised: 3. Deep-sea

Original: Deep-sea environment was introduced in (Osband et al., 2018, Section 4) and later included in Osband et al.
Revised: Deep-sea environment was introduced in (Osband et al., 2018, Section 4) and later included in Osband et al.

Original: as a benchmark for exploration under the name.
Revised: as a benchmark for exploration.

Original: The agent is located  on a N × N grid, N ∈ Nstarting at (0 , 0) .
Revised: The agent is located in the upperleft corner (position (0 , 0)) on a N × N grid, N ∈ N .

Original: In each timestep its y -coordinate isincreased, while x is the reward state is found.
Revised: In each timestep, its y -coordinate is

Original: Note that the upperright part of the board is unreachable.
Revised: [Figure]

Original:  controlled.
Revised: increased, while x is controlled.

Original: The agent issues actions  {− 1 , 1 } , which according to a prescribed mask are translated toone step ”left” orone step ”right” (decreasing, if possible, or increasing x respectively).
Revised: The agent issues actions in {− 1 , 1 } . These are translated to step left or step right (increasing x by − 1 or +1, respectively, as long as x ≥ 0; otherwise x remains unchanged) according to a prescribed action mask (not to be confused with transition masks in Algorithm 1).

Original: For each step ”right” the agent is punished with 0 . [-
-] /N .
Revised: For each step right the agent is punished with 0 . [-
-]01 /N .

Original: After N steps the game ends and the agent receives reward +1 if and only if it reaches position ( N, N ) .
Revised: After N steps, the game ends, and the agent receives reward +1 if and only if it reaches position ( N, N ).

Original: The aforementioned mask  is randomized at each ﬁeld at the beginning of training (and kept ﬁxed).
Revised: The action mask mentioned above is randomized at each ﬁeld at the beginning of training (and kept ﬁxed).

Original: Such a game is purposely constructed so that naive random exploration schemes fail already for small N ’s. Indeed, a random agent has chance (1 / 2) N of reaching the goal even is it disregards misleading rewards for ”right” steps.
Revised: Such a game is purposely constructed so that naive random exploration schemes fail already for small N ’s. Indeed, a random agent has chance (1 / 2) N of reaching the goal even if we disregard misleading rewards for step right .

Original: On Figure 1 one can observe how the exploration progresses.
Revised: In Figure 1, one can observe how the exploration progresses of our method.

Original: On Figure 2 one can see  comparison of nonensemble models, ensemble model with Thomson sampling  but without uncertainty beneﬁt κ =and our ﬁnal ensemble model withexploration the bonus for uncertainty  κ > 0 .
Revised: In Figure 2, one can see a comparison of non-ensemble models, ensemble models with Thomson sampling (see Appendix C) but without uncertainty bonus ( κ = 0) and our ﬁnal ensemble model with uncertainty bonus κ = 50. We conclude that both using sub-sampling and ensembles is essential to achieve good exploration (for details see also Appendix C).

Original: 3. T OY M ONTEZUMAS R EVENGE
Revised: 3. Toy Montezuma’s Revenge

Original: Toy Montezuma’s Revenge is a navigation maze-like environment.
Revised: Toy Montezuma’s Revenge is a navigation maze-like environment.

Original: It was introduced in Roderick et al.
Revised: It was introduced in Roderick et al.

Original: (2018) to evaluate ideas of using higher-level abstractions in long-horizon problems.
Revised: (2018) to evaluate ideas of using higher-level abstractions in long-horizon problems.

Original: While its visual layer is greatly reduced  it retains much of the  exploration problems of the original Motemzuma’s Revenge.
Revised: While its visual layer is greatly reduced version of the actual Montezuma’s Revenge Atari game, it retains much of the original’s exploration problems.

Original: This makes it a useful test environment for exploration algorithms, for example it was used in a recent self-imitation approach of Guo et al.
Revised: This makes it a useful test environment for exploration algorithms, see e.g. Guo et al.

Original: From the presets of available rooms 1 , in most of our experiments we work with  the biggest map with 24 rooms, see Figure 3.
Revised: In our experiments we work with with the biggest map with 24 rooms, see Figure 3.

Original: In order to concentrate on the evaluation of exploration we chose to work with sparse rewards.
Revised: In order to concentrate on the evaluation of exploration we chose to work with sparse rewards.

Original: The agent gets rewardonly if it reaches the treasure room , otherwise the episode is terminated after 300 steps. 
Revised: The agent gets reward 1 only if it reaches the treasure room, otherwise the episode is terminated after 300 steps. The goal is to reach the room marked with

Revised: The agent needs to avoid traps (marked in red) and pass through doors (marked in blue).

Original: Clearly, any simple exploration technique would fail in this case (we provide some baselines in Table 1).
Revised: Doors are open using keys (marked in yellow). It is expected that any simple exploration technique would fail in this case (we provide some baselines in Table 1).

Original: Guo et al.
Revised: Guo et al.

Original: (2019) benchmarks PPO, PPO with self imitation learning (PPO+SIL), PPO with count based exploration bonus (PPP+EXP) and their new technique (DTSIL).
Revised: (2019) benchmarks PPO, PPO with self imitation learning (PPO+SIL), PPO with count based exploration bonus (PPP+EXP) and their new technique (DTSIL).

Original: Only the last new technique consistently is  able to solve 24 room challenge, with PPO+EXP occasionally reaching this goal.
Revised: Only DTSIL is consistently able to solveroom challenge, with PPO+EXP occasionally reaching this goal.

Original: Our method based on ensembles solves this exploration challenge even in a harder sparse reward case.
Revised: Our method based on ensembles solves this exploration challenge even in a harder, sparse reward case.

Original: Our results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: We have three setups: ’no-ensemble’, ’ensemble, no std’, ’ensemble, std’.
Revised: We have three setups: no-ensemble ; ensemble, no std ; ensemble, std .

Original: In the ﬁrst case, we train using Algorithm 1 with  singlenetwork, neural-network.
Revised: In the ﬁrst case, we train using Algorithm 1 with a single neural-network.

Original: In the second case, for each episode we sample 10 ensembles to be used and the MCTS is guided but their mean.
Revised: In the second case, for each episode we sub-sample 10 members of ensemble of 20 to be used and the MCTS is guided by their mean.

Original: In the ﬁnal third case, we follow the same protocol but we add to the mean the standard deviation of the ensembles.
Revised: In the ﬁnal third case, we follow the same protocol but we add to the mean the standard deviation.

Original: In our experiments we observe that ’no-ensemble’ often in 30 out ofcases does not leave behind the ﬁrst room.
Revised: In our experiments we observe that no-ensemble inout of 43 cases does not leave behind the ﬁrst room.

Original: Ensembles without explicit exploration bonus perform  slightly better.
Revised: The setup without explicit exploration bonus, ensemble, no std , perform only slightly better.

Original: Finally we observe that the setup using ensembles behaves very well.
Revised: Finally, we observe that ensemble, std behaves very well.

Original: [Table] ensemble 0 (43) [Table] ensemble,
Revised: Further experimental details and the network architecture are presented in Appendix B and

Original: [Table] [Table] ensemble, std 35 (37) 17. in
Revised: C.

Original: 3. S OKOBAN
Revised: 3. Sokoban

Original: Sokoban is an environment known for its combinatorial complexity.
Revised: Sokoban is an environment known for its combinatorial complexity.

Original: The agent’s goal is to push all boxes  to the designed spots, see
Revised: The agent’s goal is to push all boxes (marked as yellow, crossed squares) to the designed spots (marked a

Original:  Figure 4.
Revised: [Table] square with red dot in the middle), see Figure 4.

Original: Apart for the navigational factor, the difﬁculty of this game is greatly increased by the fact that some actions are irreversible. 
Revised: Apart from the navigational factor, the diﬃculty of this game is greatly increased by the fact that some actions are irreversible. Sokoban board with level has

Original: An archetypal example  is pushing a box into a corner, though there are multiple less obvious cases.
Revised: A canonical example of such an action is pushing a box into a corner, though there are multiple less obvious cases.

Original: This is further conﬁrmed by the fact that it is NP-hard, see e.g. Dor & Zwick (1999).
Revised: Formally, this diﬃculty manifests itself in the fact that deciding whether a level of Sokoban is solvable or not, is NP-hard, see e.g. Dor & Zwick (1999).

Original: Due to these challenges the game has been considered as a testbed for reinforcement learning and planning methods.
Revised: Due to these challenges the game has been considered as a testbed for reinforcement learning and planning methods.

Original: The boards can be randomly procedurally generated and parameterized by their size and number of boxes. 3 Exploration problem in Sokoban can be understood at two levels: a) it appears on one board, b) at meta-level exploration is needed to ﬁnd a solution common to all boards.
Revised: Operationally, to generate Sokoban levels we use an automated procedure proposed by Racani`ere et al.

Original: In many RL approaches to Sokoban, it is used with intermediate rewards e.g. for pushing a box into a designed spot.
Revised: Some RL approaches to solving Sokoban (e.g. Racani`ere et al. Guez et al. (2019)) assume additional reward signal in the game, e.g. for pushing a box into a designed spot.

Original: In this work we use sparse setting . The agent is rewarded only when all  boxes are in place.
Revised: In this work we use sparse setting , that is the agent is rewarded only when all the boxes are put into place.

Revised: It is interesting to note, that Sokoban oﬀers two exploration problems: single-level-centric, where a level-speciﬁc exploration is needed, and multi-level-centric, where a ’metaexploration’ strategy is required, which works in a levelagnostic manner or can quickly adapt.

Original: We conducted three lines of experiments using ensembles: a) learningsolution to  randomly generated boards, b) learningsolution to  single boards, c) transfer and learnable ensembles .
Revised: As a result, we conducted three lines of experiments using ensembles: a) learning to solve randomly generated boards(dubbed as multipleboards Sokoban), b) learning to solve a single board (dubbed as single-boards Sokoban) , c) transfer and learnable ensembles.

Original: In our experiment we use Sokoban with board of size (10 , 10) and four boxes.
Revised: In our experiment we use Sokoban with board of size (10 , 10) and 4 boxes.

Original: We found that the maximal length of an episode inﬂuences the results signiﬁcantly.

Original: We use  200 steps in the experiments with learning generalized value function and 100 in others.
Revised: We use the limit of 200 steps in the experiment a) and 100 in the remaining ones. [Figure]

Original: Learning solution to randomly generated boards In this experiment we measure the ability of our approach to solve randomly generated Sokoban boards.
Revised: this experiment we measure the ability of our approach to solve randomly generated Sokoban boards.

Original: We measure the performance of the agent by computing the win rate on last 1000 games.
Revised: We measure the performance of the agent by computing the win rate on last 1000 games.

Original: In this experiment we use an ensemble value function using relative majority voting as formalized in equation 2.
Revised: In this experiment we use an ensemble value function using relative majority voting as formalized in equation 2.

Original: Relative majority voting takes into account the disagreement of ensembles when it comes to the ﬁnal outcome, not only the disagreement in assessment of particular action.
Revised: Relative majority voting takes into account the uncertainty of ensembles when it comes to the ﬁnal outcome, not only the uncertainty in assessment of particular action.

Original: Aftergames it reaches 85% win rate compared to 76% of an agent not using ensembles, see Figure 5. 
Revised: After 80000 games, our method reaches 85% win rate, compared to 76% of an agent not using ensembles, see Figure 5. As a measure of exploration [Table]

Original: As a proxy measure of ensemble disagreement, wealso present the mean standard deviation of value function across episodes (red curve in Figure 5). It suggests that the better the ensembles get, the more the agree on the values assigned to Sokoban states.
Revised: we

Original: It also indicates that ensembles reduce the amount of exploration the better they get (measured via win rate).
Revised: It shows an interesting eﬀect, which can be interpreted as a transition from exploration to exploitation approximately at a 40% win-rate mark or, equivalently, 10000 games.

Original:  In this experiment we measure the extent in which our methods can plan and learn on single board of Sokoban.
Revised: Single-board Sokoban. Learning to solve a single board In this experiment we measure the extent in which our methods can plan and learn on single boards.

Original: We note that this setting differs substantially from the one in the previous paragraph.
Revised: We note that this setting diﬀers substantially from the one in the previous paragraph.

Original: While learning a generalized value function is typically a harder task, in our case it has also positive impact on the training process.

Original: There are signiﬁcantly many boards which can be easily solved even at the starts (we tested that in our setting MCTS with randomly initialized network solves≈ 0 .
Revised: In the multipleboards Sokoban there is a possibility of generalization from easier to harder scenarios (e.g. randomly initialized network solves 0 .

Original: 7% of boards), which can be used as positive examples. This leads to emergent implicit curriculum.
Revised: 7% of boards). Such a transfer is obviously not possible in the single-board case studied here, however there are many attempts on the same board, which enables the agent to explore the board’s state space.

Original: In our experiments with single boards, we used setups similar to the one in Montezuma’s Revenge.
Revised: For the single-board Sokoban we used experimental settings similar to the one in Section 3.2, see also details in Appendix C.

Original: We observes that the setup with ensembles solves ≈ 65% compare to≈ 50% the standard one network scenario.
Revised: We observe that the setup with ensembles solves 73% compared to 50% the standard training without ensembles.

Original: To ensure statistical signiﬁcance in both of the cases we performed ≥ 250 experiments thus obtaining standard deviations ≤ 2 . 5% .

Original: The result of the standard setup might seem surprisingly good, taking into account the sparse reward.
Revised: The latter might seem surprisingly good, taking into account the sparse reward.

Original: This follow by the loop avoidance described in Section 2.
Revised: This follows by the loop avoidance described in Section 2.

Original: In this experiment if during planning the agent ﬁnds itself in a situation from which it cannot ﬁnd a novel state it is punished (with − 2 ).
Revised: If during planning the agent ﬁnds itself in a situation from which it cannot ﬁnd a novel state (i.e. encounters dead-end in Algorithm 6) a negative value, set to − 2 in our experiments, is backpropagated in Algorithm 5 to already seen vertices.

Original: This value is propagated to the learning process creating another form of  exploration.
Revised: We speculate that this introduces a form of implicit exploration.

Revised: In the singe-board Sokoban case we performed also experiments on (8 , 8) boards with 4 boxes obtaining success rate of 94% compared to 64% on the standard non-ensemble settings.

Original: Transfer and learnable ensembles Generating any new board can be seen as a cost dimension along with sample complexity.
Revised: Transfer and learnable ensembles Generating any new board can be seen as a cost dimension along with sample complexity.

Original: This quite naturally happens in meta-learning problems.
Revised: This quite naturally happens in meta-learning problems.

Original: We tested how value functions learned on small number of boards perform on new previously unseen ones.
Revised: We tested how value functions learned on small number of boards perform on new previously unseen ones.

Original: We used the following protocol, we trained value function on ﬁxed number of 10 games.
Revised: We used the following protocol, we trained value function on ﬁxed number of 10 games.

Original: To ease the training we used relabeling akin to ??
Revised: To ease the training we used relabeling akin to Andrychowicz et al.

Original: [Table]

Original: We evaluate these models on other boards.
Revised: We evaluate these models on other boards.

Original: It turns out that they are typically quite weak, arguably it is not very surprising as solving 10 boards does not give much chance to infer ’the general’ solutions.
Revised: It turns out that they are typically quite weak, arguably it is not very surprising as solving 10 boards does not give much chance to infer ’the general’ solutions.

Original: In the second phase we use ensembles of the models.
Revised: In the second phase we use ensembles of the models.

Original: More precisely, we calculate the values of n = 2 , 3 models and aggregate it using a small neural networks with one fully connected hidden layer.
Revised: More precisely, we calculate the values of n = 2 , 3 models and aggregate it using a small neural networks with one fully connected hidden layer.

Original: This network is learnable and trained using the standard setup.
Revised: This network is learnable and trained using the standard setup.

Original: We observe that the quality of such ensemble increases with the number of components as summarized in Table 2.
Revised: We observe that the quality of such ensemble increases with the number of components as summarized in Table 2.

Original: We observed high variability of the results over seeds, this is to be expected as board in Sokoban signiﬁcantly vary in difﬁculty.
Revised: We observed high variability of the results over seeds, this is to be expected as board in Sokoban signiﬁcantly vary in diﬃculty.

Original: We also observe that maximal results for transfer increase with the number of value function, being approximately 10% , 11% and 12% .
Revised: We also observe that maximal results for transfer increase with the number of value function, being approximately 10%, 11% and 12%.

Original: This further supports the claim that ensembling may lead to improved performance.
Revised: This further supports the claim that ensembling may lead to improved performance.

Original: In 5 -layer experiment we use a network withhidden cnn layers, see details in Appendix A, we compare this with an analogous 4 layers network.
Revised: In 5-layer experiment we use a network with 5 hidden cnn layers, see details in Appendix B, we compare this with an analogous 4 layers network.

Original: In the latter case, we obtain weaker result.
Revised: In the latter case, we obtain weaker result.

Original: We speculate this might be due to the fact that smaller architecture is easier to overﬁt.
Revised: We speculate this might be due to the fact that smaller architecture is easier to overﬁt.

Original: C ONCLUSIONS AND FURTHER WORK
Revised: Conclusions and further work

Original: In this paper, we introduced a reinforcement learning method that blends planning, learning, and using information about uncertainty toboost exploration.
Revised: In this paper, we introduced a reinforcement learning method that blends planning, learning, and risk-sensitive approach to exploration.

Original: We veriﬁed experimentally that such a setup is useful enabling deep exploration for time horizons spanningfor even hundreds of steps.
Revised: We veriﬁed experimentally that such a setup is useful in solving hard exploration problems, i.e. problems characterized by sparse rewards and long episodes (e.g. spanning even hundreds of steps).

Original: We believe that this opens promising future research directions.
Revised: We believe that this opens promising future research directions.

Original: There are many choices for ensembles designs some of which we tested, but there are still many others.
Revised: There are multiple ensemble design choices, and we tested only a selected few.

Original: At the moment, selecting a proper aggregation method seems somewhat problem-speciﬁc. The ultimate goal should be ﬁnding more general methods.
Revised: Additionally, there are more ways for combining the results of ensembles and it would be interesting to see if one, relatively general, method can be found.

Original: Such developments would be a step towards deep Bayesian learning.
Revised: Such a result would be a step towards deep Bayesian learning.

Original: In our work, we used MCTS with a perfect model of deterministic environments.
Revised: In our work, we used MCTS with a perfect model of deterministic environments.

Original: It would be interesting to consider problems requiring the use of learned, imperfect models.
Revised: It would be interesting to consider problems requiring the use of learned, imperfect models.

Original: This is  more demanding, though uncertainty methods might be of use, this time to avoid model imperfections.
Revised: This is a more demanding task, and it is intriguing whether the approach to uncertainty developed here could be useful in dealing with model imperfections.

Original: Equally important  would be tackling stochastic environments.
Revised: Equally important research direction would be related to solving stochastic environments.

Original: This might be considerably more difﬁcult as such a task requires disentangling epistemic (studied in this work) and aleatoric (coming from the environment) uncertainties.
Revised: This might be considerably more diﬃcult as such a task requires disentangling epistemic (studied in this work) and aleatoric (coming from the environment) uncertainties.

Original: We focused our attention on the Monte-Carlo tree search, there are many otherknown planning algorithms, which could beneﬁt from utilizing the information about uncertainty.
Revised: We focused our attention on the Monte-Carlo Tree Search, but there is a priori no reason why some other planning should not yield better results.

Original: In some initial experiments we obtained promising but yet inconclusive result using the Levine tree search (see Orseau et al.
Revised: In some initial experiments we obtained promising, but yet inconclusive, result using the Levin tree search (see Orseau et al.

Original: (2018a)).
Revised: (2018a)).

Original: Another tempting direction is training both value  and  policy, akin to methods of Silver et al.
Revised: Another tempting direction is training both value function and a policy, akin to methods of Silver et al.

Original: and shift the target spots so that they match the current location of boxes that although this operation requires the knowledge of the game mechanics (i e. its perfect model) it is only in this phase
Revised: It would also be interesting to isolate the proposed exploration method from the planner and see how it fares when coupled with model-free algorithms. This ﬁts along the lines of some recent research directions, see e.g. Agarwal et al. Going further, we speculate that it may be possible to use the methods developed for

Original: The case of Sokoban cast a problem into meta-learning and continual learning grounds.
Revised: Sokoban in the meta-learning and continual learning grounds problems.

Original: Using measures of uncertainty might enable a learning system to adapt to a changing environment.
Revised: Perhaps measures of uncertainty can be used to enable a learning system to adapt to a changing environment.

Original: In an archetypical case, this might be obtained by choosing from ensemble a model (a skill) which is useful at the moment and understanding situations that such a model is not yet present. 
Revised: In an archetypical case, this might be obtained by choosing from ensemble a model (a skill) which is useful at the moment and understanding situations that such a model is not yet present. A Algorithm In this section we detail the building blocks of Algorithm 1.

Original: A A RCHITECTURES
Revised: As mentioned in Section 2, the key elements of MCTS (see Algorithm 2) are tree traversal, leaf expansion, and backpropagate, shown in Algorithm 4, Algorithm 6 and Algorithm 5, respectively. Algorithm 7 shows the update of a tree node. In particular, Algorithm 6 shows how the value function enters the picture, by evaluating all of the leaf’s children (line 9). Algorithm 4 and Algorithm 5 both use variable penalty p .

Revised: This is a penalty corresponding to entering the same states during tree traversal (hence is operates on the planner level) and the change is applied during traversal, and undone during backpropagation.

Revised: Algorithm

Revised: Require: penalty p .

Revised: Planner penalty

Revised: Input: root 1: n ← root 2: path ← ∅ 3: while n is not a leaf do 4: n . value ← n .

Revised: value − penalty p 5: a ← choose action ( n , path ) 6: if a is None then .

Revised: Dead-end 7: break 8: path .

Revised: append (( n ,a )) 9: n ← n.child ( a ) 10: return path , n . n / ∈ path

Revised: Algorithm

Revised: Require: penalty p , γ

Revised: Input: v , path 1: for ( n , a ) in reversed(path) do 2: n .

Revised: value ← n .

Revised: value + penalty p 3: v ← n . reward + γ v 4: update ( n , v )

Revised: Algorithm

Revised: Require: dead end value , b V θ

Revised: Input: leaf 1: if leaf is terminal then 2: update ( leaf , 0 . ) 3: return 0.

Revised: 4: else if leaf is a dead-end then 5: update ( leaf , dead end value ) 6: return dead end value 7: else 8: for a ∈ A ( leaf ) do 9: v ← b V θ ( leaf .

Revised: child ( a )) 10: leaf .

Revised: child ( a ) .

Revised: value ← v 11: return leaf.value

Revised: Algorithm

Revised: Input:

Revised: [Equation]

Revised: value

Revised: [Equation]

Revised: The following blocks of code are related to the training setup.

Revised: Algorithm 8 is responsible for computing an appropriated value for each element of the episode.

Revised: There are two available modes: "bootstrap" , which utilizes the values accumulated during the MCTS phase, and "factual" , which represents the sum of discounted rewards in the episode.

Revised: In the "bootstrap" mode we undo the penalty applied during the episode generation stage (line 4 in the Algorithm 2).

Revised: The inner details of replay buﬀer, in particular Hindsight and self-imitation, is given in Algorithm 9 and Algorithm 10.

Revised: Hindsight refers to any method that processes episode, and potentially overrides states values (see lines 1-2 in Algorithm 9).

Revised: Algorithm 10 shows the way a batch is generated.

Revised: First, for each transition it is determined whether it should be sampled from a solved or unsolved episode (according to a ﬁxed ratio).

Revised: Second, a game is sampled from a population determined in the previous step, with probability proportional to the game length.

Revised: Finally, given a game a transition is chosen uniformly.

Revised: Algorithm

Revised: Require: mode .

Revised: String penalty e .

Revised: Episode penalty

Revised: Input: episode .

Revised: { ( s t , a t , r t ) } solved .

Revised: Bool γ .

Revised: Discount rate 1: T ← len ( epsiode ) 2: values ← (0 , . . .

Revised: , 0) ∈ R T 3: if mode="bootstrap" then 4: rewards ← { s t .

Revised: value + penalty e } 5: else if mode="factual" then 6: rewards ← [0 , . . .

Revised: , 0 , solved ] 7: for t =

Revised: T − 1 to 0 do 8: values t − 1 ← γ values t + rewards t 9: return values

Revised: Algorithm 9 replay buffer.add()

Revised: Require:

Revised: D .

Revised: Replay Buﬀer

Revised: H .

Revised: Hindsight mapping

Revised: Input: episode .

Revised: { ( s t , a t , r t , w t ) } value .

Revised: { v t } solved .

Revised: Bool mask .

Revised: Binary vector 1: for t = 0 to T − 1 do .

Revised: Hindsight 2: v t ← H ( episode ) t 3: D ← D ∪ ( { ( s t , a t , v t ) } , solved, mask )

Revised: Algorithm

Revised: Require:

Revised: D .

Revised: Replay Buﬀer size .

Revised: Batch size ratio .

Revised: Solved/unsolved 1: Initialize B ← ∅ 2: for b = 1 to size do 3: if b × ratio %1 = 0 then 4: select ← False 5: else 6: select ← True 7:

Revised: D 0 ← { d ∈ D : d. solved = select } 8:

Revised: Sample d ∈ D 0 with prob ( d ) ∝ len ( d. episode ) 9:

Original:  Deep-sea For Deep-see environment we encode state as one hot vector of size N 2 , and learn simple linear transformation for value estimation.
Revised: Sample ( s, v, m ) uniformly from d 10: B ← B ∪ { ( s, v, m ) } 11: return B B Architectures Deep-sea For Deep-see environment we encode state as one hot vector of size N 2 , and learn simple linear transformation for value estimation.

Original: Toy Montezuma Revenge Observation is represented as tuple containing current room location, agent position within the room and status of all keys and doors on the board.
Revised: Toy Montezuma Revenge Observation is represented as tuple containing current room location, agent position within the room and status of all keys and doors on the board.

Original: To estimate value we use fully-connected neural networks with two hidden layers consisting 50 neurons each.
Revised: To estimate value we use fully-connected neural networks with two hidden layers consistingneurons each.

Original: Single-board Sokoban Observation has  shape (10, 10, 7) where ﬁrst two coordinates are spatial and last one encodes type of ﬁeld (e.g. box, target, agent, wall).
Revised: Single-board Sokoban Observation has a shape (10, 10, 7) where ﬁrst two coordinates are spatial and the third coordinate one-hot encodes type of a state (e.g. box, target, agent, wall).

Original: To estimate value we ﬂatten the observation and apply fully-connected neural networks with two hidden layers consisting 50 neurons each.
Revised: To estimate value we ﬂatten the observation and apply fully-connected neural networks with two hidden layers consisting 50 neurons each.

Original: Multiple-boards Sokoban
Revised: Multiple-boards Sokoban

Original: Here we use  same observation  as in  single-board problem.
Revised: Here we use the same observation type as in the single-board problem.

Original: To generalize between different Sokoban levels we use convolutional neural network  with 5 hidden layers consisting 64 channels each.
Revised: Each value function network is composed of ﬁve 3 × 3 convolutional layers with with stride 1, followed by two fully connected layers with 128 units and 1 unit, respectively. C Training details Randomized priors In Osband et al.

Original: B T RAINING DETAILS
Revised: (2018, Lemma 3) showed that for Bayesian linear regression setting with Gaussian prior and noise model, generating samples from posterior distribution is equivalent to solving an appropriate optimization problem. To be exact, suppose D = { ( x i , y i ) } is the dataset, f θ ( x ) = x T θ is the regression function, (cid:15) i is a Gaussian noise, ˜ θ comes from a Gaussian prior, and ˜ y i =

Original: We considered two objectives when training value function from recorded episodes.
Revised: y i + (cid:15) i . Then the solution of the following problem [Equation] for some ζ > 0, is a sample from the posterior θ |D .

Revised: Consequently, Osband et al.

Revised: (2018) propose to use equation 3 as a training objective and include randomized prior in the value function approximator.

Revised: This objective is matched with the one used in Algorithm 1, and some of our experiments include randomized priors.

Revised: Masks An important decision for training, is how to assign transitions to the particular elements of value functions ensemble.

Revised: This is implemented using masks, see Algorithm 1.

Revised: Suppose a batch B is considered in an update step (see lines 9-10 of Algorithm 1) a let t

Revised: = ( s, v, m ) ∈ B be a transition.

Revised: A mask m ∈ { 0 , 1 } { 1 ,...,K } has the following interpretation: m i

Original: {+= 
+}
Revised: {+= 
+}

Revised: if and only if the transition t is used to train V θ i , i = 1 , . .

Revised: , K .

Revised: We experimented with the following versions of masking:

Revised: • dynamic masks: masks are generated anew whenever transition is sampled from replay buﬀer, • static masks: each transition is assigned a ﬁxed mask generated when added to the replay buﬀer.

Revised: In the cases of dynamic masks, each batch was split equally among the elements of ensemble (for this we kept the batch size to be the multiplicity of the number of ensembles).

Revised: The static masks were inspired by the Bootstrapped DQN, see (Osband et al., 2016, Appendix B), where it is a core idea.

Revised: We experimented with applying a diﬀerent mask to each transition according to the Bernoulli distribution, or to assign the same masks for all transitions in the same trajectory.

Original: Bootstap method was used in our Deep-see, Toy-Montezuma-Revenge and single-board Sokoban experiments.
Revised: We found it useful to use static masks in the Deep-sea, Toy Montezuma’s Revenge and Singleboard Sokoban experiments, and dynamic masks in Multiple-board Sokoban experiments.

Revised: Ensembles sampling Recall that equation 1 involves calculating the expected value, E θ ∼ Θ , with respect to the posterior distribution Θ.

Revised: In some experiments we instead subsample from Θ. Such an approach follows Osband et al.

Revised: (2016), which in turns is inspired by the classical Thomson sampling (see discussion (Osband et al., 2016, Section 4) and the original Thompson (1933)).

Revised: To be concrete, for a given a risk measure φ a equation 1 reduces to

Original: [Table]
Revised: [Equation]

Revised: where E = { 1 , . .

Revised: . , K } .

Revised: Sub-sampling, with a ﬁxed parameter ‘ , is equivalent to computing equation 4, with E taken as a random subset of { 1 , . .

Revised: , K } of cardinality ‘ .

Revised: Hiperparameters Table 3, below, summarizes the parameters of the training used in the

Revised: Deep-see, Toy Montezuma’s Revenge, Single-board Sokoban and Multiple-boards Sokoban experiments presented in Section 3.

Original: [Table]
Revised: [Table]