Width-Based Lookaheads Augmented with Base Policies for Stochastic Shortest Paths
Abstract
Sequential decision problems for real-world applications of-
ten need to be solved in real-time, requiring algorithms to perform well with a restricted computational budget.
Width- based lookaheads have shown state-of-the-art performance in classical planning problems as well as over the Atari games with tight budgets.
In this work we investigate width-based lookaheads over Stochastic Shortest paths (SSP).
We anal- yse
why width-based algorithms perform poorly over SSP problems, and overcome these pitfalls proposing a method to estimate costs-to-go.
We formalize width-based lookaheads as an instance of the rollout algorithm, give a deﬁnition of width for SSP problems and explain its sample complexity.
Our experimental results over a variety of SSP benchmarks show the algorithm to outperform other state-of-the-art rollout algorithms such as UCT and RTDP.
Introduction
Model-based lookahead algorithms provide the ability to autonomously solve a large variety of sequential decision making problems.
Lookaheads search for solutions by considering sequences of actions that can be made from the current state up to a certain time into the future.
For realworld applications decisions often need to be computed in real-time, requiring algorithms to perform with a restricted computational budget.
Limiting search in this way can result in considering states and trajectories which do not provide useful information.
To address this, lookaheads can be augmented with heuristics that estimate costs-to-go to prioritise states and trajectories and have been shown to perform well where computation budgets are restricted (Eyerich, Keller, and Helmert 2010).
This paper is concerned with Stochastic Shortest Path (SSP) problems which are often used to compare and evaluate search algorithms.
We consider the width-based family of planning algorithms, ﬁrst introduced by Lipovetzky and
Geffner (2012), which aim to prioritise the exploration of novel areas of the state space.
Two width-based planners,
Lipovetzky and Geffner’s breadth-ﬁrst search, IW(1), and the depth-ﬁrst search, Rollout-IW(1) (Bandres, Bonet, and
Geffner 2018), are investigated on SSP problems.
We ﬁrst provide the necessary background for SSP problems and width-based algorithms, while also formalising width-based algorithms as instances of the rollout algorithm (Bertsekas 2017).
We then show the motive to augment width-based lookaheads with cost estimates on SSP problems, deﬁne the width of SSP problems and propose a novel width-based algorithm that estimates costs-to-go by simulating a general base policy.
Our experimental study shows that the algorithm compares favourably to the original Rollout-IW(1) algorithm and to other state-of-the-art instances of the rollout algorithm.
Optimal Control and Dynamic Programming
We concern ourselves with the problem of decision under stochastic uncertainty over a ﬁnite number of stages, which we characterise following closely the presentation of Bertsekas (2017).
We are given a discrete-time dynamic system
[Equation]
where the state x k is an element of a space
S k ⊂ R d , the control u k is an element of space C k ⊂ N , and the random disturbance w k is an element of a space D k ⊂ R m 1 .
The control u k is constrained to take values in a given non-empty subset U ( x k ) ⊂
C k , which depends on the current state x k , so that u k ∈ U k ( x k ) for all x k ∈ S k and k .
The random disturbance w k is characterised by a probability distribution
P k ( ·| x k , u k ) that may depend explicitly on x k and u k but not on the values of previous disturbances w k − 1 , . .
w 0 .
We consider the class of policies , or control laws , corresponding to the sequence of functions
[Equation]
where µ k maps states x k into controls u k = µ k ( x k ) and is such
that µ k ( x k ) ∈ U ( x k ) for all x k ∈ S k .
Such policies will be called admissible .
Given an initial state x 0 and admissible policy π , the states x k and disturbances w k are random variables with distributions deﬁned through the system equation
[Equation]
Thus, for given functions g f (terminal cost) and g the expected cost of π starting at x 0 is
[Equation]
where the expectation is taken over the random variables w k and x k .
An optimal policy π ∗ is one that minimises this cost
[Equation]
where Π is the set of all admissible policies.
The optimal cost J ∗ ( x 0 ) depends on x 0 and is equal to J π ∗ ( x 0 ) .
We will refer to J ∗ as the optimal cost or optimal value function that assigns to each initial state x 0 the cost J ∗ ( x 0 ) .
Stochastic Shortest Path
We use Bertsekas’ (2017) deﬁnition, that formulates Stochastic Shortest Path (SSP) problems as the class of optimal control problems where we try to minimize
J
[Equation]
with α set to 1 and we assume there is a cost-free termination state t which ensures that J π ( x 0 ) is ﬁnite.
Once the system reaches that state, it remains there at no further cost, that is, f ( t , u, w )
= t with probability 1 and g ( t , u, w ) =
for all u ∈ U ( t ) .
We note that the optimal control problem deﬁned at the beginning of this section is a special case where states are pairs ( x k , k ) and all pairs ( x N , N ) are lumped into termination state t .
In order to guarantee termination with probability 1 , we will assume that there exists an integer m such that there is a positive probability that t will be reached in m stages or less, regardless of what π is being used and the initial state x 0 .
That is, for all admissible policies and i = 1 , ..., m it holds
[Equation]
A policy π will be proper if the condition above is satisﬁed for some m , and improper otherwise.
The Rollout Algorithm
A particularly effective on-line approach to obtain suboptimal controls is rollout , where the optimal cost-to-go from current state x k is approximated by the cost of some suboptimal policy and a d -step lookahead strategy.
The seminal
RTDP (Barto, Bradtke, and Singh 1995) algorithm, is an instance of the rollout strategy where the lookahead is uniform, d = 1 , and controls ¯ µ ( x k ) selected at stage k and for state x
k are those that attain the minimum
[Equation]
where ˜ J k +1 is an approximation on the optimal cost-to-go
J ∗ k
+1 .
If the approximation is from below, we will refer to it as a base heuristic , and can either be problem speciﬁc (Eyerich,
Keller, and Helmert 2010), domain independent (Bonet and
Geffner 2003; Yoon, Fern, and Givan 2007) or learnt from interacting with a simulator (Mnih et al. 2015).
Alternatively, ˜
J k +1 can be deﬁned as approximating the cost-to-go of a given suboptimal policy π , referred to as a base policy , where estimates are obtained via simulation (Rubinstein and Kroese 2017).
We will denote the resulting estimate of cost-to-go as H k ( x k ) 2 .
The result of combining the lookahead strategy and the base policy or heuristic is the rollout policy , ¯ π { ¯ µ 0 , ¯ µ 1 , . .
, ¯ µ N − 1 } with associated cost ¯ J ( x k ) .
Such policies have the property that for all x k and k
[Equation]
when H k is approximating from above the cost-to-go of a policy, as shown by Bertsekas (2017) from the DP algorithm that deﬁnes the costs of both the base and the rollout policy.
To compute at time
k
the rollout control ¯ µ ( x k ) , we compute and minimize over the values of the Q -factors of state and control pairs ( x l , u l ) ,
[Equation]
for admissible controls u l ∈ U ( x l ) , l = k + i , with i = 0 , ..., d − 1 , and
[Equation]
for l = k + d .
In this paper we make a number of assumptions to ensure the viability of lookaheads with d > 1 .
We will assume that we can simulate the system in Eq.
3 under the base policy, so we can generate sample system trajectories and corresponding costs consistent with probabilistic data of the problem.
We further assume that we can reset the simulator of the system to an arbitrary state.
Performing the simulation and calculating the rollout control still needs to be possible within the real-time constraints of the application, which is challenging as the number of Q -factors to estimate and minimizations to perform in Equations 9-10 is exponential on the average number of controls available per stage and d , the maximum depth of the lookahead.
We avoid the blowup of the size of the lookahead by cutting the recursion in Equation 9 and replacing the right hand side by that of Equation 10.
As detailed in the next section, we will do this when reaching states x l that are deemed not to be novel according to the notion of structural width by Lipovetzky and Geffner (2012).
This results in a selective strategy alternative to the upper conﬁdence bounds (Auer, Cesa-Bianchi, and Fischer 2002) used in popular instances of Monte-Carlo Tree Search (MCTS) algorithms like Kocsis and Szepesvari’s (2006) UCT, that also are instances of the rollout algorithm (Bertsekas 2017).
Width-Based Lookaheads
We instantiate the rollout algorithm with an l -step, depth-selective lookahead policy using Width-based
Search (Lipovetzky and Geffner 2012).
These algorithms both focus the lookahead and have good any-time behaviour.
When it comes to prioritisation of expanding states, width-based methods select ﬁrst states with novel valuations of features deﬁned over the states (Lipovetzky, Ramirez, and Geffner 2015;
Geffner and Geffner 2015).
The most basic width-based search algorithm is IW (1), a plain breadth-ﬁrst search , guaranteed to run in linear time and space as it only expands novel states.
A state x l is novel if and only if it encounters a state variable 3 x i ⊂ R , whose value v ∈ D ( x i ) , where
D ( x i ) is the domain of variable x
i , has not been seen before in the current search.
Note that novel states are independent of the objective function used, as the estimated cost-to-go
J is not used to deﬁne the novelty of the states.
IW(1) has recently been integrated as an instance of a rollout algorithm, and has been shown to perform well with respect to learning approaches with almost real-time computation budgets over the Atari games (Bandres, Bonet, and Geffner 2018).
Depth-First Width-Based Rollout
The breadth-ﬁrst search strategy underlying IW(1) ensures a state variable x
i is seen for the ﬁrst time through the shortest sequence of control steps, i.e. the shortest path assuming uniform costs g ( x , u, w ) .
On the other hand, depth-ﬁrst rollout algorithms cannot guarantee this property in general.
Rollout IW (RIW) changes the underlying search of IW into a depth-ﬁrst rollout.
In order to ensure that RIW(1) considers a state to be novel iff it reaches at least one value of a state variable x il through a shortest path, we need to adapt the deﬁnition of novelty.
Intuitively, we need to deﬁne a set of state features to emulate the property of the breadth-ﬁrst search strategy.
Let d ( x i , v ) be the best upper bound known so far on the shortest path to reach each value v ∈ D ( x i ) of a state variable from the root state x k .
Initially d ( x i , v ) =
N for all state variables, where N is the horizon which is the maximum search depth allowed for the lookahead, thus denoting no knowledge initially.
When a state x l is generated, d ( x i , v ) is set to l for all state variables where l < d ( x i , v ) .
Since RIW(1) always starts each new rollout from the current state x k , in order to prove a state x l to be novel we have to distinguish between x l being already in the lookahead tree and x l being new.
If x l is new in the tree, to conclude it is novel, it is sufﬁcient to show that there exists a state variable x i whose known shortest path value
d ( x i , v ) > l .
If x l is already in the tree, we have to prove the state contains at least one state variable value x i whose shortest path is l = d ( x i , v ) , i.e. state x l is still novel and on the shortest path to x i .
Otherwise the rollout is terminated.
In order to ensure the termination of RIW(1), non-novel states are marked with a solved label.
The label is backpropagated from a state x l +1 to x l if all the admissible control inputs u ∈ U ( x l ) yield states x l +1 =
f l ( x l , u, w l ) already labeled as solved .
RIW(1)
terminates once the root state is labeled as solved (Bandres, Bonet, and Geffner 2018).
Nonnovel states x l are treated as terminals and their cost-to-go is set to 0 .
This can induce a bias towards non-novel states rather than true terminal states.
In the next section we investigate how to overcome the ill-behaviour of RIW(1) when a state x l is non-novel.
We discuss the importance of estimating upper-bounds on the cost-to-go H l ( x l ) instead of assigning termination costs.
This turns out to be essential for RIW(1) over SSPs.
Width-Based Lookaheads on SSPs
Despite the successes of width-based algorithms on a variety of domains including the Atari-2600 games (Lipovetzky,
Ramirez, and Geffner 2015; Bandres, Bonet, and Geffner 2018), the algorithms, as will be shown, have poor performance on SSP problems.
We illustrate this with two scenarios.
First, width-based lookaheads prefer trajectories leading to non-novel states over longer ones that reach a goal.
Second, and speciﬁc to depth-ﬁrst width-based lookaheads, we show that useful information is ignored.
We can demonstrate these scenarios using a simple SSP problem with uniform and unitary action costs, shown in Figure 1.
The task is to navigate to a goal location using the least number of left, right, up or down actions.
Any action that would result in the agent moving outside of the grid produces no change in its position.
The features used by the width-based planners are the coordinates for the current agent position.
Both IW(1) and
RIW(1) algorithms, given a sufﬁcient budget, would result in the lookahead represented by yellow lines in Figure 1.
As expected, both lookaheads contain the shortest paths to make each feature of the problem true.
For both IW(1) and RIW(1), we back up the costs found in the lookahead starting from terminal and non-novel states.
In this instance a move down or left from the agent’s initial state has no effect, thus immediately producing a non-novel state.
When backing up values, down and left have an expected cost of 1, which is less than the optimal cost of 2 for up, the action that leads to the top left goal state.
This prevents both IW(1) and RIW(1) from ever achieving the goal, as they keep selecting those useless actions.
Furthermore, if the goal is the top right location in
Figure 1, RIW(1)’s random action selection can generate a trajectory that reaches the goal.
Yet, trajectories leading to the goal are pruned away, as non-novel states in later considered trajectories are treated as terminals, again resulting in the lookahead represented by the yellow lines in Figure 1.
Bandres et al.
(2018) introduced the algorithm RIW in the context of deterministic transition functions.
In this section we discuss its properties in the context of SSPs.
The set of features used to evaluate the novelty of a state is
F = { ( v, i, d ) | v ∈ D ( x i ) } where D ( x i ) is the domain of variable x
i , and d is a possible shortest path distance.
Note that the horizon N is the upper-bound of d .
The maximum number of novel states is O ( | F | ) , as the maximum number of shortest paths for a feature ( v, i, · ) ∈ F is N .
That is, in the worst case we can improve the shortest path for ( v, i, · ) by one control input at a time.
The labeling of nodes ensures the number of rollouts from the initial state in RIW(1) is at most O ( | F | × b ) , where b = max x l | U ( x l ) | is the maximum number of applicable control variables in a state, i.e. maximum branching factor.
When the labeling is applied to stochastic shortest path problems, the resulting lookahead tree is a relaxation of the original
SSP, as it allows just one possible outcome of a control input.
Alternatively, one can back-propagate the label solved to a state x l iff 1) all admissible control inputs u ∈ U ( x l ) have been applied resulting in states labeled as solved , and 2) the tree contains all the possible resulting states of each control input u ∈ U ( x l ) .
We refer to this new strategy to backpropagate labels as λ -labeling.
We denote as λ the maximum number of states that can result from applying u ∈ U ( x l − 1 ) in a state x l .
That is, λ = max x ,u,w | f
( x , u, w ) | .
RIW(1) with λ -labeling will terminate after at most O ( | F | × b × λ ) rollouts.
Furthermore, we can reconcile the notion of width over classical planning problems (Lipovetzky and Geffner 2012) with SSPs.
A terminal state t made of features f ∈ F has width 1 iff there is a trajectory x 0 , u 0 , . . .
, u n − 1 , x n for n ≤
N where x n = t , such that for each x j in the trajectory 1) the preﬁx x 0 , u 0 , . . .
, u j − 1 , x j reaches at least one feature f j = ( v, i, d ) ∈ F where all ( v, i, d (cid:48) )
∈ F for d (cid:48) <
d are unreachable, i.e., it is a shortest path possible to reach a value in x i , 2) any shortest path to f j can be extended with a single control input u into a shortest path for a feature
f j + complying with property 1) in state x j +1 , and 3) the shortest path for f n is also a shortest path for termination state t .
RIW(1)
with the new labeling strategy is guaranteed to reach all width 1 terminal states t .
Theorem 1.
Rollout IW(1) with λ -labeling is guaranteed to reach every width 1 terminal state t in polynomial time in the number of features F if λ = 1 .
If λ = ∞ , RIW(1) will not propagate any solved label, and terminate only when the computational budget is exhausted.
For simplicity, we assumed shortest paths are equivalent to the shortest sequence of control inputs.
To generalize to positive non-uniform costs, the distance d in the features should keep track of the cost of a path (cid:80) i g ( x i , u i , w i ) instead of its length, and the horizon be applied to the cost of the path.
Cost-to-go Approximation
The most successful methods for obtaining cost-to-go approximations have revolved around the idea of running a number of Monte Carlo simulations of a suboptimal base policy π (Ginsberg 1999; Coulom 2006).
This amounts to generating a given number of samples for the expression minimized in Equation 7 starting from the states x l over the set of admissible controls u l ∈ U ( x l ) in Equation 10, averaging the observed costs.
Three main drawbacks (Bertsekas 2017) follow from this strategy.
First, the costs associated with the generated trajectories may be wildly overestimating J ∗ ( x l ) yet such trajectories can be very rare events for the given policy.
Second, some of the controls u l may be clearly dominated by the rest, not warranting the same level of sampling effort.
Third, initially promising controls may turn out to be quite bad later on.
MCTS algorithms aim at combining lookaheads with stochastic simulations of policies and aim at trading off computational economy with a small risk of degrading performance.
We add two new methods to the MCTS family, by combining the multi-step, width-based lookahead strategy discussed in the previous section with two simulation-based cost-to-go approximations subject to a computational budget that limits the number of states visited by both the lookahead and base policy simulation.
Width-based Lookaheads with Random Walks
The ﬁrst method, which we call RIW-RW, uses as the base policy a random walk , a stochastic policy that assigns the same probability to each of the controls u admissible for state x , and is generally regarded as the default choice when no domain knowledge is readily available.
A rolling horizon H is set for the rollout algorithm that follows from combining the RIW(1) lookahead with the simulation of random walks.
The maximal length of the latter is set to H − l , where l is the depth of the non-novel state.
Both simulations and the unrolling of the lookahead are interrupted if the computational budget is exhausted.
While this can result in trajectories sometimes falling short from a terminal, it keeps a lid on the possibility of obtaining extremely long trajectories that eat into the computational budget allowed and preclude from further extending the lookahead or sampling other potentially more useful leaf states x l .
Worst-Case Estimates of Rollout Costs
One of the most striking properties of rollout algorithms is the cost improvement property in Equation 8, suggesting that upper bounds on costs-to-go can be used effectively to approximate the optimal costs J ∗ .
Inspired by this, the second width-based MCTS method we discuss leverages the sampling techniques known as stochastic enumeration (SE) (Rubinstein and Kroese 2017) to obtain an unbiased estimator for upper bounds on costs-to-go, or in other words, estimates the maximal costs a stochastic rollout algorithm with a large depth lookahead can incur.
SE methods are inspired by a classic algorithm by D. E.
Knuth to estimate the maximum search effort by backtracking search (1975).
Knuth’s algorithm estimates the total cost of a tree T with root u keeping track of two quantities, C the estimate of the total cost, and D the expectation on the number of nodes in T at any given level of the tree, and the number of terminal nodes once the algorithm terminates.
Starting with the root vertex u and D ← 1 , the algorithm proceeds by updating D to be D ← |S ( u ) | D and choosing randomly a vertex v from the set of successors S ( u ) of u .
The estimate C is then updated C ← C + c ( u, v ) D using the cost of the edge between vertices u and v .
These steps are then iterated until a vertex v (cid:48) is selected
s.t. S ( v (cid:48) )
= 0 .
We observe that Knuth’s C quantity would correspond to the worst-case cost-to-go ¯ J ( x ) k of a rollout algorithm using a lookahead strategy with d set to the rolling horizon H and the trivial base heuristic that assigns 0 to every leaf state.
Furthermore, we assume that the algorithm either does not ﬁnd any terminals within the limits imposed by the computational budget assigned, or if it ﬁnds one such state, it is too the very last one being visited.
Lookaheads deﬁne trees over states connected by controls, edge costs c ( u, v ) correspond directly with realisations of the random variable g ( x , u, w ) and the set of successors S ( v ) of a vertex corresponds with the set of admissible controls U ( x ) .
While Knuth’s algorithm estimates are an unbiased estimator, the variance of this estimator can be exponential on the horizon, as the accuracy of the estimator lies on the assumption that costs are evenly distributed throughout the tree (Rubinstein and Kroese 2017).
In the experiments discussed next, we use Knuth’s algorithm directly to provide H k ( x k ) , adding the stopping conditions to enforce the computational budget and limiting the length of trajectories to H − l as above.
In comparison with simulating the random walk policy, the only overhead incurred is keeping up-to-date quantities C and D with two multiplications and an addition.
Experimental Study Domains
To evaluate the different methods we use a number of
GridWorld (Sutton and Barto 2018) domains, an instance of a SSP problem.
The goal in GridWorld is to move from an initial position in a grid to a goal position.
In each state 4 actions are available: to move up, down, left or right.
Any action that causes a move outside of the grid results in no change to the agent’s position.
Actions have a cost of 1, with the exception of actions that result in reaching the goal state, that have a cost of 0.
The complexity of GridWorld can be scaled through the size of the grid and the location and number of goals.
GridWorld also allows for extensions, which we use to have domains with a stationary goal, moving goals, obstacles and partial observability.
For each instance of the GridWorld domain we have a d 0 × d 1 grid, and the state is the current location of the agent, x = ( a 0 , a 1 ) where a i is the agent’s position in dimension i .
The transition function is formalised as
[Equation]
where, ef speciﬁes the change in the agent’s position for each action, T k ⊂ S k is the set of goal states and x k +1 =
x k where the condition in Equation 11 is not met.
The cost of a transition is deﬁned as
[Equation]
otherwise, g k ( x k , u k , w k ) = 1 .
For the stationary goal setting we have a single goal state which is positioned in the middle of the grid by dividing
[Table]
and rounding d 0 and d 1 by two.
The problem setting with moving goals , has the set of goal states modiﬁed as follows
[Equation]
where δ t k gives the relative change of the goal state t k for the time step k + 1 and T k
+1 =
T k if x k ∈ T k .
We use
T
= { (0 , d 1 − 1) , ( d 0 − 1 , 0) } , d 0 = d 1 and δ t k =   (1 ,
[Equation]
1) if
[Equation] [Equation]
[Equation] [Equation]
[Equation]
if
[Equation]
δ
[Equation]
otherwise (14)
Resulting in two goals starting at opposite corners of the grid moving back and forth on the same diagonal.
The obstacles setting, uses the stationary goal, but modiﬁes S k such that,
[Equation]
where O ⊂ N 2 and is the set of obstacles, that is grid cells in which the agent is not allowed.
Having partially observable obstacles in GridWorld provides an instance of the stochastic Canadian Traveller Problem (CTP) (Papadimitriou and Yannakakis 1991).
The objective in CTP is to ﬁnd the shortest path between one location in a road map to another, however, there is a known probability for each road in the map that due to weather conditions the road is blocked.
A road in CTP can only be observed as being blocked or unblocked by visiting a location connected to it, and once a road status is observed the status remains unchanged.
In terms of the GridWorld problem, each grid cell has a known probability of being a member of the obstacle set, O .
The agent can only observe cells as being obstacles or not when it is in a neighbouring cell.
Once a grid cell is observed it is then known that it is either an obstacle or not for the remaining duration of the episode.
John Langford designed two MDP problems 5 described as Reinforcement Learning (RL) "Acid" intended to be
Dim.
Alg.
[Table]
difﬁcult to solve using common RL algorithms, such as Qlearning.
Langford’s two problems allow two actions from every state.
The state space for the problems is S k
= { i | 0 ≤ i < N } where the number of states value, N , allows the complexity of the problem to be controlled.
Langford originally speciﬁed the problems as reward-based, here we modify them to be SSP cost-based problems.
Reward shaping is commonly used to make Reinforcement Learning easier by encouraging actions, through higher rewards, towards a goal state or states.
The ﬁrst of Langford’s problems is named Antishaping and uses reward shaping to encourage actions away from the goal state.
Antishaping has the transition function
[Equation]
otherwise, the state remains unchanged, x k +
= x k .
The set containing the goal state is T k
=
N − 1 } , which can be achieved by continuously selecting u k = 0 .
The cost of each transition in Antishaping is 0.25 divided by N - x k +1 , except when x k +1 =
N − 1 where the cost is 0.
The problem becomes a large plateau where longer paths become more costly at larger rates.
The motivation behind
Langford’s second problem, Combolock , is if many actions lead back to a start state, random exploration is inadequate.
The Combolock problem has the transition function
[Equation]
otherwise x k +1 is equal to the initial position of 0.
The goal state is T k
= {
N − 1 } and sol x k is either 0 or 1 assigned to state x k which remains constant.
For each state x ∈ S , sol x has an equal chance of either being 0 or 1.
The cost of each transition in Combolock is 1 except for the transition that leads to the terminal state N − 1 where the cost is 0.
While common Reinforcement Learning algorithms such as
Q-Learning methods will struggle to solve these domains, it is claimed by Langford that the E 3 (Kearns and Singh 2002) family of algorithms, whose exploration do not rely solely on random policies or reward feedback but on exploring the maximum number of states, will perform well.
Dim.
Alg.
Heu.
Simulator Budget   1Stp
Man.
38.1 ± 2.
[Table]
Methodology
We evaluate the depth-ﬁrst width-based rollout algorithm,
RIW(1), with and without being augmented using base policies.
λ = 1 is used for the labels back-propagation.
We did not observe statistically signiﬁcant changes with λ = ∞ .
For the GridWorld domain we deﬁne the features on which
RIW(1) plans over as F = { ( a, i, d ) | a ∈ D
( x i ) } where d is the length of the control input path from the initial state, a is the agent’s position in the grid in dimension i and D ( x i ) is the domain of the agent’s position, a , in dimension i .
For
Antishaping and Combolock the feature set will be
F = { ( i, d ) | i ∈ N ) } where i is the state number the agent is in and N is the number of states of the domain.
Two additional rollout algorithms are also considered, the one-step rollout algorithm, RTDP (Barto, Bradtke, and Singh 1995) and the multi-step, selective, regret minimisation, rollout algorithm Upper Conﬁdence bounds applied to Trees (UCT) (Kocsis and Szepevari 2006).
The exploration parameter of UCT is set to 1.0 for all experiments.
For all the methods that use a base policies the maximum depth of a simulated trajectory is equal to H − l , where l is the depth at which the simulated trajectory began and H is the horizon value of the lookahead.
Also, a single, as opposed to multiple, simulated trajectory for the cost-to-go approximation is used, as initial results indicated it is favourable.
We also report the algorithms using a Manhattan distance heuristic for the
GridWorld domains that use obstacles.
Using the Manhattan distance for the GridWorld problems with obstacles provides a lower bound on the cost-to-go.
Each method on the domains is evaluated at different levels of complexity by varying the number of states.
The methods are evaluated using different simulator budgets.
The simulator budgets are the maximum simulator calls allowed for the evaluation at each time step.
For each algorithm and domain
[Table]
setting we evaluate the performance over 10 different initial states with 20 episodes per initial state, equalling a total of 200 episodes.
The values reported here for each algorithm and domain setting are the average and 95% conﬁdence interval of the costs across the 200 episodes.
Each episode was run using a single AMD Opteron 63xx class CPU @ 1.8 GHz, with an approximate runtime of 0.75 seconds per 1,000 simulator calls across the different algorithm and domain settings.
Results are also presented for the Atari-2600 game
Skiing , which is a SSP problem.
We use the OpenAI gym’s (Brockman et al. 2016) interface of the Arcade Learning Environment (ALE)
(Bellemare et al. 2013) and use the slalom game mode of Skiing .
In the slalom mode the aim is to ski down the course as fast as possible while going through all the gates.
Once the ﬁnish line is reached, a 5 second time penalty is applied for each gate that is missed.
The reward values provided by ALE for Skiing is the negative value of the total time taken plus any time penalties in centiseconds, which we use as a cost.
We use the environment settings as described by Machado et al.
(2018) with a frame skip ofand a probability of 0.25 of repeating the previous action sent to environment instead of the current one, which Machado et al.
call sticky actions.
For evaluation we use a simulator budget of 100 and partial caching as described by Bandres et al.
(2018), in that we cache simulator state-action transitions, thus assuming determinism, but clear the cached transitions when executing an action in the environment.
However, the lookahead tree itself is not cleared when executing an action in the environment as is done for the other domains trialed.
The maximum episode length is capped at 18,000 frames with a frame skip of 5 this equals 3,600 actions.
Using a simulation based cost-to-go approximation is infeasible with a simulator budget of 100 and the maximum episode length of 3,600 actions.
Therefore we report the algorithms using a heuristic cost-to-go estimate, which is the the number of gates that have either been missed or are still left times the time penalty of 500 centiseconds.
For the RIW(1) algorithms we use the pixel values from the current gray scaled screen at full resolution, that is 180 by 210 pixels, as features.
All experiments were run within the OpenAI gym frame-
[Table]
work (Brockman et al. 2016) and the code used for the algorithms and domains is available through GitHub 6 .
Results
The different H functions reported here are H NA
=
the random policy H Rnd , and the Manhattan distance H Man .
The algorithms were also evaluated using Knuth’s algorithm with a different range of rollouts for the cost-to-go estimate, however, the results are not reported here as they are either statistically indifferent or dominated by the results using H Rnd with a single rollout.
Bertsekas (2017) suggests that MCTS algorithms should readily beneﬁt from stronger algorithms to estimate costs-to-go by simulation of stochastic policies.
Our experiments showed that if synergies exist these do not manifest when using off-the-shelf stochastic estimation techniques like the ones discussed by Rubinstein and Kroese (2017).
Table 1, 2 and 3 report the results of the different lookahead algorithms on the GridWorld domain variants with a stationary goal, moving goals and obstacles respectively.
For these three domains, results were also collected for a 100x100 grid, however, the results were omitted from the tables as the simulator budgets used were not sufﬁcient to ﬁnd anything meaningful.
The results on the stationary goal GridWorld domain shown in Table 1 provide a number of insights about the rollout algorithms reported.
First, we can see RIW(1) beneﬁts from using H Rnd rather than H NA where the larger simulator budgets are used.
As the simulator budget increases, as could be expected, so does the performance of all the methods using H Rnd .
On the contrary, with H NA RIW(1)’s performance remains constant across the different budgets.
The explanation for this can be found in the motivating example we gave previously in this paper with the agent preferring the shorter trajectory of driving into the boundary of the grid.
Tablealso shows that given the largest budget and H Rnd , RIW(1) statistically outperforms the other algorithms on the three domains of different size.
Table 2 for GridWorld with moving goals has similar patterns as the stationary goal domain in that RIW(1) with
H Rnd dominates performance for the largest budget.
Also, the majority of performances for the smaller budgets, excluding
RIW(1) with H NA , are statistically indifferent.
[Table]
GridWorld with a stationary goal and obstacles results displayed in Table 3 continues the trend of results.
Using the largest budget RIW(1) with H Rnd outperforms all methods on the 10x10 and 20x20 domains.
For the 50x50 a number of results are statistically indifferent.
For this domain the algorithms using H Man as the base heuristic are also reported.
While using the largest budget on the 10x10 grid H Rnd dominates H Man , for the larger 50x50 we see H Man dominates
H Rnd for UCT, and is competitive for the other methods.
For the smallest simulator budget on CTP reported in Table 4 using H Man with RIW(1) and UCT are the dominate methods.
For the largest simulator budget RIW(1) using H Rnd is dominant over all other methods for both sized domains.
We also see that in most cases for the two smaller budgets
H Man dominates the H Rnd methods.
Table 5 and 6 show on the smaller 10 state domains
RIW(1) with H Rnd is statistically dominant over all other methods on Antishaping and Combolock for theand 1000 simulator budgets.
However, for the more complex 50 state domains, the results of all algorithms using H Rnd are statistically indifferent.
It can be observed that using H Rnd with RIW(1) does improve its performance compared with
H NA across all the domain settings with simulator budgets of 500 and 1000, besides Antishaping with 50 states.
For the Skiing Atari-2600 game results in Table 7 H Heu is the heuristic value based on the number of gates missed and remaining as described in the previous section.
RIW(1) using
H Heu dominates all other methods.
Comparing RIW(1) using
H Heu results with those reported by Machado et al.
(2018), it has similar performance to the DQN algorithm (Mnih et al. 2015) after 100 million frames of training.
Since the simulation budget per action we use here is equivalent toframes, and given that the maximum episode duration spans 3,600 actions, RIW(1) achieves the performance in Tableconsidering only 1.8 million frames.
Related Work
Bertsekas (2017) considers AlphaGo Zero (Silver et al. 2017) to be state-of-the-art in MCTS algorithms.
It combines the reasoning over conﬁdence intervals ﬁrst introduced with
UCT (Kocsis and Szepevari 2006) and the classic simulation of base policies (Ginsberg 1999), adding to both supervised learning algorithms to obtain, ofﬂine, parametric representations of costs-to-go which are efﬁcient to evaluate.
The resulting algorithm achieves super-human performance at the game of Go, long considered too hard for AI agents.
Rather
[Table]
Skiing game over 100 episodes.
than using descriptions of game states directly as we do, AlphaZero uses a CNN to extract automatically features that describe spatial relations between game pieces.
Like us, AlphaZero’s lookahead uses a stochastic policy to select what paths to expand, but rather than Q -factors, uses estimated win probabilities to prioritise controls, and simulates the opponent strategy via self-play to generate successor states.
Our simulators are always given and remain unchanged, rather than being dynamic as is the case for AlphaZero.
Junyent et al.
(2019) have recently presented a hybrid planning and learning approach that integrates Bandres et al.
(2018) rollout, with a deep neural network.
Similarly to
AlphaZero, they use it to both guide the search, and also to extract automatically the set of state features F .
Interestingly,
Junyent et al.
’s
work does not use deep neural networks to approximate costs-to-go as AlphaZero does.
A signiﬁcant improvement in performance over Bandres et al.
original rollout algorithm is reported with policies learnt after 40 million interactions with the simulator, using an overall computational budget much bigger than the one used by Bandres et al.
Discussion
MCTS approaches typically combine lookaheads and cost-togo approximations, along with statistical tests to determine what are the most promising directions and focus their sampling effort.
The width-based methods described in this paper do so too, but in ways which are, at ﬁrst sight, orthogonal to existing strategies.
It remains an area of active research to map out exactly how the width-based methods described in this paper, and those elsewhere by Junyent et al.
(2019) too, provide alternatives to the limitations of existing MCTS approaches.
Having said this, there is no general theory guiding the design of MCTS algorithms (Bertsekas 2017), and to avoid generating ad-hoc, problem dependent solutions involuntarily it is important to follow strict protocols that alert of potential lack of statistical signiﬁcance in results, while relying on a diverse set of benchmarks that can be both easily understood, and highlight limitations of existing state-of-theart methods and overcome them.
Acknowledgments
This research was supported by the Australian Government
Research Training Program Scholarship provided by the Australian Commonwealth Government and the University of
Melbourne.
The research was also supported by the Defence Science Institute, an initiative of the State Government of Victoria and Google through a Google PhD Travel Scholarship.