Inferential Induction: A Novel Framework for Bayesian Reinforcement Learning
Abstract
Bayesian Reinforcement Learning (BRL) offers a decision-theoretic solution to the reinforcement learning problem.
While “model-based” BRL algorithms have focused either on maintaining a posterior distribution on models, BRL “modelfree” methods try to estimate value function distributions but make strong implicit assumptions or approximations.
We describe a novel Bayesian framework, inferential induction , for correctly inferring value function distributions from data, which leads to a new family of BRL algorithms.
We design an algorithm, Bayesian
Backwards Induction (BBI), with this framework.
We experimentally demonstrate that BBI is competitive with the state of the art.
However, its advantage relative to existing BRL model-free methods is not as great as we have expected, particularly when the additional computational burden is taken into account.
Introduction
Many Reinforcement Learning (RL) algorithms are grounded on the application of dynamic programming to a Markov Decision Process (MDP)
[Sutton and Barto, 2018].
When the underlying MDP µ is known, efﬁcient algorithms for ﬁnding an optimal policy exist that exploit the Markov property by calculating value functions.
Such algorithms can be applied to RL, where the learning agent simultaneously acts in and learns about the MDP through e g. stochastic approximations, without explicitly reasoning about the underlying MDP.
Hence, these algorithms are called modelfree.
In Bayesian RL (BRL) [Ghavamzadeh et al., 2015] we represent our knowledge about the underlying MDP µ through some prior distribution β over a set M of possible MDPs.
This explicit representation of uncertainty admits algorithms that can perform near-optimal exploration, in contrast to methods that only rely on a single empirical model of the MDP, which require exploration heuristics.[c.f.
Vlassis et al., 2012] While model-based BRL is well-understood, many BRL algorithms try to become model-free by calculating distributions on value functions.
Unfortunately, these methods typically make implicit assumptions or approximations about the underlying MDP distribution.
In the rest of this section, we provide background in terms of setting and related work.
In Section 2, we explain our Inferential Induction framework and three different inference methods that emerge from it, before instantiating one of them into a concrete procedure.
Based on this, Section 3 describes the BBI algorithm.
In Section 4, we experimentally compare BBI with state-of-the art BRL algorithms.
In the appendix additional experimental results are available in Appendix B and some implementation details in Appendix C.
1.1 Setting and Notation
In this paper, we generally use P and E to refer to probability (measures) and expectations while allowing some abuse of notation for compactness.
Reinforcement Learning (RL) is a sequential learning problem faced by agents acting in an unknown environment µ , typically modelled as a Markov decision process [c.f. Puterman, 2005].
Deﬁnition 1.1 (Markov Decision Process (MDP)) .
An MDP µ with state space S and action space A is equipped with a reward distribution P µ ( r | s ) with corresponding expectation ρ µ ( s ) and a transition kernel P µ ( s ′ | s, a ) for states s, s ′ ∈ S and actions a ∈ A .
At time t , the agent observes 1 the environment state s t , and then selects an action a t .
Then, it receives and observes a reward r t and a next state s t +1 .
The agent is interested in the utility U t ≜ P
Tk = t γ k − t r k , i.e. the sum of future rewards r t .
Here, γ ∈ (0 , 1] is the discount factor and T ∈ [1 , ∞ ] is the problem horizon.
Typically, the agent wishes to maximise the expected utility , but other objectives are possible.
The agent acts in the environment using a policy π = ( π 1 , . . .
, π t , . . . )
that takes an action a t at time t with probability π t ( a t
| s t , r t − 1 , a t − 1 , s t − 1 , . . .
, r 1 , a 1 , s 1 ) .
Dependence on the complete observation history is necessary, if the agent is learning from experience.
However, when µ is known, the policy π ∗ µ maximising expected utility over ﬁnite horizon is Markovian 2 of the form π t ( a t
| s t ) and is computable using dynamic programming.
A useful algorithmic tool for achieving this is the value function, i.e. the expected utility of a policy π from different starting states and action:
Deﬁnition 1.2 (Value Function) .
The state value function of policy π in MDP µ is V πµ,t ( s ) ≜
E πµ ( U t | s t = s ) and the corresponding state-action (or Q-)value function is Q πµ,t ( s, a ) ≜ E πµ
( U t | s t = s, a t = a ) .
P πµ and E πµ denote probabilities and expectations under the process induced by π and µ .
Finally, the Bellman operator B πµ V ( s ) ≜ ρ µ ( s ) + γ P s ′ ∈S P πµ ( s ′ | s ) V ( s ′ ) allows us to compute the value function recursively through V πµ,t
=
B πµ V πµ,t +1 .
Bayesian RL (BRL).
In BRL, our subjective belief is represented as a probability measure β over possible MDPs.
We refer to the initial belief β as the prior distribution .
By interacting with the environment until time t , the agent obtains data D t =
( s 1 , a 1 , r 1 , . . .
, s t ) .
This data is used to calculate a posterior distribution β ( µ | D t ) that represents agent’s current knowledge about the
MDP.
For a given belief and an adaptive policy π β ( s ) , we deﬁne the Bayesian value function to be: [Equation]
[Equation]
The Bayesian value function is the expected value function under the distribution β .
The Bayesoptimal policy achieves the Bayes-optimal value function V ∗ β,t ( s ) =
sup π V πβ,t .
Calculating V πβ,t involves integrating V πµ,t for all µ , while V ∗ β,t typically requires exponential time.
Information about the value function distribution can be a useful tool for constructing near-optimal policies, as well a way to compute risk-sensitive policies.
1.2 Related Work and Our Contribution
We arrange related work in the following areas.
Firstly, “model-free” methods that do not explicitly try to take into account the uncertainty about the underlying MDP.
Secondly, “model-based” methods, that explicitly maintain a distribution on MDP models.
Finally, we discuss approximations to the Bayes-optimal solution, before we outline our contributions.
Model-free Bayesian Value Functions.
Bayesian value function distributions have been considered extensively in model-free Bayesian Reinforcement Learning (BRL).
One of the ﬁrst methods was Bayesian Q-learning [Dearden et al., 1998], which used a normal-gamma prior on the utility distribution.
However, as i.i.d.
utility samples cannot be obtained by bootstrapping from value function estimates, this idea had inherent ﬂaws.
Engel et al.
[2003] developed a more sophisticated approach, the Gaussian Process Temporal Difference (GPTD) algorithm, which has a Gaussian process (GP) prior β ( V ) on value functions.
It then combines this with the likelihood function
P ( D | V ) ∝ Q ti =1 exp {−| V ( s i ) − r
i − γV
( s i +1 ) | 2 } .
However, this makes the implicit assumption that the deterministic empirical MDP model is correct.
Engel et al.
[2005] tried to relax this assumption by allowing for correlation between sequentially visited states.
Deisenroth et al.
[2009] developed a dynamic programming algorithm with a GP prior on value functions and an explicit GP model of the MDP.
Finally, Tang and Agrawal [2018] introduced VDQN, generalising such methods to Bayesian neural networks.
The assumptions that these model-free Bayesian methods implicitly make about the MDP are hard to interpret, and we ﬁnd the use of an MDP model independently of the value function distribution unsatisfactory.
We argue that explicitly reasoning about the joint value function and MDP distribution is necessary to obtain a coherent Bayesian procedure.
Unlike the above methods, we calculate a value function posterior P ( V | D ) while simultaneously taking into account uncertainty about the MDP.
Model-based Bayesian Value Functions.
If a posterior over MDPs is available, we can calculate a distribution over value functions in two steps: a) sample from the MDP posterior and b) calculate the value function of each MDP.
Essentially, this amounts to performing posterior sampling (PSRL,
Strens [2000], Thompson [1933]) followed by policy evaluation.
Dearden et al.
[1999] suggested an early version of this approach that obtained approximate upper bounds on the Bayesian value function and sketched a Bellman-style update for performing it online.
Posterior sampling approach was later used to obtain value function distributions in the discrete case by Dimitrakakis [2011] and in the continuous case by Osband et al.
Finally, O’Donoghue et al.
[2018] derive bounds on the variance of the value function posterior.
We instead focus on whether it is possible to compute complete value function distributions exactly or approximately through a backwards induction procedure.
In particular, how can we obtain P ( V i | D ) from P ( V i +1 | D ) ?
Our Contribution.
We introduce Inferential Induction , a new Bayesian Reinforcement Learning (BRL) framework, which leads to a Bayesian form of backwards induction.
Our framework allows
Bayesian inference over value functions without any implicit assumption or approximation unlike its predecessors.
The main idea is to calculate the conditional value function distribution at step i from the value function distribution at step i + 1 analogouslto backwards induction for the expectation (Eq.
Following this, we propose a simple marginalisation techniques and design an appropriate
Monte Carlo approximation for it.
We can combine this procedure with a policy optimisation mechanism.
We use a Bayesian adaptation of dynamic programming for this and propose the Bayesian backwards induction (BBI) algorithm.
Our experimental evaluation shows that BBI is competitive to the current state of the art.
Inferential Induction framework provides the opportunity to further design more efﬁcient algorithms of this family.
Inferential Induction
Given a prior belief β , and data D t = ( s 1 , a 1 , r 1 , . . .
, s t ) obtained by interaction of the agent with the MDP, we wish to calculate the value function distribution from step t onwards, i.e. P π β ( V t | D t ) for a policy π under the belief β , conditioned on the data D t .
The main idea for doing this to inductively calculate P πβ ( V i | D t ) from P πβ ( V i +1 | D t ) for i ∈ { t, . .
, T } as follows:
[Equation]
Let ψ i +1 be a (possibly approximate) representation of P πβ ( V i +1 | D t ) .
If we can calculate the above integral, then we can also obtain ψ i ≈ P πβ ( V i | D t ) recursively, from time T up to the current time step t .
Then the estimation problem reduces to deﬁning the term P πβ
( V i | V
i +1 , D t ) appropriately.
In this paper, we describe a simple Monte Carlo method for solving the estimation problem and an approximate dynamic programming algorithm for optimising the policy within this framework.
Integrating over P πβ ( µ | V i +1 , D t ) .
A simple idea for dealing with the term linking the two value functions is to directly marginalise over the MDP as follows:
[Equation]
This equality holds because given µ , V i is uniquely determined by the policy π and V i +1 through the
Bellman operator.
However, it is crucial to note that P πβ ( µ | V i +1 , D t ) ̸
= P β ( µ | D t ) , as knowing the value function gives information about the MDP.[-
-]
This is a crucial difference with “model-free”
Bayesian value function methods, which effectively hide strong assumptions about the MDP when they perform value function inference.
We expect that this approach would give superior results.
The remaining computations are rather straightforward Monte Carlo approximations.
The backwards induction step.
To calculate the previous-step distribution from next-step distribution, let us now combine the induction step in (2) with the marginalisation in (3).
We also substitute an approximate representation ψ i +1 for the next-step belief P
( V i +1 | D t ) , to obtain the following conditional probability measure on value functions:
[Equation]
Monte-Carlo approximation of the inner integral.
We now have to leave generalities and commit to some hard choices for deﬁning and calculating P πβ ( µ |
V
i +1 , D t ) .
Expanding this term we obtain, for subsets of MDPs A ⊆ M , the following measure:
[Equation]
since P πµ ( V i +1 | D t ) =
P πµ ( V i +1 ) , as µ, π are sufﬁcient for calculating V
i +1 .
To compute P πµ ( V i +1 ) , we can sample rollouts from an arbitrary starting state distribution q and the policy π for each sampled MDP µ and then marginalise over the resulting utility values U .
In exact form, this is done through the integral:
[Equation]
In order to understand the meaning of the term P πµ ( V i +1 | U, s ) , note that
V
πi +1 ,µ ( s ) =
E πµ [ U | s i +
= s ] .
Thus, a rollout u from state s gives us partial information about the value function.
Finally, the starting state distribution q is used to measure the goodness-of-ﬁt, similarly to e.g. ﬁtted Q-iteration 6 .
As a design choice, we deﬁne the density of V i +1 given a rollout sample u m in MDP µ from state s m ∼ q to be a Gaussian with variance σ 2 .
If we generate N µ number of MDPs µ ( j ) ∼ β ( µ | D t ) and set:
[Equation]
we get E [ w jk ] =
P πβ ( µ ∈ M |
V
i +1 , D t ) .
The weight w jk can be interpreted as a measure of how well the value function V ( k ) i +1 matches the rollouts obtained from µ ( j ) .
This allows us to obtain value function samples for step i ,
[Equation]
[Equation]
each weighted by w jk , leading to the following Monte Carlo estimate of the value function distribution at step i ψ i ( B ) = N [Equation] N µ
N V X k
[Equation]
N
[Equation]
X j =  n
V ( j [Equation] k ) i ∈ B o w jk . Here, N V is the number value function samples V ( k ) i +1 .
This ends the general description of the Monte
Carlo method.
Detailed design of an algorithm depends on the representation that we use for ψ i and whether the MDP is discrete or continuous.
Section 3 gives algorithmic details, while speciﬁcations of hyperparameters and distributions are given in Section 4.
Algorithms
Algorithm 1 is a concise description of the Monte Carlo procedure that we develop.
At each time step t , the algorithm is called with the prior and data D collected so far, and it looks ahead up to some lookahead factor H .
7 We instantiate it below for discrete and continuous state spaces.
Algorithm
Discrete MDPs.
When the MDPs are discrete, the algorithm is straightforward.
Then the belief β ( µ | D ) admits a conjugate prior in the form of a Dirichlet-product for the transitions.
In that case, it is also possible to use a histogram representation for ψ i , so that it can be calculated by simply adding weights to bins according to (7).
However, as a histogram representation is not convenient for a large number of states, we model using a Gaussian ψ t .
In order to do this, we use the sample mean and covariance of the weighted value function samples V ( j,k ) i : [Equation] i
[Equation]
such that ψ i = N
( m i , Σ i ) is a multivariate normal distribution.
Continuous MDPs.
In the continuous state case, we obtain ψ through ﬁtted Q-iteration [c.f. Ernst et al., 2005].
For each action a in a ﬁnite set, we ﬁt a weighted linear model Q i ( s i , a ) =
s Ti ω a + ϵ a , where s i , ω a ∈ R d and ϵ a ∼ N (0 , σ 2 a ) .
Thus, we obtain the sample mean and covariance to be: m a i =
[Equation]
In practice, we often use a feature map ϕ : R d → R f for states.
3.1 Bayesian Backwards Induction
We now construct a policy optimisation component to use with the inferential induction based policy evaluation and the aforementioned two approximation techniques.
We use a dynamic programming algorithm that looks ahead H steps, and at each step i calculates a policy maximising the Bayesian expected utility in the next i + 1 steps.
We describe the corresponding pseudocode in Algorithm 2.
Algorithm 2 Line 8: Discrete MDPs.
Just as in standard backwards induction, at each step, we can calculate π i by keeping π i +1 , . .
, π H ﬁxed:
[Equation]
Algorithm 2 Line 8: Continuous MDPs.
As we are using ﬁtted Q-iteration, we can directly use the state-action value estimates.
So we simply set Q i ( s, a )
=
ˆ Q i ( s, a ) .
The Q i estimate is then used to select actions for every state.
We set π i ( a | s )
= 1 for a = arg max Q i
( s, a ) (Line 2.9) and calculate the value function distribution (Lines 2.10 and 2.11) for the partial policy ( π i , π i +1 , . . .
, π H ) .
1: Input: Prior β , data D , lookahead H , discount γ , N µ , N V .
2: Initialise ψ i .
Sample ˆ M ≜ (cid:8) µ ( j ) (cid:12)(cid:12) j ∈ [ N µ ] (cid:9) from β ( µ | D ) .
4: for i = H − 1 , , 1 do 5: Sample V ( k ) ∼ ψ i +1 ( v ) for k ∈ [ N V ] .
Generate n utility samples u i 7:
Calculate w jk from (5).
Calculate Q i from (10) or ﬁtted Q-iteration.
Set π i ( a | s ) = 1 for a ∈ arg max Q i ( s, a ) .
Calculate w jk from (5) and V ( j,k ) i from (6) with policy
π i : V ( j,k ) i ≜ B π i µ ( j ) V ( k ) i +1 .
Calculate ψ i from (8) or (9).
12: end for 13: return π = ( π 1 , . . .
, π H ) .
Experimental Analysis
For performance evaluation, we compare Bayesian Backwards Induction (BBI, Alg. 2) with exploration by distributional reinforcement learning [VDQN, Tang and Agrawal, 2018].
We also compare BBI with posterior sampling [PSRL, Strens, 2000, Thompson, 1933] for the discrete MDPs and with Gaussian process temporal difference [GPTD, Engel et al., 2003] for the continuous MDPs.
First, we describe the experimental setup and the priors used for implementation.
In Section 4.1, we analyse the results obtained for different environments in terms of average reward obtained over time.
Comparisons with Multi-MDP Backwards Induction [Dimitrakakis, 2011, MMBI], Bayesian
Sparse Sampling [Wang et al., 2005, BSS] and Bayesian Q-Learning [Dearden et al., 1998, BQL] are available in the appendix.
Parameters.
We run the algorithms for the inﬁnite-horizon formulation of value function with discount factor γ = 0 .
Each algorithm updates its policy at steps t k
= k ( k +1) , where k increases as 1 , 2 , . . . .
This sequence of t k ’s causes a total of 1413 updates in 10 6 steps.
We set the lookahead
H to 100 and 20 for discrete and continuous MDPs respectively.
More implementation details can be found in the appendix.
Prior.
For discrete MDPs, we use Dirichlet Dir ( α ) priors over each of the transition probabilities P ( s ′ | s, a ) .
The prior parameter α for each transition is set to 0 .
We use separate NormalGamma
1010101010 6 Time steps 1.
A v e r a g e r e w a r d
BBI PSRL VDQN (a) NChain 1010101010 6 Time steps 0.
(b) DoubleLoop 1010101010 6 Time steps31(c) LavaLake 10 ×1010101010 7 Time steps 0.00 0.
(d) Maze 10101010 5 Time steps
A v e r a g e s t
e
p
s s u r v
i v e d
BBI GPTD VDQN (e [Figure] Pendu
[Figure]
lum
NG ( µ, κ, α, β ) priors for each of the reward distributions P ( r | s, a ) . We set the prior parameters to [ µ 0 , κ 0 , α 0 , β 0 ] = For continuous MDPs, we use factored Bayesian Multivariate Regression [Minka, 2001] models as priors over transition kernels and reward functions for the continuous environments.
This implies that the transition kernel P ( s ′ | s, a ) and reward kernel P ( r | s, a ) modelled as N ( A Trans a s, Σ ) and N ( A Reward a s, σ 2 ) .
Σ is sampled from inverse Wishart distribution with corresponding d × d dimensional scale matrix, while σ is sampled from inverse Gamma with prior parameters ( 12 , 12 ) .
For transitions, we set the prior parameters to Ψ 0 = 0 .
001 I and degrees of freedom ν 0 = rank ( Ψ 0 ) .
Feature Map.
For InvertedPendulum we use the 10 dimensional feature map ϕ : R d → R f used by Lagoudakis and Parr [2003] for BBI and GPTD such that the modelling becomes P ( s ′ | ϕ ( s ) , a ) and P ( r | ϕ ( s ) , a ) .
VDQN works directly on the underlying state s as it uses a neural network.
We also add a regularizing term, λ I , λ = 0 .
01 to the diagonal of the ﬁtted Q-iteration ﬁt.
The choice of state distribution q ( s ) is of utmost importance in continuous environments.
In the continuous environment, we experimented with a few options, trading of sampling states from our history, sampling from the starting conﬁguration of the environment and sampling from the full support of the state space.
Description of Environments We evaluate the algorithms on four discrete (NChain [Strens, 2000],
DoubleLoop [Strens, 2000], LavaLake [Leike et al., 2017] and Maze [Strens, 2000]) and one continuous environment (InvertedPendulum Lagoudakis and Parr [2003]).
See the appendix for more details.
4.1 Experimental Results
The following experiments are intended to show that the general methodological idea is indeed sound, and can potentially lead to high performance algorithms.
More results, comparative experiments, as well as tests of convergence, are provided in the appendix.
Figures 1a, 1b, 1c, 1d illustrate the evolution of average reward for BBI, PSRL and VDQN on the discrete MDPs.
8 BBI is competitive with PSRL, which has good exploration properties, while
VDQN generally performs worse.
Figure 1e shows a comparison with state-of-the art algorithms, such as VDQN and GPTD.
Our algorithm is competitive, and in particular performs much better than GPTD, while it performs similarly to VDQN, which is slightly worse initially and slightly better later in terms of average steps survived.
This performance could partially be explained by the use of a linear value function Q ( ϕ ( s ) , a ) , in contrast to VDQN which uses a neural network.
4.2 Comparison of Computation Times
In Table 1 we can see the timing results for the different algorithms when run on Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz.
Here we can see that PSRL is faster and scales better due to the simplicity of using a single MDP.
It is probable that BBI could obtain speedups through improve sampling [Table] methods, alternative kernels, or a different marginalisation.
We also compared it with MMBI, which samples of multiple MDP models from the posterior and performs approximate dynamic programming to obtain a policy, with the main difference being that MDPs are sampled only once, instead of at every backwards induction step.
Discussion and Future Work
New Insights.
We offered a new perspective on Bayesian value function estimation.
The central idea is to calculate the conditional value function distribution P πβ ( V i | V
i +1 , D ) using the data and to apply it inductively for computing the marginal value function distribution P πβ ( V i | D ) .
We then designed a straightforward Monte Carlo approximation and combined it with a suitable policy optimisation mechanism and showed that it can be competitive with the state of the art.
In order to place it in context, standard backwards induction (i.e. value iteration) calculates V i
= B πµ V
i +1 and distributional reinforcement learning methods calculate P πµ ( U i
| U
i +1 ) , both for a given, underlying MDP.
In an RL setting, this can be replaced either through stochastic approximation, such as Q-learning, or through an explicit empirical model.
Inferential Induction differs from existing Bayesian value function methods, which essentially cast the problem into regression.
For example, GPTD [Engel et al., 2003] can be written as Bayesian inference with a GP prior over value functions and a data likelihood that uses a deterministic empirical model of the MDP.
While this can be relaxed by using temporal correlations as in [Engel et al., 2005], the fundamental problem remains.
Even though such methods have practical value, we show that Bayesian estimation of value functions requires us to explicitly think about the MDP distribution as well.
Algorithm Design.
We use speciﬁc approximations for discrete and continuous MDPs to propose the Bayesian Backwards Induction (BBI) algorithm.
The algorithm we developed from this family appears promising, as we are able to outperform methods that implicitly use the empirical model of the MDP, such as GPTD.
This shows that the inference is inherently sound.
We see that BBI is also competitive with state-of-the art methods like PSRL, and it generally outperforms algorithms relying on approximate inference, such as VDQN.
Limitations.
We have observed that the particular method, BBI, that we have developed requires a signiﬁcant number of samples for it to obtain a good performance, which makes it inherently slow.
Our timing analysis in Table 1 shows that methods like PSRL may work signiﬁcantly faster as they have to keep track of only one MDP and are also simpler to implement.
It is probable that BBI could obtain speedups from using sparse posteriors or sampling from state space in the importance sampling step of the algorithm.
For the continuous environments, we have used weighted linear model and linear priors for updating and tracking the value function distribution.
Thus, we rely on pre-deﬁned features (e.g. the features from Lagoudakis and Parr [2003]) which are problem speciﬁc, not always efﬁcient, and can be replaced more efﬁcient functional approximators, such as neural networks (similar to Tang and
Agrawal [2018]).
Since our aim has been to propose the framework, we have emphasised one policy evaluation and used simpler methods, such as Q-update and ﬁtted Q-iteration, for policy update.
It is possible to incorporate this method with policy gradient methods to design more efﬁcient algorithms.
This point shows the ﬂexibility of our approach as a framework and weakness as a speciﬁcally designed algorithm, which is a topic of future investigation.
Acknowledgments and Disclosure of Funding
Thank you to Nikolaos Tziortziotis for his useful discussions.
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and
Alice Wallenberg Foundation. The experiments were partly performed on resources at Chalmers Centre for Computational Science and Engineering (C3SE) provided by the Swedish National Infrastructure for Computing (SNIC). A Description of Environments
NChain.
This is a discrete stochastic MDP with 5 states, 2 actions [Strens, 2000].
Taking the ﬁrst action returns a reward 2 for all states and transitioning to the ﬁrst state.
Taking the second action returns 0 reward in the ﬁrst four states (and the state increases by one) but returns 10 for the ﬁfth state and the state remains unchained.
There is a probability of slipping of 0 . 2 with which its action has the opposite effect.
This environment requires both exploration and planning to be solved effectively and thus acts as an evaluator of posterior estimation, efﬁcient performance and effective exploration.
DoubleLoop.
This is a slightly more complex discrete deterministic MDP with with two loops of states [Strens, 2000].
Taking the ﬁrst action yields traversal of the right loop and a reward 1 for every 5 state traversal.
Taking the second action yields traversal of the left loop and a reward 2 for every 5 state traversal.
This environment acts as an evaluator of efﬁcient performance and effective exploration.
LavaLake.
This is a stochastic grid world [Leike et al., 2017] where every state gives a reward of -1, unless you reach the goal, in which case you get 50, or fall into lava, where you get -50.
We tested on the 5 × 7 and a 10 × 10 versions of the environment.
The agent moves in the direction of the action (up,down,left,right) with probability 0.8 and with probability 0.2 in a direction perpendicular to the action.
Maze.
This is a grid world with four actions (ref.
Fig.
3 in [Strens, 2000]).
The agent must obtain 3 ﬂags and reach a goal.
There are 3 ﬂags throughout the maze and upon reaching the goal state the agent obtains a reward of 1 for each ﬂag it has collected and the environment is reset.
Similar to LavaLake, the agent moves with probability 0.9 in the desired direction and 0.1 in one of the perpendicular directions.
The maze has 33 reachable locations and 8 combination of obtained ﬂags for a total of 264 states.
InvertedPendulum.
To extend our results for the continuous domain we evaluated our algorithm in a classical environment described in [Lagoudakis and Parr, 2003].
The goal of the environment is to stabilize a pendulum and to keep it from falling.
If the pendulum angle θ falls outside [ − π 2 , π 2 ] then the episode is terminated and the pendulum returned to its starting conﬁguration.
The state dimensionality is a tuple of the pendulum angle as well as its angular velocity, ˙ θ , s = ( θ, ˙ θ ) .
The environment is considered to be completed when the pendulum has been kept within the accepted range for 3000 steps.
For further environment details, see [Lagoudakis and Parr, 2003].
B Additional Results
Here we present some experiments that examine the performance of inferential induction in terms of value function estimation, inference and utility obtained.
For the latter we present additional results against other algorithms, as well as one new continuous environment.
B.1 Bayesian Value Function Estimation
In this experiment, we evaluate the Bayesian (i.e. mean) value function of the proposed algorithm (BBI) with respect to the upper bound on the Bayes-optimal value function.
The upper bound is calculated from R M max π V πµ dβ ( µ | D ) .
We estimate this bound through 100 MDP samples for
NChain.
We plot the time evolution of our value function and the simulated Bayes bound in Figurefor 10 5 steps.
We observe that this is becomes closer to the upper bound as we obtain more data.
B.2 Value Function Distribution Estimation
Here we evaluate whether inferential induction based policy evaluation (Alg. 1) results in a good approximation of the actual value function posterior.
In order to evaluate the effectiveness of estimating the value function distribution using inferential induction (Alg. 1), we compare it with the Monte Carlo distribution and the mean MDP.
We compare this for posteriors after 10, 100 andtime steps, obtained with a ﬁxed policy in NChain that visits all the states, in Figure 3 for 5 runs of Alg. 1.
The ﬁxed policy selects the ﬁrst action with probability 0.8 and the second action with probability 0.2.
The Monte Carlo estimate is done through 1000 samples of the value function vector ( γ = 0 . 99 ).
This shows that the estimate of Alg.
1 reasonably captures the uncertainty in the true
[Figure]
distribution.
For this data, we also compute the Wasserstein distance [Fournier and Guillin, 2015] between the true and the estimated distributions at the different time steps as can be found in Table 2.
There we can see that the distance to the true distribution decreases over time
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
B.3 Further Comparisons on Discrete Environments: MMBI, BSS, BQL
We have also run additional experiments with other Bayesian algorithms.
In particular, here we show comparisons with MMBI [Dimitrakakis, 2011], BSS [Wang et al., 2005] and BQL Dearden et al.
As can be seen in Figures 4 to 6, BBI performs similarly to to MMBI and PSRL.
This is to be expected, as the optimisation algorithm used in MMBI is close in spirit to BBI, with only the inference being different.
In particular, this algorithm takes k MDP samples from the posterior, and [Figure] then performs backward induction in all the MDP simultaneously to obtain a Markov policy. In turn
[Figure] PSRL can be seen as a special case of MMBI with just one sample.
This indicates that the BBI inference procedure is sound.
The near-optimal Bayesian approximation performs slightly worse in this setting, perhaps because it was not feasible to increase the planning horizon sufﬁciently.
Finally, the less principled approximations, like VDQN and BQL do not manage to have a satisfactory performance in these environments [Figure]
30 runs respectively with a length of 10 6 for each algorithm.
The runs are exponentially smoothened with a half-life 1000 before averaging.
B.4 Analysis of Variation in Performance
In Figures 7 to 11, we illustrate the variation in performance of the different algorithms for each environment.
The black lines illustrate the standard error and the 5th and 95th percentile performance is highlighted.
The results indicate that BSS is the most stable algorithm, followed by BBI, MMBI and PSRL, which nevertheless have better mean performance.
VDQN is quite unstable, however.
C Implementation Details
In this section we discuss some additional implementation details, in particular how exactly we performed the rollouts and the selection of some algorithm hyperparameters, as well as some sensitivity analysis.
C.1 Computational Details of Rollouts
To speed up the computation of rollouts, we have used three possible methods that essentially bootstrap previous rollouts or use value function samples:
[Equation]
where V t +1 ∼ ψ t +1 .
In experiments, we have found no signiﬁcant difference between them.
All results in the paper use the formulation in (13).
C.2 Hyperparameters
For the experiments, we use the following hyperparameters.
We use 10 MDP samples, a planning horizon T of 100, γ = 0 .
and we set the variance of the
Gaussian to be σ 2 =
V 2 span 10 − 4 , where V span is the span of possible values for each environment (obtained assuming maximum and minimum reward).
We use Eq.
13 for rollout computation with 10 samples from V t +1 and 50 samples from V t (20 for LavaLake 10 × 10 and Maze).
If the weights obtained in (5) are numerically unstable we attempt to resample the value functions and then double σ until it works (but is reset to original value when new data is obtained).
This is usually only a problem when very little data has been obtained.
Varying Horizon T . In order to check the sensitivity on the choice of horizon T , we perform a sensitivity analysis with T = 10 , 20 , 50 , 100 .
In Figure 12, we can see that varying the horizon has a very small impact for NChain and Maze environments
[Figure]
[Figure]
[Figure] [Figure]
[Figure]
A v e r a g e r e w a r d
[Figure] [Figure]
v
[Figure]
r a g e r
[Figure]
a r d
[Figure]
[Figure] [Figure]