Abstract. In the Bayesian approach to sequential decision making, exact calculation of the (subjective) utility is intractable. This extends to most special cases of interest, such as reinforcement learning problems. While utility bounds are known to exist for this problem, so far none of them were particularly tight. In this paper, we show how to efficiently calculate a lower bound, which corresponds to the utility of a near-optimal memoryless policy for the decision problem, which is generally different from both the Bayes-optimal policy and the policy which is optimal for the expected MDP under the current belief. We then show how these can be applied to obtain robust exploration policies in a Bayesian reinforcement learning setting. We consider decision making problems where an agent is acting in a (possibly unknown to it) environment. By choosing actions, the agent changes the state of the environment and in addition obtains scalar rewards. The agent acts so as to maximise the expectation of the utility function: U t T k=t γ k r k , where γ ∈ [0, 1] is a discount factor and where the instantaneous rewards r t ∈ [0, r max ] are drawn from a Markov decision process (MDP) µ, defined on a state space S and an action space A, both equipped with a suitable metric and σ-algebra, with a set of transition probability measures T s,a µ s ∈ S, a ∈ A on S , and a set of reward probability measures R s,a µ s ∈ S, a ∈ A on R, such that: where s t ∈ S and a t ∈ A are the state of the MDP, and the action taken by the agent at time t, respectively. The environment is controlled via a policy π ∈ P. This defines a conditional probability measure on the set of actions, such that P π (a t ∈ A | s t , a t−1 ) = π(A | s t , a t−1 ) is the probability of the action taken at time t being in A, where we use P, with appropriate subscripts, to denote probabilities of events and s t s 1 , . . . , s t and a t−1 a 1 , . . . , a t−1 denotes sequences of states and actions respectively. We use P k to denote the set of k-order Markov policies. Important special cases are the set of blind policies P 0 and the set of memoryless policies P 1 . A policy in π ∈P k ⊂ P k is stationary, when π(A | s t t−k+1 , a t−1 t−k+1 ) = π(A | s k , a k−1 ) for all t.
However, in a recent paper REF we show how to obtain near-optimal memoryless policies for the unknown MDP case, which would be applicable in this setting.
Robust Bayesian reinforcement learning through tight lower bounds
ArXiv
ArXiv
Computer Science Mathematics