In this work we present an approach for building tight model-free confidence intervals for the optimal value function V ^\star in general infinite horizon MDPs via the upper solutions. We suggest a novel upper value iterative procedure (UVIP) to construct upper solutions for a given agent's policy. UVIP leads to a model free method of policy evaluation. We analyze convergence properties of the approximate UVIPunder rather corresponding to a policy \pi does not provide reliable information on how far is the policy \pi from the optimal one. We present a novel model-free upper value iteration procedure (}{\sf  (x) - V^{\pi}(x) from above and to construct confidence intervals for V^\star. Our approach relies on upper bounds to the solution of the Bellman optimality equation via martingale approach. We provide theoretical guarantees for }{\sf   general assumptions and illustrate its performance on a number of benchmark RL problems.
Policy evaluation is an important instrument for the comparison of different algorithms in Reinforcement Learning (RL). Yet even a precise knowledge of the value function V ^{\pi corresponding to a policy \pi does not provide reliable information on how far is the policy \pi from the optimal one. We present a novel model-free upper value iteration procedure (}{\sf UVIP ) that allows us to estimate the suboptimality gap V^{\star(x) - V^{\pi}(x) from above and to construct confidence intervals for V^\star. Our approach relies on upper bounds to the solution of the Bellman optimality equation via martingale approach. We provide theoretical guarantees for }{\sf UVIP under general assumptions and illustrate its performance on a number of benchmark RL problems.
R
In this work we present an approach for building tight model-free confidence intervals for the optimal
Policy evaluation is an important instrument for the comparison of different algorithms in Reinforcement Learning (RL). Yet even a precise knowledge of the
R
^\star in general infinite horizon MDPs via the upper solutions. We suggest a novel upper value iterative procedure (UVIP) to construct upper solutions for a given agent's policy. UVIP leads to a model free method of policy evaluation. We analyze convergence properties of the approximate UVIPunder rather
^{\pi
A
UVIP
A
) that allows us to estimate the suboptimality gap V^{\star
A
UVIP
A
under