Abstract-We investigate the dual approach to dynamic programming and reinforcement learning, based on maintaining an explicit representation of stationary distributions as opposed to value functions. A significant advantage of the dual approach is that it allows one to exploit well developed techniques for representing, approximating and estimating probability distributions, without running the risks associated with divergent value function estimation. A second advantage is that some distinct algorithms for the average reward and discounted reward case in the primal become unified under the dual. In this paper, we present a modified dual of the standard linear program that guarantees a globally normalized state visit distribution is obtained. With this reformulation, we then derive novel dual forms of dynamic programming, including policy evaluation, policy iteration and value iteration. Moreover, we derive dual formulations of temporal difference learning to obtain new forms of Sarsa and Q-learning. Finally, we scale these techniques up to large domains by introducing approximation, and develop new approximate off-policy learning algorithms that avoid the divergence problems associated with the primal approach. We show that the dual view yields a viable alternative to standard value function based techniques and opens new avenues for solving dynamic programming and reinforcement learning problems. Algorithms for dynamic programming (DP) and reinforcement learning (RL) are usually formulated in terms of value functions-representations of the long run expected value of a state or state-action pair [1] . The concept of value is so pervasive in DP and RL, in fact, that it is hard to imagine that a value function representation is not a necessary component of any solution approach. Yet, linear programming (LP) methods clearly demonstrate that the value function is not a necessary concept for solving DP/RL problems. In LP methods, value functions only correspond to the primal formulation of the problem, and do not appear at all in the dual. Rather, in the dual, value functions are replaced by the notion of state (or state-action) visit distributions [2] , [3], [4]. It is entirely possible to solve DP and RL problems in the dual representation, which offers an equivalent but different approach to solving DP/RL problems without any reference to value functions. Despite the well known LP duality, dual representations have not been widely explored in DP and RL. In fact, they have only been anecdotally and partially treated in the RL literature [5] , [6] , and not in a manner that acknowledges any connection to LP duality. Nevertheless, as we will show, there exists a dual form for every standard DP and RL algorithm,
Contrary to the primal form, any feasible discounted state action visit frequency distribution maps to an actual policy REF .
Dual Representations for Dynamic Programming and Reinforcement Learning
2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning
2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning
Computer Science