Abstract-Real-world control problems are often modeled as Markov Decision Processes (MDPs) with discrete action spaces to facilitate the use of the many reinforcement learning algorithms that exist to find solutions for such MDPs. For many of these problems an underlying continuous action space can be assumed. We investigate the performance of the Cacla algorithm, which uses a continuous actor, on two such MDPs: the mountain car and the cart pole. We show that Cacla has clear advantages over discrete algorithms such as Q-Iearning and Sarsa, even though its continuous actions get rounded to actions in the same finite action space that may contain only a small number of actions. In particular, we show that Cacla retains much better performance when the action space is changed by removing some actions after some time of learning.
To deal with discrete actions that might have an underlying continuous representation, REF used policy gradients with continuous actions and selected the nearest discrete action.
Using continuous action spaces to solve discrete problems
2009 International Joint Conference on Neural Networks
2009 International Joint Conference on Neural Networks
Computer Science