We investigate the use of temporally abstract actions, or macro-actions, in the solution of Markov decision processes. Unlike current mod els that combine both primitive actions and macro-actions and leave the state space un changed, we propose a hierarchical model (using an abstract MDP) that works with macro-actions only, and that signifi cantly reduces the size of the state space. This is achieved by treating macro actions as local policies that act in certain regions of state space, and by restricting states in the ab stract MDP to those at the boundaries of regions. The abstract MDP approximates the original and can be solved more efficiently. We discuss sev eral ways in which macro-actions can be gen erated to ensure good solution quality. Finally, we consider ways in which macro-actions can be reused to solve multiple, related MDPs; and we show that this can justify the computational over head of macro-action generation.
In REF a similar result is achieved with hierarchical MDPs which operate over subsets of the state space.
Hierarchical Solution of Markov Decision Processes using Macro-actions
ArXiv
ArXiv
Computer Science