This paper develops a novel framework to analyze the convergence of Q-learning algorithm from a discrete-time switching system perspective . We prove that asynchronous Q-learning with a constant step-size can be naturally formulated as discrete-time stochastic switched linear systems. It offers novel and intuitive insights on Q-learning mainly based on control theoretic frameworks. For instance, the proposed analysis explains the overestimation phenomenon in Q-learning due to the maximization bias . Based on the control system theoretic argument and some nice structures of Q-learning , a new finite-time analysis of the Q-learning is given with a novel error bound .
This paper develops a novel framework to analyze the convergence of Q-learning algorithm by using its connections to dynamical systems . We prove that asynchronous Q-learning with a constant step-size can be naturally formulated as discrete-time stochastic switched linear systems. Moreover, the evolution of the Q-learning estimation error is over- and underestimated by trajectories of two dynamical systems . Based on the schemes , a new finite-time analysis of the Q-learning is given with a finite-time error bound. It offers novel intuitive insights on analysis of Q-learning mainly based on control theoretic frameworks. By filling the gap between both domains in a synergistic way, this approach can potentially facilitate further progress in each field .
R
from a discrete-time switching system perspective
by using its connections to dynamical systems
R
It offers novel and intuitive insights on
Moreover, the evolution of the
R
mainly based on control theoretic frameworks. For instance, the proposed analysis explains the overestimation phenomenon in Q-learning due to the maximization bias
estimation error is over- and underestimated by trajectories of two dynamical systems
R
control system theoretic argument and some nice structures of Q-learning
schemes
R
novel error bound
finite-time error bound. It offers novel intuitive insights on analysis of Q-learning mainly based on control theoretic frameworks. By filling the gap between both domains in a synergistic way, this approach can potentially facilitate further progress in each field