This paper develops a novel  framework to analyze the  convergence of Q-learning algorithm by using its connections to dynamical systems. We prove that asynchronous Q-learning with a constant step-size can be naturally formulated as  discrete-time stochastic switched linear systems . Moreover, the evolution of the Q-learning estimation error is over- and underestimated by trajectories of two  dynamical systems. Based on the schemes, a new finite-time analysis of the Q-learning is given with a finite-time error bound. It offers novel intuitive insights on analysis of Q-learning mainly based on control theoretic frameworks. By filling the gap between both domains in a synergistic way, this approach can potentially facilitate further progress in each field .
This paper develops a novel control-theoretic framework to analyze the non-asymptotic convergence of Q-learning . We show that the dynamics of asynchronous Q-learning with a constant step-size can be naturally formulated as a discrete-time stochastic affine switching system . Moreover, the evolution of the Q-learning estimation error is over- and underestimated by trajectories of two simpler dynamical systems. Based on these two systems, we derive a new finite-time error bound of asynchronous Q-learning when a constant stepsize is used. Our analysis also sheds light on the overestimation phenomenon of Q-learning . We further illustrate and validate the analysis through numerical simulations .
A
control-theoretic
A
non-asymptotic
R
algorithm by using its connections to dynamical systems. We prove that
. We show that the dynamics of
A
a
R
switched linear systems
affine switching system
A
simpler
R
the schemes,
these two systems, we derive
R
analysis of the
error bound of asynchronous
R
is given with a finite-time error bound. It offers novel intuitive insights on analysis
when a constant stepsize is used. Our analysis also sheds light on the overestimation phenomenon
R
mainly based on control theoretic frameworks. By filling the gap between both domains in a synergistic way, this approach can potentially facilitate further progress in each field
. We further illustrate and validate the analysis through numerical simulations