  We evaluate the performance of Whittle index policy for restless Markovian
bandits, when the number of bandits grows. It is proven in [30] that this
performance is asymptotically optimal if the bandits are indexable and the
associated deterministic system has a global attractor fixed point. In this
paper we show that, under the same conditions, the convergence rate is
exponential in the number of bandits, unless the fixed point is singular (to be
defined later). Our proof is based on the nature of the deterministic equation
governing the stochastic system: We show that it is a piecewise affine
continuous dynamical system inside the simplex of the empirical measure of the
bandits. Using simulations and numerical solvers, we also investigate the cases
where the conditions for the exponential rate theorem are violated, notably
when attracting limit cycles appear, or when the fixed point is singular. We
illustrate our theorem on a Markovian fading channel model, which has been well
studied in the literature. Finally, we extend our synchronous model results to
the asynchronous model.
