  We model online recommendation systems using the hidden Markov multi-state
restless multi-armed bandit problem. To solve this we present Monte Carlo
rollout policy. We illustrate numerically that Monte Carlo rollout policy
performs better than myopic policy for arbitrary transition dynamics with no
specific structure. But, when some structure is imposed on the transition
dynamics, myopic policy performs better than Monte Carlo rollout policy.
