  We consider the problem of scheduling maintenance for a collection of
machines under partial observations when the state of each machine deteriorates
stochastically in a Markovian manner. We consider two observational models:
first, the state of each machine is not observable at all, and second, the
state of each machine is observable only if a service-person visits them. The
agent takes a maintenance action, e.g., machine replacement, if he is chosen
for the task. We model both problems as restless multi-armed bandit problem and
propose the Whittle index policy for scheduling the visits. We show that both
models are indexable. For the first model, we derive a closed-form expression
for the Whittle index. For the second model, we propose an efficient algorithm
to compute the Whittle index by exploiting the qualitative properties of the
optimal policy. We present detailed numerical experiments which show that for
multiple instances of the model, the Whittle index policy outperforms myopic
policy and can be close-to-optimal in different setups.
