We consider the problem of learning in episodic finite-horizon Markov decision processes with unknown transition function, bandit feedback, and adversarial losses. We propose an efficient algorithm that achievesÕ(L|X| 2 |A|T ) regret with high probability, where L is the horizon, |X| is the number of states, |A| is the number of actions, and T is the number of episodes. To the best of our knowledge, our algorithm is the first one to ensureÕ( √ T ) regret in this challenging setting. Our key technical contribution is to introduce an optimistic loss estimator that is inversely weighted by an upper occupancy bound.
REF further achieves O(L|S| |A|T ) regret under the same setting of the unknown transition model and adversarial losses.
Learning Adversarial MDPs with Bandit Feedback and Unknown Transition
ArXiv
ArXiv
Mathematics Computer Science