Most contextual bandit algorithms minimize regret to the best fixed policy-a questionable benchmark for non-stationary environments ubiquitous in applications. In this work, we obtain efficient contextual bandit algorithms with strong guarantees for alternate notions of regret suited to these non-stationary environments. Two of our algorithms equip existing methods for i.i.d problems with sophisticated statistical tests, dynamically adapting to a change in distribution. The third approach uses a recent technique for combining multiple bandit algorithms, with each copy starting at a different round so as to learn over different data segments. We analyze several notions of regret for these methods, including the first results on dynamic regret for efficient contextual bandit algorithms. assumptions and study appropriate generalizations of regret from i.i.d. to non-stationary environments. We begin by modifying the simplest contextual bandit algorithm, EPOCH-GREEDY [16] , to work in non-stationary settings. 1 We further adapt the statistically and computationally optimal approach of Agarwal et al. [1] to this setting as well. Both of our modifications, ADA-GREEDY and ADA-ILTCB, probe for the longest period in the past over which the data distribution looks approximately i.i.d., and compute a distribution over policies based on that period. The exact test is algorithm-specific and based on verifying certain concentration inequalities which the algorithm relies upon, but the general idea might be applicable to extending other contextual bandit algorithms as well.
Recently, Luo et al REF studied the non-stationary bandit problem and proposed several bandit algorithms with statistical tests to adapt to changes in the environment.
Efficient Contextual Bandits in Non-stationary Worlds
ArXiv
ArXiv
Computer Science Mathematics