Non-stationary Causal Bandits
Abstract
The causal bandit problem is an extension of the conventional multi-armed bandit problem in which the arms available are not independent of each other, but rather are correlated within themselves in a Bayesian graph.
This extension is more natural, since day-to-day cases of bandits often have a causal relation between their actions and hence are better represented as a causal bandit problem.
Moreover, the class of conventional multi-armed bandits lies within that of causal bandits, since any instance of the former can be modeled in the latter setting by using a
Bayesian graph with all independent variables.
However, it is generally assumed that the probabilistic distributions in the Bayesian graph are stationary.
In this paper, we design non-stationary causal bandit algorithms by equipping the actual state of the art (mainly causal UCB , causal Thompson Sampling , causal KL UCB and Online Causal TS ) with the restarted Bayesian online change-point detector [2].
Experimental results show the minimization of the regret when using optimal change-point detection.
Introduction
The multi-armed bandit is a stochastic scheduling problem wherein a user can choose amongst an independent set of arms (or actions) and the reward that they will get is stochastically dependent on the arm pulled.
This problem covers many real life instances (for eg.
which advertisement will have a higher click rate) and derives its name from the one-arm slot machines in casinos.
Albeit its simple nature and relaxing assumptions, the multi-armed bandit can model a lot of problems making it widely prevalent.
However, there are several scenarios in which multi-armed bandits cannot be applied due to contradiction(s) with one or more of its fundamental assumptions.
A very common example, especially in the context of our country, is related to the cropping season.
Consider a young farmer who just started farming this season, naturally with the objective of procuring maximum yield.
He is aware of various factors affecting the same, say quality of seeds, organic manure, pesticides and fertilizers, but can afford to spend on only one due to a fixed budget; other factors are out of his reach and obtained (randomly) via government schemes.
Only at the end of the season can he estimate the quantity of each ’out-of-reach’ resource consumed and the resulting yield, and plan his approach/expenditure for the next season.
Another thing to be noted here is that the decisions of the farmer on spending his budget on a resource are not independent - if the farmer is already getting good quality of seeds (which are, say, robust to diseases) and decides not to spend on them, it is intuitive that he won’t have to spend on anti-disease fungicides/pesticides either.
Clearly, a multi-armed bandit is not the best formulation for this problem since the actions are not independent.
Instead, such instances are better modeled by causal bandits in which we assume that a Bayesian graph governs the dependencies amongst various inputs and other hidden factors, with the final ’reward’ also being a node in the graph.
Various algorithms have already been proposed for causal bandits that try to model the Bayesian graph structure and distribution, and consequently find the best regret-minimizing action [4].
However, the algorithms like causal UCB [6], causal Thompson Sampling [6] and causal KL UCB [5] assume that the distributions carried out over the graph are stationary.
Our objective in this paper is to design new algorithms to handle non-stationary environments.
We propose to equip the algorithms named causal UCB , causal Thompson Sampling , causal KL UCB and online causal Thompson Sampling with the restarted Bayesian online change-point detector in order to handle abrupt changes of the probabilistic distributions.
By doing that, the resulting strategies named Restarted causal UCB , restarted causal Thompson Sampling , restarted causal KL UCB and restarted online causal Thompson Sampling are able to adapt to the changes of the environment.
Problem Statement
A causal bandit instance can be described as follows.
We are given a set of random variables
X =
{ X 1 , X 2 . . .
X N } , a reward variable Y and a set of allowed actions A .
The dependencies between X and Y are represented using a causal graph G , and the amount of information we have beforehand on G can be varied.
Each action a t ∈
A is an intervention of the form do ( X t = x t ) (where X t = { X j 1 , X j 2 . . .
X j m } , X j i ∈ X∀ i ∈ [ m ] ) that involves assigning X j i the corresponding value in x t and removing all incoming edges for X j i in G to obtain a mutilated graph G a t (the empty intervention, do () , is also permitted).
After the intervention has been performed, values of the non-intervened variables X c = X\ X t and the reward y t are sampled from the distribution of G a t .
This process is repeated up to a fixed horizon T , and the objective is to choose { a t } Tt =1 such that the cumulative (or simple) regret is minimised.
For the purpose of this paper, we restrict ourselves to those causal bandit instances where the structure of G is known beforehand but the joint probability distribution (i.e. P ( X ∪ { Y } ) is unknown.
Moreover, each X ∈ X and Y is a { 0 , 1 } -valued random variable, and the set of allowed actions A is the set of all possible size-1 interventions (i.e. | X t | = | x t | = 1 , and hence |A| = 2 N ).
We shall minimise cumulative regret in our algorithms, which is defined as R T =
(cid:80) Tt =1 µ ⋆t − E (cid:104)(cid:80)
Tt =1 y t (cid:105) , where µ ⋆t =
E [ Y = 1 | a ⋆t ] and a ⋆t is the optimal action at time t .
Piece-wise stationary processes
We assume that the probabilistic distributions that are carried out over the Bayesian graph follow a piece-wise stationary process Indeed, we assume a global switching model that allows synchronous changes to happen, i.e. arm switches occur at the same time.
We denote the sequence of break-points up to the horizon T by: ( τ 1 = 1 , τ 2 , ..., τ K T +1 =
T + 1) .
Related Work
The causal bandit problem was first formulated in [4], where they assumed the causal graph G to be completely known beforehand.
They suggested algorithms to minimise simple regret, which is defined as R = µ ∗ − max a ∈A µ a where µ a = E
[ Y | a ] , in two different scenarios - (1) when G was parallel i.e. all non-reward variables were independent of each other, and (2) when G was arbitrary. [4] used the graph structure by looking at parents of Y and optimising a distribution over A , with the distribution itself changing to minimise the time taken to reach the best action. [7] extend Thompson Sampling to the causal bandit setting of [4] by breaking the procedure into two parts - given an intervention
a
= do ( X = x ) , they first try to estimate a distribution over the possible assignments of parent variables of Y given a , and then they estimate the reward distribution given a particular parent configuration; both distributions are updated after each trial as new samples are observed.
For many real-life causal bandit instances, it makes sense to minimise cumulative regret for a fixed horizon which is not explored in detail in these papers.
A comprehensive analysis and comparison of the proposed algorithms with classical multi-armed bandit algorithms is also lacking.
Moreover, in most practical scenarios, the structure of the causal graph is known beforehand - we know how variables affect each other as well as the reward it is the exact probability distribution which is unknown.
We believe that these algorithms can be modified to exploit this additional information, which is one of the few things we plan to explore in this paper.
Agents Implemented - Restarted Causal bandits 4.1 The restarted Bayesian online change-point detector
The authors in [2] have designed a variant of the original Bayesian online change-point detector introduced by [1].
The resulting strategy is named restarted Bayesian online change-point detector RBOCPD .
It is a pruning version of the original algorithm reinterpreted from the standpoint of forecasters aggregation and expressed as a restart procedure pruning the useless forecasters.
More formally, for a binary sequence (cid:0) y r , ..., y n (cid:1) ∈ { 0 , 1 } , the final formulation of the RBOCPD strategy takes the following form:
[Equation]
where the weight of the forecasters ϑ r,s,t are computed in a recursive way as follows (assuming an initial weight ϑ r, 1 , 1 = 1 ):
[Equation]
such that the initial weight of the forecaster takes the form of V r : t := exp (cid:16) − (cid:80) t − 1 s ′ =
r l s ′ ,t − 1 (cid:17) and the instantaneous loss l s,t := − log Lp
( y t | y s ...
y t − 1 ) is computed based on the Laplace predictor
Lp ( y t | y s ... y t − 1 ) : =
[Equation]
1 − y i ) +1 t − s + if y t = 0 .
The hyper-parameter η r,s,t is tuned as a decreasing function in t and depends also on the probability of false alarm δ .
The RBOCPD algorithm is chosen among all the sequential change-point detector algorithms in the state of the art for three main reasons.
• Well adaptability to unknown priors.
Indeed, the RBOCPD algorithm has been designed to solve the problem of sequential change-point detection in a setting where both the changepoints and the distributions before and after the change are assumed to be unknown.
This setting corresponds exactly to the situation of an agent facing a multi armed bandit whose distributions are unknown and may change abruptly at some unknown instants.
• Minimum detection delay.
This corresponds to the first criteria assessing the performance of a sequential change-point detector.
The detection delay is defined as the number of samples needed to detect a change.
In [2], the authors have shown that the detection delay of the RBOCPD strategy is asymptotically optimal in the sense that it reaches the existing lower bound stated in Theorem 3.1 in [3].
• Well controlled false alarm rate.
The false alarm rate corresponds to the probability of detecting a change at some instant where there is no change.
Again, in [2], the authors have demonstrated that ∀ δ ∈ (0 , 1) RBOCPD doesn’t make any false alarm with a probability at least 1 − δ .
4.2 Restarted causal Bandit algorithms
In order to resolve a piece-wise stationary causal multi-armed bandit, we propose to equip the classical algorithms ( causal UCB , causal Thompson Sampling and causal KL UCB ) with the restarted Bayesian online change-point detector running on each arm a ∈ A .
The resulting strategies are called Restarted causal UCB , restarted causal Thompson Sampling , restarted causal KL UCB and restarted causal OC-TS .
At some time t , the designed strategies reinitialize the parameters related to arm a when the RBOCPD associated to arm a has raised an alarm.
Restarted Bandits
Restarted Bayesian
[Figure]
(UCB, KL-UCB, Thompson sampling,
OC-TS)
[Figure]
Experiments 5.1 Graph topologies
We have considered three types of Bayesian graph models (described below) with different topologies representing the causal dependencies amongst Bernoulli variables to show that our proposed modifications/algorithms perform better than existing ones.
Figure 1 shows the representative model from each topology chosen for our experiments.
- Linear.
Every variable excluding one root variable has exactly one parent.
The idea here is that the agent must learn that the best action is the one in which the closest possible ancestor is intervened (to either a value of 0 or 1 depending on the distribution).
- Disjoint or Independent.
Here all variables (except the reward variable) are pairwise independent and are parents of the reward variable.
The best action in this case is the one in which the marginalised probability over non-intervened variables is maximum. - Random.
It refers to a completely random Bayesian graph obtained by iteratively adding variables and randomly choosing whether each previously added variable is its parent or not.
5.2 Experimental results
We present the results of our experiments here.
For each topology, we compare the performance of three designed agents ( Restarted causal UCB , restarted causal Thompson Sampling , restarted causal KL UCB and restarted causal OC-TS ) by plotting the regret (cid:80) Tt
=1 µ ⋆t −
E (cid:104)(cid:80) Tt =1 y t (cid:105) as a function of horizon.
These plots are shown in figures 3, 4 and 5.
Each plot is obtained by averaging results over 20 random seeds; we also shade the regions of uncertainty corresponding to a deviation of ± σ .
[Figure]
C u m u l a ti v e r e g r e t
Restarted causal UCB
Restarted causal KL-UCB
Restarted causal TS
Restarted causal OC-TS
Change-point
[Figure]
l a ti v e r e g r e t
Restarted causal UCB
Restarted causal KL-UCB
Restarted causal TS
Restarted causal OC-TS
Change-point
[Figure] [Figure]
Horizon
C u m u l a ti v e r e g r e t
Restarted causal UCB
Restarted causal KL-UCB
Restarted causal TS
Restarted causal OC-TS
Change-point
From the figures 3, 4 and 5, the designed algorithms ( Restarted causal UCB , restarted causal Thompson Sampling , restarted causal KL UCB and restarted causal OC-TS ) are able to adapt to the changes of the environment.
Indeed, the restarted Bayesian online changepoint detector is able to quickly detect the change-point thanks to it optimal detection delay and false alarm rate [2].
Conclusion
In this paper, we have designed non-stationary causal bandit algorithms for piece-wise stationary environments.
The designed algorithms are resulting from a combination between the causal bandit algorithm in the state of the art with the restarted Bayesian online change-point detector.
From the experiments, we show that the designed algorithms are able to quickly adapt to the change of the environment.
As future work, we are planning to mathematically analyze the designed strategies in order to compute the regret upper bound.