B ANDIT L EARNING WITH J OINT E FFECT OF I NCENTIVIZED S AMPLING , D ELAYED S AMPLING F EEDBACK , AND S ELF -R EINFORCING U SER P REFERENCES
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we consider a new multi-armed bandit (MAB) framework motivated by three common complications in online recommender systems in practice: (i) the platform (learning agent) cannot sample an intended product directly and has to incentivize customers to select this product (e.g., promotions and coupons); (ii) customer feedbacks are often received later than their selection times; and (iii) customer preferences among products are inﬂuenced and reinforced by historical feedbacks.
From the platform’s perspective, the goal of the MAB framework is to maximize total reward without incurring excessive incentive costs.
A major challenge of this MAB framework is that the loss of information caused by feedback delay complicates both user preference evolution and arm incentivizing decisions, both of which are already highly non-trivial even by themselves.
Toward this end, we ﬁrst propose a policy called “UCB-Filtering-with-Delayed-Feedback” (UCBFDF) policy for this new MAB framework.
In our analysis, we consider delayed feedbacks that can have either arm-independent or arm-dependent distributions.
In both cases, we allow unbounded support for the random delays, i.e., the random delay can be inﬁnite.
We show that the delay impacts in both cases can still be upper bounded by an additive penalty on both the regret and total incentive costs.
This further implies that logarithmic regret and incentive cost growth rates are achievable under this new MAB framework.
Experimental results corroborate our theoretical analysis on both regret and incentive costs.
I NTRODUCTION
In recent years, the multi-armed bandit (MAB) framework has received a signiﬁcant amount of interest in the learning research community.
This is partly due to the fact that, in many online ecommerce recommender systems (e.g., Amazon and Walmart), the problem of online learning of the optimal products while making proﬁts at the same time can be well formulated by an MAB problem.
However, although many MAB algorithms have been proposed in this area, it is worth noting that most of the existing MAB models in the literature have not considered the joint effect of three common phenomena in e-commerce recommender systems: (i)
In many e-commerce recommender systems, the platform (the learning agent) cannot sample an intended product (an intended arm) directly and has to incentivize customers (e.g., through promotions and coupons) to sample the product and receive the sampling feedback from the customers indirectly (e.g., ratings and reviews); (ii) Customer feedbacks are often received much later than their purchasing times (e.g., a review may or may not be submitted by a customer even months later after purchasing a product); and (iii)
Customer preferences among products are inﬂuenced and reinforced by historical feedbacks, which may even lead to various viral effects over some products (the more good reviews one product has received, the more likely that the next arriving customer will prefer this product).
The lack of a fundamental understanding and joint studies of these three important factors in MAB policy designs motivates us to ﬁll this gap in this paper.
Toward this end, we propose a new MAB framework that jointly considers i) incentivized sampling, ii) delayed sampling feedback, and iii) self-reinforcing user preferences in online recommender systems.
However, we note that the MAB policy design for the proposed new MAB framework is highly non-trivial due to the complex couplings between the aforementioned three factors.
First, similar to conventional MAB problems, there exists a dilemma between sufﬁcient exploration through sampling to learn an optimal arm (i.e., an optimal product), which may incur numerous pullings of sub-optimal arms, and the greedy exploitation to play the arm that has performed well thus far to earn proﬁts.
Second, there is another dilemma to the learning agent between offering sufﬁciently attractive incentives to mitigate biases (due to lack of initial data and self-reinforcing user preferences) and avoid spending unnecessarily high incentives that hurt the learning agent’s proﬁts.
Last but not least, the delayed sampling feedbacks may render the estimation of arms’ quality during the MAB process highly inaccurate, introducing yet another layer of uncertainty to the MAB online learning problem, which is already plagued by complications from incentivized sampling and self-reinforcing user preferences.
As in most MAB problems, we adopt “regret” as our performance metric in this paper, which is deﬁned as the cumulative reward gap between the proposed policy and an optimal policy design in hindsight.
Under the regret setting, the complications due to these three key factors naturally prompt the following fundamental questions:
(1) How should the agent design an incentivizing strategy to strike a good balance between exploration and exploitation to achieve sublinear (hopefully logarithmic) regrets?
(2) To avoid offering exceedingly high incentives, how should the agent incentivize in order to attract a user crowd that prefer an optimal arm, so that the users’ self-reinforcing preference could automatically gravitate toward this optimal arm without further incentives?
(3) Under various delayed feedback situations in the new MAB framework (e
g., unbounded random delays, heavy-tailed delay distributions, and arm-dependent delays), could we still achieve low regrets with low incentive costs?
In this paper, we answer the above fundamental questions afﬁrmatively by proposing a new “Delayed-UCB-Filtering” policy for the MAB framework that jointly considers incentivizing sampling, delayed sampling feedback, and self-reinforcing user preferences.
We show that our proposed policy achieves O (log T ) regret with O (log T ) incentive payments.
The success of our policy design hinges upon two key insights: (i) the self-reinforcing user preference effect is actually a “blessing in disguise” and can be leveraged to establish an important “dominance” condition (more on this later) that further implies O (log T ) regret and incentive costs; and (ii) the impacts of delayed feedback on regret and incentive costs can be upper bounded under appropriate statistical settings to preserve the “dominance” condition.
Our key contributions and main results are summarized as follows:
• We propose a new MAB model that jointly considers incentivized arm sampling, delayed sampling feedback, and self-reinforcing user preferences, all of which are important features of online recommender systems.
To develop efﬁcient and low-cost incentivized policy for this new MAB model, we propose a three-phase “UCB-Filtering-with-Delayed-Feedback” (UCB-FDF) policy, which contains an incentivized exploration phase, an incentivized exploitation phase, and a selfsustaining phase.
In our UCB-FDF policy, the ﬁrst two phases judiciously integrate delayed feedback information, while in third phase, the system solely relies on self-reinforcing user preferences to converge to the pulling of the optimal arm.
• We ﬁrst show a fundamental fact that, under our UCB-FDF policy, delayed sampling feedback only has an additive penalty on the regret and incentive cost performances, and that this additive penalty grows logarithmically with respect to time.
Speciﬁcally, we ﬁrst investigate the delayed feedback impact under the assumption that the feedback delay is an i.i.d.
random variable across samplings with a ﬁnite expectation.
We show that the UCB-FDF policy achieves logarithmic growth rates of regret and incentive costs under this setting.
Then, we relax the i.i.d.
feedback delay assumption to allow the feedback delay distribution to be arm-dependent.
Under this setting, we also show that similar logarithmic growth rates of regret and incentive can still be achieved.
• We conduct extensive experiments on Amazon Review Data 1 to demonstrate and verify the performance of our UCB-FDF policy as well as
the impacts of delayed feedback on real-world scenarios.
We also verify our theoretical analysis through various product categories and demonstrate the efﬁcacy of our proposed UCB-FDF MAB policy.
The rest of the paper is organized as follows.
In Section 2, we review the literature to put our work in comparative perspectives.
Then in Section 3, we formulate our new MAB model that capture the three common phenomena.
In Section 4, we present our UCB-FDF policy and analyze its performance.
Then, we present our experiment settings and results in Section 5.
Due to space limitations, the proofs are relegated to the appendix.
R ELATED W ORK
In this section, we provide a quick overview on three lines of research related to our work: i) bandits with delayed feedback, ii) bandits with random preferences, and iii) incentivized bandits.
1) Bandits with Delayed Feedback: Motivated by practical issues in the clinical trials, Eick (1988) was the ﬁrst to introduce a two-armed bandit model with delayed responses, where the patients survival time reports after the treatment are delayed.
Recently, Joulani et al.
(2013) provided a systematic study and showed that for delay τ with a ﬁnite expectation, the worst case regret scales with O ( √ KT log T + K E [ τ ]) , where K is the number of arms.
Meanwhile, Vernade et al.
(2017) showed that stochastic MAB problems with delayed feedback have a regret lower bound O
( K log T ) .
However, this work assumed that the distribution of the random delay is arm-independent.
In contrast, Joulani et al.
(2013) considered arm-dependent delay distributions that have an upper bound of the maximum random delay.
More recently, Manegueu et al.
(2020) considered arm-dependent and heavy-tailed delay distributions, where only an upper bound on the tail of the delay distribution is needed, without requiring the expectation to be ﬁnite.
Also, Lancewicki et al.
(2021) studied the case where the delay distribution is reward-dependent, which implies that the random delay in each round may also depend on the reward received on the same round.
However, most of these works on delayed bandits are based on the standard stochastic MAB framework.
In contrast, we consider delayed feedback in incentivized bandit learning with self-reinforcing user preferences, which is a more appropriate model for real-world recommender systems than the standard stochastic MAB.
2) Bandits with Random Preferences: The impacts of random user preferences in e-commerce platforms have received increasing interest in several different areas in learning and economics.
Existing works in (Agrawal et al., 2017; 2019) formulated the user preference variation given different product bundles by the multi-nomial logit model on top of the bandit learning framework and proposed a Thompson Sampling approach that achieves a worst-case regret bound of O ( √ NT log TK ) , where N is the size of recommended arm bundle.
With a different focus on preference modeling,
Barab ´ asi & Albert (1999); Chakrabarti et al. ; Ratkiewicz et al.
(2010) investigated the network evolution with “preferential attachment” that formulates the social behavior known as selfreinforcing preferences, among which the works in (Shah et al., 2018;
Zhou et al., 2021) are the closet to our work.
To our knowledge, Shah et al.
(2018) was the ﬁrst to consider self-reinforcing user preferences in bandit learning problems.
Later, Zhou et al.
(2021) incorporated self-reinforcing user preferences into the incentivized bandit learning framework.
The key difference between these two works is that, in the model in (Shah et al., 2018), only one arm is revealed to users in each round, while in the model in (Zhou et al., 2021), all arms are revealed to users and users’ arm selections are inﬂuenced by incentives.
However, both of these works fall short in modeling online recommender systems in practice as they assume that an arm-sampling feedback is observable in the same timeslot when an arm is pulled.
However, for most e-commerce recommender systems in practice, user feedbacks are often not immediately observable.
As a result, the decision on which arm to pull next has to be made without some of the feedbacks from arm-pulling actions in the past.
Incentivized Bandits: To our knowledge, Frazier et al.
(2014) was among the ﬁrst to adopt incentive schemes into a Bayesian MAB setting.
In their model, the agent seeks to maximize timediscounted total reward by incentivizing arm selections.
Later, Mansour et al.
(2015) studied the non-discounted reward setting.
For the non-Bayesian setting, Wang & Huang (2018)
; Zhou et al.
(2021) proposed policies that maximize the total non-discounted reward with bounded incentive costs.
Bandits with budget (Guha & Munagala, 2007; Goel et al., 2009) also share some similarities with our work, where the agent takes actions under resource constraints that are either ﬁxed or with a given growth rate bound.
However, none of the aforementioned works considered the impacts of delayed feedback on the regret and incentive costs performances.
Note that, due to the loss of information caused by delayed feedback, larger variances in the mean-reward estimations of the arms are inevitable.
This implies that, in order to achieve a more accurate arm quality estimation under delayed feedbacks, a higher incentive cost is necessary.
S YSTEM M ODEL
We ﬁrst introduce the MAB framework with incentivized sampling, delayed feedback, and selfreinforcing user preferences.
The system has a set of K ≥ 2 arms denoted by A = { 1 , . .
. , K } , and each arm a follows a Bernoulli reward distribution P a with an unknown mean µ a > 0 .
The bandit time horizon has T rounds.
In each time step t = 1 , 2 , . . .
, T , a user arrives and chooses an arm I t to pull.
Then, the user will receive a random reward feedback X t ∼ P I t .
Both the arm selection
I t and the feedback X t are observable to the agent.
We use T a ( t ) (cid:44) (cid:80) ti =1 1 { I i = a } to denote the number of times that arm a is pulled up to time step t .
We let T a (0) = 0 , ∀ a ∈ A .
We assume that there is a unique best arm a ∗ ∈
A in the sense that a ∗ = arg max a ∈A µ a and let µ ∗ = µ a ∗ .
Also, we deﬁne ∆ a (cid:44) µ ∗ − µ a as the gap between the mean of the optimal arm and the mean of arm a .
3. D ELAYED F EEDBACK M ODELING
In this paper, we consider delayed feedback, i.e., when an arm I t is pulled at time step t , the corresponding Bernoulli reward X t is observed after a delay period τ I t ,t , i.e., the feedback X t is observed at time step t
+ τ I t ,t .
Without loss of generality, we model the random delay time as a random variable τ a,t ∼ T a , where the delay distribution T a of arm a is unknown to the agent.
We consider two settings of delayed feedback.
We ﬁrst consider i.i.d.
delays { τ t } t ≤ T across time and arms, i.e., the delay distributions are identical for all arms.
Thus, we omit the arm index in the notations of delay feedback in this setting.
Next, we generalize the delay modeling by allowing arm-dependent delay distributions, where the delay distributions are allowed to differ across arms.
In both settings, we do not make further assumptions on the delay distributions, except that we only require a ﬁnite delay expectation.
Note that we allow the support of the delays to be unbounded, i.e., an inﬁnite delay time is possible in both settings.
This models the practical scenarios in online recommender systems that some user feedbacks (e.g., ratings and reviews) may never be received.
Under delayed feedbacks, we denote the total number of missing feedbacks from arm a up to a time step t as D a ( t ) (cid:44) (cid:80) ts =1 1 { s + τ a,s >t } .
We let D ∗ a ( t ) =
max 1 ≤ s ≤ t D a ( s ) , ∀ a ∈ A as the maximum total number of delayed feedback for arm a up to time t .
Note that D ∗ a ( t ) = 0 , ∀ a ∈ A corresponds to the non-delayed setting.
In this case, T a ( t ) denotes the total number of pulling times of arm a up to time t .
At each time step t , the agent observes a set of time-stamped feedback denoted by
S t ⊂ N × { 0 , 1 } .
In the set S t , each element is a pair of time index and a Bernoulli reward value, and the time index is the time step when the corresponding reward is observable.
Note that in this model, by observing the set S t , the agent is aware of the information of both the time step when the feedback is received, and the arm that generated the feedback.
We denote the total reward generated by arm a up to time t as S a ( t ) (cid:44) (cid:80) ts =
1 X s · 1 { I s = a,s + τ a,s ≤ t } , and let S a (0) = 0 , ∀ a ∈ A .
3. U SER P REFERENCES AND I NCENTIVE I MPACT M ODELING
In this paper, we assume that the arrival at time t has a non-zero probability p a ( t ) ∈ (0 , 1) to pull each arm a ∈ A .
We note that p a ( t ) can also be thought of as the user’s preference rate of arm a , and (cid:80) a ∈A p a ( t )
= 1 , ∀ t ≤ T .
We adopt the widely accepted multinomial logit model in the economics literature(Bawa & Shoemaker, 1987) to model arm a ’s preference rate at time step t as follows:
[Equation]
where f ( · ) : R → (0 , + ∞ ) is a feedback function that is increasing, and θ a > 0 denotes a ﬁxed initial preference bias of arm a .
We note that the preference rate modeling in (Zhou et al., 2021) is also based on the multinomial logit model, which appears to be in the same form as in (1).
However, the key difference between our preference model in (1) and that in (Zhou et al., 2021) is that the accumulative award information S i ( t − 1) in (1) accounts for reward information that can only be observed up to time t .
In other words, S i ( t − 1) in (1) is affected by feedback delays.
In fact, the preference model in (Zhou et al., 2021) can be viewed as a special case of our model with zero delay.
Since the arriving users select arms based on preferences, while the agent aims to maximize the total reward in the long run, there exists a general difference between users’ arm preferences and agent’s intended arm selection.
To induce users to pull arms following the agent’s goal, the agent needs to intervene users’ arm pulling by offering incentives on its desired arm, so as to increase the user preference of pulling the arm.
That is, the agent incentivizes arm I (cid:48) t at time step t so that p I (cid:48) t ( t ) increases accordingly.
Note that when p I (cid:48) t ( t ) increases, the preference rates on the other arms will decrease since (cid:80) a ∈A p a ( t ) = 1 , t ≤ T .
In this paper, we adopt the “coupon effect” model, which is widely used in the economics and marketing literature (Bawa & Shoemaker, 1987).
Speciﬁcally, we consider a ﬁxed incentive b in each time step and denote the time-dependent incentive impact as g ( b, t ) .
Then, the posterior preference rates of the arms with incentive b are updated as follows:
[Equation]
We remark that the deﬁnition of the posterior preference update in (2) also follows from the multinomial logit model, which is widely used to model user preferences and their variations in bandit ﬁeld (Chen & Wang, 2017; Avadhanula, 2019;
Dong et al., 2020;
Zhou et al., 2021).
Based on the deﬁned posterior preference, as incentive impact g ( b, t ) increases to inﬁnity (either the incentive value b increases to inﬁnity or the users are more sensitive to incentives as time goes by), the user preference will be induced to pulling the agent’s desired arm a with probability one.
For further detailed interpretations of the incentive impact function g ( b, t ) , we refer readers to the literature (e.g., (Zhou et al., 2021)).
Note also that, due to the random user behaviors, it is possible that
I (cid:48) t (cid:54) =
I t , i.e., the arm that the agent incentivizes is not the one that a user pulls eventually.
We deﬁne the accumulative incentive up to time step t as B t (cid:44) (cid:80) ts =1 b t , where b t ∈ { 0 , b } , ∀ t ≤ T , denotes the agent’s binary decision whether to offer incentive b at time step t .
3. R EGRET M ODELING
As in most bandit learning problems, the goal of the agent is to maximize the total expected reward E (cid:2) (cid:80) [Figure] S a ( T ) (cid:3) in the long run.
Toward this end, we need the notion of the oracle incentivized policy, where in hindsight, the agent is aware of the optimal arm a ∗ and can always offer an inﬁnite amount of payments to users with feedback being observable immediately, so that the posterior preference rate of arm a ∗ is always inﬁnitely close to one.
As a result, the expected accumulative reward generated under the oracle policy up to time T is E
[ S a ∗ ( T )]
= µ ∗ · T .
However, since the optimal arm a ∗ is unknown to the agent, the goal of the agent is to maximize the total expected reward E [Γ T ] in the long run by designing an incentivized policy with low accumulative incentive in the presence of self-reinforcing preferences and feedback delay.
Similar to conventional MAB, we measure the performance gap between our accumulative reward against that of the oracle policy, which is denoted by regret R T .
The expected (pseudo) regret is deﬁned as follows:
[Equation]
In this paper, our goal is to minimize E [ R T ] with low expected accumulative payment E [ B T ] , i.e., sub-linear growth rate regarding time horizon T .
It is clear that any policy with bounded payment cannot outperform the oracle policy.
Thus any expected regret deﬁned by comparing with boundedpayment policy is upper bounded by our regret.
B ANDIT P OLICY D ESIGN AND P ERFORMANCE A NALYSIS
In this section, we ﬁrst present the general version of the UCB-FDF policy that works with any delay distributions, where we upper bound the delay impact on the regret and incentive costs.
Based on this general result, we then study the regret and incentive costs performance of UCB-FDF under the assumptions of 1) i.i.d.
feedback delay across arms/times and 2) arm-dependent delay distributions.
In both cases, we denote the total number of missing feedbacks over all arms by
D ( t ) (cid:44) (cid:80) a ∈A D a ( t ) , and denote the maximum number of missing feedbacks during the ﬁrst t time steps by D ∗ ( t ) (cid:44) max 1 ≤ s ≤ t D ( s ) .
For arm a at time step t , we denote the number of its pulling times whose feedback is observed by T (cid:48) a ( t ) =
T a ( t ) − D a ( t ) , and denote the maximum mean gap by ∆ ∗ = max a ∈A ∆ a .
At time step t , we denote the sample mean estimation (due to delayed feedbacks) of arm a by ˆ µ a ( t ) =
S a ( t ) /T
(cid:48) a ( t ) .
Our UCB-FDF policy is illustrated in Algorithm 1.
UCB-FDF policy contains three phases: an incentivized exploration phase, an incentivized exploitation phase, and a self-sustaining phase.
UCB-FDF policy tackles feedback delays in the following two key aspects: (i) correcting the sample mean estimate of arms by only considering the number of pulling times that have observed feedback, (ii) setting the length of the exploitation phase in such a way that the outstanding rewards do not harm the emergence of “dominance” (i e., one arm receiving at least half of the rewards) of the sampled optimal arm.
Subsequently, these two aspects also inﬂuence the regret and incentive.
In order to have enough arm exploration with an unbiased sample mean estimate, the loss of counted number of pulling times necessitates a carefully designed exploration phase that incentivizes the pulling of the least informed arm a ∈ arg min i ∈U T (cid:48) a ( t ) under delayed feedbacks.
Similarly, the delay-based dominance threshold (i.e., S
[Equation]
∗
[Equation]
t )
[Equation]
in Step 3 of Algorithm 1) guarantees the dominance of sampled optimal arm, while also accounts for a longer exploitation phase to mitigate the delayed feedback effect.
We now analyze the upper bounds of the pseudo regret and expected incentive of the UCB-FDF policy.
Lemma 1.
(UCB-Filtering-with-Delayed-Feedback)
Given a ﬁxed time horizon T , if g ( b, t ) > 1 , and f ( x ) =
Θ( x α ) with α > 1 2 , then the pseudo regret of Algorithm 1 E [ R T ] is upper bounded by
[Equation]
with the expected payment E [ B T ] upper bounded by
[Equation]
Remark 1.
The UCB-FDF policy achieves a sub-linear total incentive cost by leveraging the property of self-reinforcing preference.
We can show that as long as the self-reinforcing preference function f ( x ) satisﬁes the condition f ( x ) =
Θ( x α ) with α > 1 , then “monopoly” happens with probability one (i.e., the scenario where only one arm has positive probability to be pulled, thus this particular arm is the only preferred arm).
A natural incentivizing policy is to incentivize sampled optimal arm until it achieves monopoly.
However, the key challenge here is that the onset of monopoly could take inﬁnite time steps, which implies linear total incentive.
Moreover, selfreinforcing property is not merely disrupting the system from converging to the optimal arm.
The key idea in our UCB-FDF policy design is that under the condition of the self-reinforcing preference function f ( · ) , after one arm establishes its dominance (i.e., the arm a generates at least half of the current total reward), it will have exponentially increasing probability to beat other arms and achieve monopoly.
More importantly, we can show that the onset of arm dominance takes sub-linear times, thus allowing us to achieve sub-linear total incentive costs.
Remark 2.
The feedback delay affects the observation of arm dominance, since the missing reward information from suboptimal arms, if not compensated carefully, can potentially destroy the dominance status of the optimal arm.
Thus, to guarantee dominance of the optimal arm, a longer exploitation phase is necessary, and thus a large total incentive is required.
The existence of delays in our MAB model introduces an additive term Θ( E [ D ∗ ( T )]) in both regret and incentive costs, which is dependent on the maximum accumulated delayed feedback up to time horizon T .
Based on Lemma 1, in what follows, we will analyze the upper bounds of the expected maximum accumulated delayed feedback under different assumptions on delay distributions.
4. A RM -I NDEPENDENT D ELAY WITH A F INITE E XPECTATION
We now analyze the delay impact under our ﬁrst assumption.
In the arm-independent case, we consider an i.i.d.
sequence { τ t } t of random delay regarding time step t ≤ T .
We do not make any assumption on the shape of the delay distribution, except that we only assume a ﬁnite expectation E [ τ 1 ] .
Thus, an inﬁnite random delay is possible under this assumption, which some feedbacks may never be observed by the agent.
Our results show that under this assumption, we can still achieve similar orders of the regret and incentive costs growth rates, since the key fact is that we can upper bound the expected number of such unexpectedly large random delays for every time step t .
Existing works (e.g., (Joulani et al., 2013)) provided a systematic study on the delay effect on the partial monitoring problem with side information, including the stochastic problems.
Although these works only considered the classic stochastic MAB, they share some similarities with our work in that their analysis of delay effects also leveraged the maximum number of missing feedbacks during the ﬁrst t time steps D ∗ ( t ) .
However, since our UCB-FDF policy has a different structure compared to these works on delayed stochastic MAB, their delay analysis is not applicable to our policy.
Next, we restate a result in (Joulani et al., 2013), which will be useful in our analysis.
Lemma 2 (Lemma 2 in Joulani et al.
Assume { τ 1 , . . .
, τ t } is a sequence of i.i.d.
random variables with ﬁnite expected value, and let B ( t, s ) = s + 2 log t + √ 4 s log t .
Then, it holds that
[Equation]
Theorem 3.
(Arm-Independent Delay)
Under i.i.d.
delays with a ﬁnite expectation and the conditions of Lemma 1, the pseudo regret of Algorithm 1 E [ R T ] is upper bounded by (cid:20)
[Equation]
with the expected payment E [ B T ] upper bounded by
[Equation]
We note that the gap summation of arms (cid:80) a (cid:54) =
a ∗ ∆ a plays an important role in both regret and total incentive.
As arm gaps getting smaller, it is more difﬁcult to distinguish the optimal arm from others.
Thus, a longer exploration phase is required to conduct enough sampling, which implies a larger regret and a larger total incentive costs.
On the other hand, the feedback delay causes additive terms in both regret and incentive costs in terms of the expected delay E [ τ 1 ] , and the delay impact can be upper bounded as long as the expected delay E [ τ 1 ] is no larger than time horizon T .
4. A RM -D EPENDENT D ELAY WITH F INITE E XPECTATIONS
Now, we further relax the assumption on the delay to allow arm-dependent delays.
In this case, the delay has two key impacts on the system: (i) for each arm, there is a different real-time information loss when estimating the sample mean, (ii) for the whole arm set, different scales of delay cause an uneven arm estimation, which results in a larger risk of the elimination of the optimal arm in the UCB-based exploration step.
We formally state our arm-dependent delay assumption as follows:
Assumption 1.
The delays of arm a ∈ A form an independent delay sequence { τ a,t } , where each element is a random variable satisfying τ a,t ∼ T a , with a ﬁnite expectation E [ τ a, 1 ] < + ∞ , ∀ a ∈ A .
Under Assumption 1, we show a more general result on the upper bound of E [ D ∗ ( t )] as follows:
Lemma 4.
Under Assumption 1, given a ﬁnite number of arms K > 0 , it holds that
[Equation]
The result in Lemma 4 implies larger upper bounds of the regret and incentive, due to the existence of the pre-log factor K .
This is a consequence of the situation where, as we consider arm-dependent delay distributions, the worst case could be evenly distributed expected delays E [ τ a, 1 ] of arm a with respect to time horizon T .
Formally, we state the upper bounds of regret and incentive as follows:
Theorem 5.
(Arm-Dependent Delay)
Under Assumption 1 and the conditions of Lemma 1, the pseudo regret of Algorithm 1 E [ R T ] is upper bounded by
[Equation]
with the expected payment E [ B T ] upper bounded by
[Equation]
Similar to the results under the i.i.d.
delay assumption, we can still upper bound the regret and incentive by an logarithmic growth rate O (log T ) under arm-dependent delay.
This implies that even under the weak delay assumption where only ﬁnite expectation is needed, UCB-FDF can estimate arms without too much bias, and ﬁnally achieve logarithmic regret with logarithmic incentive costs.
E XPERIMENTAL R ESULTS
We evaluate the performance of our policy and the impact of delayed feedback on real-world scenarios.
In this section, we ﬁrst introduce our experiment setting and the dataset, and then illustrate our experimental results with discussions.
5. E XPERIMENTAL S ETUP
1) System Parameters: We conduct experiments under two different delay settings.
The system parameters are set as follows: a three-armed model with Arm1 being the optimal arm, and the initial preference bias θ = [1 , 5 , 5] , i.e., the optimal arm has the least initial bias.
We choose a three-armed model since large arm set requires a proportional large time horizon to distinguish optimal arm, while in the public Amazon Review Data, the amount of reviews for most products is limited (no more than 3,000 for each product).
The self-reinforcing preference function is chosen as f ( x ) =
x α with α = 2 .
The constant incentive for each time step is set as b = 1 . 5 with an incentive impact function g ( b, t )
= b .
For the delay distribution, we use normal distributions in both assumption setting, as normal distributions have an inﬁnite support x ∈ R .
Under the arm-independent delay setting, we choose the delay distribution as τ t ∼ N (10 , 2) .
Under the arm-dependent delay setting, we choose the delay distributions as τ 1 ,t ∼ N (10 , 2) , τ 2 ,t ∼ N (15 , 2) , and τ 3 ,t ∼ N (18 , 2) for Arms 1, 2, and 3, respectively.
We only generate non-negative samples of delay under both assumptions.
[Table]
2) Dataset: We use Amazon Review Data (Ni et al., 2019) to provide a practical learning environment.
The Amazon Review Data includes 233 million customer reviews (ratings, posting times) for 29 product categories.
In the experiment, we select three products to serve as the arms that have the largest number of reviews in category Pet Supplies, Electronics, Home and Kitchen, and Books, respectively.
For each product (arm), we leverage the rating and unixReviewTime information in each review, and the total number of reviews is 3,000 for each product.
The range of ratings in Amazon Review Data is the discrete set { 1 , 2 , 3 , 4 , 5 } .
We convert the rating values to binary by setting the rating values 1 and 2 as 0 and the rating value 4 and 5 as 1 , and the reviews with rating value 3 are removed.
For each product, the binary review ratings are sorted by unixReviewTime , so the ratings come in real-world order in the experiment.
We summarize the mean values of the products by their Bernoulli ratings in the four selected categories, as shown in Table 1.
5. R ESULTS AND D ISCUSSIONS
1) Results: The experiment results are illustrated in Figure 1.
(a) and (b) show the average regret and incentive trends under arm-independent delays and arm-dependent delays, respectively.
Each curve is constructed by regret or incentive values with different time horizons from T = 150 to T = 3000 , incremented by 150 .
Each node value in curves are averaged by 100 trials.
Figure 1(c) illustrates the 100 values of regret or incentive costs with time horizon T = 3000 under arm-dependent delays.
[Figure]
2) Discussions; From the ﬁgures, we observe that our UCB-FDF policy achieves sub-linear growth rates for regret and incentive costs under our assumptions of feedback delays.
Comparing results under arm-independent delays in Figure 1(a) and arm-dependent delays in Figure 1(b), we observe that arm-dependent delays require higher incentive costs and result in larger regret, due to the longer exploration and exploitation phases.
Comparing regrets in different categories, we observe that the regret growth trends of categories “Books” and “Home and Kitchen” increase rapidly initially but quickly slow down, since the arms in these two categories have larger gap summations (cid:80) a (cid:54) =
a ∗ ∆ a .
Although a large gap summation may cause larger regret before the policy ﬁnds the optimal arm, it also allows the algorithm to distinguish the optimal arm faster.
More straightforward results can be seen by the incentive curves in different categories.
From a different angle, as we show in
Figure 1(c), some of the total incentives in category Pet Supplies are exactly 4500, implying that in many random cases, the policy is still in the exploration or exploitation phase, while in most of the random cases in categories Books and Home and Kitchen, the incentive can stop early while also reach sub-linear regret.
This difference is also caused by the variation in gap summations.
C ONCLUSION
In this work, we proposed a practical bandit model that considers the joint effect of the incentive impact, delayed feedback and self-reinforcing user preferences in real-world recommender systems.
We proposed a UCB-FDF policy that achieves logarithmic growth rates of pseudo regret and total incentive costs for a ﬁxed time horizon T .
We also analyzed how different delay assumptions inﬂuence the regret and incentive costs.
Speciﬁcally, we considered arm-independent delays and arm-dependent delays with weak assumption that only requires a ﬁnite expectation.
The evaluations with real-world customer review data showed the effectiveness of our UCB-FDF policy in achieving sub-linear regret while spending only sub-linear total incentive costs under delayed feedback.
A A PPENDIX I: P ROOFS A. P ROOF OF L EMMA
Lemma 1.
(UCB-Filtering-with-Delayed-Feedback)
Given a ﬁxed time horizon T , if g ( b, t ) > 1 , and f ( x ) =
Θ( x α ) with α > 1 3 , then the pseudo regret of Algorithm 1 E [ R T ] is upper bounded by
[Equation]
with the expected payment E [ B T ] upper bounded by
[Equation]
By the law of total expectation, the expected regret up to time T can be bounded as follows:
E [ R T ] =
E [ R t 0 ] + E [ R T | ˆ a ∗ = a ∗ ] P (ˆ a ∗ = a ∗ ) + E [ R T | ˆ a ∗ (cid:54) = a ∗ ] P (ˆ a ∗ (cid:54) = a ∗ ) ≤ E [ R t 0 ] + E [ R T | ˆ a ∗ = a ∗ ] + T · P (ˆ a ∗ (cid:54) = a ∗ ) =
E [ R t 0 ]
+ E [ R t 2 | ˆ a ∗ = a ∗ ] + E [ R T − R t 2 | ˆ a ∗ = a ∗ ] + T · P (ˆ a ∗ (cid:54) = a ∗ ) = E [ R t 0 ] + E
[Equation]
[Equation]
[Equation]
We start from bounding the term E
[ R t 0 ] .
Note that the goal of the initialization step is to ensure that the conﬁdence interval for each arm is initialized.
Then we have
[Equation]
where (i) is because that by deﬁnition of initialization, for each arm a , it contributes to t 0 in terms of its incentivizing attempts for being pulled by users once, and its delay period after which arm a will be regarded as initialized, then at time step t , when offered incentive b , arm a has probability no less than g ( b, t ) / (cid:0)
+ g ( b, t ) (cid:1) to be pulled by users, thus the number of attempts for arm a to be pulled once is a geometric random variable with parameter no less than g ( b, t ) / (cid:0)
+ g ( b, t ) (cid:1) , with expectation value (cid:0) 1 + g ( b, t ) (cid:1) /g
( b, t ) , and (ii) is because that by condition g ( b, t ) > 1 , we have (cid:0)
+ g ( b, t ) (cid:1) /g
( b, t ) < 2 .
Then we bound the term E
[ R t 1 ] .
During the exploration phase, since the regret is caused by pullings of sup-optimal arms, the expected regret after t ≤ t 1 time steps can be written as
[Equation]
Thus we can bound the expected regret during the exploration phase E
[ R t 1 ] by bounding each E
[ T a ( t 1 )] for a (cid:54) = a ∗ .
Let U ( t ) denote the set of arms that are activated, i.e., can get incentive, at time t .
Consider the following two cases during the exploration phase: (a) At time t ≤ t 1 , a ∗ ∈ U ( t ) and there exists at least one suboptimal arm a ∈ A , a (cid:54) =
a ∗ such that a ∈ U ( t ) .
Recall that c a ( t ) =
(cid:113) ln T/ 2 (cid:0) T a ( t ) − D a ( t ) (cid:1) is the conﬁdence interval of arm a at time step t .
In this case, we have:
[Equation]
where (i) is obtained since arms a, a ∗ ∈ U ( t ) implies that the upper conﬁdence bound of either arm is larger than the other arms’s lower conﬁdence bound, (ii) is because µ a +∆ a / 2 = µ ∗ − ∆ a / 2 , and the event { ˆ µ a ( t ) + c a ( t ) >
ˆ µ ∗ ( t ) − c
a ∗ ( t ) } implies either the event { ˆ µ a ( t ) + c a ( t ) > µ a + ∆ a / 2 } or the event { ˆ µ ∗ ( t ) − c a ∗ ( t ) < µ ∗ − ∆ a / 2 } .
We consider the ﬁrst probability in Eq.
(6), and similar bound will be obtained for the second probability.
By Chernoff-Hoeffding bound we have
[Equation]
Let ∆ 2 a 2 ( T a ( t ) − D a ( t ))
− ∆ a (cid:112) 2( T a ( t ) − D a ( t ))
ln T = 0 , we obtain T a ( t ) = 8 ln T/ ∆ 2 a + D a ( t ) and Eq.
(7) equals 1 /T .
Note that as ( T a ( t ) − D a ( t )) increases, Eq.
(7) decreases monotonically.
Thus, in this case, the expected regret contributed by a suboptimal arm a ∈ A during the exploration phase is bounded by
[Equation]
(b) At time t ≤ t 1 , a ∗ is eliminated by some suboptimal arm a ∈ U ( t ) , a (cid:54) =
a ∗ .
In this case, with similar technique as that in case (a) and Chernoff-Hoeffding bound, we have
[Equation]
Note that P (ˆ a ∗ (cid:54) = a ∗ ) =
P (cid:0) ∃ a (cid:54) =
a ∗ : a ∈ U ( t ) , a ∗ / ∈ U ( t ) (cid:1) .
Thus, in this case the expected regret contributed by a suboptimal arm a ∈ A is upper bounded by
[Equation]
Summing Eq.
(9) and Eq.
(11) over all suboptimal arms [Equation] the expected regret during the exploration phase is bounded by: [Equation]
[Equation]
During the exploration phase at time step t < t 1 , since the agent offers incentive payment b to the user for pulling arm i , the probability that the arm i is pulled is (cid:0) p i ( t ) + g ( b, t ) (cid:1) / (cid:0)
+ g ( b, t ) (cid:1) lower bounded by g ( b, t ) / (cid:0) 1 + g ( b, t ) (cid:1) .
Thus, the number of attempts for arm i to get pulled is a geometric random variable with expectation no larger than (cid:0)
+ g ( b, t ) (cid:1) /g
( b, t ) .
Since the above cases (a) and (b) imply that at most 8 ln T/ ∆ 2 a + E [ D ∗ a ( t 1 )] + 4 expected number of pullings from each suboptimal arm ensures a good estimation of the optimal arm, thus, the expected number of incentivizing attempts on a suboptimal arm a to guarantee 8 ln T/ ∆ 2 a + E [ D ∗ a ( t 1 )] + 4 number of pullings is upper bounded by:
[Equation]
Thus, E [ τ 1 ] is upper bounded by:
[Equation]
where (i) is due to the requirement of T a ∗ ( t 1 ) to be at most 8 ln T/ ∆ 2 min
+ E [ D ∗ a ∗ ( t 1 )] , since the exploration phase stops once the sampled strongest suboptimal arm is eliminated.
By the deﬁnition of dominance, arm ˆ a ∗ is expected to dominate at time t ≥ t 1 if
[Equation]
We tighten the condition by narrowing the left-hand-side and amplifying the right-hand-side
[Equation]
and obtain the sufﬁcient condition of dominance as follows:
[Equation]
⇒ E [ t − τ 1 ] (iii) ≥ g [Equation]
b
[Equation] [Equation] 14) where (i) is obtained since T ˆ a ∗ ( t 1 ) > 0 and D ∗ ˆ a ∗ ( t ) ≥ D ˆ a ∗ ( t ) , (ii) is because by incentivizing arm ˆ a ∗ , we have ˆ p ˆ a ∗ ( t ) ≥ g ( b, t ) / (cid:0) g ( b, t ) + 1 (cid:1) and
ˆ p a ( t ) ≤ 1 / (cid:0) g
b, t )
+ 1 (cid:1) for a (cid:54)
=
ˆ a ∗ , and (iii) is the rearrangement.
Since time t 2 is deﬁned as the earliest time to reach dominance, we can upper bound E [ t 2 − t 1 ] by [Equation]
[Equation]
Thus, we can bound the regret during the exploitation phase E
[ R t 2 − R t 1 | ˆ a ∗ = a ∗ ] in (3) by E [Equation] R
[Equation]
[Equation] R [Equation] where (i) follows because during the exploitation phase there is always a positive probability ˆ p a ( t ) which is at most 1 / (cid:0) g ( b, t ) + 1 (cid:1) to pull suboptimal arm a .
After arm ˆ a ∗ dominates, we want to prove that arm ˆ a ∗ has exponentially large probability to achieve monopoly, so as to upper bound the regret in the self-sustaining phase
E
[ R T − R t 2 | ˆ a ∗ = a ∗ ] .
Since the dominance proof only involves the accumulated reward S a ( t ) , our situation reduces to the non-delay case in expectation.
Thus, we omit the proof in Zhou et al.
and show the result by
[Equation]
) with the summation on the right hand side bounded by O (cid:0) (log T ) 1 − γ e − (log T ) γ (cid:1) and γ ∈ (0 , 1 / 4) .
Now, summing up Eqs.
(4),(10),(12),(16),(17), we obtain the regret upper bound stated in Lemma 1.
By using Eqs (13) and (15), the expected incentive E [ B T ] can also be upper bounded by
E [ B T ]
[Equation]
g [Equation] where (i) follows from D ∗ ( t 2 ) ≥ D ∗ ( t 1 ) since t 2 ≥ t 1 , and (ii) follows from rearranging of the coefﬁcients containing g ( b, 1) .
A. P ROOF OF T HEOREM
Theorem 3.
(Arm-Independent Delay)
Under i.i.d.
delays with a ﬁnite expectation and the conditions of Lemma 1, the pseudo regret of Algorithm 1 E [ R T ] is upper bounded by (cid:20) 2 g ( b, 1)∆ ∗ g ( b, 1) − 1+ (cid:88) a (cid:54) =
a ∗ 8∆ a (cid:0) g ( b, 1) − 1 (cid:1) + 8∆ ∗ (cid:0) g
( b, 1) −
[Equation]
with the expected payment E [ B T ] upper bounded by
[Equation]
Combining the results in Lemma 1 and Lemma 2, we obtain that the regret is upper bounded by
E [ R T ] ≤ (cid:88)
[Equation]
Then by some straightforward rearrangements we obtain the regret upper bound in Theorem 3.
Similarly, by replacing E [ D ∗ ( T )] with its upper bound in the incentive upper bound, we obtain the rearranged incentive upper bound in Theorem 3.
A. P ROOF OF L EMMA
Lemma 4.
Under Assumption [Equation] given a ﬁnite number of arms K > 0 , it holds that
[Equation]
We consider a two-armed model with arm a and b , with arm-dependent delay distribution T a and T b .
Without loss of generality [Equation] we assume that E [ τ a, 1 ] ≥ E [ τ b, 1 ] .
Then, the worst case where D ∗ ( t ) is maximized is that one ﬁrst consecutively pull arm a for t a times, then consecutively pull arm b for the rest t − t a times.
Now, consider another two cases where arm a and arm b are pulled for t a times and t − t a times independently, we denote their maximum outstanding feedback up to time step t a and t − t a by D ∗ a ( t a ) and D ∗ b ( t − t a ) respectively.
We obtain the relationship below at time step t :
[Equation]
Now, by the result in Lemma [Equation] we obtain that
[Equation]
where (i) is by the result in Lemma 2; (ii) comes from the inequality 2 √ ab ≤ a + b ; and (iii) comes from the fact that t/ 2 is the maximizer of function h ( x )
= log x + log( t − x ) .
This result can be straightforwardly extended to multi-armed model with K ≥ 2 by the relationship D ∗ ( t ) ≤ (cid:80) i ∈A D ∗ i ( t i ) with (cid:80) i ∈A t i = t .
A. P ROOF OF T HEOREM
Theorem 5.
(Arm-Dependent Delay)
Under Assumption 1 and the conditions of Lemma 1, the pseudo regret of Algorithm 1 E [ R T ] is upper bounded by
[Equation]
with the expected payment E [ B T ] upper bounded by
[Equation]
Combining the results in Lemma 1 and Lemma
[Equation]
we obtain that the regret is upper bounded by
E [ R T ] ≤ (cid:88)
[Equation]
Thus we obtain the regret upper bound in Theorem 5.
Similarly, by replacing E [ D ∗ ( T )] with its upper bound in the incentive upper bound, we obtain the rearranged incentive upper bound in Theorem 5. B A PPENDIX II: E XPERIMENTS We provide the full details of the experiments here. B. B ASELINE C OMPARISONS
In this section, we compare our policy UCB-FDF with UCB-List in Zhou et al.
The system parameters are almost the same as that in Section 5.1.
We use the three-armed model with the initial preference bias θ = [1 , 5 , 5] .
The self-reinforcing preference function is chosen as f ( x ) =
x α with α = 2 .
The constant incentive for each time step is set as b = 1 . 5 with an incentive impact function g ( b, t )
=
b .
The experiment results are illustrated in Figure 2.
In Figures (a) and (b), we compare the performances under the no delay setting and the arm-dependent delay setting.
Speciﬁcally,
Figure (a) shows the performance under policy UCB-FDF in the face of no delay, and Figure (b) shows the performance under policy UCB-FDF in the face of arm-dependent delays, we choose the delay distribution as τ 1 ,t ∼ N (80 , 2) , τ 2 ,t ∼ N (10 , 2) , τ 3 ,t ∼ N (10 , 2) for arm1, arm2, arm3, respectively.
We only generate non-negative samples of delay under both assumptions.
In Figures (b) and (c), we compare the performances with policy UCB-FDF and UCB-List (Zhou et al., 2021).
Speciﬁcally, Figure (c) shows the performance under policy UCB-List in the face of arm-dependent delays that is the same as that in (b).
Each curve is constructed by regret or incentive values with different time horizons from T = 150 to T = 3000 , incremented by 150 .
Each node value in curves are averaged by 100 trials.
[Figure]
i
[Figure]
Discussion: Comparing (a) and (b) in Figure 2, we can observe the arm-dependent delay impact on regret and total incentive, that both the regret and total incentive are increased due to the delayed feedback.
Comparing (b) and (c) in Figure 2, we observe that under the bandit instances in the face of same delayed feedback, our policy UCB-FDF reaches sub-linear growth rate in both regret and total incentive, except the total incentive in category Pet Supplies, since it may require more time steps to converge while our data is limited, while the policy UCB-List cannot guarantee sub-linear growth rate for both regret and total incentive.
B. D ELAY D ISTRIBUTION C OMPARISONS
In this section, we compare performance of policy UCB-FDF under different delay distributions using the data in category Books.
The experiment results are illustrated in Figure 3.
(a) in Fig- ure 3 shows the different regret under different delay distributions, and (b) shows the corresponding total incentive.
For arm-independent delays, we show the policy performance under distribution
N (20 , 5) , N (40 , 5) , and Exp (0 . 1) , Exp (0 . 01) .
For arm-dependent delays, we choose the delay distribution as τ 1 ,t ∼ N (40 , 5) , τ 2 ,t ∼ Exp (0 . 01) , τ 3 ,t = 0 for arm1, arm2, arm3, respectively.
We also show the average exploration phases ( ¯ t 1 ) and average exploitation phases ( ¯ t 2 ) under four different settings of delay distribution: i) no delay; ii) arm-independent delay with normal distribution
N (40 , 5) ; iii) arm-independent delay with exponential distribution Exp (0 . 01) ; iv) arm-dependent delay with τ 1 ,t ∼ N (40 , 5) , τ 2 ,t ∼ Exp (0 . 01) , τ 3 ,t = 0 for arm1, arm2, arm3, respectively.
The results are shown in Table 2.
[Figure]
[Figure]
t
[Figure]
[Table]
Discussion:
From Figure 3, we observe that the policy performance is impacted by different delay distributions and different distribution parameters.
It aligns with our theoretical analysis that the case with no delay has the smallest regret
[Table]
and as delay expectation getting larger, we have larger expected regret and larger total incentive.
From Table
[Table]
we ﬁrstly have the general observation that as time horizon increases, both the exploration phase and the exploitation phase are increasing, due to a more accurate estimation over arms.
On the other hand, we observe that the length of exploration phase is problem-dependent, while the length of exploitation phase is positive proportional with the value of expected delay, which is due to our exploitation mechanism where dominance threshold grows with expected delay.
B. C OMPARISON WITH D IFFERENT P ARAMETERS
In this section
[Table]
we compare the performance of policy UCB-FDF under different system parameters using the data in category Books, speciﬁcally, we compare the impact of different strengths of feedback function f ( · ) and incentive impact function g ( · ) .
We use the setting f ( x ) =
x α with α =and g ( b, t ) =
b as the comparison group.
For the other four groups of experiment, we use α =and α = 4 for feedback function with g ( b, t )
=
b , and g ( b, t ) = 2 b and g ( b, t ) = 3 b for incentive impact function with α = 2 , respectively.
(a) in Figure 4 shows the different regret under different parameters, and (b) shows the corresponding total incentive.
[Figure]
Discussion:
For the feedback function f ( · ) , as α increases, the strength of self-reinforcing preferences is enhanced, which implies that the preferences are easier to be induces to one arm, so that we observe smaller regret.
For the incentive impact function g ( b, t )
=
c · b , as the coefﬁcient c increases, the preference impact of a unit payment is enhances, implying that users are easier to be affected (controlled) by the incentive, thus we observe a slightly smaller regret, with obviously decreased total incentive.