P ARTIAL I NFORMATION AS F ULL : R EWARD I MPUTATION WITH S KETCHING IN B ANDITS
Anonymous authors
Paper under double-blind review
A BSTRACT
We focus on the setting of contextual batched bandit (CBB), where a batch of rewards is observed from the environment in each episode.
But these rewards are partial-information feedbacks where the rewards of the non-executed actions are unobserved.
Existing approaches for CBB usually ignore the potential rewards of the non-executed actions, resulting in feedback information being underutilized.
In this paper, we propose an efﬁcient reward imputation approach using sketching in CBB, which completes the unobserved rewards with the imputed rewards approximating the full-information feedbacks.
Speciﬁcally, we formulate the reward imputation as a problem of imputation regularized ridge regression, which captures the feedback mechanisms of both the non-executed and executed actions.
To reduce the time complexity of reward imputation on a large batch of data, we use randomized sketching for solving the regression problem of imputation.
We prove that the proposed reward imputation approach obtains a relative-error bound for sketching approximation, achieves an instantaneous regret with a controllable bias and a smaller variance than that without reward imputation, and enjoys a sublinear regret bound against the optimal policy.
Moreover, we present two extensions of our approach, including the rate-scheduled version and the version for nonlinear rewards, which makes our approach more feasible.
Experimental results demonstrated that our approach can outperform the state-of-the-art baselines on a synthetic dataset, the Criteo dataset, and a dataset from a commercial app.
I NTRODUCTION
Contextual bandits have been widely used in real-world sequential decision-making problems (Li et al., 2010; Lan & Baraniuk, 2016; Yom-Tov et al., 2017; Yang et al., 2021), where the agent updates the decision-making policy fully online (i.e., at each step) according to the context and corresponding reward feedback so as to maximize the cumulative reward.
In this paper, we consider a more complex setting— contextual batched bandits (CBB), where the decision process is partitioned into N episodes, the agent interacts with the environment for B steps in one episode, collects the reward feedbacks and contexts at the end of the episode, and then updates the policy using the collected data for the next episode.
CBB is more practical in some real-world applications, since updating the policy once receiving the reward feedback is rather unrealistic due to its high computational cost and decision instability.
In bandit settings, it is inevitable that the environment only reveals the rewards of the executed actions to the agent as the feedbacks, while hiding the rewards of non-executed actions.
We refer to this category of limited feedback as the partial-information feedback (also called “bandit feedback”).
Existing batched bandit approaches in the CBB setting discard the information contained in the potential rewards of the non-executed actions, address the problem of partial-information feedback using an exploitation-exploration tradeoff on the context space and reward space (Han et al., 2020;
Zhang et al.,
2020).But in the CBB setting, the agent usually estimates and maintains reward models for the action-selection policy, and the potential rewards of the non-executed actions have been somehow captured by the policy.
This additional reward structure information is estimated and available in each episode, however, are not utilized by existing batched bandit approaches.
In contextual bandit settings where the policy is updated fully online, several bias-correction approaches have been introduced to address the partial-information feedback.
Dimakopoulou et al.
(2019) presented linear contextual bandits integrating the balancing approach from causal inference, which reweight the contexts and rewards by the inverse propensity scores.
Chou et al.
(2015) designed pseudo-reward algorithms for contextual bandits, which use a direct method to estimate the unobserved rewards for the upper conﬁdence bound (UCB) strategy.
Kim & Paik (2019) focused on the feedback bias-correction for LASSO bandit with high-dimensional contexts, and applied the doubly-robust approach to the reward modiﬁcation using average contexts.
Although these approaches have been demonstrated to be effective in contextual bandit settings, little efforts have been spent to address the under-utilization of partial-information feedback in the CBB setting.
Theoretical and experimental analyses in Section 2 indicate that better performance of CBB is achievable if the rewards of the non-executed actions can be received.
Motivated by these observations, we propose a novel reward imputation approach for the non-executed actions, which mimics the reward generation mechanisms of environments.
We conclude our contributions as follows
• To fully utilize feedback information in CBB, we formulate the reward imputation as a problem of imputation regularized ridge regression, where the policy can be updated efﬁciently using sketching.
• We prove that our reward imputation approach obtains a relative-error bound for sketching approximation, achieves an instantaneous regret with a controllable bias and a smaller variance than that without reward imputation, has a lower bound of the sketch size independently of the overall number of steps, enjoys a sublinear regret bound against the optimal policy, and reduces the time complexity from O ( Bd 2 ) to O ( cd 2 ) for each action in one episode, where B denotes the batch size, c the sketch size
and d the dimensionality of inputs , satisfying d < c < B .
• We present two practical variants of our reward imputation approach, including the rate-scheduled version in which the imputation rate is set without tuning, and the version for nonlinear rewards.
• We carried out extensive experiments on the synthetic data, public benchmark
and the data collected from a real commercial product to demonstrate our performance
, empirically analyzed the inﬂuence of different parameters, and veriﬁed the correctness of the theoretical results.
Related Work.
Recently, batched bandit has become an active research topic in statistics and learning theory including 2 -armed bandit (Perchet et al., 2016), multi-armed bandit (Gao et al., 2019;
Zhang et al., 2020; Wang & Cheng, 2020), and contextual bandit (Han et al., 2020; Ren & Zhou, 2020; Gu et al., 2021).
Han et al.
(2020) deﬁned linear contextual bandits, and designed UCB-type algorithms for both stochastic and adversarial contexts, where true rewards of different actions have the same parameters.
Zhang et al.
(2020) provided methods for inference on data collected in batches using bandits, and introduced a batched least squares estimator for both multi-arm and contextual bandits.
Recently, Esfandiari et al.
(2021) proved reﬁned regret upper bounds of batched bandits in stochastic and adversarial settings.
There are several recent works that consider similar settings to
CBB, e.g., episodic Markov decision process (Jin et al., 2018), LASSO bandits (Wang & Cheng, 2020).
Sketching is another related technology that compresses a large matrix to a much smaller one by multiplying a (usually) random matrix with certain properties (Woodruff, 2014), which has been used in online convex optimization (Calandriello et al., 2017; Zhang & Liao, 2019).
P ROBLEM F ORMULATION AND A NALYSIS
First, we introduce some notations.
Let [ x ] = { 1 , 2 , . . .
, x } , S ⊆ R d be the context space,
A =
{ A j } j ∈ [ M ] the action space containing M actions, [ A ; B ] =
[ A (cid:124) , B (cid:124) ]
(cid:124) , (cid:107) A (cid:107) F , (cid:107) A (cid:107) 1 (cid:107) A (cid:107) 2 denote the Frobenius norm, 1 -norm, and spectral norm of a matrix A , respectively, (cid:107) a (cid:107) 1 and (cid:107) a (cid:107) 2 be the (cid:96) 1 -norm and the (cid:96) 2 -norm of a vector a , σ min ( A ) and σ max ( A ) denote the minimum and maximum of the of singular values of A .
In this paper, we focus on the setting of Contextual Batched Bandits (CBB) (see Algorithm 1), where the decision process is partitioned into N episodes, and in each episode, CBB consists of two phases: 1) the policy updating approximates the optimal policy based on the received contexts and rewards; 2) the online decision selects actions for execution following the updated and ﬁxed policy p for B steps (also called the batch size is B ), and stores the contextaction pairs and the observed rewards of the executed actions into a data buffer D .
The reward R in
CBB is a partial-information feedback where rewards are unobserved for the non-executed actions.
Different from existing batch bandit setting (Han et al., 2020; Esfandiari et al., 2021), where the true reward feedbacks for all actions are controlled by the same parameter vector while the contexts received differ by actions at each step, we assume that in the CBB setting, the mechanism of true reward feedback differs by actions and the received context is shared by actions.
Formally, for any context s i ∈ S ⊆ R d and action A ∈ A , we assume that the expectation of the true reward R true i,A is determined by an unknown action-speciﬁc reward parameter vector
(cid:104) θ ∗ A ,
s i (cid:105) (the linear reward will be extended to the nonlinear case in Section 5).
This setting for reward feedback matches many real-world applications, e.g., each action corresponds to a different category of candidate coupons in coupon recommendation, and the reward feedback mechanism of each category differs due to the different discount pricing strategies.
Next, we provide deeper understandings of the inﬂuence of unobserved feedbacks on the performance of policy updating in the CBB setting.
We ﬁrst conducted an empirical comparison by applying the batch UCB policy (Han et al., 2020) to environments under different proportions of received reward feedbacks.
In particular, the agent under full-information feedback can receive all the rewards of the executed and non-executed actions, called Full-Information CBB (FICBB) setting.
From Figure 1, we can observe that the partial-information feedbacks could be damaging in terms of hurting the policy updating, and batched bandit could beneﬁt from more reward feedbacks, where the performance of 80% feedback is very close to that of FI-CBB.
[Figure]
policy (Han et al., 2020) under different proportions of received reward feedbacks, interacting with the synthetic environment in
Section 6, where x % feedback means that x % of actions can receive their true rewards
Then, we demonstrate the difference of instantaneous regrets between the CBB and FI-CBB settings as in
Theorem 1.
The detailed description and proof of
Theorem 1 can be found in Appendix A.
Theorem 1.
For any action A ∈ A and context s i ∈ S , let θ nA be the reward parameter vector estimated by the batched UCB policy in the nth episode.
The upper bound of instantaneous regret (deﬁned by |(cid:104) θ nA , s i (cid:105) − (cid:104) θ ∗ A , s i (cid:105)| ) in the FI-CBB setting is tighter than that in CBB setting (i.e., using the partial-information feedback).
From Theorem 1, we can conclude that the price paid for having access only to the partial-information feedbacks is the deterioration in the regret bound.
Ideally, the policy would be updated using the fullinformation feedback.
In CBB, however, the fullinformation feedback is unaccessible.
Fortunately, in
CBB, different reward parameter vectors need to be maintained and estimated separately for each action, and the potential reward structures of the non-executed actions have been somehow captured.
So why don’t we use these maintained reward parameters to estimate the unknown rewards for the non-executed actions?
Next, we propose an efﬁcient reward imputation approach that uses this additional reward structure information for improving the performance of the bandit policy.
E FFICIENT R EWARD I MPUTATION FOR P OLICY U PDATING
In this section, we present an efﬁcient reward imputation approach tailored for policy updating in the CBB setting.
Formulation of Reward Imputation.
Since the true reward parameters differ by actions in the CBB setting, we need to maintain a different estimated reward parameter vector for each action.
As shown in Figure 2, in contrast to CBB that ignores the contexts and rewards of the non-executed steps of
A j , our reward imputation approach completes the missing values using the imputed contexts and rewards, approximating the full-information CBB setting.
Speciﬁcally, in the ( n +1) -th episode, for
[Figure]
[Equation]
where r n,b ( A j ) :
= (cid:104)
¯ θ nA j , s n,b (cid:105) denotes the imputed reward for each step b ∈
[ (cid:98) N nj ] , and s n,b is the b -th row of (cid:98) S nA j .
Then, we obtain several block matrices by concatenating the context and reward matrices from the previous episodes: L nA j = [ S
A j ; · · · ; S nA j ] ∈ R L nj × d , T nA j =
[ R
A j ; · · · ; R nA j ] ∈
R L nj , L nj = (cid:80) nk =
N kj ,
[Equation]
L
[Equation] [Equation]
[Equation]
= (cid:80) nk =0 (cid:98) N kj .
In the ( n + 1) -th episode, the estimated parameter vector ¯ θ n +1 A j of the imputed reward for action A j can be updated by solving the following imputation regularized ridge regression :
[Equation]
where γ ∈ [0 , 1] is the imputation rate which controls the degree of reward imputation and measures a trade-off between bias and variance (Remark 1&2), λ > 0 is the regularization parameter, yielding
[Equation]
that can be obtained by the closed least squares solution, where Ψ n +1 A j :
[Equation]
and η ∈ (0 , 1) is the discount parameter which controls how fast the previous imputed rewards are forgotten, and can help guaranteeing the regret bound in Theorem 2.
Efﬁcient Reward Imputation using Sketching.
As shown in the ﬁrst 4 columns in Table 1, the overall time complexity of the imputation for each action is O ( Bd 2 ) in each episode, where B represents the batch size, and d the dimensionality of the input.
Thus, for all the M actions in one episode, reward imputation increases the time complexity from O ( Bd 2 ) of the approach without imputation to O ( MBd 2 ) .
To address this issue, we design an efﬁcient reward imputation approach using sketching, which reduces the time complexity of each action in one episode from O (cid:0)
Bd 2 (cid:1) to O (cid:0) cd 2 (cid:1) , where c denotes the sketch size satisfying d < c < B and cd > B .
Speciﬁcally, in the ( n + 1) -th episode, the imputation regularized ridge regression Eq.(1) can be approximated by a sketched ridge regression as
[Equation]
[Table] [Table]
[Table] [Table]
where ˜ θ n +1 A denotes the estimated parameter vector of the imputed reward using sketching for action
A ∈ A , C nA j ∈ R c × N n j and (cid:98) C nA j ∈ R c × (cid:98) N nj are the sketch submatrices for the observed term and the imputation term, respectively, and the sketch matrices for the two terms can be represented as
[Equation]
We denote the sketches of the context matrix and the reward vector by Γ nA j :=
C nA j S nA j ∈ R c × d and Λ nA j :=
C nA j R nA j ∈ R c , the sketches of the imputed context matrix and the imputed reward vector by (cid:98) Γ nA j := (cid:98) C nA j (cid:98) S nA j ∈ R c × d and (cid:98) Λ nA j := (cid:98) C nA j (cid:98) R nA j ∈ R c , and obtain the solution of Eq.(5):
[Equation]
6) where η ∈ (0 , 1) denotes the discount parameter, W n +1 A j :=
λ I d + G n
[Equation]
γ
[Equation]
G n +
[Equation]
j , and
[Equation] [Equation]
Using the parameter ˜ θ n +1 A j , we obtain the sketched version of imputed reward as ˜ r n,b
( A j ) :
= (cid:104) ˜
θ nA j , s n,b (cid:105) at step b ∈ [ (cid:98) N nj ] .
Finally, we specify that the sketch submatrices { C nA } A ∈A ,n ∈ [ N ] and { (cid:98) C nA j } A ∈A ,n ∈ [ N ] are the block construction of Sparser Johnson-Lindenstrauss Transform (SJLT)
(Kane & Nelson, 2014), where the sketch size c is divisible by the number of blocks D 1 .
As shown in the last 4 columns in Table 1, sketching reduces the time complexity from O ( MBd 2 ) to O ( Mcd 2 ) for reward imputation of all M actions in one episode, where c < B .
When Mc ≈ B , the overall time complexity of our reward imputation using sketching is even comparable to that without reward imputation which has a O ( Bd 2 ) time complexity.
Updated Policy using Imputed Rewards.
Inspired by the UCB strategy (Li et al., 2010), the updated policy for online decision of the ( n + 1) -th episode can be formulated using the imputed rewards (parameterized by ¯ θ n +1 A in Eq.(2)) or the sketched version of imputed rewards (parameterized by ˜ θ n +1 A in Eq.(6)).
Speciﬁcally, for a new context s , origin policy ¯ p n +1 selects the action following A ← arg max A ∈A (cid:104) ¯ θ n +1 A , s (cid:105)
+ ω
[ s (cid:124)
( Ψ n +1 A ) − 1 s ] 12 , sketched policy ˜
p n +1 selects the action following A ← arg max A ∈A (cid:104) ˜ θ n +1 A , s (cid:105) + α [ s (cid:124)
( W n +1 A ) − 1 s ] 12 , where ω ≥ 0 and α ≥ 0 are the regularization parameters in policy and their theoretical values are given in Theorem 4.
We summarize the reward imputation using sketching and the sketched policy into Algorithm 2, called SPUIR.
Similarly, we call the updating of the original policy that uses reward imputation without sketching, the Policy Updating with Imputed Rewards (PUIR).
[Table] [Table]
n
[Table]
j
[Table]
nA j ,
[Table]
nA
[Table] [Table]
W
[Table]
6) } 9: end for 10: ˜ p n
[Table]
s ) selects
[Table]
arg
[Table]
12 for
[Table]
context s 11: return { ˜ θ n
[Table]
1 A } A
[Table] [Table]
W n +1 A (cid:1) − 1 } A ∈A
T HEORETICAL A NALYSIS
We provide the instantaneous regret bound, prove the approximation error of sketching, and analyze the regret in the CBB setting.
The detailed proofs can be found in Appendix B.
We ﬁrst demonstrate the instantaneous regret bound of the original solution ¯ θ nA in Eq.(1).
Theorem 2 (Instantaneous Regret Bound) .
Let η ∈ (0 , 1) be the discount parameter, γ ∈ [0 , 1] the imputation rate.
In the n -th episode, if the rewards { R n,b } b ∈ [ B ] are independent 2 and bounded by
C R , then, for any b ∈ [ B ] and ∀ A ∈ A , with probability at least 1 − δ ,
[Equation]
b
[Equation]
−
[Equation]
θ ∗
[Equation]
s
[Equation]
(9) where Ψ nA =
λ I d + Φ nA + γ (cid:98) Φ nA , ν = [2 C 2 R log(2 MB/δ )] 12 , and C Imp > 0 .
The ﬁrst term on the right-hand side of Eq.(9) can be seen as the bias term for the reward imputation, while the second term is the variance term.
The variance term of our algorithm is not larger than that without the reward imputation, i.e, for any s ∈ R d , [ s (cid:124)
( Ψ nA ) − 1 s ] 12 ≤ [ s (cid:124)
( λ I d + Φ nA ) − 1 s ] 12 .
Further, a larger imputation rate γ leads to a smaller variance term [ s (cid:124) ( Ψ nA ) − 1 s ] 12 .
Remark 1 (Smaller Variance) .
From Theorem 2, we can observe that the proposed reward imputation achieves a smaller variance ( [ s (cid:124) n,b ( Ψ nA ) − 1 s n,b ] 12 ) than that without the reward imputation.
Remark 2 (Controllable Bias) .
Our reward imputation approach incurs a bias term γ 1 2 η − 1 2 C Imp in addition to the two bias terms λ (cid:107) θ ∗ A (cid:107) 2 , ν that exist in every UCB-based policy.
But this additional bias term is controllable due to the presence of imputation rate γ that can help controlling the additional bias.
Moreover, the term C Imp in the additional bias can be replaced by a function f Imp ( n ) , and f Imp ( n ) is monotonic decreasing w.r.t.
number of episodes n provided that the mild condition √ η = Θ( d − 1 ) holds (the deﬁnition and analysis about f Imp can be found in Appendix B.1).
Overall, the imputation rate γ controls a trade-off between the bias term and the variance term, and we will design a rate-scheduled approach for choosing the imputation rate γ in Section 5.
Although some approximation error bounds using SJLT have been proposed (Nelson & Nguyˆen, 2013; Kane & Nelson, 2014; Zhang & Liao, 2019), it is still unknown what is the lower bound of the sketch size while applying SJLT to the sketched ridge regression problem in our SPUIR.
Next, we prove the approximation error as well as the lower bound of the sketch size for the sketched ridge regression problem.
For convenience, we drop all the superscripts and subscripts in this result.
Theorem 3 (Approximation Error Bound of Imputation using Sketching) .
Denote the imputation regularized ridge regression function by F ( θ ) (deﬁned in Eq.(1)) and the sketched ridge regression function by F S ( θ ) (deﬁned in Eq.(5)) for reward imputation, whose solutions (i.e., the estimated reward parameter vectors) are ¯ θ = arg min θ ∈ R d F ( θ ) and ˜ θ = arg min θ ∈ R d F S ( θ ) .
Let γ ∈ [0 , 1] be the imputation rate, λ > 0 the regularization parameter, δ ∈ (0 , 0 . 1] , ε ∈ (0 , 1) , L all =
[ L ; √ γ (cid:98) L ] , and ρ λ = (cid:107) L all (cid:107)
/ ( (cid:107) L all (cid:107) 22 + λ ) .
If Π and (cid:98) Π are SJLT, assuming that D
=
Θ( ε − 1 log 3 ( dδ − 1 )) and the sketch size c = Ω (cid:0) d polylog (cid:0) dδ − 1 (cid:1) /ε
2 (cid:1) , with probability at least 1 − δ, F (˜ θ ) ≤ (1+ ρ λ ε ) F
(¯ θ ) and (cid:107) ˜
θ − ¯ θ (cid:107) 2 = O (cid:0) √ ρ λ ε (cid:1) hold.
To measure the convergence of approximating the optimal policy in sequential decision-making, we deﬁne the regret of SPUIR against the optimal policy as follows:
[Equation]
where I n,b denotes the index of the executed action using the sketched policy ˜ p n (parameterized by { ˜ θ nA } A ∈A ) at step b in the n -th episode.
Finally, we demonstrate the regret bound of SPUIR.
Theorem 4 (Regret Bound of SPUIR) .
Let T =
BN be the overall number of steps, η ∈ (0 , 1) be the discount parameter, γ ∈ [0 , 1] the imputation rate, λ > 0 the regularization parameter,
C max θ ∗ = max A ∈A (cid:107) θ ∗ A (cid:107) 2 , C Imp be a positive constant.
Assume that the conditional independence assumption in Theorem 2 holds and the upper bound of rewards is C R , M =
O (poly( d )) , T ≥ d 2 , ν
=
[2 C 2 R log(2 MB/δ 1 )] 12 with δ 1 ∈ (0 , 1) ,
[Equation]
where C α > 0 which decreases with increase of 1 /ε and ε ∈ (0 , 1) .
Let δ 2 ∈ (0 , 0 . 1] , ρ λ < 1 be the constant deﬁned in Theorem 3, and C reg be a positive constant that decreases with increase of 1 /ε .
For the sketch matrices { Π nA } A ∈A ,n ∈ [ N ] and { (cid:98) Π } A ∈A ,n ∈ [ N ] , assuming that the number of blocks in
SJLT D
=
Θ( ε − 1 log 3 ( dδ − 1 2 )) , and the sketch size satisfying c = Ω (cid:0) d polylog (cid:0) dδ − 1 2 (cid:1) /ε
2 (cid:1) , then, for an arbitrary sequence of contexts { s n,b } n ∈ [ N ] ,b ∈ [ B ] , with probability at least 1 − N ( δ 1 + δ 2 ) ,
Reg( { A I n,b } n ∈ [ N ] ,b ∈ [ B ] ) ≤ 2 αC reg
[Equation]
Remark 3.
Setting B
= O ( (cid:112) T/d ) and ρ λ (cid:15) =
1 /d yields a sublinear upper bound of regret of order (cid:101) O ( √ MdT ) 3 provided that the sketch size c
= Ω( ρ 2 λ d 3 polylog( dδ − 1 2 )) .
We can observe that the lower bound of c is independent of the overall number of steps T , and a theoretical value of the batch size is B = C B (cid:112) T/d = C 2 B N/d , where setting C B
≈ 25 is a suitable choice that has been veriﬁed in the experiments in Section 6.
In particular, when ρ λ = O (1 /d ) , the sketch size of order c = Ω
( d polylog d ) is sufﬁcient to achieve a sublinear regret, which has been demonstrated in our experimental results.
Since the lower bound of regret for contextual batched bandit in (Han et al., 2020) assumes that there are only two actions and both the actions share the same true reward model, it can not be applied to our CBB setting where each action corresponds to a different reward model.
Despite the lack of the lower bound in CBB setting, from the theoretical results of regret, we can observe that our SPUIR admits several advantages: (a) The order of our regret bound is not higher than those in the literature of contextual bandits in the fully-online setting (i.e., B = 1 ) (Li et al., 2019; Dimakopoulou et al., 2019), which is a more simple setting than ours; (b) The ﬁrst term in the regret bound Eq.(10) measures the performance of policy updating using imputed rewards (called “policy error”).
From Theorem 2 and Remark 1&2, we obtain that, in each episode, our policy updating has a smaller variance than the policy without the reward imputation, and incurs a decreasing additional bias under mild conditions, leading to a tighter regret (i.e., smaller policy error) after some number of episodes.
(c)
The second term on the right-hand side of Eq.(10) is of order
O ( T √ ρ λ (cid:15)d/B ) , which is incurred by the sketching approximation using SJLT (called “sketching error”).
This sketching error does not have any inﬂuence on the order of regret of SPUIR, which may even
have a lower order with a suitable choice of ρ λ ε , e
g., setting ρ λ ε = T − 1 / 4 d −yields a sketching error of order O ( T 3 / 8 d 1 / 2 ) provided that c =
Ω( ρ 2 λ d 3 polylog( dδ − 1 2 ) √ T ) .
E XTENSIONS OF O UR A PPROACH
To make the proposed reward imputation approach more feasible and practical, we tackle the following two research questions by designing variants of our approach following the theoretical results:
RQ1 (Parameter Selection): Can we set the imputation rate γ without tuning?
RQ2 (Nonlinear Reward): Can we apply the proposed reward imputation approach to the case where the expectation of true rewards is nonlinear?
Rate-Scheduled Approach.
For RQ1, we equip PUIR and SPUIR with a rate-scheduled approach, called PUIR-RS and SPUIR-RS, respectively.
From Remark 1&2, a larger imputation rate γ leads to a smaller variance while increasing the bias, while the bias term includes a monotonic decreasing function w.r.t.
number of episodes under mild conditions.
Therefore, we can gradually increase γ with the number of episodes, avoiding the large bias at the beginning of reward imputation.
Speciﬁcally, we set γ
= X % for episodes from ( X − 10)% ×
N to X % × N , where X ∈ [10 , 100] .
Application to Nonlinear Rewards.
For RQ2, we provide nonlinear versions of reward imputation.
We use linearization technologies of nonlinear rewards, rather than directly setting the rewards as nonlinear functions (Valko et al., 2013; Chatterji et al., 2019), avoiding the linear regret or curse of kernelization.
Speciﬁcally, instead of using the linear imputed reward ˜ r n,b ( A j ) :
= (cid:104) ˜
θ nA j , s n,b (cid:105) , we use the following linearized nonlinear imputed rewards, denotes by T n,b ( θ , A ) :
1) SPUIR-Exp.
We assume that the expected reward is an exponential function as G E ( θ , s ) =
exp ( θ (cid:124) s ) .
Then T n,b ( θ , A ) = (cid:104) θ , ∇ θ G E
( θ , s n,b ) (cid:105) , where ∇ θ G E ( θ , s n,b ) =
exp ( θ (cid:124) s n,b ) s n,b . 2) SPUIR-Poly.
When the
expected reward is a
polynomial
function as G P ( θ , s ) =
( θ (cid:124) s ) 2 .
Then
T n,
b (
θ , A ) =
(cid:104) θ , ∇ θ G P ( θ , s n,b ) (cid:105) , where ∇ θ G P ( θ , s n,b ) =
2 ( θ (cid:124) s n,b ) s n,b .
3) SPUIR-Kernel.
Consider that the underlying expected reward in a Gaussian reproducing kernel
Hilbert space (RKHS).
We use T n,b ( θ , A ) =
(cid:104) θ , φ ( s n,b ) (cid:105) in random feature space, where the random feature mapping φ can be explicitly deﬁned as in (Rahimi & Recht, 2007).
For SPUIR-Exp and SPUIR-Poly, combining the linearization of convex functions (Shalev-Shwartz, 2011) with Theorem 4 yields the regret bounds of the same order.
For SPUIR-Kernel, using the approximation error of random features (Rahimi & Recht, 2008), we can obtain a regret bound with an additional error of order O ( B/ √ d r ) , where d r is the dimension of the random features.
E XPERIMENTS
We empirically evaluated the performance of our algorithms on 3 datasets: the synthetic dataset, publicly available Criteo dataset 4 ( Criteo-recent , Criteo-all ), and dataset collected from a real commercial app for coupon recommendation ( commercial product ).
Experimental Settings.
We compared our algorithms with: Sequential Batch UCB (SBUCB) (Han et al., 2020), Batched linear EXP3 (BEXP3) (Neu & Olkhovskaya, 2020), Batched linear EXP using Inverse Propensity Weighting (BEXP3-IPW)
(Bistritz et al., 2019), Batched Balanced Linear
Thompson Sampling (BLTS-B) (Dimakopoulou et al., 2019), and Sequential version of Delayed
Feedback Model (DFM-S) (Chapelle, 2014).
We implemented all the algorithms on Intel(R)
Xeon(R)
Silver 4114 CPU@2.20GHz, and repeated the experiments 20 times.
We tested the performance of algorithms in streaming recommendation scenarios, where the reward is represented by a linear combination of the click and conversion behaviors of users.
According to Remark 3, we set the batch size as B = C 2 B N/d , the constant C B ≈ 25 , and the sketch size c = 150 on all the datasets.
The average reward was used to evaluate the accuracy of algorithms.
Performance Evaluation.
Figure 3(a)–(c) reports the average reward of SPUIR with its variants and the baselines.
We observed that SPUIR and its variants achieved higher average rewards, demonstrating the effectiveness of our reward imputation.
Moreover, SPUIR and its rate-scheduled version
SPUIR-RS had similar performances compared with the origin PUIR, which indicates the practical effectiveness of our variants and veriﬁes the correctness of the theoretical analyses.
The results on commercial product in Table 2 indicate that SPUIR outperformed the second-best baseline with the improvements of 1.07% CVR (conversion rate) and 1.12% CTCVR (post-view clickthrough&conversion rate).
Besides, our reward imputation approaches were more efﬁcient than
DFM-S, BLTS-
B.
The variants using sketching of our algorithms (SPUIR, SPUIR-RS) signiﬁcantly reduced the time costs of reward imputation, and took less than twice as long to run compared to the baselines without reward imputation (SBUCB, BEXP3, BEXP3-IPW).
Figure 3(d) illustrates performances of SPUIR and its nonlinear variants, where SPUIR-Kernel achieved the highest rewards indicating the effectiveness of the nonlinear generalization of our approach.
For different decision tasks, a suitable nonlinear reward model needs to be selected for better performances.
Parameter Inﬂuence.
From the regret bound Eq.(10), we can observe that a larger batch size B results in a larger ﬁrst term (of order O ( B ) , called policy error) but a smaller second term (of order
[Table]
[Figure]
Criteo-recent ; (f): SPUIR and SPUIR-RS with different sketch sizes on synthetic dataset
O (1 /B ) , called sketching error), indicating that a suitable batch size B needs to be set.
This conclusion was empirically veriﬁed in Figure 3(e), where B = 1 , 000 ( C B = 25 ) yields better empirical performance in terms of the average reward.
Similar phenomenon can also be observed on Criteo dataset and commercial product .
All of the results veriﬁed the theoretical results in Remark 3:
B = C B (cid:112) T/d = C 2 B N/d is a suitable choice while setting C B ≈ 25 .
From the results in Figure 3(f) we observe that, for our SPUIR and SPUIR-RS, the performances signiﬁcantly increased when the sketch size c reached 10% B ( ≈ d log d ), which demonstrates the conclusion in Remarkthat only the sketch size of order c = Ω
( d polylog d ) is needed for satisfactory performance.
C ONCLUSION
Partial-information feedback is ubiquitous in real-world applications, where reward feedback is usually underutilized for learning.
This paper proposes a theoretically sound and computationally efﬁcient reward imputation approach for contextual batched bandits, which mimics the reward generation mechanism of the environment approximating the setting of full-information feedback.
The proposed reward imputation approach reduces the time complexity of imputation on large batches of data using sketching, achieves a relative-error bound for sketching approximation, has an instantaneous regret with a controllable bias and a smaller variance, and enjoys a sublinear regret bound against the optimal policy.
The theoretical formulation and algorithmic implementation may provide an efﬁcient reward imputation scheme for online learning under limited feedback.
E THICS S TATEMENT
To verify the effectiveness and efﬁciency of our algorithms on real products, we conducted experiments on a real dataset collected from a commercial social app.
We call this dataset commercial product , where the data were collected after the users gave consent, and did not contain any personally identiﬁable information or offensive content.
R EPRODUCIBILITY S TATEMENT
The source code of the proposed algorithms is submitted as supplementary materials.
For theoretical results, clear explanations of any assumptions and a complete proof of the claims are included as an appendix.
The detailed experimental settings are also provided in the appendix.
Since the dataset commercial product is non-public, we did not provide a URL.
We will release this non-public dataset after the publication of this paper and the link to download this non-public dataset is to be included in the ﬁnal paper.
A D ETAILED P ROBLEM F ORMULATION AND D ETAILED P ROOF IN P ROBLEM A NALYSIS
In this part, we give the detailed problem formulation and the detailed proof of Theorem 1 in problem analysis in Section 2.
A. D ETAILED P ROBLEM F ORMULATION OF CBB
In this paper, we focus on the setting of contextual batched bandits (CBB), which can be formulated as a 6 -tuple (cid:104)S , A , p, R, N, B (cid:105) :
Context space S ⊆
R d means a vector space containing the context information received at each step, e.g., context summarizes the information of both the user and items in recommendation scenarios.
Action space A =
{ A j } j ∈ [ M ] contains M candidate actions for execution.
As an example, in recommender systems, each action corresponds to a candidate item, and selecting an action means that the corresponding item is recommended.
Policy p determines which action to take at each step, which is a function of the context s ∈ S and outputs an action for execution (or a selection distribution over action space A ).
Reward R in CBB is a partial-information feedback where rewards are unobserved for the nonexecuted actions.
Consider a stochastic bandit setting, where the expectation of the true reward is assumed to be a function of the context s ∈ S .
In particular, different from the shared expectation function of true rewards in existing batch bandits (Han et al., 2020), we assume that the expectation functions of true rewards are different for each action, where each expectation function corresponds to an unknown parameter vector θ ∗ A ∈ R d , A ∈ A .
This setting for rewards matches many realworld applications, e.g., each action corresponds to a different category of candidate coupons in coupon recommendation.
Number of episodes N .
The decision process in CBB is partitioned into N episodes.
Within one episode, the agent updates the policy using the collected data, and then interacts with the environment for multiple steps using the updated and ﬁxed policy.
Batch size B is the number of steps in each episode.
That is, in each episode, the agent interacts with the environment B times using a ﬁxed policy, and stores the contexts, executed actions, and observed rewards into a data buffer D at the end of each episode.
A. D ETAILED D ESCRIPTION AND P ROOF OF T HEOREM 1 IN P ROBLEM A NALYSIS
We present some theoretical ﬁndings about the regret difference between the partial-information feedback and the full-information feedback.
Assuming that the agent in the CBB setting can observe the rewards of all the candidate actions from the environment at each step, we apply the batched UCB policy (Han et al., 2020) to this setting (see Algorithm 3), We demonstrate an instantaneous regret bound in Theorem 5, where Theorem 5 is a detailed version of Theorem 1 in Section 2.
Theorem 5 (Instantaneous Regret Bound in Full-Information CBB Setting, Detailed Version of
Theorem 1) .
Let (cid:101)
L n − 1 ∈ R nB ×
d be the matrix that stores all the context vectors till the ( n − 1) -th episode as the row vectors, and (cid:101) T n − 1 A ∈ R
nB
be the reward vector that stores all the rewards of action
A ∈
A till the ( n − 1) -th episode.
Given the action space A
=
{ A j } j ∈ [ M ] , in the n -th episode, assume that the rewards are independent and bounded by C R .
Then, with probability at least 1 − δ , for any b ∈ [ B ] and ∀ A ∈ A , we have the following instantaneous regret bound in the n -th episode
[Equation]
where Υ n = (cid:101) L (cid:124) n − 1 (cid:101)
L n − 1 and the parameter of reward model θ nA in the batched UCB policy is obtained by
[Equation]
Further, the instantaneous regret bound Eq.(11) in FI-CBB setting is tighter than that in
CBB setting (i.e., using the partial-information feedback).
In particular, the variance term (cid:113) s
[Equation]
n,b ( I d + Υ n ) − 1 s n,b is smaller than that in CBB setting.
Proof of Theorem 1.
By the formulation of θ nA and the triangle inequality
[Equation]
we ﬁrst obtain that
[Equation]
Next, we bound the two terms in the last row of Eq.(12).
[Equation]
Since E (cid:104) (cid:101) T n − 1 A (cid:105) = (cid:101)
L n − 1 θ ∗ A and the received rewards are independent, by the Azuma-Hoeffding bound, we have
[Equation]
≤
[Equation]
where ν > 0 is some constant.
Since
[Equation]
combing with Eq.(13) implies the following results
[Equation]
Combing Eq.(14) with the union bound, yields that, with probability at least 1 − δ , for any b ∈ [ B ] and ∀ A ∈ A ,
[Equation]
where the failure probability is
[Equation]
yielding that ν
= (cid:112) 2 C 2 R log(2 MB
[Equation]
δ ) .
Bounding
[Equation]
Since Υ n is positive semi-deﬁnite, combining with the H¨older inequality, we obtain
[Equation]
Combing Eq.(15) and Eq.(16) concludes the proof.
Similarly to the proof of Eq.(29), we can obtain that the variance term in full-information setting is smaller than that in partial-information setting.
B D ETAILED P ROOFS IN T HEORETICAL A NALYSIS
In this section, we provide the instantaneous regret bound in each episode, prove the approximation error of sketching, and analyze the regret for policy updating in the CBB setting.
Figure 4 describes the dependence structure of our theoretical results.
[Figure]
B. P ROOF OF T HEOREM
Before we provide the detailed proof of Theorem 2, we ﬁrst demonstrate a lemma about the convergence and monotonicity of the sum of functions, which is the main tool for analyzing the additional bias of reward imputation.
Lemma 1 (Convergence and Monotonicity) .
Let f ( n ) =
(cid:80) nj
=1 a n − j
· g ( j ) , where a ∈ (0 , 1) and n is a positive integer.
Then,
1) when g ( j ) is convergent, the limit lim n →∞ f ( n ) exists.
Moreover,
[Equation]
2) f ( n ) is a monotonic decreasing function if and only if g ( j ) satisﬁes, for any positive integer
j
[Equation] [Equation]
Proof.
Letting b ( j ) =
a − j · g ( j ) , ∀ j ∈ [ n ] , and S ( n ) = (cid:80) nj =1 b ( j ) , f ( n ) can be rewritten as f ( n ) = a n S ( n ) .
1) Rewriting f ( n ) = S ( n ) /a − n , from the Stolz’s theorem, we have
[Equation]
The condition that
f ( · ) is a monotonic decreasing function is equivalent to the following condition: for and positive integer n ,
[Equation]
From the equivalent condition Eq.(19), we obtain the following recursion formula:
[Equation]
yielding that, for any positive integer j ≥ 2 ,
[Equation]
From Eq.(20) , for a (cid:54) = 1 / 2 ,
[Equation]
and substituting the deﬁnition of b ( j ) into Eq.(21) yields the equivalent condition
[Equation]
For a = 1 / 2 , we have the condition b ( j ) ≤ ( j − 1) b (1) , which is equivalent to
[Equation]
Next, we provide the detailed proof of Theorem 2.
Proof of Theorem 2.
From the formulation of ¯ θ nA and the triangle inequality, we can obtain that, for each action A ∈ A ,
[Equation]
Next, we bound X (1) A , X (2)
A
[Equation]
and X (3) A .
For convenience, we drop all the superscripts and subscripts about n and b .
Similarly to the proof of Theorem 1, we bound X (
1) A + X (2) A as follows: with probability at least 1 − δ ,
[Equation] [Equation]
where ν = (cid:112) 2 C 2 R log(2 MB/δ ) .
For X (3) A , using the Cauchy-Schwarz inequality
[Equation]
we have
[Equation]
Now we need to bound the term (cid:13)(cid:13)(cid:13)
[Equation]
Since using the discount parameter η ∈ (0 , 1) in Eq.(4) is equivalent to multiplying both the imputed contexts and the imputed rewards by the parameter √ η in each episode, we have, in the n -th episode,
[Equation]
[Equation]
where ∆ ηn − 1 = { η ( n − i − 1) /
IR i,b } i ∈ [ n − 1] ,b ∈ [ B ] denotes an exponential-decay vector of the instantaneous regrets, and IR i,b denotes the instantaneous regret at step b in the i -th episode, i.e,
IR i,b =
[Equation] [Equation]
b
[Equation]
−
[Equation]
A , s i,b (cid:105) (cid:12)(cid:12) .
From Eq.(24), letting
[Equation]
be the cumulative instantaneous regret in the i -th episode, we can obtain the upper bound of Eq.(24) as follows:
[Equation]
where
[Equation]
From monotone bounded theorem, we have that the limit of CIR i exists.
From Eq.(17) in Lemma 1, we get that f Imp ( n ) is convergent and then has an upper bound.
We denotes the upper bound of f Imp ( n ) by C Imp > 0 , and then from Eq.(26) we have
[Equation]
Substituting Eq.(28) into Eq.(23) yields the upper bound of X (3) A .
Then, we prove that
[Equation] [Equation]
holds, which is equivalent to
[Equation]
Letting Θ
= λ I d + Φ , by Sherman-Morrison-Woodbury formula, we have
[Equation]
yielding that Eq.(30) is equivalent to
[Equation] [Equation]
where
[Equation]
Let S =
U d Σ 1 / 2 d V (cid:124) d , (cid:98) S =
(cid:98) U d (cid:98) Σ
1 / 2 d (cid:98) V (cid:124)
d be the Singular Value Decomposition (SVD) of S and (cid:98) S , respectively.
Note that Φ = V d Σ d V (cid:124) d , (cid:98) Φ = (cid:98) V d (cid:98) Σ d (cid:98) V (cid:124) d .
We can obtain that Γ is a square symmetric positive semi-deﬁnite matrix, since Γ can be decomposed into
[Equation]
where P γ Λ γ P (cid:124) γ is the SVD of I
d /γ + (cid:98) S Θ − 1 (cid:98) S
[Equation]
and
[Equation]
Thus, Eq.(32) holds, yielding that Eq.(30) also holds.
Finally, we prove that a larger imputation rate γ leads to a smaller variance term (cid:113) s (cid:124)
( Ψ ) − 1 s .
From Eq.(31), the variance term can be represented as follows:
[Equation]
where M γ = I d /γ + (cid:98) S Θ − 1 (cid:98) S (cid:124) .
Letting M γ
= U M γ Λ M γ U (cid:124)
M γ be the SVD of M γ , and z =
U
[Equation]
M γ (cid:98) S Θ − 1 s , from Eq.(33) we can written the variance term as follows:
[Equation] [Equation]
In Eq.(34), we can observed that
[Equation]
where M = (cid:98) S Θ − 1 (cid:98) S (cid:124) , which indicates that a larger imputation rate γ leads to a smaller variance term.
Finally, we provide a deeper understanding of the additional bias in Theorem 2.
Remark 4 (Controllable Bias) .
Our reward imputation approach incurs a bias term γ 12 η − 12 C Imp in addition to the two bias terms λ (cid:107) θ ∗ A (cid:107) 2 and ν that exist in every UCB-based policy.
But this additional bias term is controllable due to the presence of imputation rate γ that can help controlling the additional bias.
Moreover, from the proof of Eq.(26), we can obtain that, the term C Imp in the additional bias can be replaced by a function
f Imp ( n ) (deﬁned in Eq.(27)), and the additional bias term turns out to be γ 12 η − 12 f Imp ( n ) .
Since f Imp ( n ) has the same functional form as the function f ( n ) in Lemma 1, we can ﬁnd the conditions that f Imp ( n ) is monotonic decreasing following Eq.(18) in Lemma 1.
Speciﬁcally, letting CIR i be the cumulative instantaneous regret in the i -th episode deﬁned in Eq.(25),
1) when √ η (cid:54) = 1 / 2 , the condition of a monotonic decreasing function f Imp ( · ) is equivalent to, for any positive integer i ≥ 2 ,
[Equation]
indicating that the regret after N episodes satisﬁes
[Equation]
2) for the case √ η = 1 / 2 , the condition of a monotonic decreasing function f Imp ( · ) is equivalent to CIR i ≤ ( i − 1)( √ η ) i − 1 CIR 1 for any positive integer i ≥ 2 , indicating that the regret after
N episodes satisﬁes
[Equation]
From Eq.(35) and Eq.(36), we can conclude that a monotonic decreasing function f Imp ( · ) indicates the upper bound of regret after N episodes is of order O (CIR 1 / √ η ) .
The conclusion also indicates that setting the discount parameter as √ η = Θ(CIR 1 /N ) achieves a O ( N ) regret bound (i.e., a ˜ O ( √ dT ) regret bound following Remark 3).
Note that setting the discount parameter as √ η = Θ(CIR 1 /N ) is a mild condition, since the cumulative instantaneous regret CIR 1 is typically of order O ( B ) ( B = O ( (cid:112) T/d ) in Remark 3) yielding that √ η = Θ( d − 1 ) .
Overall, since a larger imputation rate γ leads to a smaller variance while increasing the bias (variance analysis can be found in Remark 2), γ controls a trade-off between the bias term and the variance term.
When f Imp is a monotonic decreasing function w.r.t.
number of episodes n , the additional bias term γ 12 η − 12 f Imp ( n ) can be easily controlled, e.g., gradually increasing γ with the number of episodes, avoiding the large bias from f Imp ( n ) at the beginning of reward imputation.
We design a ratescheduled approach for choosing the imputation rate γ in Section 5.
Remark 5 (Relationship to Exploration and Exploitation Trade-off) .
Exploration-exploitation dilemma is the key challenge in online learning under bandit settings.
In the full-information setting, agent (e.g., UCB policy) receives the rewards from all the actions, does not need to consider the choice of exploring the feedback mechanisms, and achieves a lower variance part in the regret upper bound (Theorem 1).
Along this line, our reward computation approach is proposed to approximate the setting of full-information feedback, which somewhat relaxes the explore/exploit dilemma and also brings a lower variance part and a controllable additional bias part in the regret.
Extra information that pushes the policy towards exploitation and away from exploration comes from the estimated reward structures of the non-executed actions maintained in each episode, and the proposed reward imputation can be seen as an effective and efﬁcient tool to capture this extra information.
B. A PPROXIMATION P ROPERTIES OF S KETCHING
Although some error bounds of approximation using SJLT have been proposed (Nelson & Nguyˆen, 2013; Kane & Nelson, 2014; Bourgain et al., 2015), it is still unknown what is the lower bound of the sketch size while applying SJLT to the sketched ridge regression problem in our SPUIR.
To address this issue, we ﬁrst prove two approximation properties of SJLT which are necessary to achieve approximation error bound of the sketched ridge regression using SJLT.
For convenience, we drop all the superscripts and subscripts in these theoretical results.
Lemma 2 ((Nelson & Nguyˆen, 2013)) .
Let U ∈ R L × d be a matrix with orthonormal columns, Π ∈ R c
× L the SJLT.
Assuming that D
= Θ( ε − 1 σ log 3 ( dδ − 1 0 ))
for Π , ε σ ∈ (0 , 1) and d ≤ c, with probability at least 1 − δ 0 all singular values of Π U
[Equation]
as long as
[Equation]
Further, this holds if the hash function h and σ deﬁning the Π is Ω (cid:0) log( dδ − 1 0 ) (cid:1) -wise independent.
Theorem 6 (Approximation Properties of SJLT) .
Let U ∈ R L × d be a matrix with orthonormal columns, and A be a matrix of any proper size.
If Π ∈ R c × L is the SJLT satisfying the assumptions in Lemma 2, and d ≤ c ≤ L , then Π has the following two properties:
1) Subspace embedding property: set c
= Ω (cid:0) d polylog (cid:0) dδ − 1 s /ε 2s (cid:1) , for ε s ∈ (0 , 1) , with probability at least 1 − δ s ,
[Equation]
2) Matrix multiplication property: set c =
Ω( d/ ( ε m δ m )) , for ε m ∈ (0 , 1) , with probability at least [Equation] m ,
[Equation]
Proof of Theorem 6. 1) From Lemma 2, by setting c = Ω (cid:0) d polylog (cid:0) dδ − 1 s (cid:1) /ε 20 (cid:1) , we can obtain the upper bounds of eigenvalues: with probability at least 1 − δ s , λ i ( U [Equation] Π
[Equation]
which yields that
[Equation]
Eq.(39) is equivalent to
[Equation]
Letting ε s = 3 ε σ and ε σ ∈ (0 , 1 / 3) yields the subspace embedding property.
2) From Lemma 1 in (Zhang & Liao, 2019), we have
[Equation]
Combining Eq.(39) with the Markov’s inequality, we obtain that, with probability at least 1 − δ m ,
[Equation]
Letting ε m = 2 d δ m c yields the matrix multiplication property.
B. P ROOF OF T HEOREM
Next, using the approximation properties of SJLT in Theorem 6, we prove that the objective function value of the imputation regularized ridge regression problem for reward imputation can be approximated well with a relative-error bound.
Moreover, we prove that the solution solving the sketched ridge regression problem for reward imputation is also a good approximation of the solution solving the imputation regularized ridge regression.
The following theorem is a detailed version of Theorem 3.
Theorem 7 (Approximation Error Bound of Imputation using Sketching, Detailed Version of Theorem 3) .
Let γ ∈ [0 , 1] be the imputation rate, λ > 0 the regularization parameter, Π ∈ R c × L and (cid:98) Π ∈ R c × (cid:98) L be the SJLT, and L ∈ R L × d , (cid:98) L ∈ R (cid:98)
L × d , T ∈ R L , (cid:98) T ∈ R (cid:98) L , θ ∈ R d .
Denote the imputation regularized ridge regression function F and sketched ridge regression function F S for reward imputation by
[Equation]
and the solutions of these regression problems by
[Equation]
Let δ ∈ (0 , 0 . 1] , ε ∈ (0 , 1) , {+ρ λ = (cid:
+}
) L all (cid:107) 22 / ( (cid:107) L all (cid:107) 22 + λ ) .
For Π and (cid:98) Π , assuming that D = Θ( ε − 1 log 3 ( dδ − 1 )) and c = Ω (cid:0) d polylog (cid:0) dδ − 1 (cid:1) /ε 2 (cid:1) , with probability at least 1 − δ,
[Equation]
where L λ all = (cid:104) L ; √ γ (cid:98) L ; √ λ I d
(cid:105) ∈
R ( L + (cid:98) L + d ) × d .
Furthermore, if there is a constant fraction of the norm of T 0all lies in the column space of L λ all , then Eq.(41) can be strengthened.
Formally, assuming that a mild structural assumption on the context matrix and the reward vector is satisﬁed, i.e., (cid:107) U all U (cid:124) all T 0all (cid:107) 2 ≥ ξ (cid:107) T 0all (cid:107) 2 with a constant ξ ∈ (0 , 1] , then with probability at least 1 − δ,
[Equation]
[Equation]
where κ ( A ) denotes the condition number of A , T 0all =
[ T ; (cid:98) T ; 0 d ] ∈ R ( L + (cid:98) L + d ) , and L λ all =
U all Σ all V (cid:124) all is the SVD of L λ all .
Proof of Theorem 7.
We ﬁrst introduce some more notation of block matrices that will simplify the proof of the theorem:
[Equation]
Then the regression functions can be rewritten as follows:
[Equation]
Obviously, Π all is still an SJLT.
Combining Theorem 6 with theorem 19 in (Wang et al., 2017), we can obtain, setting
[Equation]
with probability at least 1 − ( δ s + δ m ) ,
[Equation]
44) where ρ λ = (cid:107) L all (cid:107) 22 (cid:107) L all (cid:107)
+ λ and τ =
2max { ε 2s ,ε m } 1 − ε s .
Letting ε s = ε m := ε 0 , Eq.(44) can be rewritten as
[Equation]
Assuming that δ s = δ m := δ/ 2 ∈ (0 , 0 . 1] and ε 0 ∈ (0 , 1 / 3) , setting (cid:15) = 2 ε 0 1 − ε 0 ∈ (0 , 1) , from
Eq.(45) we obtain the upper bound Eq.(40).
Next, we bound the difference between the solutions solving the sketched ridge regression problem and the original regression problem.
Since σ 2min ( A ) (cid:107) x (cid:107) 22 ≤ (cid:107) Ax (cid:107) 22 for any A and x with proper sizes, we have
[Equation]
22 ≤ (cid:13)(cid:13)(cid:13) L all (˜ θ − ¯ θ ) (cid:13)(cid:13)(cid:13) 2 2 . The key ingredient of bounding (cid:107) ˜ θ − ¯ θ (cid:107) 2 is to bound (cid:107) L all (˜ θ − ¯ θ ) (cid:107) 2 .
Let L λ all = (cid:104) L ; √ γ (cid:98) L ; √ λ I d (cid:105) ∈
R ( L + (cid:98) L + d ) × d , T 0all =
[ T ; (cid:98) T ; 0 d ] ∈ R ( L + (cid:98) L + d ) , L λ all =
U all Σ all V (cid:124) all be the SVD of L λ all , and denote a matrix with orthonormal columns by U ⊥
all ∈ R ( L + (cid:98) L + d ) × ( L + (cid:98) L ) which satisﬁes
[Equation]
and
U (cid:124) all U ⊥ all = O .
Then, we can rewrite the solution ¯ θ as follows:
[Equation]
which yields that
[Equation]
Thus, T 0all − L λ all ¯ θ is orthogonal to U all , and consequently to L λ all (˜ θ − ¯ θ ) , and we can obtain the following equality by Pythagoras’s theorem:
[Equation]
Combining Eq.(48) with Eq.(40) yields that
[Equation]
F
[Equation]
Substituting Eq.(49) into Eq.(46) concludes the proof of Eq.(41).
If we make a mild structural assumption on the context matrix and the reward vector, we can provide a stronger bound of (cid:107) ˜ θ − ¯ θ (cid:107) 2 .
Speciﬁcally, assuming that (cid:107) U all U (cid:124) all T 0all (cid:107) 2 ≥ ξ (cid:107) T 0all (cid:107) 2 with a constant ξ ∈ (0 , 1] , from Eq.(47) and Pythagoras’s theorem we have
[Equation]
Combining Eq.(50) with Eq.(41) yields Eq.(42).
B. P ROOF OF T HEOREM
Proof of Theorem 4.
In our sketched policy [Equation] letting C max θ ∗
[Equation]
A ∈A
[Equation]
θ ∗ A
[Equation] 2 , C Imp > 0 , ν = (cid:112) 2 C 2 R log(2 MB/δ ) [Equation] and
[Equation]
Imp , from Eq.(9) in Theorem 2 we obtain that
[Equation]
Before proving the upper bound of [Equation] we need to provide a technical tool as follows.
For convenience, we also drop all the superscripts and subscripts.
The goal is to ﬁnd a constant C α such that
[Equation]
52) which is equivalent to the condition that the matrix
C 2 α W − 1 − Ψ − 1 is positive semideﬁnite.
Let L all and Π all be the matrices deﬁned in Eq.(43), L all =
(cid:101) U all (cid:101) Σ all (cid:101) V (cid:124) all be the SVD of L all , and ˜ σ 1 ≥ ˜ σ 2 · · · ≥ ˜ σ
d be the singular values of L all .
Then the i -th eigenvalue of Ψ − 1 =
( λ I d + L (cid:124) all L all ) − 1 can be represented as λ i ( Ψ − 1 )
= 1 / (˜ σ 2 i + λ ) , and the i -th eigenvalue of
W − 1 =
( λ I d + L (cid:124) all Π (cid:124) all Π all L all ) − 1 is λ i ( W − 1 )
=
1 / (ˆ λ i + λ ) , where ˆ λ i is the i -th eigenvalue of (cid:101) Σ all (cid:101)
U
(cid:124)
all Π (cid:124) all Π all (cid:101) U all (cid:101) Σ all .
From the Lidskii’s theorem and Eq.(37), we have
[Equation]
Assuming that the positive semi-deﬁniteness of C 2 α W − 1 − Ψ − 1 is satisﬁed, we obtain that
C
α λ i ( W − 1 ) − λ i ( Ψ − 1 ) ≥ 0 for i ∈
[ d ] , and combining this inequality with Eq.(53) yields that
[Equation]
From the proof of Theorem 6 and Theorem 3, we can obtain that ε σ = ε/ (6 + 3 ε ) , yielding that
[Equation]
which decreases with increase of 1 /ε .
Similarly to the proof of C α satisfying Eq.(52), we can obtain that
[Equation]
provided that
[Equation]
Obviously, C reg also decreases with increase of 1 /ε .
Then, letting α = ωC α , from Eq.(51) and Eq.(52) we have
[Equation]
where Y n,b denotes the upper bound of (cid:12)(cid:12)(cid:12)(cid:68) ˜ θ nA − ¯ θ nA , s n,b (cid:69)(cid:12)(cid:12)(cid:12) for any A ∈ A .
Next, using the compatibility of norm, we give a speciﬁc representation of the sum of Y n,b as follows:
[Equation]
Further, we give a more speciﬁc upper bound in Eq.(56) under mild structural assumption in Theorem 3.
Let κ maxall denote the maximum of the condition numbers of L λ all ( A, n ) for A ∈ A , n ∈ [ N ] , and L λ all ( A, n )
= (cid:104)
L nA ; √ γ (cid:98) L nA ; √ λ I d (cid:105) , and U all ( A, n ) be the left singular matrix of L λ all ( A, n ) .
Letting T 0all ( A, n )
=
[ T nA ; (cid:98) T nA ; 0 d ] , assuming that (cid:107) U all ( [Equation]
≥ ξ (cid:107) T 0all ( A, n ) (cid:107) 2 with a constant ξ ∈ (0 , 1] , substituting the upper bound Eq.(42) in Theorem 3 into
Eq.(56) yields that
[Equation]
where C S = max n ∈ [ [Equation] [Equation] (cid:107)
[Equation]
¯ θ
[Equation] (cid:107) [Equation] From Eq.(54), Eq.(55), Eq.(57) and the deﬁnition of our sketched policy, letting C Y =
[Equation]
, we obtain that
[Equation] [Equation]
When the structural assumption in Theorem 3 is not satisﬁed
[Equation]
from Eq.(41)
[Equation]
we can obtain that the second term in Eq.(58) is also of order O ( √
ρ λ (cid:15)d ) , which does not inﬂuence the order of the ﬁnal regret bound.
Finally, combining Eq.(58) with lemma 3 in (Han et al., 2020) gives the ﬁnal regret bound.
C D ETAILED E XPERIMENTAL S ETTINGS AND M ORE E XPERIMENTAL R ESULTS
In this section, we provide more details and results in the experiments.
C. D ESCRIPTION OF D ATASETS
Table 3 summarizes the description of datasets used in the experiments.
Next, we provide more details about the three datasets.
Synthetic Data.
Inspired by the experiments in (Saito et al., 2020), the synthetic data generation procedure was formulated as follows, which simulates the streaming recommendation environment.
• Context s i ∈ R
d : we drew elements of s i independently from a Gaussian distribution N (0 . 1 , 0 . 2 2 ) , where d = 40 ; • Click-Through-Rate (CTR): the CTRs for the 5 actions were respectively set as { 10% , 15% , 25% , 20% , 30% } ;
[Table]
• The indicator variables of click events:
[Equation]
We sampled the click index set according to the uniform distribution.
• Conversion rate (CVR) in context s i :
when C i = 1 ,
[Equation]
where the coefﬁcient vector w c ∈ R d is sampled according to a Gaussian distribution as w c ∼ N ( κ c 1 d , σ 2c I d ) , and we set different means and standard deviations for different action with κ c ∈ [0 : − 0 .
2 : − 0 . 8] and σ c ∈ [0 .
+0 .
Criteo Data.
We used the publicly available Criteo dataset 5 , consisting of Criteo’s trafﬁc on display ads over a period of two months (Chapelle, 2014), where each context consists of 8 integer features and 9 categorical features.
Following the experiments in (Yoshikawa & Imai, 2018), the categorical features were represented as one-hot vectors and then concatenated to the integer features.
We reduced the dimensionality of the feature vectors to 50 using principal component analysis (PCA).
All of the algorithms were tested in a simulated online environment that was trained on users’ logs in the Criteo dataset.
Speciﬁcally, we chose several campaigns from the Criteo dataset, where each campaign represents a category of items and corresponds to an action.
This online environment contains a prediction model for the CVR, which was well trained by applying DFM (Chapelle, 2014) using the true user feedbacks.
This environment model was trained for each chosen campaign, whose AUCs are ranging from 70% to 90% , assuring that the online environment can provide nearly realistic feedbacks.
To simulate the uncertainty of user behaviors, Gaussian noises with zero-mean were added to the model parameters.
At each step, the online environment randomly selected a campaign and samples one context from this campaign, and revealed the context to the agent with a preset CTR.
To generate a reasonable sequence of instances, the environment kept the order of timestamps of the contexts in each campaign.
We tested our algorithms and the baselines with the following two online environments on the Criteo dataset: Criteo-recent contains 5 campaigns ( 75 , 000 instances) chosen from the recent campaigns, corresponding to 5 actions; Criteo-all contains 15 campaigns ( 1 , 276 , 000 instances) chosen from all the campaigns, corresponding toactions.
Data Collected from a Real Commercial App for Coupon Recommendation.
To verify the effectiveness and efﬁciency of our algorithms on real products, we conducted experiments on a real dataset collected from a commercial social app.
We call this dataset commercial product , where the data were collected after the users gave consent, and did not contain any personally identiﬁable information or offensive content.
Since this dataset from a commercial app is proprietary, we did not provide a URL.
We will release this dataset after the publication of this paper.
In this commercial app, after clicking a recommended coupon, a user may convert the coupon after some time, or just leave it there.
The dataset was collected during a 1-month period with a subsampling, and consists of 216 , 568 instances from 5 categories of coupons, where each context is described by 86 numerical features and 16 categorical features.
The timestamps of clicks and conversions were also recorded.
Following the settings on the Criteo data, we also represented the categorical features as a one-hot vector, reduced the dimensionality of the feature vectors to 50 by PCA.
The action space contains 5 actions, where each corresponding to one coupon category.
Due to the limitation of real online experiments, in this experiment, we still trained DFM using the true user feedbacks as the online environment, where AUCs range from 75% to 90% .
To simulate the real environment under partial-information feedback, The experiments were conducted in environments where the distribution of the initialization data is atypical.
Speciﬁcally, in the experiments, we set different numbers of the initialization instances for each action. In the synthetic environment, we set the number of the initial instances as 140 , 210 , 350 , 280 ,for the 5 actions, respectively. In Criteo-recent , we set the proportion of the initial instances as 0 . 1 , 0 . 15 , 0 . 3 for the 5 actions, and set the number of the initial instances as [100 : 23 : 423] for the 15 actions in Criteo-all .
C. D ETAILED S PECIFICATION OF H YPERPARAMETERS
In these experiments, the true reward is deﬁned by R true i,A = λ c C i,A
+ (1 − λ c ) V i,A ( C i,A and V i,A denote true binary variables of user click and conversion when executing action A given context s i ), where λ c = 0 .
01 on the synthetic data, Criteo Data, and commercial product data, respectively.
As in most contextual bandit literature (Li et al., 2010; Chu et al., 2011), we set the regularization parameter λ
= 1 in the Euclidean regularization.
According to theoretical analysis in Remark 3, we set the batch size as B = C 2 B N/d , set the constant C B ≈ 25 and the sketch size c = 150 on all the datasets ( B = 1400 , 1000 , 4000 , 1700 for synthetic data , Criteo-recent , Criteo-all , and commercial product ).
The regularization parameters ω , α in our policy and that in the batch
UCB policy were tuned in [0 . 2 :
+0 . 2 : 1 . 2] .
For the SJLT in SPUIR and its variants, sketch size was set as c = 150 and the number of block D was selected in { 1 , 2 , 4 , 6 } .
Except for the rate-scheduled variants of our approaches, the imputation rate γ was selected in [0 . 1 : +0 . 2 : 0 . 9] .
Besides, the discount parameter η was tuned in [0 . 1 :
+0 . 2 : 0 . 9] .
In the nonlinear variant of our approach SPUIR-Kernel, we selected the dimension of the random features d r in { 50 , 100 , 200 } and the kernel width of Gaussian kernel in { 2 − ( i +1) / 2 , i =
[ − 12 : 2 : 12] } .
Rate-Scheduled Approach. We equip PUIR and SPUIR with a rate-scheduled approach, called PUIR-RS and SPUIR-RS, respectively. We design a rate-scheduled approach following the theoretical results about the imputation rate γ .
From Remark 1&2, we can obtain that a larger imputation rate γ leads to a smaller variance while increasing the bias.
From Remark 4, we conclude that the additional bias term includes a monotonic decreasing function w.r.t.
number of episodes under mild conditions.
Therefore, instead of using a ﬁxed imputation rate, we can gradually increase γ with the number of episodes, avoiding the large bias at the beginning of the reward imputation while achieving a small variance.
Speciﬁcally, we set γ
= X % for episodes from ( X − 10)% ×
N to X % × N , where X ∈ [10 , 100] .
C. M ORE E XPERIMENTAL R ESULTS
For better illustration, in Figure 3 of the manuscript, we omitted the curves of algorithms whose average rewards are 5% lower than the highest reward.
Now we provide the curves of all the algorithms in Figure 5.
[Figure]