EE-N ET : E XPLOITATION -E XPLORATION N EURAL N ETWORKS IN C ONTEXTUAL B ANDITS
Anonymous authors
Paper under double-blind review
A BSTRACT
Contextual multi-armed bandits have been studied for decades and adapted to various applications such as online advertising and personalized recommendation.
To solve the exploitation-exploration tradeoff in bandits, there are three main techniques: epsilon-greedy, Thompson Sampling (TS), and Upper Conﬁdence
Bound (UCB).
In recent literature, linear contextual bandits have adopted ridge regression to estimate the reward function and combine it with TS or UCB strategies for exploration.
However, this line of works explicitly assumes the reward is based on a linear function of arm vectors, which may not be true in real-world datasets.
To overcome this challenge, a series of neural-based bandit algorithms have been proposed, where a neural network is assigned to learn the underlying reward function and TS or UCB are adapted for exploration.
In this paper, we propose "EENet", a neural-based bandit approach with a novel exploration strategy.
In addition to utilizing a neural network (Exploitation network) to learn the reward function,
EE-Net adopts another neural network (Exploration network) to adaptively learn potential gains compared to currently estimated reward.
Then, a decision-maker is constructed to combine the outputs from the Exploitation and Exploration networks.
We prove that EE-Net achieves O ( √ T log T ) regret, which is tighter than existing state-of-the-art neural bandit algorithms ( O ( √ T log T ) for both UCB-based and
TS-based).
Through extensive experiments on four real-world datasets, we show that EE-Net outperforms existing linear and neural bandit approaches.
I NTRODUCTION
The stochastic contextual multi-armed bandit (MAB) (Dani et al., 2008; Lattimore and Szepesvári, 2020) has been studied for decades in machine learning community to solve sequential decision making, with applications in online advertising (Li et al., 2010), personal recommendation (Wu et al., 2016;
Ban and He, 2021b), etc.
In the standard contextual bandit setting, a set of n arms are presented to a learner in each round, where each arm is represented by a context vector.
Then by certain strategy, the learner selects and plays one arm, receiving a reward.
The goal of this problem is to maximize the cumulative rewards of T rounds.
MAB algorithms have principled approaches to address the trade-off between Exploitation and
Exploration (EE), as the collected data from past rounds should be exploited to get good rewards but also under-explored arms need to be explored with the hope of getting even better rewards.
The most widely-used approaches for EE trade-off can be classiﬁed into three main techniques: Epsilon-greedy (Langford and Zhang, 2008), Thompson Sampling (TS) (Thompson, 1933), and Upper Conﬁdence
Bound (UCB) (Auer, 2002).
Linear contextual bandits (Dani et al., 2008; Li et al., 2010; Abbasi-Yadkori et al., 2011), where the reward is assumed to be a linear function with respect to arm vectors, have been well studied and succeeded both empirically and theoretically.
Given an arm, ridge regression is usually adapted to estimate its reward based on collected data from past rounds.
UCB-based algorithms (Li et al., 2010;
Chu et al., 2011; Wu et al., 2016; Ban and He, 2021b) calculate an upper bound for the conﬁdence ellipsoid of estimated reward and determine the arm according to the sum of estimated reward and
UCB.
TS-based algorithms (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017) formulate each arm as a posterior distribution where mean is the estimated reward and choose the one with the
[Table]
With probability 1 − δ , x t = arg max i ∈ [ n ] f 1 ( x t,i ; θ 1 ) ; Otherwise, select x t randomly.
NeuralTS (Zhang et al., 2020)
For x t,i , ∀ i ∈ [ n ] , draw ˆ r t,i from N ( f 1 ( x t,i ; θ 1 ) , σ t,i 2 ) .
Then, x t = arg max i ∈ [ n ] ˆ r t,i .
NeuralUCB (Zhou et al., 2020) [Equation] = arg [Equation]
[Equation] (cid:0) [Equation] [Equation]
[Equation]
[Equation]
) + UCB
[Equation]
[Equation]
(cid:1) .
EE-Net (Our approach) ∀ i ∈ [ n ] , compute f 1 ( x t,i ; θ 1 ) , f 2 (cid:0) (cid:79) θ 1 f 1 ( x t,i ; θ 1 );
θ 2 (cid:1) (Exploration Net).
Then x t = arg max i ∈ [ n ] f 3 ( f 1 , f 2 ; θ 3 ) .
maximal sampled reward.
However, the linear assumption regarding the reward may not be true in real-world applications (Valko et al., 2013).
To learn non-linear reward functions, recent works have utilized deep neural networks to learn the underlying reward function, thanks to its powerful representation ability.
Considering the past selected arms and received rewards as training samples, a neural network f 1 is built for exploitation.
Zhou et al.
(2020) computes a gradient-based upper conﬁdence bound with respect to f 1 and uses
UCB strategy to select arms.
Zhang et al.
(2020) formulates each arm as a normal distribution where the mean is f 1 and deviation is calculated based on gradient of f 1 , and then uses the TS strategy to choose arms.
Both Zhou et al.
(2020) and Zhang et al.
(2020) achieve the near-optimal regret bound of O ( √ T log T ) .
In this paper, we propose a neural-based bandit algorithm coming with a novel exploration strategy, named "EE-Net".
Similar to other neural bandits, EE-Net has an exploitation network f 1 to estimate rewards for each arm.
The crucial difference from existing works is that EE-Net has an exploration network f 2 to predict the potential gain for each arm compared to current reward estimate.
The input to the exploration network is the gradient of f 1 and the ground-truth is residual between received reward and estimated reward.
The strategy is inspired by recent advances in the neural-based UCB (Ban et al., 2021;
Zhou et al., 2020).
Finally, a decision-maker f 3 is constructed to select arms.
fhas two modes: linear or nonlinear.
In linear mode, f 3 is a linear combination of f 1 and f 2 , inspired by the UCB strategy.
In the nonlinear mode, f 3 is formulated as a neural network with input ( f 1 , f 2 ) and the goal is to learn the probability of being an optimal arm for each arm.
Table 1 summarizes the selection criterion difference of EE-Net from other neural bandit algorithms.
To sum up, the contributions of this paper can be summarized as follows:
We propose a novel exploration strategy, EE-Net, where a neural network is assigned to learn the potential gain compared to the current estimation.
Under standard assumptions of over-parameterized neural networks, we prove that EENet can achieve the regret upper bound of O ( √ T log T ) , which is tighter than existing state-of-the-art bandit algorithms.
We conduct extensive experiments on four real-world datasets, showing that EE-Net outperforms baselines crossing (cid:15) -greedy, TS, and UCB, and becomes the new state-of-the-art exploration policy.
Next, we will show the standard problem deﬁnition and elaborate the proposed EE-Net, before we present our theoretical analysis.
In the end, we provide the empirical evaluation and conclusion.
R ELATED W ORK
Constrained Contextual bandits .
The common constrain placed on the reward function is the linear assumption, usually calculated by ridge regression (Dani et al., 2008; Li et al., 2010; Abbasi-Yadkori et al., 2011; Valko et al., 2013).
The linear UCB-based bandit algorithms (Abbasi-Yadkori et al., 2011; Li et al., 2016) and the linear Thompson Sampling (Agrawal and Goyal, 2013; Abeille and
Lazaric, 2017) can achieve successful performance and the near-optimal regret bound of ˜ O ( √ T ) .
To break the linear assumption, Filippi et al.
(2010) generalizes the reward function to a composition of linear and non-linear functions and then adopt a UCB-based algorithm to deal with it; Bubeck et al.
(2011) imposes the Lipschitz property on reward metric space and constructs a hierarchical optimistic optimization to make selections; Valko et al.
(2013) embeds the reward function into Reproducing
Kernel Hilbert Space and proposes the kernelized TS/UCB bandit algorithms.
Neural Bandits .
To learn non-linear reward functions, deep neural networks have been adapted to bandits with various variants.
Riquelme et al.
; Lu and Van Roy (2017) build L-layer DNN to learn the arm embeddings and apply Thompson sampling on the last layer for exploration.
Zhou et al.
ﬁrst introduces a provable neural-based contextual bandit algorithm with a UCB exploration strategy and then Zhang et al.
(2020) extends the neural network to Thompson sampling framework.
Their regret analysis is built on recent advances on the convergence theory in over-parameterized neural networks(Du et al., 2019; Allen-Zhu et al., 2019) and utilizes Neural Tangent Kernel (Jacot et al., 2018; Arora et al., 2019) to construct connections with linear contextual bandits (AbbasiYadkori et al., 2011).
Ban and He (2021a) further adopts convolutional neural networks with UCB exploration aiming for visual-aware applications.
Xu et al.
(2020) performs UCB-based exploration on the last layer of neural networks to reduce the computational cost brought by gradient-based UCB.
Different from the above existing works, EE-Net keeps the powerful representation ability of neural networks to learn reward function and ﬁrst assigns another neural network to determine exploration.
P ROBLEM DEFINITION
We consider the standard contextual multi-armed bandit with the known number of rounds T (Zhou et al., 2020; Zhang et al., 2020).
In each round t ∈ [ T ] , where the sequence [ T ] =
, T ] , the learner is presented with n arms, in which each arm is represented by a feature vector x t,i ∈ R d for each i ∈ [ n ] .
After playing one arm x t,i , its reward r t,i is assumed to be generated by the function:
[Equation]
where the unknown reward function h ( x t,i ) can be either linear or non-linear and the noise η t,i is drawn from certain distribution with expectation E [ η t,i ] = 0 .
Following many existing works (Zhou et al., 2020; Ban et al., 2021; Zhang et al., 2020), we consider bounded rewards, r t,i ∈
[ a, b ] .
For the brevity, we denote the selected arm in round t by x t and the reward received in t by r t .
The standard regret of T rounds is deﬁned as:
[Equation]
where x ∗ t = arg max i ∈ [ n ] h ( x t,i ) .
The goal of this problem is to minimize R T by certain selection strategy.
Notation.
We denote by { x i } t i =
the sequence ( x 1 , . . . , x t ) .
We use (cid:107) v (cid:107) 2 or (cid:107) v (cid:107) to denote the
Euclidean norm for a vector v , and (cid:107) W (cid:107) 2 and (cid:107) W (cid:107) F to denote the spectral and Frobenius norm for a matrix W .
We use (cid:104)· , ·(cid:105) to denote the standard inner product between two vectors or two matrices.
P ROPOSED M ETHOD : EE-N ET
EE-Net is composed of three independent components, while their input and output are closely correlated.
The ﬁrst component is the exploitation network , f 1 ( · ; θ 1 ) , which is to learn the unknown reward function h based on the data collected in past rounds.
The second component is the exploration network , f 2 ( · ; θ 2 ) , which is to measure the exploration efforts we should make in the present round.
The third component is the decision-maker , f 3 , which is to further trade off between exploitation and exploration, and make the selection.
1) Exploitation Net.
A neural network model f 1 is provided to learn the mapping from arms to rewards.
In round t , denote the network by f 1 ( · ; θ 1 t − 1 ) , where the superscript of θ 1 t − 1 is the index of network and the subscript represents the round where the parameters of f 1 ﬁnished the last update.
Given an arm x t,i , i ∈ [ n ] , f 1 ( x t,i ; θ 1 t − 1 ) is considered the "exploitation score" for x t,i .
By some criterion, after playing arm x t , we receive a reward r t .
Therefore, we can conduct gradient descent to
[Table]
update θ 1 based on the collected training samples { x i , r i } ti =1 and denote the updated parameters by θ 1 t .
2) Exploration Net.
Our exploration strategy is inspired by existing UCB-based neural bandits (Zhou et al., 2020;
Ban et al., 2021).
Based on the Lemma 5.2 in (Ban et al., 2021), given an arm x t,i , with probability at least 1 − δ , we have the following form:
[Equation]
where h is deﬁned in Eq.
(3.1) and Ψ is the upper conﬁdence bound represented by a function with respect to the gradient (cid:79) θ 1 f 1 ( x t,i ; θ 1 t ) (See more details and discussions in Appendix D ).
Then we have the following deﬁnition.
Deﬁnition 4.1.
Given an arm x t,i , we deﬁne h ( x t,i ) − f 1 ( x t,i ; θ 1 t ) as the " expected potential gain " for x t,i and r t,i − f 1 ( x t,i ; θ 1 t ) as the " potential gain " for x t,i .
Let y t,i =
r t,i − f 1 ( x t,i ; θ 1 t ) .
When y t,i > 0 , the arm x t,i has positive potential gain compared to the estimated reward f 1 ( x t,i ; θ 1 t ) .
A large positive y t,i makes the arm more suitable for exploration, whereas a small (or negative)
y t,i makes the arm unsuitable for exploration.
Recall that traditional approaches such as UCB effectively compute such potential gain y t,i using standard tools, e.g.,
Markov inequality, Hoeffding bounds, etc.
from large deviation bounds.
Instead of calculating a large-deviation based statistical form for Ψ , we use a neural network fto learn Ψ , where the input is (cid:79) θ 1 f 1 ( x t,i ; θ 1 t ) and the ground truth is y t,i .
Adopting gradient (cid:79) θ 1 f 1 ( x t,i ; θ 1 t ) as the input also is due to the fact that it incorporates two aspects of information: the feature of arm and the discriminative information of f 1 .
To sum up, we consider f 2 ( (cid:79) θ 1 f 1 ( x t,i ; θ 1 t ); θ 2 t ) as the "exploration score" of x t,i , because it indicates the potential gain of x t,i compared to our current exploitation score f 1 .
Given the selected arm x t , let y t =
r t − f 1 ( x t ; θ 1 t ) .
Therefore, in round t , we can use gradient descent to update θ 2 based on collected training samples { (cid:79) θ 1 f 1 ( x i ; θ 1 i ) , y i } ti .
We also provide other two heuristic forms: y t,i = | r t,i − f 1 ( x t,i ; θ 1 t ) | and y t,
i = ReLU ( r t,i − f 1 ( x t,i ; θ 1 t )) .
We compare them in an ablation study in Appendix A .
3) Decision-maker .
In round t , given an arm x t,i , i ∈ [ n ] , with the computed exploitation score f 1 ( x t,i ; θ 1 t ) and exploration score f 2 ( (cid:79) θ 1 f 1 ; θ 2 t ) , we use a function f 3 (cid:0) f 1 , f 2 ; θ 3 (cid:1) to trade off between exploitation and exploration and compute the ﬁnal score for x t,i .
The selection criterion is deﬁned as
[Equation]
Note that f 3 can be either linear or non-linear functions.
We provide the following two forms.
Linear function .
f 3 can be formulated as a linear function with respect to f 1 and f 2 :
[Equation]
where w 1 , w 2 are two weights preset by the learner.
When w 1 = w 2 = 1 , f 3 can be thought of as
UCB-type policy, where the estimated reward f 1 and potential gain f 2 are simply added together.
In experiments, we report its empirical performance in ablation study ( Appendix A ).
(2) Non-linear function .
f 3 also can be formulated as a neural network to learn the mapping from ( f 1 , f 2 ) to the optimal arm.
We transform the bandit problem into a binary classiﬁcation problem.
Input f f f
T number η η η
1: Initialize θ θ θ2: for each do 3:
Observe n arms { x t, 1 , . . .
, x t,n } 4: for each i ∈ [ n ] do 5:
Compute f 1 ( x i ; θ 1 t − 1 ) , f 2 ( (cid:79) θ 1 t − 1 f 1 ( x i ; θ 1 t − 1 ) /c √ mL ; θ 2 t − 1 ) , f 3 (( f 1 , f 2 ); θ 3 t − 1 ) 6: end for 7: x t = arg max i ∈ [ n ] f 3 (cid:0) f 1 ( x t,i ; θ 1 t − 1 ) , f 2 ( (cid:79) θ 1 f 1 ; θ 2 t − 1 ); θ 3 t − 1 (cid:1) 8:
Play x t and observe reward r t 9: θ 1 t , θ 2 t , θ 3 t =
GradientDescent( θ 10 , θ 20 , θ 30 , { x i } ti =1 , { r i } ti =1 ) 10: end for 11: 12: procedure G RADIENT D ESCENT ( θ 0 , { x i } ti =1 , { r i } ti =1 ) 13:
L 1 = 12 (cid:80) t i =1 (cid:0)
f 1 ( x i ; θ 1 ) − r i (cid:1)14: θ 1 , (0) = θ15: for i ∈ { 1 , . . .
, K 1 } do 16: θ 1 , ( i )
=
θ 1 , ( i − 1) − η 1 (cid:79) θ 1 , ( i − 1) L17: end for 18:
L 2 = 12 (cid:80) ti =1 (cid:16) f 2 ( (cid:79) θ 1 f 1 /c √ mL ; θ 2 ) − ( r i − f 1 ( x i ; θ
1 , ( K 1 ) ))
(cid:17)19: θ 2 , (0) =
θ20: for i ∈ { 1 , . .
, K 2 } do 21: θ 2 , ( i )
=
θ 2 , ( i − 1) − η 2 (cid:79) θ 2 , ( i − 1)
L22: end for 23: if Applicable then 24:
Determine label p t 25:
L 3 = − 1 t (cid:80) ti =1 (cid:2) p t log f 3 (( f 1 , f 2 ); θ 3 )
+ (1 − p t ) log(1 − f 3 (( f 1 , f 2 ); θ 3 )) (cid:3) 26: θ 3 , (0) =
θ27: for i ∈ { 1 , . .
, K 3 } do 28: θ 3 , ( i ) =
θ 3 , ( i − 1) − η 3 (cid:79) θ 3 , ( i − 1)
L29: end for 30:
Return θ 1 , ( K 1 ) , θ 2 , ( K 2 ) , θ 3 , ( K 3 ) 31: end if 32:
Return θ 1 , ( K 1 ) , θ 2 , ( K 2 ) , θ33: end procedure
Given an arm x t,i , we deﬁne p t,i as the probability of being the optimal arm for x t,i in round t .
For brevity, we denote by p t the probability of being the optimal arm for the selected arm x t in round t .
According to different reward distributions, we have different approaches to determine p t .
Binary reward .
∀ t ∈ [ T ] , suppose r t is a binary variable over a, b ( a < b ) , it is straightforward to set: p t = 1 .
if r t = b ; p t = 0 . 0 , otherwise.
Continuous reward .
∀ t ∈ [ T ] , suppose r t is a continuous variable over the range [ a, b ] , we provide two ways to determine p t .
(1) p t can be directly set as r t − a b − a .
The learner can set a threshold θ, ( a < θ < b ) .
Then p t = 1 .
if r t > θ ; p t = 0 . 0 , otherwise.
Therefore, with the collected training samples { (cid:0) f 1 ( x i ; θ 1 i ) , f 2 ( (cid:79) θ 1 f 1 ; θ 2 i ) (cid:1) , p i } t i =1 in round t , we can conduct gradient descent to update parameters of f 3 ( f 1 , f 2 ; θ 3 t ) .
Table 2 details the working structure of EE-Net and Algorithm 1 depicts the workﬂow of EE-Net.
Remark 4.1.
The networks f 1 , f 2 , f 3 can be different structures according to different applications.
For example, in the vision tasks, f 1 can be set up as convolutional layers (LeCun et al., 1995).
For the exploration network f 2 , the input (cid:79) θ 1 f 1 may have exploding dimensions when the exploitation network f 1 becomes wide and deep, which may cause huge computation cost for f 2 .
To address this challenge, we can apply dimensionality reduction techniques (Roweis and Saul, 2000; Van Der Maaten et al., 2009) to obtain low-dimensional vectors of (cid:79) θ 1 f 1 .
In the experiments, we use Roweis and Saul (2000) to acquire a 10 -dimensional vector for (cid:79) θ 1 f 1 and achieve the best performance among all baselines. Remark 4.2.
EE-Net has the ability to determine exploration direction.
Given an arm x t,i , when the estimation f 1 ( x t,i ) is lower than the expected reward h ( x t,i ) , the learner should make the "upward" exploration, i.e., increase the chance of x t,i being explored; When f 1 ( x t,i ) is higher than h ( x t,i ) , the learner should do the "downward" exploration, i.e., decrease the chance of x t,i being explored.
EE-Net uses the neural network f 2 to learn h ( x t,i ) − f 1 ( x t,i ) (which has positive and negative scores) and has the ability to determine the exploration direction.
In contrast, NeuralUCB will always make "upward" exploration and NeuralTS will randomly choose between "upward" exploration and "downward" exploration (see selection criteria in Table 1).
Remark 4.3. EE-Net reduces the space complexity from O ( p 2 ) to O ( p ) ( let θ 1 ∈ R p ). NeuralUCB and NeuralTS have to maintain the gradient outer product matrix A t = I + (cid:80) ti =1 (cid:79) θ 1 f 1 ( x i ; θ 1 i ) (cid:79) θ 1 f 1 ( x i ; θ 1 i ) (cid:62)
∈ R p × p to store the historical gradients.
On the contrary,
EE-Net does not have this matrix and only regards (cid:79) θ 1 f 1 as the input of f 2 .
R EGRET A NALYSIS
In this section, we provide the regret analysis of EE-Net when f 3 is set as the linear function f 3 = f 1 + f 2 , which can be thought of as the UCB-type trade-off between exploitation and exploration.
For the sake of simplicity, we conduct the regret analysis on some unknown but ﬁxed data distribution
D .
In each round t , n samples { ( x t, 1 , r t, 1 ) , ( x t, 2 , r t, 2 ) , . . .
, ( x t,n , r t,n ) } are drawn from D , where i ∈ [ n ] , x t,i is the representation of arm satisfying (cid:107) x t,i (cid:107) 2 = 1 and r t,i is the corresponding reward satisfying r t,i ∈
[0 , 1] , which are standard assumptions in neural bandits (Zhou et al., 2020; Zhang et al., 2020).
The analysis will focus on over-parameterized neural networks (Jacot et al., 2018; Du et al., 2019;
Allen-Zhu et al., 2019).
Given an input x ∈ R d , without loss of generality, we deﬁne the fullyconnected network f with depth L ≥ 2 and width m :
[Equation]
where σ is the ReLU activation function, W 1 ∈ R m × d , W l ∈ R m × m , for 2 ≤ l ≤ L − 1 ,
W L ∈ R 1 × m , and θ = [ vec ( W 1 ) (cid:124) , vec ( W 2 ) (cid:124) , . .
, vec ( W L ) (cid:124) ] (cid:124) .
In round t , given the collected data { x i , r i } ti =1 , the loss function is deﬁned as:
[Equation]
Initialization .
For any l ∈ [ L ] , each entry of W l is drawn from the normal distribution N (0 , 2 m ) .
Note that EE-Net at most has three networks f 1 , f 2 , f 3 .
We deﬁne them following the deﬁnition of f for brevity, although they may have different depth or width.
Then, we have the following theorem for EE-Net.
Recall that η 1 , η 2 are the learning rates for f 1 , f 2 ; K 1 is the number of iterations of gradient descent for f 1 in each round; and K 2 is the number of iterations for f 2 .
Theorem 1.
Let f 1 , f 2 follow the setting of f (Eq.
(5.1) ) with width m, m (cid:48) respectively and same depth L .
Let L 1 , L 2 be loss function deﬁned in Algorithm 1.
Set f 3 as f 3 = f 1 + f 2 .
Given two constants (cid:15) 1 , (cid:15) 2 , 0 < [Equation] < [Equation]
, assume
[Equation]
then with probability at least 1 − δ , the expected cumulative regret of EE-Net in T rounds satisﬁes [Equation] O
[Equation]
where
[Equation]
When (cid:15) 2 ≤ 1 /T , we have R T ≤ O (1) + O (cid:16) ( ξ 2 + (cid:15)
[Equation]
[Equation]
√ T − 1) (cid:112) 2 log( O ( Tn ) /δ )
(cid:17) .
Comparison with NeuralUCB/TS .
Under the same assumptions in over-parameterized neural networks, the regret bounds complexity of NeuralUCB (Zhou et al., 2020) and NeuralTS (Zhang et al., 2020) both are
[Equation]
, and ˜ d = log det ( I + H /λ [Equation] log [Equation]
[Equation]
[Equation]
where H is the neural tangent kernel matrix (NTK)
(Jacot et al., 2018; Arora et al., 2019) and λ is a regularization parameter.
Remark 5.1.
It is easy to observe that the regret bound of EE-Net is tighter than NeuralUCB/TS, which roughly improves by a multiplicative factor of √ log T , because our proof of EE-Net is directly built on recent advances in convergence theory (Allen-Zhu et al., 2019) and generalization bound (Cao and Gu, 2019) of over-parameterized neural networks.
Instead, the analysis for NeuralUCB/TS contains three parts of approximation error by calculating the distances between the expected reward and ridge regression, ridge regression and NTK, and NTK and network function.
Remark 5.2.
The regret bound of EE-Net does not have the effective dimension ˜ d which is a considerable multiplicative factor when the input dimension d is extremely large.
The effective dimension is ﬁrst introduced by Valko et al.
(2013) to measure the underlying dimensions of observed context.
Although ˜ d can be upper bounded to some dimensional subspace of reproducing kernel hilbert space (RKHS) by NTK (Zhang et al., 2020), their regret bound still has the multiplicative factor ˜ d , but EE-Net does not have this factor.
The proof of Theorem 1 is in Appendix B .
Moreover, we provide the regret analysis of greedy approach with the only exploitation network f 1 , i.e. f 3 = f 1 , showing that EE-Net theoretically outperforms the greedy approach (see details in Appendix C) .
E XPERIMENTS
In this section, we evaluate EE-Net on four real-world datasets comparing with strong state-of-the-art baselines.
We ﬁrst present the setup of experiments, then show regret comparison and report ablation study.
For the reproducibility, all the code has been released anonymously 1 .
MNIST dataset .
MNIST is a well-known image dataset (LeCun et al., 1998) for the 10-class classiﬁcation problem.
Following the evaluation setting of existing works (Valko et al., 2013; Zhou et al., 2020; Zhang et al., 2020), we transform this classiﬁcation problem into bandit problem.
Consider an image x ∈ R d , we aim to classify it from 10 classes.
First, in each round, the image x is transformed into 10 arms and presented to the learner, represented by 10 vectors in sequence x 1 = ( x , 0 , . . . , 0 ) , x 2 = ( 0 , x , . . . , 0 ) , . . .
, x
10 = ( 0 , 0 , . . .
, x ) ∈ R 10 d .
The reward is deﬁned asif the index of selected arm matches the index of x ’s ground-truth class; Otherwise, the reward is 0 .
Yelp 2 and Movielens (Harper and Konstan, 2015) datasets .
Yelp is a dataset released in the Yelp dataset challenge, which consists of 4.7 million rating entries for 1 .
57 × 10 5 restaurants by 1 .million users.
MovieLens is a dataset consisting of 25 million ratings between 1 . 6 × 10 5 users and 6 × 10 4 movies.
We build the rating matrix by choosing the top 2000 users and toprestaurants(movies) and use singular-value decomposition (SVD) to extract a 10 -dimension feature vector for each user and restaurant(movie).
In these two datasets, the bandit algorithm is to choose the restaurants(movies) with bad ratings.
We generate the reward by using the restaurant(movie)’s gained stars scored by the users.
In each rating record, if the user scores a restaurant(movie) less than 2 stars (5 stars totally), its reward is 1 ; Otherwise, its reward is 0 .
In each round, we set 10 arms as follows: we randomly choose one with reward 1 and randomly pick the other 9 restaurants(movies) withrewards; then, the representation of each arm is the concatenation of corresponding user feature vector and restaurant(movie) feature vector.
Disin (Ahmed et al., 2018) dataset .
Disin is a fake news dataset on kaggle 3 including 12600 fake news articles and 12600 truthful news articles, where each article is represented by the text.
To transform the text into vectors, we use the approach (Fu and He, 2021) to represent each article by a 300-dimension vector.
Similarly, we form a 10-arm pool in each round, where 9 real news and 1 fake news are randomly selected.
If the fake news is selected, the reward is 1 ; Otherwise, the reward is 0 .
Baselines .
To comprehensively evaluate EE-Net, we choose 4 neural-based bandit algorithms, one linear and one kernelized bandit algorithms.
LinUCB (Li et al., 2010) explicitly assumes the reward is a linear function of arm vector and unknown user parameter and then applies the ridge regression and un upper conﬁdence bound to determine selected arm.
KernelUCB (Valko et al., 2013) adopts a predeﬁned kernel matrix on the reward space combined with a UCB-based exploration strategy.
Neural-NoExplore only uses the exploitation network f 1 and selects an arm by the greedy strategy x t = arg max i ∈ [ n ] f 1 ( x t,i ; θ 1 ) .
Neural-Epsilon adapts the epsilon-greedy exploration strategy on exploitation network f 1 .
I.e., with probability 1 − (cid:15) , the arm is selected by x t = arg max i ∈ [ n ] f 1 ( x t,i ; θ 1 ) and with probability (cid:15) , the arm is chosen randomly.
NeuralUCB (Zhou et al., 2020) uses the exploitation network f 1 to learn the reward function coming with an UCB-based exploration strategy.
NeuralTS
(Zhang et al., 2020) adopts the exploitation network f 1 to learn the reward function coming with an Thompson Sampling exploration strategy.
Note that we do not report results of LinTS and KernelTS in experiments, because of the limited space in ﬁgures, but LinTS and KernelTS have been signiﬁcantly outperformed by NeuralTS (Zhang et al., 2020).
Setup for EE-Net .
To compare fairly, for all the neural-based methods including EE-Net, the exploitation network f 1 is built by a 2-layer fully-connected network with 100 width.
For the exploration network f 2 , we use a 2-layer fully-connected network with 100 width as well.
For the decision maker f 3 , by comprehensively evaluate both linear and nonlinear functions, we found that the most effective approach is combining them together, which we call " hybrid decision maker ".
In detail, for rounds t ≤ 500 , f 3 is set as f 3 = f 2 + f 1 , and for t > 500 , f 3 is set as a neural network with two 20-width fully-connected layers.
Setting f 3 in this way is because the linear decision maker can maintain stable performance in each running (robustness) and the non-linear decision maker lacks the stability while can further improve the performance (see details in Appendix A ).
The hybrid decision maker can combine these two advantages together.
For all the neural networks, we use the
Adam optimizer (Kingma and Ba, 2014) and learning rate is set as 0 . 0001 .
With the same exploitation network f 1 , EE-Net outperforms all baselines.
6. R EGRET C OMPARISON
Conﬁgurations .
For LinUCB, following (Li et al., 2010), we do a grid search for the exploration constant α over ( 0 . 01 , 0 .
1 , 1) which is to tune the scale of UCB.
For KernelUCB (Valko et al., 2013), we use the radial basis function kernel and stop adding contexts after 1000 rounds, following (Valko et al., 2013; Zhou et al., 2020).
For the regularization parameter λ and exploration parameter ν in KernelUCB, we do the grid search for λ over ( 0 . 1 , 1 , 10) and for ν over (0 . 01 , 0 . 1 , 1) .
For NeuralUCB and NeuralTS, following setting of (Zhou et al., 2020; Zhang et al., 2020), we use the exploiation network f 1 and conduct the grid search for the exploration parameter ν over ( 0 . 001 , 0 . 01 , 0 . 1 , 1) and for the regularization parameter λ over (0 . 01 , 0 . 1 , 1) .
For NeuralEpsilon, we use the same neural network f 1 and do the grid search for the exploration probability (cid:15) over ( 0.01 , 0 . 1 , 0 . 2) .
Neural-Noexplore uses the same neural network f 1 as well.
For the neural bandits
NeuralUCB/TS, following their setting, as they have expensive computation cost to store and compute the whole gradient matrix, we use a diagonal matrix to make approximation.
For all grid-searched parameters, we choose the best of them for the comparison and report the averaged results of 10 runs for all methods.
Results .
Figure 1 and Figure 2 show the regret comparison on these four datasets.
EE-Net consistently outperforms all baselines across all datasets.
For LinUCB and KernelUCN, the simple linear reward function or predeﬁned kernel cannot properly formulate ground-truth reward function existed in real-world datasets.
In particular, on Mnist and Disin datasets, the correlations between rewards and arm feature vectors are not linear or some simple mappings.
Thus, LinUCB and KernelUCB barely exploit the past collected data samples and fail to select correct arms.
For neural-based bandit algorithms, as Neural-Noexplore does not have exploration portion, its rates of collecting new samples and learning new knowledge are unstable and usually delayed.
Therefore, Neural-Noexplore usually is inferior to the methods with exploration.
The exploration probability of Neural-Epsilon is ﬁxed and difﬁcult to be adjustable.
Thus it is usually hard to make effective exploration.
To make exploration,
NeuralUCB statistically calculates a gradient-based upper conﬁdence bound and NeuralTS draws each arm
’s predicted reward from a normal distribution where the standard deviation is computed by gradient.
However, the conﬁdence bound or standard deviation they calculated only consider the worst cases and thus may not be able represent the actual potential of each arm.
Instead, EE-Net uses a neural network f 2 to learn each arm’s potential by neural network’s powerful representation ability.
Therefore, EE-Net can outperform these two state-of-the-art bandit algorithms.
Note that
NeuralUCB/TS does need two parameters to tune UCB/TS according to different scenarios while
EE-Net only needs to set up a neural network and automatically learns it.
Ablation Study .
In Appendix A , we conduct ablation study regarding the label function y of f 2 and the different setting of f 3 .
C ONCLUSION
In this paper, we propose a novel exploration strategy, EE-Net.
In addition to a neural network that exploits collected data in past rounds , EE-Net has another neural network to learn the potential gain compared to current estimation for exploration.
Then, a decision maker is built to make selections to further trade off between exploitation and exploration.
We demonstrate that EE-Net outperforms
NeuralUCB and NeuralTS both theoretically and empirically, becoming the new state-of-the-art exploration policy.
B P ROOF OF T HEOREM
In this section, we provide the proof of Theorem 1 and related lemmas.
Theorem 1 .
Let f 1 , f 2 follow the setting of f (Eq.
(5.1) ) with width m, m (cid:48) respectively and same depth L .
Let L 1 , L 2 be loss functions deﬁned in Algorithm 1.
Set f 3 as f 3 = f 1 + f 2 .
Given two constants (cid:15) 1 , (cid:15) 2 , 0 < (cid:15) 1 ,
[Equation]
2 < 1 , suppose
[Equation]
then with probability at least 1 − δ , the expected cumulative regret of EE-Net in T rounds satisﬁes
[Equation]
where
[Equation]
When (cid:15) 2 ≤ 1 /T , we have
[Equation]
Proof.
In round t , given the n arms ( x t, 1 , . . .
x t,n ) , let H t
= ( h ( x t,i ) , . . .
, h ( x t,n )) denote their expected rewards.
For brevity, for the selected arm x t in round t , let h ( x t ) be its expected reward and x ∗ t be the optimal arm in round t .
Then, the expected regret of round t is computed by
[Equation]
where I 1 is because f 3 ( x t ) =
max i ∈ [ n ] f 3 ( x t,i ) and f 3 ( x t ) − f 3 ( x ∗ t ) ≥ 0 .
Then, based on Lemma B.1, with probability at least 1 − δ , we have
[Equation]
Then expected regret of T rounds is computed by
[Equation]
where I 2 is because (cid:80) Tt =1 1 √ t ≤
[Equation] [Equation]
√
[Equation]
Chlebus
[Equation]
When K 2 =
Θ( poly ( T,n,L ) δ· log T ) , according to Lemma B.3 (1), with probability at least 1 − δ , we have (cid:15) 2 = 1 /T .
Therefore, we have
[Equation]
As the choice of m , we have ξ 1 ≤ 1 .
Therefore, we have
[Equation]
The proof is completed.
Lemma B.1.
Given 0 < (cid:15) 1 , (cid:15) 2 < 1 , suppose m, η, K 1 , K 2 satisfy the conditions in Eq. (C.1).
Then, with probability at least 1 − δ , for any t ∈ [ T ] , i ∈ [ n ] , it holds uniformly that E [Equation] x t
[Equation]
i
[Equation]
r t,
i
[Equation]
Proof.
Using the Lemma B.4 (2) and Lemma B.3 (2), for t ∈ [ T ] , with probability at least 1 − δ , we have (cid:107) (cid:79) θ 1 t f ( x ; θ 1 t ) (cid:107) 2 ≤ O ( √ mL ) for any (cid:107) x (cid:107) ≤ 1 .
Now, consider the exploration network f 2 .
There must exist a constant c 1 such that
[Equation]
Consider (cid:79) θ 1 t f ( x ; θ 1 t ) ∈
R p where p
= m + md + m
( L − 1) .
Then applying Lemma B.2 and union bound on f 2 , replacing d by p , with probability at least 1 − 2 δ , for any t ∈
[ T ] , we have
[Equation]
Then, applying the Lemma B.3 (1) on f 1 , with probability at least 1 − δ , for any i ∈ [ t ] , we have
[Equation]
Applying union bound, with probability at least 1 − 3 δ , we have
[Equation]
[Equation] Then, using the Lemma B.3 (1) and applying union bound [Equation] with probability at least 1 − 4 δ
[Equation]
we have
[Equation]
Set
[Equation]
f
[Equation]
θ 1 t ) (cid:1) | .
We have
[Equation]
Then, we have
[Equation]
where F i denotes the σ -algebra generated by ( f 1 ( θ 11 ) , f 2 ( θ 21 )) , . .
, ( f 1 ( θ 1 i ) , f 2 ( θ 2 i )) and f 1 ( θ 1 i ) , f 2 ( θ 2 i )) are the hypothesis of neural networks f 1 , f 2 .
Then applying the Hoeffding-Azuma inequality to V 1 , . .
. , V t , with probability at least 1 − 5 δ , we have
[Equation]
Then, applying union bound
[Equation]
with probability at least 1 − O ( Tn ) δ , Eq(B.12) always holds for any t ∈
[ T ]
[Equation]
i ∈ [ n ] .
Rescaling the δ completes the proof
[Equation]
Lemma B.2.
Let f ( · ; θ t ) follow the gradient descent of f 1 in Algorithm 1.
Suppose m, η, K satisfy the conditions in Eq.
(C.1). With probability at least 1 − δ , for any x with (cid:107) x (cid:107) 2 = 1 and t ∈ [ T ] , it holds that
[Equation]
Proof.
Considering an inequality | a − b | ≤ c
[Equation]
we have | a | ≤ | b | + c .
Let θ 0 be randomly initialized.
Then applying Lemma B [Equation] 4 (1), for any x ∼ D , (cid:107) x (cid:107) 2 = 1 and (cid:107) θ t − θ [-
-]0 (cid:107) ≤ w , we have
[Equation]
where I 1 is an application of Cauchy–Schwarz inequality, I 2 is by initializing f ( x ; θ 0 )
=
and
Lemma B.3 (3), and I 3 is due to Lemma B.3 (2).
Then, apply the union bound and rescale δ .
The proof is completed.
Lemma B.3.
Suppose m ≥ max (cid:16) poly ( n, L, δ − 1 · d ) , Ω( e √ log1 /δ ) (cid:17) , the learning rate η = Ω( δd poly ( T,n,L ) m ) , the number of iterations K satisﬁes the conditions in Eq.
(C.1) , then with probability at least 1 − δ , given a constant 0 < (cid:15) < 1 , starting from random initialization,
(1) (Theorem 1 in (Allen-Zhu et al., 2019))
The loss satisﬁes L ≤ (cid:15) (Eq.
[Equation]
2)) in
[Equation]
(2) (Theorem 1 in (Allen-Zhu et al., 2019))
For any k ∈ [ K ] , it holds uniformly that (cid:107) θ ( k ) t
[Equation]
(3) Following the initialization, it holds that
[Equation]
where θ ( k ) t represents the parameters of f after k ∈ [ K ] iterations if gradient descent in round t .
Proof.
For (1) and (2), the only different setting from (Allen-Zhu et al., 2019) is that the initialization of last layer W L ∼ N (0 , 2 m ) in this paper while W L ∼ N (0 , 1 d ) in (Allen-Zhu et al., 2019).
Because m > d , the upper bound in (Allen-Zhu et al., 2019) still holds for W L : with probability at least 1 − exp ( − Ω( m/L )) , (cid:107) W L (cid:107) ≤ (cid:112) m/d .
Therefore, it is easy to verify that (1) and (2) still hold for the initialization of this paper.
For (3), based on Lemma 7.1 in Allen-Zhu et al.
(2019), we have | f ( x i ; θ 0 )
| ≤ O (1) .
Denote by D the ReLU function.
For any l ∈ [ L ] ,
[Equation]
where the inequality is according to Lemma 7.2 in Allen-Zhu et al.
Therefore, we have
[Equation]
(cid:79) θ 0 f ( x i ; θ 0 ) (cid:107) ≤ O ( L ) .
Lemma B.4 (Lemma 4.1, (Cao and Gu, 2019)) .
Suppose O ( m − 3 / 2 L − 3 / 2 [log( TnL 2 /δ )] 3 / 2 ) ≤ w ≤
O
( L − 6 [log m ] − 3 / 2 ) .
Then, with probability at least 1 − δ over randomness of θ 0 , for any t ∈ [ T ] , (cid:107) x (cid:107) 2 = 1 , and θ , θ (cid:48) satisfying (cid:107) θ − θ 0 (cid:107) ≤ w and (cid:107) θ (cid:48) − θ 0 (cid:107) ≤ w , it holds uniformly that
[Equation]
w 1 / 3 L 2 (cid:112) m log( m ))
(cid:107) θ − θ (cid:48)
(cid:107)(2) (cid:107) (cid:79) θ f
( x ; θ ) (cid:107) F ≤ O ( √ mL ) .
C C OMPARISON WITH S INGLE R EWARD F UNCTION
In this section, we provide the regret analysis of pure-exploitation approach, denoted by EE-Net’,
i
[Equation]
e., the EE-Net with f 3 = f 1 .
Theorem 2.
Let f 1 follow the setting of f (Eq.
(5.1) ) and f 3 = f 1 .
Given one constant (cid:15) 1 , 0 < (cid:15) 1 < 1 , suppose
[Equation]
then with probability at least 1 − δ
[Equation]
the expected regret of EE-Net’ in T rounds satisﬁes
[Equation]
where
[Equation]
When (cid:15) 1 ≤ 1 /T
[Equation]
we have
[Equation]
Comparison with Theorem 1 .
With the regret upper bounds of EE-Net and EE-Net’, we have
[Equation]
Note that ξ 1 , ξ 2 < 1 by setting of m, m (cid:48) .
It is easy to observe that regret bound of EE-Net is tighter than EE-Net’, because (cid:15) 1 is a given number and can be (cid:15) 1 (cid:28) 1 .
Therefore, by adding a exploration network f 2 , EE-Net can achieve a tighter bound than pure-exploitation approach.
Proof of Theorem 2.
In round t , given the n arms ( x t, 1 , . . .
x t,n ) , let H t
= ( h ( x t,i ) , . . .
, h ( x t,n )) denote their expected rewards.
For brevity, for the selected arm x t in round t , let h ( x t ) be its expected reward and x ∗ t be the optimal arm in round t .
Then, based on the proof of Theorem 1, the expected regret of round t is computed by
[Equation]
Then, based on Lemma C.1, with probability at least 1 − δ , we have
R
[Equation]
≤(cid:32)(cid:114) 2 (cid:15)
[Equation]
) (cid:114) 2 log( O ( Tn ) /δ ) t (cid:33)
Then expected regret of T rounds is computed by
[Equation]
(cid:32)(cid:114)
[Equation]
t + ( ξ 1 + 1) (cid:114) 2 log( O ( Tn ) /δ ) t (cid:33) <
2(2 √ T − 1) √ 2 (cid:15) 1 + 2( ξ 1 + 1)(2 √ T − 1) (cid:112) 2 log( O ( Tn ) /δ )
(cid:124) (cid:123)(cid:122) (cid:125) I(C.3) where I 2 is because (cid:80)
[Equation]
(Chlebus
[Equation]
When K = Ω( poly ( T,n,L ) δ· log T ) , according to Lemma 3.4, with probability at least 1 − δ , we have (cid:15) 1 = 1 /T .
Therefore, we have
[Equation]
The proof is completed.
Lemma C.1.
Given 0 < (cid:15) 1 < 1 , suppose m, η satisfy the conditions in Eq.
(C.1) and K 1 =
Ω( poly ( T,n,L ) δ· log (cid:15) − 1 1 ) .
Then, with probability at least 1 − δ , for any t ∈ [ T ] , i ∈ [ n ] , it holds uniformly that
[Equation]
Proof.
Applying Lemma B.2 on f 1 , with probability at least 1 − δ , for any t ∈
[ T ] , it holds that | f 1 ( x ; θ t ) | ≤ ξ 1 .
And due to | r | ≤ 1 for any ( x , r ) ∼ D , we have
[Equation]
Then, we have
[Equation]
where I 1 is an application of Lemma B.3 (1).
Then, based on Lemma 1 in Cesa-Bianchi et al.
, applying the union bound, with probability at least 1 − 2 δ , we have
[Equation]
Finally, rescaling δ completes the proof.
D G RADIENT - BASED U PPER C ONFIDENCE B OUND
In this section, we list one gradient-based UCB from existing works (Ban et al., 2021;
Zhou et al., 2020), which motivates our design of exploration network f 2 .
Let g ( x t ; θ t ) =
(cid:79) θ t f ( x t ; θ t ) .
Lemma D.1.
(Lemma 5.2 in (Ban et al., 2021)).
Given a set of context vectors { x t } Tt =1 and the corresponding rewards { r t } Tt =1 , E ( r t ) =
h ( x t ) for any x t ∈ { x t } Tt =1 .
Let f ( x t ; θ ) be the L -layers fully-connected neural network where the width is m , the learning rate is η , the number of iterations of gradient descent is K .
Then, there exist positive constants C 1 , C 2 , S , such that if m ≥ poly ( T, n, L,
[Equation]
then, with probability at least 1 − δ , for any x t ∈ { x t } Tt =1 , we have the following upper conﬁdence bound: [Equation] D
[Equation]
where
[Equation]
Note that g ( x t ; θ 0 ) is the gradient at initialization, which can be initialized as constants.
Therefore, the above UCB can be represented as the following form for exploitation network f 1 : | h ( x t,i ) − f 1 ( x t,i ; θ 1 t ) | ≤ Ψ( g ( x t ; θ t )) .
Disadvantages of UCB .
The above UCB tries to measure the potential gain for x t compared to current estimation
f 1 ( x t ) , i.e., y = h ( x t ) − f ( x t ; θ t ) .
However, this UCB only considers the worst case for y and this UCB usually is a very large number.
This is why they need to add turning parameters in their implementation (Ban et al., 2021;
Zhou et al., 2020).
For these arms with h ( x t ) < f ( x t ; θ t ) , their exploration score should be lower than the arms with h ( x t ) >
f ( x t ; θ t ) , while their added points are same in UCB strategy.
In contrast, EE-Net has a neural network f 2 to learn y and intends to predict y accurately.
Moreover, f 2 can discriminate the arms with h ( x t ) >
f ( x t ; θ t ) from the arms with h ( x t ) <
f
( x t ; θ t ) .
E E XPERIMENTS (A DDED )
[Figure]
Figure 5 reports the average decision-making time cost of 5 runs for neural-based methods.
NeuralUCB and NeuralTS are about 1.5 to 2 times slower than EE-Net, because NeuralUCB and NeuralTS need additional computational cost for the gradient kernel matrix.
Neural-Epsilon is the fastest approach because it only needs to compute one network output when making selections, while EE-Net needs to compute two network outputs.
Figure 6 reports the average overall time cost of 5 runs for neural-based methods.
EE-Net is about 1.2-1.6 times slower than NeuralUCB and NeuralTS, because EE-Net needs to train two neural networks in each round while NeuralUCB and NeuralTS need to train one.
Neural-Epsilon is still the fastest method because it only needs to train one neural network and it does not have the extra calculation of gradient.
Figure 7 reports the regret comparison on Movielens and Yelp data sets, where each method was run 30 times.
These results are very similar to the results in Figure 1.
[Figure]