EE-N ET : E XPLOITATION -E XPLORATION N EURAL N ETWORKS IN C ONTEXTUAL B ANDITS
A BSTRACT
In this paper, we propose a novel neural exploration strategy in contextual bandits,
EE-Net, distinct from the standard UCB-based and TS-based approaches.
Contextual multi-armed bandits have been studied for decades with various applications.
To solve the exploitation-exploration tradeoff in bandits, there are three main techniques: epsilon-greedy, Thompson Sampling (TS), and Upper Conﬁdence Bound (UCB).
In recent literature, linear contextual bandits have adopted ridge regression to estimate the reward function and combine it with TS or UCB strategies for exploration.
However, this line of works explicitly assumes the reward is based on a linear function of arm vectors, which may not be true in real-world datasets.
To overcome this challenge, a series of neural bandit algorithms have been proposed, where a neural network is used to learn the underlying reward function and TS or
UCB are adapted for exploration.
Instead of calculating a large-deviation based statistical bound for exploration like previous methods, we propose "EE-Net", a novel neural-based exploration strategy.
In addition to using a neural network (Exploitation network) to learn the reward function, EE-Net uses another neural network (Exploration network) to adaptively learn potential gains compared to the currently estimated reward for exploration.
Then, a decision-maker is constructed to combine the outputs from the Exploitation and Exploration networks.
We prove that EE-Net can achieve O ( √ T log T ) regret and show that EE-Net outperforms existing linear and neural contextual bandit baselines on real-world datasets.
I NTRODUCTION
The stochastic contextual multi-armed bandit (MAB) (Lattimore and Szepesvári, 2020) has been studied for decades in machine learning community to solve sequential decision making, with applications in online advertising (Li et al., 2010), personal recommendation (Wu et al., 2016;
Ban and He, 2021b), etc.
In the standard contextual bandit setting, a set of n arms are presented to a learner in each round, where each arm is represented by a context vector.
Then by certain strategy, the learner selects and plays one arm, receiving a reward.
The goal of this problem is to maximize the cumulative rewards of T rounds.
MAB algorithms have principled approaches to address the trade-off between Exploitation and
Exploration (EE), as the collected data from past rounds should be exploited to get good rewards but also under-explored arms need to be explored with the hope of getting even better rewards.
The most widely-used approaches for EE trade-off can be classiﬁed into three main techniques: Epsilon-greedy (Langford and Zhang, 2008), Thompson Sampling (TS) (Thompson, 1933), and Upper Conﬁdence
Bound (UCB) (Auer, 2002;
Ban and He, 2020).
Linear bandits (Li et al., 2010; Dani et al., 2008; Abbasi-Yadkori et al., 2011), where the reward is assumed to be a linear function with respect to arm vectors, have been well studied and succeeded both empirically and theoretically.
Given an arm, ridge regression is usually adapted to estimate its reward based on collected data from past rounds.
UCB-based algorithms (Li et al., 2010;
Chu et al., 2011; Wu et al., 2016;
Ban and He, 2021b) calculate an upper bound for the conﬁdence ellipsoid of estimated reward and determine the arm according to the sum of estimated reward and UCB.
TS-based algorithms (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017) formulate each arm as a posterior distribution where mean is the estimated reward and choose the one with the maximal
[Figure]
r
[Figure]
! " ∇ ! "
" ($;& " )
[Figure]
r − ! !
[Figure]
Exploitation Network ! !
Exploration Network ! "
ℎ(!) & !
( " ℎ(!) & !
(∇! ! (∇! !
[Figure]
sampled reward.
However, the linear assumption regarding the reward may not be true in real-world applications (Valko et al., 2013).
To learn non-linear reward functions, recent works have utilized deep neural networks to learn the underlying reward functions, thanks to its powerful representation ability.
Considering the past selected arms and received rewards as training samples, a neural network f 1 is built for exploitation.
Zhou et al.
(2020) computes a gradient-based upper conﬁdence bound with respect to f 1 and uses
UCB strategy to select arms.
Zhang et al.
(2021) formulates each arm as a normal distribution where the mean is f 1 and deviation is calculated based on gradient of f 1 , and then uses the TS strategy to choose arms.
Both Zhou et al.
(2020) and Zhang et al.
(2021) achieve the near-optimal regret bound of O ( √ T log T ) .
In this paper, we propose a novel neural exploration strategy, named "EE-Net".
Similar to other neural bandits, EE-Net has another exploitation network f 1 to estimate rewards for each arm.
The crucial difference from existing works is that EE-Net has an exploration network f 2 to predict the potential gain for each arm compared to current reward estimate.
The input to the exploration network is the gradient of f 1 and the ground-truth is residual difference between the true received reward and the estimated reward from f 1 .
The strategy is inspired by recent advances in the neural UCB strategies (Zhou et al., 2020;
Ban et al., 2021).
Finally, a decision-maker f 3 is constructed to select arms.
f 3 has two modes: linear or nonlinear.
In linear mode, f 3 is a linear combination of f 1 and f 2 , inspired by the UCB strategy.
In the nonlinear mode, f 3 is formulated as a neural network with input ( f 1 , f 2 ) and the goal is to learn the probability of being an optimal arm for each arm.
Figure 1 depicts the workﬂow of EE-Net and its advantages for exploration compared to UCB or TS-based methods (see more details in Appendix D).
To sum up, the contributions of this paper can be summarized as follows:
We propose a novel neural exploration strategy, EE-Net, where another neural network is assigned to learn the potential gain compared to the current reward estimate.
Under standard assumptions of over-parameterized neural networks, we prove that EE-Net can achieve the regret upper bound of O ( √ T log T ) , which improves a multiplicative factor of √ log T and is independent of either input or effective dimension, compared to existing state-of-the-art neural bandit algorithms.
We conduct extensive experiments on four real-world datasets, showing that EE-Net outperforms baselines including linear and neural versions of (cid:15) -greedy, TS, and UCB.
Next, we discuss the problem deﬁnition in Sec.3, elaborate on the proposed EE-Net in Sec.4, and present our theoretical analysis in Sec.5.
In the end, we provide the empirical evaluation (Sec.6) and conclusion.
R ELATED W ORK
Constrained Contextual bandits .
The common constrain placed on the reward function is the linear assumption, usually calculated by ridge regression (Li et al., 2010; Abbasi-Yadkori et al., 2011; Valko et al., 2013; Dani et al., 2008).
The linear UCB-based bandit algorithms (Abbasi-Yadkori et al., 2011; Li et al., 2016) and the linear Thompson Sampling (Agrawal and Goyal, 2013; Abeille and
Lazaric, 2017) can achieve successful performance and the near-optimal regret bound of ˜ O ( √ T ) .
To break the linear assumption, Filippi et al.
(2010) generalizes the reward function to a composition of linear and non-linear functions and then adopt a UCB-based algorithm to deal with it; Bubeck et al.
(2011) imposes the Lipschitz property on reward metric space and constructs a hierarchical optimistic optimization to make selections; Valko et al.
(2013) embeds the reward function into Reproducing
Kernel Hilbert Space and proposes the kernelized TS/UCB bandit algorithms.
Neural Bandits .
To learn non-linear reward functions, deep neural networks have been adapted to bandits with various variants.
Riquelme et al.
; Lu and Van Roy (2017) build L-layer DNN to learn the arm embeddings and apply Thompson Sampling on the last layer for exploration.
Zhou et al.
ﬁrst introduces a provable neural-based contextual bandit algorithm with a UCB exploration strategy and then Zhang et al.
(2021) extends the neural network to Thompson sampling framework.
Their regret analysis is built on recent advances on the convergence theory in over-parameterized neural networks(Du et al., 2019; Allen-Zhu et al., 2019) and utilizes Neural Tangent Kernel (Jacot et al., 2018; Arora et al., 2019) to construct connections with linear contextual bandits (AbbasiYadkori et al., 2011).
Ban and He (2021a) further adopts convolutional neural networks with UCB exploration aiming for visual-aware applications.
Xu et al.
(2020) performs UCB-based exploration on the last layer of neural networks to reduce the computational cost brought by gradient-based UCB.
Different from the above existing works, EE-Net keeps the powerful representation ability of neural networks to learn reward function and ﬁrst assigns another neural network to determine exploration.
P ROBLEM DEFINITION
We consider the standard contextual multi-armed bandit with the known number of rounds T (Zhou et al., 2020; Zhang et al., 2021).
In each round t ∈ [ T ] , where the sequence [ T ] =
, T ] , the learner is presented with n arms, X t
= { x t, 1 , . . .
, x t,n } , in which each arm is represented by a feature vector x t,i ∈ R d for each i ∈ [ n ] .
After playing one arm x t,i , its reward r t,i is assumed to be generated by the function:
[Equation]
where the unknown reward function h ( x t,i ) can be either linear or non-linear and the noise η t,i is drawn from certain distribution with expectation E [ η t,i ] = 0 .
Following many existing works (Zhou et al., 2020; Ban et al., 2021; Zhang et al., 2021), we consider bounded rewards, r t,i ∈
[ a, b ] .
For the brevity, we denote the selected arm in round t by x t and the reward received in t by r t .
The pseudo regret of T rounds is deﬁned as:
[Equation]
where E [ r ∗ t | X t ] = max i ∈ [ n ] h ( x t,i ) is the maximal expected reward in the round t .
The goal of this problem is to minimize R T by certain selection strategy.
Notation.
We denote by { x i } ti =1 the sequence ( x 1 , . . .
, x t ) .
We use (cid:107) v (cid:107) 2 to denote the Euclidean norm for a vector v , and (cid:107) W (cid:107) 2 and (cid:107) W (cid:107) F to denote the spectral and Frobenius norm for a matrix
W .
We use (cid:104)· , ·(cid:105) to denote the standard inner product between two vectors or two matrices.
We may use (cid:79) θ 1 t f 1 ( x t,i ) or (cid:79) θ 1 t f 1 to represent the gradient (cid:79) θ 1 t f 1 ( x t,i ; θ 1 t ) for brevity.
We use { x τ , r τ } tτ = to represent the collected data up to round t .
P ROPOSED M ETHOD : EE-N ET
EE-Net is composed of three components.
The ﬁrst component is the exploitation network , f 1 ( · ; θ 1 ) , which focuses on learning the unknown reward function h based on the data collected in past rounds.
[Table]
The second component is the exploration network , f 2 ( · ; θ 2 ) , which focuses on characterizing the level of exploration needed for each arm in the present round.
The third component is the decision-maker , f 3 , which focuses on suitably combining the outputs of the exploitation and exploration networks leading to the arm selection.
1) Exploitation Net.
The exploitation net f 1 is a neural network which learns the mapping from arms to rewards.
In round t , denote the network by f 1 ( · ; θ 1 t − 1 ) , where the superscript of θ 1 t − 1 is the index of network and the subscript represents the round where the parameters of f 1 ﬁnished the last update.
Given an arm x t,i , i ∈ [ n ] , f 1 ( x t,i ; θ 1 t − 1 ) is considered the "exploitation score" for x t,i .
By some criterion, after playing arm x t , we receive a reward r t .
Therefore, we can conduct gradient descent to update θ 1 based on the collected training samples { x τ , r τ } tτ =1 and denote the updated parameters by θ 1 t .
2) Exploration Net.
Our exploration strategy is inspired by existing UCB-based neural bandits (Zhou et al., 2020;
Ban et al., 2021).
Based on the Lemma 5.2 in (Ban et al., 2021), given an arm x t,i , with probability at least 1 − δ , we have the following UCB form:
[Equation]
where h is deﬁned in Eq.
(3.1) and Ψ is an upper conﬁdence bound represented by a function with respect to the gradient (cid:79) θ 1 t − 1 f 1 (see more details and discussions in Appendix D).
Then we have the following deﬁnition.
Deﬁnition 4.1.
In round t , given an arm x t,i , we deﬁne h ( x t,i ) − f 1 ( x t,i ; θ 1 t − 1 ) as the " expected potential gain " for x t,i and r t,i − f 1 ( x t,i ; θ 1 t − 1 ) as the " potential gain " for x t,i .
Let y t,i =
r t,i − f 1 ( x t,i ; θ 1 t − 1 ) .
When y t,i > 0 , the arm x t,i has positive potential gain compared to the estimated reward f 1 ( x t,i ; θ 1 t − 1 ) .
A large positive y t,i makes the arm more suitable for exploration, whereas a small (or negative)
y t,i makes the arm unsuitable for exploration.
Recall that traditional approaches such as UCB intend to estimate such potential gain y t,i using standard tools, e.g., Markov inequality, Hoeffding bounds, etc., from large deviation bounds.
Instead of calculating a large-deviation based statistical bound for y t,i , we use a neural network f 2 ( · ; θ 2 ) to represent Ψ , where the input is (cid:79) θ 1 t − 1 f 1 ( x t,i ) and the ground truth is r t,i − f 1 ( x t,i ; θ 1 t − 1 ) .
Adopting gradient (cid:79) θ 1 t − 1 f 1 ( x t,i ) as the input also is due to the fact that it incorporates two aspects of information: the feature of the arm and the discriminative information of f 1 .
Moreover, in the upper bound of NeuralUCB or the variance of NeuralTS, there is a recursive term
A t − 1 =
I + (cid:80) t − 1 τ =1 ∇ θ 1 τ − 1 f 1 ( x τ ) ∇ θ 1 τ − 1 f 1 ( x τ ) (cid:62) which is a function of past gradients up to ( t − 1) and incorporates relevant historical information.
On the contrary, in EE-Net, the recursive term which depends on past gradients is θ 2 t − 1 in the exploration network f 2 because we have conducted gradient descent for θ 2 t − 1 based on {∇ θ 1 τ − 1 f 1 ( x τ ) } t − 1 τ =1 .
Therefore, this form θ 2 t − 1 is similar to A t −in neuralUCB/TS, but EE-net does not (need to) make a speciﬁc assumption about the functional form of past gradients, and is also more memory-efﬁcient.
To sum up, in round t , we consider f 2 ( (cid:79) θ 1 t − 1 f 1 ( x t,i ); θ 2 t − 1 ) as the "exploration score" of x t,i , because it indicates the potential gain of x t,i compared to our current exploitation score
f 1 ( x t,i ; θ 1 t − 1 ) .
Therefore, after receiving the reward r t , we can use gradient descent to update θ 2 based on collected training samples { (cid:79) θ 1 τ − 1 f 1 ( x τ ) , r τ − f 1 ( x τ ; θ 1 τ − 1 } tτ =1 .
We also provide other two heuristic forms for f 2 ’s ground-truth label: | r t,i − f 1 ( x t,i ; θ 1 t − 1 ) | and ReLU ( r t,i − f 1 ( x t,i ; θ 1 t − 1 )) .
We compare them in an ablation study in Appendix B.
Input
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
: Initialize
[Table] [Table]
[Table] [Table]
[Table]
do 3:
Observe
[Table]
] do
[Table]
Compute f
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
=
[Table]
Play observe reward
9: θ
[Table]
θ
[Table]
θ
[Table]
D
[Table] [Table]
[Table] [Table]
[Table] [Table]
10: end for 11: 12: procedure G
D ESCENT
[Table] [Table]
[Table] [Table]
14: θ 1 , (0) = θ15: for k ∈ { 1 , . .
, K 1 } do 16: θ 1 , ( k )
=
θ 1 , ( k − 1) − η 1 (cid:79) θ 1 , ( k − 1) L17: end for 18: (cid:98) θ 1 t = θ 1 , ( K 1 ) 19:
L 2 = 12 (cid:80) tτ
=1 (cid:16) f 2 ( φ ( (cid:79)
[Equation]
1 f 1 ( x
[Equation]
)); θ
[Equation] [Equation]
( r
[Equation]
f
[Equation]
( x
[Equation]
; θ
[Equation]
τ
[Equation] [Equation]
(cid:17)20: θ 2 , (0) =
θ21: for k ∈ { 1 , . .
, K 2 } do 22: θ 2 , ( k )
=
θ 2 , ( k − 1) − η 2 (cid:79) θ 2 , ( k − 1) L23: end for 24: (cid:98) θ 2 t = θ 2 , ( K 2 ) 25:
Determine label p t 26:
L 3 = − 1 t (cid:80) ti =1 (cid:2) p t log f 3 (( f 1 , f 2 ); θ 3 )
+ (1 − p t ) log(1 − f 3 (( f 1 , f 2 ); θ 3 )) (cid:3) 27: θ 3 , (0) =
θ28: for k ∈ { 1 , . .
, K 3 } do 29: θ 3 , ( k )
=
θ 3 , ( k − 1) − η 3 (cid:79) θ 3 , ( k − 1) L30: end for 31: (cid:98) θ 3 t = θ 3 , ( K 3 ) 32:
Randomly choose ( θ 1 t , θ 2 t ) uniformly from { ( (cid:98) θ 1 0 , (cid:98) θ 2 0 ) , ( (cid:98) θ 1 1 , (cid:98) θ 2 1 ) , . . .
, ( (cid:98) θ 1 t , (cid:98) θ 2 t ) } 33:
Randomly choose θ 3 t uniformly from { (cid:98) θ 3 0 , (cid:98) θ 3 1 , . . .
, (cid:98) θ 3 t } 34:
Return θ 1 t , θ 2 t , θ 3 t 35: end procedure 3) Decision-maker .
In round t , given an arm x t,i , i ∈ [ n ] , with the computed exploitation score f 1 ( x t,i ; θ 1 t − 1 ) and exploration score f 2 ( (cid:79) θ 1 t − 1 f 1 ; θ 2 t − 1 ) , we use a function f 3 (cid:0) f 1 , f 2 ; θ 3 (cid:1) to trade off between exploitation and exploration and compute the ﬁnal score for x t,i .
The selection criterion is deﬁned as
[Equation]
Note that f 3 can be either linear or non-linear functions.
We provide the following two forms.
(1) Linear function .
f 3 can be formulated as a linear function with respect to f 1 and f 2 :
[Equation]
where w 1 , w 2 are two weights preset by the learner.
When w 1 = w 2 = 1 , f 3 can be thought of as
UCB-type policy, where the estimated reward f 1 and potential gain f 2 are simply added together.
In experiments, we report its empirical performance in ablation study (Appendix B).
(2) Non-linear function .
f 3 also can be formulated as a neural network to learn the mapping from ( f 1 , f 2 ) to the optimal arm.
We transform the bandit problem into a binary classiﬁcation problem.
Given an arm x t,i , we deﬁne p t,i as the probability of being the optimal arm for x t,i in round t .
For brevity, we denote by p t the probability of being the optimal arm for the selected arm x t in round t .
According to different reward distributions, we have different approaches to determine p t .
Binary reward .
∀ t ∈ [ T ] , suppose r t is a binary variable over a, b ( a < b ) , it is straightforward to set: p t = 1 .
if r t = b ; p t = 0 . 0 , otherwise.
Continuous reward .
∀ t ∈ [ T ] , suppose r t is a continuous variable over the range [ a, b ] , we provide two ways to determine p t .
(1) p t can be directly set as r t − a b − a .
The learner can set a threshold θ, ( a < θ < b ) .
Then p t = 1 .
if r t > θ ; p t = 0 . 0 , otherwise.
Therefore, with the collected training samples (cid:110)(cid:16) f 1 ( x τ ; θ 1 τ − 1 ) , f 2 ( (cid:79) θ 1 τ − 1 f 1 ; θ 2 τ − 1 ) (cid:17) , p τ (cid:111) t τ =1 in round t , we can conduct gradient descent to update parameters of f 3 ( · ; θ 3 ) .
Table 1 details the working structure of EE-Net.
Algorithm 1 depicts the workﬂow of EE-Net, where the input of f 2 is normalized, i.e., φ ( (cid:79) θ 1 t − 1 f 1 ( x t,i )) .
Algorithm 1 provides a version of gradient descent (GD) to update EE-Net, where drawing ( θ 1 t , θ 2 t ) uniformly from their stored historical parameters is for the sake of analysis.
One can easily extend EE-Net to stochastic GD to update the parameters incrementally.
Remark 4.1 ( Network structure ) .
The networks f 1 , f 2 , f 3 can be different structures according to different applications.
For example, in the vision tasks, f 1 can be set up as convolutional layers (LeCun et al., 1995).
For the exploration network f 2 , the input (cid:79) θ 1 f 1 may have exploding dimensions when the exploitation network f 1 becomes wide and deep, which may cause huge computation cost for f 2 .
To address this challenge, we can apply dimensionality reduction techniques to obtain lowdimensional vectors of (cid:79) θ 1 f 1 .
In the experiments, we use Roweis and Saul (2000) to acquire a 10 -dimensional vector for (cid:79) θ 1 f 1 and achieve the best performance among all baselines.
Remark 4.2 ( Exploration direction ) .
EE-Net has the ability to determine exploration direction.
Given an arm x t,i , when the estimation f 1 ( x t,i ) is lower than the expected reward h ( x t,i ) , the learner should make the "upward" exploration, i.e., increase the chance of x t,i being explored; When f 1 ( x t,i ) is higher than h ( x t,i ) , the learner should do the "downward" exploration, i.e., decrease the chance of x t,i being explored.
EE-Net uses the neural network f 2 to learn h ( x t,i ) − f 1 ( x t,i ) (which has positive and negative scores) and has the ability to determine the exploration direction.
In contrast,
NeuralUCB will always make "upward" exploration and NeuralTS will randomly choose between "upward" exploration and "downward" exploration (see selection criteria in Table 2 and more details in Appendix D).
Remark 4.3 ( Space complexity ) .
NeuralUCB and NeuralTS have to maintain the gradient outer product matrix (e.g., A t =
(cid:80) tτ
=1 (cid:79) θ 1 f 1 ( x τ ; θ 1 τ ) (cid:79) θ 1 f 1 ( x τ ; θ 1 τ ) (cid:62) ∈
R p × p ) and, for θ 1 ∈ R p , have a space complexity of O ( p 2 ) to store the outer product.
On the contrary, EE-Net does not have this matrix and only regards (cid:79) θ 1 f 1 as the input of f 2 .
Thus, EE-Net reduces the space complexity from
O ( p 2 ) to O ( p ) .
R EGRET A NALYSIS
In this section, we provide the regret analysis of EE-Net when f 3 is set as the linear function f 3 = f 1 + f 2 , which can be thought of as the UCB-type trade-off between exploitation and exploration.
For the sake of simplicity, we conduct the regret analysis on some unknown but ﬁxed data distribution
D .
In each round t , n samples { ( x t, 1 , r t, 1 ) , ( x t, 2 , r t, 2 ) , . . .
, ( x t,n , r t,n ) } are drawn i.i.d.
from D .
This is standard distribution assumption in over-parameterized neural networks (Cao and Gu, 2019).
Then, for the analysis, we have the following assumption, which is a standard input assumption in neural bandits and over-parameterized neural networks(Zhou et al., 2020; Allen-Zhu et al., 2019).
Assumption 5.1 ( ρ -Separability) .
For any t ∈ [ T ] , i ∈ [ n ] , (cid:107) x t,i (cid:107) 2 = 1 , and r t,i ∈ [0 , 1] .
Then, for every pair x t,i , x t (cid:48) ,i (cid:48) , t (cid:48) ∈
[ T ] , i (cid:48) ∈ [ k ] , and ( t, i ) (cid:54) =
( t (cid:48) , i (cid:48) ) , (cid:107) x t,i − x t (cid:48) ,i (cid:48) (cid:107) 2 > ρ , and suppose there exists an operator such that (cid:107) φ ( · ) (cid:107) 2 = 1 and (cid:107) φ ( (cid:79) θ
[Table]
f 1 ( x t,i )) − φ ( (cid:79) θ
[Table]
f 1 ( x t (cid:48) ,i (cid:48) ))
(cid:107) 2 ≥ ρ
For example, the operator can be designed as φ ( (cid:79) θ 1 f 1 ( x t,i ))
=
[Table]
) (cid:107)
[Table]
The analysis will focus on over-parameterized neural networks (Jacot et al., 2018; Du et al., 2019; AllenZhu et al., 2019).
Given an input x ∈ R d , without loss of generality, we deﬁne the fully-connected network f with depth L ≥ 2 and width m :
[Equation]
(5.1) where σ is the ReLU activation function, W 1 ∈ R m × d , W l ∈ R m × m , for 2 ≤ l ≤ L − 1 ,
W L ∈ R 1 × m , and θ = [ vec
[Equation]
Initialization .
For any l ∈ [ L − 1] , each entry of W l is drawn from the normal distribution N (0 , 2 m ) and W L is drawn from the normal distribution N (0 , 1 m ) .
Note that EE-Net at most has three networks f 1 , f 2 , f 3 .
We deﬁne them following the deﬁnition of f for brevity, although they may have different depth or width.
Then, we have the following theorem for EE-Net.
Recall that η 1 , η 2 are the learning rates for f 1 , f 2 ; K 1 is the number of iterations of gradient descent for f 1 in each round; and K 2 is the number of iterations for f 2 .
Theorem 1.
Let f 1 , f 2 follow the setting of f (Eq.
with the same width m and depth L .
Let
L 1 , L 2 be loss functions deﬁned in Algorithm 1.
Set f 3 as f 3 = f 1 + f 2 .
For any δ ∈ (0 , 1) , (cid:15) ∈ (0 , O ( 1 T )] , ρ ∈ (0 , O ( 1 L )] , suppose
[Equation]
Then, with probability at least 1 − δ over the initialization, the pseudo regret of EE-Net in T rounds satisﬁes
[Equation]
Comparison with existing works .
Under the similar assumptions in over-parameterized neural networks, the regret bounds complexity of NeuralUCB (Zhou et al., 2020) and NeuralTS (Zhang et al., 2021) both are
[Equation]
where H is the neural tangent kernel matrix (NTK)
(Jacot et al., 2018; Arora et al., 2019) and λ is a regularization parameter.
Similarly, in linear contextual bandits, Abbasi-Yadkori et al.
(2011) achieve
O ( d √ T log T ) and Li et al.
(2017) achieve O ( √ dT log T ) .
Remark 5.1.
Compared to NeuralUCB/TS, EE-Net roughly improves by a multiplicative factor of √ log T , because our proof of EE-Net is directly built on recent advances in convergence theory (Allen-Zhu et al., 2019) and generalization bound (Cao and Gu, 2019) of over-parameterized neural networks.
Instead, the analysis for NeuralUCB/TS contains three parts of approximation error by calculating the distances between the expected reward and ridge regression, ridge regression and
NTK, and NTK and network function.
Remark 5.2.
The regret bound of EE-Net does not have the effective dimension ˜ d or input dimension d .
˜ d or d may cause signiﬁcant error, when the determinant of H is extremely large or d > T .
The proof of Theorem 1 is in Appendix C and mainly based on the following generalization bound.
The bound results from an online-to-batch conversion while using convergence guarantees of deep learning optimization.
Lemma 5.1.
For any δ ∈ (0 , 1) , (cid:15) ∈ (0 , 1) , ρ ∈ (0 , O ( 1 L )) , suppose m, η 1 , η 2 , K 1 , K 2 satisfy the conditions in Eq.
(5.2) and ( x τ,i , r τ,i ) ∼ D , ∀ τ ∈
[ t ] , i ∈ [ n ] .
Let
[Equation]
max x t,i ,i
[Equation]
[Equation]
and r t is the corresponding reward, given ( x t,i , r t,i ) , i ∈ [ n ] .
Then, with probability at least (1 − δ ) over the random of the initialization, it holds that
[Equation]
where the expectation is also taken over ( θ 1 t − 1 , θ 2 t − 1 ) that are uniformly drawn from ( (cid:98) θ 1 τ , (cid:98) θ
[Equation]
) , τ
[Equation]
[ t − 1] .
Remark 5.3.
Lemma 5.1 provides a ﬁxed ˜ O ( 1 √ t ) -rate generalization bound for exploitationexploration networks f 1 , f 2 in contrast with the relative bound w.r.t.
the Neural Tangent Random
Feature (NTRF) benchmark (Cao and Gu, 2019).
We achieve this by working in the regression rather than classiﬁcation setting and utilizing the convergence guarantees for square loss (Allen-Zhu et al., 2019).
Note that the bound in Lemma 5.1 holds in the setting of bounded (possibly random) rewards r ∈ [0 , 1] instead of a ﬁxed function in the conventional classiﬁcation setting.
E XPERIMENTS
In this section, we evaluate EE-Net on four real-world datasets comparing with strong state-of-the-art baselines.
We ﬁrst present the setup of experiments, then show regret comparison and report ablation study.
Codes are available at 1 .
We use four real-world datasets: Mnist, Yelp, Movielens, and Disin , the details and settings of which are attached in Appendix A.
Baselines .
To comprehensively evaluate EE-Net, we choose 3 neural-based bandit algorithms, one linear and one kernelized bandit algorithms.
LinUCB (Li et al., 2010) explicitly assumes the reward is a linear function of arm vector and unknown user parameter and then applies the ridge regression and un upper conﬁdence bound to determine selected arm.
KernelUCB (Valko et al., 2013) adopts a predeﬁned kernel matrix on the reward space combined with a UCB-based exploration strategy.
Neural-Epsilon adapts the epsilon-greedy exploration strategy on exploitation network f 1 .
I.e., with probability 1 − (cid:15) , the arm is selected by x t = arg max i ∈ [ n ] f 1 ( x t,i ; θ 1 ) and with probability (cid:15) , the arm is chosen randomly.
NeuralUCB
uses
coming
UCB
based
NeuralTS
(Zhang et al., 2021) adopts the exploitation network f 1 to learn the reward function coming with an Thompson Sampling exploration strategy.
Note that we do not report results of LinTS and KernelTS in experiments, because of the limited space in ﬁgures, but LinTS and KernelTS have been signiﬁcantly outperformed by NeuralTS (Zhang et al., 2021).
Setup for EE-Net .
To compare fairly, for all the neural-based methods including EE-Net, the exploitation network f 1 is built by a 2-layer fully-connected network with 100 width.
For the exploration network f 2 , we use a 2-layer fully-connected network with 100 width as well.
For the decision maker f 3 , by comprehensively evaluate both linear and nonlinear functions, we found that the most effective approach is combining them together, which we call " hybrid decision maker ".
In detail, for rounds t ≤ 500 , f 3 is set as f 3 = f 2 + f 1 , and for t > 500 , f 3 is set as a neural network with two 20-width fully-connected layers.
Setting f 3 in this way is because the linear decision maker can maintain stable performance in each running (robustness) and the non-linear decision maker can further improve the performance (see details in Appendix B).
The hybrid decision maker can combine these two advantages together.
The conﬁgurations of all methods are attached in Appendix A.
Results .
Figure 2 and Figure 3 show the regret comparison on these four datasets.
EE-Net consistently outperforms all baselines across all datasets.
For LinUCB and KernelUCN, the simple linear reward function or predeﬁned kernel cannot properly formulate ground-truth reward function existed in real-world datasets.
In particular, on Mnist and Disin datasets, the correlations between rewards and arm feature vectors are not linear or some simple mappings.
Thus, LinUCB and KernelUCB barely exploit the past collected data samples and fail to select correct arms.
For neural-based bandit algorithms, the exploration probability of Neural-Epsilon is ﬁxed and difﬁcult to be adjustable.
Thus it is usually hard to make effective exploration.
To make exploration, NeuralUCB statistically calculates a gradient-based upper conﬁdence bound and NeuralTS draws each arm
’s predicted reward from a normal distribution where the standard deviation is computed by gradient.
However, the conﬁdence bound or standard deviation they calculated only consider the worst cases and thus may not be able represent the actual potential of each arm, and they cannot make "upward" and "downward" exploration properly.
Instead, EE-Net uses a neural network f 2 to learn each arm’s potential by neural network’s powerful representation ability.
Therefore, EE-Net can outperform these two state-of-the-art bandit algorithms.
Note that NeuralUCB/TS does need two parameters to tune UCB/TS according to different scenarios while EE-Net only needs to set up a neural network and automatically learns it.
Ablation Study .
In Appendix B, we conduct ablation study regarding the label function y of f 2 and the different setting of f 3 .
C ONCLUSION
In this paper, we propose a novel exploration strategy, EE-Net.
In addition to a neural network that exploits collected data in past rounds , EE-Net has another neural network to learn the potential gain compared to current estimation for exploration.
Then, a decision maker is built to make selections to further trade off between exploitation and exploration.
We demonstrate that EE-Net outperforms
NeuralUCB and NeuralTS both theoretically and empirically, becoming the new state-of-the-art exploration policy.
Acknowledgements: We are grateful to Shiliang Zuo and Yunzhe Qi for the valuable discussions in the revisions of EE-Net.
This research work is supported by National Science Foundation under Awards
No.
IIS-1947203, IIS-2002540, IIS-2137468, IIS-1908104, OAC- and DBI- and a grant from C3.ai.
The views and conclusions are those of the authors and should not be interpreted as representing the ofﬁcial policies of the funding agencies or the government.
A D ATASETS AND S ETUP
MNIST dataset .
MNIST is a well-known image dataset (LeCun et al., 1998) for the 10-class classiﬁcation problem.
Following the evaluation setting of existing works (Valko et al., 2013; Zhou et al., 2020; Zhang et al., 2021), we transform this classiﬁcation problem into bandit problem.
Consider an image x ∈ R d , we aim to classify it from 10 classes.
First, in each round, the image x is transformed into 10 arms and presented to the learner, represented by 10 vectors in sequence x 1 = ( x , 0 , . . . , 0 ) , x 2 = ( 0 , x , . . . , 0 ) , . . .
, x
10 = ( 0 , 0 , . . .
, x ) ∈ R 10 d .
The reward is deﬁned asif the index of selected arm matches the index of x ’s ground-truth class; Otherwise, the reward is 0 .
Yelp 2 and Movielens (Harper and Konstan, 2015) datasets .
Yelp is a dataset released in the Yelp dataset challenge, which consists of 4.7 million rating entries for 1 .
57 × 10 5 restaurants by 1 .million users.
MovieLens is a dataset consisting of 25 million ratings between 1 . 6 × 10 5 users and 6 × 10 4 movies.
We build the rating matrix by choosing the top 2000 users and toprestaurants(movies) and use singular-value decomposition (SVD) to extract a 10 -dimension feature vector for each user and restaurant(movie).
In these two datasets, the bandit algorithm is to choose the restaurants(movies) with bad ratings.
We generate the reward by using the restaurant(movie)’s gained stars scored by the users.
In each rating record, if the user scores a restaurant(movie) less than 2 stars (5 stars totally), its reward is 1 ; Otherwise, its reward is 0 .
In each round, we set 10 arms as follows: we randomly choose one with reward 1 and randomly pick the other 9 restaurants(movies) withrewards; then, the representation of each arm is the concatenation of corresponding user feature vector and restaurant(movie) feature vector.
Disin (Ahmed et al., 2018) dataset .
Disin is a fake news dataset on kaggle 3 including 12600 fake news articles and 12600 truthful news articles, where each article is represented by the text.
To transform the text into vectors, we use the approach (Fu and He, 2021) to represent each article by a 300-dimension vector.
Similarly, we form a 10-arm pool in each round, where 9 real news and 1 fake news are randomly selected.
If the fake news is selected, the reward is 1 ; Otherwise, the reward is 0 .
Conﬁgurations .
For LinUCB, following (Li et al., 2010), we do a grid search for the exploration constant α over (0 . 01 , 0 .
1 , 1) which is to tune the scale of UCB.
For KernelUCB (Valko et al., 2013), we use the radial basis function kernel and stop adding contexts after 1000 rounds, following (Valko et al., 2013; Zhou et al., 2020).
For the regularization parameter λ and exploration parameter ν in
KernelUCB, we do the grid search for λ over (0 . 1 , 1 , 10) and for ν over (0 . 01 , 0 . 1 , 1) .
For NeuralUCB and NeuralTS, following setting of (Zhou et al., 2020; Zhang et al., 2021), we use the exploiation network f 1 and conduct the grid search for the exploration parameter ν over (0 . 001 , 0 . 01 , 0 .
1 , 1) and for the regularization parameter
λ over (0 . 01 , 0 . 1 , 1) .
For NeuralEpsilon, we use the same neural network f 1 and do the grid search for the exploration probability (cid:15) over (0 . 01 , 0 . 1 , 0 . 2) .
For the neural bandits NeuralUCB/TS, following their setting, as they have expensive computation cost to store and compute the whole gradient matrix, we use a diagonal matrix to make approximation.
For all neural networks, we conduct the grid search for learning rate over (0 . 01 , 0 .
For all grid-searched parameters, we choose the best of them for the comparison and report the averaged results of 10 runs for all methods.
Create exploration samples for f 2 .
When the selected arm is not optimal in a round, the optimal arm must exist among the remaining arms, and thus the exploration consideration should be added to the remaining arms.
Based on this fact, we create additional samples for the exploration network f 2 in practice.
For example, in setting of binary reward, e.g., 0 or 1 reward, if the received reward r t = 0 while select x t , we add new train samples for f 2 , ( x t,i , c r ) for each i ∈
[ i ] ∩ x t,i (cid:54) = x t , where c r ∈ (0 , 1) usually is a small constant.
This measure can further improve the performance of EE-Net in our experiments.
B A BLATION S TUDY
In this section, we conduct ablation study regarding the label function y for exploration network fand seting of decision maker f 3 on two representative datasets Movielens and Mnist.
Label function y .
In this paper, we use y 1 = r − f 1 to measure the potential gain of an arm, as the label of f 2 .
Moreover, we provide other two intuitive form y 2 = | r − f 1 | and y 3 = ReLU ( r − f 1 ) .
Figure 4 shows the regret with different y , where "EE-Net" denotes our method with default y 1 , "EE-Net-abs" represents the one with y 2 and "EE-Net-ReLU" is with y 3 .
On Movielens and Mnist datasets, EE-Net slightly outperforms EE-Net-abs and EE-Net-ReLU.
In fact, y 1 can effectively represent the positive potential gain and negative potential gain, such that f 2 intends to score the arm with positive gain higher and score the arm with negative gain lower.
However, y 2 treats the positive/negative potential gain evenly, weakening the discriminative ability.
y 3 can recognize the positive gain while neglecting the difference of negative gain.
Therefore, y 1 usually is the most effective one for empirical performance.
Setting of f 3 .
f 3 can be set as an either linear function or non-linear function.
In the experiment, we test the simple linear function f 3 = f 1 + f 2 , denoted by "EE-Net-Lin", and a non-linear function represented by a 2-layer 20-width fully-connected neural network, denoted by "EE-Net-NoLin".
For the default hybrid setting, denoted by "EE-Net", when rounds t ≤ 500 , f 3 = f 1 + f 2 ; Otherwise, fis the neural network.
Figure 5 reports the regret with these three different modes.
EE-Net achieves the best performance with small standard deviation.
In contrast, EE-Net-NoLin obtains the worst performance and largest standard deviation.
However, notice that EE-Net-NoLin can achieve the best performance in certain running (the green shallow) but it is erratic.
Because in the begin phase, without enough training samples, EE-Net-NoLin strongly relies on the quality of collected samples.
With appropriate training samples, gradient descent can lead f 3 to global optimum.
On the other hand, with misleading training samples, gradient descent can deviate f 3 from global optimum.
Therefore,
EE-Net-NoLin shows very unstable performance.
In contrast, EE-Net-Lin is inspired by the UCB strategy, i.e., the exploitation plus the exploration, exhibiting stable performance.
To combine their advantages together, we propose the hybrid approach, EE-Net, achieving the best performance with strong stability.
C P ROOF OF T HEOREM
In this section
[Equation]
we provide the proof of Theorem 1 and related lemmas.
Proof.
For brevity
[Equation]
for the selected arm x t in round t
[Equation]
let h ( x t ) be its expected reward and x ∗ t = arg max x t,i ,i ∈ [ n ] h ( x t,i ) be the optimal arm in round t .
Let f 3 ( x ; θ t
[Equation]
Note that ( x t,i , r t,i ) ∼ D , for each i ∈ [ n ] .
Then, the expected regret of round t is given by
[Equation]
where I 1 is because f 3 ( x t ) =
max i ∈ [ n ] f 3 ( x t,i ) and f 3 ( x t ) − f 3 ( x ∗ t ) ≥ 0 and ( a ) introduces the additional parameters θ ∗ t − 1 =
( θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 ) which will be suitably chosen.
Because, for each i ∈ [ n ] , ( x t,i , r t,i ) ∼ D , applying Lemma C.1 and Corollary C.1, with probability at least (1 − δ ) over the randomness of initialization, for I 2 , I 5 , we have
[Equation]
I
[Equation]
where
[Equation]
(C
[Equation]
) and we apply the union bound over round τ, ∀ τ ∈ [ t ] to make Lemma C.1 and Corollary C.1 hold for each round τ, τ ∈ [ t ] .
For I 3 , I 4 , based on Lemma C.2, with probability at least 1 − δ , we have
I 3 , I 4 ≤
[Equation]
To sum up, with probability at least 1 − δ , we have
[Equation]
Then expected regret of T rounds is computed by
[Equation]
where I 2 is because (cid:80) Tt =1 1 √ t ≤ (cid:82) T 1 1 √ t dx + 1 = 2 √
T − 1 (Chlebus, 2009) and the bound of ξ 1 is due to the choice of m , i.e., since ξ 1 = (cid:101)
O
(1 /m 1 / 6 ) and m ≥ (cid:101) Ω( poly ( T )) , m can be chosen so that
Tξ 1 = (cid:101) O
( T/m 1 / 6 ) ≤ O (1) .
Then, when (cid:15) ≤ 1 /T , we have
[Equation]
As the choice of m
[Equation]
we have ξ ≤ O (1) .
Therefore, we have
[Equation]
The proof is completed.
Lemma C
[Equation]
[ Lemma 5.1 restated ] For any δ, (cid:15) ∈ (0 , 1) , ρ ∈ (0 , O ( 1 L )) , suppose m, η 1 , η 2 , K 1 , Ksatisfy the conditions in Eq.
(5.2) and ( x τ,i , r τ,i ) ∼ D , ∀ τ ∈
[ t ] , i ∈ [ n ] .
Let
[Equation]
and r t is the corresponding reward, given ( x t,i , r t,i ) , i ∈ [ n ] .
Then, with probability at least (1 − δ ) over the random of the initialization, it holds that
[Equation]
τ
[Equation]
−
[Equation]
where the expectation is also taken over ( θ 1 t − 1 , θ 2 t − 1 ) that are uniformly drawn from ( (cid:98) θ 1 τ , (cid:98) θ 2 τ ) , τ ∈ [ t − 1] .
Proof.
In this proof, we consider the collected data of up to round t − 1 , { x τ , r τ } t − 1 τ =1 , as the training dataset and then obtain a generalization bound for it, inspired by Cao and Gu (2019).
For convenience, we use x := x t,i , r :=
r t,i , noting that the same analysis holds for each i ∈ [ n ] .
Consider the exploration network f 2 , applying Lemma C.3.
With probability at least 1 − δ , for any τ ∈ [ t ] , we have
[Equation]
Similarly, applying Lemma C.3 again, with probability at least 1 − δ , for any t ∈ [ T ] , we have
[Equation]
C.
[Equation]
Because for any r ∼ D r , | r | ≤ 1 , with Eq.
(C.10) and (C.11), applying union bound, with probability at least (1 − 2 δ ) over the random initialization, we have
[Equation]
Noting that (C.12) is for x = x τ,i for a speciﬁc τ ∈ [ t ] , i ∈ [ n ] .
By union bound, (C.12) holds ∀ τ ∈
[ t ] , i ∈ [ n ] with probability at least (1 − ntδ ) .
For brevity, let f 2 ( x ; (cid:98) θ 2 τ ) represent
[Equation]
Recall that, for each τ ∈ [ t − 1] , (cid:98) θ 1 τ and (cid:98) θ 2 τ are the parameters training on { x τ (cid:48) , r τ (cid:48) } ττ (cid:48)
=1 according to Algorithm 1.
In round τ ∈ [ t ] , let x τ = arg max x τ,i ,i ∈
[ n ] [ f 1 ( x τ,i ; θ 1 τ − 1 ) + f 2 ( x τ,i ; θ 2 τ − 1 )] , given ( x τ,i , r τ,i ) ∼ D , i ∈ [ n ] .
Let r τ be the corresponding reward.
Let ( x (cid:48) τ,i , r (cid:48) τ,i ) ∼ D , i ∈ [ n ] be shadow samples from the same distribution and let x (cid:48)
τ = arg max x (cid:48) τ,i ,i ∈ [ n ] [ f
[Equation]
x
[Equation]
i ; θ
[Equation]
) + f 2 ( x (cid:48)
[Equation]
θ
[Equation]
1 )] , with r (cid:48) τ being the corresponding reward.
Then, we deﬁne
[Equation]
C.13)
Then, as ( x τ,i , r τ,i ) ∼ D , i ∈ [ n ] , based on the deﬁnition of ( x τ , r τ ) , we have
E [ V τ | F τ
[Equation]
1 ] =
[Equation]
) where F τ − 1 denotes the σ -algebra generated by the history H τ − 1 = { x τ
[Equation]
, r τ
[Equation]
} τ − 1 τ
[Equation]
=1 .
Moreover
[Equation]
we have
[Equation]
Since { V τ } tτ =1 is a martingale difference sequence, inspired by Lemma 1 in (Cesa-Bianchi et al., 2001), applying the Hoeffding-Azuma inequality, with probability at least 1 − 3 δ , we have
[Equation]
where I 1 =
according to (C.14) and I 2 is because of (C.12).
According to Algorithm
[Equation]
( θ 1 t − 1 , θ 2 t − 1 ) is uniformly drawn from { ( (cid:98) θ 1 τ , (cid:98) θ 2 τ ) } t − 1 τ =0 .
Thus, with probability 1 − 3 δ , we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
2 log(1 /δ ) t .
(C.16)
For I 3 , according to Lemma C.6, for any (cid:101) θ 2 satisfying (cid:107) (cid:101) θ 2 − θ 20 (cid:107) 2 ≤ O ( t 3 ρ √ m log m ) , with probability 1 − δ , we have t t (cid:88)
[Equation] =
[Equation]
t [Equation] (cid:12)(cid:12)(cid:12)
[Equation] (cid:17)(cid:12)(cid:12)(cid:12) (cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
+ O (cid:18) 3 L √ 2 t (cid:19) .
(C.17)
For I 4 , according to Lemma C.4 (1), these exists (cid:101) θ 2 satisfying (cid:107) (cid:101) θ 2 − θ 20 (cid:107) 2 ≤ O ( t 3 ρ √ m log m ) , with probability 1 − δ , such that
[Equation]
18) where I 5 follows by a direct application of Lemma C.4 (1) by deﬁning the loss L ( (cid:101) θ 2 ) =
[Equation]
≤ (cid:15) .
Combining Eq.(C.16), Eq.(C.17) and Eq.(C.18), with probability (1 − 5 δ ) we have
[Equation]
where the expectation over ( θ 1 t − 1 , θ 2 t − 1 ) that is uniformly drawn from { ( (cid:98) θ 1 τ , (cid:98) θ 2 τ ) } t − 1 τ =0 .
Then, applying union bound to t, n and rescaling the δ complete the proof.
Corollary C.1.
For any δ, (cid:15) ∈ (0 , 1) , ρ ∈ (0 , O ( 1 L )) , suppose m, η 1 , η 2 , K 1 , K 2 satisfy the conditions in Eq.
(5.2) and ( x τ,i , r τ,i ) ∼ D , ∀ τ ∈
[ t ] , i ∈ [ n ] .
For any τ ∈ [ t ] , let
[Equation]
and r ∗ τ is the corresponding reward, given ( x τ,i , r τ,i ) , i ∈ [ n ] .
Then, with probability at least (1 − δ ) over the random of the initialization, there exist θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 , s.t., (cid:107) θ 1 , ∗ t − 1 − θ 10 (cid:107) 2 ≤ O ( t 3 ρ √ m log m ) and (cid:107) θ 2 , ∗ t − 1 − θ 20 (cid:107) 2 ≤ O ( t 3 ρ √ m log m ) , such that
[Equation]
where the expectation is also taken over ( θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 ) that are uniformly drawn from ( (cid:98) θ 1 , ∗ τ , (cid:98) θ 2 , ∗ τ ) , τ ∈ [ t − 1] .
Proof.
This a direct corollary of Lemma C.1, given the optimal historical pairs { x ∗ τ , r ∗ τ } t − 1 τ =1 .
For brevity, let f 2 ( x ; (cid:98) θ 2 , ∗ τ ) represent f 2 (cid:16) φ (
[Equation]
Suppose that, for each τ ∈ [ t − 1] , (cid:98) θ 1 , ∗ τ and (cid:98) θ 2 , ∗ τ are the parameters training on { x ∗ τ (cid:48) , r ∗ τ (cid:48) } ττ (cid:48)
= according to Algorithm 1.
Note that these pairs { x ∗ τ (cid:48) , r ∗ τ (cid:48) } ττ (cid:48)
=1 are unknown to the algorithm we run, and the parameters ( (cid:98) θ 1 , ∗ τ , (cid:98) θ 2 , ∗ τ ) are not estimated.
However, for the analysis, it is sufﬁcient to show that there exist such parameters so that the conditional expectation of the error can be bounded.
In round τ ∈ [ t ] , let x ∗ τ = arg max x τ,i i ∈ [ n ] [ h ( x τ,i )] , given ( x τ,i , r τ,i ) ∼ D , i ∈ [ n ] .
Let r ∗ τ be the corresponding reward.
Let ( x (cid:48) τ,i , r (cid:48) τ,i ) ∼ D , i ∈ [ n ] be shadow samples from the same distribution and let x (cid:48)∗ τ = arg max x (cid:48)
τ,i ,i ∈ [ n ] h ( x (cid:48)
τ,i ) , with r (cid:48)∗ τ being the corresponding reward.
Then, we deﬁne
V
[Equation]
[Equation]
Then, as ( x τ,i , r τ,i ) ∼ D
[Equation]
i ∈ [ n ]
[Equation]
we have
[Equation]
= [Equation]
C [Equation] [Equation] where F τ − 1 denotes the σ -algebra generated by the history { x ∗ τ (cid:48) , r ∗ τ
[Equation]
} τ − 1 τ (cid:48)
=1 .
Therefore, { V τ } tτ =1 is a martingale difference sequence.
Similarly, applying the Hoeffding-Azuma inequality to V τ , with probability 1 − 3 δ , we have
[Equation]
t ; θ
[Equation]
∗ t −
[Equation]
(cid:17)
[Equation]
(cid:16) r (cid:48)∗ t − f 1 ( x (cid:48)∗ t ; θ 1 , ∗ t − 1 ) (cid:17)(cid:12)(cid:12)(cid:12)(cid:105) =
t t (cid:88) τ =
[Equation]
(cid:12)
[Equation]
(cid:98) θ 2 ,
[Equation]
−
[Equation]
(cid:17) − (cid:16) r (cid:48)∗ τ − f 1 ( x (cid:48)∗ τ ; (cid:98) θ 1 , ∗ τ − 1 ) (cid:17)(cid:12) (cid:12) (cid:12)(cid:105) ≤t t (cid:88) τ = (cid:12)(cid:12)(cid:12) f 2 (cid:16) x ∗ τ ; (cid:98) θ 2 , ∗ τ − 1 (cid:17) − (cid:16) r ∗ τ − f 1 ( x ∗ τ ; (cid:98) θ 1 , ∗ τ − 1 ) (cid:17)(cid:12)(cid:12)(cid:12) (cid:124) (cid:123)(cid:122) (cid:125) I+(1 + 2 ξ ) (cid:114) 2 log(1 /δ ) t . (C.23) For I 3 , according to Lemma C.6, for any (cid:101) θ 2 , ∗ satisfying (cid:107) (cid:101) θ 2 , ∗ − θ 20 (cid:107) 2 ≤ O ( t 3 ρ √ m log m ) , with probability 1 − δ , we have t
[Equation]
[Equation]
For I 4 , according to Lemma C.4 (1), these exists (cid:101) θ 2 , ∗ satisfying
[Equation]
θ
[Equation]
≤ O (
[Equation]
3 ρ √ m log m ) , with probability 1 − δ , such that
[Equation]
≤
[Equation]
t √
[Equation]
where I 5 follows by a direct application of Lemma C
[Equation]
4 (1) by deﬁning the loss L ( (cid:101) θ 2 , ∗ ) = 12 (cid:80) tτ = [Equation] f
[Equation]
x
[Equation]
θ
[Equation]
−
[Equation]
−
[Equation]
( x [Equation] (cid:98) θ
[Equation]
1 ) (cid:17)(cid:17) 2 ≤ (cid:15) .
Combining above inequalities
[Equation]
with probability (1 − 5 δ ) we have
[Equation]
Then, applying union bound to t, n and rescaling the δ complete the proof.
Lemma C.2.
Given δ, (cid:15) ∈ (0 , 1) , ρ ∈ (0 , O ( 1 L )) , suppose m, η 1 , η 2 , K 1 , K 2 satisfy the conditions in Eq.
Then, with probability at least 1 − δ , in each round t ∈ [ T ] , for any (cid:107) x (cid:107) 2 = 1 , we have
[Equation]
Proof.
According to Lemma C.4 (2), (cid:107) (cid:98) θ
[Equation]
[Equation]
[Equation]
2 ≤ O
[Equation]
[Equation] m log m ) , ∀ τ ∈ [ t ] .
Thus, we have [Equation]
First, based on Triangle inequality, for any (cid:107) x (cid:107) 2 = 1 , we have
[Equation]
where the last inequality is because of Lemma C.4 (3) and Lemma C.7.
Applying Lemma C.5 (1), for any x ∼ D , (cid:107) x (cid:107) 2 = 1 and (cid:107) θ
[Equation]
−
[Equation]
≤ O
[Equation]
m log m )
= w , we have
[Equation]
Similarly, we can use the same way to prove the lemmas for f 2 .
Lemma C.3.
Let f ( · ; (cid:98) θ t ) follow the stochastic gradient descent of f 1 or f 2 in Algorithm 1.
Suppose m, η 1 , η 2 satisfy the conditions in Eq.
With probability at least 1 − δ , for any x with (cid:107) x (cid:107) 2 =and t ∈ [ T ] , it holds that
[Equation]
Proof.
Considering an inequality | a − b | ≤ c , we have | a | ≤ | b | + c .
Let θ 0 be randomly initialized.
Then applying Lemma C.5 (1), for any x ∼ D , (cid:107) x (cid:107) 2 = 1 and (cid:107) (cid:98) θ t − θ 0 (cid:107) ≤ w , we have | f ( x ; (cid:98) θ t ) | ≤ | f ( x ; θ 0 ) | + |(cid:104) (cid:79) θ 0 f
( x i ; θ 0 ) , (cid:98) θ t − θ [-
-]0 (cid:105)|
[Equation] O ( L 2 (cid:112) m
[Equation] ( m )) (cid:107) (cid:98) θ t −[-θ 0 (cid:107) 2 w 
-]
[Equation] 0 (cid:107) 2 w 1 /≤ O (1) (cid:124) [Equation]
I
[Equation]
(C.32) where: I 0 is based on the Lemma C.4 (3); I 1 is an application of Cauchy–Schwarz inequality; I 2 is according to Lemma C.4 (2) and (3) in which (cid:98) θ t can be considered as one step gradient descent; I 3 is due to Lemma C.4 (2).
Then, the proof is completed.
Lemma C.4.
Given a constant 0 < (cid:15) < 1 , suppose m satisﬁes the conditions in Eq.
(5.2), the learning rate η = Ω( ρ poly ( t,n,L ) m ) , the number of iterations K = Ω(
[Equation]
· log (cid:15) − 1 ) .
Then, with probability at least 1 − δ , starting from random initialization θ 0 , (1) (Theorem 1 in (Allen-Zhu et al., 2019))
In round t ∈ [ T ] , given the collected data { x τ , r τ } t i = τ , the loss function is deﬁned as: L ( θ ) = 1 2 (cid:80) tτ
=
( f ( x τ ; θ ) − r τ ) 2 .
Then,
there exists (cid:101) θ satisfying (cid:107) (cid:101) θ − θ 0 (cid:107) 2 ≤ O (cid:16) t 3 ρ √ m log m (cid:17) , such that L ( (cid:101) θ ) ≤ (cid:15) in K = Ω( poly ( t,n,L ) ρ· log
[Equation]
iterations; (2) (Theorem 1 in (Allen-Zhu et al., 2019))
For any k ∈ [ K ] , it holds uniformly that (cid:107) θ ( k ) t − θ 0 (cid:107) 2 ≤ O (cid:16) t 3 ρ √ m log m (cid:17) ;
(3) Following the initialization, given (cid:107) x (cid:107) 2 = 1 , it holds that
[Equation]
where θ ( k ) t represents the parameters of f after k ∈ [ K ] iterations of gradient descent in round t .
Proof.
Note that the output dimension d in (Allen-Zhu et al., 2019) is removed because the output of network function in this paper always is a scalar.
For (1) and (2), the only different setting from (Allen-Zhu et al., 2019) is that the initialization of last layer W L ∼ N (0 , 2 m ) in this paper while
W L ∼ N (0 , 1 d ) in (Allen-Zhu et al., 2019).
Because d = 1 and m > d here, the upper bound in (Allen-Zhu et al., 2019) still holds for W L : with probability at least 1 − exp ( − Ω( m/L )) , (cid:107) W L (cid:107) F ≤ (cid:112) m/d .
Therefore, (1) and (2) still hold for the initialization of this paper.
For (3), based on Lemma 7.1 in Allen-Zhu et al.
(2019), we have | f ( x ; θ 0 )
| ≤ O (1) .
Denote by D the ReLU function.
For any l ∈ [ L ] ,
[Equation]
where the inequality is according to Lemma 7.2 in Allen-Zhu et al.
Therefore, we have (cid:107)
[Equation]
x
[Equation] θ [Equation] ≤ O ( L ) .
Lemma C.5 (Lemma 4.1, (Cao and Gu, 2019)) .
For any δ ∈ (0 , 1) , if w satisﬁes
[Equation]
then, with probability at least 1 − δ over randomness of θ 0 , for any t ∈ [ T ] , (cid:107) x (cid:107) 2 = 1 , and θ , θ (cid:48) satisfying (cid:107) θ − θ 0 (cid:107) 2 ≤ w and (cid:107) θ (cid:48) − θ 0 (cid:107) 2 ≤ w , it holds uniformly that [Equation]
[Equation]
Lemma C.6.
For any δ > 0 , suppose
[Equation]
(cid:17) .
Then, with probability at least 1 − δ , setting η 2 = Θ( t 5 δ 2 √ 2 m ) for algorithm 1, for any (cid:101) θ 2 satisfying
[Equation]
θ 2 − θ
[Equation]
2 ≤ O ( t 3 ρ √ m log m ) , it holds that
[Equation]
Proof.
This is a direct application of Lemma 4.3 in (Cao and Gu, 2019) by setting R =
t 3 ρ log m, (cid:15) =
LR √ 2 νt , and ν = ν (cid:48) R 2 , where ν (cid:48) is some small enough absolute constant.
We set L τ ( (cid:98) θ 2 τ − 1 ) =
[Equation] f 2 ( (cid:79) [Equation] 1 f 1 ; (cid:98) θ 2 τ − 1 ) − (cid:16) r τ − f 1 ( x τ ; (cid:98) θ 1 τ − 1 ) (cid:17)(cid:12)(cid:12)(cid:12) .
Based on Lemma C.4 (2), for any τ ∈ [ t ] , we have (cid:107) [Equation] θ
[Equation]
−
[Equation] θ [Equation] −
[Equation] ≤ [Equation] θ
[Equation]
(cid:107)
[Equation] + [Equation] θ
[Equation]
− (cid:98) θ
[Equation]
(cid:107) [Equation] ≤ O ( t 3 ρ √ m log m ) .
[Table]
With probability 1 − (cid:15) , x t = arg max x t,i ,i ∈ [ n ] f 1 ( x t,i ; θ 1 ) ; Otherwise, select x t randomly.
NeuralTS (Zhang et al., 2021)
For x t,i , ∀ i ∈ [ n ] , draw ˆ r t,i from N ( f 1 ( x t,i ; θ 1 ) , σ t,i 2 ) .
Then, select x t, ˆ i , ˆ i = arg max i ∈ [ n ] ˆ r t,i .
NeuralUCB (Zhou et al., 2020) [Table] = arg [-max x t,i ,i ∈ [ n ] (cid:0) f 
-][Table]
[Table] [Table] [Table]
[Table] EE-Net (Our approach) ∀ i ∈ [ n ] , compute f 1 ( x t,i ; θ 1 ) , f 2 (cid:0) (cid:79) θ 1 f 1 ( x t,i ; θ 1 );
θ 2 (cid:1) (Exploration Net).
Then x t = arg max x t,i i ∈ [ n ] f 3 ( f 1 , f 2 ; θ 3 ) .
Then, according to Lemma 4.3 in (Cao and Gu, 2019), then, for any (cid:101) θ 2 satisfying (cid:107) (cid:101) θ 2 − θ 20 (cid:107) 2 ≤ O ( t 3 ρ √ m log m ) , there exist a small enough absolute constant ν (cid:48) , such that t (cid:88) τ = L τ ( (cid:98) θ 2 τ − 1 ) ≤ t (cid:88) τ = [Table]
L [Table] ( (cid:101) θ 2 )
+ [Table]
(C.34) Then, replacing (cid:15) completes the proof. Lemma C.7 (Theorem 5, Allen-Zhu et al. For any δ ∈ (0 , 1) , if w satisﬁes that
O [Equation] then, with probability at least 1 − δ , for all (cid:107) θ − θ 0 (cid:107) 2 ≤ w , we have
[Equation]
D M OTIVATION OF E XPLORATION N ETWORK
In this section, we list one gradient-based UCB from existing works (Ban et al., 2021;
Zhou et al., 2020), which motivates our design of exploration network f 2 .
Let g ( x t ; θ t ) =
(cid:79) θ t f ( x t ; θ t ) .
Lemma D.1.
(Lemma 5.2 in (Ban et al., 2021)).
Given a set of context vectors { x t } Tt =1 and the corresponding rewards { r t } Tt =1 , E ( r t ) =
h ( x t ) for any x t ∈ { x t } Tt =1 .
Let f ( x t ; θ ) be the L -layers fully-connected neural network where the width is m , the learning rate is η , the number of iterations of gradient descent is K .
Then, there exist positive constants C 1 , C 2 , S , such that if
[Equation]
poly [Equation] n
[Equation]
then, with probability at least 1 − δ , for any x t ∈ { x t } Tt =1 , we have the following upper conﬁdence bound:
[Equation]
where
[Equation]
Note that g ( x t ; θ 0 ) is the gradient at initialization, which can be initialized as constants.
Therefore, the above UCB can be represented as the following form for exploitation network f 1 : | h ( x t,i ) − f 1 ( x t,i ; θ 1 t ) | ≤ Ψ( g ( x t ; θ t )) . [Table]
EE-Net has smaller approximation error.
Given an arm x , let f 1 ( x ) be the estimated reward and h ( x ) be the expected reward.
The exploration network f 2 in EE-Net is to learn h ( x ) − f 1 ( x ) , i.e., the residual between expected reward and estimated reward, which is the ultimate goal of making exploration.
There are advantages of using a network f 2 to learn h ( x ) − f 1 ( x ) in EE-Net, compared to giving a statistical upper bound for it such as NeuralUCB, (Ban et al., 2021), and NeuralTS (in
NeuralTS, the variance ν can be thought of as the upper bound).
For EE-Net, the approximation error for h ( x ) − f 1 ( x ) is caused by the genenalization error of the neural network (Lemma B.1.
in the manuscript).
In contrast, for NeuralUCB, (Ban et al., 2021), and NeuralTS, the approximation error for h ( x ) − f 1 ( x ) includes three parts.
The ﬁrst part is caused by ridge regression.
The second part of the approximation error is caused by the distance between ridge regression and Neural Tangent
Kernel (NTK).
The third part of the approximation error is caused by the distance between NTK and the network function.
Because they use the upper bound to make selections, the errors inherently exist in their algorithms.
By reducing the three parts of the approximation errors to only the neural network convergence error, EE-Net achieves tighter regret bound compared to them (improving by roughly √ log T ).
[Figure]
EE-Net has the ability to determine exploration direction .
The two types of exploration are described by Figure 6.
When the estimated reward is larger than the expected reward, i.e., h ( x ) − f 1 ( x ) < 0 , we need to do the ‘downward exploration’, i.e., lowering the exploration score of x to reduce its chance of being explored; when h ( x ) − f 1 ( x ) > 0 , we should do the ‘upward exploration’, i.e., raising the exploration score of x to increase its chance of being explored.
For EE-Net, f 2 is to learn h ( x ) − f 1 ( x ) .
When h ( x ) − f 1 ( x ) > 0 , f 2 ( x ) will also be positive to make the upward exploration.
When h ( x ) − f 1 ( x ) < 0 , f 2 ( x ) will be negative to make the downward exploration.
In contrast, NeuralUCB will always choose upward exploration, i.e., f 1 ( x ) + UCB ( x ) where UCB ( x ) is always positive.
In particular, when h ( x ) − f 1 ( x ) < 0 , NeuralUCB will further amplify the mistake.
NeuralTS will randomly choose upward or downward exploration for all cases, because it draws a sampled reward from a normal distribution where the mean is f 1 ( x ) and the variance ν is the upper bound.