Original: L EARNING N EURAL C ONTEXTUAL B ANDITS THROUGH P ERTURBED R EWARDS
Revised: L EARNING N EURAL C ONTEXTUAL B ANDITS THROUGH P ERTURBED R EWARDS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Thanks to the power of representation learning, neural contextual bandit algorithms demonstrate remarkable performance improvement against their classical counterparts.
Revised: Thanks to the power of representation learning, neural contextual bandit algorithms demonstrate remarkable performance improvement against their classical counterparts.

Original: But because their exploration has to be performed in the entire neural network parameter space to obtain nearly optimal regret, the resulting computational cost is prohibitively high.
Revised: But because their exploration has to be performed in the entire neural network parameter space to obtain nearly optimal regret, the resulting computational cost is prohibitively high.

Original: We perturb the rewards when updating the neural network to eliminate the need of explicit exploration and the corresponding computational overhead.
Revised: We perturb the rewards when updating the neural network to eliminate the need of explicit exploration and the corresponding computational overhead.

Original: We prove that a (cid:101) O
Revised: We prove that a (cid:101) O

Original: ( (cid:101) d √ T ) regret upper bound is still achievable under standard regularity conditions, where T is the number of rounds of interactions and (cid:101)
Revised: ( (cid:101) d √ T ) regret upper bound is still achievable under standard regularity conditions, where T is the number of rounds of interactions and (cid:101)

Original: d is the effective dimension of a neural tangent kernel matrix.
Revised: d is the effective dimension of a neural tangent kernel matrix.

Original: Extensive comparisons with several benchmark contextual bandit algorithms, including two recent neural contextual bandit models, demonstrate the effectiveness and computational efﬁciency of our proposed neural bandit algorithm.
Revised: Extensive comparisons with several benchmark contextual bandit algorithms, including two recent neural contextual bandit models, demonstrate the effectiveness and computational efﬁciency of our proposed neural bandit algorithm.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Contextual bandit is a well-formulated abstraction of many important real-world problems, including content recommendation (Li et al., 2010;
Revised: Contextual bandit is a well-formulated abstraction of many important real-world problems, including content recommendation (Li et al., 2010;

Original: Wu et al., 2016), online advertising (Schwartz et al., 2017; Nuara et al., 2018), and mobile health (Lei et al., 2017; Tewari & Murphy, 2017).
Revised: Wu et al., 2016), online advertising (Schwartz et al., 2017; Nuara et al., 2018), and mobile health (Lei et al., 2017; Tewari & Murphy, 2017).

Original: In such problems, an agent iteratively interacts with an environment to maximize its accumulated rewards over time.
Revised: In such problems, an agent iteratively interacts with an environment to maximize its accumulated rewards over time.

Original: Its essence is sequential decision-making under uncertainty.
Revised: Its essence is sequential decision-making under uncertainty.

Original: Because the reward from the environment for a chosen action (also referred to as an arm in literature) under each context is stochastic, a no-regret learning algorithm needs to explore the problem space for improved reward estimation, i.e., learning the mapping from an arm and its context 1 to the expected reward.
Revised: Because the reward from the environment for a chosen action (also referred to as an arm in literature) under each context is stochastic, a no-regret learning algorithm needs to explore the problem space for improved reward estimation, i.e., learning the mapping from an arm and its context 1 to the expected reward.

Original: Linear contextual bandit algorithms (Abbasi-Yadkori et al., 2011; Li et al., 2010), which assume the reward mapping is a linear function of the context vector, dominate the community’s attention in the study of contextual bandits.
Revised: Linear contextual bandit algorithms (Abbasi-Yadkori et al., 2011; Li et al., 2010), which assume the reward mapping is a linear function of the context vector, dominate the community’s attention in the study of contextual bandits.

Original: Though theoretically sound and practically effective, their linear reward mapping assumption is incompetent to capture possible complex non-linear relations between the context vector and reward.
Revised: Though theoretically sound and practically effective, their linear reward mapping assumption is incompetent to capture possible complex non-linear relations between the context vector and reward.

Original: This motivated the extended studies in parametric bandits, such as generalized linear bandits (Filippi et al., 2010; Faury et al., 2020) and kernelized bandits (Chowdhury & Gopalan, 2017; Krause & Ong, 2011).
Revised: This motivated the extended studies in parametric bandits, such as generalized linear bandits (Filippi et al., 2010; Faury et al., 2020) and kernelized bandits (Chowdhury & Gopalan, 2017; Krause & Ong, 2011).

Original: Recently, to unleash the power of representation learning, deep neural networks (DNN) have also been introduced to learn the underlying reward mapping directly.
Revised: Recently, to unleash the power of representation learning, deep neural networks (DNN) have also been introduced to learn the underlying reward mapping directly.

Original: In (Zahavy & Mannor, 2019; Riquelme et al., 2018; Xu et al., 2020), a deep neural network is applied to provide a feature mapping, and exploration is performed at the last layer.
Revised: In (Zahavy & Mannor, 2019; Riquelme et al., 2018; Xu et al., 2020), a deep neural network is applied to provide a feature mapping, and exploration is performed at the last layer.

Original: NeuralUCB (Zhou et al., 2020) and NeuralTS (Zhang et al., 2020) explore the entire neural network parameter space to obtain nearly optimal regret using the neural tangent kernel technique (Jacot et al., 2018).
Revised: NeuralUCB (Zhou et al., 2020) and NeuralTS (Zhang et al., 2020) explore the entire neural network parameter space to obtain nearly optimal regret using the neural tangent kernel technique (Jacot et al., 2018).

Original: These neural contextual bandit algorithms signiﬁcantly boosted empirical performance compared to their classical counterparts.
Revised: These neural contextual bandit algorithms signiﬁcantly boosted empirical performance compared to their classical counterparts.

Original: Nevertheless, a major practical concern of existing neural contextual bandit algorithms is their added computational cost when performing exploration.
Revised: Nevertheless, a major practical concern of existing neural contextual bandit algorithms is their added computational cost when performing exploration.

Original: Take the recently developed NeuralUCB and NeuralTS for example.
Revised: Take the recently developed NeuralUCB and NeuralTS for example.

Original: Their construction of the high-probability conﬁdence set for model exploration depends on the dimensionality of the network parameters and the learned context vectors’ representations, which is often very large for DNNs.
Revised: Their construction of the high-probability conﬁdence set for model exploration depends on the dimensionality of the network parameters and the learned context vectors’ representations, which is often very large for DNNs.

Original: For instance, a performing neural bandit solution often has the number of parameters in the order of 100 thousands (if not less).
Revised: For instance, a performing neural bandit solution often has the number of parameters in the order of 100 thousands (if not less).

Original: It is prohibitively expensive to compute inverse of the induced covariance matrix on such a huge number of parameters, as required by their construction of conﬁdence set.
Revised: It is prohibitively expensive to compute inverse of the induced covariance matrix on such a huge number of parameters, as required by their construction of conﬁdence set.

Original: As a result, approximations, e.g., only using the diagonal of the covariance matrix (Zhou et al., 2020; Zhang et al., 2020), are employed to make such algorithms operational in practice.
Revised: As a result, approximations, e.g., only using the diagonal of the covariance matrix (Zhou et al., 2020; Zhang et al., 2020), are employed to make such algorithms operational in practice.

Original: But there is no theoretical guarantee for such diagonal approximations, which directly leads to the gap between the theoretical and empirical performance of the neural bandit algorithms.
Revised: But there is no theoretical guarantee for such diagonal approximations, which directly leads to the gap between the theoretical and empirical performance of the neural bandit algorithms.

Original: In this work, to alleviate the computational overhead caused by the expensive exploration, we propose to eliminate explicit model exploration by learning a neural bandit model with perturbed rewards.
Revised: In this work, to alleviate the computational overhead caused by the expensive exploration, we propose to eliminate explicit model exploration by learning a neural bandit model with perturbed rewards.

Original: At each round of model update, we inject pseudo noise generated from a zero-mean Gaussian distribution to the observed reward history.
Revised: At each round of model update, we inject pseudo noise generated from a zero-mean Gaussian distribution to the observed reward history.

Original: With the induced randomization, sufﬁcient exploration is achieved when simply pulling the arm with the highest estimated reward.
Revised: With the induced randomization, sufﬁcient exploration is achieved when simply pulling the arm with the highest estimated reward.

Original: This brings in considerable advantage over existing neural bandit algorithms: no additional computational cost is needed to obtain no regret.
Revised: This brings in considerable advantage over existing neural bandit algorithms: no additional computational cost is needed to obtain no regret.

Original: We rigorously prove that with a high probability the algorithm obtains a (cid:101)
Revised: We rigorously prove that with a high probability the algorithm obtains a (cid:101)

Original: O
Revised: O

Original: ( (cid:101) d √ T ) regret, where (cid:101) d is the effective dimension of a neural tangent kernel matrix and T is the number of rounds of interactions.
Revised: ( (cid:101) d √ T ) regret, where (cid:101) d is the effective dimension of a neural tangent kernel matrix and T is the number of rounds of interactions.

Original: This result recovers existing regret bounds for the linear setting where the effective dimension equals to the input feature dimension.
Revised: This result recovers existing regret bounds for the linear setting where the effective dimension equals to the input feature dimension.

Original: Besides, our extensive empirical evaluations demonstrate the strong advantage in efﬁciency and effectiveness of our solution against a rich set of state-of-the-art contextual bandit solutions over both synthetic and real-world datasets.
Revised: Besides, our extensive empirical evaluations demonstrate the strong advantage in efﬁciency and effectiveness of our solution against a rich set of state-of-the-art contextual bandit solutions over both synthetic and real-world datasets.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Most recently, attempts have been made to incorporate DNNs with contextual bandit algorithms.
Revised: Most recently, attempts have been made to incorporate DNNs with contextual bandit algorithms.

Original: Several existing work study neural-linear bandits (Riquelme et al., 2018; Zahavy & Mannor, 2019), where exploration is performed on the last layer of the DNN.
Revised: Several existing work study neural-linear bandits (Riquelme et al., 2018; Zahavy & Mannor, 2019), where exploration is performed on the last layer of the DNN.

Original: Under the neural tangent kernel (NTK) framework (Jacot et al., 2018), NeuralUCB (Zhou et al., 2020) constructs conﬁdence sets with DNN-based random feature mappings to perform upper conﬁdence bound based exploration.
Revised: Under the neural tangent kernel (NTK) framework (Jacot et al., 2018), NeuralUCB (Zhou et al., 2020) constructs conﬁdence sets with DNN-based random feature mappings to perform upper conﬁdence bound based exploration.

Original: NeuralTS (Zhang et al., 2020) samples from the posterior distribution constructed with a similar technique.
Revised: NeuralTS (Zhang et al., 2020) samples from the posterior distribution constructed with a similar technique.

Original: However, as the exploration is performed in the induced random feature space, the added computational overhead is prohibitively high, which makes such solutions impractical.
Revised: However, as the exploration is performed in the induced random feature space, the added computational overhead is prohibitively high, which makes such solutions impractical.

Original: The authors suggested diagonal approximations of the resulting covariance matrix, which however leave the promised theoretical guarantees of those algorithms up in the air.
Revised: The authors suggested diagonal approximations of the resulting covariance matrix, which however leave the promised theoretical guarantees of those algorithms up in the air.

Original: Reward perturbation based exploration has been studied in a number of classical bandit models (Kveton et al., 2019a; 2020; 2019b).
Revised: Reward perturbation based exploration has been studied in a number of classical bandit models (Kveton et al., 2019a; 2020; 2019b).

Original: In a context-free k -armed bandit setting, Kveton et al.
Revised: In a context-free k -armed bandit setting, Kveton et al.

Original: (2019b) proposed to estimate each arm’s reward over a perturbed history and select the arm with the highest estimated reward at each round.
Revised: (2019b) proposed to estimate each arm’s reward over a perturbed history and select the arm with the highest estimated reward at each round.

Original: Such an arm selection strategy is proved to be optimistic with a sufﬁciently high probability.
Revised: Such an arm selection strategy is proved to be optimistic with a sufﬁciently high probability.

Original: Later this strategy has been extended to linear and generalized linear bandits (Kveton et al., 2019a; 2020).
Revised: Later this strategy has been extended to linear and generalized linear bandits (Kveton et al., 2019a; 2020).

Original: In (Kveton et al., 2020), the authors suggested its application to neural network models; but only some simple empirical evaluations were provided, without any theoretical justiﬁcations.
Revised: In (Kveton et al., 2020), the authors suggested its application to neural network models; but only some simple empirical evaluations were provided, without any theoretical justiﬁcations.

Original: Our work for the ﬁrst time provides a rigorous regret analysis of neural contextual bandits with the perturbation-based exploration: a sublinear regret bound is still achievable in terms of the number of interactions between the agent and environment.
Revised: Our work for the ﬁrst time provides a rigorous regret analysis of neural contextual bandits with the perturbation-based exploration: a sublinear regret bound is still achievable in terms of the number of interactions between the agent and environment.

Original:  N EURAL B ANDIT L EARNING WITH P ERTURBED R EWARDS
Revised:  N EURAL B ANDIT L EARNING WITH P ERTURBED R EWARDS

Original: We study the problem of contextual bandit with ﬁnite K arms, where each arm is associated with a d -dimensional context vector: x i ∈ R d for i ∈
Revised: We study the problem of contextual bandit with ﬁnite K arms, where each arm is associated with a d -dimensional context vector: x i ∈ R d for i ∈

Original: [ K ] .
Revised: [ K ] .

Original: At each round t ∈ [ T ] , the agent needs to select one of the arms, denoted as a t , and receives its reward r a t ,t , which is generated as r a t ,t = h ( x a t ) + η t .
Revised: At each round t ∈ [ T ] , the agent needs to select one of the arms, denoted as a t , and receives its reward r a t ,t , which is generated as r a t ,t = h ( x a t ) + η t .

Original: In particular, h ( x ) represents the unknown underlying reward mapping function satisfying 0 ≤ h ( x ) ≤ 1 for any x , and η t is an R -sub-Gaussian random variable that satisﬁes E [exp( µη t )] ≤ exp[ µ 2 R 2 ] for all µ ≥ 0 .
Revised: In particular, h ( x ) represents the unknown underlying reward mapping function satisfying 0 ≤ h ( x ) ≤ 1 for any x , and η t is an R -sub-Gaussian random variable that satisﬁes E [exp( µη t )] ≤ exp[ µ 2 R 2 ] for all µ ≥ 0 .

Original: The goal is to minimize pseudo regret over T rounds:
Revised: The goal is to minimize pseudo regret over T rounds:

Original: [Equation]
Revised: [Equation]

Original: where a ∗ is the optimal arm with the maximum expected reward.
Revised: where a ∗ is the optimal arm with the maximum expected reward.

Original: To deal with the potential non-linearity of h ( x ) and unleash the representation learning power of DNNs, we adopt a fully connected neural network f ( x ; θ ) to approximate h ( x ) : f ( x ; θ ) =
Revised: To deal with the potential non-linearity of h ( x ) and unleash the representation learning power of DNNs, we adopt a fully connected neural network f ( x ; θ ) to approximate h ( x ) : f ( x ; θ ) =

Original: √ m W L φ (cid:16) W L − 1 φ (cid:0) · · · φ ( W 1 x ) (cid:1)(cid:17) , where φ ( x ) =
Revised: √ m W L φ (cid:16) W L − 1 φ (cid:0) · · · φ ( W 1 x ) (cid:1)(cid:17) , where φ ( x ) =

Original: ReLU ( x ) , θ = [ vec ( W 1 ) , . .
Revised: ReLU ( x ) , θ = [ vec ( W 1 ) , . .

Original: , vec ( W L )] ∈
Revised: , vec ( W L )] ∈

Original: R p with p = m + md + m 2 ( L − 1) , and depth L ≥ 2 .
Revised: R p with p = m + md + m 2 ( L − 1) , and depth L ≥ 2 .

Original: Each hidden layer is assumed to have the same width (i.e., m ) for convenience in later analysis; but this does not affect the conclusion of our theoretical analysis.
Revised: Each hidden layer is assumed to have the same width (i.e., m ) for convenience in later analysis; but this does not affect the conclusion of our theoretical analysis.

Original: Existing neural bandit solutions perform explicit exploration in the entire model space (Zhou et al., 2020;
Revised: Existing neural bandit solutions perform explicit exploration in the entire model space (Zhou et al., 2020;

Original: Zhang et al., 2020; Zahavy & Mannor, 2019; Riquelme et al., 2018; Xu et al., 2020), which introduces prohibitive computational cost.
Revised: Zhang et al., 2020; Zahavy & Mannor, 2019; Riquelme et al., 2018; Xu et al., 2020), which introduces prohibitive computational cost.

Original: And oftentimes the overhead is so high that approximation has to be employed (Zhou et al., 2020; Zhang et al., 2020), which unfortunately breaks the theoretical promise of these algorithms.
Revised: And oftentimes the overhead is so high that approximation has to be employed (Zhou et al., 2020; Zhang et al., 2020), which unfortunately breaks the theoretical promise of these algorithms.

Original: In our proposed model, to eliminate such explicit model exploration in neural bandit, a randomization strategy is introduced in the neural network update.
Revised: In our proposed model, to eliminate such explicit model exploration in neural bandit, a randomization strategy is introduced in the neural network update.

Original: We name the resulting solution as Neural bandit with Perturbed Rewards, or NPR in short.
Revised: We name the resulting solution as Neural bandit with Perturbed Rewards, or NPR in short.

Original: In NPR, at round t , the neural model is learned with the t rewards perturbed with designed perturbations:
Revised: In NPR, at round t , the neural model is learned with the t rewards perturbed with designed perturbations:

Original: [Equation]
Revised: [Equation]

Original: where { γ ts } ts =1 ∼ N (0 , ν 2 ) are Gaussian random variables that are independently sampled in each round t , and ν is a hyper-parameter that controls the strength of perturbation (and thus the exploration) in NPR.
Revised: where { γ ts } ts =1 ∼ N (0 , ν 2 ) are Gaussian random variables that are independently sampled in each round t , and ν is a hyper-parameter that controls the strength of perturbation (and thus the exploration) in NPR.

Original: We use an l 2 -regularized square loss for model estimation, where the regularization centers at the randomly initialization θ 0 with the trade-off parameter λ .
Revised: We use an l 2 -regularized square loss for model estimation, where the regularization centers at the randomly initialization θ 0 with the trade-off parameter λ .

Original: The detailed procedure of NPR is given in Algorithm 1.
Revised: The detailed procedure of NPR is given in Algorithm 1.

Original: The algorithm starts by pulling all candidate arms once.
Revised: The algorithm starts by pulling all candidate arms once.

Original: This guarantees that for any arm, NPR is sufﬁciently optimistic compared to the true reward with respect to its approximation error (Lemma 4.4).
Revised: This guarantees that for any arm, NPR is sufﬁciently optimistic compared to the true reward with respect to its approximation error (Lemma 4.4).

Original: When all the K arms have been pulledfor once, the algorithm pulls the arm with the highest estimated reward, a t = argmax i f ( x i ; θ t − 1 ) .
Revised: When all the K arms have been pulled once, the algorithm pulls the arm with the highest estimated reward, a t = argmax i f ( x i ; θ t − 1 ) .

Original: Once received the feedback r a t ,t , the model perturbs the entire reward history so far via a freshly sampled noise sequence { γ ts } ts =1 , and updates the neural network by { ( x a s , r a s ,s + γ ts ) } ts =1 using gradient descent.
Revised: Once received the feedback r a t ,t , the model perturbs the entire reward history so far via a freshly sampled noise sequence { γ ts } ts =1 , and updates the neural network by { ( x a s , r a s ,s + γ ts ) } ts =1 using gradient descent.

Original: In the regret analysis in Section 4, we prove that the variance from the added { γ ts } s ∈ [ t ] will lead to the necessary optimism for exploration (Abbasi-Yadkori et al., 2011).
Revised: In the regret analysis in Section 4, we prove that the variance from the added { γ ts } s ∈ [ t ] will lead to the necessary optimism for exploration (Abbasi-Yadkori et al., 2011).

Original: We adopt gradient descent for analysis convenience, while stochastic gradient descent can also be used to solve the optimization problem with a similar theoretical guarantee based on recent works (AllenZhu et al., 2019; Zou et al., 2019).
Revised: We adopt gradient descent for analysis convenience, while stochastic gradient descent can also be used to solve the optimization problem with a similar theoretical guarantee based on recent works (AllenZhu et al., 2019; Zou et al., 2019).

Original: Compared with existing neural contextual bandit algorithms (Zhou et al., 2020; Zhang et al., 2020;
Revised: Compared with existing neural contextual bandit algorithms (Zhou et al., 2020; Zhang et al., 2020;

Original: Zahavy & Mannor, 2019; Riquelme et al., 2018; Xu et al., 2020), NPR does not need any added computation for model exploration, besides the regular neural network update.
Revised: Zahavy & Mannor, 2019; Riquelme et al., 2018; Xu et al., 2020), NPR does not need any added computation for model exploration, besides the regular neural network update.

Original: This greatly alleviates the overhead for the computation resources (both space and time) and makes NPR sufﬁciently general to be applied for practical problems.
Revised: This greatly alleviates the overhead for the computation resources (both space and time) and makes NPR sufﬁciently general to be applied for practical problems.

Original: More importantly, our theoretical analysis directly corresponds to its actual behavior when applied, as no approximation is needed.
Revised: More importantly, our theoretical analysis directly corresponds to its actual behavior when applied, as no approximation is needed.

Original:  R EGRET A NALYSIS
Revised:  R EGRET A NALYSIS

Original: In this section, we provide ﬁnite time regret analysis of NPR, where the time horizon T is set beforehand.
Revised: In this section, we provide ﬁnite time regret analysis of NPR, where the time horizon T is set beforehand.

Original: The theoretical analysis is built on the recent studies about the generalization of DNN models (Cao & Gu, 2020; 2019; Chen et al., 2019; Daniely, 2017; Arora et al., 2019), which illustrate that with (stochastic) gradient descent, the learned parameters of a DNN locate in a particular regime with the generalization error being characterized by the best function in the corresponding neural tangent kernel (NTK) space (Jacot et al., 2018).
Revised: The theoretical analysis is built on the recent studies about the generalization of DNN models (Cao & Gu, 2020; 2019; Chen et al., 2019; Daniely, 2017; Arora et al., 2019), which illustrate that with (stochastic) gradient descent, the learned parameters of a DNN locate in a particular regime with the generalization error being characterized by the best function in the corresponding neural tangent kernel (NTK) space (Jacot et al., 2018).

Original: We leave the background of NTK in the appendix and focus on the key steps of our proof in this section.
Revised: We leave the background of NTK in the appendix and focus on the key steps of our proof in this section.

Original: The analysis starts with the following lemma for the set of context vectors { x i } Ki =1 .
Revised: The analysis starts with the following lemma for the set of context vectors { x i } Ki =1 .

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: There exists a positive constant ¯ C such that for any δ ∈ (0 , 1) , when m ≥ ¯
Revised: There exists a positive constant ¯ C such that for any δ ∈ (0 , 1) , when m ≥ ¯

Original: CK
Revised: CK

Original: [Equation]
Revised: [Equation]

Original: /λ 40 , with probability at least 1 − δ , there exists a θ ∗ ∈ R p , for all i ∈
Revised: /λ 40 , with probability at least 1 − δ , there exists a θ ∗ ∈ R p , for all i ∈

Original: [ K ] ,
Revised: [ K ] ,

Original: [Equation]
Revised: [Equation]

Original: where H is the NTK matrix deﬁned on the context set and h =
Revised: where H is the NTK matrix deﬁned on the context set and h =

Original: ( h ( x 1 ) , . . .
Revised: ( h ( x 1 ) , . . .

Original: , h ( x K )) .
Revised: , h ( x K )) .

Original: Details of H can be found in the appendix.
Revised: Details of H can be found in the appendix.

Original: This lemma suggests that with a satisﬁed neural network width m , with high probability, the underlying reward mapping function can be approximated by a linear function over g ( x i ; θ 0 ) , parameterized by θ ∗ − θ 0 , where g ( x i ; θ 0 )
Revised: This lemma suggests that with a satisﬁed neural network width m , with high probability, the underlying reward mapping function can be approximated by a linear function over g ( x i ; θ 0 ) , parameterized by θ ∗ − θ 0 , where g ( x i ; θ 0 )

Original: = ∇ θ f ( x ; θ 

Revised: = ∇ θ f ( x ; θ

Original: ) ∈ R
Revised: ) ∈ R

Original: d is the gradient of the initial neural network.
Revised: d is the gradient of the initial neural network.

Original: We also deﬁne the covariance matrix A t at round t as,
Revised: We also deﬁne the covariance matrix A t at round t as,

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 is the l 2 regularization parameter in Eq (3.2).
Revised: where λ > 0 is the l 2 regularization parameter in Eq (3.2).

Original: Discussed before, there exist two kinds of noises in the training samples: the observation noise and the perturbation noise.
Revised: Discussed before, there exist two kinds of noises in the training samples: the observation noise and the perturbation noise.

Original: To analyze the effect of each noise in model estimation, two auxiliary least square solutions are introduced, A − 1 t ¯ b t and A − 1 t b t with
Revised: To analyze the effect of each noise in model estimation, two auxiliary least square solutions are introduced, A − 1 t ¯ b t and A − 1 t b t with

Original: [Equation]
Revised: [Equation]

Original: These two auxiliary solutions isolate the induced deviations: the ﬁrst least square solution only contains the deviation caused by observation noise { η s } t − 1 s =1 ; and the second least square solution has an extra deviation caused by { γ t − 1 s
Revised: These two auxiliary solutions isolate the induced deviations: the ﬁrst least square solution only contains the deviation caused by observation noise { η s } t − 1 s =1 ; and the second least square solution has an extra deviation caused by { γ t − 1 s

Original: [Equation]
Revised: [Equation]

Original: To analysis the estimation error of the neural network, we ﬁrst deﬁne the following events.
Revised: To analysis the estimation error of the neural network, we ﬁrst deﬁne the following events.

Original: At round t , we have event E t, 1 deﬁned as:
Revised: At round t , we have event E t, 1 deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 4.1 and the deﬁnition of ¯ b t , under event E t, 1 , the deviation caused by the observation noise { η s } t − 1 s =1 can be controlled by α t .
Revised: According to Lemma 4.1 and the deﬁnition of ¯ b t , under event E t, 1 , the deviation caused by the observation noise { η s } t − 1 s =1 can be controlled by α t .

Original: We also need event E t, 2 at round t deﬁned as
Revised: We also need event E t, 2 at round t deﬁned as

Original: E t, 2 = {∀ i ∈
Revised: E t, 2 = {∀ i ∈

Original: [ K ] : | f ( x t,i ; θ t − 1 )
Revised: [ K ] : | f ( x t,i ; θ t − 1 )

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) ( m ) is deﬁned as
Revised: where (cid:15) ( m ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: with constants { C (cid:15),i } 3 i =1 .
Revised: with constants { C (cid:15),i } 3 i =1 .

Original: Event E t, 2 considers the situation that the deviation caused by the added noise { γ t − 1 s } t − 1 s =1 is under control with parameter β t at round t , and the approximation error (cid:15) ( m ) is carefully tracked by properly setting the neural network width m .
Revised: Event E t, 2 considers the situation that the deviation caused by the added noise { γ t − 1 s } t − 1 s =1 is under control with parameter β t at round t , and the approximation error (cid:15) ( m ) is carefully tracked by properly setting the neural network width m .

Original: Next, the following lemmas show that with proper setting, given the history F t −= σ { x 1 , r 1 , x 2 , r 2 , . . .
Revised: Next, the following lemmas show that with proper setting, given the history F t −= σ { x 1 , r 1 , x 2 , r 2 , . . .

Original: , x t − 1 , r t − 1 , x t } , which is σ -algebra generated by the previously pulled arms and the observed rewards, events E t, 1 and E t, 2 happen with a high probability.
Revised: , x t − 1 , r t − 1 , x t } , which is σ -algebra generated by the previously pulled arms and the observed rewards, events E t, 1 and E t, 2 happen with a high probability.

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: For any t ∈ [ T ] , λ > 

Revised: For any t ∈ [ T ] , λ >

Original: and δ > 0 , with α t = (cid:113) R 2 log (cid:0) det( A t ) /
Revised: and δ > 0 , with α t = (cid:113) R 2 log (cid:0) det( A t ) /

Original: ( δ 2 det( λ I ))
Revised: ( δ 2 det( λ I ))

Original: (cid:1) + √ λS , we have P ( E t, 1 |F t − 1 ) ≥ 1 − δ .
Revised: (cid:1) + √ λS , we have P ( E t, 1 |F t − 1 ) ≥ 1 − δ .

Original: In particular, S denotes the upper bound of √ 2 h (cid:62) H − 1 h , which is a constant if the reward function h ( · ) belongs to the RKHS space induced by the neural tangent kernel.
Revised: In particular, S denotes the upper bound of √ 2 h (cid:62) H − 1 h , which is a constant if the reward function h ( · ) belongs to the RKHS space induced by the neural tangent kernel.

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: There exist positive constants { C i } 3 i =1 , such that with step size and the neural network satisfy that η = C 1 ( mλ + mLT ) − 1 , m ≥ C 2 √ λL − 3 / 2 [log( TKL 2 /δ )] 3 / 2 , and m [log m ] − 3 ≥
Revised: There exist positive constants { C i } 3 i =1 , such that with step size and the neural network satisfy that η = C 1 ( mλ + mLT ) − 1 , m ≥ C 2 √ λL − 3 / 2 [log( TKL 2 /δ )] 3 / 2 , and m [log m ] − 3 ≥

Original: C 3 max { TL 12 λ − 1 , T 7 λ − 8 L 18 ( λ + LT ) 6 , L 21 T 7 λ − 7 (1+ (cid:112) T/λ ) 6 } , given history F t − 1 , by choosing β t = σ √ 4 log t + 2 log K , we have P ( E t, 2 |F t − 1 ) ≥ 1 − t − 2 .
Revised: C 3 max { TL 12 λ − 1 , T 7 λ − 8 L 18 ( λ + LT ) 6 , L 21 T 7 λ − 7 (1+ (cid:112) T/λ ) 6 } , given history F t − 1 , by choosing β t = σ √ 4 log t + 2 log K , we have P ( E t, 2 |F t − 1 ) ≥ 1 − t − 2 .

Original: We leave the detailed proof in the appendix.
Revised: We leave the detailed proof in the appendix.

Original: According to Lemma 4.2 and Lemma 4.3, with more observations become available, thethe neural network based reward estimator f ( x ; θ t ) will gradually converge to the true expected reward h ( x ) .
Revised: According to Lemma 4.2 and Lemma 4.3, with more observations become available, the neural network based reward estimator f ( x ; θ t ) will gradually converge to the true expected reward h ( x ) .

Original: In the meantime, the variance of the added perturbation leads to reward overestimation, which will compensate the potential underestimation caused by the observation noise and the approximation error.
Revised: In the meantime, the variance of the added perturbation leads to reward overestimation, which will compensate the potential underestimation caused by the observation noise and the approximation error.

Original: The following lemma describes this anticoncentration behavior of the added noise { γ t − 1 s } t − 1 s =1 .
Revised: The following lemma describes this anticoncentration behavior of the added noise { γ t − 1 s } t − 1 s =1 .

Original: Lemma 4.4.
Revised: Lemma 4.4.

Original: For any t ∈ [ T ] , with ν = α t (1 − λλ − 1 K
Revised: For any t ∈ [ T ] , with ν = α t (1 − λλ − 1 K

Original: ( A K ))
Revised: ( A K ))

Original: − 1 / 2 , α t deﬁned in Lemma 4.2, (cid:15) ( m ) deﬁned in Lemma 4.3, given history F t − 1 , we have P (cid:0) E t, 3 ) =
Revised: − 1 / 2 , α t deﬁned in Lemma 4.2, (cid:15) ( m ) deﬁned in Lemma 4.3, given history F t − 1 , we have P (cid:0) E t, 3 ) =

Original: P ( f ( x t,i ; θ t − 1 ) ≥ h ( x t,i ) − (cid:15) ( m ) |F
Revised: P ( f ( x t,i ; θ t − 1 ) ≥ h ( x t,i ) − (cid:15) ( m ) |F

Original: t − 1 , E t, 1 (cid:1) ≥ (4 e √ π ) − 1 , where A K is the covariance matrix constructed after the initial
Revised: t − 1 , E t, 1 (cid:1) ≥ (4 e √ π ) − 1 , where A K is the covariance matrix constructed after the initial

Original: K -round arm pulling (line 8 to 10 in Algorithm 1) by Eq (4.2), and λ K ( A K ) represents the K -th largest eigenvalue of matrix A K .
Revised: K -round arm pulling (line 8 to 10 in Algorithm 1) by Eq (4.2), and λ K ( A K ) represents the K -th largest eigenvalue of matrix A K .

Original: With α t deﬁned in Lemma 4.2, and (cid:15) ( m ) and β t deﬁned in Lemma 4.3, we divide the arms into two groups.
Revised: With α t deﬁned in Lemma 4.2, and (cid:15) ( m ) and β t deﬁned in Lemma 4.3, we divide the arms into two groups.

Original: More speciﬁcally, we deﬁne the set of sufﬁciently sampled arms in round t as:
Revised: More speciﬁcally, we deﬁne the set of sufﬁciently sampled arms in round t as:

Original: [Equation]
Revised: [Equation]

Original: Accordingly, the set of undersampled arms is deﬁned as ¯Ω t =
Revised: Accordingly, the set of undersampled arms is deﬁned as ¯Ω t =

Original: [ K ] \ Ω t .
Revised: [ K ] \ Ω t .

Original: Note that by this deﬁnition, the best arm a ∗ belongs to ¯Ω t .
Revised: Note that by this deﬁnition, the best arm a ∗ belongs to ¯Ω t .

Original: In the following lemma, we show that at any time step t > K , the expected one-step regret is upper bounded in terms of the regret due to playing an undersampled arm a t ∈ ¯Ω t .
Revised: In the following lemma, we show that at any time step t > K , the expected one-step regret is upper bounded in terms of the regret due to playing an undersampled arm a t ∈ ¯Ω t .

Original: Lemma 4.5.
Revised: Lemma 4.5.

Original: With m , η , ν satisfying the conditions in Lemma 4.2, 4.3, 4.4, and (cid:15) ( m ) deﬁned in
Revised: With m , η , ν satisfying the conditions in Lemma 4.2, 4.3, 4.4, and (cid:15) ( m ) deﬁned in

Original: Lemma 4.3, with probability at least 1 − δ the one step regret of NPR is upper bounded,
Revised: Lemma 4.3, with probability at least 1 − δ the one step regret of NPR is upper bounded,

Original: [Equation]
Revised: [Equation]

Original: α t ) (cid:0) 1 + 2 / P [Equation] a t ∈ ¯Ω t ) (cid:1) (cid:107) g ( x t 
Revised: Furthermore, P ( a t ∈ ¯Ω t ) ≥ P t ( E t, 3 ) − P t ( ¯ E t, 2 ) .

Original: [Equation]

Original: a t ; θ 0 ) / √ m (cid:107) A − 1 t .

Original: Furthermore, P ( a t ∈ ¯Ω t ) ≥ P t ( E t, 3 ) − P t ( ¯ E t, 2 ) .

Original: To bound the cumulative regret of NPR, we also need the following lemma.
Revised: To bound the cumulative regret of NPR, we also need the following lemma.

Original: Lemma 4.6.
Revised: Lemma 4.6.

Original: With (cid:101) d
Revised: With (cid:101) d

Original: as the effective dimension of the NTK matrix H , we have
Revised: as the effective dimension of the NTK matrix H , we have

Original: [Equation]
Revised: [Equation]

Original: The effective dimension is ﬁrst proposed in (Valko et al., 2013) to analyze the kernelized contextual bandit, which roughly measures the number of directions in the kernel space where the data mostly lies.
Revised: The effective dimension is ﬁrst proposed in (Valko et al., 2013) to analyze the kernelized contextual bandit, which roughly measures the number of directions in the kernel space where the data mostly lies.

Original: The detailed deﬁniton of (cid:101) d can be found in Deﬁnition ?? in the appendix.
Revised: The detailed deﬁniton of (cid:101) d can be found in Deﬁnition B.3 in the appendix.

Original: Finally, with proper neural network setup, we provide an m -independent regret upper bound.
Revised: Finally, with proper neural network setup, we provide an m -independent regret upper bound.

Original: Theorem 4.7.
Revised: Theorem 4.7.

Original: Let (cid:101) d be the effective dimension of the NTK kernel matrix H .
Revised: Let (cid:101) d be the effective dimension of the NTK kernel matrix H .

Original: There exist positive constants C 1 and C r , such that for any δ ∈ (0 , 1) , when network structure and the step size for model update satisfy m ≥ poly ( T, L, K, λ − 1 , S − 1 , log(1 /δ )) , η = C 1 ( mTL + mλ ) − 1 and λ ≥ max { 1 , S − 2 } , with probability at least 1 − δ , the cumulative regret of NPR satisﬁes[Equation]
Revised: There exist positive constants C 1 and C r , such that for any δ ∈ (0 , 1) , when network structure and the step size for model update satisfy m ≥ poly ( T, L, K, λ − 1 , S − 1 , log(1 /δ )) , η = C 1 ( mTL + mλ ) − 1 and λ ≥ max { 1 , S − 2 } , with probability at least 1 − δ , the cumulative regret of NPR satisﬁes

Original: [Equation]
Revised: [Equation]

Original: ) where Γ = 44 e √ π (1 + (cid:113) (4 log T + 2 log K ) / (1 − λλ − 1 K ( A K ))) .
Revised: where Γ = 44 e √ π (1 + (cid:113) (4 log T + 2 log K ) / (1 − λλ − 1 K ( A K ))) .

Original: Proof.
Revised: Proof.

Original: By Lemma 4.2, E t, 1 holds for all t ∈
Revised: By Lemma 4.2, E t, 1 holds for all t ∈

Original: [ T ] with probability at least 1 − δ .
Revised: [ T ] with probability at least 1 − δ .

Original: Therefore, with probability at least 1 − δ , we have
Revised: Therefore, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: The second inequality is due to the analysis of one-step regret in Lemma 4.5.
Revised: The second inequality is due to the analysis of one-step regret in Lemma 4.5.

Original: According to Lemma 4.3, by choosing δ = σc t , with c t = √ 4 log t + 2 log K , P t ( ¯ E t, 2 ) ≤ t − 2 .
Revised: According to Lemma 4.3, by choosing δ = σc t , with c t = √ 4 log t + 2 log K , P t ( ¯ E t, 2 ) ≤ t − 2 .

Original: Hence, (cid:80) Tt =1 P t ( ¯ E t, 2 ) ≤ π 2 / 6 .
Revised: Hence, (cid:80) Tt =1 P t ( ¯ E t, 2 ) ≤ π 2 / 6 .

Original: Based on Lemma 4.4, with σ = α t (1 − λλ − 1 K ( A K )) − 1 / 2 , P t ( E t, 3 ) ≥ 1 4 e √ π .
Revised: Based on Lemma 4.4, with σ = α t (1 − λλ − 1 K ( A K )) − 1 / 2 , P t ( E t, 3 ) ≥ 1 4 e √ π .

Original: Therefore, 1 + 2 / ( P t ( E t, 3 ) − P t ( ¯ E t, 2 )) ≤ 44 e √ π .
Revised: Therefore, 1 + 2 / ( P t ( E t, 3 ) − P t ( ¯ E t, 2 )) ≤ 44 e √ π .

Original: By chaining all the inequalities, we have the cumulative regret of NPR upper bounded as,
Revised: By chaining all the inequalities, we have the cumulative regret of NPR upper bounded as,

Original: [Equation]
Revised: [Equation]

Original: where Γ = 44 e √ π (1 + (cid:113) (4 log T + 2 log K ) / (1 − λλ − 1 K ( A K )))
Revised: where Γ = 44 e √ π (1 + (cid:113) (4 log T + 2 log K ) / (1 − λλ − 1 K ( A K )))

Original: The second inequality is due to Lemma 4.6, and the bounds of events ¯ E t, 2 and E t, 3 .
Revised: The second inequality is due to Lemma 4.6, and the bounds of events ¯ E t, 2 and E t, 3 .

Original: By setting η = c ( mλ + mLT ) − 1 , and
Revised: By setting η = c ( mλ + mLT ) − 1 , and

Original: J = (1+ LT/λ )(log( C (cid:15), 2 )+log T (cid:112) TL/λ ) /c , C (cid:15), 2 (1 − ηmλ )
Revised: J = (1+ LT/λ )(log( C (cid:15), 2 )+log T (cid:112) TL/λ ) /c , C (cid:15), 2 (1 − ηmλ )

Original: J T (cid:112) TL/λ ≤ 1 .
Revised: J T (cid:112) TL/λ ≤ 1 .

Original: Then by choosing the satisﬁed m , C (cid:15), 1 m
Revised: Then by choosing the satisﬁed m , C (cid:15), 1 m

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: √
Revised: √

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: √ log m (1+ (cid:112) T/λ ) ≤ 2 / 3 .
Revised: √ log m (1+ (cid:112) T/λ ) ≤ 2 / 3 .

Original: R T can be further bounded by R T ≤
Revised: R T can be further bounded by R T ≤

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: K
Revised: K

Original: This completes the proof.
Revised: This completes the proof.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically evaluate the proposed neural bandit algorithm NPR against several state-of-the-art baselines, including: linear and neural UCB (LinUCB (Abbasi-Yadkori et al., 2011) and NeuralUCB (Zhou et al., 2020)), linear and neural Thompson Sampling
Revised: In this section, we empirically evaluate the proposed neural bandit algorithm NPR against several state-of-the-art baselines, including: linear and neural UCB (LinUCB (Abbasi-Yadkori et al., 2011) and NeuralUCB (Zhou et al., 2020)), linear and neural Thompson Sampling

Original: (LinTS (Agrawal &
Revised: (LinTS (Agrawal &

Original: Goyal, 2013) and NeuralTS (Zhang et al., 2020)), perturbed reward for linear bandits (LinFPL)
Revised: Goyal, 2013) and NeuralTS (Zhang et al., 2020)), perturbed reward for linear bandits (LinFPL)

Original: (Kveton et al., 2019a).
Revised: (Kveton et al., 2019a).

Original: To demonstrate the real impact of using diagonal approximation in conﬁdence set construction, we also include NeuralUCB and NeuralTS with diagonal approximation as suggested in their original papers for comparisons.
Revised: To demonstrate the real impact of using diagonal approximation in conﬁdence set construction, we also include NeuralUCB and NeuralTS with diagonal approximation as suggested in their original papers for comparisons.

Original: We evaluated all the algorithms on 1) synthetic dataset via simulation, 2) six K -class classiﬁcation datasets from UCI machine learning repository (Beygelzimer et al., 2011), and 3) two real-world datasets extracted from the social bookmarking web service Delicious and music streaming service
Revised: We evaluated all the algorithms on 1) synthetic dataset via simulation, 2) six K -class classiﬁcation datasets from UCI machine learning repository (Beygelzimer et al., 2011), and 3) two real-world datasets extracted from the social bookmarking web service Delicious and music streaming service

Original: LastFM (Wu et al., 2016).
Revised: LastFM (Wu et al., 2016).

Original: We implemented all the algorithms in PyTorch and performed all the experiments on a server equipped with Intel Xeon Gold 6230 2.10GHz CPU, 128G RAM, four
Revised: We implemented all the algorithms in PyTorch and performed all the experiments on a server equipped with Intel Xeon Gold 6230 2.10GHz CPU, 128G RAM, four

Original: NVIDIA GeForce RTX 2080Ti graphical cards.
Revised: NVIDIA GeForce RTX 2080Ti graphical cards.

Original: 5. E XPERIMENT ON SYNTHETIC DATASET
Revised: 5. E XPERIMENT ON SYNTHETIC DATASET

Original: In our simulation, we ﬁrst generate a size- K arm pool, in which each arm i is associated with a d -dimensional feature vector x i .
Revised: In our simulation, we ﬁrst generate a size- K arm pool, in which each arm i is associated with a d -dimensional feature vector x i .

Original: The contextual vectors are sampled uniformly at random from a unit ball.
Revised: The contextual vectors are sampled uniformly at random from a unit ball.

Original: We generate the rewards via the following two nonlinear functions:
Revised: We generate the rewards via the following two nonlinear functions:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where each element of Σ ∈ R d × d is randomly sampled from N (0 , 1)
Revised: where each element of Σ ∈ R d × d is randomly sampled from N (0 , 1)

Original: [Figure]
Revised: [Figure]

Original: and θ is randomly sampled from a unit ball.
Revised: and θ is randomly sampled from a unit ball.

Original: At each round t , only a subset of k arms out of the total K arms are sampled without replacement and disclosed to all algorithms for selection.
Revised: At each round t , only a subset of k arms out of the total K arms are sampled without replacement and disclosed to all algorithms for selection.

Original: The ground-truth reward r a is corrupted by Gaussian noise η = N (0 , ξ 2 ) before feeding back to the bandit algorithms.
Revised: The ground-truth reward r a is corrupted by Gaussian noise η = N (0 , ξ 2 ) before feeding back to the bandit algorithms.

Original: We ﬁxed the feature dimension d = 50 and the pool size K = 100 with k = 20 .
Revised: We ﬁxed the feature dimension d = 50 and the pool size K = 100 with k = 20 .

Original: ξ was set to 0.1 for both h 1 and h 2 .
Revised: ξ was set to 0.1 for both h 1 and h 2 .

Original: Cumulative regret is used to compare the performance of different algorithms.
Revised: Cumulative regret is used to compare the performance of different algorithms.

Original: Here the best arm is deﬁned as the one with the highest expected reward in the presented k candidate arms.
Revised: Here the best arm is deﬁned as the one with the highest expected reward in the presented k candidate arms.

Original: All the algorithms were executed up to 10000 rounds in simulation, and the averaged results over 10 runs are reported in Figure 1(a) to 1(e).
Revised: All the algorithms were executed up to 10000 rounds in simulation, and the averaged results over 10 runs are reported in Figure 1(a) to 1(e).

Original: For the neural bandit algorithms, we adopted a 3-layer neural network with m = 64 units in each hidden layer.
Revised: For the neural bandit algorithms, we adopted a 3-layer neural network with m = 64 units in each hidden layer.

Original: We did a grid search on the ﬁrst 1000 rounds for regularization parameter λ over { 10 − i } 4 i =0 , exploration parameter ν over { 10 − i } 4 i =0 , and step size η over { 10 − i } 3 i =0 .
Revised: We did a grid search on the ﬁrst 1000 rounds for regularization parameter λ over { 10 − i } 4 i =0 , and step size η over { 10 − i } 3 i =0 . We also searched for the concentration parameter δ so that the exploration parameter ν is equivalently searched over { 10 − i } 4 i =0 .

Original: The best set of hyper-parameters for each algorithm are applied for the full simulation over 10000 rounds, i.e., only that trial will be continued for the rest of 9000 rounds.
Revised: The best set of hyper-parameters for each algorithm are applied for the full simulation over 10000 rounds, i.e., only that trial will be continued for the rest of 9000 rounds.

Original: In Figure 1(a) and 1(b), we can observe that linear bandit algorithms clearly failed to learn nonlinear mapping and suffered much higher regret compared to the neural algorithms.
Revised: In Figure 1(a) and 1(b), we can observe that linear bandit algorithms clearly failed to learn nonlinear mapping and suffered much higher regret compared to the neural algorithms.

Original: NPR achieved similar performance as NeuralUCB and NeuralTS, which shows the effectiveness of our randomized exploration strategy in neural bandit learning.
Revised: NPR achieved similar performance as NeuralUCB and NeuralTS, which shows the effectiveness of our randomized exploration strategy in neural bandit learning.

Original: Figure 1(c) shows the average running time for all the neural bandit models in this experiment.
Revised: Figure 1(c) shows the average running time for all the neural bandit models in this experiment.

Original: With the same network structure, the time spent for model update was similar across different neural bandit models.
Revised: With the same network structure, the time spent for model update was similar across different neural bandit models.

Original: But we can clearly observe that NeuralUCB and NeuralTS took much more time in arm selection.
Revised: But we can clearly observe that NeuralUCB and NeuralTS took much more time in arm selection.

Original: When choosing an arm, NeuralUCB and NeuralTS, and their corresponding versions with diagonal approximation, need to repeatedly compute: 1) the inverse of the covariance matrix based on the pulled arms in history; and 2) the conﬁdence set of their reward estimation on each candidate arm.
Revised: When choosing an arm, NeuralUCB and NeuralTS, and their corresponding versions with diagonal approximation, need to repeatedly compute: 1) the inverse of the covariance matrix based on the pulled arms in history; and 2) the conﬁdence set of their reward estimation on each candidate arm.

Original: The size of the covariance matrix depends on the number of parameters in the neural network (e.g., 7460-by-7460 in this experiment).
Revised: The size of the covariance matrix depends on the number of parameters in the neural network (e.g., 7460-by-7460 in this experiment).

Original: With diagonal approximation, the computational cost is signiﬁcantly reduced.
Revised: With diagonal approximation, the computational cost is signiﬁcantly reduced.

Original: But, as mentioned before, there is no theoretical guarantee about the approximation’s impact on the model’s performance.
Revised: But, as mentioned before, there is no theoretical guarantee about the approximation’s impact on the model’s performance.

Original: Intuitively, the impact depends on how the full covariance matrix concentrates on its diagonal, e.g., whether the random feature mappings of the network on the pulled arms are correlated among their dimensions.
Revised: Intuitively, the impact depends on how the full covariance matrix concentrates on its diagonal, e.g., whether the random feature mappings of the network on the pulled arms are correlated among their dimensions.

Original: Unfortunately, this cannot be determined in advance.
Revised: Unfortunately, this cannot be determined in advance.

Original: Figure 1(a) and 1(b) show that the diagonal approximation leads to a signiﬁcant performance drop.
Revised: Figure 1(a) and 1(b) show that the diagonal approximation leads to a signiﬁcant performance drop.

Original: This conﬁrms our concern of such an approximation.
Revised: This conﬁrms our concern of such an approximation.

Original: In contrast, asshown in Figure 1(c), there is no extra computation overhead in NPR; in the meantime, it provides comparable performance to NeuralUCB and NeuralTS with a full covariance matrix.
Revised: In contrast, as

Original: [Figure]
Revised: [Figure]

Revised: shown in Figure 1(c), there is no extra computation overhead in NPR; in the meantime, it provides comparable performance to NeuralUCB and NeuralTS with a full covariance matrix.

Original: To further investigate the efﬁciency advantage of NPR, we reported the total running time for the neural bandit algorithms under different sizes of the arm pool and different structures of the neural network under the same simulation setting.
Revised: To further investigate the efﬁciency advantage of NPR, we reported the total running time for the neural bandit algorithms under different sizes of the arm pool and different structures of the neural network under the same simulation setting.

Original: With the same neural network structure, models with different sizes of candidate arms should share the same running time for model update.
Revised: With the same neural network structure, models with different sizes of candidate arms should share the same running time for model update.

Original: Hence,
Revised: Hence,

Original: Figure 1(d) reports how the size of arm pool affects the time spent on arm selection.
Revised: Figure 1(d) reports how the size of arm pool affects the time spent on arm selection.

Original: As we can ﬁnd, with more candidate arms, NeuralUCB, NeuralTS, and their corresponding diagonal approximations spent more time on arm selection, or more speciﬁcally on constructing the conﬁdence set of the reward estimation for each arm.
Revised: As we can ﬁnd, with more candidate arms, NeuralUCB, NeuralTS, and their corresponding diagonal approximations spent more time on arm selection, or more speciﬁcally on constructing the conﬁdence set of the reward estimation for each arm.

Original: With a ﬁxed size arm pool, the complexity of the neural network will strongly affect the time for computing the inverse of the covariance matrix.
Revised: With a ﬁxed size arm pool, the complexity of the neural network will strongly affect the time for computing the inverse of the covariance matrix.

Original: In Figure 1(e), the x-axis shows the structure for the hidden layers.
Revised: In Figure 1(e), the x-axis shows the structure for the hidden layers.

Original: The running time for NeuralUCB and NeuralTS signiﬁcantly increased with enlarged neural networks.
Revised: The running time for NeuralUCB and NeuralTS signiﬁcantly increased with enlarged neural networks.

Original: The running time of NPR stayed stable, across varying sizes of arm pools and the network complexity.
Revised: The running time of NPR stayed stable, across varying sizes of arm pools and the network complexity.

Original: This shows the strong advantage of
Revised: This shows the strong advantage of

Original: NPR in efﬁciency and effectiveness, especially for large-scale problems.
Revised: NPR in efﬁciency and effectiveness, especially for large-scale problems.

Original: 5. E XPERIMENT ON CLASSIFICATION DATASETS
Revised: 5. E XPERIMENT ON CLASSIFICATION DATASETS

Original: Following the setting in (Zhou et al., 2020), we evaluated the bandit algorithms on K -class classiﬁcation problem with six public benchmark datasets, including adult , mushroom , and shuttle from UCI Machine Learning Repository (Dua & Graff, 2017).
Revised: Following the setting in (Zhou et al., 2020), we evaluated the bandit algorithms on K -class classiﬁcation problem with six public benchmark datasets, including adult , mushroom , and shuttle from UCI Machine Learning Repository (Dua & Graff, 2017).

Original: Due to space limit, we report the results on these three datasets in this section, and leave the other three, letter , covertype , and MagicTelescope in the appendix.
Revised: Due to space limit, we report the results on these three datasets in this section, and leave the other three, letter , covertype , and MagicTelescope in the appendix.

Original: We adopted the disjoint model (Li et al., 2010) to build the context vectors for candidate arms: given an input instance with feature x ∈ R d of a k -class classiﬁcation problem, we constructed the context features for k candidate arms as: x 1 = ( x , 0 , . . . , 0 ) , . . .
Revised: We adopted the disjoint model (Li et al., 2010) to build the context vectors for candidate arms: given an input instance with feature x ∈ R d of a k -class classiﬁcation problem, we constructed the context features for k candidate arms as: x 1 = ( x , 0 , . . . , 0 ) , . . .

Original: , x k = ( 0 , . . .
Revised: , x k = ( 0 , . . .

Original: , 0 , x ) ∈ R d × k .
Revised: , 0 , x ) ∈ R d × k .

Original: The model receives reward 1 if it selects the correct class by pulling the corresponding arm, otherwise 0.
Revised: The model receives reward 1 if it selects the correct class by pulling the corresponding arm, otherwise 0.

Original: The cumulative regret measures the total mistakes made by the model over T rounds.
Revised: The cumulative regret measures the total mistakes made by the model over T rounds.

Original: Similar to the experiment in synthetic datasets, we adopted a 3-layer neural network with m = 64 units in each hidden layer.
Revised: Similar to the experiment in synthetic datasets, we adopted a 3-layer neural network with m = 64 units in each hidden layer.

Original: We used the same grid search setting as in our experiments on synthetic datasets.
Revised: We used the same grid search setting as in our experiments on synthetic datasets.

Original: Figure 2 shows the averaged results across 10 runs for 10000 rounds, except for mushroom which only contains 8124 data instances.
Revised: Figure 2 shows the averaged results across 10 runs for 10000 rounds, except for mushroom which only contains 8124 data instances.

Original: There are several key observations in Figure 2.
Revised: There are several key observations in Figure 2.

Original: First, the improvement of applying a neural bandit model depends on the nonlinearity of the dataset.
Revised: First, the improvement of applying a neural bandit model depends on the nonlinearity of the dataset.

Original: For example, the performance on mushoom and shuttle was highly boosted by the neural models, while the improvement on adult was limited.
Revised: For example, the performance on mushoom and shuttle was highly boosted by the neural models, while the improvement on adult was limited.

Original: Second, the impact of the diagonal approximation for NeuralUCB and NeuralTS also depends on the dataset.
Revised: Second, the impact of the diagonal approximation for NeuralUCB and NeuralTS also depends on the dataset.

Original: It did not hurt their performance too much on shuttle , while using the full covariance matrix led to much better performance on Mushroom .
Revised: It did not hurt their performance too much on shuttle , while using the full covariance matrix led to much better performance on Mushroom .

Original: Overall, NPR showed consistently better or at least comparable performance to the best neural bandit baselines in all six datasets.
Revised: Overall, NPR showed consistently better or at least comparable performance to the best neural bandit baselines in all six datasets.

Original: Such robust performance and high efﬁciency make it more advantageous in practice.
Revised: Such robust performance and high efﬁciency make it more advantageous in practice.

Original: 5. E XPERIMENT ON L AST FM & D ELICIOUS
Revised: 5. E XPERIMENT ON L AST FM & D ELICIOUS

Original: The LastFM dataset was extracted from the music streaming service website Last.fm, and the Delicious data set was extracted from the social bookmark sharing service website Delicious (Wu et al., 2016).
Revised: The LastFM dataset was extracted from the music streaming service website Last.fm, and the Delicious data set was extracted from the social bookmark sharing service website Delicious (Wu et al., 2016).

Original: In particular, the LastFM dataset contains 1,892 users and 17,632 items (artists).
Revised: In particular, the LastFM dataset contains 1,892 users and 17,632 items (artists).

Original: We treat the “listened artists” in each user as positive feedback.
Revised: We treat the “listened artists” in each user as positive feedback.

Original: The Delicious dataset contains 1,861 users and  69,226 items (URLs).
Revised: The Delicious dataset contains 1,861 users and [Figure] 69,226 items (URLs).

Original: We treat the bookmarked URLs for each user as positive feedback.
Revised: We treat the bookmarked URLs for each user as positive feedback.

Original: Following the standard setting (Cesa-Bianchi et al., 2013), we pre-processed these two datasets.
Revised: Following the standard setting (Cesa-Bianchi et al., 2013), we pre-processed these two datasets.

Original: For the item[Figure] features, we used all the tags associated with each item to create its TF-IDF feature vector.
Revised: For the item features, we used all the tags associated with each item to create its TF-IDF feature vector.

Original: Then
Revised: Then

Original: PCA was applied to reduce the dimensionality of the features.
Revised: PCA was applied to reduce the dimensionality of the features.

Original: In both datasets, we used the ﬁrstprinciple components to construct the item feature vectors.
Revised: In both datasets, we used the ﬁrstprinciple components to construct the item feature vectors.

Original: On the user side, we created user features by running node2vec (Grover & Leskovec, 2016) on the user relation graph extracted from the social network provided by the datasets.
Revised: On the user side, we created user features by running node2vec (Grover & Leskovec, 2016) on the user relation graph extracted from the social network provided by the datasets.

Original: Each of the users was represented with a 50-dimensional feature vector.
Revised: Each of the users was represented with a 50-dimensional feature vector.

Original: We generated the candidate pool as follows: we ﬁxed the size of candidate arm pool to k = 25 for each round; for each user, we picked one item from those non-zero payoff items according to the complete observations in the dataset, and randomly picked the other 24 from those zero-payoff items.
Revised: We generated the candidate pool as follows: we ﬁxed the size of candidate arm pool to k = 25 for each round; for each user, we picked one item from those non-zero payoff items according to the complete observations in the dataset, and randomly picked the other 24 from those zero-payoff items.

Original: We constructed the context vectors by concatenating the user and item feature vectors.
Revised: We constructed the context vectors by concatenating the user and item feature vectors.

Original: In our experiment, a 3-layer neural network with m = 128 units in each hidden layer was applied to learn the bandit model.
Revised: In our experiment, a 3-layer neural network with m = 128 units in each hidden layer was applied to learn the bandit model.

Original: And the same gird search was performed to ﬁnd the best set of hyper-parameters.
Revised: And the same gird search was performed to ﬁnd the best set of hyper-parameters.

Original: We performed the experiment with 10000 rounds.
Revised: We performed the experiment with 10000 rounds.

Original: Because of the high demand of GPU memory for the matrix inverse computation, we only compared NPR to NeuralUCB and
Revised: Because of the high demand of GPU memory for the matrix inverse computation, we only compared NPR to NeuralUCB and

Original: NeuralTS with the diagonal approximation.
Revised: NeuralTS with the diagonal approximation.

Original: We compared the cumulative reward from each algorithm in Figure 3.
Revised: We compared the cumulative reward from each algorithm in Figure 3.

Original: Besides the neural contextual bandit algorithms, we also include the best reported model on these two datasets, hLinUCB (Wang et al., 2016) for comparison.
Revised: Besides the neural contextual bandit algorithms, we also include the best reported model on these two datasets, hLinUCB (Wang et al., 2016) for comparison.

Original: To improve visibility, we normalized the cumulative reward by a random strategy’s cumulative reward (Wu et al., 2016), where the arm was randomly sampled from the candidate pool.
Revised: To improve visibility, we normalized the cumulative reward by a random strategy’s cumulative reward (Wu et al., 2016), where the arm was randomly sampled from the candidate pool.

Original: All the neural models showed the power to learn the potential non-linear reward mapping on these two real-world datasets.
Revised: All the neural models showed the power to learn the potential non-linear reward mapping on these two real-world datasets.

Original: NPR showed its advantage over the baselines (especially on LastFM).
Revised: NPR showed its advantage over the baselines (especially on LastFM).

Original: As NPR has no additional requirement on the computation resource (e.g., no need for the expensive matrix operation as in the baselines), we also experienced much shorter running in
Revised: As NPR has no additional requirement on the computation resource (e.g., no need for the expensive matrix operation as in the baselines), we also experienced much shorter running in

Original: NPR than the other two neural bandit models.
Revised: NPR than the other two neural bandit models.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Existing neural contextual bandit algorithms sacriﬁce computation efﬁciency for effective exploration in the entire neural network parameter space.
Revised: Existing neural contextual bandit algorithms sacriﬁce computation efﬁciency for effective exploration in the entire neural network parameter space.

Original: To eliminate the computational overhead caused by such an expensive operation, we appealed to randomization-based exploration and obtained the same level of learning outcome.
Revised: To eliminate the computational overhead caused by such an expensive operation, we appealed to randomization-based exploration and obtained the same level of learning outcome.

Original: No extra computational cost or resources are needed besides the regular neural network update.
Revised: No extra computational cost or resources are needed besides the regular neural network update.

Original: We proved that the proposed solution obtains (cid:101) O
Revised: We proved that the proposed solution obtains (cid:101) O

Original: ( (cid:101) d √ T ) regret for T rounds of interactions between the system and user.
Revised: ( (cid:101) d √ T ) regret for T rounds of interactions between the system and user.

Original: Extensive empirical results on both synthetic and real-world datasets support our theoretical analysis, and demonstrate the strong advantage of our model in efﬁciency and effectiveness, which suggest the potential of deploying powerful neural bandit model online in practice.
Revised: Extensive empirical results on both synthetic and real-world datasets support our theoretical analysis, and demonstrate the strong advantage of our model in efﬁciency and effectiveness, which suggest the potential of deploying powerful neural bandit model online in practice.

Original: Our empirical efﬁciency analysis shows that the key bottleneck of learning a neural model online is at the model update step.
Revised: Our empirical efﬁciency analysis shows that the key bottleneck of learning a neural model online is at the model update step.

Original: Our current regret analysis depends on (stochastic) gradient descent over the entire training set for model update in each round, which is prohibitively expensive.
Revised: Our current regret analysis depends on (stochastic) gradient descent over the entire training set for model update in each round, which is prohibitively expensive.

Original: We would like to investigate the possibility of more efﬁcient model update, e.g., online stochastic gradient descent or continual learning, and the corresponding effect on model convergence and regret analysis.
Revised: We would like to investigate the possibility of more efﬁcient model update, e.g., online stochastic gradient descent or continual learning, and the corresponding effect on model convergence and regret analysis.

Original: Currently, our analysis focuses on the K -armed setting, and it is important to extend our solution and analysis to inﬁnitely many arms in our future work.
Revised: Currently, our analysis focuses on the K -armed setting, and it is important to extend our solution and analysis to inﬁnitely many arms in our future work.