Original: A NTI -C ONCENTRATED C ONFIDENCE B ONUSES FOR S CALABLE E XPLORATION
Revised: A NTI -C ONCENTRATED C ONFIDENCE B ONUSES FOR S CALABLE E XPLORATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Intrinsic rewards play a central role in handling the exploration-exploitation trade-off when designing sequential decision-making algorithms, in both foundational theory and state-of-the-art deep reinforcement learning.
Revised: Intrinsic rewards play a central role in handling the exploration-exploitation trade-off when designing sequential decision-making algorithms, in both foundational theory and state-of-the-art deep reinforcement learning.

Original: The LinUCB algorithm, a centerpiece of the stochastic linear bandits literature, prescribes an elliptical bonus which addresses the challenge of leveraging shared information in large action spaces.
Revised: The LinUCB algorithm, a centerpiece of the stochastic linear bandits literature, prescribes an elliptical bonus which addresses the challenge of leveraging shared information in large action spaces.

Original: This bonus scheme cannot be directly transferred to high-dimensional exploration problems, however, due to the computational cost of maintaining the inverse covariance matrix of action features.
Revised: This bonus scheme cannot be directly transferred to high-dimensional exploration problems, however, due to the computational cost of maintaining the inverse covariance matrix of action features.

Original: We introduce anti-concentrated conﬁdence bounds for efﬁciently approximating the elliptical bonus, using an ensemble of regressors trained to predict random noise from policy network-derived features.
Revised: We introduce anti-concentrated conﬁdence bounds for efﬁciently approximating the elliptical bonus, using an ensemble of regressors trained to predict random noise from policy network-derived features.

Original: Using this approximation, we obtain stochastic linear bandit algorithms which obtain ˜ O
Revised: Using this approximation, we obtain stochastic linear bandit algorithms which obtain ˜ O

Original: ( d √ T ) regret bounds for poly( d ) ﬁxed actions.
Revised: ( d √ T ) regret bounds for poly( d ) ﬁxed actions.

Original: We develop a practical variant for deep reinforcement learning that is competitive with contemporary intrinsic reward heuristics on Atari benchmarks
Revised: We develop a practical variant for deep reinforcement learning that is competitive with contemporary intrinsic reward heuristics on Atari benchmarks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Optimism in the face of uncertainty (OFU) is a ubiquitous algorithmic principle for online decisionmaking in bandit and reinforcement learning problems.
Revised: Optimism in the face of uncertainty (OFU) is a ubiquitous algorithmic principle for online decisionmaking in bandit and reinforcement learning problems.

Original: Broadly, optimistic decision-making algorithms augment their reward models with a bonus (or intrinsic reward ) proportional to their uncertainty about an action’s outcome, ideally balancing exploration and exploitation.
Revised: Broadly, optimistic decision-making algorithms augment their reward models with a bonus (or intrinsic reward ) proportional to their uncertainty about an action’s outcome, ideally balancing exploration and exploitation.

Original: A vast literature is dedicated to developing and analyzing the theoretical guarantees of these algorithms (Lattimore & Szepesvári, 2020).
Revised: A vast literature is dedicated to developing and analyzing the theoretical guarantees of these algorithms (Lattimore & Szepesvári, 2020).

Original: In fundamental settings such as stochastic multi-armed and linear bandits, optimistic algorithms are known to enjoy minimax-optimal regret bounds.
Revised: In fundamental settings such as stochastic multi-armed and linear bandits, optimistic algorithms are known to enjoy minimax-optimal regret bounds.

Original: In modern deep reinforcement learning, many approaches to exploration have been developed with the same principle of optimism, with most empirical successes coming from uncertainty-based intrinsic reward modules (Burda et al., 2018b; Pathak et al., 2017; Osband et al., 2016).
Revised: In modern deep reinforcement learning, many approaches to exploration have been developed with the same principle of optimism, with most empirical successes coming from uncertainty-based intrinsic reward modules (Burda et al., 2018b; Pathak et al., 2017; Osband et al., 2016).

Original: Such bonuses can be very useful, with prior work demonstrating impressive results on a wide array of challenging exploration problems.
Revised: Such bonuses can be very useful, with prior work demonstrating impressive results on a wide array of challenging exploration problems.

Original: Several of these methods draw inspiration from theoretical work on multiarmed bandits, using ideas like count-based exploration bonuses.
Revised: Several of these methods draw inspiration from theoretical work on multiarmed bandits, using ideas like count-based exploration bonuses.

Original: However, a related body of work on linear bandits provides tools for extending exploration bonuses to large but structured action spaces, a paradigm which may be appropriate for deep reinforcement learning.
Revised: However, a related body of work on linear bandits provides tools for extending exploration bonuses to large but structured action spaces, a paradigm which may be appropriate for deep reinforcement learning.

Original: The Linear UCB (LinUCB) algorithm (Auer, 2002; Dani et al., 2008; Li et al., 2010; Abbasi-Yadkori et al., 2011) is attractive in this setting because it enjoys minimax-optimal statistical guarantees.
Revised: The Linear UCB (LinUCB) algorithm (Auer, 2002; Dani et al., 2008; Li et al., 2010; Abbasi-Yadkori et al., 2011) is attractive in this setting because it enjoys minimax-optimal statistical guarantees.

Original: To obtain these, LinUCB leverages a so-called elliptical bonus, the computation of which requires maintaining an inverse covariance matrix over action features.
Revised: To obtain these, LinUCB leverages a so-called elliptical bonus, the computation of which requires maintaining an inverse covariance matrix over action features.

Original: The principal challenge in generalizing the elliptical potential used in bandits to the deep setting lies in computing and storing this object.
Revised: The principal challenge in generalizing the elliptical potential used in bandits to the deep setting lies in computing and storing this object.

Original: Due to the moving internal representation of the policy network, and the number of parameters used to compose it, a naive implementation of the LinUCB algorithm would require remembering all of the agent’s experience and constantly recomputing and inverting this matrix, which is likely extremely large.
Revised: Due to the moving internal representation of the policy network, and the number of parameters used to compose it, a naive implementation of the LinUCB algorithm would require remembering all of the agent’s experience and constantly recomputing and inverting this matrix, which is likely extremely large.

Original: Clearly, such an approach is too computationally intensive to be useful.
Revised: Clearly, such an approach is too computationally intensive to be useful.

Original: As we discuss in the next section, several works have used neural features for LinUCB-style bonuses, but all require an inversion of this sort, limiting their ability to scale to high dimensions.
Revised: As we discuss in the next section, several works have used neural features for LinUCB-style bonuses, but all require an inversion of this sort, limiting their ability to scale to high dimensions.

Original: Towards bridging foundational algorithms with empirical frontiers, we develop a scalable strategy for computing LinUCB’s elliptical bonus, enabling us to investigate its effectiveness as an intrinsic reward in deep reinforcement learning.
Revised: Towards bridging foundational algorithms with empirical frontiers, we develop a scalable strategy for computing LinUCB’s elliptical bonus, enabling us to investigate its effectiveness as an intrinsic reward in deep reinforcement learning.

Original: We use an ensemble of least-squares regressors to approximate these bonuses without explicitly maintaining the covariance matrix or its inverse.
Revised: We use an ensemble of least-squares regressors to approximate these bonuses without explicitly maintaining the covariance matrix or its inverse.

Original: Our algorithm is both theoretically principled and computationally tractable, and we demonstrate that its empirical performance on Atari games is often competitive with popular baselines.
Revised: Our algorithm is both theoretically principled and computationally tractable, and we demonstrate that its empirical performance on Atari games is often competitive with popular baselines.

Original: 1. O UR CONTRIBUTIONS
Revised: 1. O UR CONTRIBUTIONS

Original: We propose the use of anti-concentrated conﬁdence bounds (ACB) to efﬁciently approximate the
Revised: We propose the use of anti-concentrated conﬁdence bounds (ACB) to efﬁciently approximate the

Original: LinUCB bonus.
Revised: LinUCB bonus.

Original: ACB estimates per-action elliptical conﬁdence intervals by regressing random targets on policy features.
Revised: ACB estimates per-action elliptical conﬁdence intervals by regressing random targets on policy features.

Original: It anti-concentrates these bonuses by taking a maximum over the predictions from an ensemble of these regressors.
Revised: It anti-concentrates these bonuses by taking a maximum over the predictions from an ensemble of these regressors.

Original: First, we introduce ACB in the basic stochastic linear bandit setting.
Revised: First, we introduce ACB in the basic stochastic linear bandit setting.

Original: We show that these bonuses provably approximate LinUCB’s elliptical potential; thus, optimistic exploration with ACB directly inherits standard analysis techniques for LinUCB.
Revised: We show that these bonuses provably approximate LinUCB’s elliptical potential; thus, optimistic exploration with ACB directly inherits standard analysis techniques for LinUCB.

Original: We derive near-optimal high-probability regret bounds for ACB, when the size of the action space is polynomial in the action feature dimension.
Revised: We derive near-optimal high-probability regret bounds for ACB, when the size of the action space is polynomial in the action feature dimension.

Original: We derive sufﬁcient ensemble sizes for the special cases of multi-armed bandits and ﬁxed actions, as well as the general case of changing actions.
Revised: We derive sufﬁcient ensemble sizes for the special cases of multi-armed bandits and ﬁxed actions, as well as the general case of changing actions.

Original: These follow from lower tail bounds for the maximum of independent Gaussians; we conjecture that they are improvable using more sophisticated analytical tools.
Revised: These follow from lower tail bounds for the maximum of independent Gaussians; we conjecture that they are improvable using more sophisticated analytical tools.

Original: The main contribution of this work is empirical: we ﬁnd that ACB provides a viable exploration bonus for deep reinforcement learning.
Revised: The main contribution of this work is empirical: we ﬁnd that ACB provides a viable exploration bonus for deep reinforcement learning.

Original: After deﬁning a suitable nonlinear analogue using action features from the policy network, we demonstrate that the intrinsic rewards produced by ACB are competitive with those from state-of-the-art algorithms in deep RL on a variety of Atari benchmarks (Figure 1).
Revised: After deﬁning a suitable nonlinear analogue using action features from the policy network, we demonstrate that the intrinsic rewards produced by ACB are competitive with those from state-of-the-art algorithms in deep RL on a variety of Atari benchmarks (Figure 1).

Original: To the best of our knowledge, our work is the ﬁrst to scalably study bonuses from the linear bandits literature in these deep reinforcement learning settings.
Revised: To the best of our knowledge, our work is the ﬁrst to scalably study bonuses from the linear bandits literature in these deep reinforcement learning settings.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Linear bandits.
Revised: Linear bandits.

Original: Stochastic linear bandits were ﬁrst introduced by Abe & Long (1999).
Revised: Stochastic linear bandits were ﬁrst introduced by Abe & Long (1999).

Original: Optimistic algorithms are fundamental in this setting (Auer, 2002;
Revised: Optimistic algorithms are fundamental in this setting (Auer, 2002;

Original: Dani et al., 2008; Li et al., 2010; Rusmevichientong & Tsitsiklis, 2010; Abbasi-Yadkori et al., 2011), and provide minimax-optimal regret bounds.
Revised: Dani et al., 2008; Li et al., 2010; Rusmevichientong & Tsitsiklis, 2010; Abbasi-Yadkori et al., 2011), and provide minimax-optimal regret bounds.

Original: Several works are concerned with designing more scalable optimism-based algorithms, with a focus on empirical bandit settings (as opposed to deep RL).
Revised: Several works are concerned with designing more scalable optimism-based algorithms, with a focus on empirical bandit settings (as opposed to deep RL).

Original: Jun et al.
Revised: Jun et al.

Original: (2017) consider streaming conﬁdence bound estimates (obtaining per-iteration update costs independent of t ) in the generalized linear bandits model, but still incur the cost of maintaining the inverse covariance matrix via online Newton step updates.
Revised: (2017) consider streaming conﬁdence bound estimates (obtaining per-iteration update costs independent of t ) in the generalized linear bandits model, but still incur the cost of maintaining the inverse covariance matrix via online Newton step updates.

Original: Ding et al.
Revised: Ding et al.

Original: (2021) give strong regret guarantees for epoch-batched SGD with Thompson sampling, under more stringent assumptions (i.i.d. action features, and a “diversity” condition to induce strong convexity).
Revised: (2021) give strong regret guarantees for epoch-batched SGD with Thompson sampling, under more stringent assumptions (i.i.d. action features, and a “diversity” condition to induce strong convexity).

Original: Korda et al.
Revised: Korda et al.

Original: (2015) demonstrate that variance-reduced algorithms succeed in practice for estimating the LinUCB bonus.
Revised: (2015) demonstrate that variance-reduced algorithms succeed in practice for estimating the LinUCB bonus.

Original: Most closely related to our work, Kveton et al.
Revised: Most closely related to our work, Kveton et al.

Original: (2019b;a); Ishfaq et al.
Revised: ; Ishfaq et al.

Original: (2021) study perturbed regressions for exploration, obtaining similar regret bounds to ours (including the suboptimal √ log A factor in the regret bounds).
Revised: (2021) study perturbed regressions for exploration, obtaining similar regret bounds to ours (including the suboptimal √ log A factor in the regret bounds).

Original: The LinPHE algorithm in (Kveton et al., 2019a) uses a single random regressor’s
Revised: The LinPHE algorithm in (Kveton et al., 2019) uses a single random regressor’s

Original: constant-probability anti-concentration to approximate the elliptical bonus.
Revised: constant-probability anti-concentration to approximate the elliptical bonus.

Original: In work concurrent to ours, the main algorithm proposed by Ishfaq et al.
Revised: In work concurrent to ours, the main algorithm proposed by Ishfaq et al.

Original: (2021) also takes a maximum over random regressors, and plugs their analysis into downstream end-to-end results for linear
Revised: (2021) also takes a maximum over random regressors, and plugs their analysis into downstream end-to-end results for linear

Original: MDPs.
Revised: MDPs.

Original: The algorithms presented in these works resemble the “always-rerandomizing” variant of
Revised: The algorithms presented in these works resemble the “always-rerandomizing” variant of

Original: ACB (Algorithm 1), which is still not suitable for large-scale deep RL.
Revised: ACB (Algorithm 1), which is still not suitable for large-scale deep RL.

Original: Our work introduces the “lazily-rerandomizing” and “never-rerandomizing” variants in an effort to close this gap, and presents corresponding open theoretical problems.
Revised: Our work introduces the “lazily-rerandomizing” and “never-rerandomizing” variants in an effort to close this gap, and presents corresponding open theoretical problems.

Original: Unlike the aforementioned works, we consider our primary contribution as empirical, with the deep learning variant of ACB (Algorithm 2) offering performance comparable to commonly used deep
Revised: Unlike the aforementioned works, we consider our primary contribution as empirical, with the deep learning variant of ACB (Algorithm 2) offering performance comparable to commonly used deep

Original: RL bonuses on a wide array of Atari benchmarks without sacriﬁcing theoretical transparency.
Revised: RL bonuses on a wide array of Atari benchmarks without sacriﬁcing theoretical transparency.

Original: The resulting algorithmic choices deviate somewhat from the bandit version (as well as LinPHE/LSVIPHE), towards making ACB a viable drop-in replacement for typical bonuses used in deep RL.
Revised: The resulting algorithmic choices deviate somewhat from the bandit version (as well as LinPHE/LSVIPHE), towards making ACB a viable drop-in replacement for typical bonuses used in deep RL.

Original: The use of ensembles in this work bears a resemblance to Monte Carlo techniques for posterior sampling (Thompson, 1933; Agrawal & Goyal, 2013).
Revised: The use of ensembles in this work bears a resemblance to Monte Carlo techniques for posterior sampling (Thompson, 1933; Agrawal & Goyal, 2013).

Original: This is a starting point for many empiricallymotivated exploration algorithms, including for deep RL (see below), but theory is limited.
Revised: This is a starting point for many empiricallymotivated exploration algorithms, including for deep RL (see below), but theory is limited.

Original: (Lu & Van Roy, 2017) analyze an ensemble-based approximation of Thompson sampling (without considering a max over the ensemble), and obtain a suboptimal regret bound scaling with A log A .
Revised: (Lu & Van Roy, 2017) analyze an ensemble-based approximation of Thompson sampling (without considering a max over the ensemble), and obtain a suboptimal regret bound scaling with A log A .

Original: Exploration in deep reinforcement learning.
Revised: Exploration in deep reinforcement learning.

Original: In deep reinforcement learning, most popular approaches use predictability as a surrogate for familiarity.
Revised: In deep reinforcement learning, most popular approaches use predictability as a surrogate for familiarity.

Original: The Intrinsic Curiosity Module (ICM), for example, does this by training various neural machinery to predict proceeding states from current state-action pairs (Pathak et al., 2017).
Revised: The Intrinsic Curiosity Module (ICM), for example, does this by training various neural machinery to predict proceeding states from current state-action pairs (Pathak et al., 2017).

Original: The L 2 error of this prediction is used as a bonus signal for the agent.
Revised: The L 2 error of this prediction is used as a bonus signal for the agent.

Original: The approach can be viewed as a modiﬁcation to Intelligent Adaptive Curiosity, but relying on a representation that is trained to encourage retaining only causal information in transition dynamics (Oudeyer et al., 2007).
Revised: The approach can be viewed as a modiﬁcation to Intelligent Adaptive Curiosity, but relying on a representation that is trained to encourage retaining only causal information in transition dynamics (Oudeyer et al., 2007).

Original: Stadie et al.
Revised: Stadie et al.

Original: (2015) instead use a representation learned by an autoencoder.
Revised: (2015) instead use a representation learned by an autoencoder.

Original: A large-scale study of these “curiosity-based” approaches can be found in Burda et al.
Revised: A large-scale study of these “curiosity-based” approaches can be found in Burda et al.

Original: (2018a).
Revised: (2018a).

Original: Similar to these algorithms, Random Network Distillation (RND) relies on the inductive bias of a random and ﬁxed network, where the exploration bonus is computed as the prediction error between this random network and a separate model trained to mimic its outputs (Burda et al., 2018b).
Revised: Similar to these algorithms, Random Network Distillation (RND) relies on the inductive bias of a random and ﬁxed network, where the exploration bonus is computed as the prediction error between this random network and a separate model trained to mimic its outputs (Burda et al., 2018b).

Original: In tabular settings, count-based bonuses are often used to encourage exploration (Strehl & Littman, 2008).
Revised: In tabular settings, count-based bonuses are often used to encourage exploration (Strehl & Littman, 2008).

Original: This general approach has been extended to the deep setting as well, where states might be high-dimensional and infrequently visited (Bellemare et al., 2016).
Revised: This general approach has been extended to the deep setting as well, where states might be high-dimensional and infrequently visited (Bellemare et al., 2016).

Original: Algorithms of this type usually rely on some notion of density estimation to group similar states (Ostrovski et al., 2017; Tang et al., 2016; Martin et al., 2017).
Revised: Algorithms of this type usually rely on some notion of density estimation to group similar states (Ostrovski et al., 2017; Tang et al., 2016; Martin et al., 2017).

Original: Other work proposes more traditional uncertainty metrics to guide the exploration process.
Revised: Other work proposes more traditional uncertainty metrics to guide the exploration process.

Original: For example, Variational Intrinsic Control (Gregor et al., 2016) uses a variational notion of uncertainty while
Revised: For example, Variational Intrinsic Control (Gregor et al., 2016) uses a variational notion of uncertainty while

Original: Exploration via Disagreement (Pathak et al., 2019) and Bootstrapped DQN (Osband et al., 2016) model uncertainty over the predictive variance of an ensemble.
Revised: Exploration via Disagreement (Pathak et al., 2019) and Bootstrapped DQN (Osband et al., 2016) model uncertainty over the predictive variance of an ensemble.

Original: More broadly, various approximations of Thompson sampling (Thompson, 1933) have been used to encourage exploration in deep reinforcement learning (Osband et al., 2013; Guez et al., 2012; Zhang et al., 2020; Strens, 2000;
Revised: More broadly, various approximations of Thompson sampling (Thompson, 1933) have been used to encourage exploration in deep reinforcement learning (Osband et al., 2013; Guez et al., 2012; Zhang et al., 2020; Strens, 2000;

Original: Henaff et al., 2019).
Revised: Henaff et al., 2019).

Original: These methods have analogues in the related space of active learning, where ensemble and variational estimates of uncertainty are widely used (Gal et al., 2017; Beluch et al., 2018).
Revised: These methods have analogues in the related space of active learning, where ensemble and variational estimates of uncertainty are widely used (Gal et al., 2017; Beluch et al., 2018).

Original: Separate from these, there has been some work intended to more-directly deliver UCB-inspired bonuses to reinforcement learning (Zhou et al., 2020; Nabati et al., 2021; Zahavy & Mannor, 2019; Bai et al., 2021).
Revised: Separate from these, there has been some work intended to more-directly deliver UCB-inspired bonuses to reinforcement learning (Zhou et al., 2020; Nabati et al., 2021; Zahavy & Mannor, 2019; Bai et al., 2021).

Original: These efforts have various drawbacks, however, with many requiring matrix inversions, limiting their ability to scale, or simply not being competitive with more conventional
Revised: These efforts have various drawbacks, however, with many requiring matrix inversions, limiting their ability to scale, or simply not being competitive with more conventional

Original: deep
Revised: deep

Original: RL approaches.
Revised: RL approaches.

Original: ACB removes the covariance matrix inversion requirement, and as we demonstrate in Section A.2, is competitive with RND and ICM on several Atari benchmarks.
Revised: ACB removes the covariance matrix inversion requirement, and as we demonstrate in Section A.2, is competitive with RND and ICM on several Atari benchmarks.

Original: 5 Timesteps 1e
Revised: 5 Timesteps 1e

Original: C u m u l a t
Revised: C u m u l a t

Original: i v e R e g r e t 1e2 OLS ensemble updates
Revised: i v e R e g r e t 1e2 OLS ensemble updates

Original: M=

Revised: M=

Original: M=
Revised: M=

Original: M=4 M=
Revised: M=4 M=

Original: M=
Revised: M=

Original: M=32 M=
Revised: M=32 M=

Original: M=128 M=[-
-]
Revised: M=128 M=[-
-]

Original:  Timesteps 1e       6 1e2 SGD ensemble updates
Revised: LinUCB     5 Timesteps 1e       6 1e2 SGD ensemble updates

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we provide a brief review and establish notation for the stochastic linear bandit setting and the LinUCB algorithm.
Revised: In this section, we provide a brief review and establish notation for the stochastic linear bandit setting and the LinUCB algorithm.

Original: For a more comprehensive treatment, see Agarwal et al.
Revised: For a more comprehensive treatment, see Agarwal et al.

Original: ; Lattimore & Szepesvári (2020).
Revised: ; Lattimore & Szepesvári (2020).

Original: At round t = 1 , . . .
Revised: At round t = 1 , . . .

Original: , T , the learner is given an action set A t ⊆
Revised: , T , the learner is given an action set A t ⊆

Original: A with features { x t,a ∈ R d } a ∈A t , chooses an action a t ∈
Revised: A with features { x t,a ∈ R d } a ∈A t , chooses an action a t ∈

Original: A t , and receives reward r t ( a t ) :
Revised: A t , and receives reward r t ( a t ) :

Original: = (cid:104) x t,a t , θ ∗ (cid:105) + ε t , where ε t is a martingale difference sequence.
Revised: = (cid:104) x t,a t , θ ∗ (cid:105) + ε t , where ε t is a martingale difference sequence.

Original: We use A to denote max t |A t | .
Revised: We use A to denote max t |A t | .

Original: The performance of the learner is quantiﬁed by its regret,
Revised: The performance of the learner is quantiﬁed by its regret,

Original: [Equation]
Revised: [Equation]

Original: This setting encompasses two commonly-considered special cases:
Revised: This setting encompasses two commonly-considered special cases:

Original: • Fixed actions: the action sets A t are the same for all t .
Revised: • Fixed actions: the action sets A t are the same for all t .

Original: • Multi-armed bandits: ﬁxed actions, with A t =
Revised: • Multi-armed bandits: ﬁxed actions, with A t =

Original: [ d ] and x t,a = e a .
Revised: [ d ] and x t,a = e a .

Original: Here, θ ∗ is the vector of per-arm means.
Revised: Here, θ ∗ is the vector of per-arm means.

Original: An OFU-based algorithm plays at round t
Revised: An OFU-based algorithm plays at round t

Original: [Equation]
Revised: [Equation]

Original: where ˆ r t ( a ) is an estimate for the reward r t ( a ) , and bonus t ( a ) is selected to be a valid upper conﬁdence bound for ˆ r t ( a ) , such that  r t ( a ) ≤ ˆ r t ( a ) +bonus t ( a ) with high probability.
Revised: where ˆ r t ( a ) is an estimate for the reward r t ( a ) , and bonus t ( a ) is selected to be a valid upper conﬁdence bound for ˆ r t ( a ) , such that E [ r t ( a )] ≤ ˆ r t ( a ) + bonus t ( a ) with high probability.

Original: The LinUCB algorithm uses ˆ r t ( a ) :
Revised: The LinUCB algorithm uses ˆ r t ( a ) :

Original: = (cid:68) x t,a , ˆ θ t (cid:69) , where ˆ θ t = arg min θ ∈ R d (cid:80) t − 1 τ =1 ( (cid:104) x τ,a τ , θ (cid:105) − r i ( a i ))
Revised: = (cid:68) x t,a , ˆ θ t (cid:69) , where ˆ θ t = arg min θ ∈ R d (cid:80) t − 1 τ =1 ( (cid:104) x τ,a τ , θ (cid:105) − r i ( a i ))

Original: 2 + λ (cid:107)
Revised: 2 + λ (cid:107)

Original: θ (cid:107)is the(cid:96) 2 -regularized least-squares regressor for the reward, and sets bonus t ( a )
Revised: θ (cid:107)is the 2 -regularized least-squares regressor for the reward, and sets bonus t ( a )

Original: = β (cid:113) x
Revised: = β (cid:113) x

Original: (cid:62) t,a ˜ Σ − 1 t x t,a , where ˜Σ t := λI + (cid:80) t − 1 τ =1 x t,a t x (cid:62) t,a t .
Revised: (cid:62) t,a ˜ Σ − 1 t x t,a , where ˜Σ t := λI + (cid:80) t − 1 τ =1 x t,a t x (cid:62) t,a t .

Original: In an online learning context, this is known as the elliptical potential (Cesa-Bianchi & Lugosi, 2006; Carpentier et al., 2020).
Revised: In an online learning context, this is known as the elliptical potential (Cesa-Bianchi & Lugosi, 2006; Carpentier et al., 2020).

Original: We will quantify the theoretical performance of ACB by comparing it to LinUCB.
Revised: We will quantify the theoretical performance of ACB by comparing it to LinUCB.

Original: Theorem 1 gives a high-probability regret bound for LinUCB, which is optimal up to logarithmic factors:
Revised: Theorem 1 gives a high-probability regret bound for LinUCB, which is optimal up to logarithmic factors:

Original: Theorem 1 (LinUCB regret bound; Theorem 5.3, (Agarwal et al., 2019)) .
Revised: Theorem 1 (LinUCB regret bound; Theorem 5.3, (Agarwal et al., 2019)) .

Original: If || θ ∗ || , || x t,a || ≤ O (1) , and each ε t is O (1) -subgaussian, then there are choices of λ, β such that, with probability at least 1 − δ , the regret of the LinUCB algorithm satisﬁes
Revised: If || θ ∗ || , || x t,a || ≤ O (1) , and each ε t is O (1) -subgaussian, then there are choices of λ, β such that, with probability at least 1 − δ , the regret of the LinUCB algorithm satisﬁes

Original: [Equation]
Revised: [Equation]

Original: The ˜ O ( · ) suppresses factors polynomial in log(1 /δ ) and log T .
Revised: The ˜ O ( · ) suppresses factors polynomial in log(1 /δ ) and log T .

Original: LinUCB can be implemented in O ( d 2 |A t | ) time per iteration, by maintaining ˜Σ − 1 t using rankupdates, and using this to compute the least-squares estimator ˆ θ t =
Revised: LinUCB can be implemented in O ( d 2 |A t | ) time per iteration, by maintaining ˜Σ − 1 t using rankupdates, and using this to compute the least-squares estimator ˆ θ t =

Original: ˜Σ − 1 t (cid:80) t − 1 τ =1 x τ,a τ r τ ( a τ ) as well as bonus t ( a ) .
Revised: ˜Σ − 1 t (cid:80) t − 1 τ =1 x τ,a τ r τ ( a τ ) as well as bonus t ( a ) .

Original: Parameters: ensemble size M ; β, λ 1:
Revised: Parameters: ensemble size M ; β, λ 1:

Original: For all i = 1 , . . .
Revised: For all i = 1 , . . .

Original: d , set warm-start features x − i +

Revised: d , set warm-start features x − i +

Original: = √ λe corresponding warm-start targets y ( j ) −
Revised: = √ λe corresponding warm-start targets y ( j ) −

Original: i +1 ∼ N (0 , 1) for each j
Revised: i +1 ∼ N (0 , 1) for each j

Original: ] 2: for
Revised: ] 2: for

Original: T do 3:
Revised: T do 3:

Original: Observe action features { x t,a } a ∈A t 4:
Revised: Observe action features { x t,a } a ∈A t 4:

Original: Update reward estimator:
Revised: Update reward estimator:

Original: [Table] [Table]
Revised: [Table] [Table]

Original: t −
Revised: t −

Original: [Table]
Revised: [Table]

Original: τ = ( (cid:104)
Revised: τ = ( (cid:104)

Original: [Table]
Revised: [Table]

Original: (cid:105)
Revised: (cid:105)

Original: [Table]
Revised: [Table]

Original: Sample new random ensemble targets: (re-randomized) y
Revised: Sample new random ensemble targets: (re-randomized) y

Original: ( j ) τ ∼ N (0 , 1) for each j ∈
Revised: ( j ) τ ∼ N (0 , 1) for each j ∈

Original: [ M
Revised: [ M

Original: , τ ∈ {− d + 1 , . . .
Revised: , τ ∈ {− d + 1 , . . .

Original: , t − 1 } (incremental) y ( j ) t
Revised: , t − 1 } (incremental) y ( j ) t

Original: [Table]
Revised: [Table]

Original: ∼ N (0 , 1) for each j ∈ [ M ] 6:
Revised: ∼ N (0 , 1) for each j ∈ [ M ] 6:

Original: Update bonus ensembles for j
Revised: Update bonus ensembles for j

Original: M ] , w ( j ) t := arg min w t − 1 (cid:88) τ = − d + (cid:16) (cid:104) x
Revised: M ] , w ( j ) t := arg min w t − 1 (cid:88) τ = − d + (cid:16) (cid:104) x

Original: τ,a τ , w (cid:105) − y ( j ) τ (cid:17)7:
Revised: τ,a τ , w (cid:105) − y ( j ) τ (cid:17)7:

Original: Compute
Revised: Compute

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: ] (cid:12)(cid:12)(cid:12)(cid:68)
Revised: ] (cid:12)(cid:12)(cid:12)(cid:68)

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (cid:69)(cid:12)(cid:12)(cid:12) for each a ∈
Revised: (cid:69)(cid:12)(cid:12)(cid:12) for each a ∈

Original: A t 8:
Revised: A t 8:

Original: Choose action a
Revised: Choose action a

Original: [Table]
Revised: [Table]

Original: := arg
Revised: := arg

Original: [Table]
Revised: [Table]

Original: (cid:104)
Revised: (cid:104)

Original: [Table]
Revised: [Table]

Original: t,a ,
Revised: t,a ,

Original: [Table]
Revised: [Table]

Original: θ
Revised: θ

Original: [Table]
Revised: [Table]

Original: (cid:105)
Revised: (cid:105)

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: ( a ) 9:
Revised: ( a ) 9:

Original: Observe reward r t ( a t ) =
Revised: Observe reward r t ( a t ) =

Original: (cid:104)
Revised: (cid:104)

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: a t , θ ∗ (cid:105) + ε t
Revised: a t , θ ∗ (cid:105) + ε t

Original:  A NTI - CONCENTRATED CONFIDENCE BOUNDS FOR LINEAR BANDITS
Revised:  A NTI - CONCENTRATED CONFIDENCE BOUNDS FOR LINEAR BANDITS

Original: We begin by presenting the ACB intrinsic reward for the stochastic linear bandit setting.
Revised: We begin by presenting the ACB intrinsic reward for the stochastic linear bandit setting.

Original: It is well-known that the weights of a least-squares regressor trained to predict i.i.d.
Revised: It is well-known that the weights of a least-squares regressor trained to predict i.i.d.

Original: noise will capture information about about the precision matrix of the data.
Revised: noise will capture information about about the precision matrix of the data.

Original: Our key idea is to amplify the deviation in the predictions of an ensemble of such regressors by taking a maximum over their outputs, allowing us to construct upper conﬁdence bounds for ˆ r t ( a ) .
Revised: Our key idea is to amplify the deviation in the predictions of an ensemble of such regressors by taking a maximum over their outputs, allowing us to construct upper conﬁdence bounds for ˆ r t ( a ) .

Original: ACB maintains M linear regressors trained to predict i.i.d.
Revised: ACB maintains M linear regressors trained to predict i.i.d.

Original: standard Gaussian noise, and sets the bonus to be proportional to the maximum deviation over this ensemble from its mean (which is 0 ).
Revised: standard Gaussian noise, and sets the bonus to be proportional to the maximum deviation over this ensemble from its mean (which is 0 ).

Original: This procedure is formally provided in Algorithm 1.
Revised: This procedure is formally provided in Algorithm 1.

Original: With the correct choice of ensemble size, the concentration of the maximum of independent Gaussians implies that each bonus t ( a ) is large enough to be a valid upper conﬁdence bound, yet small enough to inherit the regret bound of LinUCB.
Revised: With the correct choice of ensemble size, the concentration of the maximum of independent Gaussians implies that each bonus t ( a ) is large enough to be a valid upper conﬁdence bound, yet small enough to inherit the regret bound of LinUCB.

Original: ACB can be instantiated with freshly resampled history; in this case, the theoretical guarantees on the sufﬁcient ensemble size are strongest.
Revised: ACB can be instantiated with freshly resampled history; in this case, the theoretical guarantees on the sufﬁcient ensemble size are strongest.

Original: A more computationally efﬁcient variant, which is the one we scale up to deep RL, samples random targets only once, enabling the use of streaming algorithms to estimate regression weight w jt .
Revised: A more computationally efﬁcient variant, which is the one we scale up to deep RL, samples random targets only once, enabling the use of streaming algorithms to estimate regression weight w jt .

Original: For these two variants, the following regret bounds result from straightforward modiﬁcations to the analysis of LinUCB (Abbasi-Yadkori et al., 2011), which are deferred (along with more precise statements) to Appendix B:
Revised: For these two variants, the following regret bounds result from straightforward modiﬁcations to the analysis of LinUCB (Abbasi-Yadkori et al., 2011), which are deferred (along with more precise statements) to Appendix B:

Original: Proposition 2 (ACB regret bound with re-randomized history) .
Revised: Proposition 2 (ACB regret bound with re-randomized history) .

Original: Then, Algorithm 1, with rerandomized history and ensemble size M = Θ(log( T/δ )) , obtains a regret bound of ˜ O
Revised: Then, Algorithm 1, with rerandomized history and ensemble size M = Θ(log( T/δ )) , obtains a regret bound of ˜ O

Original: ( d √ T log A ) , with probability at least 1 − δ .
Revised: ( d √ T log A ) , with probability at least 1 − δ .

Original: For A ≤ O (poly( d )) , our regret bound matches that of LinUCB up to logarithmic factors.
Revised: For A ≤ O (poly( d )) , our regret bound matches that of LinUCB up to logarithmic factors.

Original: We note that the expected regret guarantee of the LinPHE algorithm in Kveton et al.
Revised: We note that the expected regret guarantee of the LinPHE algorithm in Kveton et al.

Original: (2019a) also has the same dependence on log( AT ) .
Revised: (2019) also has the same dependence on log( AT ) .

Original: Our lower-tail analysis on the number of regressors required for the maximum deviation to be optimistic is slightly sharper that that found in Ishfaq et al.
Revised: Our lower-tail analysis on the number of regressors required for the maximum deviation to be optimistic is slightly sharper than that found in Ishfaq et al.

Original: (2021), by a factor of log A ; this is due to requiring optimism only on a ∗ in Lemma 2.
Revised: (2021), by a factor of log A ; this is due to requiring optimism only on a ∗ in Lemma 2.

Original: Proposition 3 (ACB regret bound with incremental updates) .
Revised: Proposition 3 (ACB regret bound with incremental updates) .

Original: Algorithm 1, with incrementally randomized history and an ensemble size of M = Θ(log
Revised: Algorithm 1, with incrementally randomized history and an ensemble size of M = Θ(log

Original: ( T/δ )) , obtains a regret bound of ˜ O
Revised: ( T/δ )) , obtains a regret bound of ˜ O

Original: ( A √ T ) in the multi-armed bandit setting, with probability at least 1 − δ .
Revised: ( A √ T ) in the multi-armed bandit setting, with probability at least 1 − δ .

Original: We conjecture that the incrementally-updated version guarantees can be extended to linear bandits beyond MAB for reasonable ensemble sizes.
Revised: We conjecture that the incrementally-updated version guarantees can be extended to linear bandits beyond MAB for reasonable ensemble sizes.

Original: In between the extremes of always vs. never re-

Original: θ update τ

Original: L ( j

Original: [-= (( w ( j ) t ) (cid:62) g t − y ( j ) t ) 
 + λ || w ( j ) t − w ( j ) 
 ||-] randomizing history, it is possible to analyze a “rarely re-randomizing” variant of the algorithm, which obtains the smaller sufﬁcient ensemble size of Proposition 2 while only re-randomizing ˜
Revised: [-= (( w ( j ) t ) (cid:62) g t − y ( j ) t ) 
 + λ || w ( j ) t − w ( j ) 
 ||-]In between the extremes of always vs. never re- randomizing history, it is possible to analyze a “rarely re-randomizing” variant of the algorithm, which obtains the smaller sufﬁcient ensemble size of Proposition 2 while only re-randomizing ˜

Original: O ( d log T ) times:
Revised: O ( d log T ) times:

Original: Theorem 4 (Lazy re-randomization) .
Revised: Theorem 4 (Lazy re-randomization) .

Original: Algorithm 3, with lazy re-randomized history, obtains a regret bound of ˜ O
Revised: Algorithm 3, with lazy re-randomized history, obtains a regret bound of ˜ O

Original: ( d (cid:112) log( AT/δ ) T ) in the case of linear bandits with ﬁxed actions, while resampling the random targets y ( j ) τ only ˜ O
Revised: ( d (cid:112) log( AT/δ ) T ) in the case of linear bandits with ﬁxed actions, while resampling the random targets y ( j ) τ only ˜ O

Original: ( d log( AT/δ ) log( T )) times.
Revised: ( d log( AT/δ ) log( T )) times.

Original: We defer the modiﬁed algorithm and analysis to Appendix C. We note that the lazy version is computationally more efﬁcient than the bonus approximation procedures in LinPHE (Kveton et al., 2019a) and LSVI-PHE (Ishfaq et al., 2021), and the expected regret guarantees obtained by LinPHE are not sufﬁcient to successfully implement a lazy strategy.
Revised: We defer the modiﬁed algorithm and analysis to Appendix C. We note that the lazy version is computationally more efﬁcient than the bonus approximation procedures in LinPHE (Kveton et al., 2019) and LSVI-PHE (Ishfaq et al., 2021), and the expected regret guarantees obtained by LinPHE are not sufﬁcient to successfully implement a lazy strategy.

Original: Like other works proposing more scalable algorithms in the same setting (Korda et al., 2015; Ding et al., 2021), with theoretically sound hyperparameter choices, we do not exhibit end-to-end theoretical improvements over the O ( d 2 |A t | ) per-iteration cost of rank-1 updates for implementing LinUCB.
Revised: Like other works proposing more scalable algorithms in the same setting (Korda et al., 2015; Ding et al., 2021), with theoretically sound hyperparameter choices, we do not exhibit end-to-end theoretical improvements over the O ( d 2 |A t | ) per-iteration cost of rank-1 updates for implementing LinUCB.

Original: Both lines 4 and 6 in Algorithm 1 depend on recursive least squares optimization oracles.
Revised: Both lines 4 and 6 in Algorithm 1 depend on recursive least squares optimization oracles.

Revised: For our main empirical results (as well as Figure 2 (right), we implement this oracle using Polyak-averaged

Revised: SGD updates to each regressor, requiring O ( Md |A t | ) time per iteration to compute all the bonuses.

Original: Synthetic bandit experiments.
Revised: Synthetic bandit experiments.

Original: As a simple empirical validation of our method, we demonstrate the performance of Algorithm 1 (with incremental sampling) in a simulated multi-armed bandit setting with A = 50 actions.
Revised: As a simple empirical validation of our method, we demonstrate the performance of Algorithm 1 (with incremental sampling) in a simulated multi-armed bandit setting with A = 50 actions.

Original: One arm has a mean reward 0 .
Revised: One arm has a mean reward 0 .

Original: 75 while the others’ are set to 0 .
Revised: 75 while the others’ are set to 0 .

Original: 25 ; the noise ε t for each arm is sampled independently from N (0 , 0 .
Revised: 25 ; the noise ε t for each arm is sampled independently from N (0 , 0 .

Original: As predicted by the theory, the ensemble size M must be chosen to be large enough for the bonuses to be sufﬁciently optimistic, small enough to avoid too many upper-tail events.
Revised: As predicted by the theory, the ensemble size M must be chosen to be large enough for the bonuses to be sufﬁciently optimistic, small enough to avoid too many upper-tail events.

Original: Further details are provided in Appendix A.1.
Revised: Further details are provided in Appendix A.1.

Original:  E XTENDING ACB TO DEEP REINFORCEMENT LEARNING
Revised:  E XTENDING ACB TO DEEP REINFORCEMENT LEARNING

Original: Outlined as Algorithm 2, the RL version of ACB deviates from the version mentioned earlier in several ways.
Revised: Outlined as Algorithm 2, the RL version of ACB deviates from the version mentioned earlier in several ways.

Original: First, like other work assigning exploratory bonuses in deep reinforcement learning, we assign intrinsic rewards on states, rather than on state-action pairs (Pathak et al., 2017; Burda et al., 2018b).
Revised: First, like other work assigning exploratory bonuses in deep reinforcement learning, we assign intrinsic rewards on states, rather than on state-action pairs (Pathak et al., 2017; Burda et al., 2018b).

Original: Second, to help deal with non-stationarity, we compute gradient features with respect to a tail-averaged set of policy weights, rather than on the policy weights being used to interact with the environment.
Revised: Second, to help deal with non-stationarity, we compute gradient features with respect to a tail-averaged set of policy weights, rather than on the policy weights being used to interact with the environment.

Original: Because neural networks learn their own internal feature representation, extending ACB to the neural setting gives us some ﬂexibility in the way we choose to represent data.
Revised: Because neural networks learn their own internal feature representation, extending ACB to the neural setting gives us some ﬂexibility in the way we choose to represent data.

Original: One option is to use the penultimate layer representation of the network, an approach that has seen success in the closely related active learning setting (Sener & Savarese, 2018).
Revised: One option is to use the penultimate layer representation of the network, an approach that has seen success in the closely related active learning setting (Sener & Savarese, 2018).

Original: In the deep active learning classiﬁcation setting, a more recent approach is to instead represent data using a hallucinated gradient, embedding data as the gradient that would be induced by the sample if the most likely label according to the model were the true label (Ash et al., 2020).
Revised: In the deep active learning classiﬁcation setting, a more recent approach is to instead represent data using a hallucinated gradient, embedding data as the gradient that would be induced by the sample if the most likely label according to the model were the true label (Ash et al., 2020).

Original: Treating the policy network as a classiﬁer, we ﬁnd that this representation tends to perform better than using just the penultimate layer.
Revised: Treating the policy network as a classiﬁer, we ﬁnd that this representation tends to perform better than using just the penultimate layer.

Original: Speciﬁcally, this gradient is computed as
Revised: Speciﬁcally, this gradient is computed as

Original: [Equation]
Revised: [Equation]

Original: where f is the policy network, θ is its current parameters, (cid:96) ce
Revised: where f is the policy network, θ is its current parameters, (cid:96) ce

Original: (˜ y, ˆ y ) is the log loss of the prediction ˜
Revised: (˜ y, ˆ y ) is the log loss of the prediction ˜

Original: y given true label ˆ y , and ˆ y is the action to which the most probability mass is assigned.
Revised: y given true label ˆ y , and ˆ y is the action to which the most probability mass is assigned.

Original: Using this representation, each seen state x contributes g x g (cid:62) x to the ACB covariance matrix, a quantity that can viewed as a sort of rank-one approximation of the pointwise Fisher information,
Revised: Using this representation, each seen state x contributes g x g (cid:62) x to the ACB covariance matrix, a quantity that can viewed as a sort of rank-one approximation of the pointwise Fisher information,

Original: I ( x, θ ) =
Revised: I ( x, θ ) =

Original: E y ∼ f
Revised: E y ∼ f

Original: ( x,θ ) ∇ (cid:96) ( f ( x, θ ) , y )( ∇ (cid:96) ( f ( x, θ ) , y ))
Revised: ( x,θ ) ∇ (cid:96) ( f ( x, θ ) , y )( ∇ (cid:96) ( f ( x, θ ) , y ))

Original: (cid:62) (Ash et al., 2021).
Revised: (cid:62) (Ash et al., 2021).

Original: This connection draws a parallel between ACB with gradient features and well-understood, Fisher-based objectives in the active learning literature (Zhang & Oles, 2000; Gu et al., 2014; ? ).
Revised: This connection draws a parallel between ACB with gradient features and well-understood, Fisher-based objectives in the active learning literature (Zhang & Oles, 2000; Gu et al., 2014; Chaudhuri et al., 2015).

Original: The ability to use such a gradient representation highlights the scalability of ACB—policy networks used in RL are typically composed of over a million parameters, and explicitly maintaining and inverting a covariance matrix of this size is not feasible.
Revised: The ability to use such a gradient representation highlights the scalability of ACB—policy networks used in RL are typically composed of over a million parameters, and explicitly maintaining and inverting a covariance matrix of this size is not feasible.

Original: ACB circumvents this need and handles high-dimensional features without issue.
Revised: ACB circumvents this need and handles high-dimensional features without issue.

Original: Figure 3 shows examples of environments for which the gradient representation produces better exploration than the penultimate-layer representation, according to the number of unique states encountered by the agent (using a hash).
Revised: Figure 3 shows examples of environments for which the gradient representation produces better exploration than the penultimate-layer representation, according to the number of unique states encountered by the agent (using a hash).

Original: In both cases, representations are computed on a Polyak-averaged version of the policy with α = 10 − 6 , which we also use in Section 5 below.
Revised: In both cases, representations are computed on a Polyak-averaged version of the policy with α = 10 − 6 , which we also use in Section 5 below.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section compares ACB with state-of-the-art approaches to exploration bonuses in deep reinforcement learning on Atari benchmarks.
Revised: This section compares ACB with state-of-the-art approaches to exploration bonuses in deep reinforcement learning on Atari benchmarks.

Original: We inherit hyperparameters from RND, including the convolutional policy network architecture, which can be found in Appendix Section A.2.
Revised: We inherit hyperparameters from RND, including the convolutional policy network architecture, which can be found in Appendix Section A.2.

Original: Policy optimization is done with PPO (Schulman et al., 2017).
Revised: Policy optimization is done with PPO (Schulman et al., 2017).

Original: In all experiments, we use 128 parallel agents and rollouts of 128 timesteps, making τ in Algorithm 2 128 2 .
Revised: In all experiments, we use 128 parallel agents and rollouts of 128 timesteps, making τ in Algorithm 2 128 2 .

Original: Like other work on intrinsic bonuses, we consider the non-episodic setting, so the agent is unaware of when it the episode is terminated.
Revised: Like other work on intrinsic bonuses, we consider the non-episodic setting, so the agent is unaware of when it the episode is terminated.

Original: Consequently, even in games for which merely learning to survive is equivalent to learning to play well, random or constant rewards will not elicit good performance.
Revised: Consequently, even in games for which merely learning to survive is equivalent to learning to play well, random or constant rewards will not elicit good performance.

Original: All experiments are run for ﬁve replicates, and plots show mean behavior and standard error.
Revised: All experiments are run for ﬁve replicates, and plots show mean behavior and standard error.

Original: In our investigation, we identiﬁed several tricks that seem to help mitigate the domain shift in the gradient representation.
Revised: In our investigation, we identiﬁed several tricks that seem to help mitigate the domain shift in the gradient representation.

Original: Among these, as mentioned above, we tail-average auxiliary policy network parameters, slowly moving them towards the deployment policy.
Revised: Among these, as mentioned above, we tail-average auxiliary policy network parameters, slowly moving them towards the deployment policy.

Original: This approach has been used to increase stability in both reinforcement learning and generative adversarial networks (Lillicrap et al., 2015; Bell, 1934).
Revised: This approach has been used to increase stability in both reinforcement learning and generative adversarial networks (Lillicrap et al., 2015; Bell, 1934).

Original: In experiments shown here, α is ﬁxed at 10 − 6 .
Revised: In experiments shown here, α is ﬁxed at 10 − 6 .

Original: We additionally normalize intrinsic bonuses by a running estimate of their standard deviation, an implementation detail that is also found in both RND and ICM.
Revised: We additionally normalize intrinsic bonuses by a running estimate of their standard deviation, an implementation detail that is also found in both RND and ICM.

Original: Batch normalization is applied to gradient features before passing them to the auxiliary weight ensemble, and, like in the bandit setting, we compute bonuses using a Polyak-averaged version of the ensemble rather than the ensemble itself.
Revised: Batch normalization is applied to gradient features before passing them to the auxiliary weight ensemble, and, like in the bandit setting, we compute bonuses using a Polyak-averaged version of the ensemble rather than the ensemble itself.

Original: We use the RMSprop variant of SGD in optimizing auxiliary weights.
Revised: We use the RMSprop variant of SGD in optimizing auxiliary weights.

Original: We use an ensemble of 128 auxiliary weights and regularize aggressively towards their initialization with λ = 10 3 .
Revised: We use an ensemble of 128 auxiliary weights and regularize aggressively towards their initialization with λ = 10 3 .

Original: As with the bandit setting, larger ensembles tend to produce bonuses that both are more stable and elicit more favorable behavior from the agent (Figure 4).
Revised: As with the bandit setting, larger ensembles tend to produce bonuses that both are more stable and elicit more favorable behavior from the agent (Figure 4).

Original: Experiments here consider eight different Atari games.
Revised: Experiments here consider eight different Atari games.

Original: Environments were chosen to include both games that are known to be challenging in the way of exploration (Private Eye, Pitfall!
Revised: Environments were chosen to include both games that are known to be challenging in the way of exploration (Private Eye, Pitfall!

Original: and Gravitar (Burda et al., 2018b)), and games that are not—remaining environments overlap with those from Table 1 of Mnih et al.
Revised: and Gravitar (Burda et al., 2018b)), and games that are not—remaining environments overlap with those from Table 1 of Mnih et al.

Original: Intrinsic rewards only.
Revised: Intrinsic rewards only.

Original: In our ﬁrst set of experiments, we train agents exclusively on bonuses generated by ACB, rather than on any extrinsic reward signal.
Revised: In our ﬁrst set of experiments, we train agents exclusively on bonuses generated by ACB, rather than on any extrinsic reward signal.

Original: In Figure 1, we visualize some of these bonuses as gameplay progresses in two Atari games.
Revised: In Figure 1, we visualize some of these bonuses as gameplay progresses in two Atari games.

Original: In both, large intrinsic bonuses are obtained when the agent reaches the periphery of its experience.
Revised: In both, large intrinsic bonuses are obtained when the agent reaches the periphery of its experience.

Original: Correspondingly, when the agent dies, and the game state is reset to something familiar, the ACB bonus typically drops by a signiﬁcant margin.
Revised: Correspondingly, when the agent dies, and the game state is reset to something familiar, the ACB bonus typically drops by a signiﬁcant margin.

Original: These intrinsic bonuses encourage the agent to avoid the start state, and to continually push the boundary of what has been observed so far.
Revised: These intrinsic bonuses encourage the agent to avoid the start state, and to continually push the boundary of what has been observed so far.

Original: Figure 5 compares the per-episode extrinsic rewards collected by agents trained only on intrinsic rewards across eight popular Atari games and three different bonus schemes.
Revised: Figure 5 compares the per-episode extrinsic rewards collected by agents trained only on intrinsic rewards across eight popular Atari games and three different bonus schemes.

Original: Even though extrinsic reward is not observed, many of these agents obtain non-trivial performance on the game, suggesting that the pursuit of novelty is often enough to play well.
Revised: Even though extrinsic reward is not observed, many of these agents obtain non-trivial performance on the game, suggesting that the pursuit of novelty is often enough to play well.

Original: Extrinsic reward does not necessarily correlate with exploration quality, but ACB is clearly competitive with other intrinsic reward methods along this axis.
Revised: Extrinsic reward does not necessarily correlate with exploration quality, but ACB is clearly competitive with other intrinsic reward methods along this axis.

Original: In Figure 6, we measure the number of unique states encountered, as measured by a hash, as exploration progresses.
Revised: In Figure 6, we measure the number of unique states encountered, as measured by a hash, as exploration progresses.

Original: ACB is competitive from this perspective as well, visiting as many or more states than baselines in most of the environments considered.
Revised: ACB is competitive from this perspective as well, visiting as many or more states than baselines in most of the environments considered.

Original: Intrinsic and extrinsic rewards.
Revised: Intrinsic and extrinsic rewards.

Original: Next, while still in the non-episodic setting, we experiment with training PPO jointly on intrinsic and extrinsic rewards.
Revised: Next, while still in the non-episodic setting, we experiment with training PPO jointly on intrinsic and extrinsic rewards.

Original: Following the PPO modiﬁcation proposed by Burda et al.
Revised: Following the PPO modiﬁcation proposed by Burda et al.

Original: (2018b), we use two value heads, one for intrinsic and another for extrinsic rewards.
Revised: (2018b), we use two value heads, one for intrinsic and another for extrinsic rewards.

Original: total reward is computed as the sum of intrinsic and extrinsic returns.
Revised: total reward is computed as the sum of intrinsic and extrinsic returns.

Original: Figure 7 compares RND and
Revised: Figure 7 compares RND and

Original: ACB agents trained with both intrinsic and extrinsic rewards on eight Atari games, which shows ACB again performing competitively in comparison to RND.
Revised: ACB agents trained with both intrinsic and extrinsic rewards on eight Atari games, which shows ACB again performing competitively in comparison to RND.

Original: We ﬁnd that ACB performs at least as well as
Revised: We ﬁnd that ACB performs at least as well as

Original: RND on most of the games we considered, and sometimes, for particular timesteps and environments, even offers an improvement.
Revised: RND on most of the games we considered, and sometimes, for particular timesteps and environments, even offers an improvement.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This article introduced ACB, a scalable and computationally-tractable algorithm for estimating the elliptical potential in both bandit and deep reinforcement learning scenarios.
Revised: This article introduced ACB, a scalable and computationally-tractable algorithm for estimating the elliptical potential in both bandit and deep reinforcement learning scenarios.

Original: ACB avoids the need to invert a large covariance matrix by instead storing this information in an ensemble of weight vectors trained to predict random noise.
Revised: ACB avoids the need to invert a large covariance matrix by instead storing this information in an ensemble of weight vectors trained to predict random noise.

Original: This work aims to bridge the gap between the empirical successes of deep reinforcement learning and the theoretical transparency of bandit algorithms.
Revised: This work aims to bridge the gap between the empirical successes of deep reinforcement learning and the theoretical transparency of bandit algorithms.

Original: We believe ACB is a big step along this direction, but there is still more work to be done to further this effort.
Revised: We believe ACB is a big step along this direction, but there is still more work to be done to further this effort.

Original: For one, ACB is still not able to perform as well as RND on Montezuma’s Revenge, an Atari benchmark for which exploration is notoriously difﬁcult, and for which RND performs exceptionally well.
Revised: For one, ACB is still not able to perform as well as RND on Montezuma’s Revenge, an Atari benchmark for which exploration is notoriously difﬁcult, and for which RND performs exceptionally well.

Original: Our experiments indicate that our proposed technique of anti-concentrating optimism can improve empirical performance well beyond our current theoretical understanding.
Revised: Our experiments indicate that our proposed technique of anti-concentrating optimism can improve empirical performance well beyond our current theoretical understanding.

Original: We leave improving our regret guarantees (such as eliminating dependences on log( A ) , matching the optimal rate in all regimes), as well as extending our analysis to more difﬁcult settings (such as the incremental case) as exciting open problems.
Revised: We leave improving our regret guarantees (such as eliminating dependences on log( A ) , matching the optimal rate in all regimes), as well as extending our analysis to more difﬁcult settings (such as the incremental case) as exciting open problems.

Original: A E XPERIMENTAL DETAILS A. B ANDIT EXPERIMENTS
Revised: A E XPERIMENTAL DETAILS A. B ANDIT EXPERIMENTS

Original: Figure 2 shows the cumulative regret of Algorithm 1 (with incremental sampling) in a simulated multi-armed bandit setting with A = 50 actions, for different values of M (powers of 2 from 1 to 256 ).
Revised: Figure 2 shows the cumulative regret of Algorithm 1 (with incremental sampling) in a simulated multi-armed bandit setting with A = 50 actions, for different values of M (powers of 2 from 1 to 256 ).

Original: One randomly sampled arm is chosen to have a mean reward of 0 .
Revised: One randomly sampled arm is chosen to have a mean reward of 0 .

Original: 75 , with the others 0 .
Revised: 75 , with the others 0 .

Original: 25 ; the noise ε t for each arm is sampled independently from N (0 , 0 .
Revised: 25 ; the noise ε t for each arm is sampled independently from N (0 , 0 .

Original: Standard deviations are shown over 100 independent trials.
Revised: Standard deviations are shown over 100 independent trials.

Original: The bonus scaling factor β was selected in each run by grid search over { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 } ; the learning rate for SGD was selected by grid search over the same values.
Revised: The bonus scaling factor β was selected in each run by grid search over { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0 } ; the learning rate for SGD was selected by grid search over the same values.

Original: Each batch of experiments (one choice of algorithm and M ; all replicas and hyperparameter sweeps) took around 1 CPU hour on an internal cluster.
Revised: Each batch of experiments (one choice of algorithm and M ; all replicas and hyperparameter sweeps) took around 1 CPU hour on an internal cluster.

Original: A. A TARI EXPERIMENTS
Revised: A. A TARI EXPERIMENTS

Original: All policy network architectures used, for ICM, RND, and ACB, have the convolutional architecture described in Burda et al.
Revised: All policy network architectures used, for ICM, RND, and ACB, have the convolutional architecture described in Burda et al.

Original: (2018b), consisting of three convolutional layers followed by two linear layers, and leaky ReLU activations.
Revised: (2018b), consisting of three convolutional layers followed by two linear layers, and leaky ReLU activations.

Original: For ACB and RND, we use PPO hyperparameters also from Burda et al.
Revised: For ACB and RND, we use PPO hyperparameters also from Burda et al.

Original: (2018b), including 4 update epochs, an entropy coefﬁcient of 0.001, γ E =  and γ I = 0.99.
Revised: (2018b), including 4 update epochs, an entropy coefﬁcient of 0.001, γ E = 0 . 999 and γ I = 0.99.

Original: Our ICM implementation uses the default parameters of Pathak et al.
Revised: Our ICM implementation uses the default parameters of Pathak et al.

Original: (2017), which instead use 3 update epochs.
Revised: (2017), which instead useupdate epochs.

Original: We use a learning rate of 0.0001 and 128 simultaneous agents universally.
Revised: We use a learning rate of 0.0001 and 128 simultaneous agents universally.

Original: All model updates are performed via Adam except for the ACB auxiliary weights, which use RMSprop.
Revised: All model updates are performed via Adam except for the ACB auxiliary weights, which use RMSprop.

Original: See attached code for more details.
Revised: See attached code for more details.

Original: Compute resources per job included a single CPU and either a P100 of V100 NVIDIA GPU, with experiments each taking between three and ﬁve days.
Revised: Compute resources per job included a single CPU and either a P100 of V100 NVIDIA GPU, with experiments each taking between three and ﬁve days.

Original: B P ROOFS OF REGRET BOUNDS
Revised: B P ROOFS OF REGRET BOUNDS

Original: In this section, we prove Propositions 2 and 3.
Revised: In this section, we prove Propositions 2 and 3.

Original: We use the following notation in the analysis:
Revised: We use the following notation in the analysis:

Original: • ˜Σ t denotes λI + (cid:80) t − 1 τ =1 x t,a t x (cid:62) t,a t .
Revised: • ˜Σ t denotes λI + (cid:80) t − 1 τ =1 x t,a t x (cid:62) t,a t .

Original: • regret t denotes the instantaneous regret (cid:104) x t,a ∗ , θ ∗ (cid:105) − (cid:104) x t,a t , θ ∗ (cid:105) .
Revised: • regret t denotes the instantaneous regret (cid:104) x t,a ∗ , θ ∗ (cid:105) − (cid:104) x t,a t , θ ∗ (cid:105) .

Original: • log + ( · ) denotes log(max(1 , · )) .
Revised: • log + ( · ) denotes log(max(1 , · )) .

Original: Furthermore, we assume that || x t,a || ≤ B for all t, a , || θ ∗ || ≤ W , and that the η t are a martingale difference sequence independent of the decisions made by the algorithm.
Revised: Furthermore, we assume that || x t,a || ≤ B for all t, a , || θ ∗ || ≤ W , and that the η t are a martingale difference sequence independent of the decisions made by the algorithm.

Original: Assume that the η t are σ 2 -subgaussian, with σ > 0 .
Revised: Assume that the η t are σ 2 -subgaussian, with σ > 0 .

Original: B. U SEFUL LEMMAS
Revised: B. U SEFUL LEMMAS

Original: Lemma 5 (Tail bounds for a maximum of Gaussians; Tu (2017)) .
Revised: Lemma 5 (Tail bounds for a maximum of Gaussians; Tu (2017)) .

Original: Let z (1) , . .
Revised: Let z (1) , . .

Original: , z ( M ) be i.i.d.
Revised: , z ( M ) be i.i.d.

Original: N (0 , 1) random variables.
Revised: N (0 , 1) random variables.

Original: Let Z denote max j =1 ,...,M | z ( j ) | .
Revised: Let Z denote max j =1 ,...,M | z ( j ) | .

Original: Then, for all 0 < δ < 1 , each of these inequalities holds with probability at least 1 − δ :
Revised: Then, for all 0 < δ < 1 , each of these inequalities holds with probability at least 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 (Proposition 5.5, Agarwal et al.
Revised: Lemma 6 (Proposition 5.5, Agarwal et al.

Original: At any time t ∈ [ T ] , for all a ∈ A t , with probability 1 − δ t , 
Revised: At any time t ∈ [ T ] , for all a ∈ A t , with probability 1 − δ t , (cid:68) [Equation] [Equation] ˜Σ

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for β t = √ λ (cid:107) θ ∗ (cid:107) + (cid:113) 2 σ 2 log(det( ˜ Σ t ) λ − d /δ t ) .
Revised: for β t = √ λ (cid:107) θ ∗ (cid:107) + (cid:113) 2 σ 2 log(det( ˜ Σ t ) λ − d /δ t ) .

Original: Lemma 7 (Sufﬁcient conditions for the bonus) .
Revised: Lemma 7 (Sufﬁcient conditions for the bonus) .

Original: Suppose that for all t ≤ T , the bonuses satisfy
Revised: Suppose that for all t ≤ T , the bonuses satisfy

Original: [Equation]
Revised: [Equation]

Original: where a ∗ is the optimal action.
Revised: where a ∗ is the optimal action.

Original: If we choose actions according to this bonus at every time step t , then with probability 1 − δ ,
Revised: If we choose actions according to this bonus at every time step t , then with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: for
Revised: for

Original: [Equation]
Revised: [Equation]

Original: as
Revised: as

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using the assumptions on the bonus and Lemma 6 with δ t = δ/T , 1 with probability 1 − δ , for all t ≤ T
Revised: Using the assumptions on the bonus and Lemma 6 with δ t = δ/T , 1 with probability 1 − δ , for all t ≤ T

Original: [Equation]
Revised: [Equation]

Original: ˜Σ − 1 t − γ 1 (cid:107) x t,a ∗ (cid:107) ˜Σ − 1 t + β t (cid:107) x t,a ∗ (cid:107) ˜Σ − 1 t + β t (cid:107) x t,a t (cid:107) ˜Σ − 1 t ≤ ( γ 2 + β t ) (cid:107) x t,a t (cid:107) ˜Σ − 1 t .
Revised: ˜Σ − 1 t − γ 1 (cid:107) x t,a ∗ (cid:107) ˜Σ − 1 t + β t (cid:107) x t,a ∗ (cid:107) ˜Σ − 1 t + β t (cid:107) x t,a t (cid:107) ˜Σ − 1 t ≤ ( γ 2 + β t ) (cid:107) x t,a t (cid:107) ˜Σ − 1 t .

Original: Summing up, we have
Revised: Summing up, we have

Original: R ( T ) =
Revised: R ( T ) =

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: t = regret t ≤
Revised: t = regret t ≤

Original: T (cid:88)
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: t ≤
Revised: t ≤

Original: [Equation]
Revised: [Equation]

Original: T (cid:88) t = ( γ
Revised: T (cid:88) t = ( γ

Original: [Equation]
Revised: [Equation]

Original: (cid:118)(cid:117)(cid:117)(cid:116)
Revised: (cid:118)(cid:117)(cid:117)(cid:116)

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:107) x t,a t (cid:107) 2˜Σ − 1 t ≤
Revised: = (cid:107) x t,a t (cid:107) 2˜Σ − 1 t ≤

Original: [Equation]
Revised: [Equation]

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (
Revised: = (

Original: [Equation]
Revised: [Equation]

Original: The above follows from Cauchy-Schwarz and observing that β t ≤ ¯ β for all t ≤ T .
Revised: The above follows from Cauchy-Schwarz and observing that β t ≤ ¯ β for all t ≤ T .

Original: B. R EGRET BOUNDS
Revised: B. R EGRET BOUNDS

Original: Proposition 2 (ACB regret bound with re-randomized history) .
Revised: Proposition 2 (ACB regret bound with re-randomized history) .

Original: Algorithm 1, with rerandomized history, ensemble size M = (cid:100) log(2 T/δ ) (cid:101) , λ = σ 2 /W 2 , and β = √ λW + √ 2 σ (cid:113) d log (cid:0) 1 + TBλd (cid:1) + log( T/δ ) , as long as γ 1 ≥ ¯ β , obtains a regret bound of R ( T )
Revised: Algorithm 1, with rerandomized history, ensemble size M = (cid:100) log(2 T/δ ) (cid:101) , λ = σ 2 /W 2 , and β = √ λW + √ 2 σ (cid:113) d log (cid:0) 1 + TBλd (cid:1) + log( T/δ ) , as long as γ 1 ≥ ¯ β , obtains a regret bound of R ( T )

Original: [Equation]
Revised: [Equation]

Original: , with probability at least 1 − δ
Revised: , with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: At time t , since we re-sample all the noisy targets, we have,
Revised: At time t , since we re-sample all the noisy targets, we have,

Original: [Equation]
Revised: [Equation]

Original: (cid:68) w ( j ) t , x t,a (cid:69) is therefore distributed as N (cid:16) 0 , (cid:107) x t,a (cid:107) 2˜Σ − 1 t (cid:17) .
Revised: (cid:68) w ( j ) t , x t,a (cid:69) is therefore distributed as N (cid:16) 0 , (cid:107) x t,a (cid:107) 2˜Σ − 1 t (cid:17) .

Original: Thus by Lemma 5, for all a ∈ A t with probability 1 − δ ,
Revised: Thus by Lemma 5, for all a ∈ A t with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: , and for a ∗ ,
Revised: , and for a ∗ ,

Original: [Equation]
Revised: [Equation]

Original: where c 1 , c 2 > 0 are ﬁxed constants.
Revised: where c 1 , c 2 > 0 are ﬁxed constants.

Original: Since bonus t ( a ) =
Revised: Since bonus t ( a ) =

Original: β max j ∈ [ M ]
Revised: β max j ∈ [ M ]

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: x t,a
Revised: x t,a

Original: [Equation]
Revised: [Equation]

Original: , we get a lower and upper bound on the bonus in terms of the LinUCB bonus.
Revised: , we get a lower and upper bound on the bonus in terms of the LinUCB bonus.

Original: Lastly, by the choice of M , with probability 1 − δ , we satisfy conditions of Lemma 7 with c 1 √ log 2 β = √ λW + √
Revised: Lastly, by the choice of M , with probability 1 − δ , we satisfy conditions of Lemma 7 with c 1 √ log 2 β = √ λW + √

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: 1 + TBλd (cid:1) + log( T/δ ) .
Revised: 1 + TBλd (cid:1) + log( T/δ ) .

Original: This gives us
Revised: This gives us

Original: [Equation]
Revised: [Equation]

Original: Proposition 3 (ACB regret bound with incremental updates) .
Revised: Proposition 3 (ACB regret bound with incremental updates) .

Original: Assume further that the η t are i.i.d.
Revised: Assume further that the η t are i.i.d.

Original: σ 2 -subgaussian random variables.
Revised: σ 2 -subgaussian random variables.

Original: Algorithm 1, with incrementally randomized history, ensemble size M = (cid:100) log( T/δ ) (cid:101) , λ = σ 2 /W 2 , and β = √ λW + √ 2 σ (cid:113)
Revised: Algorithm 1, with incrementally randomized history, ensemble size M = (cid:100) log( T/δ ) (cid:101) , λ = σ 2 /W 2 , and β = √ λW + √ 2 σ (cid:113)

Original: A log (cid:0) 1 + TBλA (cid:1) + log( T/δ ) , as long as γ 1 ≥ ¯ β , obtains a regret bound of
Revised: A log (cid:0) 1 + TBλA (cid:1) + log( T/δ ) , as long as γ 1 ≥ ¯ β , obtains a regret bound of

Original: [Equation]
Revised: [Equation]

Original: in the multi-armed bandit setting, with probability at least 1 − δ .
Revised: in the multi-armed bandit setting, with probability at least 1 − δ .

Original: Proof.
Revised: Proof.

Original: Recall that the action set A is ﬁxed, and d = A ; we will use x a to denote x t,a .
Revised: Recall that the action set A is ﬁxed, and d = A ; we will use x a to denote x t,a .

Original: We follow a “reward tape” argument (see, e.g., Slivkins (2019), Section 1.3.1) to show that the conditions of
Revised: We follow a “reward tape” argument (see, e.g., Slivkins (2019), Section 1.3.1) to show that the conditions of

Original: Lemma 7 hold.
Revised: Lemma 7 hold.

Original: Notice that in the incremental version of Algorithm 1, bonus t ( a ) depends only on the targets
Revised: Notice that in the incremental version of Algorithm 1, bonus t ( a ) depends only on the targets

Original: y ( j ) τ for − d + 1 ≤ τ < t , the number of times a (call this N t ( a ) ) has been selected up to time t − 1 , and the reward noise variables ε τ for 1 ≤ τ < t , such that a τ = a .
Revised: y ( j ) τ for − d + 1 ≤ τ < t , the number of times a (call this N t ( a ) ) has been selected up to time t − 1 , and the reward noise variables ε τ for 1 ≤ τ < t , such that a τ = a .

Original: This is because (cid:68) w ( j ) t , x
Revised: This is because (cid:68) w ( j ) t , x

Original: [Equation]
Revised: [Equation]

Original: (cid:69) =
Revised: (cid:69) =

Original: [Equation]
Revised: [Equation]

Original: − d +1 1 a τ = a y ( j ) τ in the case where all the x a are orthogonal.
Revised: − d +1 1 a τ = a y ( j ) τ in the case where all the x a are orthogonal.

Original: Consider the following procedure: sample TA (1 + M ) independent random variables ahead of time: η τ,a , an i.i.d copy of η t for each τ ∈ [ T ] , a ∈ [ A ] , and y ( j ) τ,a ∼ N (0 , 1) for each τ ∈ [ T ] , a ∈ [ A ] , j ∈ [ M ] .
Revised: Consider the following procedure: sample TA (1 + M ) independent random variables ahead of time: η τ,a , an i.i.d copy of η t for each τ ∈ [ T ] , a ∈ [ A ] , and y ( j ) τ,a ∼ N (0 , 1) for each τ ∈ [ T ] , a ∈ [ A ] , j ∈ [ M ] .

Original: Then, run a deterministic version of Algorithm 1, which at time t uses these y ( j ) τ,a : τ = 1 , . .
Revised: Then, run a deterministic version of Algorithm 1, which at time t uses these y ( j ) τ,a : τ = 1 , . .

Original: . , N t ( a ) as the random regression targets to compute bonus t ( a ) , and observes reward (cid:104) x a t , θ (cid:63) (cid:105) + η N t ( a t )+1 ,a t .
Revised: . , N t ( a ) as the random regression targets to compute bonus t ( a ) , and observes reward (cid:104) x a t , θ (cid:63) (cid:105) + η N t ( a t )+1 ,a t .

Original: Then, for any θ ∗ , the distribution of action sequences taken by this procedure is identical to that of Algorithm 1 with sequentially sampled η t and y ( j ) t , and Lemmaholds.
Revised: Then, for any θ ∗ , the distribution of action sequences taken by this procedure is identical to that of Algorithm 1 with sequentially sampled η t and y ( j ) t , and Lemmaholds.

Original: By Lemma 5, with probability at least 1 − δ (cid:48) , we have that for any t ∈
Revised: By Lemma 5, with probability at least 1 − δ (cid:48) , we have that for any t ∈

Original: [ T ] , a ∈ [ A ] ,
Revised: [ T ] , a ∈ [ A ] ,

Original: [Equation]
Revised: [Equation]

Original: and for a ∗ ,
Revised: and for a ∗ ,

Original: [Equation]
Revised: [Equation]

Original: for absolute constants c 1 , c 2 > 0 , so that the conditions of Lemma 7 hold with the same choice of ¯ β .
Revised: for absolute constants c 1 , c 2 > 0 , so that the conditions of Lemma 7 hold with the same choice of ¯ β .

Original: Selecting δ (cid:48)
Revised: Selecting δ (cid:48)

Original: =
Revised: =

Original: δ/ (2 AT ) , we have by the union bound that this condition holds for all t ∈ [ T ] , a ∈ [ A ] , and the proof reduces to that of Proposition 2, with an identical regret bound of
Revised: δ/ (2 AT ) , we have by the union bound that this condition holds for all t ∈ [ T ] , a ∈ [ A ] , and the proof reduces to that of Proposition 2, with an identical regret bound of

Original: [Equation]
Revised: [Equation]

Original: Note here that A = d , and that T ≥ A is necessary for a non-vacuous regret bound, so that log A ≤ O (log T ) factors in the main paper are suppressed by the ˜ O ( · ) notation.
Revised: Note here that A = d , and that T ≥ A is necessary for a non-vacuous regret bound, so that log A ≤ O (log T ) factors in the main paper are suppressed by the ˜ O ( · ) notation.

Original: C L AZILY RE - RANDOMIZED ACB
Revised: C L AZILY RE - RANDOMIZED ACB

Original: Parameters: ensemble size M ; β, γ, λ , ﬁxed action set A 1:
Revised: Parameters: ensemble size M ; β, γ, λ , ﬁxed action set A 1:

Original: For all i = 1 , . . .
Revised: For all i = 1 , . . .

Original: , d , set warm-start features a − i +1 = √
Revised: , d , set warm-start features a − i +1 = √

Original: λe
Revised: λe

Original: i 2: Set τ =3:
Revised: i 2: Set τ =3:

Original: Set ω =4: for t = 1 , · · · , T : do 5: if ω = 0 then 6:
Revised: Set ω =4: for t = 1 , · · · , T : do 5: if ω = 0 then 6:

Original: Update parameters:
Revised: Update parameters:

Original: ˆ θ t := arg min θ t − 1 (cid:88) i =
Revised: ˆ θ t := arg min θ t − 1 (cid:88) i =

Original: ( (cid:104)
Revised: ( (cid:104)

Original: a i , θ (cid:105) − r i ( a i ))
Revised: a i , θ (cid:105) − r i ( a i ))

Original: 2 + λ (cid:107) θ
Revised: 2 + λ (cid:107) θ

Original: (cid:107)7:
Revised: (cid:107)7:

Original: Sample new targets y ( j ) t (cid:48) ∼ N (0 , 1) for each j ∈ [ M ] , t (cid:48) ∈ {− d + 1 , . .
Revised: Sample new targets y ( j ) t (cid:48) ∼ N (0 , 1) for each j ∈ [ M ] , t (cid:48) ∈ {− d + 1 , . .

Original: , t − 1 } 8:
Revised: , t − 1 } 8:

Original: Update bonus ensemble for each j ∈
Revised: Update bonus ensemble for each j ∈

Original: [ M ] , w ( j ) t := arg min w t − 1 (cid:88) t (cid:48)
Revised: [ M ] , w ( j ) t := arg min w t − 1 (cid:88) t (cid:48)

Original: = − d + (cid:16) (cid:104) a t (cid:48) , w (cid:105) − y
Revised: = − d + (cid:16) (cid:104) a t (cid:48) , w (cid:105) − y

Original: ( j ) t (cid:48)
Revised: ( j ) t (cid:48)

Original: (cid:17)9:
Revised: (cid:17)9:

Original: Compute per-arm bonuses bonus t ( a ) :
Revised: Compute per-arm bonuses bonus t ( a ) :

Original: = β · max j ∈
Revised: = β · max j ∈

Original: [ M ] (cid:12)(cid:12)(cid:12)(cid:68) a, w ( j ) τ (cid:69)(cid:12)(cid:12)(cid:12) for each a ∈
Revised: [ M ] (cid:12)(cid:12)(cid:12)(cid:68) a, w ( j ) τ (cid:69)(cid:12)(cid:12)(cid:12) for each a ∈

Original: A 10:
Revised: A 10:

Original: Set s t := arg max a ∈
Revised: Set s t := arg max a ∈

Original: [ A ] (cid:104) a, ˆ θ t (cid:105) + bonus t ( a ) 11:
Revised: [ A ] (cid:104) a, ˆ θ t (cid:105) + bonus t ( a ) 11:

Original: Set ω = (cid:108) γ bonus 2 ( s t ) (cid:109) 12: τ = t 13:
Revised: Set ω = (cid:108) γ bonus 2 ( s t ) (cid:109) 12: τ = t 13:

Original: Choose action a t = a τ 14:
Revised: Choose action a t = a τ 14:

Original: Observe reward r t ( a t ) =
Revised: Observe reward r t ( a t ) =

Original: (cid:104) s t , θ ∗ (cid:105) + ε t 15:
Revised: (cid:104) s t , θ ∗ (cid:105) + ε t 15:

Original: Decrease ω by
Revised: Decrease ω by

Original: In this section, we present the algorithm (see Algorithm 3) and regret analysis of a lazy-version of
Revised: In this section, we present the algorithm (see Algorithm 3) and regret analysis of a lazy-version of

Original: ACB which only rarely re-randomizes history.
Revised: ACB which only rarely re-randomizes history.

Original: C. U SEFUL L EMMAS
Revised: C. U SEFUL L EMMAS

Original: Lemma 8 (Ensemble Bounds) .
Revised: Lemma 8 (Ensemble Bounds) .

Original: For M = log(2 AT/δ ) , at all updates τ , with probability 1 − δ , for all a ∈ A , β (cid:107) a (cid:107) ˜Σ − 1 t (cid:114)
Revised: For M = log(2 AT/δ ) , at all updates τ , with probability 1 − δ , for all a ∈ A , β (cid:107) a (cid:107) ˜Σ − 1 t (cid:114)

Original: π log ≤ bonus τ ( a ) ≤ β (cid:107) a (cid:107) ˜Σ − 1 t (cid:115) 8 log (cid:18) 2 AT δ (cid:19) .
Revised: π log ≤ bonus τ ( a ) ≤ β (cid:107) a (cid:107) ˜Σ − 1 t (cid:115) 8 log (cid:18) 2 AT δ (cid:19) .

Original: Proof.
Revised: Proof.

Original: Follows by applying Lemma 5 and taking union bound over A and t ≤ T .
Revised: Follows by applying Lemma 5 and taking union bound over A and t ≤ T .

Original: Lemma 9.
Revised: Lemma 9.

Original: Let τ be the time of an update and γ = β 2 π log22 .
Revised: Let τ be the time of an update and γ = β 2 π log22 .

Original: Then, for τ (cid:48) =
Revised: Then, for τ (cid:48) =

Original: τ + (cid:108) γ bonus 2 τ ( a τ ) (cid:109) , det(˜Σ τ (cid:48) ) det( ˜ Σ τ ) ≥ (cid:115) 

Revised: τ + (cid:108) γ bonus 2 τ ( a τ ) (cid:109) , det(˜Σ τ (cid:48) ) det( ˜ Σ τ ) ≥ (cid:115)

Original: + π log16 log(2 AT/δ ) .
Revised: + π log16 log(2 AT/δ ) .

Original: Proof.
Revised: Proof.

Original: Note that from τ to τ (cid:48) , the action taken remains the same.
Revised: Note that from τ to τ (cid:48) , the action taken remains the same.

Original: For the upper bound, using Lemma 11 from Abbasi-Yadkori et al.
Revised: For the upper bound, using Lemma 11 from Abbasi-Yadkori et al.

Original: (2011) and some elementary calculus:
Revised: (2011) and some elementary calculus:

Original: [Equation]
Revised: [Equation]

Original: Lemma 10.
Revised: Lemma 10.

Original: Let τ t be the largest index ≤ t when the update happened and γ = β 2 π log22 , then det
Revised: Let τ t be the largest index ≤ t when the update happened and γ = β 2 π log22 , then det

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Suppose t = τ t , then the bound trivially holds, thus WLOG, we assume t > τ t .
Revised: Suppose t = τ t , then the bound trivially holds, thus WLOG, we assume t > τ t .

Original: By our algorithm, we know that, 0 < t − τ t ≤ (cid:108) γ bonus 2 τt ( a τ
Revised: By our algorithm, we know that, 0 < t − τ t ≤ (cid:108) γ bonus 2 τt ( a τ

Original: [Equation]
Revised: [Equation]

Original: (cid:109) .
Revised: (cid:109) .

Original: This implies, bonus 2 τ ( a τ ) ≤ γ .
Revised: This implies, bonus 2 τ ( a τ ) ≤ γ .

Original: Now, using
Revised: Now, using

Original: Lemma 11 from Abbasi-Yadkori et al.
Revised: Lemma 11 from Abbasi-Yadkori et al.

Original: (2011) and some elementary calculus:
Revised: (2011) and some elementary calculus:

Original: [Equation]
Revised: [Equation]

Original: C.
Revised: C.

Original: R
Revised: R

Original: EGRET
Revised: EGRET

Original: B
Revised: B

Original: OUND
Revised: OUND

Original: Theorem 4 (ACB regret bound with lazy updates) .
Revised: Theorem 4 (ACB regret bound with lazy updates) .

Original: Algorithm 3, with lazy rerandomized history, ensemble size M
Revised: Algorithm 3, with lazy rerandomized history, ensemble size M

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , and β =
Revised: , and β =

Original: [Equation]
Revised: [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: and γ =
Revised: and γ =

Original: [Equation]
Revised: [Equation]

Original: log22 , obtains a regret bound of
Revised: log22 , obtains a regret bound of

Original: [Equation]
Revised: [Equation]

Original: in the ﬁxed action linear bandit setting, with probability at least 1 − δ .
Revised: in the ﬁxed action linear bandit setting, with probability at least 1 − δ .

Original: The algorithm re-randomizes at most O ( d log + ( TBW/σd ) log
Revised: The algorithm re-randomizes at most O ( d log + ( TBW/σd ) log

Original: ( AT/δ )) times.
Revised: ( AT/δ )) times.

Original: Proof.
Revised: Proof.

Original: Let τ t be the largest index ≤ t when the update happened, then
Revised: Let τ t be the largest index ≤ t when the update happened, then

Original: [Equation]
Revised: [Equation]

Original: 8 log (cid:18) 2 AT δ (cid:19) + β τ t (cid:33) (cid:107) a τ t (cid:107) ˜Σ − 1 τt + (cid:32)
Revised: 8 log (cid:18) 2 AT δ (cid:19) + β τ t (cid:33) (cid:107) a τ t (cid:107) ˜Σ − 1 τt + (cid:32)

Original: β t − β (cid:114) π log (cid:33) (cid:107) a ∗ (cid:107) ˜Σ − 1 τt ≤ (cid:32) β (cid:115) 8 log (cid:18) 

Revised: β t − β (cid:114) π log (cid:33) (cid:107) a ∗ (cid:107) ˜Σ − 1 τt ≤ (cid:32) β (cid:115) 8 log (cid:18)

Original: AT δ (cid:19) + β τ t (cid:33) (cid:107) a t (cid:107) ˜Σ − 1 t det(˜Σ t ) det(˜Σ τ t ) + (cid:32) β t − β (cid:114) π log (cid:33) (cid:107) a ∗ (cid:107) ˜Σ − 1 τt ≤ 2 e (cid:32) β (cid:115) 8 log (cid:18) 

Revised: AT δ (cid:19) + β τ t (cid:33) (cid:107) a t (cid:107) ˜Σ − 1 t det(˜Σ t ) det(˜Σ τ t ) + (cid:32) β t − β (cid:114) π log (cid:33) (cid:107) a ∗ (cid:107) ˜Σ − 1 τt ≤ 2 e (cid:32) β (cid:115) 8 log (cid:18)

Original: AT δ (cid:19) + β τ t (cid:33) (cid:107) a t (cid:107) ˜Σ − 1 t + (cid:32) β t − β (cid:114) π log (cid:33) (cid:107) a ∗ (cid:107) ˜Σ − 1 τt where the ﬁrst inequality comes from the validity of the conﬁdence bound for a ∗ , the second from the fact that a τ t = arg max a (cid:68) a, ˆ θ τ t (cid:69) + bonus τ t ( a ) , the third from the validity of the conﬁdence bound for a τ t , the fourth follows from Lemma 8, the ﬁfth follows from Lemma 12 Abbasi-Yadkori et al.
Revised: AT δ (cid:19) + β τ t (cid:33) (cid:107) a t (cid:107) ˜Σ − 1 t + (cid:32) β t − β (cid:114) π log (cid:33) (cid:107) a ∗ (cid:107) ˜Σ − 1 τt where the ﬁrst inequality comes from the validity of the conﬁdence bound for a ∗ , the second from the fact that a τ t = arg max a (cid:68) a, ˆ θ τ t (cid:69) + bonus τ t ( a ) , the third from the validity of the conﬁdence bound for a τ t , the fourth follows from Lemma 8, the ﬁfth follows from Lemma 12 Abbasi-Yadkori et al.

Original: (2011) and lastly the sixth follows from Lemma 10.
Revised: (2011) and lastly the sixth follows from Lemma 10.

Original: Setting δ t = δ/T we have β τ t ≤ β (cid:113)
Revised: Setting δ t = δ/T we have β τ t ≤ β (cid:113)

Original: π
Revised: π

Original: log22 , therefore, regret t ≤ 2 e (cid:32)(cid:115) 8 log (cid:18) 2 AT δ (cid:19)
Revised: log22 , therefore, regret t ≤ 2 e (cid:32)(cid:115) 8 log (cid:18) 2 AT δ (cid:19)

Original: [Equation]
Revised: [Equation]

Original: Using the above, we have
Revised: Using the above, we have

Original: [Equation]
Revised: [Equation]

Original: Lastly, we need to bound the number of times we refresh randomness.
Revised: Lastly, we need to bound the number of times we refresh randomness.

Original: By Lemma 9, each time we refresh, we increase det(˜Σ t ) by a factor of (cid:113) 

Revised: By Lemma 9, each time we refresh, we increase det(˜Σ t ) by a factor of (cid:113)

Original: + π log2 16log(

Revised: + π log2 16log(

Original: AT/δ ) .
Revised: AT/δ ) .

Original: Since det(˜Σ t ) / det( λI ) is bounded by (1+ TB/λd ) d , the number of times we refresh is k
Revised: Since det(˜Σ t ) / det( λI ) is bounded by (1+ TB/λd ) d , the number of times we refresh is k

Original: = O ( d log + ( TBW/σd ) log
Revised: = O ( d log + ( TBW/σd ) log

Original: ( AT/δ )) .
Revised: ( AT/δ )) .

Original: Formally, k must satisfy (cid:16) 1 +
Revised: Formally, k must satisfy (cid:16) 1 +

Original: [Equation]
Revised: [Equation]