Original: Scalable Representation Learning in Linear Contextual Bandits with Constant Regret Guarantees
Revised: Scalable Representation Learning in Linear Contextual Bandits with Constant Regret Guarantees

Revised: Anonymous Author(s)

Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: We study the problem of representation learning in stochastic contextual linearbandits.
Revised: We study the problem of representation learning in stochastic contextual linearbandits.

Original: While the primary concern in this domain is usually to find realizable representations (i.e., those that allow predicting the reward function at any context-actionpair exactly), it has been recently shown that representations with certain spectralproperties (called HLS ) may be more effective for the exploration-exploitation task,enabling LinUCB to achieve constant (i.e., horizon-independent) regret.
Revised: While the primary concern in this domain is usually to find realizable representations (i.e., those that allow predicting the reward function at any context-actionpair exactly), it has been recently shown that representations with certain spectralproperties (called HLS ) may be more effective for the exploration-exploitation task,enabling LinUCB to achieve constant (i.e., horizon-independent) regret.

Original: In thispaper, we propose B ANDIT SRL, a representation learning algorithm that combines a novel constrained optimization problem to learn a realizable representationwith good spectral properties with a generalized likelihood ratio test to exploit therecovered representation and avoid excessive exploration.
Revised: In thispaper, we propose B ANDIT SRL, a representation learning algorithm that combines a novel constrained optimization problem to learn a realizable representationwith good spectral properties with a generalized likelihood ratio test to exploit therecovered representation and avoid excessive exploration.

Original: We prove that B ANDIT SRL can be paired with any no-regret algorithm and achieve constant regretwhenever an HLS representation is available.
Revised: We prove that B ANDIT SRL can be paired with any no-regret algorithm and achieve constant regretwhenever an HLS representation is available.

Original: Furthermore, B ANDIT SRL can beeasily combined with deep neural networks and we show how regularizing towards
Revised: Furthermore, B ANDIT SRL can beeasily combined with deep neural networks and we show how regularizing towards

Original: HLS representations is beneficial in standard benchmarks.
Revised: HLS representations is beneficial in standard benchmarks.

Original:  Introduction
Revised:  Introduction

Original: The contextual bandit is a general framework to formalize the exploration-exploitation dilemmaarising in sequential decision-making problems such as recommendation systems, online advertising,and clinical trials [e.g., 1].
Revised: The contextual bandit is a general framework to formalize the exploration-exploitation dilemmaarising in sequential decision-making problems such as recommendation systems, online advertising,and clinical trials [e.g.

Original: When solving real-world problems where contexts and actions arecomplex and high-dimensional (e g., users’ social graph, items’ visual description), it is crucialto provide the bandit algorithm with a suitable representation of the context-action space.
Revised: When solving real-world problems where contexts and actions arecomplex and high-dimensional (e g., users’ social graph, items’ visual description), it is crucialto provide the bandit algorithm with a suitable representation of the context-action space.

Original: Whileseveral representation learning algorithms have been proposed in supervised learning and obtainedimpressing empirical results [e.g., 2, 3], how to efficiently learn representations that are effective forthe exploration-exploitation problem is still relatively an open question.
Revised: Whileseveral representation learning algorithms have been proposed in supervised learning and obtainedimpressing empirical results [e.g., 2, 3], how to efficiently learn representations that are effective forthe exploration-exploitation problem is still relatively an open question.

Original: The primary objective in representation learning is to find features that map the context-action spaceinto a lower-dimensional embedding that allows fitting the reward function accurately, i e., realizable representations [e.g., 4–10].
Revised: The primary objective in representation learning is to find features that map the context-action spaceinto a lower-dimensional embedding that allows fitting the reward function accurately, i e., realizable representations [e.g., 4–9].

Original: Within the space of realizable representations, bandit algorithmsleveraging features of smaller dimension are expected to learn faster and thus have smaller regret.
Revised: Within the space of realizable representations, bandit algorithmsleveraging features of smaller dimension are expected to learn faster and thus have smaller regret.

Original: Nonetheless, Papini et al.
Revised: Nonetheless, Papini et al.

Original: [11] have recently shown that, even among realizable features, certain representations are naturally better suited to solve the exploration-exploitation problem.
Revised: [10] have recently shown that, even among realizable features, certainrepresentations are naturally better suited to solve the exploration-exploitation problem.

Original: In particular, they proved that L IN UCB [12, 13] can achieve constant regret when provided with a “good” repre- sentation.
Revised: In particular,they proved that L IN UCB [11, 12] can achieve constant regret when provided with a “good” representation.

Original: Interestingly, this property is not related to “global” characteristics of the feature map (e.g.,dimension, norms), but rather on a spectral property of the representation (the space associated tooptimal actions should cover the context-action space, see HLS property in Def. 2.1).
Revised: Interestingly, this property is not related to “global” characteristics of the feature map (e.g.,dimension, norms), but rather on a spectral property of the representation (the space associated tooptimal actions should cover the context-action space, see HLS property in Def. 2.1).

Original: This naturallyraises the question whether it is possible to learn such representation at the same time as solving the contextual bandit problem.
Revised: This naturallyraises the question whether it is possible to learn such representation at the same time as solvingthe contextual bandit problem.

Original: Papini et al.
Revised: Papini et al.

Original: [11] provided a first positive answer with the L EADERalgorithm, which is proved to perform as well as the best realizable representation in a given set up to a logarithmic factor in the number of representations.
Revised: [10] provided a first positive answer with the L EADERalgorithm, which is proved to perform as well as the best realizable representation in a given set up to a logarithmic factor in the number of representations.

Original: While this allows constant regret when a realizable HLS representation is available, the algorithm suffers from two main limitations: 1) itis entangled with L IN UCB and it can hardly be generalized to other bandit algorithms; 2) it learnsa different representation for each context-action pair, thus making it hard to extend beyond finiterepresentations to arbitrary functional space (e.g., deep neural networks).
Revised: While this allows constant regret when arealizable HLS representation is available, the algorithm suffers from two main limitations: 1) itis entangled with L IN UCB and it can hardly be generalized to other bandit algorithms; 2) it learnsa different representation for each context-action pair, thus making it hard to extend beyond finiterepresentations to arbitrary functional space (e.g., deep neural networks).

Original: In this paper, we address those limitations through B ANDIT SRL, a novel algorithm that decouplesrepresentation learning and exploration-exploitation so as to work with any no-regret contextual banditalgorithm and to be easily extended to general representation spaces.
Revised: In this paper, we address those limitations through B ANDIT SRL, a novel algorithm that decouplesrepresentation learning and exploration-exploitation so as to work with any no-regret contextual banditalgorithm and to be easily extended to general representation spaces.

Original: B ANDIT SRL combines twocomponents: 1) a representation learning mechanism based on a constrained optimization problemthat promotes “good” representations while preserving realizability; and 2) a generalized likelihoodratio test (GLRT) to avoid over exploration and fully exploit the properties of “good” representations.
Revised: B ANDIT SRL combines twocomponents: 1) a representation learning mechanism based on a constrained optimization problemthat promotes “good” representations while preserving realizability; and 2) a generalized likelihoodratio test (GLRT) to avoid over exploration and fully exploit the properties of “good” representations.

Original: The main contributions of the paper can be summarized as follows:
Revised: The main contributions of the paper can be summarized as follows:

Original: We show that adding a GLRT on the top of any no-regret algorithm enables it to exploit the
Revised: We show that adding a GLRT on the top of any no-regret algorithm enables it to exploit the

Original: properties of a HLS representation and achieve constant regret.
Revised: properties of a HLS representation and achieve constant regret.

Original: This generalizes the constant regret result for L IN UCB in [11] to any no-regret algorithm.2.
Revised: This generalizes the constantregret result for L IN UCB in [10] to any no-regret algorithm.2.

Original: Similarly, we show that B ANDIT SRL can be paired with any no-regret algorithm and performeffective representation selection, including achieving constant regret whenever a HLS representation is available in a given set.
Revised: Similarly, we show that B ANDIT SRL can be paired with any no-regret algorithm and performeffective representation selection, including achieving constant regret whenever a HLS representation is available in a given set.

Original: This generalizes the result of L EADER beyond L IN UCB.In doing this we also improve the analysis of the misspecified case and prove a tighter boundon the time to converge to realizable representations.
Revised: This generalizes the result of L EADER beyond L IN UCB. In doing this we also improve the analysis of the misspecified case and prove a tighter boundon the time to converge to realizable representations.

Original: Furthermore, numerical simulations in synthetic problems confirm that B ANDIT SRL is empirically competitive with L EADER .
Revised: Furthermore, numerical simulations insynthetic problems confirm that B ANDIT SRL is empirically competitive with L EADER .

Original: Finally, in contrast to L EADER , B ANDIT SRL can be easily scaled to complex problems where representations are encoded through deep neural networks.
Revised: Finally, in contrast to L EADER , B ANDIT SRL can be easily scaled to complex problems whererepresentations are encoded through deep neural networks.

Original: In particular, we show that the La- grangian relaxation of the constrained optimization problem for representation learning becomes a regression problem with an auxiliary representation loss promoting HLS-like representations.
Revised: In particular, we show that the Lagrangian relaxation of the constrained optimization problem for representation learning becomesa regression problem with an auxiliary representation loss promoting HLS-like representations.

Original: We test different variants of the resulting NN-B ANDIT SRL algorithm showing how the auxiliaryrepresentation loss improves performance in a number of dataset-based benchmarks.
Revised: We test different variants of the resulting NN-B ANDIT SRL algorithm showing how the auxiliaryrepresentation loss improves performance in a number of dataset-based benchmarks.

Original:  Preliminaries
Revised:  Preliminaries

Original: We consider a stochastic contextual bandit problem with context space X and finite action set A .
Revised: We consider a stochastic contextual bandit problem with context space X and finite action set A .

Original: At each round t ≥ 1 , the learner observes a context x t sampled i.i.d.
Revised: At each round t ≥ 1 , the learner observes a context x t sampled i.i.d.

Original: from a distribution ρ over X ,selects an action a t ∈ A , and receives a reward y t = µ ( x t , a ) + η t where η t is a zero-mean noise andµ :
Revised: from a distribution ρ over X ,selects an action a t ∈ A , and receives a reward y t = µ ( x t , a ) + η t where η t is a zero-mean noise andµ :

Original: X × A → R is the expected reward.
Revised: X × A → R is the expected reward.

Original: The objective of a learner A is to minimize its pseudo-regret
Revised: The objective of a learner A is to minimize its pseudo-regret

Original: R T := (cid:80) Tt =1 (cid:0) µ ⋆ ( x t ) − µ ( x t , a t ) (cid:1) for any T ≥ 1 , where µ ⋆ ( x t ) :
Revised: R T := (cid:80) Tt =1 (cid:0) µ ⋆ ( x t ) − µ ( x t , a t ) (cid:1) for any T ≥ 1 , where µ ⋆ ( x t ) :

Original: = max a ∈A µ ( x t , a ) .
Revised: = max a ∈A µ ( x t , a ) .

Original: We assumethat for any x ∈ X the optimal action
Revised: We assumethat for any x ∈ X the optimal action

Original: a ⋆x := argmax a ∈A µ ( x, a ) is unique and we define the gap∆( x, a ) :
Revised: a ⋆x := argmax a ∈A µ ( x, a ) is unique and we define the gap∆( x, a ) :

Original: = µ ⋆ ( x ) − µ ( x, a ) .
Revised: = µ ⋆ ( x ) − µ ( x, a ) .

Original: We say that A is a no-regret algorithm if, for any instance of µ , itachieves sublinear regret, i.e., R T
Revised: We say that A is a no-regret algorithm if, for any instance of µ , itachieves sublinear regret, i.e., R T

Original: =
Revised: =

Original: o ( T ) .
Revised: o ( T ) .

Original: We consider the problem of representation learning in given a candidate function space Φ ⊆ 73 (cid:8) ϕ :
Revised: We consider the problem of representation learning in given a candidate function space Φ ⊆ 73 (cid:8) ϕ :

Original: X × A → R d ϕ (cid:9) , where the dimensionality d ϕ may depend on the feature ϕ .
Revised: X × A → R d ϕ (cid:9) , where the dimensionality d ϕ may depend on the feature ϕ .

Original: Letθ ⋆ ϕ = argmin θ ∈ R d ϕ E x ∼ ρ (cid:2) (cid:80) a ( ϕ ( x, a ) T θ − µ ( x, a )) 2 (cid:3) be the best linear fit of µ for representation ϕ .
Revised: Letθ ⋆ϕ = argmin θ ∈ R dϕ E x ∼ ρ (cid:2) (cid:80) a ( ϕ ( x, a ) T θ − µ ( x, a )) 2 (cid:3) be the best linear fit of µ for representation ϕ .

Original: We assume that Φ contains a linearly realizable representation.
Revised: We assume that Φ contains a linearly realizable representation.

Original: Assumption 1 (Realizability) .
Revised: Assumption 1 (Realizability) .

Original: There exists an (unknown) subset Φ ⋆ ⊆
Revised: There exists an (unknown) subset Φ ⋆ ⊆

Original: Φ such that, for each ϕ ∈ Φ ⋆ ,µ ( x, a )
Revised: Φ such that, for each ϕ ∈ Φ ⋆ ,µ ( x, a )

Original: =
Revised: =

Original: ϕ
Revised: ϕ

Original: ( x, a ) T θ ⋆ϕ , ∀ x ∈ X , a ∈ A .
Revised: ( x, a ) T θ ⋆ϕ , ∀ x ∈ X , a ∈ A .

Original: Assumption 2 (Regularity) .
Revised: Assumption 2 (Regularity) .

Original: Let B ϕ :
Revised: Let B ϕ :

Original: = { θ ∈ R d ϕ : ∥ θ ∥ 2 ≤ B ϕ } be a ball in R d ϕ .
Revised: = { θ ∈ R d ϕ : ∥ θ ∥ 2 ≤ B ϕ } be a ball in R d ϕ .

Original: We assume that, for each ϕ ∈ Φ , sup x,a ∥ ϕ ( x, a ) ∥ 2 ≤ L ϕ , ∥ θ ⋆ϕ ∥ 2 ≤ B ϕ , sup x,a | ϕ ( x, a ) T θ | ≤ 1for any θ ∈ B ϕ and | y t | ≤ 1 almost surely for all t .
Revised: We assume that,for each ϕ ∈ Φ , sup x,a ∥ ϕ ( x, a ) ∥ 2 ≤ L ϕ , ∥ θ ⋆ϕ ∥ 2 ≤ B ϕ , sup x,a | ϕ ( x, a ) T θ | ≤ 1 for any θ ∈ B ϕand | y t | ≤ 1 almost surely for all t .

Original: We assume parameters L ϕ and B ϕ are81 known.
Revised: We assume parameters L ϕ and B ϕ are known.

Original: We also  assume the minimum gap ∆ = inf x ∈X : ρ ( x ) > 0 ,a ∈A , ∆( x,a ) >{ ∆( x, a ) } > 0 and thatλ min (cid:16) 1 |A| (cid:80) a E x ∼ ρ
Revised: We also 81 assume the minimum gap ∆ = inf [Equation] [Equation]

Original: [ ϕ ( x, a ) ϕ
Revised: [Equation]

Original: ( x, a ) T ] (cid:17) > 0 , for any ϕ ∈ Φ ⋆ , i.e, all realizable representations arenon-redundant,
Revised: x, a ) } > 0 and that all realizablerepresentations are non-redundant, i.e., λ [Equation] (cid:16)

Original:  Under Asm. 1, when | Φ | = 1 , the problem reduces to a stochastic linear contextual bandit and can besolved using standard algorithms, such as L IN UCB/OFUL [12, 13], LinTS [14], and ϵ -greedy
Revised: [Equation] 0 , for any ϕ ∈ Φ ⋆ . Under Asm. 1, when | Φ | = 1 , the problem reduces to a stochastic linear contextual bandit and can besolved using standard algorithms, such as L IN UCB/OFUL [11, 12], LinTS [13], and ϵ -greedy

Original: [15],which enjoy sublinear regret and, in some cases, logarithmic problem-dependent regret.
Revised: [14],which enjoy sublinear regret and, in some cases, logarithmic problem-dependent regret.

Original: Recently,
Revised: Recently,

Original: Papini et al.
Revised: Papini et al.

Original: [11] showed that L IN UCB only suffers constant regret when a realizable representationis HLS, i.e., when the features of optimal actions span the entire d ϕ -dimensional space.HLS
Revised: [10] showed that L IN UCB only suffers constant regret when a realizable representationis HLS, i.e., when the features of optimal actions span the entire d ϕ -dimensional space.

Original: Definition 2.1 (HLS Representation) .
Revised: Definition 2.1 (HLS Representation) .

Original: A representation ϕ is HLS(refers to the intials of the authorsof [16]) if
Revised: A representation ϕ is HLS if

Original: [Equation]
Revised: [Equation]

Original: where λ min ( A ) denotes the minimum eigenvalue of a matrix A .
Revised: where λ min ( A ) denotes the minimum eigenvalue of a matrix A .

Original: Papini et al.
Revised: Papini et al.

Original: showed that HLS, together with realizability, is a sufficient and necessary property forachieving constant regret in contextual stochastic linear bandits for non-redundant representations.
Revised: showed that HLS, together with realizability, is a sufficient and necessary property forachieving constant regret in contextual stochastic linear bandits for non-redundant representations.

Original: In order to deal with the general case where Φ may contain non-realizable representations, we rely onthe following misspecification assumption from [11].
Revised: In order to deal with the general case where Φ may contain non-realizable representations, we rely onthe following misspecification assumption from [10].

Original: Assumption 3 (Misspecification) .
Revised: Assumption 3 (Misspecification) .

Original: For each ϕ / ∈ Φ ⋆ , there exists ϵ ϕ > 0 such that
Revised: For each ϕ / ∈ Φ ⋆ , there exists ϵ ϕ > 0 such that

Original: [Equation]
Revised: [Equation]

Original: This assumption states that any non-realizable representation has a minimum level of misspecificationon average over contexts and for any context-action policy.
Revised: This assumption states that any non-realizable representation has a minimum level of misspecificationon average over contexts and for any context-action policy.

Original: In the finite-context case, a sufficientcondition for Asm. 3, for each ϕ / ∈ Φ ⋆ , there exists a context x ∈ X with ρ ( x ) > 0 such thatϕ ( x, a ) T θ ̸ = µ ( x, a ) for all a ∈ A and θ ∈ B ϕ .
Revised: In the finite-context case, a sufficientcondition for Asm. 3, for each ϕ / ∈ Φ ⋆ , there exists a context x ∈ X with ρ ( x ) > 0 such thatϕ ( x, a ) T θ ̸ = µ ( x, a ) for all a ∈ A and θ ∈ B ϕ .

Original: Related work.
Revised: Related work.

Original: Several papers have focused on contextual bandits with an arbitrary function spaceto estimate the reward function under realizability assumptions [e.g., 4, 5, 7].
Revised: Several papers have focused on contextual bandits with an arbitrary function spaceto estimate the reward function under realizability assumptions [e.g., 4, 5, 7].

Original: While these worksconsider a similar setting to ours, they do not aim to learn “good” representations, but rather focus onthe exploration-exploitation problem to obtain sublinear regret guarantees.
Revised: While these worksconsider a similar setting to ours, they do not aim to learn “good” representations, but rather focus onthe exploration-exploitation problem to obtain sublinear regret guarantees.

Original: This often correspondsto recovering the maximum likelihood representation, which may not lead to the best regret.
Revised: This often correspondsto recovering the maximum likelihood representation, which may not lead to the best regret.

Original: Afterthe work in [11], the problem of representation learning with constant regret guarantees has alsobeen studied in reinforcement learning [17, 18].
Revised: Afterthe work in [10], the problem of representation learning with constant regret guarantees has alsobeen studied in reinforcement learning [15, 16].

Original: As these approaches build on the ideas in [11], theyinherit the same limitations as [11].
Revised: As these approaches build on the ideas in [10], theyinherit the same limitations as [10].

Original: Another related literature is the one of expert learning and model selection in bandits [e.g., 19–25],where the objective is to select the best candidate among a set of base learning algorithms or experts.
Revised: Another related literature is the one of expert learning and model selection in bandits [e.g., 17–23],where the objective is to select the best candidate among a set of base learning algorithms or experts.

Original: While these algorithms are general and can be applied to different settings, including representationlearning with a finite set of candidates, they may not be able to effectively leverage the specificstructure of the problem.
Revised: While these algorithms are general and can be applied to different settings, including representationlearning with a finite set of candidates, they may not be able to effectively leverage the specificstructure of the problem.

Original: Furthermore, at the best of our knowledge, these algorithms suffers apolynomial dependence in the number of base algorithms ( | Φ | in our setting) and are limited toworst-case regret guarantees.
Revised: Furthermore, at the best of our knowledge, these algorithms suffers apolynomial dependence in the number of base algorithms ( | Φ | in our setting) and are limited toworst-case regret guarantees.

Original: Whether the √ T or poly( | Φ | ) dependency can be improved in generalis an open question (see [25] and [11, App.
Revised: Whether the √ T or poly( | Φ | ) dependency can be improved in generalis an open question (see [23] and [10, App.

Original: A]).
Revised: A]).

Original: Finally, [8, 26] studied the specific problem ofmodel selection with nested linear representations, where the best representation is the one with thesmallest dimension for which the reward is realizable.
Revised: Finally, [8, 24] studied the specific problem ofmodel selection with nested linear representations, where the best representation is the one with thesmallest dimension for which the reward is realizable.

Original: Several works have recently focused on theoretical and practical investigation of contextual banditswith neural networks (NNs) [27–29].
Revised: Several works have recently focused on theoretical and practical investigation of contextual banditswith neural networks (NNs) [25–27].

Original: While their focus was leveraging the representation power of
Revised: While their focus was leveraging the representation power of

Original: NNs to correctly predict the rewards, here we focus on learning representations with good spectralproperties through a novel auxiliary loss.
Revised: NNs to correctly predict the rewards, here we focus on learning representations with good spectralproperties through a novel auxiliary loss.

Original: A related approach to our is [29] where the authors leverageself-supervised auxiliary losses for representation learning in image-based bandit problems.
Revised: A related approach to our is [27] where the authors leverageself-supervised auxiliary losses for representation learning in image-based bandit problems.

Original:  A General Framework for Representation Learning
Revised:  A General Framework for Representation Learning

Original: We introduce B ANDIT SRL ( Bandit Spectral Representation Learner ), an algorithm for stochastic contextual linear bandit that efficiently decouples representation learning from exploration-exploitation.
Revised: We introduce B ANDIT SRL ( Bandit Spectral Representation Learner ), an algorithm for stochastic contextual linear bandit that efficiently decouples representation learning from exploration-exploitation.

Original: As illustrated in Alg. 1, B ANDIT SRL has access to a fixed-representation contextual bandit algorithm
Revised: As illustrated in Alg. 1, B ANDIT SRL has access to a fixed-representation contextual bandit algorithm

Original: A , the base algorithm , and it is built around two key mechanisms: ❶ a constrained optimizationproblem where the objective is to minimize a representation loss L to favor representations withHLSproperties, whereas the constraint ensures realizability; ❷ a generalized likelihood ratio test (GLRT)to ensure that, if a HLSrepresentation is learned, the base algorithm A does not over-explore and the“good” representation is exploited to obtain constant regret.
Revised: A , the base algorithm , and it is built around two key mechanisms: ❶ a constrained optimizationproblem where the objective is to minimize a representation loss L to favor representations with HLS

Original: Mechanism ❶ (line 12).
Revised: Mechanism ❶ (line 12).

Original: The first challenge when provided with a generic set Φ is to ensure thatthe algorithm does not converge to selecting misspecified representations, which may lead to linearregret.
Revised: The first challenge when provided with a generic set Φ is to ensure thatthe algorithm does not converge to selecting misspecified representations, which may lead to linear

Original: This is achieved by introducing a hard constraint in the representation optimization, so that
Revised: Algorithm

Original: B ANDIT SRL only selects representations in the set (see also [11, App. F]),
Revised: 1: Input: representations Φ , no-regret algorithm A , confidence δ ∈ (0 , 1) , update schedule γ >2: Initialize j = 0 , ϕ j , θ ϕ j , 0 arbitrarily, V 0 ( ϕ j )

Original: [Equation]
Revised: =

Original: Input 2: Initialize d  t j δ j : (2( j
Revised: λI d ϕj , t j =3: for t = 1 , . . .

Original: ) 3: for t do 4:
Revised: do 4:

Original: Observe context x t 5: if GLR t − 1 (x t ; ϕ j ) > β t − 1 ,δ/ | Φ | ( ϕ j ) then 6:
Revised: Observe context x t 5: if GLR t − 1 (x t ; ϕ j ) > β t − 1 ,δ/ | Φ | ( ϕ j ) then 6:

Original: Play a t = argmax a ∈A (cid:8)
Revised: Play a t = argmax a ∈A (cid:8)

Original: ϕ j ( x t , a ) T θ ϕ j ,t − 1 (cid:9) and observe reward y t 7: else 8:
Revised: ϕ j ( x t , a ) T θ ϕ j ,t − 1 (cid:9) and observe reward y t 7: else 8:

Original: Play a t =
Revised: Play a t =

Original: A t (cid:0) x t ; ϕ j , δ j / | Φ | (cid:1) , observe reward y t , and feed it into A 9: end if 10: if t = ⌈ γt j ⌉ and | Φ | > 1 then 11:
Revised: A t ( x t ; ϕ j , δ/ | Φ | ) , observe reward y t , and feed it into A 9: end if 10: if t = ⌈ γt j ⌉ and | Φ | > 1 then 11:

Original: Set j = j + 1 and t j = t 12:
Revised: Set j = j + 1 and t j = t 12:

Original: Compute ϕ j = argmin ϕ ∈
Revised: Compute ϕ j = argmin ϕ ∈

Original: Φ t (cid:8) L t ( ϕ ) (cid:9) and reset A 13: end if 14: end for where E t ( ϕ, θ ) := 1 t (cid:80) t s =1 (cid:0) ϕ
Revised: Φ t (cid:8) L t ( ϕ ) (cid:9) and reset A 13: end if 14: end for regret. This is achieved by introducing a hard constraint in the representation optimization, so that

Revised: B ANDIT SRL only selects representations in the set (see also [10, App. F]),

Original:  ( x s , a s ) T θ − y s (cid:1) 2 is the empirical mean-square error (MSE) of model( ϕ, θ ) and α t,δ ( ϕ ) :
Revised: [Equation] (1) where E t ( ϕ, θ ) : = 1 t (cid:80) ts =1 (cid:0) ϕ ( x s , a s ) T θ − y s (cid:1) 2 is the empirical mean-square error (MSE) of model( ϕ, θ ) and α t,δ ( ϕ ) :

Original: = 40 t log (cid:16) 8 | Φ | 2 (12 L ϕ B ϕ t ) dϕ t 3 δ (cid:17) + 2 t .
Revised: = 40 t log (cid:16) 8 | Φ | 2 (12 L ϕ B ϕ t ) dϕ t 3 δ (cid:17) + 2 t .

Original: This condition leverages the existence of arealizable representation in Φ t to eliminate representations whose MSE is not compatible with theone of the realizable representation, once accounted for the statistical uncertainty (i.e., α t,δ ( ϕ ) ).
Revised: This condition leverages the existence of arealizable representation in Φ t to eliminate representations whose MSE is not compatible with theone of the realizable representation, once accounted for the statistical uncertainty (i.e., α t,δ ( ϕ ) ).

Original: Subject to realizability constraint, the representation loss L t ( ϕ ) favours learning a HLS representation(if possible).
Revised: Subject to realizability constraint, the representation loss L t ( ϕ ) favours learning a HLS representation(if possible).

Original: As illustrated in Def. 2.1, a HLS representation is such that the expected design matrixassociated to the optimal actions has a positive minimum eigenvalue.
Revised: As illustrated in Def. 2.1, a HLS representation is such that the expected design matrixassociated to the optimal actions has a positive minimum eigenvalue.

Original: Unfortunately it is not possibleto directly optimize for this condition, since we have access to neither the context distribution ρ northe optimal action in each context.
Revised: Unfortunately it is not possibleto directly optimize for this condition, since we have access to neither the context distribution ρ northe optimal action in each context.

Original: Nonetheless, we can design a loss that works as a proxy for the
Revised: Nonetheless, we can design a loss that works as a proxy for the

Original: HLS property whenever A is a no-regret algorithm.
Revised: HLS property whenever A is a no-regret algorithm.

Original: Let V t ( ϕ ) =
Revised: Let V t ( ϕ ) =

Original: λI d ϕ + (cid:80) ts =1 ϕ ( x s , a s ) ϕ
Revised: λI d ϕ + (cid:80) ts =1 ϕ ( x s , a s ) ϕ

Original: ( x s , a s )
Revised: ( x s , a s )

Original: Tbe the empirical design matrix built on the context-actions pairs observed up to time t , then we define
Revised: Tbe the empirical design matrix built on the context-actions pairs observed up to time t , then we define

Original: L eig ,t ( ϕ ) :
Revised: L eig ,t ( ϕ ) :

Original: = − λ min (cid:0) V t ( ϕ ) − λI d ϕ (cid:1) /L
Revised: = − λ min (cid:0) V t ( ϕ ) − λI d ϕ (cid:1) /L

Original: 2 ϕ , where the normalization factor ensures invariance w.r.t.the feature norm.
Revised: 2 ϕ , where the normalization factor ensures invariance w.r.t.the feature norm.

Original: Intuitively, the empirical distribution of contexts ( x t ) t ≥ 1 converges to ρ and thefrequency of optimal actions selected by a no-regret algorithm increases over time, thus ensuringthat V t ( ϕ ) /t tends to behave as the design matrix under optimal arms E x ∼ ρ [ ϕ ( x, a ⋆x ) ϕ
Revised: Intuitively, the empirical distribution of contexts ( x t ) t ≥ 1 converges to ρ and thefrequency of optimal actions selected by a no-regret algorithm increases over time, thus ensuringthat V t ( ϕ ) /t tends to behave as the design matrix under optimal arms E x ∼ ρ [ ϕ ( x, a ⋆x ) ϕ

Original: ( x, a ⋆x ) T ] .
Revised: ( x, a ⋆x ) T ] .

Original: Asdiscussed in Sect.
Revised: Asdiscussed in Sect.

Original: 5 alternative losses can be used to favour learning HLS representations.
Revised: 5 alternative losses can be used to favour learning HLS representations.

Original: Mechanism ❷ (line 5).
Revised: Mechanism ❷ (line 5).

Original: While Papini et al.
Revised: While Papini et al.

Original: [11] proved that L IN UCB is able to exploit HLSrepresentations, other algorithms such as ϵ -greedy may keep forcing exploration and not fully takeadvantage of HLS properties, thus failing to achieve constant regret.
Revised: [10] proved that L IN UCB is able to exploit HLSrepresentations, other algorithms such as ϵ -greedy may keep forcing exploration and not fully takeadvantage of HLS properties, thus failing to achieve constant regret.

Original: In order to prevent this, weintroduce a generalized likelihood ratio test (GLRT).
Revised: In order to prevent this, weintroduce a generalized likelihood ratio test (GLRT).

Original: At each round t , let ϕ t − 1 be the representationused at time t , then B ANDIT SRL decides whether to act according to the base algorithm A withrepresentation ϕ t − 1 or fully exploit the learned representation and play greedily w.r.t.
Revised: At each round t , let ϕ t − 1 be the representationused at time t , then B ANDIT SRL decides whether to act according to the base algorithm A withrepresentation ϕ t − 1 or fully exploit the learned representation and play greedily w.r.t.

Original: it.
Revised: it.

Original: Denoteby θ ϕ,t − 1 =
Revised: Denoteby θ ϕ,t − 1 =

Original: V t − 1 ( ϕ ) − 1 (cid:80) t − 1 s =1 ϕ
Revised: V t − 1 ( ϕ ) − 1 (cid:80) t − 1 s =1 ϕ

Original: ( x s , a s )
Revised: ( x s , a s )

Original: y s the regularized least-squares parameter at time t forrepresentation ϕ and by π ⋆t − 1 ( x ; ϕ ) = argmax a ∈A (cid:8) ϕ
Revised: y s the regularized least-squares parameter at time t forrepresentation ϕ and by π ⋆t − 1 ( x ; ϕ ) = argmax a ∈A (cid:8) ϕ

Original: ( x, a ) T θ ϕ,t − 1 (cid:9) the associated greedy policy.
Revised: ( x, a ) T θ ϕ,t − 1 (cid:9) the associated greedy policy.

Original: Then, B ANDIT SRL selects the greedy action π ⋆t − 1 ( x t ; ϕ t − 1 ) when the GLR test is active, otherwiseit selects the action proposed by the base algorithm A .
Revised: Then, B ANDIT SRL selects the greedy action π ⋆t − 1 ( x t ; ϕ t − 1 ) when the GLR test is active, otherwiseit selects the action proposed by the base algorithm A .

Original: Formally, for any ϕ ∈ Φ and x ∈ X , we define 163 the generalized likelihood ratio as
Revised: Formally, for any ϕ ∈ Φ and x ∈ X , we define 161 the generalized likelihood ratio as

Original: [Equation]
Revised: [Equation]

Original: and, given β t − 1 ,δ ( ϕ ) = σ (cid:113) 2 log(1 /δ )
Revised: and, given β t − 1 ,δ ( ϕ ) = σ (cid:113) 2 log(1 /δ )

Original: + d ϕ log(1 + ( t − 1)
Revised: + d ϕ log(1 + ( t − 1)

Original: L 2 ϕ / ( λd ϕ ))
Revised: L 2 ϕ / ( λd ϕ ))

Original: + √ λB ϕ , the GLR test is
Revised: + √ λB ϕ , the GLR test is

Original: GLR t − 1 ( x ; ϕ ) > β t − 1 ,δ/ | Φ | ( ϕ ) [16, 30, 31].
Revised: GLR t − 1 ( x ; ϕ ) > β t − 1 ,δ/ | Φ | ( ϕ ) [28–30].

Original: If this happens at time t and ϕ t − 1 is realizable, then wehave enough confidence to conclude that the greedy action is optimal, i.e., π ⋆t − 1 ( x t ; ϕ t − 1 )
Revised: If this happens at time t and ϕ t − 1 is realizable, then we have enough confidence to conclude that the greedy action is optimal, i.e., π ⋆t − 1 ( x t ; ϕ t − 1 )

Original: = a ⋆x t .
Revised: = a ⋆x t .

Original: An important aspect of this test is that it is run on the current context x t and it does not requireevaluating global properties of the representation.
Revised: An important aspect of this test is that it is run on the current context x t and it does not requireevaluating global properties of the representation.

Original: While at any time t it is possible that a non-HLSnon-realizable representation may pass the test, the GLRT is sound as 1) exploration through A andthe representation learning mechanism work in synergy to guarantee that eventually a realizable representation is always provided to the GLRT; 2) only HLS representations are guaranteed toconsistently trigger the test at any context x .
Revised: While at any time t it is possible that a non-HLSnon-realizable representation may pass the test, the GLRT is sound as 1) exploration through A andthe representation learning mechanism work in synergy to guarantee that eventually a realizablerepresentation is always provided to the GLRT; 2) only HLS representations are guaranteed toconsistently trigger the test at any context x .

Original: In practice, B ANDIT SRL does not update the representation at each step but in phases.
Revised: In practice, B ANDIT SRL does not update the representation at each step but in phases.

Original: This isnecessary to avoid too frequent representation changes and control the regret, but also to make thealgorithm more computationally efficient and practical.
Revised: This isnecessary to avoid too frequent representation changes and control the regret, but also to make thealgorithm more computationally efficient and practical.

Original: Indeed, updating the representation may becomputationally expensive in practice (e.g., retraining a NN) and a phased scheme with γ parameterreduces the number of representation learning steps to J ≈ ⌈ log γ ( T ) ⌉ .
Revised: Indeed, updating the representation may becomputationally expensive in practice (e.g., retraining a NN) and a phased scheme with γ parameterreduces the number of representation learning steps to J ≈ ⌈ log γ ( T ) ⌉ .

Original: The algorithm A is reset at the 178 beginning of a phase j when the representation is selected and it is run on the samples collected duringthe current phase when the base algorithm is selected.
Revised: The algorithm A is reset at the 176 beginning of a phase j when the representation is selected and it is run on the samples collected duringthe current phase when the base algorithm is selected.

Original: If A is able to leverage off-policy data, at thebeginning of a phase j , we can warm-start it by providing ϕ j and all the past data ( x s , a s , y s ) s ≤ t j .
Revised: If A is able to leverage off-policy data, at thebeginning of a phase j , we can warm-start it by providing ϕ j and all the past data ( x s , a s , y s ) s ≤ t j .

Original: While the reset is necessary for dealing with any no-regret algorithm, it can be removed for algorithmssuch as L IN UCB and ϵ -greedy without affecting the theoretical guarantees.
Revised: While the reset is necessary for dealing with any no-regret algorithm, it can be removed for algorithmssuch as L IN UCB and ϵ -greedy without affecting the theoretical guarantees.

Original: Comparison to L EADER .
Revised: Comparison to L EADER .

Original: We first recall the basic structure of L EADER .
Revised: We first recall the basic structure of L EADER .

Original: Denote by UCB t ( x, a, ϕ )the upper-confidence bound computed by L IN UCB for the context-action pair ( x, a ) and representation ϕ after t steps.
Revised: Denote by UCB t ( x, a, ϕ )the upper-confidence bound computed by L IN UCB for the context-action pair ( x, a ) and representation ϕ after t steps.

Original: Then L EADER selects the action a t ∈ argmax a ∈A min
Revised: Then L EADER selects the action a t ∈ argmax a ∈A min

Original: ϕ ∈
Revised: ϕ ∈

Original: Φ t UCB t ( x t , a, ϕ ) .
Revised: Φ t UCB t ( x t , a, ϕ ) .

Original: Unlike the constrained optimization problem in B ANDIT SRL, this mechanism couples representationlearning and exploration-exploitation
Revised: Unlike the constrained optimization problem in B ANDIT SRL, this mechanism couples representationlearning and exploration-exploitation

Original: and it requires optimizing a representation for the currentx t and for each action a .
Revised: and it requires optimizing a representation for the currentx t and for each action a .

Original: Indeed, L EADER does not output a single representation and possiblychooses different representations for each context-action pair.
Revised: Indeed, L EADER does not output a single representation and possiblychooses different representations for each context-action pair.

Original: While this enables L EADER to mixrepresentations and achieve constant regret in some cases even when Φ does not include any HLSrepresentation, it leads to two major drawbacks: 1) the representation selection is directly entangledwith the L IN UCB exploration-exploitation strategy, 2) it is impractical in problems where Φ is aninfinite functional space (e.g., a deep neural network).
Revised: While this enables L EADER to mixrepresentations and achieve constant regret in some cases even when Φ does not include any HLSrepresentation, it leads to two major drawbacks: 1) the representation selection is directly entangledwith the L IN UCB exploration-exploitation strategy, 2) it is impractical in problems where Φ is aninfinite functional space (e.g., a deep neural network).

Original: The mechanisms ❶ and ❷ successfully addressthese limitations and enable B ANDIT SRL to be paired with any no-regret algorithm and to be scaledto any representation class as illustrated in the next section.
Revised: The mechanisms ❶ and ❷ successfully addressthese limitations and enable B ANDIT SRL to be paired with any no-regret algorithm and to be scaledto any representation class as illustrated in the next section.

Original: 3.1 Extension to Neural Networks
Revised: 3.1 Extension to Neural Networks

Original: We now consider a representation space Φ defined by the last layer of a NN.
Revised: We now consider a representation space Φ defined by the last layer of a NN.

Original: We denote by ϕ :
Revised: We denote by ϕ :

Original: X ×
Revised: X ×

Original: A → R d the last layer and by f ( x, a )
Revised: A → R d the last layer and by f ( x, a )

Original: =
Revised: =

Original: ϕ
Revised: ϕ

Original: ( x, a ) T θ the full NN, where θ are the last-layerweights.
Revised: ( x, a ) T θ the full NN, where θ are the last-layerweights.

Original: We show how B ANDIT SRL can be easily adapted to work with deep neural networks (NN).
Revised: We show how B ANDIT SRL can be easily adapted to work with deep neural networks (NN).

Original: First , the GLRT requires only to have access to the current context x t and representation ϕ j , i.e.,the features defined by the last layer of the current network, and its cost is linear in the number ofactions.
Revised: First , the GLRT requires only to have access to the current context x t and representation ϕ j , i.e.,the features defined by the last layer of the current network, and its cost is linear in the number ofactions.

Original: Second , the phased scheme allows lazy updates, where we retrain the network only log γ ( T )times.
Revised: Second , the phased scheme allows lazy updates, where we retrain the network only log γ ( T )times.

Original: Third , we can run any bandit algorithm with a representation provided by the NN, including
Revised: Third , we can run any bandit algorithm with a representation provided by the NN, including

Original: L IN UCB, LinTS, and ϵ -greedy.
Revised: L IN UCB, LinTS, and ϵ -greedy.

Original: Fourth , the representation learning step can be adapted to allowefficient optimization of a NN.
Revised: Fourth , the representation learning step can be adapted to allowefficient optimization of a NN.

Original: We consider a regularized problem obtained through an approximationof the constrained problem:
Revised: By Lagrangian relaxation, we move from constrained optimization toargmin ϕ

Original: [Equation]
Revised: [Equation]

Original: 3) where c reg ≥ 0 is a tunable parameter.
Revised: where c reg ≥ 0 is a tunable parameter.

Original: The fact we consider c reg constant allows us to ignore termsthat do not depend on either ϕ or θ .

Original: This leads to a convenient regularized loss that aims to minimizethe MSE (second term) while enforcing some spectral property on the last layer of the NN (first term).
Revised: This leads to a convenient regularized loss that aims tominimize the MSE (second term) while enforcing some spectral property on the last layer of the NN(first term).

Original: In practice, we can optimize this loss by stochastic gradient descent over a replay buffer containingthe samples observed over time.
Revised: In practice, we can optimize this loss by stochastic gradient descent over a replay buffercontaining the samples observed over time.

Original: The resulting algorithm, called NN-B ANDIT SRL, is a direct andelegant generalization of the theoretically-grounded algorithm.
Revised: The resulting algorithm, called NN-B ANDIT SRL, is adirect and elegant generalization of the theoretically-grounded algorithm.

Original: While in theory we can optimize the regularized loss (3) with all the samples, in practice it isimportant to better control the sample distribution.
Revised: While in theory we can optimize the regularized loss (2) with all the samples, in practice it isimportant to better control the sample distribution.

Original: As the algorithm progresses, we expect the replaybuffer to contain an increasing number of samples obtained by optimal actions, which may lead therepresentation to solely fit optimal actions while increasing misspecification on suboptimal actions.
Revised: As the algorithm progresses, we expect the replaybuffer to contain an increasing number of samples obtained by optimal actions, which may lead therepresentation to solely fit optimal actions while increasing misspecification on suboptimal actions.

Original: This may compromise the behavior of the algorithm and ultimately lead to high regret.
Revised: This may compromise the behavior of the algorithm and ultimately lead to high regret.

Original: This is aninstance of catastrophic forgetting induced by a biased/shifting sample distribution [e.g., 32].
Revised: This is an instance of catastrophic forgetting induced by a biased/shifting sample distribution [e.g., 31].

Original: Toprevent this phenomenon, we store two replay buffers: i) an explorative buffer D A ,t with samplesobtained when A was selected; ii) an exploitative buffer D glrt ,t with samples obtained when GLRT triggered and greedy actions were selected.
Revised: Toprevent this phenomenon, we store two replay buffers: i) an explorative buffer D A ,t with samplesobtained when A was selected; ii) an exploitative buffer D glrt ,t with samples obtained when GLRTtriggered and greedy actions were selected.

Original: The explorative buffer D A ,t is used to compute the MSE
Revised: The explorative buffer D A ,t is used to compute the MSE

Original: E t ( ϕ, θ ) .
Revised: E t ( ϕ, θ ) .

Original: While this reduces the number of samples, it improves the robustness of the algorithmby promoting realizability.
Revised: While this reduces the number of samples, it improves the robustness of the algorithmby promoting realizability.

Original: On the other hand, we use all the samples D t = D A ,t ∪ D glrt ,t for the 224 representation loss
Revised: On the other hand, we use all the samples D t = D A ,t ∪ D glrt ,t for the 220 representation loss

Original: L ( ϕ ) .
Revised: L ( ϕ ) .

Original: This is coherent with the intuition that mechanism ❶ works when thedesign matrix V t drifts towards the design matrix of optimal actions, which is at the core of the HLSproperty.
Revised: This is coherent with the intuition that mechanism ❶ works when thedesign matrix V t drifts towards the design matrix of optimal actions, which is at the core of the HLSproperty.

Original: Refer to App.
Revised: Refer to App.

Original: C for a more detailed description of NN-B ANDIT SRL.
Revised: C for a more detailed description of NN-B ANDIT SRL.

Original:  Theoretical Guarantees
Revised:  Theoretical Guarantees

Original: In this section, we provide a complete characterization of the theoretical guarantees of B ANDIT SRLwhen Φ is a finite set of representations, i.e., | Φ | < ∞ .
Revised: In this section, we provide a complete characterization of the theoretical guarantees of B ANDIT SRLwhen Φ is a finite set of representations, i.e., | Φ | < ∞ .

Original: We consider the update scheme with γ = 2 .
Revised: We consider the update scheme with γ = 2 .

Original: 4.1 Constant Regret Bound for HLS Representations
Revised: 4.1 Constant Regret Bound for HLS Representations

Original: We first study the case where a realizable HLS representation is available.
Revised: We first study the case where a realizable HLS representation is available.

Original: For the characterization ofthe behavior of the algorithm, we need to introduce the following times:
Revised: For the characterization ofthe behavior of the algorithm, we need to introduce the following times:

Original: • τ elim : an upper-bound to the time at which all non-realizable representations are eliminated,
Revised: • τ elim : an upper-bound to the time at which all non-realizable representations are eliminated,

Original: i
Revised: i

Original: e., for all t ≥ τ elim , Φ t = Φ ⋆ ;• τ HLS : an upper-bound to the time (if it exists) after which the HLS representation is selected,i
Revised: e.,for all t ≥ τ elim , Φ t = Φ ⋆ ; • τ HLS : an upper-bound to the time (if it exists) after which the HLS representation is selected,i.e., ϕ t = ϕ

Original: e., ϕ t = ϕ

Original: ⋆ for all t ≥ τ HLS , where ϕ ⋆ ∈ Φ ⋆ is the unique HLS realizable representation;• τ glrt : an upper-bound to the time (if it exists) such that the GLR test triggers for the HLSrepresentation ϕ
Revised: ⋆ for all t ≥ τ HLS , where ϕ ⋆ ∈ Φ ⋆ is the unique HLS realizable representation;• τ glrt : an upper-bound to the time (if it exists) such that the GLR test triggers for the HLSrepresentation ϕ

Original: ⋆ for all t ≥ τ glrt .
Revised: ⋆ for all t ≥ τ glrt .

Original: We begin by deriving a constant problem-dependent regret bound for B ANDIT SRL with HLSrepresentations.
Revised: We begin by deriving a constant problem-dependent regret bound for B ANDIT SRL with HLSrepresentations.

Original: The proof and explicit values of the constants are reported in App.
Revised: The proof and explicit values of the constants are reported in App.

Original: B. 

Revised: B.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let A be any no-regret algorithm for stochastic contextual linear bandits, Φ satisfy
Revised: Let A be any no-regret algorithm for stochastic contextual linear bandits, Φ satisfy

Original: Asm.
Revised: Asm.

Original: 1- 3, | Φ | < ∞ , γ = 2 , and L t ( ϕ ) =
Revised: 1- 3, | Φ | < ∞ , γ = 2 , and L t ( ϕ ) =

Original: L eig ,t ( ϕ ) :
Revised: L eig ,t ( ϕ ) :

Original: = − λ min ( V t ( ϕ ) − λI d ϕ ) /L
Revised: = − λ min ( V t ( ϕ ) − λI d ϕ ) /L

Original: 2 ϕ .
Revised: 2 ϕ .

Original: Moreover, letΦ ⋆ contains a unique HLS representation ϕ ⋆ .
Revised: Moreover, letΦ ⋆ contains a unique HLS representation ϕ ⋆ .

Original: Then, for any δ ∈ (0 , 1) and T ∈ N , the regret of
Revised: Then, for any δ ∈ (0 , 1) and T ∈ N , the regret of

Original: B ANDIT SRL is bounded, with probability at least 1 − 4 δ , as 

Revised: B ANDIT SRL is bounded, with probability at least 1 − 3 δ , as

Original: [Equation]
Revised: [Equation]

Original: whereδ j := δ/ (2( j + 1) 2 ) and
Revised: where

Original: [Equation]
Revised: [Equation]

Original: with τ alg a finite (independent from the horizon T ) constant depending on algorithm A (see Tab.
Revised: with τ alg a finite (independent from the horizon T ) constant depending on algorithm A (see Tab.

Original: 1)and R A ( τ, ϕ, δ ) an anytime bound (non-decreasing in τ and 1 /δ )
Revised: 1)and R A ( τ, ϕ, δ ) an anytime bound (non-decreasing in τ and 1 /δ )

Original: on the regret accumulated over τsteps by A using representation ϕ and confidence level δ .
Revised: on the regret accumulated over τsteps by A using representation ϕ and confidence level δ .

Original: The key finding of the previous result is that B ANDIT SRL achieves constant regret whenever arealizable HLS representation is available in the set Φ , which may contain non-realizable as wellas realizable non-HLS representations.
Revised: The key finding of the previous result is that B ANDIT SRL achieves constant regret whenever arealizable HLS representation is available in the set Φ , which may contain non-realizable as wellas realizable non-HLS representations.

Original: The regret bound above also illustrates the “dynamics”of the algorithm and three main regimes.
Revised: The regret bound above also illustrates the “dynamics”of the algorithm and three main regimes.

Original: In the early stages, non-realizable representations maybe included in Φ t , which may lead to suffering linear regret until time τ elim when the constraintin the representation learning step filters out all non-realizable representations (first term in theregret bound).
Revised: In the early stages, non-realizable representations maybe included in Φ t , which may lead to suffering linear regret until time τ elim when the constraintin the representation learning step filters out all non-realizable representations (first term in theregret bound).

Original: At this point, B ANDIT SRL leverages the loss L to favor HLS representationsand the base
Revised: At this point, B ANDIT SRL leverages the loss L to favor HLS representationsand the base

Original: algorithm A to perform effective exploration-exploitation.
Revised: algorithm A to perform effective exploration-exploitation.

Original: This leads to the secondterm in the bound, which corresponds to an upper-bound to the sum of the regrets of A in eachphase in between τ elim and τ glrt ∨ τ HLS , which is roughly (cid:80) j τ elim <
Revised: This leads to the secondterm in the bound, which corresponds to an upper-bound to the sum of the regrets of A in each Algorithm R A ( T, ϕ, δ/ | Φ | ) τ alg L IN UCB d

Revised: [Table]

Revised: ϵ -greedy with ϵ t =

Revised: t − 1 /(cid:112)

Revised: d ϕ |A| log( |

Revised: [Table]

Revised: | /

Revised: [Table]

Revised: ) T

Revised: [Table]

Revised: [Table]

Revised: phase in between τ elim and τ glrt ∨ τ HLS , which is roughly (cid:80) j τ elim <

Original: j<j τ opt R A ( t j +1 − t j , ϕ j ) ≤max ϕ ∈
Revised: j<j [Equation] opt R A ( t j +1 − t j , ϕ j ) ≤max ϕ ∈

Original: Φ ⋆ R A ( τ opt − τ elim , ϕ ) log 2 ( τ opt ) .
Revised: Φ ⋆ R A ( τ opt − τ elim , ϕ ) log 2 ( τ opt ) .

Original: In this second regime, in some phases the algorithm 260 may still select non-HLS representations, which leads to a worst-case bound over all realizablerepresentations in Φ ⋆ .
Revised: In this second regime, in some phases the algorithm 256 may still select non-HLS representations, which leads to a worst-case bound over all realizablerepresentations in Φ ⋆ .

Original: Finally, after τ glrt ∨ τ HLS the GLRT consistently triggers over time.During Algorithm
Revised: Finally, after τ glrt ∨ τ HLS the GLRT consistently triggers over time.

Original: R A ( T, ϕ, δ/ | Φ | ) τ alg

Original: L IN UCB [Table] this last regime, B ANDIT SRL has reached enough accuracy and confidence so that the greedy policyof the HLS representation is indeed optimal and no additional regret is incurred.
Revised: Duringthis last regime, B ANDIT SRL has reached enough accuracy and confidence so that the greedy policyof the HLS representation is indeed optimal and no additional regret is incurred.

Original: We notice that the only dependency on the number of representations | Φ | in Thm.
Revised: We notice that the only dependency on the number of representations | Φ | in Thm.

Original: 4.1 is due tothe rescaling of the confidence level δ
Revised: 4.1 is due tothe rescaling of the confidence level δ

Original: (cid:55)→ δ/ | Φ | .
Revised: (cid:55)→ δ/ | Φ | .

Original: Since standard algorithms have a logarithmic 266 dependence in 1 /δ , this only leads to a logarithmic dependency in | Φ | .
Revised: Since standard algorithms have a logarithmic 262 dependence in 1 /δ , this only leads to a logarithmic dependency in | Φ | .

Original: On the other hand, due to the 267 resets, B ANDIT SRL has an extra logarithmic factor in the effective regret horizon τ opt .
Revised: On the other hand, due to the 263 resets, B ANDIT SRL has an extra logarithmic factor in the effective regret horizon τ opt .

Original: Single HLS representation.
Revised: Single HLS representation.

Original: A noteworthy consequence of Thm.
Revised: A noteworthy consequence of Thm.

Original: 4.1 is that any no-regret algorithmequipped with GLRT achieves constant regret when provided with a realizable HLS representation.
Revised: 4.1 is that any no-regret algorithmequipped with GLRT achieves constant regret when provided with a realizable HLS representation.

Original: Corollary 4.2.
Revised: Corollary 4.2.

Original: Let Φ
Revised: Let Φ

Original: =
Revised: =

Original: Φ ⋆ = { ϕ ⋆ } and ϕ ⋆ is HLS .
Revised: Φ ⋆ = { ϕ ⋆ } and ϕ ⋆ is HLS .

Original: Then, τ elim = τ
Revised: Then, τ elim = τ

Original: HLS = 0 and, with probabilityat least 1 − 4 δ , B ANDIT SRL suffers constant regret: R T ≤ R A ( τ glrt ∧ T, ϕ ⋆ , δ ) .
Revised: HLS = 0 and, with probabilityat least 1 − 3 δ , B ANDIT SRL suffers constant regret: R T ≤ R A ( τ glrt ∧ T, ϕ ⋆ , δ ) .

Original: This corollary also illustrates that the performance of A is not affected when ϕ ⋆ is non-HLS (i.e.,τ glrt = ∞ ), as B ANDIT SRL achieves the same regret of the base algorithm.
Revised: This corollary also illustrates that the performance of A is not affected when ϕ ⋆ is non-HLS (i.e.,τ glrt = ∞ ), as B ANDIT SRL achieves the same regret of the base algorithm.

Original: Note that there is noadditional logarithmic factor in this case since we do not need any reset for representation learning.
Revised: Note that there is noadditional logarithmic factor in this case since we do not need any reset for representation learning.

Original: 4.2 Additional Results
Revised: 4.2 Additional Results

Original: No HLS representation.
Revised: No HLS representation.

Original: A consequence of Thm.
Revised: A consequence of Thm.

Original: 4.1 is that when | Φ | > 1 but no realizable HLSexists ( τ glrt = ∞ ), B ANDIT SRL still enjoys a sublinear regret.
Revised: 4.1 is that when | Φ | > 1 but no realizable HLSexists ( τ glrt = ∞ ), B ANDIT SRL still enjoys a sublinear regret.

Original: Corollary 4.3 (Regret bound without HLS representation) .
Revised: Corollary 4.3 (Regret bound without HLS representation) .

Original: Consider the same setting in Thm.
Revised: Consider the same setting in Thm.

Original: 4.1and assume that Φ ⋆ does not contain any HLS representation.
Revised: 4.1and assume that Φ ⋆ does not contain any HLS representation.

Original: Then, for any δ ∈ (0 , 1) and T ∈ N ,the regret of B ANDIT SRL is bounded, with probability at least 1 − 4 δ , as follows:
Revised: Then, for any δ ∈ (0 , 1) and T ∈ N ,the regret of B ANDIT SRL is bounded, with probability at least 1 − 3 δ , as follows:

Original: [Equation]
Revised: [Equation]

Original: This shows that the regret of B ANDIT SRL is of the same order of the base no-regret algorithm A whenrunning with the worst realizable representation.
Revised: This shows that the regret of B ANDIT SRL is of the same order of the base no-regret algorithm A whenrunning with the worst realizable representation.

Original: While such worst-case dependency is undesirable, itis common to many representation learning algorithms, both in bandits and reinforcement learning [e.g.4, 33].
Revised: While such worst-case dependency is undesirable, itis common to many representation learning algorithms, both in bandits and reinforcement learning [e.g.4, 32].

Original: In App.
Revised: In App.

Original: C, we show that an alternative representation loss could address this problemand lead to a bound scaling with the regret of the best realizable representation ( R T ≤ 2 τ elim +
Revised: C, we show that an alternative representation loss could address this problemand lead to a bound scaling with the regret of the best realizable representation ( R T ≤ 2 τ elim +

Original: min ϕ ∈
Revised: min ϕ ∈

Original: Φ ⋆ R A ( T, ϕ, δ/ | Φ | ) log 2 ( T ) ), while preserving the guarantees for the HLS case.
Revised: Φ ⋆ R A ( T, ϕ, δ/ | Φ | ) log 2 ( T ) ), while preserving the guarantees for the HLS case.

Original: Since the 287 representation loss requires an upper-bound on the number of suboptimal actions and a carefullytuned schedule for guessing the gap ∆ , it is less practical than the smallest eigenvalue, which we useas the basis for our practical version of B ANDIT SRL.
Revised: Since the 283 representation loss requires an upper-bound on the number of suboptimal actions and a carefullytuned schedule for guessing the gap ∆ , it is less practical than the smallest eigenvalue, which we useas the basis for our practical version of B ANDIT SRL.

Original: 290 Algorithm-dependent instances and comparison to L EADER .
Revised: Algorithm-dependent instances and comparison to L EADER .

Original: Table 1 reports the regret bound of
Revised: Table 1 reports the regret bound of

Original: B ANDIT SRL for different base algorithms.
Revised: B ANDIT SRL for different base algorithms.

Original: These results make explicit the dependence in the numberof representations | Φ | and show that the cost of representation learning is only logarithmic.
Revised: These results make explicit the dependence in the numberof representations | Φ | and show that the cost of representation learning is only logarithmic.

Original: In thespecific case of L IN UCB for HLS representations, we highlight that the upper-bound to the time τ optin Thm.
Revised: In thespecific case of L IN UCB for HLS representations, we highlight that the upper-bound to the time τ optin Thm.

Original: 4.1 improves over the result of L EADER .
Revised: 4.1 improves over the result of L EADER .

Original: While L EADER has no explicit concept of τ alg ,a term with the same dependence of τ alg in Tab. 1 appears also in the L EADER analysis.
Revised: While L EADER has no explicit concept of τ alg , a term with the same dependence of τ alg in Tab. 1 appears also in the L EADER analysis.

Original: This termencodes an upper bound to the pulls of suboptimal actions and depends on the L IN UCB strategy.
Revised: This termencodes an upper bound to the pulls of suboptimal actions and depends on the L IN UCB strategy.

Original: Asa result, the first three terms in Eq.
Revised: Asa result, the first three terms in Eq.

Original: 4 are equivalent to the ones of L EADER .
Revised: 3 are equivalent to the ones of L EADER .

Original: The improvement comesfrom the last term ( τ elim ), where, thanks to a refined analysis of the elimination condition, we are ableto improve the dependence on the inverse minimum misspecification ( 1 / min ϕ/ ∈
Revised: The improvement comesfrom the last term ( τ elim ), where, thanks to a refined analysis of the elimination condition, we are ableto improve the dependence on the inverse minimum misspecification ( 1 / min ϕ/ ∈

Original: Φ ⋆ ϵ ϕ ) from quadratic to linear (see App.
Revised: Φ ⋆ ϵ ϕ ) from quadraticto linear (see App.

Original: B for a detailed comparison).
Revised: B for a detailed comparison).

Original: On the other hand, B ANDIT SRL suffers from theworst regret among realizable representations, whereas L EADER scales with the best representation.
Revised: On the other hand, B ANDIT SRL suffers from theworst regret among realizable representations, whereas L EADER scales with the best representation.

Original: As discussed above, this mismatch can be mitigated by using by a different choice of representationloss.
Revised: As discussed above, this mismatch can be mitigated by using by a different choice of representationloss.

Original: In the case of ϵ -greedy, the T 2 / 3 regret upper-bound induces a worse τ alg due to a largernumber of suboptimal pulls.
Revised: In the case of ϵ -greedy, the T 2 / 3 regret upper-bound induces a worse τ alg due to a largernumber of suboptimal pulls.

Original: This in turns reflects into a higher regret to the constant regime.
Revised: This in turns reflects into a higher regret to the constant regime.

Original: Finally,
Revised: Finally,

Original: L EADER is still guaranteed to achieve constant regret by selecting different representations at differentcontext-action pairs whenever non-HLS representations satisfy a certain mixing condition [cf. 11,
Revised: L EADER is still guaranteed to achieve constant regret by selecting different representations at differentcontext-action pairs whenever non-HLS representations satisfy a certain mixing condition [cf. 10,

Original: Sec. 5.2].
Revised: Sec. 5.2].

Original: This result is not possible with B ANDIT SRL, where one representation is selected in eachphase.
Revised: This result is not possible with B ANDIT SRL, where one representation is selected in eachphase.

Original: At the same time, it is the single-representation structure of B ANDIT SRL that allows us toaccommodate different base algorithms and scale it to any representation space.
Revised: At the same time, it is the single-representation structure of B ANDIT SRL that allows us toaccommodate different base algorithms and scale it to any representation space.

Original:  Experiments
Revised:  Experiments

Original: We provide an empirical validation of B ANDIT SRL both in synthetic contextual linear banditproblems and on non-linear contextual problems [see e.g., 6, 27].
Revised: We provide an empirical validation of B ANDIT SRL both in synthetic contextual linear banditproblems and on non-linear contextual problems [see e.g., 6, 25].

Original: Linear Benchmarks.
Revised: Linear Benchmarks.

Original: We first evaluate B ANDIT SRL on synthetic linear problems to empiricallyvalidate our theoretical findings.
Revised: We first evaluate B ANDIT SRL on synthetic linear problems to empiricallyvalidate our theoretical findings.

Original: In particular, we test B ANDIT SRL with different base algorithms andrepresentation learning loss and we compare it with L EADER .
Revised: In particular, we test B ANDIT SRL with different base algorithms andrepresentation learning loss and we compare it with L EADER .

Original: 4 We consider the “varying dimension”problem introduced in [11] which consists of six realizable representations with dimension from 2to 6 .
Revised: 4 We consider the “varying dimension”problem introduced in [10] which consist of six realizable representations with dimension from 2to 6 .

Original: Of the two representations of dimension d = 6 , one is HLS.
Revised: Of the two representations of dimension d = 6 , one is HLS.

Original: In addition seven misspecifiedrepresentations are available.
Revised: In addition seven misspecifiedrepresentations are available.

Original: Details are provide in App.
Revised: Details are provide in App.

Original: D. We consider L IN UCB and ϵ -greedyas base algorithms and we use the theoretical parameters, but we perform warm start using all thepast data when a new representation is selected.
Revised: D. We consider L IN UCB and ϵ -greedyas base algorithms and we use the theoretical parameters, but we perform warm start using all thepast data when a new representation is selected.

Original: Similarly, for B ANDIT SRL we use the theoreticalparameters ( γ = 2 ) and L t ( ϕ ) :
Revised: Similarly, for B ANDIT SRL we use the theoreticalparameters ( γ = 2 ) and L t ( ϕ ) :

Original: = L eig ,t ( ϕ ) .
Revised: = L eig ,t ( ϕ ) .

Original: Fig.
Revised: Fig.

Original: 1 shows that, as expected, B ANDIT SRL with bothbase algorithms is able to achieve constant regret when a HLS representation exists.
Revised: 1 shows that, as expected, B ANDIT SRL with bothbase algorithms is able to achieve constant regret when a HLS representation exists.

Original: As expected fromthe theoretical analysis, ϵ -greedy leads to a higher regret than L IN UCB.
Revised: As expected fromthe theoretical analysis, ϵ -greedy leads to a higher regret than L IN UCB.

Original: Furthermore, empirically
Revised: Furthermore, empirically

Original: B ANDIT SRL with L IN UCB obtains a performance that is comparable with the one of L EADERboth with and without realizable HLS representation.
Revised: B ANDIT SRL with L IN UCB obtains a performance that is comparable with the one of L EADERboth with and without realizable HLS representation.

Original: Note that when no-HLS exists, the regret of
Revised: Note that when no-HLS exists, the regret of

Original: B ANDIT SRL with ϵ -greedy is T 2 / 3 , while L IN UCB-based algorithms are able to achieve log( T )regret.
Revised: B ANDIT SRL with ϵ -greedy is T 2 / 3 , while L IN UCB-based algorithms are able to achieve log( T )regret.

Original: When Φ contains misspecified representations (Fig.
Revised: When Φ contains misspecified representations (Fig.

Original: 1(center-left))
Revised: 1(center-left))

Original: , we can observe that inthe first regime [1 , τ elim ] the algorithm suffers linear regret, after that we have the  regime of the basealgorithm ( [ τ elim , τ glrt ∨ τ HLS ] ) up to the point where the GLRT leads to select only optimal actions.
Revised: , we can observe that in thefirst regime [1 , τ elim ] the algorithm suffers linear regret, after that we have the classical regime ofthe base algorithm ( [ τ elim , τ glrt ∨ τ HLS ] ) up to the point where the GLRT leads to select only optimalactions.

Original: Weak HLS.
Revised: Weak HLS.

Original: Papini et al.
Revised: Papini et al.

Original: [11] showed that when realizable representations are redundant (i.e., λ ⋆ ( ϕ ⋆ ) =0 ), it is still possible to achieve constant regret if the representation is “weakly”-HLS, i.e., the featuresof the optimal actions spans the features ϕ ( x, a ) associated to any context-action pair, but not necessarily R d ϕ .
Revised: [10] showed that when realizable representations are redundant (i.e., λ ⋆ ( ϕ ⋆ ) =0 ), it is still possible to achieve constant regret if the representation is “weakly”-HLS, i.e., the featuresof the optimal actions spans the features ϕ ( x, a ) associated to any context-action pair, but not necessarily R d ϕ .

Original: To test this case, we pad a 5-dimensional vector of ones to all the features of the six realizablerepresentations in the previous experiment.
Revised: To test this case, we pad a 5-dimensional vector of ones to all the features of the six realizablerepresentations in the previous experiment.

Original: To deal with the weak-HLS condition, we introduce thealternative representation loss L weak ,t( ϕ )
Revised: To deal with the weak-HLS condition, we introduce thealternative representation loss L weak ,t

Original: = − min s ≤ t (cid:8) ϕ ( x s , a s ) T ( V t ( ϕ ) − λI d ϕ ) ϕ ( x s , a s ) /L 2 ϕ (cid:9) .
Revised: [Equation]

Original: Since, V t ( ϕ ) − λI d ϕ tends to behave as E x
Revised: Since, V t ( ϕ ) − λI d ϕ tends to behave as E x

Original: [ ϕ ⋆ ( x ) ϕ ⋆ ( x ) T ] , this loss encourages representations whereall the observed features are spanned by the optimal arms, thus promoting weak-HLS representations(see App.
Revised: [ ϕ ⋆ ( x ) ϕ ⋆ ( x ) T ] , this loss encourages representations whereall the observed features are spanned by the optimal arms, thus promoting weak-HLS representations(see App.

Original: C for more details).
Revised: C for more details).

Original: As expected, Fig. 1(right) shows that the min-eigenvalue loss L eig ,tfails in identifying the correct representation in this domain.
Revised: As expected, Fig. 1(right) shows that the min-eigenvalue loss L eig ,tfails in identifying the correct representation in this domain.

Original: On the other hand, B ANDIT SRL withthe novel loss is able to achieve constant regret and converge to constant regret (we cut the figure forreadability), and behaves as L EADER when using L IN UCB.
Revised: On the other hand, B ANDIT SRL withthe novel loss is able to achieve constant regret and converge to constant regret (we cut the figure forreadability), and behaves as L EADER when using L IN UCB.

Original: Non-Linear Benchmarks.
Revised: Non-Linear Benchmarks.

Original: We study the performance of NN-B ANDIT SRL in classical benchmarkswhere non-linear representations are required.
Revised: We study the performance of NN-B ANDIT SRL in classical benchmarkswhere non-linear representations are required.

Original: We only consider the weak-HLS loss L weak ,t ( ϕ ) as it ismore general than full HLS.
Revised: We only consider the weak-HLS loss L weak ,t ( ϕ )as it is more general than full HLS.

Original: As base algorithms we consider ϵ -greedy with ϵ t = t − 1 / 3 and L IN UCB,L IN TS with theoretical parameters, where the algorithms are run on the representation ϕ
Revised: As base algorithms we consider ϵ -greedy with ϵ t = t − 1 / 3and L IN UCB, where the algorithm is run on the representation ϕ

Original: j providedby the NN at each phase j .
Revised: j provided by the NN at each [Figure]

Original:  We compare NN-B ANDIT SRL against the base algorithms using themaximum-likelihood representation (i.e., Neural-( ϵ -greedy, L IN TS) [6] and Neural-L IN UCB [28]),inverse gap weighting strategy [e.g., 7, 10] and NeuralUCB [27].
Revised: [Figure] Results are averaged over 10 seeds. phase j (similar to [26]). We compare NN-B ANDIT SRL against the base algorithms using themaximum-likelihood representation and GLRT (MLE- ϵ -greedy and MLE-L IN UCB).

Original: See App.
Revised: See App.

Original: C-D for details.
Revised: C-Dfor details.

Original: [Figure]

Original: [Figure] [Figure] Time Misspecified and one HLS 0 .
Revised: We first consider the wheel domain introduced in [6], which is particularly challenging as the rewardfunction is highly non-linear w.r.t.

Original: Time

Original: [-Realizable, no HLS     · 
-]
Revised: [-Realizable, no HLS     · 
-]

Original: Time

Original: Weak HLS

Original: [Figure] C u m u l a t i v e R
Revised: to contexts and actions.

Original: [Figure]

Original: g r

Original: [Figure]

Original: t

Original: Realizable and [Figure] In all the problems 5 the reward function is highly non-linear w.r.t. to contexts and actions and we usea network composed by layers of dimension [50 , 50 , 50 , 50 , 10 , 1] and ReLu activation to learn therepresentation (i.e., d = 10 ).
Revised: Fig. 2(left) shows that both ϵ -greedyand L IN UCB achieves better performance through representation learning, outperforming the basealgorithms.

Original: Fig. 2 shows that all the base algorithms ( ϵ - GREEDY , L IN UCB, L IN TS)achieves better performance through representation learning, outperforming the base algorithms. Thisprovide evidence that NN-B ANDIT SRL is effective even beyond the theoretical scenario. For the baseline algorithms (NeuralUCB, IGW) we report the regret of the best configuration on eachindividual dataset, while for NN-B ANDIT SRL we fix the parameters across datasets (i.e., α GLRT =5 ).
Revised: In all the datasets –statlog, magic, covertype [6, 25, 33]– obtained from the standardmulticlass-to-bandit conversion, not only NN-B ANDIT SRL is able to outperform the algorithmswithout explicit representation learning but it is also able to achieve constant regret.

Original: While this comparison clearly favours the baselines, it also shows that NN-B ANDIT SRL is arobust algorithm that behaves better or on par with the state-of-the-art algorithms. In particular, NNB ANDIT SRL uses theoretical parameters while the baselines uses tuned configurations. Optimizingthe parameters of NN-B ANDIT SRL  is outside the scope of these experiments.
Revised: Since the otheralgorithms (that are also equipped with the GLRT) are not able to achieve constant regret, these resultsshow that NN-B ANDIT SRL learns a representation that is (weak) HLS, and show the importance ofour regularized loss in practical application.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a novel algorithm, B ANDIT SRL, for representation selection in stochastic contextuallinear bandits.
Revised: We proposed a novel algorithm, B ANDIT SRL, for representation selection in stochastic contextuallinear bandits.

Original: B ANDIT SRL combines a mechanism for representation learning that aims to recoverrepresentations with good spectral properties, with a generalized likelihood ratio test to exploit therecovered representation.
Revised: B ANDIT SRL combines a mechanism for representation learning that aims to recoverrepresentations with good spectral properties, with a generalized likelihood ratio test to exploit therecovered representation.

Original: We proved that, thanks to these mechanisms, B ANDIT SRL is not only ableto achieve sublinear regret with any no-regret algorithm A but, when a HLS representation exists, itis able to achieve constant regret.
Revised: We proved that, thanks to these mechanisms, B ANDIT SRL is not only ableto achieve sublinear regret with any no-regret algorithm A but, when a HLS representation exists, itis able to achieve constant regret.

Original: We demonstrated that B ANDIT SRL can be implemented using
Revised: We demonstrated that B ANDIT SRL can be implemented using

Original: NNs and showed its effectiveness in standard benchmarks.
Revised: NNs and showed its effectiveness in standard benchmarks.

Original: A direction for future investigation is to extend the approach to a weaker misspecification assumptionthan Asm. 3.
Revised: A direction for future investigation is to extend the approach to a weaker misspecification assumptionthan Asm. 3.

Original: Another direction is to leverage the technical and algorithmic tools introduced in thispaper for representation learning in reinforcement learning, e.g., in low-rank problems [e.g. 38].
Revised: Another direction is to leverage the technical and algorithmic tools introduced in thispaper for representation learning in reinforcement learning, e.g., in low-rank problems [e.g. 34]

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results..
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [No]
Revised: [No]

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi- ments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No]
Revised: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N /A]
Revised: [N/A]