Efﬁcient Frameworks for Generalized Low-Rank Matrix Bandit Problems
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In the stochastic contextual low-rank matrix bandit problem, the expected rewardof an action is given by the inner product between the action’s feature matrix andsome ﬁxed, but initially unknown d 1 by d 2 matrix Θ ∗ with rank r (cid:28) { d 1 , d 2 } ,and an agent sequentially takes actions based on past experience to maximizethe cumulative reward.
In this paper, we study the generalized low-rank matrixbandit problem, which has been recently proposed in [26] under the Generalized
Linear Model (GLM) framework.
To overcome the computational infeasibility andtheoretical restrain of existing algorithms on this problem, we ﬁrst propose the
G-ESTT framework that modiﬁes the idea from [17] by using Stein’s method on thesubspace estimation and then leverage the estimated subspaces via a regularizationidea.
Furthermore, we remarkably improve the efﬁciency of G-ESTT by using anovel exclusion idea on the estimated subspace instead, and propose the G-ESTSframework.
We also show that both of our methods are the ﬁrst algorithm to achievethe optimal ˜ O
(( d 1 + d 2 ) r √ T ) bound of regret presented in [26] up to logarithmterms, which improves upon the current regret of ˜ O
(( d 1 + d 2 ) 3 / 2 √ rT ) [26].
And
G-ESTS could even attain a better ˜ O ( √ d 1 d 2 rT ) regret bound under the ﬁnite-armcase.
For completeness, we conduct experiments to illustrate that our proposedalgorithms, especially G-ESTS, are computationally tractable and consistentlyoutperform other state-of-the-art (generalized) linear matrix bandit methods.
Introduction
The contextual bandit has proven to be a powerful framework for sequential decision-making problems, with great applications to clinical trials [36], recommendation system [22] and personalizedmedicine [4].
This class of problems evaluates how an agent should choose an action from the potential action set at each round based on an updating policy on-the-ﬂy so as to maximize the cumulativereward or minimize the overall regret.
With high dimensional sparse data becoming ubiquitous invarious ﬁelds nowadays, the most fundamental (generalized) linear bandit framework, although hasbeen extensively studied, becomes inefﬁcient in practice.
This fact consequently leads to a line ofwork on stochastic high dimensional bandit problems with low dimensional structures [16, 24], suchas the LASSO bandit and low-rank matrix bandit.
In this work, we investigate on the generalized low-rank matrix bandit problem ﬁrstly studiedin [26]: at round t = 1 , . .
, T , the algorithm selects an action represented by a d 1 by d 2 matrix X tfrom the admissible action set X t ( X t may be ﬁxed), and receives its associated noisy reward y t =µ ( (cid:104) Θ ∗ , X t (cid:105) )
+ η t where Θ ∗ ∈ R
d 1 × d 2 is some unknown low-rank matrix with rank r (cid:28) { d 1 , d 2 } andµ ( · ) is the inverse link function.
More details about this setting are deferred to Section 3.
This problemhas vast applicability in real world applications.
On the one hand, matrix inputs are appropriate whendealing with paired contexts which are omnipresent in practice.
For instance, to design a personalized
[Table]
movie recommendation system, we can formulate each user as m d 1 -dimensional feature vectors( x 1 , . . .
, x m ∈ R d 1 ) and each movie as m d 2 -dimensional feature vectors ( y 1 , . .
y m ∈ R d 2 ).
Auser-item pair can then be naturally represented by a feature matrix deﬁned as the outer productssummation (cid:80) mk =1 x k y (cid:62)
k ∈ R d 1 × d 2 , which will become the contextual feature observed by the banditalgorithm.
Other applications involve interaction features between two groups, such as ﬂight-hotelbundles [26] and dating service [17] can also be similarly established.
Besides, low-rank modelshave gained tremendous success in various areas [5].
In particular, our problem can be regarded as anextension of the inductive matrix factorization problem [14, 38], which estimates low-rank matriceswith contextual information, under the online learning scenario.
Our study is inspired by a line of work on stochastic contextual low-rank matrix bandit [15, 17, 26].
To design an algorithm for matrix bandit problems, a naïve approach is to ﬂatten the d 1 by d 2 featurematrices into vectors and then apply any (generalized) linear bandit algorithms, which, however,would be inefﬁcient when d 1 d 2 is large.
To take advantage of the low-rank structure, [17] haveintroduced the bilinear low-rank bandit problem and proposed a two-stage algorithm named ESTRwhich could achieve a regret bound of ˜ O
(( d 1 + d 2 ) 3 / 2 √ rT ) 1 .
Subsequently, [15] constructed a newalgorithm called (cid:15) -FALB for bilinear bandits and achieved a better regret of ˜ O
( (cid:112) d 1 d 2 ( d 1 + d 2 ) T ) .
However, they only studied the linear reward framework and also restricted the feature matrix as arank-one matrix.
As a follow-up work, [26] further released the rank-one restriction on the actionfeature matrices, and they introduced an algorithm LowGLOC based on the online-to-conﬁdenceset conversion [2] for generalized low-rank matrix bandits with ˜ O
( (cid:112) ( d 1 + d 2 ) 3 rT ) regret bound.
However, this regret bound is still loose compared with the optimal rate ˜ O
(( d 1 + d 2 ) r √ T ) deducedin [15], and the arm set is still assumed ﬁxed at each round.
This algorithm is also computationallyprohibitive since it requires to calculate weights of a self-constructed covering of the admissibleparameter space at each iteration.
And how to ﬁnd this covering for low-rank matrices is also unclear.
In this work, we propose two efﬁcient methods called G-ESTT and G-ESTS for this problem by modifying two stages of ESTR appropriately from different perspectives.
To the best of our knowledge,the proposed methods are the ﬁrst two generalized (contextual) low-rank bandit algorithms that arecomputationally feasible, and achieves an improved regret bound of ˜ O
(( d 1 + d 2 ) r √ T ) upon all existing works on (bilinear) low-rank bandits.
The main contributions of this paper can be summarized as: 1) we propose two novel two-stage frameworks G-ESTT and G-ESTS under some mild assumptions.
Compared with ESTR in [17], (cid:15) -FALB in [15] and LowESTR in [ ], our algorithms are designedfor nonlinear reward framework.
Compared with LowGLOC in [26], our algorithms achieve a betterregret bound, can work with varying action sets and are computationally feasible.
For G-ESTT,we extend the GLM-UCB algorithms [11] via a regularization technique along with some noveltechniques.
Our proposed G-ESTS is simple and could be easily implemented based on anystate-of-the-art generalized linear bandit algorithms.
In particular, when we combine G-ESTS withsome efﬁcient algorithms (e.g. SGD-TS [9]), the total time complexity after a warm-up stage scalesas O
( Tr ( d 1 + d 2 )) .
We verify that G-ESTT and G-ESTS are the ﬁrst two algorithms to attain the˜ O
(( d 1 + d 2 ) r √ T ) optimal regret bound of low-rank matrix bandit problems up to logarithmic terms.
And their practical superiority is also validated in experiments.
Notations: For a vector x ∈ R n , we use (cid:107) x (cid:107) p to denote the l p -norm of the vector x and(cid:107) x (cid:107) H = √ x (cid:62) Hx to denote its weighted 2 − norm with regard to a positive deﬁnite matrix H ∈ R n × n .
For matrices X, Y ∈ R n 1 × n 2 , we use (cid:107) X (cid:107) op , (cid:107) X (cid:107) nuc and (cid:107) X (cid:107) F to deﬁne the operator norm, nuclearnorm and Frobenious norm of matrix X respectively, and we denote (cid:104) X, Y (cid:105) :
= trace ( X (cid:62) Y ) as theinner product between X and Y .
We write f ( n ) (cid:16) g ( n ) if f ( n ) =
O ( g ( n )) and g ( n ) =
O ( f ( n )) .
Related Work
In this section, we brieﬂy discuss some previous algorithms on low-rank matrix bandit problems.
Besides the works we have discussed in the former section, [18, 33] considered the rank-one banditproblems where the expected reward forms a rank-one matrix and the player selects an element fromthis matrix as the expected reward at each round.
In addition, [19] also studied the rank-one matrixbandit via an elimination-based algorithm.
Alternatively, [12, 20, 25] considered the general low-rankmatrix bandit, and furthermore [13] considered stochastic low-rank tensor bandit.
However, for allthese works the feature matrix of an action could be ﬂattened into a one-hot basis vector, and ourwork yields a more general structure.
Additionally, [24] extended some previous works [16] and presented a uniﬁed algorithm based ongreedy search for high-dimensional bandit problems.
But it’s nontrivial to extend the framework tothe matrix bandit problem.
For example, they assume that the minimum eigenvalue of the covariancematrix could be strictly lower bounded, but this lower bound would mostly depend on the size offeature matrices, and hence would affect the regret bound consequently.
Preliminaries
In this section we review our problem setting and introduce the assumptions for our theoreticalanalysis.
Let T be the total number of rounds and X t be the action set
( X t could be ﬁxed or not).
Throughout this paper, we denote the action set X t =
X as ﬁxed for notation simplicity, while ourframeworks also work with the same regret bound when X t varies over time (see Appendix H.3 formore details.)
Algorithms along with theory could be identically obtained when the action set varies(Appendix H.3).
At each round t ∈ [ T ] , The agent selects an action X t ∈ X t and gets the payoff y twhich is conditionally independent of the past payoffs and choices.
For the generalized low-rankmatrix bandits, we assume the payoff y t follows a canonical exponential family such that:
[Equation]
β =
[Equation]
X
[Equation]
where Θ ∗ ⊆
Θ is a ﬁxed but unknown matrix with rank r (cid:28) { d 1 , d 2 } and Θ is some admissiblecompact subset of R d 1 × d 2 .
We also call µ ( (cid:104)
X t , Θ (cid:105) ) as the reward of action X t .
In addition, one can represent model (1) in the following Eqn.
Note that if we relax the deﬁnitionof µ ( · ) to any real univariate function with some centered exogenous random noise η t , the modelshown in Eqn.
(2) generalizes our problem setting to a single index model (SIM) matrix bandit, andthe generalized low-rank matrix bandit problem is a special case of this model.
[Equation]
Here, η t follows the sub-Gaussian property with some constant parameter σ 0 conditional on theﬁltration F t = { X t , X t − 1 , η t − 1 , . . .
, X 1 , η 1 } .
We also denote d = max { d 1 , d 2 } .
And it is naturalto evaluate the agent’s strategy based on the regret [3], deﬁned as the difference between the totalreward of optimal policy and the agent’s total reward in practice:
[Equation]
We also present the following two deﬁnitions to facilitate further analysis via Stein’s method:
Deﬁnition 3.1.
Let p : R → R be a univariate probability density function deﬁned on R .
The score function S p : R → R regarding density p ( · ) is deﬁned as:
[Equation]
In particular, for a matrix probability density p = ( p ij ) :
R d 1 × d 2 → R d 1 × d 2 , we deﬁne the scorefunction S p
= ( S p ij ) :
R d 1 × d 2 → R d 1 × d 2 as S p ij ( x ) =
S p ij ( x ) by applying the univariate scorefunction to each entry of p .
Deﬁnition 3.2.
(Fact 2.6, [27])
Given a rectangular matrix A ∈ R d 1 × d 2 , the (Hermitian) dilation
H : R d 1 × d 2 → R ( d 1 + d 2 ) × ( d 1 + d 2 ) is deﬁned as:
[Equation]
We would omit the subscipt x of ∇ and the superscipt p of S when the underlying distribution isclear.
For any real-value function f ( · ) and symmetric matrix A with its SVD: A = UDU (cid:62) , we writef ( A ) =
Uf ( D ) U (cid:62) .
With these deﬁnitions, we make the following mild assumptions:
Assumption 3.3.
(Finite second-moment score)
There exists a sampling distribution D over X suchthat for the random matrix X drawn from D with its associated density p : R d 1 × d 2 → R d 1 × d 2 , wehave E [( S p ( X )) 2 ij ] ≤ M, ∀ i, j .
Assumption 3.4.
The norm of true parameter Θ ∗ and feature matrices in X is bounded: there exists
S ∈ R + such that for all arms X ∈ X , (cid:107) X (cid:107) F , (cid:107) Θ ∗ (cid:107) F ≤ S .
Assumption 3.5.
The inverse link function µ ( · ) in GLM is continuously differentiable and thereexist two constants c µ , k µ such that 0 < c µ ≤ µ (cid:48)
( x ) ≤ k µ for all | x | ≤ S .
Assumption 3.3 is commonly used in Stein’s method [7], and easily satisﬁed by a wide range ofdistribution that are non-zero-mean or even non sub-Gaussian thereby allowing us to work with casesnot previously possible.
For example, to ﬁnd D we only need the convex hull of X contains a ballwith radius R , and then we can use p ij as centerer normal p.d.f.
with variance R 2 / ( d i d j ) .
Thischoice works well in our experiments and please refer to Appendix I for more details.
Furthermore,
Assumption 3.4 and 3.5 are also standard in contextual generalized bandit literature, and they explicitlyimply that we have an upper bound as | µ ( (cid:104) X, Θ (cid:105) )
| ≤ | µ (0) | + max {| c µ | , | k µ |} S :=
S f .
Main Results
In this section, we present our novel two-stage frameworks, named Generalized Explore Subspace
Then Transform (G-ESTT) and Generalized Explore Subspace
Then Subtract (G-ESTS) respectively.
These two algorithms are inspired by the two-stage algorithm ESTR proposed in [17].
ESTR estimatesthe row and column subspaces for the true parameter Θ ∗ in stage 1.
In stage 2, it exploits the estimatedsubspaces and tranforms the original matrix bandits into linear bandits with sparsity, and then invokea penalized approach called LowOFUL.
To extend their work into the nonlinear reward setting, we ﬁrstly adapt stage 1 into the GLMframework by estimating Θ ∗ via the following quadratic optimization problem in Eqn.
(4) inspiredby a line of work in nonlinear signal estimation [29, 37].
And then we propose two different methodsbased on the estimated subspaces to deal with GLM bandits for stage 2: for G-ESTT we adapt theidea of LowOFUL into the GLM framework and propose an improved algorithms with regularization.
For G-ESTS we innovatively drop all negligible entries to get a low-dimensional bandit, which couldefﬁciently be solved by any modern generalized linear bandit algorithm with much less computation.
4.1 Stage 1: Subspace Exploration
To explore the valid subspace of the parameter matrix Θ ∗ , we ﬁrstly deﬁne a function ψ : R → R [27] in Eqn.
(3) and subsequently we deﬁne ˜ ψ ν : R d 1 × d 2 → R d 1 × d 2 as ˜ ψ ν ( A ) =
ψ ( ν H ( A )) 1: d 1 , ( d 1 +1):( d 1 + d 2 ) /ν for some parameter ν ∈ R + .
[Equation]
We consider the following well-deﬁned regularized minimization problem with nuclear norm penalty:
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
6) and similarly deﬁne the parameter set:
[Equation]
vec
[Equation]
8: For T 2 = T − T 1 rounds, invoke (P)LowGLM-UCB with X 0 , Θ 0 , k = ( d 1 + d 2 ) r − r 2 , ( λ 0 , λ ⊥ ) .
An interesting fact is that our estimator is invariant under different choices of function µ ( · ) , and wecould present the following oracle inequality regarding the estimation error (cid:13)(cid:13)(cid:13)(cid:98) Θ − µ ∗ Θ ∗ (cid:13)(cid:13)(cid:13) F for somenonzero constant µ ∗ by adapting generalized Stein’s Method [7].
Theorem 4.1.
(Bounds for GLM)
For any low-rank generalized linear model with samples
X 1 . . .
, X T 1 drawn from X according to D in Assumption 3.3, and assume Assumption 3.4 and 3.5 hold, then for the optimal solution to the nuclear norm regularization problem (4) with ν = (cid:113) 2 log(2( d 1 + d 2 ) /δ ) /
((4 σ 20 + S 2 f ) MT 1 d 1 d 2 ) and
[Equation]
with probability at least 1 − δ it holds that:
[Equation]
for C 1 = 36(4 σ 20 + S 2 f ) M and some nonzero constant µ ∗ .
The proof of Theorem 4.1 is based on novelly adapting Stein-typed Lemma and is deferred to
Appendix B. We believe this oracle bound is non-trivial since the rate of convergence is better thanthat deduced from the restricted strong convexity (details in Appendix J) even without assuming thesub-Gaussian property.
In addition, this bound also holds under a more general SIM in Eqn.
(2) otherthan just GLM.
Furthermore, although there exists a non-zero constant µ ∗ in the error term, it willnot affect the singular vectors and subspace estimation of Θ ∗ at all.
After acquiring the estimated (cid:98) Θ in stage 1, we can obtain the corresponding SVD as (cid:98) Θ =[ (cid:98) U, (cid:98)
U ⊥ ] (cid:98) D
[ (cid:98) V , (cid:98) V ⊥ ] (cid:62) , where (cid:98) U ∈ R d 1 × r , (cid:98)
U ⊥ ∈ R d 1 × ( d 1 − r ) , (cid:98)
V ∈ R d 2 × r and (cid:98) V ⊥ ∈ R d 2 × ( d 2 − r ) .
And we assume the SVD of the matrix Θ ∗ can be represented as Θ ∗ =
UDV (cid:62) where U ∈ R d 1 × rand V ∈ R
d 2 × r .
To transform the original generalized matrix bandits into generalized linear banditproblems, we follow the works in [17] and penalize those covariates that are complementary to (cid:98) Uand (cid:98) V .
Speciﬁcally, we could orthorgonally rotate the parameter space Θ and the action set X as:
[Equation]
Deﬁne the total dimension p :=
d 1 d 2 , the effective dimension k :=
d 1 d 2 − ( d 1 − r )( d 2 − r ) andthe r -th largest singular value for Θ ∗ as D rr , and vectorize the new arm space X (cid:48) and admissible
Algorithm
Input: T 2 , k, X 0 , the probability rate δ , penalization parameters ( λ 0 , λ ⊥ ) .
Initialize M 1 ( c µ ) =
(cid:80) T 1 i =
1 x s 1 ,i x (cid:62) s 1 ,i + Λ /c µ .
for t ≥ 1 do
Estimate ˆ θ t according to (10).
Choose the arm x t = arg max x ∈X 0 { µ ( x (cid:62) ˆ θ t ) + ρ t ( δ ) (cid:107) x (cid:107) M − 1 t ( c µ ) } , receive y t ,
Update M t +1 ( c µ ) ←− M t ( c µ ) + x t x (cid:62) t .
end for parameter space as shown in Eqn.
(6) and (7).
Then for the true parameter θ ∗ after transformation,we know that θ ∗ k
+1: p = vec (Θ ∗ , (cid:48)
r +1: d 1 ,r +1: d 2 ) is almost null based on results in [32] and Theorem 4.1:
[Equation]
Therefore, this problem degenerates to an equivalent d 1 d 2 − dimensional generalized linear banditwith a sparse structure (i.e. last p − k entries of θ ∗ are almost null according to Eqn.
To reloadthe notation we deﬁne X 0 , Θ 0 as the new feature set and parameter space as shown in Algorithm 1.
4.2 Stage 2 of G-ESTT
After reducing the original generalized matrix bandit problem into an identical p -dimensional generalized linear bandit problem in stage 2, we can reformulate the problem in the following way: ateach round t , the agent chooses a vector x t of dimension p from the transformed action set X 0 , andobserves a noisy reward y t = µ ( x t (cid:62) θ ∗ )+ η t .
To make use of our additional knowledge shown in Eqn.(8), we propose LowGLM-UCB as an extension of the standard generalized linear bandit algorithm
GLM-UCB [11] combined with self-normalized martingale technique [1].
Speciﬁcally, we considerthe following maximum quasi-likelihood estimation problem shown in Eqn.
(9) for each round with aweighted regularizer, where the regularizer is (cid:107) θ (cid:107) 2Λ / 2 = θ (cid:62) Λ θ/ 2 for some positive deﬁnite diagonalmatrix Λ
= diag ( λ 0 , . . .
, λ 0 , λ ⊥ , . .
. , λ ⊥ ) with λ 0 only applied to the ﬁrst k diagonal entries.
Byenlarging λ ⊥ , we ensure more penalization forced on the last p − k element of θ ∗ as desired.
[Equation]
where x s 1 ,i is the special vectorization shown in Eqn.
(6) of [ (cid:98) U, (cid:98) U ⊥ ] (cid:62) X
i [ (cid:98) V , (cid:98) V ⊥ ] and X i is the armwe randomly pull at i -th step in stage 1, and y s 1 ,i is the corresponding payoff we observe.
Since(cid:101)
L Λ t ( θ ) is a strictly concave function of θ , we have its gradient equal to 0 at the maximum ˆ θ t , i.e.∇ θ (cid:101)
L Λ t ( θ ) | ˆ θ t = 0 .
In what follows, for t ≥ 2 , θ ∈ R p we deﬁne the function g t ( θ ) and have that
[Equation]
We also deﬁne a matrix function
[Equation]
1 x[-s 1 ,i x (cid:62) s 
 ,i + (cid:80)-] t x (cid:62) t + Λ /s for s ∈ R + anddenote V t :
= M t (1) .
Furthermore, a remarkable beneﬁt of reusing the actions { X i } T 1 i
=1 we randomlypull in stage 1 is that they contain more randomness and are preferable to the ones we select basedon some strategy in stage 2 regarding the parameter estimation because most vector recovery theoryrequires sufﬁcient randomness during sampling.
More inspiring, the projection step in the tradition
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
, invoke any generalized linear bandit algorithm with X 0 ,sub , Θ 0 ,sub , k = ( d 1 + d 2 ) r − r 2 .
GLM-UCB [11], which might be nonconvex and hence hard to solve, is no longer required due theconsistency of ˆ θ t after reutilizing { X i } T 1 i =1 .
Speciﬁcally, if we assume the true parameter θ ∗ lies inthe interior of Θ 0 and the sampling distribution D satisﬁes sub-Gaussian property with parameterσ , and Assumption 3.4, 3.5 held, then we can show that (cid:13)(cid:13)(cid:13)
ˆ θ t − θ ∗ (cid:13)(cid:13)(cid:13) 2 ≤ 1 holds with probabilityat least 1 − δ as long as T 1 ≥ (( ˆ C 1 √ p + ˆ C 2 (cid:112) log(1 /δ ))
[-/σ 
 +-]/λ 1 ) 2 + 2 B/λ 1 holds for some absoluteconstants ˆ C 1 , ˆ C 2 with the deﬁnition B := 16 σ 20 ( p + log(1 /δ )) /c
2 µ .
An intuitive explanation alongwith rigorous proof are deferred to Appendix D due to the space limit.
The proposed LowGLM-UCBis shown in Algorithm 2, and its regret analysis presented in Theorem C.1 in Appendix.
Notice that we can simply replace M t ( c µ ) by V t in Algorithm 2, and the regret bound would increaseat most up to a constant factor (Appendix G).
A potential drawback of Algorithm 2 is that in eachiteration we have to calculate ˆ θ t , which might be computationally expensive.
We could resolve thisproblem by only recomputing ˆ θ t whenever | M t ( c µ ) | increases signiﬁcantly, i.e. by a constant factor
C > 1 in scale.
And consequently we only need to solve the Eqn.
(10) for O (log( T 2 )) times up tothe horizon T 2 , which remarkably saves the computation.
Meanwhile, the bound of the regret wouldonly increase by a constant multiplier √ C .
We call this modiﬁed algorithm as PLowGLM-UCBwith initial letter “P” standing for “Parsimonious”.
Its pseudo-code and regret analysis are given in
Appendix H.1.
Equipped with LowGLM-UCB in stage 2, we deduce the overall regret of G-ESTT:
4.2.1 Overall regret of G-ESTT
To quantify the performance of our algorithm, we ﬁrst deﬁne α xt ( · ) and β xt ( · ) as
[Equation]
And the following Theorem 4.2 exhibits the overall regret bound for G-ESTT.
Theorem 4.2.
(Regret of G-ESTT)
Suppose we set T 1 (cid:16) (cid:112)
d 1 d 2 rT log((
d 1 + d 2 )
/δ ) /D
rr , andwe invoke (P)LowGLM-UCB in stage 2 with ρ t ( δ ) =
α t + T 1 ( δ/ 2) , p = d 1 d 2 , k = ( d 1 + d 2 ) r − r 2 ,λ ⊥ =
c µ S 2 T/ ( k log(1 + c µ S 2 T/ ( kλ 0 ))) , and the rotated arm sets X 0 and available parameterspace Θ 0 .
The overall regret of G-ESTT is, with probability at least 1 − δ ,
[Equation]
Note G-ESTT improves all existing regret bounds by a √ d factor based on Table 1 given that r (cid:28) d .
4.3 Stage 2 of G-ESTS
Although G-ESTT is more efﬁcient than all existing algorithms on our problem setting, it still needsto calculate the MLE in high dimensional space which might be increasingly formidable with largesizes of feature matrices.
Note this computational issue remains ubiquitous among most banditalgorithms on high dimensional problems with sparsity, not to mention these algirthms rely onmultiple unspeciﬁed hyperparameters.
Therefore, to handle this practical issue, we propose anotherfast and efﬁcient framework called G-ESTS in this section.
Inspired by the success of dimension reduction in machine learning [34], we propose G-ESTS asshown in Algorithm 3.
And we summarize the core idea of G-ESTS as: After rearranging thevectorization of the action set X (cid:48) and the unknown Θ (cid:48)∗ as we have shown in Eqn.
(6) and (7) for
G-ESTT, we can simply exclude, rather than penalize, the subspaces that are complementary to therows and columns of (cid:98) Θ .
In other words, we could remove the last p − k entries directly, i.e. Eqn.
(13)and (14).
Intriguingly, not only can we get a low-dimensional ( k ) generalized linear bandit problemin stage 2, where redundant dimensions are excluded and hence any state-of-the-art algorithms couldbe readily invoked.
But also we obtain the optimal regret bound of scale ˜ O
( dr √ T ) again as shownin Table 1.
Besides, G-ESTS requires exceedingly less computation compared with other existingalgorithms since consequently we get a low-dimensional bandit problem.
In summary, we believe our
G-ESTS is superior regarding both theoretical guarantee and computational efﬁciency.
4.3.1 Overall regret of G-ESTS
Theorem 4.3.
(Regret of G-ESTS)
Suppose we set T 1 (cid:16) (cid:112)
d 1 d 2 rT log((
d 1 + d 2 )
/δ ) /D
rr , and weinvoke any efﬁcient generalized linear bandit algorithm with regret bound ˜ O ( √ kT ) 2 in stage 2 withp =
d 1 d 2 , k = ( d 1 + d 2 ) r − r 2 , and the reduced arm sets X 0 ,sub and available parameter spaceΘ 0 ,sub .
The overall regret of G-ESTS is, with probability at least 1 − δ ,
[Equation]
According to Theorem 4.2 and 4.3, both G-ESTT and G-ESTS could achieve an improved ˜ O ( dr √ T )regret bound.
Surprisingly, this matches the lower bound of low-rank matrix bandits presented in [26]up to logarithm terms.
To the best of our knowledge, our methods the ﬁrst two methods that attainthis optimal rate.
In addition, given the ﬁnite arm set, our G-ESTS framework can even achieve abetter regret bound of order ˜ O ( √ d 1 d 2 rT ) (see Appendix F.2 for detailed proof), and this result isbetter than the lower regret bound for the case of inﬁnite arms by a √ r factor.
In following experments, We will implement the SGD-TS algorithm [9] in the stage 2 of G-ESTSsince SGD-TS could efﬁciently proceed with only O ( dT ) complexity for d − dimensional featuresover T rounds.
Therefore, the total computational complexity of stage 2 is at most O ( T 2 ( d 1 + d 2 ) r ) ,which is signiﬁcantly less than that of other methods for low-rank matrix bandits (e.g. LowESTR [26]).
And the total time complexity of G-ESTS would only scale O ( T 1 d 1 d 2 /(cid:15) 2 + T 2 ( d 1 + d 2 ) r ) where(cid:15) is the accuracy for subgradient methods in stage 1.
This fact also ﬁrmly validates the practicalsuperiority of our G-ESTS approach.
We naturally believe that this G-ESTS framework can be easilyimplemented in the linear setting as a special case of GLM, where in stage 2 one can utilize any linearbandit algorithm accordingly.
In addition, we can easily modify our approaches for the contextualsetting by merely transforming the action sets at each iteration with the same regret bound.
Moredetails with pseudo-codes for the contextual case are in Appendix H.3.
Experiments
In this section, we show by simulation experiments that our proposed G-ESTT (with LowGLM-UCB),
G-ESTS (with SGD-TS) outperform existing algorithms for the generalized low-rank matrix banditproblems.
Since we are the ﬁrst to propose a practical algorithm for this problem, currently there isno existing literature for comparison.
In order to validate the advantage of utilizing low-rank structureand generalized reward functions, we compare with the original SGD-TS after naïvely ﬂattening the
[Table]
d 1 by d 2 matrices without using the low-rank structure, and LowESTR [26], which works well forlinear low-rank matrix bandits.
We simulate a dataset with d 1 = d 2 = 10 (12) and r = 1 (2) : when r = 1 , we set the diagonalmatrix Θ ∗ as diag (Θ ∗ ) =
When r = 2 , we set Θ ∗ = v 1 v (cid:62) 1 + v 2 v (cid:62) 2 for two randomorthorgonal vectors v 1 , v 2 with (cid:107) v 1 (cid:107) 2 = (cid:107) v 2 (cid:107) 2 = 3 .
For arms we draw 480 (1000)
random matricesfrom { X ∈ R
d
1 × d 2 : (cid:107) X (cid:107) F ≤ 1 } , and we build a logistic model where the payoff y t is drawn froma Bernoulli distribution with mean µ ( X (cid:62) t θ ∗ ) .
More details on the hyper-parameter tuning are in
Appendix I.
Each experiment is repeated for 100 times for credibility and the average regret, alongwith standard deviation, is displayed in Figure 1.
Note that our experiments are more comprehensivethan those in [26].
And due to the expensive time complexity of UCB-based baselines (Table 2), it isformidable for us to increase d here.
From the plots, we observe that our algorithms G-ESTT and G-ESTS always achieve less regretcompared with LowESTR and SGD-TS in all four scenarios consistently.
Intriguingly, in the warm-upperiod SGD-TS incurs less regret compared with our methods due to the sacriﬁce of random samplingin stage 1, but our proposed framework quickly overtakes SGD-TS after utilizing the low-rankstructure as desired.
This phenomenon exactly coincides with our theory.
Notice that G-ESTT isslightly better than G-ESTS in the case for r = 2 especially in the very beginning of stage 2, and webelieve it is because that our G-ESTT could reutilize the actions in stage 1 and hence could yieldmore robust performance when switching to stage 2.
However, G-ESTS would gradually catch upwith G-ESTT in the long run as expected.
Besides, it costs G-ESTS extremely less running time thanother existing methods to update the decisions due to its dimensional reduction as shown in Table2.
We also observe that the cumulative regret of G-ESTS tends to become better eventually if weincrease T 1 decently.
(Further investigation and plots for 1000 arms are in Appendix I.)
[Figure]
Conclusion
In this paper, we discussed the generalized linear low-rank matrix bandit problem.
We proposed twonovel and efﬁcient frameworks called G-ESTT and G-ESTS, and both methods could achieve animproved (cid:101)
O ( dr √ T ) bound of regret under some mild conditions.
There are several directions for our future work.
Firstly, although the GLM has achieved tremendoussuccess under various settings, recently some other models [39] that are proved to be powerfulwere proposed, so we can study these modern frameworks in low-rank matrix case.
Secondly, itseems more reasonable to continuously update the subspace estimation via online SVD over T [17]on-the-ﬂy with randomized strategy.
Therefore, we can consider something like arbitrarily choosingan admissible subset of the action set at each round [31] without hurting the regret too much.