Efﬁcient Frameworks for Generalized Low-Rank Matrix Bandit Problems
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In the stochastic contextual low-rank matrix bandit problem, the expected rewardof an action is given by the inner product between the action’s feature matrix andsome ﬁxed, but initially unknown d 1 by d 2 matrix Θ ∗ with rank r (cid:28) { d 1 , d 2 } ,and an agent sequentially takes actions based on past experience to maximizethe cumulative reward.
In this paper, we study the generalized low-rank matrixbandit problem, which has been recently proposed in [26] under the Generalized
Linear Model (GLM) framework.
To overcome the computational infeasibility andtheoretical restrain of existing algorithms on this problem, we ﬁrst propose the
G-ESTT framework that modiﬁes the idea from [17] by using Stein’s method on thesubspace estimation and then leverage the estimated subspaces via a regularizationidea.
Furthermore, we remarkably improve the efﬁciency of G-ESTT by using anovel exclusion idea on the estimated subspace instead, and propose the G-ESTSframework.
We also show that both of our methods are the ﬁrst algorithm to achievethe optimal ˜ O
(( d 1 + d 2 ) r √ T ) bound of regret presented in [26] up to logarithmterms, which improves upon the current regret of ˜ O
(( d 1 + d 2 ) 3 / 2 √ rT ) [26].
And
G-ESTS could even attain a better ˜ O ( √ d 1 d 2 rT ) regret bound under the ﬁnite-armcase.
For completeness, we conduct experiments to illustrate that our proposedalgorithms, especially G-ESTS, are computationally tractable and consistentlyoutperform other state-of-the-art (generalized) linear matrix bandit methods.
Introduction
The contextual bandit has proven to be a powerful framework for sequential decision-making problems, with great applications to clinical trials [36], recommendation system [22] and personalizedmedicine [4].
This class of problems evaluates how an agent should choose an action from the potential action set at each round based on an updating policy on-the-ﬂy so as to maximize the cumulativereward or minimize the overall regret.
With high dimensional sparse data becoming ubiquitous invarious ﬁelds nowadays, the most fundamental (generalized) linear bandit framework, although hasbeen extensively studied, becomes inefﬁcient in practice.
This fact consequently leads to a line ofwork on stochastic high dimensional bandit problems with low dimensional structures [16, 24], suchas the LASSO bandit and low-rank matrix bandit.
In this work, we investigate on the generalized low-rank matrix bandit problem ﬁrstly studiedin [26]: at round t = 1 , . .
, T , the algorithm selects an action represented by a d 1 by d 2 matrix X tfrom the admissible action set X t ( X t may be ﬁxed), and receives its associated noisy reward y t =µ ( (cid:104) Θ ∗ , X t (cid:105) )
+ η t where Θ ∗ ∈ R
d 1 × d 2 is some unknown low-rank matrix with rank r (cid:28) { d 1 , d 2 } andµ ( · ) is the inverse link function.
More details about this setting are deferred to Section 3.
This problemhas vast applicability in real world applications.
On the one hand, matrix inputs are appropriate whendealing with paired contexts which are omnipresent in practice.
For instance, to design a personalized
[Table]
movie recommendation system, we can formulate each user as m d 1 -dimensional feature vectors( x 1 , . . .
, x m ∈ R d 1 ) and each movie as m d 2 -dimensional feature vectors ( y 1 , . .
y m ∈ R d 2 ).
Auser-item pair can then be naturally represented by a feature matrix deﬁned as the summation ofthe outer products (cid:80) mk =1 x k y (cid:62)
k ∈ R d 1 × d 2 , which will become the contextual feature observed bythe bandit algorithm.
Other applications involve interaction features between two groups, such asﬂight-hotel bundles [26] and dating service [17] can also be similarly established.
Besides, low-rankmodels have gained tremendous success in various areas [5].
In particular, our problem can beregarded as an extension of the inductive matrix factorization problem [14, 38], which estimateslow-rank matrices with contextual information, under the online learning scenario.
Our study is inspired by a line of work on stochastic contextual low-rank matrix bandit [15, 17, 26].
To design an algorithm for matrix bandit problems, a naïve approach is to ﬂatten the d 1 by d 2 featurematrices into vectors and then apply any (generalized) linear bandit algorithms, which, however,would be inefﬁcient when d 1 d 2 is large.
To take advantage of the low-rank structure, [17] haveintroduced the bilinear low-rank bandit problem and proposed a two-stage algorithm named ESTRwhich could achieve a regret bound of ˜ O
(( d 1 + d 2 ) 3 / 2 √ rT ) 1 .
Subsequently, [15] constructed a newalgorithm called (cid:15) -FALB for bilinear bandits and achieved a better regret of ˜ O
( (cid:112) d 1 d 2 ( d 1 + d 2 ) T ) .
However, they only studied the linear reward framework and also restricted the feature matrix as arank-one matrix.
As a follow-up work, [26] further released the rank-one restriction on the actionfeature matrices, and they introduced an algorithm LowGLOC based on the online-to-conﬁdenceset conversion [2] for generalized low-rank matrix bandits with ˜ O
( (cid:112) ( d 1 + d 2 ) 3 rT ) regret bound.
However, this regret bound is still loose compared with the optimal rate ˜ O
(( d 1 + d 2 ) r √ T ) deducedin [15], and the arm set is still assumed ﬁxed at each round.
This algorithm is also computationallyprohibitive since it requires to calculate the weights of a self-constructed covering of the admissibleparameter space at each iteration.
And how to ﬁnd this covering for low-rank matrices is also unclear.
In this work, we propose two efﬁcient methods called G-ESTT and G-ESTS for this problem bymodifying two stages of ESTR appropriately from different perspectives.
To the best of our knowledge,the proposed methods are the ﬁrst two generalized (contextual) low-rank bandit algorithms that arecomputationally feasible, and achieve an improved regret bound of ˜ O
(( d 1 + d 2 ) r √ T ) upon all existingworks on (bilinear) low-rank bandits.
The main contributions of this paper can be summarized as: 1)we propose two novel two-stage frameworks G-ESTT and G-ESTS under some mild assumptions.
Compared with ESTR in [17], (cid:15) -FALB in [15] and LowESTR in [26], our algorithms are proposed forthe nonlinear reward framework with arbitrary action matrices.
Compared with LowGLOC in [26],our algorithms not only achieve a better regret bound in theory, but also are computationally feasiblein practice.
For G-ESTT, we extend the GLM-UCB algorithms [11] via a novel regularizationtechnique.
Our proposed G-ESTS is simple and could be easily implemented based on anystate-of-the-art generalized linear bandit algorithms.
In particular, when we combine G-ESTS withsome efﬁcient algorithms (e.g. SGD-TS [9]), the total time complexity after a warm-up stage scalesas O
( Tr ( d 1 + d 2 )) .
We verify that G-ESTT and G-ESTS are the ﬁrst two algorithms to attain the˜ O
(( d 1 + d 2 ) r √ T ) optimal regret bound of low-rank matrix bandit problems up to logarithmic terms.
And their practical superiority is also validated in experiments.
Notations: For a vector x ∈ R n , we use (cid:107) x (cid:107) p to denote the l p -norm of the vector x and (cid:107) x (cid:107) H = 77 √ x (cid:62) Hx to denote its weighted 2 − norm with regard to a positive deﬁnite matrix H ∈ R n × n .
For1 ˜ O ignores the polylogarithmic factors.
matrices X, Y ∈ R n 1 × n 2 , we use (cid:107) X (cid:107) op , (cid:107) X (cid:107) nuc and (cid:107) X (cid:107) F to deﬁne the operator norm, nuclearnorm and Frobenious norm of matrix X respectively, and we denote (cid:104) X, Y (cid:105) :
= trace ( X (cid:62) Y ) as theinner product between X and Y .
We write f ( n ) (cid:16) g ( n ) if f ( n ) =
O ( g ( n )) and g ( n ) =
O ( f ( n )) .
Related Work
In this section, we brieﬂy discuss some previous algorithms on low-rank matrix bandit problems.
Besides the works we have discussed in the former section, [18, 33] considered the rank-one banditproblems where the expected reward forms a rank-one matrix and the player selects an element fromthis matrix as the expected reward at each round.
In addition, [19] also studied the rank-one matrixbandit via an elimination-based algorithm.
Alternatively, [12, 20, 25] considered the general low-rankmatrix bandit, and furthermore [13] considered a stochastic low-rank tensor bandit.
However, for allthese works the feature matrix of an action could be ﬂattened into a one-hot basis vector, and ourwork yields a more general structure.
Additionally, [24] extended some previous works [16] and presented a uniﬁed algorithm based on agreedy search for high-dimensional bandit problems.
But it’s nontrivial to extend the framework tothe matrix bandit problem.
For example, they assume that the minimum eigenvalue of the covariancematrix could be strictly lower bounded, but this lower bound would mostly depend on the size offeature matrices, and hence would affect the regret bound consequently.
Preliminaries
In this section we review our problem setting and introduce the assumptions for our theoreticalanalysis.
Let T be the total number of rounds and X t be the action set
( X t could be ﬁxed or not).
Throughout this paper, we denote the action set X t =
X as ﬁxed for notation simplicity, while ourframeworks also work with the same regret bound when X t varies over time (see Appendix H.3 formore details.)
Algorithms along with theory could be identically obtained when the action set varies(Appendix H.3).
At each round t ∈ [ T ] , The agent selects an action X t ∈ X t and gets the payoff y twhich is conditionally independent of the past payoffs and choices.
For the generalized low-rankmatrix bandits, we assume the payoff y t follows a canonical exponential family such that:
[Equation]
β
[Equation]
X
[Equation]
where Θ ∗ ⊆
Θ is a ﬁxed but unknown matrix with rank r (cid:28) { d 1 , d 2 } and Θ is some admissiblecompact subset of R d 1 × d 2 .
We also call µ ( (cid:104)
X t , Θ (cid:105) ) as the reward of action X t .
In addition, one can represent model (1) in the following Eqn.
Note that if we relax the deﬁnitionof µ ( · ) to any real univariate function with some centered exogenous random noise η t , the modelshown in Eqn.
(2) generalizes our problem setting to a single index model (SIM) matrix bandit, andthe generalized low-rank matrix bandit problem is a special case of this model.
[Equation]
Here, η t follows the sub-Gaussian property with some constant parameter σ 0 conditional on theﬁltration F t = { X t , X t − 1 , η t − 1 , . . .
, X 1 , η 1 } .
We also denote d = max { d 1 , d 2 } .
And it is naturalto evaluate the agent’s strategy based on the regret [3], deﬁned as the difference between the totalreward of optimal policy and the agent’s total reward in practice:
[Equation]
We also present the following two deﬁnitions to facilitate further analysis via Stein’s method:
Deﬁnition 3.1.
Let p : R → R be a univariate probability density function deﬁned on R .
The score function S p : R → R regarding density p ( · ) is deﬁned as:
[Equation]
In particular, for a random matrix with its entrywise probability density p = ( p ij ) :
R d 1 × d 2 → R d 1 × d 2 , we deﬁne its score function S p
= ( S p ij ) :
R d 1 × d 2 → R d 1 × d 2 as S p [Equation] ( x ) =
S p ij ( x ) byapplying the univariate score function to each entry of p independently.
Deﬁnition 3.2.
(Fact 2.6, [27])
Given a rectangular matrix A ∈ R d 1 × d 2 , the (Hermitian) dilation
H : R d 1 × d 2 → R ( d 1 + d 2 ) × ( d 1 + d 2 ) is deﬁned as: [Equation]
[Equation]
We would omit the subscript x of ∇ and the superscript p of S when the underlying distribution isclear.
With these deﬁnitions, we make the following mild assumptions:
Assumption 3.3.
(Finite second-moment score)
There exists a sampling distribution D over X suchthat for the random matrix X drawn from D with its associated density p : R d 1 × d 2 → R d 1 × d 2 , wehave E [( S p ( X )) 2 ij ] ≤ M, ∀ i, j .
Assumption 3.4.
The norm of true parameter Θ ∗ and feature matrices in X is bounded: there exists
S ∈ R + such that for all arms X ∈ X , (cid:107) X (cid:107) F , (cid:107) Θ ∗ (cid:107) F ≤ S 0 .
Assumption 3.5.
The inverse link function µ ( · ) in GLM is continuously differentiable and thereexist two constants c µ , k µ such that 0 < c µ ≤ µ (cid:48)
( x ) ≤ k µ for all | x | ≤ S 0 .
Assumption 3.3 is commonly used in Stein’s method [7], and easily satisﬁed by a wide range ofdistributions that are non-zero-mean or even non sub-Gaussian thereby allowing us to work withcases not previously possible.
For example, to ﬁnd D we only need the convex hull of X contains aball with radius R , and then we can use p ij as centered normal p.d.f.
with variance R 2 / ( d i d j ) .
Thischoice works well in our experiments and please refer to Appendix I for more details.
Furthermore,
Assumption 3.4 and 3.5 are also standard in contextual generalized bandit literature, and they explicitlyimply that we have an upper bound as | µ ( (cid:104) X, Θ (cid:105) )
| ≤ | µ (0) | + max {| c µ | , | k µ |} S 0 :
= S f .
Main Results
In this section, we present our novel two-stage frameworks, named Generalized Explore Subspace
Then Transform (G-ESTT) and Generalized Explore Subspace
Then Subtract (G-ESTS) respectively.
These two algorithms are inspired by the two-stage algorithm ESTR proposed in [17].
ESTR estimatesthe row and column subspaces for the true parameter Θ ∗ in stage 1.
In stage 2, it exploits the estimatedsubspaces and transforms the original matrix bandits into linear bandits with sparsity, and then invokea penalized approach called LowOFUL.
To extend their work into the nonlinear reward setting, we ﬁrstly adapt stage 1 into the GLMframework by estimating Θ ∗ via the following quadratic optimization problem in Eqn.
(4) inspiredby a line of work in nonlinear signal estimation [29, 37].
And then we propose two different methodsbased on the estimated subspaces to deal with GLM bandits for stage 2: for G-ESTT we adapt theidea of LowOFUL into the GLM framework and propose an improved algorithm with regularization.
For G-ESTS we innovatively drop all negligible entries to get a low-dimensional bandit, which couldefﬁciently be solved by any modern generalized linear bandit algorithm with much less computation.
4.1 Stage 1: Subspace Exploration
For any real-value function f ( · ) and symmetric matrix A ∈ R d with its SVD: A = UDU (cid:62) , wedeﬁne f
( A ) :=
U diag ( f ( D 11 ) , . . . ,
f ( D dd )) U (cid:62) . To explore the valid subspace of the parametermatrix Θ ∗ , we ﬁrstly deﬁne a function ψ : R → R [27] in Eqn.
(3) and subsequently we deﬁne˜ ψ ν : R d 1 × d 2 → R d 1 × d 2 as ˜ ψ ν ( A ) =
ψ ( ν H ( A )) 1: d 1 , ( d 1 +1):( d 1 + d 2 ) /ν for some parameter ν ∈ R + .
[Equation]
We consider the following well-deﬁned regularized minimization problem with nuclear norm penalty:
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
6) and similarly deﬁne the parameter set:
[Equation]
vec
[Equation]
8: For T 2 = T − T 1 rounds, invoke (P)LowGLM-UCB with X 0 , Θ 0 , k = ( d 1 + d 2 ) r − r 2 , ( λ 0 , λ ⊥ ) .
An interesting fact is that our estimator is invariant under different choices of function µ ( · ) , and wecould present the following oracle inequality regarding the estimation error (cid:13)(cid:13)(cid:13)(cid:98) Θ − µ ∗ Θ ∗ (cid:13)(cid:13)(cid:13) F for somenonzero constant µ ∗ by adapting generalized Stein’s Method [7].
Theorem 4.1.
(Bounds for GLM)
For any low-rank generalized linear model with samples
X 1 . . .
, X T 1 drawn from X according to D in Assumption 3.3, and assume Assumption 3.4 and 3.5 hold, then for the optimal solution to the nuclear norm regularization problem (4) with ν = (cid:113) 2 log(2( d 1 + d 2 ) /δ ) /
((4 σ 20 + S 2 f ) MT 1 d 1 d 2 ) and
[Equation]
with probability at least 1 − δ it holds that:
[Equation]
for C 1 = 36(4 σ 20 + S 2 f ) M and some nonzero constant µ ∗ .
The proof of Theorem 4.1 is based on novelly adapting Stein-typed Lemma and is deferred to
Appendix B. We believe this oracle bound is non-trivial since the rate of convergence is better thanthat deduced from the restricted strong convexity (details in Appendix J) even without assuming thesub-Gaussian property.
In addition, this bound also holds under a more general SIM in Eqn.
(2) otherthan just GLM.
Furthermore, although there exists a non-zero constant µ ∗ in the error term, it willnot affect the singular vectors and subspace estimation of Θ ∗ at all.
After acquiring the estimated (cid:98) Θ in stage 1, we can obtain the corresponding SVD as (cid:98) Θ =[ (cid:98) U, (cid:98)
U ⊥ ] (cid:98) D
[ (cid:98) V , (cid:98) V ⊥ ] (cid:62) , where (cid:98) U ∈ R d 1 × r , (cid:98)
U ⊥ ∈ R d 1 × ( d 1 − r ) , (cid:98)
V ∈ R d 2 × r and (cid:98) V ⊥ ∈ R d 2 × ( d 2 − r ) .
And we assume the SVD of the matrix Θ ∗ can be represented as Θ ∗ =
UDV (cid:62) where U ∈ R d 1 × rand V ∈ R
d 2 × r .
To transform the original generalized matrix bandits into generalized linear banditproblems, we follow the works in [17] and penalize those covariates that are complementary to (cid:98) Uand (cid:98) V .
Speciﬁcally, we could orthogonally rotate the parameter space Θ and the action set X as:
[Equation]
Deﬁne the total dimension p :=
d 1 d 2 , the effective dimension k :=
d 1 d 2 − ( d 1 − r )( d 2 − r ) andthe r -th largest singular value for Θ ∗ as D rr , and vectorize the new arm space X (cid:48) and admissible
Algorithm
Input: T 2 , k, X 0 , the probability rate δ , penalization parameters ( λ 0 , λ ⊥ ) .
Initialize M 1 ( c µ ) =
(cid:80) T 1 i =
1 x s 1 ,i x (cid:62) s 1 ,i + Λ /c µ .
for t ≥ 1 do
Estimate ˆ θ t according to (10).
Choose the arm x t = arg max x ∈X 0 { µ ( x (cid:62) ˆ θ t ) + ρ t ( δ ) (cid:107) x (cid:107) M − 1 t ( c µ ) } , receive y t ,
Update M t +1 ( c µ ) ←− M t ( c µ ) + x t x (cid:62) t .
end for parameter space as shown in Eqn.
(6) and (7).
Then for the true parameter θ ∗ after transformation,we know that θ ∗ k
+1: p = vec (Θ ∗ , (cid:48)
r +1: d 1 ,r +1: d 2 ) is almost null based on results in [32] and Theorem 4.1:
[Equation]
Therefore, this problem degenerates to an equivalent d 1 d 2 − dimensional generalized linear banditwith a sparse structure (i.e. last p − k entries of θ ∗ are almost null according to Eqn.
To reloadthe notation we deﬁne X 0 , Θ 0 as the new feature set and parameter space as shown in Algorithm 1.
4.2 Stage 2 of G-ESTT
After reducing the original generalized matrix bandit problem into an identical p -dimensional generalized linear bandit problem in stage 2, we can reformulate the problem in the following way: ateach round t , the agent chooses a vector x t of dimension p from the transformed action set X 0 , andobserves a noisy reward y t = µ ( x t (cid:62) θ ∗ )+ η t .
To make use of our additional knowledge shown in Eqn.(8) , we propose LowGLM-UCB as an extension of the standard generalized linear bandit algorithm
GLM-UCB [11] combined with self-normalized martingale technique [1].
Speciﬁcally, we considerthe following maximum quasi-likelihood estimation problem shown in Eqn.
(9) for each round with aweighted regularizer, where the regularizer is (cid:107) θ (cid:107) 2Λ / 2 = θ (cid:62) Λ θ/ 2 for some positive deﬁnite diagonalmatrix Λ
= diag ( λ 0 , . . .
, λ 0 , λ ⊥ , . .
. , λ ⊥ ) with λ 0 only applied to the ﬁrst k diagonal entries.
Byenlarging λ ⊥ , we ensure more penalization forced on the last p − k element of θ ∗ as desired.
[Equation]
Here x s 1 ,i in Eqn. (9) is the special vectorization shown in Eqn.
(6) of [ (cid:98) U, (cid:98) U ⊥ ] (cid:62) X
i [ (cid:98) V , (cid:98) V ⊥ ] where X i is the arm we randomly pull at i -th step in stage 1, and y s 1 ,i is the corresponding payoff weobserve.
x i in the second summation of Eqn. (9) refers to the arm we pull at i -th step in stage 2.
Since (cid:101) L Λ t ( θ ) is a strictly concave function of θ , we have its gradient equal to 0 at the maximum ˆ θ t ,i.e. ∇ θ (cid:101) L Λ t ( θ )
| ˆ θ t = 0 .
In what follows, for t ≥ 2 , θ ∈ R p we deﬁne the function g t ( θ ) and have that [Equation]
[Equation]
We also deﬁne a matrix function M t ( s )
= (cid:80) T 1 i =
1 x s 1 ,i x (cid:62) {+s 
 ,i + (cid:80)+} t − 1 k =1 x k x (cid:62) k + Λ /s for s ∈ R + anddenote V t :=
M t (1) .
Furthermore, a remarkable beneﬁt of reusing the actions { X i } T 1 i
=1 we randomlypull in stage 1 is that they contain more randomness and are preferable to the ones we select basedon some strategy in stage 2 regarding the parameter estimation because most vector recovery theoryrequires sufﬁcient randomness during sampling.
More inspiring, the projection step in the tradition
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
, invoke any generalized linear bandit algorithm with X 0 ,sub , Θ 0 ,sub , k = ( d 1 + d 2 ) r − r 2 .
GLM-UCB [11], which might be nonconvex and hence hard to solve, is no longer required due theconsistency of ˆ θ t after reutilizing { X i } T 1 i =1 .
Speciﬁcally, if we assume the true parameter θ ∗ lies in theinterior of Θ 0 and the sampling distribution D satisﬁes sub-Gaussian property with parameter σ , and Assumption 3.4, 3.5 held, then we can show that (cid:13)(cid:13)(cid:13)
ˆ θ t − θ ∗ (cid:13)(cid:13)(cid:13) 2 ≤ 1 holds with probability at least 1 − δas long as T 1 ≥ (( ˆ C 1 √ p + ˆ C 2 (cid:112) log(1 /δ ))
{+/σ 
+} + 2 B/σ 2 holds for some absolute constants ˆ C 1 , ˆ C 2with the deﬁnition B := 16 σ 20 ( p + log(1 /δ )) /c
2 µ .
An intuitive explanation along with a rigorousproof are deferred to Appendix D due to the space limit.
The proposed LowGLM-UCB is shown in Algorithm 2, and its regret analysis is presented in Theorem C.1 in Appendix.
Notice that we can simply replace M t ( c µ ) by V t in Algorithm 2, and the regret bound would increaseat most up to a constant factor (Appendix G).
A potential drawback of Algorithm 2 is that in eachiteration we have to calculate ˆ θ t , which might be computationally expensive.
We could resolve thisproblem by only recomputing ˆ θ t whenever | M t ( c µ ) | increases signiﬁcantly, i.e. by a constant factor
C > 1 in scale.
And consequently we only need to solve the Eqn.
(10) for O (log( T 2 )) times up tothe horizon T 2 , which remarkably saves the computation.
Meanwhile, the bound of the regret wouldonly increase by a constant multiplier √ C .
We call this modiﬁed algorithm as PLowGLM-UCB withthe initial letter “P” standing for “Parsimonious”.
Its pseudo-code and regret analysis are given in
Appendix H.1.
Equipped with LowGLM-UCB in stage 2, we deduce the overall regret of G-ESTT:
4.2.1 Overall regret of G-ESTT
To quantify the performance of our algorithm, we ﬁrst deﬁne α xt ( · ) and β xt ( · ) as [Equation]
[Equation]
And the following Theorem 4.2 exhibits the overall regret bound for G-ESTT.
Theorem 4.2.
(Regret of G-ESTT)
Suppose we set T 1 (cid:16) (cid:112)
d 1 d 2 rT log((
d 1 + d 2 )
/δ ) /D
rr , and weinvoke LowGLM-UCB (or PLowGLM-UCB) in stage 2 with ρ t ( δ ) =
α t + T 1 ( δ/ 2) , p = d 1 d 2 , k =( d 1 + d 2 ) r − r 2 , λ ⊥ =
c µ S 20 T/ ( k log(1+ c µ S 20 T/ ( kλ 0 ))) , and the rotated arm sets X 0 and availableparameter space Θ 0 .
The overall regret of G-ESTT is, with probability at least 1 − δ ,
[Equation]
Note G-ESTT improves all existing regret bounds by a √ d factor based on Table 1 given that r (cid:28) d .
4.3 Stage 2 of G-ESTS
Although G-ESTT is more efﬁcient than all existing algorithms on our problem setting, it still needsto calculate the MLE in high dimensional space which might be increasingly formidable with largesizes of feature matrices.
Note this computational issue remains ubiquitous among most banditalgorithms on high dimensional problems with sparsity, not to mention these algorithms rely onmultiple unspeciﬁed hyperparameters.
Therefore, to handle this practical issue, we propose anotherfast and efﬁcient framework called G-ESTS in this section.
Inspired by the success of dimension reduction in machine learning [34], we propose G-ESTS asshown in Algorithm 3.
And we summarize the core idea of G-ESTS as: After rearranging thevectorization of the action set X (cid:48) and the unknown Θ (cid:48)∗ as we have shown in Eqn.
(6) and (7) for
G-ESTT, we can simply exclude, rather than penalize, the subspaces that are complementary to therows and columns of (cid:98) Θ .
In other words, we could remove the last p − k entries directly, i.e. Eqn.
(13)and (14).
Intriguingly, not only can we get a low-dimensional ( k ) generalized linear bandit problemin stage 2, where redundant dimensions are excluded and hence any state-of-the-art algorithms couldbe readily invoked.
But also we obtain the optimal regret bound of scale ˜ O
( dr √ T ) again as shownin Table 1.
Besides, G-ESTS requires exceedingly less computation compared with other existingalgorithms since consequently we get a low-dimensional bandit problem.
In summary, we believe our
G-ESTS is superior regarding both theoretical guarantee and computational efﬁciency.
4.3.1 Overall regret of G-ESTS
Theorem 4.3.
(Regret of G-ESTS)
Suppose we set T 1 (cid:16) (cid:112)
d 1 d 2 rT log((
d 1 + d 2 )
/δ ) /D
rr , and weinvoke any efﬁcient generalized linear bandit algorithm with regret bound ˜ O ( k √ T ) 2 in stage 2 withp =
d 1 d 2 , k = ( d 1 + d 2 ) r − r 2 , and the reduced arm sets X 0 ,sub and available parameter spaceΘ 0 ,sub .
The overall regret of G-ESTS is, with probability at least 1 − δ ,
[Equation]
According to Theorem 4.2 and 4.3, both G-ESTT and G-ESTS could achieve an improved ˜ O ( dr √ T )regret bound.
Surprisingly, this matches the lower bound of low-rank matrix bandits presented in [26]up to logarithm terms.
To the best of our knowledge, our methods are the ﬁrst two methods that attainthis optimal rate.
In addition, given the ﬁnite arm set, our G-ESTS framework can even achieve abetter regret bound of order ˜ O ( √ d 1 d 2 rT ) (see Appendix F.2 for detailed proof), and this result isbetter than the lower regret bound for the case of inﬁnite arms by a √ r factor.
In the following experiments, We will implement the SGD-TS algorithm [9] in stage 2 of G-ESTSsince SGD-TS could efﬁciently proceed with only O ( dT ) complexity for d − dimensional featuresover T rounds.
Therefore, the total computational complexity of stage 2 is at most O ( T 2 ( d 1 + d 2 ) r ) ,which is signiﬁcantly less than that of other methods for low-rank matrix bandits (e.g. LowESTR [26]).
And the total time complexity of G-ESTS would only scale O ( T 1 d 1 d 2 /(cid:15) 2 + T 2 ( d 1 + d 2 ) r ) where(cid:15) is the accuracy for subgradient methods in stage 1.
This fact also ﬁrmly validates the practicalsuperiority of our G-ESTS approach.
We naturally believe that this G-ESTS framework can be easilyimplemented in the linear setting as a special case of GLM, where in stage 2 one can utilize any linearbandit algorithm accordingly.
In addition, we can easily modify our approaches for the contextualsetting by merely transforming the action sets at each iteration with the same regret bound.
Moredetails with pseudo-codes for the contextual case are in Appendix H.3.
Experiments
In this section, we show by simulation experiments that our proposed G-ESTT (with LowGLM-UCB),
G-ESTS (with SGD-TS) outperform existing algorithms for the generalized low-rank matrix banditproblems.
Since we are the ﬁrst to propose a practical algorithm for this problem, currently there isno existing literature for comparison.
In order to validate the advantage of utilizing low-rank structureand generalized reward functions, we compare with the original SGD-TS after naïvely ﬂattening the
[Table]
d 1 by d 2 matrices without using the low-rank structure, and LowESTR [26], which works well forlinear low-rank matrix bandits.
We simulate a dataset with d 1 = d 2 = 10 (12) and r = 1 (2) : when r = 1 , we set the diagonalmatrix Θ ∗ as diag (Θ ∗ ) =
When r = 2 , we set Θ ∗ = v 1 v (cid:62) 1 + v 2 v (cid:62) 2 for two randomorthogonal vectors v 1 , v 2 with (cid:107) v 1 (cid:107) 2 = (cid:107) v 2 (cid:107) 2 = 3 .
For arms we draw 480 (1000)
random matricesfrom { X ∈ R
d
1 × d 2 : (cid:107) X (cid:107) F ≤ 1 } , and we build a logistic model where the payoff y t is drawn froma Bernoulli distribution with mean µ ( X (cid:62) t θ ∗ ) .
More details on the hyper-parameter tuning are in
Appendix I.
Each experiment is repeated 100 times for credibility and the average regret, along withstandard deviation, is displayed in Figure 1.
Note that our experiments are more comprehensive thanthose in [26].
And due to the expensive time complexity of UCB-based baselines (Table 2), it isformidable for us to increase d here.
From the plots, we observe that our algorithms G-ESTT and G-ESTS always achieve less regretcompared with LowESTR and SGD-TS in all four scenarios consistently.
Intriguingly, in the warm-upperiod SGD-TS incurs less regret compared with our methods due to the sacriﬁce of random samplingin stage 1, but our proposed framework quickly overtakes SGD-TS after utilizing the low-rankstructure as desired.
This phenomenon exactly coincides with our theory.
Notice that G-ESTT isslightly better than G-ESTS in the case for r = 2 especially in the very beginning of stage 2, and webelieve it is because that our G-ESTT could reutilize the actions in stage 1 and hence could yieldmore robust performance when switching to stage 2.
However, G-ESTS would gradually catch upwith G-ESTT in the long run as expected.
Besides, it costs G-ESTS extremely less running time thanother existing methods to update the decisions due to its dimensional reduction as shown in Table2.
We also observe that the cumulative regret of G-ESTS tends to become better eventually if weincrease T 1 decently.
(Further investigation and plots for 1000 arms are in Appendix I.) [Figure] (d
[Figure]
Conclusion
In this paper, we discussed the generalized linear low-rank matrix bandit problem.
We proposed twonovel and efﬁcient frameworks called G-ESTT and G-ESTS, and both methods could achieve animproved (cid:101)
O ( dr √ T ) bound of regret under some mild conditions.
There are several directions for our future work.
Firstly, although the GLM has achieved tremendoussuccess in various settings, recently some other models [39] that are proved to be powerful wereproposed, so we can study these modern frameworks in low-rank matrix case.
Secondly, it seemsmore reasonable to continuously update the subspace estimation via online SVD over T [17] on-theﬂy with a randomized strategy.
Therefore, we can consider something like arbitrarily choosing anadmissible subset of the action set at each round [31] without hurting the regret too much.