C ONTEXTUAL BANDITS WITH CONCAVE REWARDS , AND AN APPLICATION TO FAIR RANKING
Elvis Dohmatob,
Pirotta, Lazaric, {dohmatob,pirotta,lazaric,usunier}@meta.com
A BSTRACT
We consider C ontextual B andits with C oncave R ewards ( CBCR ), a multi-objective bandit problem where the desired trade-off between the rewards is deﬁned by a known concave objective function, and the reward vector depends on an observed stochastic context
We present the ﬁrst algorithm with provably vanishing regret for
CBCR without restrictions on the policy space, whereas prior works were restricted to ﬁnite policy spaces or tabular representations.
Our solution is based on a geometric interpretation of CBCR algorithms as optimization algorithms over the convex set of expected rewards spanned by all stochastic policies.
Building on Frank-Wolfe analyses in constrained convex optimization, we derive a novel reduction from the
CBCR regret to the regret of a scalar-reward bandit problem.
We illustrate how to apply the reduction off-the-shelf to obtain algorithms for CBCR with both linear and general reward functions, in the case of non-combinatorial actions.
Motivated by fairness in recommendation we describe a special case of CBCR with rankings and fairness aware objectives, leading to the ﬁrst algorithm with regret guarantees for contextual combinatorial bandits with fairness of exposure
I NTRODUCTION
Contextual bandits are a popular paradigm for online recommender systems that learn to generate personalized recommendations from user feedback.
These algorithms have been mostly developed to maximize a single scalar reward which measures recommendation performance for users.
Recent fairness concerns have shifted the focus towards item producers whom are also impacted by the exposure they receive (Biega et al., 2018; Geyik et al., 2019), leading to optimize trade-offs between recommendation performance for users and fairness of exposure for items (Singh & Joachims, 2019;
Zehlike & Castillo, 2020).
More generally, there is an increasing pressure to insist on the multiobjective nature of recommender systems (Vamplew et al., 2018; Stray et al., 2021), which need to optimize for several engagement metrics and account for multiple stakeholders’ interests (Mehrotra et al., 2020; Abdollahpouri et al., 2019).
In this paper, we focus on the problem of contextual bandits with multiple rewards, where the desired trade-off between the rewards is deﬁned by a known concave objective function, which we refer to as C ontextual B andits with C oncave R ewards ( CBCR ).
Concave rewards are particularly relevant to fair recommendation, where several objectives can be expressed as (known) concave functions of the (unknown) utilities of users and items (Do et al., 2021).
Our CBCR problem is an extension of B andits with C oncave R ewards ( BCR ) (Agrawal & Devanur, 2014) where the vector of multiple rewards depends on an observed stochastic context.
We address this extension because contexts are necessary to model the user/item features required for personalized recommendation.
Compared to BCR , the main challenge of CBCR is that optimal policies depend on the entire distribution of contexts and rewards.
In BCR , optimal policies are distributions over actions, and are found by direct optimization in policy space (Agrawal & Devanur, 2014; Berthet & Perchet, 2017).
In CBCR , stationary policies are mappings from a continuous context space to distributions over actions.
This makes existing BCR approaches inapplicable to CBCR because the policy space is not amenable to tractable optimization without further assumptions or restrictions.
As a matter of fact, the only prior theoretical work on CBCR is restricted to a ﬁnite policy set (Agrawal et al., 2016).
We present the ﬁrst algorithms with provably vanishing regret for CBCR without restriction on the policy space.
Our main theoretical result is a reduction where the CBCR regret of an algorithm is bounded by its regret on a proxy bandit task with single (scalar) reward.
This reduction shows that it is straightforward to turn any contextual (scalar reward) bandits into algorithms for CBCR .
We prove this reduction by ﬁrst re-parameterizing CBCR as an optimization problem in the space of feasible rewards, and then revealing connections between Frank-Wolfe (FW) optimization in reward space and a decision problem in action space.
This bypasses the challenges of optimization in policy space.
To illustrate how to apply the reduction, we provide two example algorithms for CBCR with noncombinatorial actions, one for linear rewards based on LinUCB (Abbasi-Yadkori et al., 2011), and one for general reward functions based on the SquareCB algorithm (Foster & Rakhlin, 2020) which uses online regression oracles.
In particular, we highlight that our reduction can be used together with any exploration/exploitation principle, while previous FW approaches to BCR relied exclusively on upper conﬁdence bounds (Agrawal & Devanur, 2014; Berthet & Perchet, 2017; Cheung, 2019).
Since fairness of exposure is our main motivation for CBCR , we show how our reduction also applies to the combinatorial task of fair ranking with contextual bandits, leading to the ﬁrst algorithm with regret guarantees for this problem, and we show it is computationally efﬁcient .
We compare the empirical performance of our algorithm to relevant baselines on a music recommendation task.
Related work.
Agrawal et al.
(2016) address a restriction of CBCR to a ﬁnite set of policies, where explicit search is possible.
Cheung (2019) use FW for reinforcement learning with concave rewards, a similar problem to CBCR .
However, they rely on a tabular setting where there are few enough policies to compute them explicitly.
Our approach is the only one to apply to CBCR without restriction on the policy space, by removing the need for explicit representation and search of optimal policies.
Our work is also related to fairness of exposure in bandits.
Most previous works on this topic either do not consider rankings (Celis et al., 2018; Wang et al., 2021; Patil et al., 2020; Chen et al., 2020), or apply to combinatorial bandits without contexts (Xu et al., 2021).
Both these restrictions are impractical for recommender systems.
Mansoury et al.
; Jeunen & Goethals (2021) propose heuristics with experimental support that apply to both ranking and contexts in this space, but they lack theoretical guarantees.
We present the ﬁrst algorithm with regret guarantees for fair ranking with contextual bandits.
We provide a more detailed discussion of the related work in Appendix A.
M AXIMIZATION OF CONCAVE REWARDS IN CONTEXTUAL BANDITS
Notation.
For any n 2 N , we denote by
J n
K = { 1 , . . . , n } .
The dot product of two vectors x and y in
R n is either denoted x | y or using braket notation h x | y i , depending on which one is more readable.
Setting.
We deﬁne a stochastic contextual bandit (Langford & Zhang, 2007) problem with D rewards.
At each time step t , the environment draws a context x t ⇠ P , where x 2 X ✓ R q and P is a probability measure over X .
The learner chooses an action a t 2 A where A ✓ R K is the action space, and receives a noisy multi-dimensional reward r t 2 R D , with expectation E
[ r t | x t , a t ] = µ ( x t ) a t , where µ : X !
R D ⇥ K is the matrix-value contextual expected reward function.
The trade-off between the D cumulative rewards is speciﬁed by a known concave function f : R D !
R [ {± 1 } .
Let A denote the convex hull of A and ⇡ : X !
A be a stationary policy, 2 then the optimal value for the problem is deﬁned as f ⇤ =
sup
[Equation]
We rely on either of the following assumptions on f :
Assumption A f is closed proper concave 3 on R D and A is a compact subset of R K .
Moreover, there is a compact convex set K ✓ R D such that
• (Bounded rewards) 8 ( x, a ) 2 X ⇥ A , µ ( x ) a 2 K and for all t 2 N ⇤ , r t 2 K with probability 1 .
• (Local Lipschitzness) f is L -Lipschitz continuous with respect to k .
k 2 on an open set containing K .
Assumption B Assumption A holds and f has C -Lipschitz-continuous gradients w.r.t.
k .
k 2 on K .
The most general version of our algorithm, described in Appendix D, removes the need for the smoothness assumption using smoothing techniques.
We describe an example in Section 3.3.
In the rest of the paper, we denote by D K = sup z,z2 K k z
z
k 2 the diameter of K , and use ˜ C
= C 2 D 2 K .
We now give two examples of this problem setting, motivated by real-world applications in recommender systems, and which satisfy Assumption A.
Example 1 (Optimizing multiple metrics in recommender systems.)
Mehrotra et al.
(2020) formalized the problem of optimizing D engagement metrics (e.g. clicks, streaming time) in a banditbased recommender system.
At each t , x t represents the current user’s features.
The system chooses one arm among K , represented by a vector a t in the canonical basis of R K which is the action space
A .
Each entry of the observed reward vector ( r t,i ) Di =1 corresponds to a metric’s value.
The trade-off between the metrics is deﬁned by the Generalized Gini Function: f ( z ) =
P
Di =1 w i
z
" i , where ( z " i ) Di = denotes the values of z sorted increasingly and w 2 R D is a vector of non-increasing weights.
Example 2 (Fairness of exposure in rankings.)
The goal is to balance the traditional objective of maximizing user satisfaction in recommender systems and the inequality of exposure between item producers (Singh & Joachims, 2018; Zehlike & Castillo, 2020).
For a recommendation task with m items to rank, this leads to a problem with D = m + 1 objectives, which correspond to the m items’ exposures, plus the user satisfaction metric.
The context x t 2 X ⇢ R md is a matrix where each x t,i 2 R d represents a feature vector of item i for the current user.
The action space A is combinatorial, i.e. it is the space of rankings represented by permutation matrices:
[Equation]
For a 2 A , a i,k = 1 if item i is at rank k .
Even though we use a double-index notation and call a a permutation matrix , we ﬂatten a as a vector of dimension K = m 2 for consistency of notation.
We now give a concrete example for f , which is concave as usual for objective functions in fairness of exposure (Do et al., 2021).
It is inspired by Morik et al.
(2020), who study trade-offs between average user utility and inequality 4 of item exposure:
[Equation]
[Figure]
where  > 0 is a trade-off parameter.
[Equation]
[Equation]
The learning problem.
In the bandit setting, P and µ are unknown and the learner can only interact online with the environment.
Let h T =
x t , a t , r t  t
J
T
K be the history of contexts, actions, and reward observed up to time T  1 and  0 > 0 be a conﬁdence level, then at step t a bandit algorithm
A receives in input the history h t , the current context x t , and it returns a distribution over actions A and selects an action a t ⇠ A ( h t , x t ,  0 ) .
The objective of the algorithm is to minimize the regret
[Equation]
Note that our setting subsumes classical stochastic contextual bandits: when D = 1 and f ( z ) = z , maximizing f (ˆ s T ) amounts to maximizing a cumulative scalar reward
P
Tt =1 r t .
In Lem.
9 (App. C.3), we show that alternative deﬁnitions of regret, with different choices of comparator or performance measure, would yield a difference of order O (1 / p T ) , and hence not substantially change our results.
A GENERAL REDUCTION - BASED APPROACH FOR CBCR
In this section we describe our general approach for CBCR .
We ﬁrst derive our key reduction from
CBCR to a speciﬁc scalar-reward bandit problem.
We then instantiate our algorithm to the case of linear and general reward functions for smooth objectives f .
Finally, we extend to the case of non-smooth objective functions using Moreau-Yosida regularization (Rockafellar & Wets, 2009).
3. R EDUCTION FROM CBCR TO SCALAR - REWARD CONTEXTUAL BANDITS
There are two challenges in the CBCR problem: 1) the computation of the optimal policy sup ⇡
[Equation]
even with known µ ; 2) the learning problem when µ is unknown.
1: Reparameterization of the optimization problem.
The ﬁrst challenge is that optimizing directly in policy space for the benchmark problem sup ⇡ : X !
A f
[Equation]
is intractable without any restriction, because the policy space includes all mappings from the continuous context space X to distributions over actions.
Our solution is to rewrite the optimization problem as a standard convex constrained problem by introducing the convex set S of feasible rewards:
[Equation]
Under Assumption A, S is a compact subset of K (see Lemma 7 in App.
C)
so f attains its maximum over S .
We have thus reduced the complex initial optimization problem to a concave optimization problem over a compact convex set.
2: Reducing the learning problem to scalar-reward bandits.
Unfortunately, since P and µ are unknown, the set S is unknown.
This precludes the possibility of directly using standard constrained optimization techniques, including gradient descent with projections onto S .
We consider FrankWolfe, a projection-free optimization method robust to approximate gradients (Lacoste-Julien et al., 2013; Kerdreux et al., 2018).
At each iteration t of FW, the update direction is given by the linear subproblem: argmax s 2 S hr f ( z t  1 ) | s i , where z t  1 is the current iterate.
Our main technical tool, Lemma 1, allows to connect the FW subproblem in the unknown reward space
[Equation] to a workable decision problem in the action space (see Lemma 13 in Appendix E for a proof):
Lemma 1 Let E t ⇥ .
⇤ be the expectation conditional on h t .
Let z t 2 K be a function of contexts, actions and rewards up to time t .
Under Assumption A, we have:
[Equation]
For all  2 (0 , 1] , with probability at least 1   , we have:
[Equation]
Lemma 1 shows that FW for CBCR operates closely to a sequence of decision problems of the form (max a 2 A hr f
( z t  1 ) | µ ( x t ) a i ) Tt =1 .
However, we have yet to address the problem that P and µ are unknown.
To solve this issue, we introduce a reduction to scalar-reward contextual bandits .
We can notice that solving for the sequence of actions maximizing
[Equation] ) | µ
[Equation]
x t ) a i corresponds to solving a contextual bandit problem with adversarial contexts and stochastic rewards.
Formally, using z t = ˆ s t 5 , we deﬁne the extended context ˜ x t =
( r f (ˆ s t  1 ) , x t ) , the average scalar reward ˜ µ (˜ x t ) =
r f (ˆ s t  1 ) | µ ( x t ) and the observed scalar reward ˜ r t = hr f
(ˆ s t  1 ) | r t i .
This fully deﬁnes a contextual bandit problem with scalar reward .
Then, the objective of the algorithm is to minimize the following scalar regret :
[Equation]
In this framework, the only information observed by the learning algorithm is ˜ h t :=
˜ x t 0 , a t 0 , ˜ r t t
J t
K .
This regret minimization problem has been extensively studied (see e.g.,
Slivkins, 2019, Chap. 8 for an overview).
The following key reduction result 6 relates R scal T to
R T , the regret of the original CBCR problem:
Theorem 2 Under Assmpt.
B, for every T 2 N ⇤ and  > 0 , algorithm A satisﬁes, with prob.
1   :
[Equation]
The reduction shown in Thm.
2 hints us at how to use or adapt scalar bandit algorithms for CBCR .
In particular, any algorithm with sublinear regret will lead to a vanishing regret for CBCR .
Since the worst-case regret of contextual bandits is ⌦ ( p T ) (Dani et al., 2008), we obtain near minimax optimal algorithms for CBCR .
We illustrate this with two algorithms derived from our reduction in Sec. 3.2.
Proof sketch of Theorem 2: CBCR and Frank-Wolfe algorithms (full proof in Appendix E).
Although the set S is not known, the standard telescoping sum argument for the analysis of
Frank-Wolfe algorithms (see Lemma 14 in Appendix E, and e.g., (Berthet & Perchet, 2017, Lemma 12) for similar derivations) gives that under Assumption B, denoting g t =
r f (ˆ s t  1 ) :
[Equation]
The result is true for every sequence ( r t ) t
[Equation]
K T , and only tracks the trajectory of ˆ s t in reward space.
We introduce now the reference of the scalar regret:
[Equation]
Lemma 1 bounds the leftmost term, from which Theorem 2 immediately follows using (8).
3. P RACTICAL APPLICATION : T WO ALGORITHMS FOR MULTI - ARMED CBCR
To illustrate the effectiveness of the reduction from CBCR to scalar-reward bandits, we focus on the case where the action space
A is the canonical basis of R K (as in Example 1).
We ﬁrst study the case of linear rewards.
Then, for general reward functions, we introduce the FW-SquareCB algorithm, the ﬁrst example of a FW-based approach combined with an exploration principle other than optimism.
This shows our approach has a much broader applicability to solve ( C ) BCR than previous strategies.
From LinUCB to FW-LinUCB (details in Appendix G).
We consider a CBCR with linear reward function, i.e., µ ( x ) =
✓ x where ✓ 2 R D ⇥ d (recall we have D rewards) and x 2 R d ⇥ K , where d is the number of features.
Let ˜ ✓ := ﬂatten( ✓ ) and g t =
r f (ˆ s t  1 ) .
Using [ . ; . ] to denote the vertical concatenation of matrices, the expected reward for action a in context x at time t can be written h g t | µ ( x ) a i = g | t ✓ xa =
h ˜
✓ | ˜ x t a i where ˜ x t 2 R Dd ⇥ K is the extended context with entries ˜ x t =
[ g t, 0 x t ; . .
; g t,D x t ] 2 R Dd ⇥ K .
This is an instance of a linear bandit problem, where at each time t , action a is associated to the vector ˜ x t a and its expected reward is h ˜ ✓ | ˜ x t a i .
As a result, we can immediately derive a LinUCB-based algorithm for linear CBCR by leveraging the equivalence FW-LinUCB ( h t , x t ,  0 ) =
LinUCB (˜ h t , ˜ x t ,  0 ) .
LinUCB’s regret guarantees imply
R scal T
=
O
( d p T ) with high probability, which, in turn give a O (1 / p T ) for R T .
From SquareCB to FW-SquareCB (details in Appendix H).
We now consider a CBCR with general reward function µ ( x ) .
The SquareCB algorithm (Foster & Rakhlin, 2020) is a randomized exploration strategy that delegates the learning of rewards to an arbitrary online regression algorithm.
The scalar regret of SquareCB is bounded depending on the regret of the base regression algorithm.
For FW-SquareCB, we have access to an online regression oracle ˆ µ t , an estimate of µ which is a function of h t , which has regression regret bounded by R oracle ( T ) .
The exploration strategy of FWSquareCB follows the same principles as SquareCB: let g t =
r
f
(ˆ s t  1 ) and denote ˆ µ t = g | t ˆ µ t ( x t ) , so that ˆ µ | t a = h g t | ˆ µ t ( x t ) a i .
Let A t = FW-SquareCB ( h t , x t ,  0 ) deﬁned as
[Equation]
where a t 2 argmax a 2 A ˆ µ | t a and ˆ µ ⇤ t
= ˆ µ t a t
Then FW-SquareCB has R T in O ( p
R oracle ( T ) / p T ) with high probability.
[Table]
3. T HE CASE OF NONSMOOTH f
When f is nonsmooth
[Table] we use a smoothing technique where the scalar regret is not measured using r f (ˆ s t  1 ) [Table] but rather using gradients of a sequence ( f t ) t 2 N of smooth approximations of f , whose smoothness decrease over time (see e
[Table]
g., Lan, 2013, for applications of smoothing to FW).
We provide a comprehensive treatment of smoothing in our general approach described in Appendix D [Table] while speciﬁc smoothing techniques are discussed in Appendix F [Table]
We now describe the use of Moreau-Yosida regularization (Rockafellar & Wets, 2009, Def. 1.22):
[Equation]
⌘ .
It is well-known that f t is concave and L -Lipschitz whenever f is, and f t is p t +1  0 -smooth
(see Lemma 15 in Appendix F).
A related smoothing method was used by Agrawal & Devanur (2014) for (non-contextual) BCR .
Our treatment of smoothing is more systematic than theirs, since we use a smoothing factor  0 / p t + 1 that decreases over time rather than a ﬁxed smoothing factor that depends on a pre-speciﬁed horizon.
Our regret bound for
CBCR is based on a scalar regret R scal , sm T where r f t  1 (ˆ s t  1 ) is used instead of r f (ˆ s t  1 ) :
[Equation]
scal , sm T
[Equation]
Theorem 3 Under Assumptions A, for every z 0 2 K , every T  1 and every  > 0 ,  0 > 0 ,
Algorithm A satisﬁes, with probability at least 1     0 :
[Equation]
The proof is given in Appendix F. Taking
= D K L leads to a simpler bound where D K L  0 +3 L  0 D K = 4 .
C ONTEXTUAL RANKING BANDITS WITH FAIRNESS OF EXPOSURE
In this section, we apply our reduction to the combinatorial bandit task of fair ranking, and obtain the ﬁrst algorithm with regret guarantees in the contextual setting.
This task is described in Example(Sec.
We remind that there is a ﬁxed set of m items to rank at each timestep t , and that actions are ﬂattened permutation matrices ( A is deﬁned in Ex. 2, Eq.
The context x t ⇠ P is a matrix x t = ( x t,i )
i
J m
K where each x t,i 2 R d represents a feature vector of item i for the current user.
Observation model.
The user utility u ( x t ) is given by a position-based model with position weights b ( x t ) 2 [0 , 1] m and expected value for each item v ( x t ) 2 [0 , 1] m .
Denoting u ( x t ) the ﬂattened version of v ( x t ) b
[Equation]
x t ) | 2 R m ⇥ m , the user utility is (Lagrée et al., 2016; Singh & Joachims, 2018):
[Equation]
In this model, b k ( x t ) 2 [0 , 1] is the probability that the user observes the item at rank k .
The quantity
P m k
=1 a
i,k b k ( x t ) is thus the probability that the user observes item i given ranking a .
We denote input :  0 > 0 ,  > 0 , ˆ s 0 2 K V
=  I d , y
=
d , ˆ ✓ 0 =
d 1 for t = 1 , . . .
do 2 Observe context x t ⇠ P 3 8 i, ˆ v t,i ˆ ✓ | t  1 x t,i + ↵ t   0 k x t,i k V  1 t // UCB on v i ( x t ) (def. of ↵ t in Lem.
26, App. I)
4 a t top- k { @ f @
z m +
(ˆ s t  1 )ˆ v t,i + @ f @
z i (ˆ s t  1 ) } mi = //
FW linear optimization step 5 Observe exposed items e t 2 { 0 , 1 } m and user feedback c t 2 { 0 , 1 } m 6 Update ˆ s t ˆ s t  1 + 1 t ( r t  ˆ s t  1 ) 7 V t
V t  1 + m
X i =
e t,i x t,i x | t,i , y t
y t  1 + m
X i =
c t,i x t,i and ˆ ✓ t
V  1 t y t // regression 8 end k = max x 2 X k b ( x ) k 0  m the maximum rank that can be exposed to any user.
In most practical applications, k ⌧ m .
As formalized in Assumption D below, the position weights b k ( x ) are always non-increasing with k since the user browses the recommended items in order of their rank.
We use a linear assumption for item values, where D X and D ✓ are known constants:
Assumption
[Equation]
We propose an observation model where values v i ( x ) and position weights b ( x ) are unknown .
However, we assume that at each time step t , after computing the ranking a t , we have two types of feedback: ﬁrst, e t,i 2 { 0 , 1 } is 1 if item i has been exposed to the user, and 0 otherwise.
Second c t,i 2 { 0 , 1 } which represents a binary like / dislike feedback from the user.
We have
[Equation]
This observation model captures well applications such as newsfeed ranking on mobile devices or dating applications where only one post/proﬁle is shown at a time.
What we gain with this model is that b ( x ) can depend arbitrarily on the context x , while previous work on bandits in the position-based model assumes b known and context-independent (Lagrée et al., 2016).
Fairness of exposure.
There are D = m + 1 rewards, i.e., µ ( x ) 2 R ( m +1) ⇥ m 2 .
Denoting µ i ( x ) the i th-row of µ ( x ) , seen as a column vector, each of the m ﬁrst rewards is the exposure of a speciﬁc item, while the m + 1 -th reward is the user utility:
[Equation]
and µ
[Equation]
The observed reward vector r t
R D is deﬁned by 8 i
J m
K , r t,i =
e t,i and r t,m +1 =
P mi =1 c t,i .
Notice that E ⇥ [Equation]  [Equation]
= [Equation]
Let K be the convex hull of { z 2 { 0 , 1 } m +1 :
P mi =1 z i  k and z m +1 
P mi =
z
i } , we have D K  p k p k + 2  k + 1 and r t 2 K with probability 1 .
The objective function f : R D !
R makes a trade-off between average user utility and inequalities in item exposure (we gave an example in Eq.
The remaining assumptions of our framework are that the objective function is non-decreasing with respect to average user utility.
This is not required but it is natural (see Example 2) and slightly simpliﬁes the algorithm.
Assumption D
The assumptions of the framework described above hold, as well as Assumption B.
Moreover, 8 z 2 K @ f @
z m +1 ( z ) > 0 , and 8 x 2 X , 1  b 1 ( x )  . . .
b k ( x ) = . . .
=
b m ( x ) = 0 .
Algorithm and results.
We present the algorithm in the setting of linear contextual bandits, using
LinUCB (Abbasi-Yadkori et al., 2011; Li et al., 2010) as scalar exploration/exploitation algorithm in
Algorithm 1.
It builds reward estimates based on Ridge regression with regularization parameter  .
As in the previous section, we focus on the case where f is smooth but the extension to nonsmooth f is straightforward, as described in Section 3.
Appendix I provides the analysis for the general case.
As noted by Do et al.
(2021), Frank-Wolfe algorithms are particularly suited for fair ranking in the position-based model.
This is illustrated by line 4 of Alg. 1, where for ˜ u
2 R m , top- k
(˜ u ) outputs a permutation (matrix) of
J m
K that sorts the top- k elements of ˜ u. Alg. 1 is thus computationally fast , with a cost dominated by the top- k sort. It also has an intuitive interpretation as giving items an adaptive bonus depending on r f (e.g., boosting the scores of items which received low exposure in previous steps).
The following result is a consequence of (Do et al., 2021, Theorem 1):
Proposition 4 Let t 2 N ⇤ and ˆ µ t such that 8 i
J m
K , ˆ µ t,i = µ i ( x t ) and ˆ µ t,m +
=
ˆ v t b ( x t ) | viewed as a column vector, with ˆ v deﬁned in line 3 of Algorithm 1.
Then, under Assumption D, a t deﬁned on line 4 of Algorithm 1 satisﬁes: hr f (ˆ s t
[Equation] ) |
[Equation]
The proposition says that even though computing a t as in line 4 of Alg. 1 does not require the knowledge of b ( x t ) , we still obtain the optimal update direction according to ˆ µ t .
Together with the usage of the observed reward r t in FW iterates (instead of e.g., ˆ µ t a t as would be done by Agrawal &
Devanur (2014)), this removes the need for explicit estimates of µ ( x t ) .
This is how our algorithm works without knowing the position weights b ( x t ) , which are then allowed to depend on the context.
The usage of ˆ v t to compute a t follows the usual conﬁdence-based approach to explore/exploitation principles for linear bandits, which leads to the following result (proven in Appendix I):
Theorem 5 Under Assumptions B, C and D, for every  0 > 0 , every T 2 N ⇤ , every   D 2 X k , with probability at least 1   0 , Algorithm 1 has scalar regret bounded by
[Equation]
Thus, considering only d, T, k and  =  0 Alg. 1 has regret R T  O
[Equation]
E XPERIMENTS
We present two experimental evaluations of our approach, which are fully detailed in App.
B.
5. M ULTI - ARMED CBCR : A PPLICATION TO MULTI - OBJECTIVE BANDITS
We ﬁrst focus on the multi-objective recommendation task of Example 1 where f ( z ) =
P
Di =1 w i z
" i .
Algorithms.
We evaluate our two instantiations presented in Sec.
3.2 with the Moreau-Yosida smoothing technique of Sec.
3.3: (i) FW-SquareCB with Ridge regression and (ii) FW-LinUCB, where exploration is controlled by a scaling variable ✏ on the exploration bonus of each arm.
We compare them to MOLinCB from (Mehrotra et al., 2020).
Environments.
We reproduce the synthetic environments of Mehrotra et al.
(2020), where the context and reward parameters are generated randomly, and
w i = 1 2 i  1 .
We set K = 50 and D 2 { 5 , 20 } (we also vary K in App. B).
Each simulation is repeated with 100 random seeds.
Results.
Following (Mehrotra et al., 2020), we evaluate the algorithms’ performance by measuring the value of f ( 1 T
P
Tt =1 µ ( x t ) a t ) over time.
Our results are shown in Figure 1 (left).
We observe that our algorithm FW-SquareCB obtains comparable performance with the baseline MOLinCB.
These algorithms converge after ⇡ 100 rounds.
In this environment from (Mehrotra et al., 2020), only little exploration is needed, hence FW-LinUCB obtains better performance when ✏ is smaller ( ✏ = 0 . 01) .
The advantage of using an FW instantiation for the multi-objective bandit optimization task is that unlike MOLinCB, its convergence is also supported by our theoretical regret guarantees.
5. R ANKING CBCR : A PPLICATION TO FAIRNESS OF EXPOSURE IN RANKINGS
We now tackle the ranking problem of Section 4.
We show how FW-LinUCBRank allows to fairly distribute exposure among items on a music recommendation task with bandit user feedback.
Environment.
Following (Patro et al., 2020), we use the Last.fm music dataset from (Cantador et al., 2011), from which we extract the top 50 users and items with the most listening counts.
We use a protocol similar to Li et al.
(2016) to generate context and rewards from those.
We use ¯ k =ranking slots, and exposure weights b k ( x ) =
log(2)1+log( k ) .
Simulations are repeated with 10 seeds.
Algorithms.
Our algorithm is FW-LinUCBRank with the nonsmooth objective f of Eq.
(2), which trades off between user utility and item inequality.
We study other fairness objectives in App.
B. Our ﬁrst baseline is LinUCBRank (Ermis et al., 2020), designed for ranking without fairness.
Then, we study two baselines with amortized fairness of exposure criteria.
Mansoury et al.
(2021) proposed a fairness module for UCB-based ranking algorithms, which we plug into LinUCBRank .
We refer to this baseline as Unbiased-LinUCBRank .
Finally, the FairLearn ( c, ↵ ) algorithm (Patil et al., 2020) enforces as fairness constraint that the pulling frequency of each arm be  c , up to a tolerance ↵ .
We implement as third baseline a simple adaptation of FairLearn to contextual bandits and ranking.
Dynamics.
Figure 1 (middle) represents the values of f over time achieved by the competing algorithms, for ﬁxed  = 1 .
As expected, compared to the fairness-aware and -unaware baselines, our algorithm FW-LinUCBRank reaches the best values of f .
Interestingly, Unbiased-LinUCBRank also obtains high values of f on the ﬁrst 10 4 rounds, but its performance starts decreasing after more iterations.
This is because Unbiased-LinUCBRank is not guaranteed to converge to an optimal trade-off between user fairness and item inequality.
At convergence.
We analyse the trade-offs achieved after 5 · 10 6 rounds between user utility and item inequality measured by the Gini index.
We vary  in the objective f of Eq.
(2) for FW-LinUCBRank and the strength c in FairLearn ( c, ↵ ) , with tolerance ↵ = 1 .
In Fig.
1 (right), we observe that compared to FairLearn , FW-LinUCBRank converges to much higher user utility at all levels of inequality among items.
In particular, it achieves zero-unfairness at little cost for user utility.
C ONCLUSION
We presented the ﬁrst general approach to contextual bandits with concave rewards.
To illustrate the usefulness of the approach, we show that our results extend randomized exploration with generic online regression oracles to the concave rewards setting, and extend existing ranking bandit algorithms to fairness-aware objective functions.
The strength of our reduction is that it can produce algorithms for CBCR from any contextual bandit algorithm, including recent extensions of SquareCB to inﬁnite compact action spaces (Zhu & Mineiro, 2022; Zhu et al., 2022) and future ones.
In our main application to fair ranking, the designer sets a fairness trade-off f to optimize.
An interesting open problem is the integration of recent elicitation methods for f (e.g., Lin et al., 2022) in the bandit setting.
Another interesting issue is the generalization of our framework to include constraints (Agrawal & Devanur, 2016).
Finally, we note that the deployment of our algorithms requires to carefully design the whole machine learning setup, including the speciﬁcation of reward functions (Stray et al., 2021), the design of online experiments (Bird et al., 2016), while taking feedback loops into account (Bottou et al., 2013; Jiang et al., 2019; Dean & Morgenstern, 2022). A CKNOWLEDGMENTS
The authors would like to thank Clément Vignac, Marc Jourdan, Yaron Lipman, Levent Sagun and the anonymous reviewers for their helpful comments