Group Meritocratic Fairness in Linear Contextual Bandits
Anonymous Author(s)
AffiliationAddress email
Abstract
We study the linear contextual bandit problem where an agent has to select onecandidate from a pool and each candidate belongs to a sensitive group.
In thissetting, candidates’ rewards may not be directly comparable between groups, forexample when the agent is an employer hiring candidates from different ethnicgroups and some groups have a lower reward due to discriminatory bias and/orsocial injustice.
We propose a notion of fairness that states that the agent’s policyis fair when it selects a candidate with highest relative rank, which measures howgood the reward is when compared to candidates from the same group.
This is avery strong notion of fairness, since the relative rank is not directly observed bythe agent and depends on the underlying reward model and on the distribution ofrewards.
Thus we study the problem of learning a policy which approximates a fairpolicy under the condition that the contexts are independent between groups andthe distribution of rewards of each group is absolutely continuous.
In particular, wedesign a greedy policy which at each round constructs a ridge regression estimatorfrom the observed context-reward pairs, and then computes an estimate of therelative rank of each candidate using the empirical cumulative distribution function.
We prove that the greedy policy achieves, after T rounds, up to log factors andwith high probability, a fair pseudo-regret of order √ dT , where d is the dimensionof the context vectors.
The policy also satisfies demographic parity at each roundwhen averaged over all possible information available before the selection.
Wefinally show with a proof of concept simulation that our policy achieves sub-linearfair pseudo-regret also in practice.
Introduction
Consider a sequential decision making problem where at each time-step an employer has to selectone candidate from a pool to hire for a job.
The employer does not know how a candidate willperform if hired, but she can learn it over time by measuring the performance of previously selectedsimilar candidates.
This scenario can be formalized as a (linear) contextual bandit problem (see[2, 12, 26] and references therein), where each candidate is represented by a context vector and afterthe employer (or agent) chooses a candidate, it receives a reward, i.e. a scalar value measuring thetrue performance of the candidate, which depends (linearly) on the context.
In the above framework, the typical objective is to find a policy for the employer to select candidateswith the highest rewards [1–3, 26].
However, in some important scenarios this objective may notbe appropriate; if candidates belong to different sensitive groups (e.g. based on ethnicity, gender,etc.)
the resulting policy might discriminate or even exclude some groups completely in the selection process.
This may happen when some groups have lower expected reward than others, e.g. becausethey acquired less skills due lower financial support.
Another example arises when each candidate inthe pool, if selected, will perform a different kind of job, and the associated reward is job-specific.
For instance, if the employer is a university and each candidate is a researcher in a different discipline,then the rewards associated to different disciplines will be substantially different and be incomparable,e.g. citations counts greatly vary among different subjects; see [23] for a discussion.
In both of theabove scenarios, it is unfair to directly compare rewards of candidates belonging to different groups.
A simple way to deal with this issue would be to select the candidate to hire uniformly at random.
This policy satisfies a notion of fairness called demographic parity (see [7, 31] and references therein),which requires the probability of selecting a candidate from a certain group to be equal for all groups.
However, as it is apparent, this approach completely ignores the employer’s goal of selecting goodcandidates and is also unfair to candidates who spent effort acquiring credentials for the job.
In thiswork, we provide a fair way of comparing candidates from different groups via the relative rank , thatis the rank (or quantile) of the reward w.r.t.
the rewards distribution of the candidate’s group.
We calla policy group meritocratic fair (GMF) if it always selects a candidate with the highest relative rank .
Such a policy is meritocratic but only in terms of the within-group performance.
A closely relatedidea has been introduced in [23] for settings where the candidates’ rewards are available before theselection, while we are not aware of a similar notion in the multi-armed bandits literature.
A GMF policy requires the knowledge of the relative rank of each candidate which is not directlyobserved by the agent and depends on the underlying reward model and on the distribution of rewards.
Moreover, to estimate the relative rank from the observed rewards and contexts it is necessary tolearn the cumulative distribution function (CDF) of the rewards of each group, which introduces anadditional challenge to the standard linear contextual bandit framework where only the linear relationbetween contexts and rewards has to be learned.
Due to necessity of exploration, a learned policycannot be GMF at all rounds, thus we study the problem of learning a policy which minimizes thefair regret , that is the cumulative difference between the relative rank of the candidate chosen by a
GMF policy and the candidate chosen by our policy.
For this purpose, we design a greedy policy, which at each round uses the following two-stagestrategy.
Firstly, it constructs a ridge regression estimator which maps contexts to rewards.
Secondly,it computes an estimate of the relative rank of each candidate using the empirical CDF of the estimatedrewards.
We show that the proposed policy achieves, under some reasonable assumptions and after Ttime-steps,
˜ O
( K 3 + √ dT ) fair regret with high probability, where d is the dimension of the contextvectors and K is the number of candidates in the pool.
The policy also satisfies demographic parityat each time-step when averaged over all possible random draws of the available information.
Contributions and Organization.
After a review of previous work in Sec. 2, we proceed to introducethe learning problem and the proposed fairness notion in Sec. 3.
We then propose a learning policyto solve the problem in Sec. 4.
Notably, the policy jointly learns the underlying regression modeland the cumulative distribution function of each group.
Finally we derive a ˜ O ( √ dT ) regret boundin Sec. 5.
Sec.
6 presents an illustrative simulation experiment with diverse reward distributions,highlighting the efficiency of our algorithm.
Conclusions are drawn in Sec. 7.
Notation.
We use ⟨· , ·⟩ for the scalar product.
For K ∈ N we have [ K ] = { 1 , . . .
, K } .
Let ψ bea scalar random variable, for each a ∈ [ K ] and µ ∗ ∈ R d , we denote with F ψ , F a the CDF of ψand ⟨ µ ∗ , X a ⟩ respectively.
If X is a continuous random vector with values in R d , we denote withf X : R d → [0 , ∞ ) its probability density function.
For any positive integer s , we denote I s as thes × s identity matrix.
For a random variable Y ∈ R s , we call Y an absolutely continuous randomvariable,
Related Works
In recent years algorithmic fairness has received a lot of attention, becoming a large area of machinelearning research.
The potential for learning algorithms to amplify pre-existing bias and cause harmto human beings has triggered researchers to study solutions to mitigate or remove unfairness ofthe learned predictor, see [4, 8, 11, 14, 16, 19, 20, 22, 24, 25, 29, 35–37, 39] and references therein.
Fairness in sequential decision problems (see [38] for a survey) is usually divided into two categories:group fairness (GF) and individual fairness.
We give an overview of these notions below.
GF requires some statistical measure to be (approximately) equal across different sensitive groups.
Aprominent example relevant to this work is demographic parity , which requires that the probabilitythat the policy selects a candidate from a certain group should be the same for all groups.
A similarnotion is used by [10, 32], where the probability that the policy selects a candidate has to alwaysbe greater than a given threshold for all candidates.
[27] impose a weaker requirement concerningthe expected fraction of candidates selected from each group.
Other examples of GF in sequentialdecision problems are equal opportunity [5] and equalized odds [6].
Under some assumptions on thedistributions of the contexts, the GMF policy and greedy policy that we propose satisfy variants ofdemographic parity at each round.
Individual fairness can be divided in two categories: fairness through awareness (FA) [28, 34] andmeritocratic fairness (MF) [21, 22].
FA is based on the idea that similar individuals should be treatedsimilarly and is designed to avoid “winner takes all” scenarios where some individuals cannot beselected when they have a lower reward than others in the pool, even if the difference between rewardsis very small.
For example, [34] propose a policy where the probability of selecting a context overanother is lower when the context has a lower reward, but is never zero.
MF instead requires thatless qualified individual should not be favored over more qualified ones, which could happen duringthe learning process.
For example [22] proposes an algorithm based on confidence intervals, whereif the uncertainty between the best arms is too high the arm is selected uniformly at random.
Thisguarantees meritocratic fairness at each round but comes at a cost in terms of regret.
Our definition of fairness falls between group and meritocratic fairness.
It is meritocratic becauseit states that a candidate with a worse relative rank than another should never be selected.
It is alsobased on groups since the relative ranks directly depend on the distribution of rewards of each group.
A similar idea of fairness based on relative rank has been introduced in [23], which study the problemof selecting candidates from different groups based on their scalar-valued score when the scoresbetween groups are incomparable (e.g. number of citations in different research areas).
Contraryto our work, where the (noisy) rewards are observed only for the selected candidates, in [23] thenoiseless scores for all candidates can be accessed before the selection.
This difference makes theestimation of the relative rank simpler as the rewards CDFs can be estimated more efficiently.
Group Meritocratic Fairness in Linear Contextual Bandits
We consider the linear contextual bandit setup [2] where at each round t ∈ [ T ] , an agent receives aset of feature vectors { X t,a } Ka =1 with X t,a ⊂ R d , sampled from the environment, one for each arma ∈
[ K ] .
We assume that context (or individual) X t,a has an associated reward ⟨ µ ∗ , X t,a ⟩ whereµ ∗ is a vector in R d unknown to the agent.
After the agent selects the arm a t , it receives a noisyreward associated to X t,a t equal to r t,a t = ⟨ µ ∗ , X t,a t ⟩ + η t , where η t is some scalar noise, whichwe will formally specify later.
In addition, we assume that each arm represents a sensitive group (e.g.ethnicity, gender, etc.).
Usually, the goal of the agent is to maximise the expected cumulative reward (cid:80)
Tt =1 ⟨ µ ∗ , X t,a t ⟩ .
Sinceas we previously explained, this objective might be unfair to some of the sensitive groups, we insteaduse a different kind of reward which measures the relative performance of an individual comparedto others of the same arm/group.
First, we additionally assume, for each arm a , that { X t,a } T t =1 arei.i.d and have the same distribution of X a , which we define to be a random variable with unknowndistribution.
We call the distribution of ⟨ µ ∗ , X a ⟩ the reward distribution of arm a and denote with F aits CDF, i.e. F a ( r ) =
P ( ⟨ µ ∗ , X a ⟩ ≤ r ) for every r ∈ R .
Then, we introduce the relative rank ofcandidate X t,a as F a ( ⟨ µ ∗ , X t,a ⟩ ) , that is the probability that a sample from the reward distribution of arm a is lower than the reward of X t,a .
We argue that the relative rank, allows to have a fair wayof comparing individuals from different groups and introduce the following fairness definition.
Definition 3.1 (Group Meritocratic Fairness) .
A policy { a ∗ t } ∞ t =1 is group meritocratic fair (GMF) iffor all t ∈ N , a ∈ [ K ] it satisfies
[Equation]
A GMF policy chooses individuals with the highest reward compared to individuals from the samegroup.
This is a strong definition of fairness which is impossible to satisfy at each round for a learnedpolicy.
As in standard linear contextual bandits, µ ∗ is unknown and must be learned.
In this settinghowever, we have the additional challenge of learning the CDF for the rewards of each arm, F a .
Thus, we will focus on how to learn a GMF policy by introducing the following regret definition.
Definition 3.2 (Fair Pseudo-Regret) .
Let T ∈ N , { a t } Tt =1 be the evaluated policy and { a ∗ t } Tt =1 be a
GMF policy.
Then we denote by (cumulative) fair pseudo-regret the quantity
[Equation]
The goal of the learned policy will be to minimize the fair pseudo-regret, since a policy with sublinearfair pseudo-regret will get closer and closer to a GMF fair policy over time.
Remark 3.1.
Note that the fair pseudo-regret resembles the standard pseudo-regret defined as
R ( T ) :=
T
[Equation]
where rewards are replaced by relative ranks and a opt t by the GMF policy a ∗ t .
Furthermore, sincethe CDF restricted to the support is strictly increasing, when the reward distributions are the samefor each arm, i.e. F a = F a ′ for all a , a ′ ∈ [ K ] , then a policy minimizing the fair pseudo-regret alsominimizes the standard pseudo-regret and vice versa.
This is not true in the general case, wherefair and standard pseudo-regrets are often competing objectives.
For example, when { X a } Ka =1 aremutually independent, ⟨ µ ∗ , X a ⟩ is absolutely continuous for every a ∈ [ K ] and there exists ˆ a suchthat ⟨ µ ∗ , X ˆ a ⟩ > ⟨ µ ∗ , X a ⟩ for every a ̸
= ˆ a , then for every t , a opt t = ˆ a , while as we will show in
Proposition 3.1, a ∗ t selects each arm with equal probability.
Thus, with non-zero probability a opt t hasa linear fair pseudo-regret while a ∗ t has a linear standard pseudo-regret.
Moreover, in Appendix F,for K = 2 , we show that if ⟨ µ ∗ , X 1 ⟩ and ⟨ µ ∗ , X 2 ⟩ are independent, absolutely continuous, but notidentically distributed, then the GMF policy has a linear standard regret.
Learning a GMF policy brings several challenges.
The relative rank is not directly observed by theagent, which receives instead only the noisy reward.
This implies that the agent has to estimate F a , which in general might not even be Lipschitz continuous.
This is the main reason why werestrict our analysis to the case where the rewards {⟨ µ ∗ , X a ⟩} Ka =1 are independent and absolutelycontinuous.
In particular, let, for any t ≥ 0 , H − t := ∪ ti =1 (cid:8) { X i,a } Ka =1 , r i,a i , a i (cid:9) with H − 0 = ∅ and
H t := H − t ∪ {{ X t +1 ,a } Ka =1 } be respectively the the history and the information available for thedecision at round t + 1 , then the following holds.
Proposition 3.1 (GMF policy satisfies history-agnostic demographic parity ) .
Let {⟨ µ ∗ , X a ⟩} Ka =1 beindependent and absolutely continuous and for every a ∈ [ K ] , t ∈ N , let X t,a be an i.i.d.
copy of X a .
Then for every t ∈ N , {F a ( ⟨ µ ∗ , X t,a ⟩ ) } Ka =1 are i.i.d.
uniform on [0 , 1] and
P ( a ∗ t = a | H − t − 1 ) =
[Equation]
K ∀ a ∈ [ K ] , (2) for any GMF policy { a ∗ t } ∞ t =1 .
Note, the randomness lies exclusively in the current contexts { X t,a } Ka =1 .
Proof.
Let ψ a :=
F a ( ⟨ µ ∗ , X t,a ⟩ ) .
From the assumptions { ψ a } Ka =1 are i.i.d random variables,independent from H − t − 1 , with uniform distribution on [0 , 1] (see [9, Theorem 2.1.10]).
Hence∀ a 1 , a 2 ∈
[ K ] : P ( ψ a 1 = ψ a 2 ) = 0 , P ( a ∗ t = a | H − t − 1 ) = P ( a ∗ t = a ) and P ( a ∗ t = a 1 ) = P ( ψ a 1 > ψ a ′ , ∀ a ′ ̸ = a 1 ) =
P ( ψ a 2 > ψ a ′ , ∀ a ′ ̸ = a 2 ) = P ( a ∗ t = a 2 ) = 1 /K .
We call property (2) history-agnostic demographic parity since it states that, at each round, the policyselects all groups with equal probability regardless of the history. Recall that in our setup each armcorresponds to a sensitive group.
Proposition 3.1 ensures that a GMF policy will keep exploringregardless of the history.
This fact plays a key role in the design of our greedy policy
Remark 3.2.
Note that in the standard linear contextual bandit setting, the optimal policy ( a opt t ) doesnot necessarily satisfy (2) even when we assume that {⟨ µ ∗ , X a ⟩} Ka =1 are independent and absolutelycontinuous.
This is true since when the rewards of one arm are always lower than at least one of theother arms, that arm will never be selected by the optimal policy.
In the following, we properly state and then discuss all the assumptions made in this work.
Assumption A. Let µ ∗ ∈ R
d be the underlying reward model.
We assume that:
(i)
The noise random variable η t is zero mean R -subgaussian, conditioned on H t − 1 .
(ii) Let X a be a random variable with values in R d and
such that ∥ X a ∥ 2 ≤ L almost surely.
For any a ∈ [ K ] , { X i,a } Ti =1 are i.i.d.
copies X a .(iii)
The random variables { X a } Ka =1 are mutually independent.
(iv) For every a ∈ [ K ] , there exist an s a ≥ 1 , an absolutely continuous random variable Y a withvalues in R s a admitting a density f a , B a ∈ R d
× s a and c a ∈ R d such that B ⊤ a B a
= I s a ,X a =
B a Y a
+ c a and µ ∗⊤ B a ̸ = 0 .
Assumption A(i) is a standard assumption on the noise in stochastic bandits.
Assumption A(ii)implies that the actions taken by the policy do not affect future contexts.
This is needed to allow thelearning of the distribution of rewards for each group and is also used in [10, 27].
Assumption A(iv)implies that ⟨ µ ∗ , X a ⟩ is absolutely continuous and is satisfied when X a is absolutely continuousin a subspace of R d which is not orthogonal to µ ∗ 1 .
This fact combined with A(iii) ensures that Proposition 3.1 holds.
Assumptions A(iii)-(iv) are specific to our setting and a current limitation ofthe analysis.
Notice however, that assuming that the rewards ⟨ µ ∗ , X a ⟩ are absolutely continuous isnatural when the context contain continuous attributes, while A(iii) is reasonable when the groupsare sufficiently isolated, e.g. each context is sourced from a different country/group.
Note that wealso make the implicit assumption that at each round the agent receives exactly one candidate fromeach group.
We study a natural extension of our bandit setup where we remove this assumption in
Appendix E, and show that our results can easily be adapted to this more realistic setting.
In the following section, we present a greedy policy which achieves sublinear fair pseudo-regret withhigh probability, while selecting each arm with the same probability at each round when averagedover all the available information.
Note that we use arm and group interchangeably in all that follows.
Fair Greedy Policy
If Proposition 3.1 holds, then there is no arm with relative rank always strictly worse than theothers and a policy with sub-linear fair pseudo-regret (1) will never stop selecting one or more arms.
Hence, using confidence intervals will not help in decreasing the probability that one arm is selected.
Furthermore, estimating the relative ranks {F a ( ⟨ µ ∗ , X t,a ⟩ ) } Ka =1 is challenging, since they are notdirectly observed and using the past noisy rewards { r i,a i } t − 1 i =1 to construct the empirical CDF foreach group, similarly to [23], can be inaccurate due to the presence of noise.
For these reasons we propose a greedy policy, outlined in Alg. 1, which uses the following two-stageprocedure at each round t .
First it assembles the previously selected contexts and correspondingrewards from iterate 1 up to t 0 =
⌊ ( t − 1) / 2 ⌋ (line 4) in order to construct an estimate µ t 0 of µ ∗ (line 6).
As we shall see, this estimate is a noisy version of the ridge regression estimator.
Secondly, for each arm a , our policy computes an estimate of the relative rank F a ( ⟨ µ ∗ , X t,a ⟩ ) ,namely ˆ F t,a ( ⟨ µ t 0 , X t,a ⟩ ) , which is the empirical CDF value of ⟨ µ t 0 , X t,a ⟩ and is constructed usingµ t 0 and the contexts from round t 0 + 1 up to t (line 7).
Lastly, it selects a t uniformly at randomamong the arms in argmax a ∈ [ K ] ˆ F t,a ( ⟨ µ t 0 , X t,a ⟩ ) (line 8).
Remark 4.1 (Computational cost of the proposed policy) .
Compared to common approaches basedon the ridge regression estimator, Alg. 1 has an higher computational and memory cost which growlinearly with t .
µ t 0 requires to compute the product between V − 1 t 0 and X ⊤ 1: t 0 r 1: t 0 , which can be storedusing d 2 and d values respectively and updated online using the sherman-morrison formula [17].
However, Alg. 1 also requires, at each round t , to keep in memory K ( t − t 0 ) d -dimensional contextsand to compute the same number of scalar products to construct the empirical CDF for all K groups.227 Algorithm
Set t 0 =
⌊ ( t − 1) / 2 ⌋ , X 1: t 0 =
( X 1 ,a 1 , . . .
, X t 0 ,a t 0 ) ⊤ , r 1: t 0 = ( r 1 ,a 1 , . . .
, r t 0 ,a t 0 ) .
Generate γ
0 ∼ N (0 , I d ) .
If t 0 = 0 set µ t 0 = 0 , else let V t 0 :
= X ⊤ 1: t 0 X 1: t 0 + λ I d and compute µ t 0 :
[Equation]
For each a ∈ [ K ] compute
[Equation]
Sample
[Equation]
Observe r t,a t = ⟨ µ, X t,a t ⟩ + η t .
10: end for
The algorithm has two hyperparameters λ
and ρ , although the latter can be set arbitrarily smallwithout affecting the regret.
The small additive noise at line 5 of Alg. 1 has negligible impact on theestimation of µ ∗ but it ensures that the following holds.
Lemma 4.1.
Let a t be the action taken by the policy defined in Alg.
1 at time t and let Assumption Abe satisfied.
Then, for all t ≥ 1 we have
[Equation]
Proof sketch (proof in Appendix B).
The noise term in µ t 0 ensures that µ t 0 is absolutely continuousand hence µ ⊤ t 0 B a ̸ = 0 almost surely.
Combining this with Assumption A(iv) we obtain that ⟨ µ t [-
-]0 , X a ⟩is also absolutely continuous (see Lemma A.1).
Moreover, thanks to Assumption A(ii)(iii) we canshow that the random variables in { ˆ F t,a ( ⟨ µ t 0 , X t,a ⟩ ) } Ka =1 are i.i.d.
when conditioned on µ t 0 .
Notethat a t is sampled uniformly form the argmax of i.i.d.
random variables, when conditioned on µ t 0 ,which implies P ( a t = a | µ t 0 ) =
1 /K .
The statement follows by taking the expectation over µ t 0 .
Since P ( a t = a ) =
E H t − 1 [ P ( a t = a | H t − 1 )] , with H t − 1 being the information available tothe policy before making a decision at round t , we call the property in (3) information-averageddemographic parity , which is weaker than history-agnostic demographic parity (in (2)).
However,our analysis requires a lower bound on P ( a t = a | H − t − 1 ) which is presented in the next section.
Regret Analysis
In this section we present the analysis leading to the high probability ˜ O
( K 3 + √ dT ) upper bound onthe fair pseudo-regret of the greedy policy in Alg.
We start by showing two key properties of CDFfunctions in the following lemma.
Recall that for a continuous random variable Z we denote by f Zthe associated probability density function (PDF) (proof in Appendix C.1).
Lemma 5.1.
Let Assumption A (iv) hold and set ∀ a ∈ [ K ] , Z a := ⟨ µ ∗ , X a ⟩ , and M :=max a ∈ [ K ] ,z ∈ R
f Z a ( z ) as the maximum PDF value of the rewards of all groups.
Then, the followingtwo statements are true.
(i) F a is Lipschitz continuous for every a ∈ [ K ] , and in particular for any r, r ′ ∈ R we have
[Equation]
(ii)
For every a ∈ [ K ] , let µ ∈ R d , Z := ⟨ µ, X a ⟩ .
Then we havesup a ∈ [ K ] ,r ∈ R |F a ( r ) − F Z ( r ) | ≤ 2 M ∥ µ ∗ − µ ∥∥ x max ∥ ∗ , for any norm ∥·∥ with dual norm ∥·∥ ∗ , where ∥ x max ∥ ∗ := sup x ∈∪ K a =1 Supp( X a )
∥ x ∥ ∗ ,where Supp( X a ) is the support of X a .
Lemma 5.1(i) characterizes the Lipschitz constant of F a and its derivation is straightforward.
Lemma 5.1(ii) is needed since we only have access to an estimate of µ ∗ , which will take therole of µ .
Its derivation is more subtle and could be of independent interest.
By using Lemma 5.1 andthe Dvoretzky–Kiefer–Wolfowitz-Massart (DKWM) inequality [15, 30] to bound the gap between
CDF and empirical CDF, we obtain the following result.
Lemma 5.2 (Instant regret bound) .
Let Assumption A (ii)(iv) hold and a t to be generated by Alg. 1.
Then with probability at least 1 − δ/ 4 , for all t such that 3 ≤ t ≤ T we have
[Equation]
where ∥ x max ∥ V − 1 t 0 : = sup x ∈∪ Ka =1 Supp [Equation] ∥ V −
[Equation]
[Equation]
Proof.
Let Z t := ⟨ µ t 0 , X a t ⟩ , Z ∗ t := ⟨ µ t 0 , X a ∗ t ⟩ and F Z t , F Z ∗ t be their CDF conditioned on µ t 0 , a t ,and a ∗ t .
Let also R inst ( t ) :
= F a ∗ t ( ⟨ µ ∗ , X t,a ∗ t ⟩ ) − F a t ( ⟨ µ ∗ , X t,a t ⟩ ) .
Then we can write
R inst ( t ) =
F a ∗ t ( ⟨ µ ∗ , X t,a ∗ t ⟩ ) − F a ∗ t ( ⟨ µ t 0 , X t,a ∗ t ⟩ ) (cid:124) (cid:123)(cid:122) (cid:125)
(I)
[Equation]
[Equation]
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) (
[Equation]
[Equation]
Since a t is chosen greedily in Alg.
1 we have (IV) ≤ 0 .
Then, applying Lemma 5.1(i), CauchySchwarz and ∥ X t,a ∥ ∗ ≤ ∥ x max ∥ ∗ for (I) and (VII) and Lemma 5.1(ii) for (II) and (VI), we obtain(I)
+ (VII) ≤ 2 M ∥ µ ∗ − µ t
∥∥ x max ∥ ∗ , [-(II) + (VI) ≤ 4 M ∥ µ ∗ − µ t 
-][Equation]
∥∥ [Equation] max
[Equation] ∗
[Equation]
By noticing that ˆ F t,a ( · ) is the empirical CDF of the random variable ⟨ µ t 0 , X a ⟩ conditioned to µ t 0 ,we can bound (III) and (V) directly using the DKWM inequality (see Lemma C.1), which gives thatwith probability at least 1 − δ/ 4 and for all t such that 3 ≤ t ≤ T we have
[Equation]
We conclude the proof by combining the previous bounds and setting ∥·∥ = ∥·∥ V t 0 .
We proceed by controlling the term ∥ µ ∗ − µ t
∥ V t
∥ x max ∥ V − 1 t 0 in Lemma 5.2.
The quantity ∥ µ ∗ −µ t
∥ V t 0 can be bounded using the OFUL confidence bounds [1, Theorem 2], since the noise term inµ t 0 decreases at an appropriate rate.
Controlling ∥ x max ∥ V − 1 t 0 requires instead different results thanthe ones in [1], since it depends on the distributions of { X a } Ka =1 and not only on previous contextsand rewards.
Hence, to provide an upper bound for ∥ x max ∥ V − 1 t 0 which decreases with t , we also relyon Assumption A(iii) and the structure of Alg. 1, which enable the following history-agnostic lowerbound on the probability of selecting one arm.
Proposition 5.1.
Let Assumption A hold, a t be generated by Alg.
1 and c ∈ [0 , 1) .
Then withprobability at least 1 − δ/ 4 , for all a ∈ [ K ] and all t ≥ 3 + 8 log 3 / 2 (cid:0) 5 K e /δ (cid:1) / (cid:0) 1 − K √ c (cid:1) 3 we have P ( a [Equation]
=
[Equation]
where we recall that H − t = ∪ ti =1 (cid:8) { X i,a } Ka =1 , r i,a i , a i (cid:9) .
Proof sketch (proof in Appendix C.2).
For any a ∈ [ K ] , let ˆ r t,a = ⟨ µ t 0 , X t,a ⟩ be the estimatedreward for arm a at round t , denote with F ˆ r t,a the CDF of ˆ r t,a conditioned on µ t 0 , and letϕ t,a :=
F ˆ r t,a (ˆ r t,a ) , and ˆ ϕ t,a :=
ˆ F t,a (ˆ r t,a ) , where ˆ F t,a (ˆ r t,a ) is defined in line 7 of Alg.
Now, by the definition of a t (line 8 of Alg. 1), we have
[Equation]
where we introduced C t :
= argmax a ∈ [ K ] ˆ ϕ t,a .
Let ϵ t > 0 and continue the analysis conditioning onthe events where sup a ∈ [ K ] | ϕ t,a − ˆ ϕ t,a | ≤ ϵ t .
Then, we can write
[Equation]
where in the first inequality we considered the case when a ∈ C t and | C t | = 1 , and in the secondinequality we considered the worst case scenario where ˆ ϕ t,a = ϕ t,a − ϵ t and ˆ ϕ t,a ′ =
ϕ t,a ′ + ϵ t .
Assumption A(iv) and the additive noise in µ t 0 imply that ⟨ µ t 0 , X a ⟩ is an absolutely continuousrandom variable for each a ∈ [ K ] , which yields that { ϕ t,a } a ∈ [ K ] is uniformly distributed on [0 , 1] .
Furthermore, { ϕ t,a } a ∈ [ K ] are also independent due to Assumption A(iii).
Thus we have
[Equation]
Finally, thanks to Assumption A(ii) we can invoke the DKWM inequality to appropriately bound ϵ tin high probability for all t sufficiently large.
The property in Proposition 5.1 guarantees that, for sufficiently large t , the policy can get arbitrarilyclose to satisfy history-agnostic demographic parity in (3).
In particular this allows us to control∥ x max ∥ V − 1 t 0 by using a standard matrix concentration inequality [33, Theorem 3.1] on a specialdecomposition of V t 0 , thereby enabling the following result (proof in Appendix C.3).
Lemma 5.3.
Let Assumption A hold, a t be generated by Alg. 1, τ 1 = 32 K 3 log 3 / 2 (cid:0) 5 K e /δ (cid:1) ,τ 2 = {+
+} L 2 λ +min (Σ) log( 4 dδ ) and τ = 4 max( τ 1 , τ 2 ) + 3 .
Then, with probability at least 1 − 3 δ 4 , for allt ≥ τ we have
[Equation]
where b 1 = λ 12 + R + L , Σ :=
K − 1 (cid:80) Ka
=1 E (cid:2)
X a X ⊤ a (cid:3) and λ +min (Σ) is its smallest nonzero eigenvalue.
[Figure]
Finally we obtain the desired high probability regret bound by combining Lemma 5.2 with Lemma 5.3and summing over the T rounds (see Appendix C.4 for a proof).
Theorem 1.
Let Assumption A hold and a t be generated by Alg.
Then, with probability at least1 − δ , for any T ≥ 1 we have
[Equation]
with τ defined in Lemma 5.3.
Hence R F ( T ) =
O ( K 3 log 3 / 2 ( K/δ ) + (cid:112) dT log( KT/δ )) .
The regret bound in Thm.
1 has two terms.
The O ( K 3 log 3 / 2 ( K )) term describes the rounds neededto satisfy Proposition 5.1 with c = 1 / 2 .
The remaining part, which is of order O ( (cid:112) dT log( KT )) isinstead associated to the convergence of the empirical CDF and to the bandit performance.
Indeed, itrecalls the standard regret bound holding for finite-action linear contextual bandits [2, 12, 26].
Simulation with Diverse Reward Distributions
We present an illustrative proof of concept experiment which simulates groups with diverse rewarddistributions.
We set K = 4 , η t = 2 ξ t , where each ξ t has standard normal distribution, X a =
B a Y a + c a where each coordinate of Y a ∈ R 4 is an independent sample from the uniform distributionon [0 , 1] , B a ∈ R
(4 K +1)
× 4 is such that X a contains Y a starting from the 4 a -th coordinate and c a hasall the coordinates set to zero except for the last which is set to 3 a to simulate a group bias.
In thissetup µ ∗ acts differently on each group, in particular, we note that µ ∗ ∈ R 4 K +1 has its last coordinatemultiplying the group bias in c a , which we set to 1 , and 4 group-specific coordinates, which we set tomanually picked values between 0 and 9 .
Results are shown in Fig.
1, where we compare our greedypolicy in Alg. 1 with OFUL [1], both with regularization parameter set to 0 .
1 , and with the uniformrandom policy.
We observe that, as expected from our analysis, our policy achieves sublinear fairpseudo-regret, while also having better-than random, although linear, standard regret.
Conclusions
In this work, we introduced the concept of group meritocratic fairness in linear contextual bandits,which states that a fair policy should select, at each round, the candidate with the highest relativerank in the pool.
This allows us to compare candidates coming from different sensitive groups, but itis hard to satisfy since the relative rank is not directly observed and depends on both the underlyingreward model and on the rewards distribution for each group.
After defining an appropriate fairpseudo-regret we analyzed a greedy policy and proved that its fair pseudo-regret is sublinear with highprobability.
This result was possible since we restricted the analysis to the case where the contextsof different groups are independent random variables and the rewards are absolutely continuous.
Relaxing these assumptions is a challenging avenue for future work on this topic.
Another interestingdirection will be to study the optimality of the proposed results and establishing lower bounds for anyalgorithm which minimises the fair pseudo-regret.