Group Meritocratic Fairness in Linear Contextual Bandits
Abstract
We study the linear contextual bandit problem where an agent has to select one candidate from a pool and each candidate belongs to a sensitive group.
In this setting, candidates’ rewards may not be directly comparable between groups, for example when the agent is an employer hiring candidates from different ethnic groups and some groups have a lower reward due to discriminatory bias and/or social injustice.
We propose a notion of fairness that states that the agent’s policy is fair when it selects a candidate with highest relative rank, which measures how good the reward is when compared to candidates from the same group.
This is a very strong notion of fairness, since the relative rank is not directly observed by the agent and depends on the underlying reward model and on the distribution of rewards.
Thus we study the problem of learning a policy which approximates a fair policy under the condition that the contexts are independent between groups and the distribution of rewards of each group is absolutely continuous.
In particular, we design a greedy policy which at each round constructs a ridge regression estimate from the observed context-reward pairs, and then computes an estimate of the relative rank of each candidate using the empirical cumulative distribution function.
We prove that, despite its simplicity and the lack of an initial exploration phase, the greedy policy achieves, up to log factors and with high probability, a fair pseudoregret of order √ dT after T rounds, where d is the dimension of the context vectors.
The policy also satisfies demographic parity at each round when averaged over all possible information available before the selection.
Finally, we use simulated settings and experiments on the US census data to show that our policy achieves sub-linear fair pseudo-regret also in practice.
Introduction
Consider a sequential decision making problem where at each round an employer has to select one candidate from a pool to hire for a job.
The employer does not know how well a candidate will perform if hired, but they can learn it over time by measuring the performance of previously selected similar candidates.
This scenario can be formalized as a (linear) contextual bandit problem (see [2, 12, 26] and references therein), where each candidate is represented by a context vector, and after the employer (or agent) chooses a candidate, it receives a reward, i.e. a scalar value measuring the true performance of the candidate, which depends (linearly) on the context.
In the above framework, the typical objective is to find a policy for the employer to select candidates with the highest rewards [1–3, 26].
However, in some important scenarios this objective may not be appropriate; if candidates belong to different sensitive groups (e.g. based on ethnicity, gender, etc.)
the resulting policy might discriminate or even exclude some groups completely in the selection process.
This may happen when some groups have lower expected reward than others, e.g. because they acquired less skills due lower financial support.
Another example arises when each candidate in the pool, if selected, will perform a different kind of job, and the associated reward is job-specific.
For instance, if the employer is a university and each candidate is a researcher in a different discipline, then the rewards associated to different disciplines will be substantially different and incomparable, e.g. citation counts vary greatly among different subjects; see [23] for a discussion.
In both of the above scenarios, it is unfair to directly compare rewards of candidates belonging to different groups.
A simple way to deal with this issue would be to select the candidate to hire uniformly at random.
This policy satisfies a notion of fairness called demographic parity (see [7, 31] and references therein), which requires the probability of selecting a candidate from a given group to be equal for all groups.
However, as is apparent, this approach completely ignores the employer’s goal of selecting good candidates and is also unfair to candidates who spent effort acquiring credentials for the job.
In this work, we provide a fair way of comparing candidates from different groups via the relative rank , that is the cumulative distribution function (CDF) value of the reward of the candidate where the distribution is that of the rewards of the candidate’s group.
We call a policy group meritocratic fair (GMF) if it always selects a candidate with the highest relative rank.
Such a policy is meritocratic but only in terms of the within-group performance.
A closely related idea has been introduced in [23] for settings where the candidates’ rewards are available before the selection, while we are not aware of a similar notion in the multi-armed bandits literature.
A GMF policy requires the knowledge of the relative rank of each candidate which is not directly observed by the agent and depends on the underlying reward model and on the distributions of rewards.
Moreover, to estimate the relative rank from the observed rewards and contexts it is necessary to learn the CDF of the rewards of each group, which adds a challenge to the standard linear contextual bandit framework where only the linear relation between contexts and rewards has to be learned.
Due to this, a learned policy cannot be GMF at all rounds, thus we study the problem of learning a policy which minimizes the fair regret , that is the cumulative difference between the relative rank of the candidate chosen by a GMF policy and the candidate chosen by a learning policy.
For this purpose, we design a greedy policy, which at each round uses the following two-stage strategy.
Firstly, it constructs a ridge regression estimate which maps contexts to rewards.
Secondly, it computes an estimate of the relative rank of each candidate using the empirical CDF of the estimated rewards.
We show that the proposed policy achieves, under some reasonable assumptions and after
T rounds, ˜ O
( K 3 + √ dT ) fair regret with high probability, where d is the dimension of the context vectors and K is the number of candidates in the pool.
Notably, our policy does not require an initial exploration phase and satisfies demographic parity at each round when averaged over all possible random draws of the information avaliable to the agent before the decision, i.e. current contexts and previously received contexts, actions and rewards.
Contributions and Organization.
After a review of previous work in Sec. 2, we introduce the learning problem and the proposed fairness notion in Sec. 3.
To simplify the exposition, we assume that each arm corresponds to a sensitive group.
In Sec.
4 we propose a greedy policy which jointly learns the underlying regression model and the CDF of each group.
We derive a ˜ O ( √ dT ) regret bound for our policy in Sec. 5.
In Sec.
6 we present an illustrative simulation experiment with diverse reward distributions.
In Sec. 7, we extend our policy and results to the case where candidates from the same arm can belong to different groups and show the efficacy of our approach with an experiment on the US census data where the sensitive group (ethnicity) is drawn at random together with the context.
We draw conclusions in Sec. 8.
Code at https://github.com/CSML-IIT-UCL/GMFbandits
Notation.
We use ⟨· , ·⟩ for the scalar product.
For K ∈ N we have [ K ] = { 1 , . . .
, K } .
Let ψ be a scalar random variable, for each a ∈ [ K ] and µ ∗ ∈ R d , we denote with F ψ , F a the CDF of ψ and ⟨ µ ∗ , X a ⟩ respectively.
If X is a continuous random vector with values in R d , we denote with f X : R d → [0 , ∞ ) its probability density function.
For any s ∈ N , we denote I s as the s × s identity matrix.
For a random variable Y ∈ R s , we call Y an absolutely continuous random variable, if its distribution is an absolutely continuous measure with respect to the Lebesgue measure on R s .
For a positive semi-definite matrix D , we denote λ min ( D ) and λ
+min ( D ) the smallest eigenvalue of D , and the smallest non-zero eigenvalue of D respectively.
Supp( X ) indicates the support of a random variable X .
We also denote with U [ S ] , the uniform distribution over the set S .
Related Works
In recent years algorithmic fairness has received a lot of attention, becoming a large area of machine learning research.
The potential for learning algorithms to amplify pre-existing bias and cause harm to human beings has triggered researchers to study solutions to mitigate or remove unfairness of the learned predictor, see [4, 8, 11, 14, 16, 19, 20, 22, 24, 25, 29, 35–37, 39] and references therein.
Fairness in sequential decision problems (see [38] for a survey) is usually divided into two categories: group fairness (GF) and individual fairness.
We give an overview of these notions below.
GF requires some statistical measure to be (approximately) equal across different sensitive groups.
A prominent example relevant to this work is demographic parity , which requires that the probability that the policy selects a candidate from a given group should be the same for all groups.
A similar notion is used by [10, 32], where the probability that the policy selects a candidate has to always be greater than a given threshold for all candidates.
[27] impose a weaker requirement concerning the expected fraction of candidates selected from each group.
Other examples of GF in sequential decision problems are equal opportunity [5] and equalized odds [6].
Under some assumptions on the distributions of the contexts, our GMF and greedy policies satisfy variants of demographic parity at each round.
Individual fairness can be divided in two categories: fairness through awareness (FA) [28, 34] and meritocratic fairness (MF) [21, 22].
FA is based on the idea that similar individuals should be treated similarly and is designed to avoid “winner takes all” scenarios where some individuals cannot be selected when they have a lower reward than others in the pool, even if the difference between rewards is very small.
For example, [34] propose a policy where the probability of selecting a context over another is lower when the context has a lower reward, but is never zero.
MF instead requires that less qualified individual should not be favored over more qualified ones, which could happen during the learning process.
For example [22] proposes an algorithm where the policy selects the arm uniformly at random among the best arms with overlapping confidence intervals.
This guarantees meritocratic fairness at each round but comes at a cost in terms of regret.
Our definition of fairness falls between group and meritocratic fairness.
It is meritocratic because it states that a candidate with a worse relative rank than another should never be selected.
It is also based on groups since the relative ranks directly depend on the distribution of rewards of each group.
A similar idea of fairness based on relative rank has been introduced in [23], which study the problem of selecting candidates from different groups based on their scalar-valued score when the scores between groups are incomparable (e.g. number of citations in different research areas).
Contrary to our work, where the (noisy) rewards are observed only for the selected candidates, in [23] the noiseless scores for all candidates can be accessed before the selection.
This difference makes the estimation of the relative rank simpler in [23], as the rewards CDFs can be estimated more efficiently.
Group Meritocratic Fairness in Linear Contextual Bandits
We consider the linear contextual bandit setup [2] where at each round t ∈ [ T ] , an agent receives a set of feature vectors { X t,a } Ka =1 with X t,a ⊂ R d sampled from the environment, one for each arm a ∈ [ K ] .
We assume that context (or candidate) X t,a has an associated reward ⟨ µ ∗ , X t,a ⟩ where µ ∗ ∈ R
d is unknown to the agent.
After the agent selects the arm a t , it receives the noisy reward equal to r t,a t = ⟨ µ ∗ , X t,a t ⟩ + η t , where η t is some scalar noise (formally specified later).
In addition, we assume that each arm represents a fixed sensitive group (e.g. based on ethnicity, gender, etc.).
The latter assumption simplifies the presentation but implies that at each round the agent receives exactly one candidate for each group.
This can be too restrictive
e.g. when candidates are sampled i.i.d.
together with their group and/or some groups are minorities.
However, our results can be easily adapted to more realistic settings without such assumption, as we show in Sec.
7 and more rigorously in Appendix E.
Excluding these sections, we use arm and group interchangeably in all that follows.
Usually, the goal of the agent is to maximise the expected cumulative reward (cid:80) T t =1 ⟨ µ ∗ , X t,a t ⟩ .
Since as we previously explained, this objective might be unfair to some of the sensitive groups, we instead use a different kind of reward which measures the relative performance of a candidate compared to others of the same arm/group.
First, we additionally assume, for each group a , that { X t,a } Tt =1 are i.i.d and have the same distribution of X a , which we define to be a random variable with unknown distribution.
We call the distribution of ⟨ µ ∗ , X a ⟩ the reward distribution of arm a and denote with F a its CDF, i.e. F a ( r ) =
P ( ⟨ µ ∗ , X a ⟩ ≤ r ) for every r ∈ R .
Then, we introduce the relative rank of candidate X t,a as F a ( ⟨ µ ∗ , X t,a ⟩ ) , that is the probability that a sample from the reward distribution of arm a is lower than the reward of X t,a .
We argue that the relative rank, allows to have a fair way of comparing candidates from different groups and introduce the following fairness definition.
Definition 3.1 (Group Meritocratic Fairness) .
A policy { a ∗ t } ∞ t =1 is group meritocratic fair (GMF) if for all t ∈ N , a ∈ [ K ]
[Equation]
A GMF policy chooses candidates with the highest reward compared to candidates from the same group.
This is a strong definition of fairness which is impossible to satisfy at each round for a learned policy.
As in standard linear contextual bandits, µ ∗ is unknown and must be learned.
In this setting however, we have the additional challenge of learning the CDF for the rewards of each arm, F a .
Thus, we will focus on how to learn a GMF policy by introducing the following regret definition.
Definition 3.2 (Fair Pseudo-Regret) .
Let T ∈ N , { a t } Tt =1 be the evaluated policy and { a ∗ t } Tt =1 be a
GMF policy.
Then we denote by (cumulative) fair pseudo-regret the quantity
[Equation]
The goal of the learned policy will be to minimize the fair pseudo-regret, since a policy with sublinear fair pseudo-regret will get closer and closer to a GMF fair policy over time.
Remark 3.1.
The fair pseudo-regret resembles the standard pseudo-regret defined as
R ( T
[Equation]
:=
[Equation]
(cid:88)
[Equation]
where rewards are replaced by relative ranks and a opt t by the GMF policy a ∗ t .
Furthermore, since the CDF restricted to the support is strictly increasing, when the reward distributions are the same for each arm, i.e. F a = F a ′ for all a, a ′ ∈
[ K ] , then a policy minimizing the fair pseudo-regret also minimizes the standard pseudo-regret and vice versa.
This is not true in the general case, where fair and standard pseudo-regrets are often competing objectives.
For example, when {⟨ µ ∗ , X a ⟩} Ka =1 are independent and absolutely continuous and there exists ˆ a such that ⟨ µ ∗ , X ˆ a ⟩ > ⟨ µ ∗ , X a ⟩ for every a ̸
= ˆ a , then for every t , a opt t = ˆ a , while as we will show in Proposition 3.1, a ∗ t selects each arm with equal probability.
Thus, with non-zero probability a opt t has a linear fair pseudo-regret while a ∗ t has a linear standard pseudo-regret.
Moreover, in Appendix F, for K = 2 , we show that if ⟨ µ ∗ , X 1 ⟩ and ⟨ µ ∗ , X 2 ⟩ are independent, absolutely continuous, but not identically distributed, then the GMF policy has a linear standard regret and { a opt t } ∞ t =1 has a linear fair regret with positive probability.
Learning a GMF policy brings several challenges.
The relative rank is not directly observed by the agent, which receives instead only the noisy reward.
This implies that the agent has to estimate
F a , which in general might not even be Lipschitz continuous.
This is the main reason why we restrict our analysis to the case where the rewards {⟨ µ ∗ , X a ⟩} Ka =1 are independent and absolutely continuous.
In particular, for any t ≥ 0 , let H − t := ∪ ti =1 (cid:8) { X i,a } Ka =1 , r i,a i , a i (cid:9) with H − 0 = ∅ and H t :
= H − t ∪ {{ X t +1 ,a } Ka =1 } be respectively the history and the information available for the decision at round t + 1 , then the following holds.
Proposition 3.1 (GMF policy satisfies history-agnostic demographic parity ) .
Let {⟨ µ ∗ , X a ⟩} Ka =1 be independent and absolutely continuous and for every a ∈ [ K ] , t ∈ N , let X t,a be an i.i.d.
copy of X a .
Then for every t ∈ N , {F a ( ⟨ µ ∗ , X t,a ⟩ ) } Ka =1 are i.i.d.
uniform on [0 , 1] and
[Equation]
for any GMF policy { a ∗ t } ∞ t =1 .
Note, the randomness lies exclusively in the current contexts { X t,a } Ka =1 .
Proof.
Let ψ a :=
F a ( ⟨ µ ∗ , X t,a ⟩ ) .
From the assumptions { ψ a } K a =1 are i.i.d random variables, independent from H − t − 1 , with uniform distribution on [0 , 1] (see [9, Theorem 2.1.10]).
Hence ∀ a 1 , a 2 ∈
[ K ] : P ( ψ a 1 = ψ a 2 ) = 0 , P ( a ∗ t = a | H − t − 1 ) =
P ( a ∗ t = a ) and
P ( a ∗ t = a 1 ) = P ( ψ a 1 > ψ a ′ , ∀ a ′ ̸
= a 1 ) =
P ( ψ a 2 > ψ a ′ , ∀ a ′ ̸ = a 2 ) = P ( a ∗ t = a 2 ) = 1 /K .
We call property (2) history-agnostic demographic parity since it states that, at each round, the policy selects all groups with equal probability regardless of the history.
Recall that in our setup each arm corresponds to a sensitive group.
Proposition 3.1 ensures that a GMF policy will keep exploring regardless of the history.
This fact plays a key role in the design of our policy, which is greedy without the need of an exploration phase.
Remark 3.2.
Note that in the standard linear contextual bandit setting, the optimal policy a opt t does not necessarily satisfy (2) even when we assume that {⟨ µ ∗ , X a ⟩} Ka =1 are independent and absolutely continuous.
This is true since when the rewards of one arm are always lower than at least one of the other arms, that arm will never be selected by the optimal policy.
In the following, we state and discuss the assumptions made for the analysis of our greedy policy.
Assumption A. Let µ ∗ ∈ R
d be the underlying reward model.
We assume that:
(i)
The noise random variable η t is zero mean R -subgaussian, conditioned on H t − 1 .
(ii) Let X a be a random variable with values in R d and
such that ∥ X a ∥ 2 ≤ L almost surely.
For any a ∈ [ K ] , { X i,a } Ti =1 are i.i.d.
copies of X a .
(iii)
The random variables { X a } Ka =1 are mutually independent.
(iv) For every a ∈ [ K ] , there exist d a ≥ 1 , an absolutely continuous random variable Y a with values in R d a admitting a density f a , B a ∈ R d × d a and c a ∈ R d
such that B ⊤ a B a
= I d a , X a =
B a Y a
+ c a and µ ∗⊤ B a ̸ = 0 .
Assumption A(i) is a standard assumption on the noise in stochastic bandits.
A(ii) implies that the actions taken by the policy do not affect future contexts.
This is needed to allow the learning of the distribution of rewards for each group and is also used in [10, 27].
A(iv) implies that ⟨ µ ∗ , X a ⟩ is absolutely continuous and is satisfied when X a is absolutely continuous in a subspace of R d which is not orthogonal to µ ∗ 2 .
This fact combined with A(iii) ensures that Proposition 3.1 holds.
Assumptions A(iii)-(iv) are specific to our setting and a current limitation of the analysis.
Notice however, that A(iii) is reasonable when the groups are sufficiently isolated, e.g. each context is sourced from a different country/group, while assuming that the rewards ⟨ µ ∗ , X a ⟩ are absolutely continuous is natural when the contexts contain continuous attributes.
Furthermore A(iv) allows µ ∗ to act differently on each group, similarly to the case when there is a different reward vector for each sensitive group.
An example of this is showed in the simulation experiment in Sec. 6.
The Fair-Greedy Policy
If Proposition 3.1 holds, then there is no arm with relative rank always strictly worse than the others and any learned policy with sub-linear fair pseudo-regret will select all arms with equal probability in the limit when the number of rounds goes to infinity.
Hence, using confidence intervals will not help in decreasing the probability that one arm is selected.
Furthermore, estimating the relative ranks {F a ( ⟨ µ ∗ , X t,a ⟩ ) } Ka =1 is challenging, since they are not directly observed and using the past noisy rewards { r i,a i } t − 1 i =1 to construct the empirical CDF for each group, similarly to [23], can be inaccurate due to the presence of noise.
For the reasons above, we propose the greedy approach in Alg. 1, which uses the following two-stage procedure at each round t .
First it assembles the previously selected contexts and corresponding rewards from iterate 1 up to ˜ t =
⌊ ( t − 1) / 2 ⌋ (line 4) in order to construct an estimate µ ˜ t of µ ∗ (line 5), which is a noisy version of the ridge regression estimate.
Secondly, for each arm a , our policy computes an estimate of the relative rank F a ( ⟨ µ ∗ , X t,a ⟩ ) , namely ˆ F t,a ( ⟨ µ ˜ t , X t,a ⟩ ) , which is the empirical CDF value of ⟨ µ ˜ t , X t,a ⟩ and is constructed using µ ˜ t and the contexts from round ˜ t +up to t (line 6).
Lastly, it selects a t uniformly at random among the arms maximizing the relative rank estimate (line 7).
1: Requires regularization parameter λ > 0 and noise magnitude ρ ∈ (0 , 1] .
2: for t = 1 . . .
T do 3:
Receive contexts { X t,a } Ka = 4:
Set ˜ t =
⌊ ( t − 1) /
[Equation]
, X
[Equation]
X
[Equation] [Equation]
X
[Equation]
t
[Equation] [Equation]
If ˜ t = 0 set µ ˜ t = 0 , else let V ˜ t := X ⊤ 1:˜ t X 1:˜ t
+ λ I d , generate γ
[Equation]
t ∼ N (0 , I
[Equation]
) and compute
[Equation]
For each a ∈ [ K ] compute
[Equation]
Sample action
[Equation]
Observe noisy reward r t,a t = ⟨ µ, X t,a t ⟩ + η t .
9: end for
Fair-Greedy has two hyperparameters λ
and ρ , although the latter can be set arbitrarily small without affecting the regret.
Moreover, it is greedy as at each time t , it always selects from the arms the one with the highest currently estimated relative rank.
However, contrary to standard greedy approaches in bandits, Fair-Greedy does not require an initial exploration phase because it naturally explores all arms, as the following lemma and remark show.
Lemma 4.1 (Fair-Greedy satisfies information averaged demographic parity ) .
Let a t be the action taken by Fair-Greedy at time t and let Assumption A be satisfied.
Then, for all t ≥ 1 we have
[Equation]
Proof sketch (proof in Appendix B).
The noise term in µ ˜ t ensures that µ ˜ t is absolutely continuous and hence
µ ⊤ ˜ t B a ̸ = 0 almost surely.
Combining this with Assumption A(iv) we obtain that ⟨ µ ˜ t , X a ⟩ is also absolutely continuous (see Lemma A.1).
Moreover, thanks to Assumption A(ii)(iii) we can show that the random variables in { ˆ F t,a ( ⟨ µ ˜ t , X t,a ⟩ ) } Ka =1 are i.i.d.
when conditioned on µ ˜ t .
Note that a t is sampled uniformly form the argmax of i.i.d.
random variables, when conditioned on µ ˜ t , which implies P ( a t = a | µ ˜ t ) = 1 /K .
The statement follows by taking the expectation over µ ˜ t .
Remark 4.1.
It is easy to verify (through Lemma 4.1)
that at any number of rounds T , the Fair-Greedy policy selects in expectation T/K candidates from every group, i.e. E (cid:2) (cid:80) Tt =1 1 { a t = a } (cid:3) =
TK for every a ∈ [ K ] .
This also holds for the GMF policy and the one selecting arms uniformly at random.
Since P ( a t = a ) =
E H t − 1 [ P ( a t = a | H t − 1 )] , with H t − 1 being the information available to the policy before making a decision at round t , we call the property in (3) information-averaged demographic parity, which is weaker than history-agnostic demographic parity (in (2)).
However, our analysis still requires a lower bound on P ( a t = a | H − t − 1 ) which is presented in the next section.
Remark 4.2 (Computational cost of Fair-Greedy) .
Compared to common linear contextual bandits approaches based on ridge regression, Alg. 1 has an higher computational and memory cost which grow linearly with t .
µ ˜ t requires us to compute the product of V − 1 ˜ t and X ⊤ 1:˜ t r 1:˜ t , which can be stored using d 2 and d values respectively and updated online (via sherman-morrison [17]).
However, Alg.also requires, at each round t , to keep in memory K ( t − 1 − ˜ t )
d -dimensional contexts and to compute the same number of scalar products to construct the empirical CDF for all K groups.
Regret Analysis
In this section we present the analysis leading to the high probability
˜ O
( K 3 + √ dT ) upper bound on the fair pseudo-regret of the greedy policy in Alg.
We start by showing two key properties of
CDF functions in the following lemma (proof in Appendix C.1).
Recall that for a continuous random variable Z we denote by f Z the associated probability density function (PDF).
Lemma 5.1.
Let Assumption A (iv) hold and set ∀ a ∈ [ K ] , Z a := ⟨ µ ∗ , X a ⟩ so that F a = F Z a and
M := max a ∈ [ K ] ,z ∈ R f
Z a ( z ) <
+ ∞ as the maximum PDF value of the rewards of all groups.
Then, the following two statements are true.
(i) F a is Lipschitz continuous for every a ∈ [ K ] , and in particular for any r, r ′ ∈ R we have
[Equation]
(ii)
For every a ∈ [ K ] , let µ ∈ R d , ˜ Z a := ⟨ µ, X a ⟩ .
Then we have sup a ∈ [ K ] ,r ∈ R |F a ( r )
[Equation] [Equation]
∥
[Equation]
∥∥ x max ∥
[Equation]
for any norm ∥·∥ with dual norm ∥·∥ ∗ , where ∥ x max ∥ ∗ :
= sup x ∈∪ K a =1 Supp( X a )
∥ x ∥ ∗ and Supp( X a ) is the support of the random variable X a .
Lemma 5.1(i) bounds the Lipschitz constant of F a and its derivation is straightforward.
Lemma 5.1(ii) is needed since we only have access to an estimate of µ ∗ , which will take the role of µ .
Its derivation is more subtle and could be of independent interest.
By using Lemma 5.1 and the
Dvoretzky–Kiefer–Wolfowitz-Massart (DKWM) inequality [15, 30] to bound the gap between CDF and empirical CDF, we obtain the following result.
Lemma 5.2 (Instant regret bound) .
Let Assumption A (ii)(iv) hold and a t to be generated by Alg. 1.
Then with probability at least 1 − δ/ 4 , for all t such that 3 ≤ t ≤ T we have
[Equation]
where ∥ x
[Equation]
Proof.
Let Z t := ⟨ µ ˜ t , X a t ⟩ , Z ∗ t := ⟨ µ ˜ t , X a ∗ t ⟩ and F Z t , F Z ∗ t be their CDF conditioned on µ ˜ t , a t , and a ∗ t .
Let also R inst ( t ) :
= F a ∗ t ( ⟨ µ ∗ , X t,a ∗ t ⟩ ) − F a t ( ⟨ µ ∗ , X t,a t ⟩ ) .
Then we can write
R inst
[Equation]
Since a t is chosen greedily in Alg.
1 we have (IV) ≤ 0 .
Then, applying Lemma 5.1(i), CauchySchwarz and ∥ X t,a ∥ ∗ ≤ ∥ x max ∥ ∗ for (I) and (VII) and Lemma 5.1(ii) for (II) and (VI), we obtain
[Equation]
By noticing that ˆ F t,a ( · ) is the empirical CDF of the random variable ⟨ µ ˜ t , X a ⟩ conditioned to µ ˜ t , we can bound (III) and (V) directly using the DKWM inequality (see Lemma C.1), which gives that with probability at least 1 − δ/ 4 and for all t such that 3 ≤ t ≤ T we have
[Equation]
We conclude the proof by combining the previous bounds and setting ∥·∥ = ∥·∥ V ˜ t .
We proceed by controlling the term ∥ µ ∗ − µ ˜ t ∥ V ˜ t
∥ x max ∥ V − 1 ˜ t in Lemma 5.2.
The quantity ∥ µ ∗ − µ ˜ t ∥ V ˜ t can be bounded using the OFUL confidence bounds [1, Theorem 2], since the noise term in µ ˜ t decreases at an appropriate rate.
Controlling ∥ x max ∥ V − 1 ˜ t requires instead different results than the ones in [1], since it depends on the distributions of { X a } Ka =1 and not only on previous contexts and rewards.
Hence, to provide an upper bound for ∥ x max ∥ V − 1 ˜ t which decreases with t , we also rely on
Assumption A(iii) and the structure of Alg. 1, which enable the following history-agnostic lower bound on the probability of selecting one arm.
Proposition 5.1.
Let Assumption A hold, a t be generated by Alg.
1 and c ∈ [0 , 1) .
Then with probability at least 1 − δ/ 4 , for all a ∈ [ K ] and all t ≥ 3 + 8 log 3 / 2 (cid:0) 5 K e /δ (cid:1) / (cid:0) 1 − K √ c (cid:1) 3 we have
P
[Equation]
, where we recall that H − t = ∪ ti =1 (cid:8) { X i,a } Ka =1 , r i,a i , a i (cid:9) .
Proof sketch (proof in Appendix C.2).
For any a ∈ [ K ] , let ˆ r t,a = ⟨ µ ˜ t , X t,a ⟩ be the estimated reward for arm a at round t , denote with F ˆ r t,a the CDF of ˆ r t,a conditioned on µ ˜ t , and let ϕ t,a :=
F ˆ r t,a (ˆ r t,a ) , and ˆ ϕ t,a :=
ˆ F t,a (ˆ r t,a ) , where ˆ F t,a (ˆ r t,a ) is defined in line 6 of Alg.
Now, by the definition of a t (line 7 of Alg. 1), we have
P
[Equation]
where we introduced C t :
= argmax a ∈ [ K ] ˆ ϕ t,a .
Let ϵ t > 0 and continue the analysis conditioning on the events where sup a ∈ [ K ] | ϕ t,a − ˆ ϕ t,a | ≤ ϵ t .
Then, we can write
[Equation]
where in the first inequality we considered the case when a ∈ C t and | C t | = 1 , and in the second inequality we considered the worst case scenario where ˆ ϕ t,a = ϕ t,a − ϵ t and ˆ ϕ t,a ′ =
ϕ t,a ′ + ϵ t .
Assumption A(iv) and the additive noise in µ ˜ t imply that ⟨ µ ˜ t , X a ⟩ is an absolutely continuous random variable for each a ∈ [ K ] , which yields that { ϕ t,a } a ∈ [ K ] is uniformly distributed on [0 , 1] .
Furthermore, { ϕ t,a } a ∈ [ K ] are also independent due to Assumption A(iii).
Thus we have
P ( a t = a | H − t −
[Equation]
Finally, thanks to Assumption A(ii) we can invoke the DKWM inequality to appropriately bound ϵ t in high probability for all t sufficiently large.
The property in Proposition 5.1 guarantees that, for sufficiently large t , the policy can get arbitrarily close to satisfy history-agnostic demographic parity in (3).
In particular this allows us to control ∥ x max ∥ V − 1 ˜ t by using a standard matrix concentration inequality [33, Theorem 3.1] on a special decomposition of V ˜ t , thereby enabling the following result (proof in Appendix C.3).
Lemma 5.3.
Let Assumption A hold, a t be generated by Alg. 1, τ 1 = 32 K 3 log 3 / 2 (cid:0) 5 K e /δ (cid:1) , τ 2 =
L 2 λ +min (Σ) log( 4 dδ ) and τ = 4 max( τ 1 , τ 2 ) + 3 .
Then, with probability at least 1 − 3 δ 4 , for all t ≥ τ we have
[Equation]
where b 1 = λ 12 + R + L , Σ :=
K − 1 (cid:80) Ka
=1 E (cid:2)
X a X ⊤ a (cid:3) and λ +min (Σ) is its smallest nonzero eigenvalue.
Finally we obtain the desired high probability regret bound by combining Lemma 5.2 with Lemma 5.
and summing over the T rounds (see Appendix C.4 for a proof).
[Figure]
Theorem 1.
Let Assumption A hold and a t be generated by Alg.
Then, with probability at least 1 − δ , for any T ≥ 1 we have
[Equation]
with τ defined in Lemma 5.3.
Hence R F ( T ) =
O ( K 3 log 3 / 2 ( K/δ ) + (cid:112) dT log( KT/δ )) .
The regret bound in Thm.
1 has two terms.
The O ( K 3 log 3 / 2 ( K )) term describes the rounds needed to satisfy Proposition 5.1 with c = 1 / 2 .
The remaining part, which is of order O ( (cid:112) dT log( KT )) is instead associated to the convergence of the empirical CDF and to the bandit performance.
Indeed, it recalls the standard regret bound holding for finite-action linear contextual bandits [2, 12, 26].
Simulation with Diverse Reward Distributions
We present an illustrative proof of concept experiment which simulates groups with diverse reward distributions.
We set K = 4 , η t = 2 ξ t , where ξ t has standard normal distribution, X a =
B a Y a + c a where each coordinate of Y a ∈ R 4 is an independent sample from the uniform distribution on [0 , 1] ,
B a ∈ R (4 K +1)
× 4 is such that X a contains Y a starting from the 4 a -th coordinate and c a has all the coordinates set to zero except for the last which is set to 3 a to simulate a group bias.
In this setup µ ∗ acts differently on each group, in particular, we note that µ ∗ ∈ R 4 K +1 has its last coordinate multiplying the group bias in c a , which we set to 1 , and 4 group-specific coordinates, which we set to manually picked values between 0 and 9 .
Results are shown in Fig.
1, where we compare our greedy policy in Alg. 1 with OFUL [1], both with regularization parameter set to 0 .
1 , and with the Uniform
Random policy.
We observe that, as expected from our analysis, our policy achieves sublinear fair pseudo-regret, while also having better-than random, although linear, standard regret.
Additional details and an experiment on US census data with gender as the sensitive group are in Appendix D.
Multiple Candidates for Each Group
In this section, we analyze the more realistic case where contexts from a given arm do not necessarily belong to the same group.
The complete analysis is presented in Appendix E.
In particular, we assume that at each round t , the agent receives { ( X t,a , s t,a ) } Ka =1 , which are K i.i.d.
random variables where s t,a ∈ [ G ] is the sensitive group of the context X t,a ∈ R d and G is the total number of groups.
This setting can model for example a hiring scenario where at each round the employer has to choose among candidates belonging to different ethnic groups, some of which are minorities and hence have a small probability P ( s t,a = i ) of being in the pool of received candidates.
By naturally adapting the definition of fair-regret R F ( T ) , the Fair-Greedy policy and Assumption A to this setting, with probability 1 − δ we obtain the following regret bound (see Cor.
E.1 in Appendix E).
[Equation] [Figure]
where q min = min i ∈ [ G ] P ( s t,a = i ) G , so that q min = 1 if and only if each group has equal probability of being sampled and q min > 0 without loss of generality.
(4) is similar to Thm. 1, having the same dependency on δ and T but an improved dependency on the number of arms K when K > G , since contexts from all arms can be used to estimate the CDF of each group.
The first term in (4) comes from the application of the Chernoff bound to lower bound the number of candidates in each group received by the agent, which is now random.
US Census experiments.
Group = Ethnicity.
We test this setting in practice by simulating the hiring scenario discussed above with data from the US Census containing the income and other useful indicators of several individuals in the United States.
This data is accessed via the FolkTables library [13].
In particular, at each round, we sample K = 10 candidates at random from the population containing the G = 6 largest ethnic groups 3 , the reward is a previously computed linear estimate of the income, while the noisy reward is the true reward plus some small gaussian noise.
We compare the Fair-Greedy Policy with OFUL [1], Greedy (selects the candidate with the best estimated reward) and Uniform Random in Fig.
Similarly to the synthetic experiment in Sec. 6, the Fair-greedy policy achieves the best fair pseudo-regret and standard regret better than Uniform Random.
Note that Greedy outperforms OFUL, which is too conservative in this scenario.
Furthermore, the FairGreedy policy selects approximately the same percentage of candidates from each group, similarly to
Uniform Random, while OFUL and Greedy select smaller percentages from G2, G3, G5 and G6.
In
Appendix E.1 we provide more details and a comparison with two oracle fair policies which shows that knowing µ ∗ plays a more important role than knowing the true reward CDFs of each group.
Conclusions and Future Work
We introduced the concept of group meritocratic fairness in linear contextual bandits, which states that a fair policy should select, at each round, the candidate with the highest relative rank in the pool.
This allows us to compare candidates coming from different sensitive groups, but it is hard to satisfy since the relative rank is not directly observed and depends on both the underlying reward model and on the rewards distribution for each group.
After defining an appropriate fair pseudo-regret we analyzed a greedy policy and proved that its fair pseudo-regret is sublinear with high probability.
This result was possible since we restricted the analysis to the case where the contexts of different groups are independent random variables and the rewards are absolutely continuous.
Relaxing these assumptions is a challenging avenue for future work.
In particular, without the independence of contexts across arms, different approaches relying on confidence intervals might be necessary.
Other two interesting directions are (i) to study the optimality of the proposed results and establishing lower bounds for any algorithm which minimises the fair pseudo-regret and (ii) to design a learning policy which aims at achieving a tradeoff between group meritocratic fairness and reward maximization.
Appendices
Below we give an overview of the structure of the Appendices.
• In Appendix A we present some auxiliary lemmas which are used to prove our results.
• In Appendix B we present the proof of Lemma 4.1.
• Appendix C presents the proofs for the results in Sec. 5.
• In Appendix D we include additional details on the experimental setting in Sec.
6 and an experiment on the US Census Data with gender as sensitive attribute.
• In Appendix E we provide a more rigorous treatment of the setting introduced in Sec.
7, where we can have multiple condidates for each group at any given round.
We also provide more details and additional comparisons for the experiment on the US Census Data in Sec.
. • In Appendix F we investigate the tradeoff between fair an standard regret.
A Auxiliary Lemmas
Lemma A.1.
Let n ∈ N , and assume that Y, ν are independent random variables in R n , such that Y is absolutely continuous and ν ̸ = 0 , almost surely.
Then, ν ⊤ Y is an absolutely continuous random variable.
Proof.
It is enough to show that for any A ⊂ R with zero Lebesgue measure, P ( ν ⊤ Y ∈ A ) = 0 .
Let
A ⊆ R , then we can write
[Equation]
We proceed the proof by controlling the term P ( ν ⊤ Y ∈
A | ν ) .
We know that ν ̸ = 0 almost surely.
Now, since Y and ν are independent, let ν = w for a fixed w ∈ R n such that w ̸ = 0 , then we have that
[Equation]
where we defined A ′ :
= (cid:110) x
∥ w ∥ 2 : x ∈ A (cid:111) .
Now consider the change of basis matrix R
=
( v 1 , . .
. , v n ) ⊤ , such that v 1 = w ∥ w ∥ 2 , with RR ⊤ = I n .
By assigning ˆ Y = R ⊤ Y , we can write
[Equation]
Since we assume that Y is an absolutely continuous random variable, there exists M Y > 0 , such that sup
[Equation]
n f Y
[Equation]
y ) ≤ M Y almost surely, which allows us to write
[Equation]
Finally, it is straightforward to check that if A has a zero Lebesgue measure, then A ′ also has a a zero
Lebesgue measure, which gives P ( w ⊤ Y ∈ A ) = 0 .
Lemma A.2 (Lipschitz CDF) .
Let n ∈ N , ν ∈ R n / { 0 } and b ∈ R .
Let also Y be an absolutely continuous random variable with values in R n , with probability density function f Y .
Then the CDF of Z = ⟨ ν, Y ⟩ + b , namely F Z , is Lipschitz continuous.
More specifically
[Equation]
where M ′ = max z ∈ R f Z ( z ) .
Proof.
Since ν ̸ = 0 and Y is absolutely continuous, Z is also absolutely continuous with probability density f
Z (see Lemma A.1).
Furthermore, if r ′ ≤ r , we can write
[Equation]
Applying the same reasoning to the case when r ≤ r ′ concludes the proof.
Lemma A.3.
Let { X a } Ka =1 be K random variables with values in R d and such that they are allwith probability strictly less than one.
Define Σ
= K − 1 (cid:80) Ka
=1 E [ X a X ⊤ a ] and let Σ = USU ⊤ be its compact eigenvalue decomposition with U ∈ R d × r , S ∈ R r × r with 1 ≤ r ≤ d .
Assume that S is invertible.
Then, for any y ∈ ∪ Ka =1 Supp
( X a ) , we have UU ⊤ y
=
y and λ +min (Σ) ∥
y ∥ 22 ≤ y ⊤ Σ y , where λ +min (Σ) is the smallest non-zero eigenvalue of the matrix Σ .
Proof.
Let X be a random variable with the distribution P ( X ) =
K − 1 (cid:80) Ka
=1 P ( X a ) .
It is straightforward to check that Σ
= E [ XX ⊤ ] , and y ∈ Supp( X ) .
We can also write y =
y
+ y 2 where
y 2 ∈ Ker(Σ) :
= { z ∈ R d : Σ z = 0 } and y 1 ∈ Ker(Σ) ⊥ :
= { z ∈ R d : ⟨ z, x ⟩ = 0 , ∀ x ∈ Ker(Σ) } .
This implies that
[Equation]
Now, let f ( x ) =
( y ⊤ 2 x ) 2 .
Then, f ( x ) ≥ 0 , for any x ∈ R d and f ( y ) =
∥
y 2 ∥ 42 .
Furthermore, since, f ( x ) is a continuous function there exists ϵ > 0 , such that for any z ∈ B ( y, ϵ ) =
{ x ∈ R d : ∥ x − y ∥ 2 < ϵ } , f ( z ) ≥ ∥
y 2 ∥ 42 2 .
On the other hand, since y ∈ Supp( X ) , P ( X ∈ B ( y, ϵ )) > 0 .
Hence, we can write
[Equation]
therefore y 2 = 0 which implies that y ∈ Ker(Σ) ⊥ .
Since UU ⊤ y is the orthogonal projection of y onto Ker(Σ) ⊥
we conclude that y = UU ⊤ y , y ⊤ =
y ⊤ UU ⊤ and
[Equation]
B Proof of Lemma 4.
Proof.
If t < 3 then µ ˜ t = 0 and a t ∼ U [[ K ]] and the statement follows.
If t ≥ 3 , Let µ ∈ S :
= { µ ′ ∈ R d : µ ′⊤ B a ̸ = 0 ∀ a ∈ [ K ] } ,
ˆ r i,a = ⟨ µ, X i,a ⟩ and t ′ =
t − ˜ t − 1 .
Then by Lemma A.1 ˆ r i,a is absolutely continuous.
Given a permutation of indices j
= ( j 1 , . . . , j t ′ ) where j i ∈ { ˜ t + 1 , . . .
, t } , for i ∈ [ t ′ ] .
Let Ω a be the set of the events of { X i,a } ti = ˜
t +1 and P be the set of all permutations of the indices { ˜ t + 1 , . . .
, t } .
Consider the event
[Equation]
Since { ˆ r i,a } ti =˜ t
+1 are absolutely continuous, we have for all k ̸
= i , P (ˆ r j i ,a = ˆ r j k ,a ) =
and this yields Ω a = ∪ j ∈ P E a,j and E a,j ∩ E a,j ′ = ∅ for all j ̸ = j ′ .
Furthermore, since { ˆ r i,a } ti =˜
t +1 are i.i.d.
we have that p a :=
P ( E a,j ) =
P ( E a,j ′ ) for all j ̸ = j ′ .
In particular, since | P | = t ′ !
we have p a = 1 / ( t ′ !) .
Let ϕ a =
( t ′ − 1) − 1 (cid:80) t − 1 i =
˜ t +1 1 { ˆ r i,a < ˆ r t,a } .
Let b ∈ { 0 , . .
, t − 1 } and let P b = { j ∈ P : j b +1 = t } .
We have that | P b | = ( t ′ − 1)!
and
[Equation]
[Equation] As a consequence, for all a ∈ [ K ] , ϕ a is uniform over { 0 , 1 / ( t ′ − 1) , . . .
Since { r i,a } i ∈ [ t +1] ,a ∈ [ K ] are mutually independent we have that { ϕ a } a ∈ [ K ] are i.i.d.
discrete uniform random variables.
As a consequence, let ˆ a = U (cid:104) argmax a ′ ∈
[ K ] ˆ ϕ a (cid:105) we have that P (ˆ a = a ) = 1 /K .
Using the definition of ˆ a we have
[Equation]
where the last equality is derived by the fact that by the construction of µ ˜ t , P ( µ ˜ t ∈ S ) = 1 .
C Proofs of Results in Sec.
The following result is used in the Proof of Lemma 5.2.
Its proof is obtained by using the
Dvoretzky–Kiefer–Wolfowitz-Massart inequality [15, 30] combined with a union bound.
Lemma C.1.
Let Assumption A (ii) hold and ˆ F t,a ( r ) , and µ ˜ t to be generated by Alg. 1.
Let Z a :=
⟨ µ ˜ t , X a ⟩ and denote with F Z a its CDF, conditioned on µ ˜ t .
Then, with probability at least 1 − δ we have that for all 3 ≤ t ≤ T
[Equation]
Proof.
Let 3 ≤ t ∈ T and recall that ˜ t =
⌊ ( t − 1) / 2 ⌋ .
Note that from Assumption A(ii), for all a ∈ [ K ] , {⟨ µ ˜ t , X i,a ⟩} t − 1 i =
˜ t
+1 are i.i.d copies of Z a , conditioned on µ ˜ t .
Since ˆ F t,a ( r ) is the empirical
CDF of Z a conditioned on µ ˜ t , we can apply the Dvoretzky–Kiefer–Wolfowitz-Massart inequality [15, 30] to obtain
[Equation]
Therefore, since ˜ t ≤ ( t − 1) / 2 we deduce that P [ E t,a ] ≤ δ ′ , where
[Equation]
Consequently, by applying a union bound we obtain
[Equation]
Finally, by substituting δ ′ =
δ/ ( KT ) and computing the probability of the complement of ∪ Ti =1 ∪ Ka =
E t,a , we obtain the desired result.
C.1 Proof of Lemma 5.
Proof.
For every a ∈ [ K ] , by Assumption A(iv), we have that X
a =
B a Y a + c a where Y a ∈ R d a is absolutely continuous with density f a .
Let ν ∗ :
= µ ∗⊤ B a , ν := µ ⊤ B a , b ∗ :
= ⟨ µ ∗ , c a ⟩ , b := ⟨ µ, c a ⟩ .
Then we have
[Equation]
and
[Equation]
From Assumption A(iv) we also have that ν ∗ ̸ = 0 , hence, by applying Lemma A.2 with ν = ν ∗ and
Y =
Y a and by taking the maximum over a ∈ [ K ] , the statement (i) follows.
We now prove (ii).
Since Y a is absolutely continuous we can write for any r ∈ R
[Equation]
Now, by adding and subtracting q ( y ) :
= ⟨ ν ∗ − ν, y ⟩ + b ∗ − b and letting r ′ :
= r − b ∗ we have
[Equation]
By Cauchy-Schwarz inequality, for any y ∈ Supp( Y a ) , we get
[Equation]
where we defined ∥ x max ∥ ∗ :=
max x
[Equation]
Ka
[Equation]
X
[Equation]
∥ x ∥
[Equation]
=
[Equation]
B
[Equation]
y + c a ∥ ∗ .
We now let κ = ∥ µ ∗ − µ ∥∥ x max ∥ ∗ , and note that
[Equation]
To control the above integral, we provide a proper change of variables.
To this end, since by the assumption ν ∗ ̸ = 0 , we let { v 1 , . . .
, v d a } be an orthonormal basis of R d a , with v 1 = ν ∗ /
∥ ν ∗ ∥ 2 .
Moreover, let R =
( v 1 , . .
. , v d a ) to be the corresponding change of basis matrix.
Then, for all y ∈ R d a , we can always write y =
R ˆ y , where ˆ y i =
⟨ y, v i ⟩ , with ˆ y 1 = ⟨ ν ∗ ,y ⟩ ∥ ν ∗ ∥ 2 .
Hence we denote with ˆ Y a = R ⊤ Y a which now has the first coordinate parallel to ν ∗ .
Using the change of variables formula for multivariate integrals and noting that we are applying a rotation and hence | det ( R ) | =and f R ˆ Y a ( R ˆ y )
=
f ˆ Y a (ˆ y ) / | det ( R ) |
= f ˆ Y a (ˆ y )
we get
[Equation]
Let z =
(ˆ y 2 , . .
, ˆ y d a ) .
By Fubini’s Theorem, and with the convention that f ˆ Y a (ˆ y 1 , z ) =
f ˆ Y (ˆ y 1 , z 1 , . . .
, z d a − 1 ) we have
[Equation]
where f ˆ Y 1 (ˆ y 1 ) :
= (cid:82)
z ∈ R da − 1 f ˆ Y (ˆ y 1 , z 1 , . . .
, z d a − 1 ) d z is the marginal density of ˆ Y 1 = ⟨ ν ∗ ,Y a ⟩ ∥ ν ∗ ∥ 2 , and we highlight that ˆ Y 1 =
( Z a − b ∗ ) /
∥ ν ∗ ∥ 2 .
Finally note that
[Equation]
which yields
[Equation]
and (ii) follows by substituting the definition of κ .
C.2 Proof of Proposition 5.
Proof.
Recall the definition of Alg. 1.
For any a ∈ [ K ] let ˆ r t,a = µ ⊤ ˜ t X t,a , which is the estimated reward for arm a , at round t .
Note that µ ˜ t and X t,a are independent random variables.
Furthermore, denote with F ˆ r t,a the CDF of ˆ r t,a conditioned on µ ˜ t , and let ϕ t,a :=
F ˆ r t,a (ˆ r t,a ) , and ˆ ϕ t,a :=
ˆ F t,a (ˆ r t,a ) .
Now, by the definition of the algorithm, we have
[Equation]
where we introduced C t :
= argmax a ∈ [ K ] ˆ ϕ t,a .
Let ϵ t > 0 and continue the analysis conditioning on the events where sup a ∈ [ K ] | ϕ t,a − ˆ ϕ t,a | ≤ ϵ t .
Then, we can write
P
[Equation]
, where in the first inequality we considered the case when a ∈ C t and | C t | = 1 .
In the second inequality we considered the worst case scenario where ˆ ϕ t,a = ϕ t,a − ϵ t and ˆ ϕ t,a ′ =
ϕ t,a ′ + ϵ t .
Recall that by the construction of the algorithm µ ˜ t =
V
[Equation]
for all
[Equation]
[ K ] , the additive noise (1 / √ d ˜ t ) γ ˜ t assures that µ ⊤ ˜ t B a ̸ = 0 , almost surely.
Therefore, by Lemma A.
ˆ r t,a = ⟨ µ ˜ t , X t,a ⟩ conditioned on µ ˜ t is absolutely continuous.
Assumption A(iii) and [9, Theorem 2.1.10] yield that { ϕ t,a } a ∈ [ K ] are independent and uniformly distributed on [0 , 1] and in turn that
[Equation]
We continue by computing an ϵ t for which sup a ∈ [ K ] | ϕ t,a − ˆ ϕ t,a | ≤ ϵ t holds with high probability.
Observing that, conditioned on µ ˜ t , ˆ F t,a is the empirical CDF of F ˆ r t,a , we can use the
Dvoretzky–Kiefer–Wolfowitz-Massart inequality to obtain, for any a ∈ [ K ] , t ≥ 3 , and s ≥
[Equation]
Now, let τ 0 :
= 3 + 8 log 3 / 2 (cid:0) 5 K e /δ (cid:1) / (cid:0) 1 − K √ c (cid:1) 3 .
By applying the union bound, we can write
P
[Equation]
Since ˜ t =
⌊ t − 12 ⌋ , it is straightforward to check that
[Equation]
Now, for any δ ∈ (0 , 1) , by assigning s = (cid:113) log(4 K ( τ
[Equation]
− 1) /δ )
τ
[Equation]
−, we get
P  
[Equation]
where from τ 0 ≥ 3 , δ < 1 = ⇒ 4 K ( τ 0 − 1) /δ
≥ 8 ≥ e 2 = ⇒ log (4 K ( τ 0 − 1) /δ )
≥
we obtain the last inequality.
From (6), it follows that
[Equation]
Moreover, form (7), by letting ϵ t = (cid:113) log(4 K ( τ 0 − 1) /δ )
τ 0 −, with probability at least 1 − δ 4 , we have
[Equation]
For the term (I) in the above, using log( x ) ≤ log(5 e / 4) x 1 / 3 and x ≥ x 2 / 3 for any x ≥ 1 we deduce that
[Equation]
Now, by substituting τ 0 = 3 + 8 log 3 / 2 (cid:0) 5 K e /δ (cid:1) / (cid:0) 1 − K √ c (cid:1) 3 , we get that (I) ≤ 14 (1 − K √ c ) 2 and conclude the proof by plugging this inequality in (8).
C.3 Proof of Lemma 5.
We start by establishing some required lemmas.
Lemma C.2.
Let Σ , τ 1 , τ 2 be defined in Lemma 5.3, τ 3 = max ( τ 1 , τ 2 ) , Σ
= USU ⊤ be the compact eigenvalue decomposition of Σ , with U ∈ R d × r , S ∈ R r × r is a diagonal matrix with non-zero diagonal elements, and U ⊤ U
= I r .
Denote ˆ S t
= (cid:80) ˜
ti =
1 U ⊤ X
i,a i X ⊤ i,a i U, where for i ∈
[˜ t ] , a i is given by Alg.
Then with probability at least 1 − δ 2 , for any t ≥ 2 τ 3 + 3 we have
[Equation] [Equation]
Proof.
Let ˜ S t 0 :
= (cid:80) ˜
ti =1 E (cid:2)
U ⊤ X
i,a i X ⊤ i,a i U |H
− i − 1 (cid:3) .
First, note that for any τ 3 ≤ i ≤ ˜ t , we can write
[Equation]
where the last equality holds based on the fact that X i,a i X ⊤ i,a i conditioned on a i , is independent from H − i − 1 .
Then, since t ≥ 3 + 64 K 3 log 3 / 2 (cid:0) 5 K e /δ (cid:1) , by utilizing Proposition 5.1 with c =and noting that 1 / (1 − K (cid:112) 1 / 2) ≤ 2 K for all K ≥ 1 , with probability at least 1 − δ 4 , we have
P ( a i = a | H − i − 1 ) ≥ 12 K .
Therefore, with probability at least 1 − δ 4 , we obtain
[Equation]
and consequently, with probability at least 1 − δ 4 we have
[Equation]
where in the last two displays we used the concavity attribute of the function λ min ( · ) .
Note that { X i,a i } ∞ i =1 , is an adaptive sequence with respect to the filtration (cid:8) H − i (cid:9) ∞ i =0 , with
[Equation]
for any i ∈ [˜ t ] .
Let ι = ˜ t − τ 3 .
Now, by invoking [33, Theorem 3.1] (with δ = 12 and µ = λ +min (Σ) / 2 , where δ, µ are constants that appear in the latter theorem), we have
[Equation]
d ·   e
[Equation]
12  ιλ +min(Σ
[Equation]
≤ q , where we introduced q = d · exp( − ιλ +min (Σ) 27 L 2 ) , and we used the inequality e − 12 · 12 − 12 ≤ e − 427 .
Note that since ˜ t ≥ τ 3 = 54 L 2 λ +min (Σ) log( 4 dδ ) , we have q ≤ δ 4 .
Let p =
P [ λ min ( ˜ S t 0 ) ≥ ιλ
+min (Σ)] , then we can write
[Equation]
and accordingly
[Equation]
where we used p ≥ 1 − δ 4 , which follows from (9).
Substituting ι = ˜ t − τ 3 gives the final result.
Lemma C.3.
Let x ∈ ∪ Ka
=1 Supp
( X a ) and τ 3 be defined in Lemma C.2, then with probability at least 1 − δ 2 , for all t ≥ 2 τ 3 +
we have
[Equation]
Proof.
Note that if x = 0 it is straightforward to check that the statement holds.
So without loss of generality we assume that x ∈ S , where S = ∪ Ka =1 Supp
( X t,a ) − { 0 } .
Consider the compact singular value decomposition Σ
= USU ⊤ where U ∈ R d × r , S ∈ R r × r is a diagonal matrix with non-zero diagonal elements (due to Assumption A(iv)) and U ⊤ U
= I r .
Denote ˆ S ˜ t =
U ⊤ ˆΣ ˜ t U .
For any x ∈ S we have from Lemma A.3 that UU ⊤ x = x , and x ⊤ UU ⊤ = x ⊤ .
First, we claim that
[Equation]
To prove the above claim, it is enough to show that
[Equation]
Note that
[Equation]
With similar steps one can show that U ⊤ (ˆΣ ˜ t
+ λ I d ) − 1 U ( ˆ S ˜ t
+ λ I r )
=
I r , and therefore U ⊤ (ˆΣ ˜ t + λ I d ) − 1 U
=
( ˆ S ˜ t
+ λ I r ) − 1 .
By exploiting this fact, we can write
[Equation]
where the second and last equations are results of Lemma A.3, and consequently
[Equation]
On the other hand, from Lemma C.2, with probability at least 1 − δ 2 , we have
[Equation]
Finally, by combining (10) and (11) with probability at least 1 − δ 2 we have
[Equation]
Lemma C.4.
With probability at least 1 − δ 4 , for all t ≥ 3 we have
[Equation]
Proof.
Recall that by the definition of Alg. 1, we have µ ˜ t
=
V − 1 ˜ t X ⊤ 1:˜ t r
[Equation]
˜ t +
[Equation] [Equation]
t .
Therefore, we can write
[Equation]
We proceed the proof by providing upper bounds for (I) and (II).
For (I), by invoking [1, Theroem 2], with probability at least 1 − δ 8 , for all t ≥ 3 , which implies ˜ t ≥ 1 we have
[Equation]
On the other hand, since ρ ≤ 1 , for term (II) we have
[Equation]
Thus, by applying the union bound with probability at least 1 − δ 8 , for all t ≥ 3 we have
[Equation]
) (cid:113) d log((8 + 8˜ t max( L 2 /λ, 1)) /δ ) .
Proof of Lemma 5.3.
Recall that τ 3 = max( τ 1 , τ 2 ) .
From Lemma C.3, with probability at least 1 − δfor all t ≥ 2 τ 3 +
we have ∥
[Equation]
From Lemma C.4, with probability at least 1 − δ 4 for all t ≥
[Equation]
Thus, combining Lemmas C.3 and C.4, with probability at least 1 − 3 δ 4 for all t ≥ 2 τ 3 + 3 we have ∥
[Equation]
≤
[Equation]
By the fact that t ≥ 4 τ 3 + 3 , we have ˜ t ≥ 2 τ 3 , which implies
[Equation]
2˜ t .
We conclude the proof by using the inequality ˜ t ≥ t − 32 ≥ t 8 , for all t ≥ 4 .
C.4 Proof of Theorem
Proof.
Combining Lemma 5.2 with Lemma 5.3 and using 1 / ( t − 1) ≤ 3 / (4 t ) for all t ≥ 4 we obtain, with probability at least 1 − δ and for all τ ≤ t ≤ T
[Equation]
+ 48 ML
[Equation]
By summing up the last inequality, with probability at least 1 − δ we get
T (cid:88)
[Equation]
where the last display is obtained by the inequality (cid:80)
Tt =1 t − 12 ≤ 2 T 12 .
On the other hand, for t ∈ [ T ] ,
F a ∗ t ( ⟨ µ ∗ , X t,a ∗ t ⟩ ) − F a t ( ⟨ µ ∗ , X t,a t ⟩ ) ≤ 1 , and we can write τ (cid:88)
[Equation]
By combining (12) and (13), we conclude the proof.
D Experiments
In this section we include additional details on the simulation experiments in Sec. 6 and an experiment on the US census data.
D.1 Additional Details on the Simulation
We use the following value for the underlying linear model used in Fig. 1.
[Equation]
Each slice of 4 coordinates of µ ∗ affects a different group.
Furthermore, since each coordinate of
Y a follows a standard uniform distribution, the resulting reward distributions for each group follow weighted variants of the Irwin-Hall distribution [18].
D.2 Experiments on US Census data
In this section, we present an experiment performed using the US Census data and the FalkTables library 4 [13].
In particular we construct a dataset with features similar to the UCI Adult dataset but where the target is the person’s income instead of the binary variable indicating if the income is more or less than 50 K dollars.
We use this target as a possibly inaccurate proxy for how well a candidate will perform on the job, hence it will be used as the noisy reward for the bandit problem.
Setup and Preprocessing.
To setup the bandit problem, we construct 2 datasets, namely D 1 and
D 2 , by selecting 500 K males and 500 K females random samples first from the 2017 US Census
Survey, to assemble D 1 , and then from the 2018 survey to assemble D 2 .
We use D 1 to find mean and standard deviation for each feature and also for the target.
After that we normalize features and target from D 2 by subtracting the mean and dividing by the standard deviation previously computed on D 1 .
We then construct µ ∗ as a ridge regression estimate on the samples from D 2 with the regularization parameter equal to 10 − 8 .
The regression vector µ ∗ will be used to compute the (true) rewards for the samples.
We construct the bandit problem with K = 2 arms/groups which correspond to the gender identities male and female.
At each round, the context vectors of one male and one female candidate are sampled from D 2 and after one of the two is selected by the policy, its corresponding noisy reward (i.e. its income) is received by the agent.
Baselines.
We compare our method, namely Fair-greedy (Alg. 1), with the following baselines.
• Uniform Random , which selects an arm uniformly at random at each round.
• OFUL [1], with exploration parameter set to 0 .
[Figure]
• Greedy , which computes the ridge regression estimate for the reward vector using all the selected contexts and noisy rewards in the history and then selects the arm maximising the estimated reward.
• Fair-greedy (Oracle CDF) , which is a variant of Fair-greedy where all the selected contexts and noisy rewards in the history are used to compute the ridge regression estimate and the empirical CDF of each group is replaced by the true CDF.
• Fair-greedy (Oracle rewards) , which is another variant of Fair-greedy where the ridge regression estimate is replaced by the true reward model µ ∗ and all contexts in the history are used to compute the empirical CDF for each group.
Note that the last two methods are oracle methods because they rely either on the true CDF of the rewards for each group or on µ ∗ , which are unknown to the agent.
All methods using a ridge regression estimate have the regularization parameter set to 0 .
We observed that varying this parameter did not affect much the relative performance of the methods
[Figure]
Results
The results and the reward distributions are illustrated in Fig.
We note that in this case,
Greedy performs much better than OFUL, which appears to be too conservative for this problem.
In particular, the standard pseudo-regret of Greedy is unrivaled after 1000 rounds.
Furthermore, since there is a large overlap in the distributions of rewards, our Fair-greedy policy performs much better than the Uniform Random policy even in terms of standard pseudo-regret, while it outperforms all non-oracle methods in terms of fair pseudo-regret.
As expected, the oracle methods both achieve a lower fair pseudo-regret than Fair-greedy , and we note that knowing only the underlying model µ ∗ is significantly more advantageous than knowing only the CDF for each group.
E Multiple Candidates for Each Group
This section contains a rigorous treatement of the content in Sec.
We consider the more realistic case where contexts from a given arm do not necessarily belong to the same group.
In particular, we assume that at each round t , the agent receives tuples { ( X t,a , s t,a ) } Ka =1 , where s t,a ∈
[ G ] is the sensitive group of the context X t,a ∈ R d
and G is the total number of groups.
After that the agent selects action a t and subsequently receives the noisy reward ⟨ µ ∗ , X t,a ⟩ + η t
Note that we recover the original setting discussed in Sec. 3 when G = K and s t,a = a for every a ∈ [ K ] , t ∈ N .
A more realistic scenario is when { ( X t,a , s t,a ) } Ka =1 are i.i.d., and the distribution represents e.g. the underlying population of candidates, where P ( s t,a = i ) is the same for all a ∈ [ K ] and can be small when the group i is a minority.
The following analysis applies to both cases.
We impose the following assumption, which is a natural extension of Assumption A.
Assumption B. Let µ ∗ ∈ R
d be the underlying reward model.
We assume that:
(i)
The noise random variable η t is zero mean R -subgaussian, conditioned on H t − 1 .
(ii)
For any a ∈ [ K ] , let ( X a , s a ) be a random variable with values in R d × [ G ] and ∥ X a ∥ 2 ≤ L almost surely.
{ ( X t,a , s t,a ) }
Tt =1 are i.i
d. copies of ( X a , s a ) .
X a conditioned to s a =
i is a copy of the random variable ˆ X i which is independent on the arm, for every a ∈ [ K ] .
(iii)
For every a ∈ [ K ] X a conditioned to s a is independent from ( X a ′ , s a ′ ) for any a ′ ̸
= a .
(iv) For every i ∈ [ G ] , then there exist d i ≥ 1 , an absolutely continuous random variable Y i with values in R d
i admitting a density f i , B i ∈ R
d × d i and c i ∈ R
d such that B ⊤ i B i =
I d i , ˆ X i
=
B i Y i
+ c i and µ ∗⊤ B i ̸
= 0 .
We define F ( r, i ) =
P ( ⟨ µ ∗ , ˆ X i ⟩ ≤ r ) = P ( ⟨ µ ∗ , X a ⟩ ≤ r | s a = i ) for any r ∈ R , i ∈ G .
Hence we can extend the definition of group meritocratic fairness as follows.
Definition E.1 (GMF policy) .
a policy { a ∗ t } ∞ t =1 is group meritocratic fair (GMF) if for all t ∈ N , a ∈ [ K ] it satisfies
[Equation]
The fair pseudo-regret is now defined as
[Equation]
We can adapt Proposition 3.1 to this setting as follows.
Proposition E.1 (GMF policy satisfies history-agnostic demographic parity ) .
Let {⟨ µ ∗ , X a ⟩} Ka = conditioned to { s a } Ka =1 be independent and absolutely continuous and for every a ∈ [ K ] , t ∈ N , let ( X t,a , s t,a ) be an i.i.d.
copy of ( X a , s a ) .
Then for every t ∈ N , {F ( ⟨ µ ∗ , X t,a ⟩ , s t,a ) } Ka =1 conditioned to { s t,a } Ka =1 are i.i.d.
uniform on [0 , 1] and
[Equation]
15) for any GMF policy { a ∗ t } ∞ t =1 .
Proof.
Let ψ a :=
F ( ⟨ µ ∗ , X t,a ⟩ , s t,a ) .
From the assumptions { ψ a } K a =1 conditioned to { s t,a } K a = are i.i.d random variables, independent from H − t − 1 , with uniform distribution on [0 , 1] (see [9,
Theorem 2.1.10]).
Let ˜ P
=
P ( · | { s t,a } Ka =1 , H − t − 1 ) , we have that ∀ a 1 , a 2 ∈ [ K ] :
˜ P ( ψ a 1 = ψ a 2 ) = 0 , ˜ P ( a ∗ t = a | H − t − 1 ) =
˜ P ( a ∗ t = a ) and ˜ P
[Equation]
Let S t, ̸ a = { s t,a : a ∈ [ K ] /a } , then the statement follows from
[Equation]
Proposition E.1 states that the probability of selecting an arm does not change based on group membership.
Fair-Greedy V2 in Alg.
2 is the extension of the fair-greedy policy to this new setting.
Algorithm
1: Requires regularization parameter λ > 0 , and noise magnitude ρ ∈ (0 , 1] 2: for t = 1 . . .
T do 3:
Receive { ( X t,a , s t,a ) } Ka = 4:
Set ˜ t =
⌊ ( t − 1) / 2 ⌋ , X 1:˜ t =
( X 1 ,a 1 , . . .
, X ˜ t,a ˜ t ) ⊤ , r 1:˜ t =
( r 1 ,a 1 , . .
, r ˜ t,a ˜ t ) .
If ˜ t = 0 set µ ˜ t = 0 , else let V ˜ t := X ⊤ 1:˜ t X 1:˜ t
+ λ I d , generate γ ˜ t ∼ N (0 , I d ) and compute
[Equation]
˜ t .
For each a ∈ [ K ] , let i :=
s t,a and N t,i = (cid:80) t − 1 j = ˜ t +1 (cid:80) K a ′ =1 1 { s j,a ′ =
i } , compute ˆ F t ( ⟨ µ ˜ t , X t,a ⟩ , i ) :
= N − 1 t,i t − 1 (cid:88) j =˜ t +
K (cid:88) a ′ = 1 {⟨ µ ˜ t , X j,a ′ ⟩ ≤ ⟨ µ ˜ t , X t,a ⟩} 1 { s j,a ′ = i } .
Sample action a t ∼ U (cid:2) argmax a ∈ [ K ] ˆ F t ( ⟨ µ ˜ t , X t,a ⟩ , s t,a ) (cid:3) .
Observe noisy reward r t,a t = ⟨ µ, X t,a t ⟩ + η t .
9: end for
Notice that the number of contexts used for the CDF approximation for group i ∈ [ G ] is now the random variable N t,i .
Furthermore, we are now using contexts from all the arms to estimate the
CDFs, which as we will see it can improve the dependency on K in the fair pseudo-regret bound.
We observe that the information averaged demographic parity property of Lemma 4.1 does not transfer directly to Fair-Greedy V2, because at each round, there can be a different number of candidates for each group.
However, as we will see, the regret is still similar to the original case.
The following Lemma establishes an high probability lower bound on N t,i .
Lemma E.1.
Let q
K
= min i ∈
[ G ] (cid:80) Ka =1 P ( s a = i ) and let
R = 1 {∃ a ∈ [ K ] such that ∀ i ∈
[ G ] P ( s a = i ) < 1 } .
R = 1 means that the sensitive attribute is random for at least one arm, while is deterministic if
R = 0 .
Then, let α = R b
+ (1 − R ) with b ∈ (0 , 1) and t N :=
+ R⌈
[Equation]
K log( GT/δ ) ⌉ , with
N t,i defined at Line 6 of Alg.
2 and,without loss of generality, q K > 0 .
For simiplicity we let R x =when R = 0 , x = ∞ .
We have that with probablity at least 1 − R δ , for every t ∈ { t N , . . .
, T }
[Equation]
Proof.
If R = 0 , then P ( s a = i )
=
1 { s a = i } and the result follows.
If R = 1 instead, note that for every i ∈ [ G ] we have that
[Equation]
Applying the Chernoff bound we have that with probability at least 1 − δ , for all t > t N
[Equation]
and the statement follows
Let S T ( t N , α ) :
= (cid:8) {{ s t,a } Ka =1 } Tt =1 : min i N t,i ≥ ( t − ˜ t − 1) αq K for all t N ≤ t ≤ T (cid:9) be the event when Lemma E.1 is satisfied.
We can then proceed the analysis assuming that S T ( t N , α ) holds.
Noticing that the maximum number of approximate CDFs to be computed at each round is G we can adapt Lemma C.1 as follows.
Lemma E.2.
Let Assumption B (ii) hold and ˆ F t ( r, i ) , and µ ˜ t to be generated by Alg. 2.
Let Z i := ⟨ µ ˜ t , ˆ X i ⟩ and denote with F Z i ( · ) its CDF, conditioned on µ ˜ t .
Then, if the event S T ( t N , α ) is satisfied, with probability at least 1 − δ we have that for all t N ≤ t ≤ T
[Equation]
Then, following the steps in Lemma 5.2, we obtain the following bound on the instantaneous regret.
Lemma E.3 (Instant regret bound) .
Let Assumption B (ii)(iv) hold and a t to be generated by Alg. 2.
Then, if the event S T ( t N , α ) is satisfied, with probability at least 1 − δ/ 4 , for all t such that t N ≤ t ≤ T we have
[Equation]
where ∥ x max ∥
[Equation]
˜ t
[Equation]
Proof sketch.
Uses the decomposition in the proof of Lemma 5.2, then Lemma E.2 and a version of
Lemma 5.1 adapted to this more general setting.
We can bound ∥ µ ∗ − µ ˜ t ∥ V ˜ t using the confidence bounds in OFUL [1].
To bound ∥ x max ∥ V − 1 ˜ t instead, we first provide an adaptation of Proposition 5.1, which guarantees sufficient exploration of all arms.
The proof is very similar to that of Proposition 5.1 and we report it here for completeness.
Proposition E.2.
Let Assumption A hold, a t be generated by Alg.
1 and c ∈ [0 , 1) .
Then, if
S T ( t N , α ) is satisfied, with probability at least 1 − δ/ 4 , for all a ∈ [ K ] and all t ≥ max (cid:0) t N , 3 + 8 log 3 / 2 (cid:0) 5 G e /δ (cid:1)(cid:0) 1 − K √ c (cid:1) − 3 ( q K α ) − 3 / 2 (cid:1) we have
[Equation]
where we recall that H − t = ∪ ti =1 (cid:8) { ( X i,a , s i,a ) } Ka =1 , r i,a i , a i (cid:9) .
Proof.
Recall the definition of Alg. 2.
For any a ∈ [ K ] let ˆ r t,a = µ ⊤ ˜ t X t,a , which is the estimated reward for arm a , at round t .
Note that µ ˜ t and X t,a are independent random variables.
Furthermore, denote with F ˆ r t,a ( · , s t,a ) the CDF of ˆ r t,a conditioned on µ ˜ t and s t,a , and let ϕ t,a :=
F ˆ r t,a (ˆ r t,a , s t,a ) , and ˆ ϕ t,a :=
ˆ F t (ˆ r t,a , s t,a ) .
Let C t := argmax a ∈ [ K ] ˆ ϕ t,a .
Now, by the definition of the algorithm, we have
P ( a t = a | { s t,a } Ka =1 , H − t − 1 ) =
K (cid:88) m =  m P ( a ∈ C t , | C t | = m | H − t − 1 ) ,
Let ϵ t > 0 and ˜ P ( · )
= P ( · | { s t,a } Ka =1 , H − t − 1 , sup a ∈ [ K ] | ϕ t,a − ˆ ϕ t,a | ≤ ϵ t ) .
Then, we can write ˜ P ( a t = a ) ≥ ˜ P
(ˆ ϕ t,a > ˆ ϕ t,a ′ , ∀ a ′ ̸ = a ) ≥ ˜ P ( ϕ t,a > ϕ t,a ′ + 2 ϵ t , ∀ a ′ ̸
= a ) , where in the first inequality we considered the case when a ∈ C t and | C t | = 1 .
In the second inequality we considered the worst case scenario where ˆ ϕ t,a = ϕ t,a − ϵ t and ˆ ϕ t,a ′ =
ϕ t,a ′ + ϵ t .
Recall that by the construction of the algorithm µ ˜ t = V − 1 ˜ t
X ⊤ 1:˜ t r 1:˜ t + (1 / √ d ˜ t ) · γ ˜ t .
for all i ∈ [ G ] , the additive noise (1 / √ d ˜ t ) γ ˜ t assures that µ ⊤ ˜ t B i ̸ = 0 , almost surely.
Therefore, by Lemma A.
ˆ r t,a = ⟨ µ ˜ t , X t,a ⟩ conditioned on µ ˜ t is absolutely continuous.
Assumption A(iii) and [9, Theorem 2.1.10] yield that { ϕ t,a } a ∈ [ K ] conditioned to { s t,a } Ka =1 are independent and uniformly distributed on [0 , 1] and in turn that
[Equation]
We continue by computing an ϵ t for which sup a ∈ [ K ] | ϕ t,a − ˆ ϕ t,a | ≤ ϵ t holds with high probability.
Observing that, conditioned on µ ˜ t and { s t,a } Ka =1 , ˆ F t,a ( · , s t,a ) is the empirical CDF of F ˆ r t,a ( , s t,a ) , we can use Lemma E.1 and the Dvoretzky–Kiefer–Wolfowitz-Massart inequality to obtain, for any a ∈ [ K ] , t ≥ t N , and s ≥
[Equation]
Now, let τ 0 :
=
max (cid:0) t N , 3 + 8 log 3 / 2 (cid:0) 5 G e /δ (cid:1)(cid:0) 1 − K √ c (cid:1) − 3 ( αq K ) − 3 / 2 (cid:1) .
By applying the union bound and noticing that we have max of G CDFs and approximate CDFs, we can write
[Equation]
Since ˜ t =
⌊ t − 12 ⌋ , it is straightforward to check that
[Equation]
Now, for any δ ∈ (0 , 1) , by assigning s = (cid:113) log(4 G ( τ 0 − 1) /δ )
( τ 0 − 1) αq K , we get
P
[Equation]
[Equation] where from τ 0 ≥ 3 , δ < 1 = ⇒ 4 G ( τ 0 − 1)
/δ ≥ 8 ≥ e 2 = ⇒ log (4 G ( τ 0 − 1) /δ )
≥
we obtain the last inequality.
From (16), it follows that
[Equation]
Moreover, form (17), by letting ϵ t = (cid:113) log(4 G ( τ 0 − 1) /δ )
[Equation]
τ
[Equation]
− 1) αq K , with probability at least 1 − δ 4 , we have inf
[Equation]
For the term (I) in the above, using log( x ) ≤ log(5 e / 4) x 1 / 3 and x ≥ x 2 / 3 for any x ≥ 1 we deduce that
[Equation]
Now, since τ 0 ≥ 3 + 8 log 3 /
[Equation]
5 G e /δ (cid:1)(cid:0) 1 − K √ c (cid:1) − 3 ( αq K ) − 3 / 2 , we get that (I) ≤ 14 (1 − K √ c ) 2 and conclude the proof by plugging this inequality in (18).
Furthermore, for fixed t , let ˜ E
= E [ · | H − t − 1 ] and ˜ P = P [ · | H − t − 1 ] .
Note that if the assumptions of
Proposition E.2 are satisfied, then
[Equation]
where we applied Proposition E.2 in the last line.
We can bound ∥ x max ∥ V − 1 ˜ t in Lemma E.3 in the same way as in Lemma C.3 using (19) with c = 1 / 2 when needed in the proof of Lemma C.2.
Combining the previous results we obtain the following regret bound.
Theorem 2.
Let Assumption B hold, a t be generated by Alg.
2 and Σ :=
G − 1 (cid:80) Gi =1 E [ ˆ X
i ˆ X ⊤
i ]
Then, with probability at least 1 − δ , for any T ≥ 1 we have
[Equation]
where δ = δ 1 + R δ 2 , τ = 4 max
τ 1 , τ 2 , R τ 3 ) + 3 and τ 1 = 32 K(
[Equation]
where q K , R and α are defined in Lemma E.1 and we use the convention R τ 3 = 0 if R = 0 , τ 3 = ∞ .
Hence
[Equation]
Proof Sketch.
First, assume S T ( t N , α ) holds and use a similar strategy of Thm.
1 to get a bound w.p.
at least 1 − δ 1 .
Then combine this result with Lemma E.1.
Notice that in the case where each arm corresponds to a different sensitive group, i.e. when G = K , s a = a and therefore q K = 1 , R = 0 and α = 1 , we recover Thm. 1.
Moreover, we have the following corollary which shows an advantage for higher number of arms compared to the bound in
Thm. 1 when { ( X a , s a ) } Ka =1 are i.i.d..
Corollary E.1.
Let { ( X a , s a ) } Ka =1 be i.i.d.
and q min :=
min i ∈ [ G ] P ( s a = i ) G .
If Assumption B holds and a t is generated via Alg.
2 we have that with probability at least 1 − δ , for any T ≥ 1 and α ∈ (0 , 1) we have that
[Equation]
where δ =
δ 1 + δ 2 , τ = 4 max( τ 1 , τ 2 , τ 3 ) + 3 and τ 1 = 32( KG ) 3 /( αq min ) 3 / 2 log
[Equation] [Equation]
[Figure]
Hence
[Equation]
Note that in Cor.
E.1, q min > 0 without loss of generality and q min = 1 if
and only if each group has the same probability of being sampled.
Furthermore q min /G is the probability that a context belongs to the less common group, which depends on the problem at hand.
Note that there is an advantage compared to Thm. 1 in terms of number of arms when K > G .
This is because context coming from all arms can be use to estimate the CDF of a given group.
E.1 Additional Details on the US Census Experiments
This experiment is introduced in Sec.
7 and similarly to that of Appendix D.2, is performed using the US Census data.
However, candidates are sampled from the original dataset at random together with their sensitive group (their ethnicity).
Hence, we use Fair-greedy V2 (Alg. 2).
Differently from
Appendix D.2 where we use the target income as noisy reward, here we add artificial noise with standard deviation 0 .
2 directly to the true reward.
Setup and Preprocessing.
To setup the bandit problem, we construct two datasets: D 1 and D 2 .
We first load all the data from the 2017 US Census Survey to assemble D 1 , and then from the survey to assemble D 2 .
Then we retain only candidates from 6 ethnic groups containing at least 5 × 10 3 candidates, in order to accurately compute the true CDF for each group.
We use D 1 to find mean and standard deviation for each feature and also for the target.
After that we normalize features and target of D 2 by subtracting the mean and dividing by the standard deviation previously computed on D 1 .
We then construct µ ∗ as a ridge regression estimate on the samples from D 2 with the regularization parameter equal to 10 − 8 .
The regression vector µ ∗ will be used to compute the (true) rewards for the samples.
We construct the bandit problem as follows.
At each round, the context vectors of K = 10 individual are sampled from D 2 and after one is selected by the policy, its corresponding noisy reward, obtained by adding gaussian noise with standard deviation 0 .
2 to the true reward, is received by the agent.
Baselines.
We compare our method with the same baselines as in Appendix D.2, where the two oracle policies are now variants of Fair-Greedy V2.
Moreover, we set the regularization parameter for all policies using a ridge estimate to 0.1 and the exploration parameter of OFUL to 0.01.
Results (Fig. 4).
We draw similar conclusions as in Appendix D.2.
In particular, Greedy performing better than OFUL and the Fair-Greedy policy achieving sublinear fair pseudo-regret, but worse than
Oracle methods.
Additionaly we can see that knowing µ ∗ plays a more important role than knowing the true reward CDFs.
In this case, the gap between the Uniform random policy and the others is even larger since K = 10 .
Moreover, as expected, Fair-greedy selects more candidates from underperforming (in terms of reward) minority groups, when compared with OFUL and Greedy.
F Trade off between Fairness and Reward Maximization
In this section, we show for which problems the GMF policy and the optimal policy have competing goals.
in particular, for the case of K = 2 , when the rewards are absolutely continuous and independent across arms, whenever they are not identically distributed, the GMF policy achieves linear standard pseudo-regret with nonzero probability.
The following theorem proves this result.
Theorem 3.
Let Assumption A hold with K = 2 , and assume that F 1 ̸ = F 2 .
Let ¯ r t,a :=
⟨ µ ∗ , X t,a ⟩ , { a ∗ t } Tt =1 be the GMF policy (see Definition 3.1) and { a opt t } Tt =1 be the optimal policy (see Remark 3.1).
Then, there exists ϵ > 0 , such that
[Equation]
Furthermore with probability at least ϵp 4 L ∥ µ ∗ ∥ , for any T > 0 , we have
[Equation]
Proof.
Let ¯ r a := ⟨ µ ∗ , X a ⟩ , q a = F a ( r a ) be the CDF value of r a and F − 1 a be the quantile function, i.e. such that F − 1 a ( x ) =
inf { y ∈ R : x ≤ F a ( y ) } .
For ϵ > 0 consider the set E ϵ :=
E ϵ 1 ∪ E ϵ 2 where
[Equation]
Note that we can write
E ϵ
= { ( x, y ) ∈ [0 , 1] 2 : y < x < F 1 ( F − 1 2 ( y ) − ϵ ) } ,
E {+ϵ 
+}
= { ( x, y ) ∈ [0 , 1] 2 : F 1 ( F − 1 2 ( y ) + ϵ ) < x < y } .
Now, let g 1 , 2 ( y, ϵ )
=
F 1 ( F − 1 2 ( y ) + ϵ ) .
Since from Assumption A(ii)(iv), q 1 an q 2 are i.i.d uniform on [0 , 1] we have that
[Equation]
[Equation]
Since F 1 ̸ = F 2 , and F 1 , F 2 are absolutely continuous, there exists ϵ ′ > 0 , such that F − 1 2 ( y ) −
F − 1 1 ( y ) > ϵ ′ , or F − 1 2 ( y ) − F − 1 1 ( y ) <
ϵ ′ for y inside a closed interval, and hence P (( q 1 , q 2 ) ∈
E ϵ ′ ) > 0 .
This yields (20) by letting ϵ = ϵ ′ , and p = P (( q 1 , q 2 ) ∈
E ϵ ) .
Now, let q t,a = F a (¯ r t,a ) , then for the expected value of the instantaneous standard regret, at round t , we can write
[Equation]
F − 1 1 ( x ) | d x d y ≥ ϵ P (( q t, 1 , q t,
[Equation]
E ϵ )
=
ϵp > 0 , and for the standard regret, we have
[Equation]
Finally, let Ω be the event that 12 · (cid:80) Tt =1 E (cid:2) ¯ r t,a opt t − ¯ r t,a ∗ t (cid:3) ≤ (cid:80)
Tt =1 [¯ r t,a opt t − ¯ r t,a ∗ t ] .
Considering the fact that (cid:80) Tt =1 [¯ r t,a opt t − ¯ r t,a ∗ t ] ≤ 2 L ∥ µ ∗ ∥ T , we deduce
T
[Equation]
and we get
[Equation]
r
[Equation]
t ] /
[Equation]
4 L ∥ µ ∗ ∥ T ) ≤ P (Ω) , which finishes the proof.
Remark F.1.
In Thm. 3, ϵ ≤ 2 L ∥ µ ∗ ∥ , otherwise p = 0 .
On the other hand, by the definition p ≤ 1 , and accordingly ϵp 4 L ∥ µ ∗ ∥ ≤ 1 / 2 .
Remark F.2.
With similar reasoning as in the proof of Thm.
3, we can show that if F 1 ̸ =
Fthe optimal policy (see Remark 3.1) has linear fair pseudo-regret with positive probability, that is independent of T .
In particular, there exist c, c ′ > 0 , such that for any T > 0 , P ( T · c ′ ≤ (cid:80) Tt =
[ F a ∗ t (¯ r t,a ∗ t ) − F a opt t (¯ r t,a opt t )]) > c .
Example 1 (Disjoint supports) .
As an example consider the case when K = 2 and ¯ r t, 1 − ¯ r t, 2 ≥ ϵ > 0 , for all t ≥ 1 , almost surely.
Then, F 1 ( F − 1 2 ( y ) − ϵ )
= F 1 ( F − 1 2 ( y ) + ϵ ) =
for every y ∈ [0 , 1] .
Hence we have
[Equation]
Then by Thm. 3, with probability at least ϵ
[Equation]
∥ , for any T > 0 , we have (cid:80) Tt =1 [¯ r t,a opt t − ¯ r t,a ∗ t ] ≥ Tϵ 4 .