E QUAL E XPERIENCE IN R ECOMMENDER S YSTEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
We explore the fairness issue that arises in recommender systems.
Biased data due to inherent stereotypes of particular groups (e.g., male students’ average rating on mathematics is often higher than that on humanities, and vice versa for females) may yield a limited scope of suggested items to a certain group of users.
Our main contribution lies in the introduction of a novel fairness notion (that we call equal experience ), which can serve to regulate such unfairness in the presence of biased data.
The notion captures the degree of the equal experience of item recommendations across distinct groups.
We propose an optimization framework that incorporates the fairness notion as a regularization term, as well as introduce computationally-efﬁcient algorithms that solve the optimization.
Experiments on synthetic and benchmark real datasets demonstrate that the proposed framework can indeed mitigate such unfairness while exhibiting a minor degradation of recommendation accuracy.
I NTRODUCTION
Recommender systems are everywhere, playing a crucial role to support decision making and to decide what we experience in our daily life.
One recent challenge concerning fairness arises when the systems are built upon biased historical data.
Biased data due to polarized preferences of particular groups for certain items may often yield limited recommendation service.
For instance, if female students exhibit high ratings on literature subjects and less interest in math and science relative to males, the subject recommender system trained based on such data may provide a narrow scope of recommended subjects to the female group, thereby yielding unequal experience.
This unequal experience across groups may result in amplifying the gender gap issue in science, technology, engineering, and mathematics (STEM) ﬁelds.
Among various works for fair recommender systems (Yao & Huang, 2017; Li et al., 2021; Kamishima & Akaho, 2017; Xiao et al., 2017; Beutel et al., 2019; Burke, 2017), one recent and most relevant work is Yao & Huang (2017).
They focus on a scenario in which unfairness occurs mainly due to distinct recommendation accuracies across different groups.
They propose novel fairness measures that quantify the degree of such unfairness via the difference between recommendation accuracies, and also develop an optimization framework that well trades the fairness measures against the average accuracy.
However, it comes with a challenge in ensuring fairness w.r.t.
the unequal experience .
This is because similar accuracy performances between different groups do not guarantee a variety of recommendations to an underrepresented group with historical data bearing low preferences and/or scarce ratings for certain items.
For instance, in the subject recommendation, the fairness notion may not serve properly, as long as female students exhibit low ratings (and/or lack of ratings) on math and science subjects due to societal/cultural inﬂuences (and/or sampling biases).
Furthermore, if the recommended items are selected only according to the overall preference, the biased preference for a speciﬁc item group will further increase, and the exposure to the unpreferred item group will gradually decrease.
Contribution:
In an effort to address the challenge, we introduce a new fairness notion that we call equal experience .
At a high level, the notion represents how equally various items are suggested even for an underrepresented group preserving such biased historical data.
Inspired by an informationtheoretic notion “mutual information” (Cover, 1999) and its key property “chain rule” , we quantify our notion so as to control the level of independence between preference predictions and items for any group of users.
Speciﬁcally, the notion encourages prediction (cid:101) Y (e.g., 1 if a user prefers an item; 0 otherwise) to be independent of the following two: (i) user group Z user (e.g., 0 for male; and 1 for female); and (ii)
item group Z item (e.g., 0 for mathematics; and 1 for literature).
In other words, it promotes (cid:101)
Y ⊥
( Z user , Z item ) ; which in turns ensures all of the following four types of independence that one can think of: (cid:101)
Y ⊥ Z item , (cid:101)
Y ⊥ Z user , (cid:101)
Y ⊥ Z item |
Z user , and (cid:101)
Y ⊥ Z user | Z item .
This is inspired by the fact that mutual information being zero is equivalent to the independence between associated random variables, as well as the chain rule:
[Equation]
See Section 3.1 for details.
The higher independence, the more diverse recommendation services are offered for every group.
We also develop an optimization framework that incorporates the quantiﬁed notion as a regularization term into a conventional optimization for collaborative ﬁltering in recommender systems (e.g., the one based on matrix completion Koren (2008); Koren et al.
Here one noticeable feature of our framework is that the fairness performances w.r.t.
the above four types of independence conditions can be gracefully controlled via a single uniﬁed regularization term.
This is in stark contrast to prior works (Yao & Huang, 2017; Li et al., 2021; Kamishima & Akaho, 2017; Mehrotra et al., 2018), each of which promotes only one independence condition or two via two separate regularization terms.
See below Related works for details.
In order to enable an efﬁcient implementation of the fairness constraint, we employ recent methodologies developed in the context of fair classiﬁers, such as the ones building upon kernel density estimation (Cho et al., 2020a), mutual information (Zhang et al., 2018; Kamishima et al., 2012; Cho et al., 2020b), or covariance (Zafar et al., 2017a;b).
We also conduct extensive experiments both on synthetic and two benchmark real datasets: MovieLens 1M (Harper & Konstan, 2015) and Last FM 360K (Celma, 2010).
As a result, we ﬁrst identify two primary sources of biases that incur unequal experience : population imbalance and observation bias (Yao & Huang, 2017).
In addition, we demonstrate that our fairness notion can help improve the fairness measure w.r.t.
equal experience (to be deﬁned in Section 3.1; see
Deﬁnition 2) while exhibiting a small degradation of recommendation accuracy.
Related works: In addition to Yao & Huang (2017), numerous fairness notions and algorithms have been proposed for fair recommender systems (Xiao et al., 2017; Beutel et al., 2019; Singh & Joachims, 2018; Zehlike et al., 2017; Narasimhan et al., 2020; Biega et al., 2018;
Li et al., 2021; Kamishima & Akaho, 2017; Mehrotra et al., 2018; Schnabel et al., 2016).
Xiao et al.
(2017) develop fairness notions that encourage similar recommendations for users within the same group.
Beutel et al.
(2019) consider similar metrics as that in Yao & Huang (2017) yet in the context of pairwise recommender systems wherein pairewise preferences are given as training data.
Li et al.
(2021) propose a fairness measure that quantiﬁes the irrelevancy of preference predictions to user groups, like demographic parity in the fairness literature (Feldman et al., 2015; Zafar et al., 2017a;b).
Speciﬁcally, they consider the independence condition between prediction (cid:101) Y and
user group Z user : (cid:101)
Y ⊥ Z user .
Actually this was also considered as another fairness measure in Yao & Huang (2017).
Similarly, other works with a different direction consider the similar notion concerning the independence w.r.t.
item group Z item : (cid:101)
Y ⊥ Z item (Kamishima & Akaho, 2017; Singh & Joachims, 2018; Biega et al., 2018).
Mehrotra et al.
(2018) incorporate both measures to formulate a multi-objective optimization.
In Section 2.2, we will elaborate on why the above prior fairness notions cannot fully address the challenge w.r.t.
unequal experience.
There has been a proliferation of fairness notions in the context of fair classiﬁers: (i) group fairness (Feldman et al., 2015; Zafar et al., 2017b;
Hardt et al., 2016; Woodworth et al., 2017); (ii) individual fairness (Dwork et al., 2012; Garg et al., 2018); (iii) causality-based fairness (Kusner et al., 2017; Nabi & Shpitser, 2018; Russell et al., 2017; Wu et al., 2019; Zhang & Bareinboim, 2018b;a).
Among various prominent group fairness notions, demographic parity and equalized odds give an inspiration to our work in the process of applying the chain rule, reﬂected in equation 1.
Concurrently, a multitude of fairness algorithms have been developed with the use of covariance (Zafar et al., 2017a;b), mutual information (Zhang et al., 2018;
Kamishima et al., 2012; Cho et al., 2020b), kernel density estimation (Cho et al., 2020a) or Rényi correlation (Mary et al., 2019) to name a few.
In this work, we also demonstrate that our proposed framework (to be presented in Section 3) embraces many of these approaches; See Remark 1 for details.
P ROBLEM F ORMULATION
As a key technique for operating recommender systems, we consider collaborative ﬁltering which estimates user ratings on items.
We ﬁrst formulate an optimization problem for collaborative ﬁltering building upon one prominent approach, matrix completion .
We then introduce a couple of fairness measures proposed by recent prior works (Yao & Huang, 2017; Li et al., 2021; Kamishima & Akaho, 2017), and present an extended optimization framework that incorporates the fairness measures as regularization terms.
2. O PTIMIZATION BASED ON MATRIX COMPLETION
As a well-known approach for operating recommender systems, we consider matrix completion (Fazel, 2002; Koren et al., 2009; Candès & Recht, 2009).
Let M ∈ R n × m be the ground-truth rating matrix where n and m denote the number of users and items respectively.
Each entry, denoted by M ij , can be of any type.
It could be binary, ﬁve-star rating, or any real number.
Denote by Ω the set of observed entries of M .
For simplicity, we assume noiseless observation.
Denote by (cid:99) M ∈ R n × m an estimate of the rating matrix.
Matrix completion can be done via the rank minimization that exploits the low-rank structure of the rating matrix.
However, since the problem is NP-hard (Fazel, 2002), we consider a well-known relaxation approach that intends to minimize instead the squared error between M and (cid:99) M in the observed entries:
[Equation]
There are two well-known approaches for solving the optimization in equation 2: (i) matrix factorization (Abadir & Magnus, 2005; Koren et al., 2009); and (ii) neural-net-based parameterization (Salakhutdinov et al., 2007; Sedhain et al., 2015;
He et al., 2017).
Matrix factorization assumes a certain structure on the rating matrix: M = LR where L ∈ R n × r and R ∈ R r × m .
One natural way to search for optimal L ∗ and R ∗ is to apply gradient descent (Robbins & Monro, 1951) w.r.t.
all of the L ij ’s and R ij ’s, although it does not ensure the convergence of the optimal point due to non-convexity.
The second approach is to parameterize (cid:99) M via neural networks such as restricted
Boltzmann machine (Salakhutdinov et al., 2007) and autoencoder (Sedhain et al., 2015; Lee et al., 2018).
For instance, one may employ an autoencoder-type neural network which outputs a completed matrix (cid:99) M fed by the partially-observed version of M .
For a user-based autoencoder (Sedhain et al., 2015), an observed row vector of M is fed into the autoencoder, while an observed column vector serves as an input for an item-based autoencoder (Sedhain et al., 2015).
In this work, we consider the two approaches in our experiments: matrix factorization with gradient descent; and autoencoder-based parameterization.
One common way to promote a fair recommender system is to incorporate a fairness measure, say
L fair (which we will relate to an estimated matrix (cid:99) M ), as a regularization term into the above base optimization in equation 2: min
[Equation]
where λ ∈ [0 , 1] denotes a normalized regularization factor that balances prediction accuracy against the fairness constraint.
For the fairness-regularization term L fair , several fairness measures have been introduced.
2. F AIRNESS MEASURES IN PRIOR WORKS (Y AO & H UANG , 2017; K AMISHIMA & A KAHO , 2017; L I ET AL ., 2021)
We list three of them, which are mostly relevant to our framework to be presented in Section 3.
For illustrative purpose, we will explain them in a simple setting where there are two groups of users, say the male group M and the female group F .
The ﬁrst is value unfairness proposed by Yao & Huang (2017).
It quantiﬁes the difference between prediction errors across the two groups of users over the entire items:
[Equation]
where M Ω and F Ω denote the male and female group w.r.t.
observed entries, respectively.
While the measure promotes fairness w.r.t.
prediction accuracy across distinct groups, it may not ensure fairness w.r.t.
the diversity of recommended items to users.
To see this clearly, consider an extreme scenario in which the ground truth rating is very small M ij ∗ ≈ 0 for a certain item j ∗ (say science subject) for all i ∈ F .
In this case, minimizing VAL may encourage (cid:99) M ij ∗ ≈ 0 for all i ∈ F .
This then incurs almost no recommendation of the science subject to the females, thus giving no opportunity to experience the subject.
This motivates us to propose a new fairness measure (to be presented in Section 3.1) that helps mitigate such unfairness.
On the other hand, Kamishima & Akaho (2017) introduce another fairness measure, which bears a similar spirit to demographic parity in the fairness literature (Feldman et al., 2015; Zafar et al., 2017a;b).
The measure, named Calders and Verwer’s discrimination score ( CVS ), quantiﬁes the level of irrelevancy between preference predictions and item groups.
To describe it in detail, let us introduce some notations.
Let Z item be a sensitive attribute w.r.t.
item groups, e.g., Z item =(literature) and Z item = 1 (science).
Let (cid:98) Y be a generic random variable w.r.t.
estimated ratings (cid:99) M ij ’s. To capture the preference prediction, let us consider a simple binary preference setting
in which (cid:101) Y :
= 1 { (cid:98) Y ≥ τ } where τ indicates a certain threshold.
Specializing the measure into the one like demographic parity, it can be quantiﬁed as:
[Equation]
Minimizing the measure encourages the independence between (cid:101) Y and Z item , thereby promoting the same rating statistics across different groups.
However, it does not necessarily ensure the same statistics when we focus on a certain group of users.
It guarantees the independence only in the average sense.
To see this clearly, consider a simple scenario in which there are two groups of users, say female and male.
Let Z user be another sensitive attribute w.r.t.
user groups, e.g., Z user =(female) and Z user = 1 (male).
Fig.
1 illustrates a concrete example where (cid:101) Y is independent of Z item .
Notice that the number of 1’s w.r.t.
Z item = 0 (over the entire user groups) is the same as that w.r.t.
Z item = 1 .
However, focusing on a certain user group, say Z user = 0 , (cid:101) Y is highly correlated with
Z item .
Observe in the case Z user = 0 that the number of 1’s is 9 for Z item = 0 , while it reads 0 for
Z item = 1 .
Li et al.
(2021) consider a similar measure, named user-oriented group fairness ( UGF ), yet which targets the independence w.r.t.
user groups.
Similar to CVS , we can deﬁne it by replacing Z item with
Z user in equation 5:
[Equation]
[Equation]
However, by symmetry, the high correlation issue discussed via Fig.
1 still arises.
P ROPOSED F RAMEWORK
We ﬁrst propose new fairness measures that can regulate fairness w.r.t.
the opportunity to experience inherently-low preference items, as well as address the high correlation issue discussed as above.
We then develop an integrated optimization framework that uniﬁes the fairness measures as a single regularization term.
Finally we introduce concrete methodologies that can implement the proposed optimization.
3. N EW FAIRNESS MEASURES
The common limitation of the prior fairness measures (Yao & Huang, 2017; Kamishima & Akaho, 2017; Li et al., 2021) is that the independence between preference predictions and item groups may not be guaranteed for a certain group of users.
This motivates us to consider the conditional independence as a new fairness notion, formally deﬁned as below.
Deﬁnition 1 (Equalized Recommendation)
A recommender system is said to respect “equalized recommendation” if its prediction (cid:101) Y is independent of item’s sensitive attribute Z item given user’s sensitive attribute Z user : (cid:101)
Y ⊥ Z item |
Z user .
Inspired by the quantiﬁcation methods w.r.t.
equalized odds in the fairness literature (Jiang et al., 2019; Donini et al., 2018;
Hardt et al., 2016; Woodworth et al., 2017), we quantify the new notion via:
[Equation]
for arbitrary alphabet sizes |Z user | and |Z item | .
Here DER stands for the difference w.r.t.
two interested probabilities that arise in equalized recommendation, and this naming is similar to those in prior fairness metrics (Donini et al., 2018; Jiang et al., 2019).
It captures the degree of violating equalized recommendation via the difference between the conditional probability and its marginal given Z user .
Notice that the minimum DER = 0 is achieved under “equalized recommendation”.
One may consider another measure which takes “max” operation instead of “ (cid:80) ” in equation 7 or a different measure based on the ratio of the two associated probabilities.
We focus on DER in equation 7 for tractability of an associated optimization problem that we will explain in Section 3.2.
The constraint of “equalized recommendation” encourages the same prediction statistics of items for every user group, thereby promoting the equal chances of experiencing a variety of items for all individuals.
However, the notion comes with a limitation.
The limitation comes from the fact that conditional independence does not necessarily imply independence (Cover, 1999):
[Equation]
Actually, the ultimate goal of a fair recommender system is to ensure all of the following four types of independence:
[Equation]
One natural question that arises is then: What is a proper fairness notion which allows us to respect all of the above four conditions preferably in one shot?
In an attempt to succinctly represent all of the four conditions, we invoke an information-theoretic notion, mutual information (Cover, 1999).
One key property of mutual information, called the chain rule, gives an insight:
[Equation]
From this, we can readily see that
[Equation]
This is due to the non-negativity property of mutual information.
The key observation in equationmotivates us to propose a new fairness notion that we call equal experience .
Deﬁnition 2 (Equal Experience)
A recommender system is said to respect “equal experience” if its prediction (cid:101) Y is independent of both Z item and Z user : (cid:101) Y ⊥ ( Z item , Z user ) .
Similar to DER , we also quantify the notion as the difference between the conditional probability and its marginal:
[Equation]
12) for arbitrary alphabet sizes |Z user | and |Z item | .
We also coin the similar naming: DEE (difference w.r.t.
two interested probabilities that arise in equal experience).
3. F AIRNESS - REGULARIZED OPTIMIZATION
Taking DEE as the fairness-regularization term L fair in the focused framework (equation 3), we get:
[Equation]
where λ ∈ [0 , 1] denotes a normalized regularization factor.
Here one challenge that arises in equation 13 is that expressing DEE in terms of an optimization variable (cid:99) M is not that straightforward.
To overcome the challenge, we take the kernel density estimation (KDE) technique (Cho et al., 2020a) which enables faithful quantiﬁcation of fairness-regularization terms.
One key beneﬁt of the KDE approach is that the computed measures based on KDE is differentiable w.r.t.
model parameters, thus enjoying a family of gradient-based optimizers (Géron, 2019; Kingma & Ba, 2014b).
Since the problem context where the KDE technique (Cho et al., 2020a) was introduced is different from ours, we describe below details on the technique, tailoring it to our framework.
Implementation of DEE via the KDE technique (Cho et al., 2020a):
We ﬁrst parameterize prediction output (cid:99) M via matrix factorization or a neural network.
Let w be a collection of parameters w.r.t.
(cid:99) M .
It could be a collection of matrix entries of L and R when employing matrix factorization (cid:99) M = LR .
Or it could be a collection of neural network parameters in the latter case.
The key idea of the KDE technique is to approximate the interested probability distributions via kernel density estimator deﬁned below:
Deﬁnition 3 (Kernel Density Estimator (KDE) (Davis et al., 2011))
Let (ˆ y (1) , . . .
, ˆ y ( s ) ) be i.i.d.
examples drawn from a distribution with an unknown density f .
Its KDE is deﬁned as: (cid:98) f
(ˆ y ) :
= 1 sh
[Equation]
si =
[Equation]
k (cid:16)
[Equation]
i
[Equation]
h (cid:17) where f k is a kernel function (e.g., Gaussian kernel function (Davis et al., 2011)) and h > 0 is a smoothing parameter called bandwidth.
For DEE , the interested probability distributions are P ( (cid:101) Y = 1) and P ( (cid:101)
Y = 1 | Z item =
z 2 , Z user =
z 1 ) .
Let us ﬁrst consider P ( (cid:101) Y = 1) .
Remember (cid:101) Y :
= 1 { (cid:98) Y ≥ τ } , so (cid:98) Y should be taken into consideration initially.
Using the KDE, we can estimate the probability density function of (cid:98) Y , say f (cid:98) Y
(ˆ y ) :
[Equation]
This together with (cid:101) Y := 1 { (cid:98) Y ≥ τ } gives:
[Equation]
where F k (ˆ y ) := (cid:82) ∞
ˆ y f k ( t ) dt .
Since the approach relies upon a family of gradient-based optimizers, the gradients of (cid:98) P ( (cid:101) Y = 1) and (cid:98) P ( (cid:101)
Y = 1 | Z item =
z 2 , Z user =
z 1 ) need to be computed explicitly.
Using the technique in Cho et al.
(2020a)
(Proposition 1 therein), we can readily approximate ∇ w DEE .
See Appendix A.2 for details.
Remark 1 (Other choices for a measure of “equal experience”)
Instead of DEE , one can resort to other measures based on prominent tools employed in the fairness literature: covariance (Zafar et al., 2017a;b), mutual information (Zhang et al., 2018; Kamishima et al., 2012; Cho et al., 2020b),
Wasserstein distance (Jiang et al., 2020) and Rényi correlation (Mary et al., 2019).
We leave a more detailed explanation in Appendix A.3.
(cid:4)
E XPERIMENTS
We conduct experiments both on synthetic and two benchmark real datasets: MovieLens 1M (Harper & Konstan, 2015) and Last FM 360K (Celma, 2010).
We generate synthetic data so as to pose fairness issues.
Algorithms are implemented in PyTorch (Paszke et al., 2019), and experiments are performed on a server with Titan RTX GPUs.
All the simulation results (to be reported) are the ones averaged over ﬁve trials with distinct random seeds.
In Appendix A.6, we also present the running times of our algorithm and baselines on the synthetic and real datasets.
4. S YNTHETIC DATASET
Here we highlight two types of bias: population imbalance and observation bias (Yao & Huang, 2017).
For illustrative purpose, let us explain them in a simple subject-recommendation example where there are two user groups ( Z user = 0 for male and Z user = 1 for female) and two item groups ( Z item = 0 for science and Z item = 1 for literature).
Population imbalance refers to the difference in the ground-truth preferences between two user groups, e.g., for the science subject, male students exhibit higher ratings relative to females.
Observation bias is the one that occurs due to the stereotype formed by societal and cultural inﬂuences.
To understand what it means, let us consider a scenario where a male student equally likes science and literature subjects.
But due to the stereotype that male students prefer science to literature in general, there may be very sparse ratings from male students for literature.
The system trained based on such data might incorrectly interpret as if male students dislike literature.
Such data is said to have observation bias .
We generate synthetic data that bear the two biases in the context of binary ratings, i.e., M ij ∈ { +1 ( like ) , − 1 ( dislike ) } .
We divide n users into male and female groups each of n 2 , say M and F .
Items are also divided into two groups of m 2 , say male-preferred group M (cid:48) and female-preferred group F (cid:48) .
To account for population imbalance, we ﬁrst generate the ground-truth rating matrix
M ∈ R n × m using the following four probabilities: { p MM (cid:48) , p MF (cid:48) , p FM (cid:48) , p FF (cid:48) } where p MM (cid:48) indicates the probability that a male student likes a male-preferred subject (science).
More precisely, for i ∈ M and j ∈ M (cid:48) ,
[Equation]
Similarly the other probabilities are deﬁned.
To ensure the low-rank structure, say rank r , of the rating matrix, we generate r 2 basis rating vectors for male group as per the above preference probabilities, and similarly another set of r 2 basis rating vectors is generated for female group.
Every male student picks one of the r 2 basis vectors w.r.t.
the male group uniformly at random, and similarly for every female student.
This then yields rank ( M ) =
r .
To control observation bias, we introduce another probability set: { q MM (cid:48) , q MF (cid:48) , q FM (cid:48) , q FF (cid:48) } where q MM (cid:48) denotes the probability that a male student’s rating is observed for a male-preferred subject.
More precisely, for i ∈ M and j ∈ M (cid:48) ,
[Equation]
Similarly the other probabilities are deﬁned.
For simplicity, throughout all the synthetic data simulations, we assume a symmetric setting in which p MM (cid:48)
=
p FF (cid:48)
(= p 0 ) , p MF (cid:48)
= p FM (cid:48)
(= p 1 ) , q MM (cid:48)
=
q FF (cid:48)
(= q 0 ) , and q MF (cid:48)
=
q FM (cid:48)
(= q 1 ) .
We consider a setting in which ( r, n, m ) =
We leave a more detailed explanation of experiments on the synthetic dataset in Appendix A.4.
First, we check whether the bias actually incurs unfair recommendations.
For ease of understanding, we consider two scenarios: (i) population imbalance varies without observation bias; and (ii) observation bias varies without population
[Figure]
[Figure]
[Table]
imbalance.
Fig.
2 (Left) presents the 1st scenario, demonstrating that the fairness performance measured in DEE decreases with an increase in population imbalance, controlled by | p 0 − p 1 | .
Fig.(Right)
considers the 2nd scenario.
We see the same trend yet now w.r.t.
the variation of observation bias.
Table 1 presents the prediction error (RMSE) and fairness performances on the synthetic dataset having observation bias ( q 0 , q 1 ) =
(0 . 2 , 0 . 01) yet without population imbalance ( p 0 , p 1 ) =
We consider four fairness measures: (i) DEE (in equation 12); (ii) DER (in equation 7);
(iii) VAL (in equation 4); (iv) UGF (in equation 6); (v) CVS (in equation 5).
We also compare our algorithm with four baselines:
(i) unfair (no fairness constraint); (ii) VAL -based algorithm
(Yao & Huang, 2017); (iii) UGF -based algorithm
(Li et al., 2021); (iv) CVS -based algorithm (Kamishima & Akaho, 2017).
Each baseline, say VAL -based algorithm, achieves the best fairness performance only for VAL , while it does not work well under the other fairness measures.
On the other hand, our algorithm offers great fairness performances for all the measures, except for VAL , which our framework does not target.
4. R EAL DATASETS
We consider two benchmark datasets: MovieLens 1M (Harper & Konstan, 2015), and Last FM 360K (Celma, 2010).
We run experiments employing both matrix factorization (MF) based and autoencoder (AE) based techniques.
In Appendix A.5, we leave a more detailed explanation of experiments on the real datasets and the results of AE-based technique.
The results of real data experiments are listed in Tables 2 and 3.
As in the synthetic data setting, we can make two relevant observations.
All baseline algorithms fail to respect our metric ( DEE ) while meeting their own.
We also see that our algorithm exhibits signiﬁcant performances for all the fairness measures except for
VAL which does not have close relationship with the equal experience that we aim at.
[Table] [Table]
E XTENSION
In this section
[Table] we discuss the extension of our work: (i) introducing a new notion that bears a similar spirit to equalized odds [Table] and (ii) applying our notion to the fair ranking context.
Our fairness notion equal experience aims at recommending a variety of items for all user groups.
Demographic parity is similar to our notion in the sense of considering the irrelevancy of predictions to groups.
On the other hand, equalized odds is the fairness notion that concerns equal error rates (e.g., true/negative positive rate) across user groups by employing the ground-truth label Y .
Similar to equalized odds , our measure in recommender systems can readily be extended to a setting in which the ground-truth label is available to exploit.
The key idea is to promote (cid:101)
Y ⊥
( Z user , Z item ) | Y .
See
Appendix A.7 for details.
Many end-to-end recommender systems offer a recommendation list via two processes: (i) candidate generation, and (ii) ranking.
In this work, we focus on the ﬁrst candidate generation which we built collaborative ﬁltering for.
But our proposed notion can also be applicable in generating an end ranked list.
The idea behind the end-ranked list generation is to deﬁne (cid:101) Y as an indicator function which returns 1 when the item of interest belongs to, say top-k item set (0 otherwise).
In this case, the same notion (cid:101) Y ⊥
( Z user , Z item ) serves a proper role.
We leave detailed implementation of this notion under the fair ranking context for a future work.
C ONCLUSION
We introduced a novel fairness notion, equal experience , capable of respecting the desired requirements for fair recommender systems: independence between preference predictions and user groups; conditional independence for a certain user group, and vice versa for item groups.
The notion also seamlessly integrates into prior fairness algorithms.
Extensive experiments revealed the existence of unfairness (or bias) w.r.t.
equal experience , and our fair optimization framework successfully mitigates such unfairness with minimal prediction accuracy loss.
Our future work of interest is four-folded: (i) merging our notion with unexamined algorithms relying upon Rényi correlation or
Wasserstein distance; (ii) constructing a robust and fair recommender system in the presence of data poisoning; (iii) developing a blind fair recommender system without sensitive attributes; and (iv) extending our notion to fair ranking.
E THICS S TATEMENT
The proposed fairness notion and framework (which faithfully respects the notion) will enable a recommender system to offer an enough opportunity for users to experience various groups of items against stereotypes.
Users are further inﬂuenced by the various experiences when making more essential choices such as choosing a job.
Hence, it can give signiﬁcant impacts upon users who can expand their options via diverse experiences.
On the other hand, users provide sensitive information to a service provider of fair recommender systems, and there is a possibility that the user’s sensitive information could be abused.
Hence, one future work of great potential might be to develop a framework that ensures fairness even in the blind setting where sensitive information is not available
R EPRODUCIBILITY S TATEMENT
To ensure the reproducibility of experiments, we provide a step-by-step guideline as to synthetic data construction in Section 4.1.
We also provide a thorough description regarding two real datasets,
MovieLens 1M (Harper & Konstan, 2015) and Last FM 360K (Celma, 2010), along with data preprocessing methods in Section 4.2.
All the hyperparameter settings w.r t. synthetic and real data experiments are listed in Appendix A.4 and A.5, respectively.
We present the average running time of all algorithms together with speciﬁc computer conﬁguration details in Appendix A.6.
Lastly, the codes are available at https://github.com/cjw2525/FairRec .
A A PPENDIX A. O UTLINE
In the appendix, we ﬁrst provide further explanation for the implementation of KDE technique.
We also present other choices for a measure of equal experience in further detail.
Next, we provide a detailed explanation of experimental settings on both synthetic and real datasets: MovieLens 1M (Harper & Konstan, 2015) and Last FM 360K (Celma, 2010).
We then present additional experimental results which are not included in the main paper due to space limitation, and as well as provide a complexity analysis of our approach.
We also provide a detailed explanation for the extension of our measure to the fairness notion: (cid:101) Y ⊥
( Z user , Z item ) | Y , and present experimental results for this extension.
A. I MPLEMENTATION OF KDE TECHNIQUE
In order to approximate ∇ w DEE , we ﬁrst calculate the gradient of (cid:98) P ( (cid:101) Y = 1) employing the technique in Cho et al.
(2020a): ∇ w (cid:98) P ( (cid:101) Y = 1) = nmh n (cid:88) i = m (cid:88) j = f k
[Equation]
We can apply the same procedures w.r.t.
the second interested probability (cid:98) P ( (cid:101)
Y = 1 | Z item =
z 2 , Z user =
z 1 ) .
Merging equation 17 and the counterpart w.r.t.
the second probability, we can readily obtain:
[Equation]
where H δ ( x ) denotes the Huber loss (Huber, 1992) that takes 12 x 2 when | x | ≤ δ ; otherwise δ ( | x | − 12 δ ) .
A. O THER CHOICES FOR A MEASURE OF "equal experience"
Instead of DEE in equation 12, one can resort to other measures based on prominent tools employed in the fairness literature: covariance (Zafar et al., 2017a;b), mutual information (Zhang et al., 2018;
Kamishima et al., 2012; Cho et al., 2020b), Wasserstein distance (Jiang et al., 2020) and Rényi correlation (Mary et al., 2019).
For instance, the covariance-based approach allows us to take L fair as:
[Equation]
where Z : = ( Z [Equation] , Z
[Equation]
Here we use (cid:98) Y instead of (cid:101) Y , as (cid:101)
Y incurs non-differentiability, hindering implementation.
In the case of mutual information, one can take L fair as:
[Equation]
[Equation] Again, for ease of implementation, we employ (cid:98) Y .
This choice is also relevant because it serves as an upper bound of I ( (cid:101) Y ; Z item , Z user ) .
Notice that (cid:101) Y is a function of (cid:98) Y .
Reducing I ( (cid:98) Y ; Z item , Z user ) yields the minimization of the interested quantity
I ( (cid:101) Y ; Z item , Z user ) .
For implementation of
I ( (cid:98) Y ; Z item , Z user ) , we may employ the variational optimization technique in (Zhang et al., 2018; Cho et al., 2020b) to translate it into a function optimization which can also be parameterized.
Other choices can also be dealt with properly relying upon the associated techniques in (Jiang et al., 2020;
Mary et al., 2019).
A. S YNTHETIC DATASET EXPERIMENTS
We consider a setting in which ( r, n, m ) =
The synthetic data generated under the setting is randomly split into two subsets: 90% train set and 10% test set.
Since M ij ∈ { +1 , − 1 } , we set the threshold τ = 0 , i.e., (cid:101) Y = 1 { (cid:98) Y ≥ 0 } .
We train a matrix factorization (MF) based recommender system with the same rank as that of the dataset, i.e., L ∈ R 600 × 20 and
R ∈ R 20 × 400 .
We set hyperparameters ( δ, h ) =
(0 . 01 , 0 . 01) and λ = 0 .
99 for KDE-based algorithm implementation.
We use Adam optimizer for 1,000 iterations using full gradient.
The learning rate is set to 1e-3 and ( β 1 , β 2 ) =
The additional experimental results in a variety of scenarios are listed from Table 4 to 8.
We also visualize how the predicted preference rate of item groups for every user group Pr( ˜ Y = 1 | Z user , Z item ) changes under our framework. See Fig. 3 and 4.
[Table]
synthetic dataset. The dataset preserves observation bias ( q 0 , q 1 ) = (0 . 2 , 0 . 03) while exhibiting no population imbalance ( p 0 , p 1 ) =
[Table] [Table] [Table]
[Table]
A. R EAL DATASETS EXPERIMENTS
• MovieLens 1M : The associated task is to predict the movie rating on a 5-star scale.
This dataset contains 6,040 users, 3,900 movies, and 1,000,209 ratings,
i
e., rating matrix is 4.26% [Table] full.
1 We divide user and item groups based on gender and genre, respectively.
Action, crime, ﬁlme-noir, war are selected as male-preferred genre, whereas children, fantasy, musical, romance are selected as female-preferred genre.
We can select male-preferred and femalepreferred genres in a variety of ways based on ratings and observations.
For various scenarios, the experimental results with similar trends are obtained, so we report the results for one representative scenario.
If we assume that the real dataset is generated from the same model as the synthetic dataset, we can estimate the following probabilities.
We empirically estimate the interested probabilities w.r.t.
population imbalance as: (cid:98) p MM (cid:48)
= 0 .
627 , (cid:98) p MF (cid:48)
= 0 .
517 , (cid:98) p FM (cid:48)
= 0 .
622 , and (cid:98) p FF (cid:48)
= 0 .
Similarly we obtain the estimates for the other probabilities w.r.t.
observation bias: (cid:98) q MM (cid:48)
= 0 .
053 , (cid:98) q MF (cid:48)
= 0 .
037 , (cid:98) q FM (cid:48)
= 0 .
, and (cid:98) q FF (cid:48)
= 0 .
• Last FM 360K : The associated task is to predict whether the user likes the artist or not.
This dataset contains 359,347 users, 294,015 artists, and 17,559,530 play counts, i.e., rating matrix is 0.02% full.
The data for play counts is converted to binary rating:
+1 if counts > average, otherwise −
We divide user and item groups based on gender and genre, respectively.
Since this dataset only contains gender information, we use Last.fm APIto collect the genre of corresponding artist’s music; the tag was associated with 5, artists.
We also randomly select 5000 male and 5000 female users.
Among 10 genres, we choose hip-hop and musical for male and female preferred genres, respectively.
The ﬁnal rating matrix of 10,000 users and 5,706 artists is 0.55% full.
From the real data, we obtain empirical estimates for the interested probabilities w.r.t.
population imbalance: (cid:98) p MM (cid:48)
= 0 .
548 , (cid:98) p MF (cid:48)
= 0 .
421 , (cid:98) p FM (cid:48)
= 0 .
438 and (cid:98) p FF (cid:48)
= 0 .
Similarly we obtain the estimates for the other probabilities w.r.t.
observation bias: (cid:98) q MM (cid:48)
= 0 .
0054 , (cid:98) q MF (cid:48)
= 0 .
0011 , (cid:98) q FM (cid:48)
= 0 .
0036 and (cid:98) q FF (cid:48)
= 0 .
We randomly split the real datasets into 90% train set and 10% test set.
In case of MovieLens data, the rating is ﬁve-star based, so we set the threshold τ = 3 , i.e., (cid:101) Y = 1 { (cid:98) Y ≥ 3 } .
On the other hand, for LastFM dataset, we set τ = 0 as M ij ∈ { +1 , − 1 } .
We run experiments employing both matrix factorization (MF) based and autoencoder (AE) based techniques.
We set the rank asfor MF-based algorithm as was found by hyperparameter search.
The structure of the employed autoencoder (Sedhain et al., 2015) is as follows: (i) encoder has two linear layers: 512 nodes with
ReLU actiavation and 512 nodes with dropout layer (rate = 0 .
7 ) and ReLU activation
; (ii) decoder has one layer with 512 nodes.
For MovieLens 1M data (ﬁve-star ratings), we apply the clipping to the decoder output to ﬁt into the range.
For LastFM 360K data (binary rating: +1 and − 1 ), we apply tanh activation.
Hyperparameters for KDE-based algorithm are set to ( δ, h ) =
(0 . 01 , 0 . 01) and λ = 0 .
We use Adam optimizer for 1,000 iterations using full gradient, and the learning rate is http:/ set to 1e-3.
Since the main paper contains mostly MF-based experiments, here we only present the performances of autoencoder based algorithm on both real datasets.
We observe the same performance trends as those in
[Table] [Table]
[Figure]
ours
[Figure] [Figure] [Figure] unfair ours
[Figure]
[Figure]
[Table]
A. C OMPLEXITY ANALYSIS
We do complexity analysis of ours in light of other baselines.
For comparison, we consider the running time measured under Pytorch on Xeon Silver 4210R CPU and TITAN RTX GPU.
Tablepresents the running times of matrix factorization based algorithms on the synthetic and real datasets.
While our approach provides better fairness performance w.r.t.
DEE (as in the above tables), it comes at a cost of an increased complexity, around twice relative to the CVS -based algorithm [Table] [Table] DEE)
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] A. E XTENSION TO THE FAIRNESS NOTION : (cid:101) Y ⊥ Z user , Z item | Y
Similar to DEE in equation 12, we quantify the notion via:
[Equation]
for arbitrary alphabet sizes |Z user | and |Z item | .
Using the KDE approach, similarly we can obtain:
[Equation]
where F k (ˆ y ) := (cid:82) ∞
ˆ y f k ( t ) dt ; Y
ij := 1 { M ij ≥ τ } ; and I y :
= { ( i, j ) : ( i, j ) ∈ Ω , Y ij = y } .
We can then compute the gradien w.r.t.
w as:
[Equation]
We can then enjoy a family of gradient-based optimizers (Géron, 2017; Kingma & Ba, 2014a).
We provide experimental results for the extension on MovieLens 1M (Harper & Konstan, 2015) real dataset.
We run experiments employing both matrix factorization (MF) based and autoencoder (AE) based techniques.
We demonstrate that the framework based on the extended notion can indeed mitigate such unfairness while exhibiting a minor degradation of recommendation accuracy.
The results are listed in Table 12 and 13.
[Table] [Table]