Incorporating Bias-aware Margins into Contrastive Loss for Collaborative Filtering
Anonymous Author(s)
Abstract
Collaborative filtering (CF) models easily suffer from popularity bias, which makesrecommendation deviate from users’ actual preferences.
However, most currentdebiasing strategies are prone to playing a trade-off game between head and tailperformance, thus inevitably degrading the overall recommendation accuracy.
Toreduce the negative impact of popularity bias on CF models, we incorporate Biasaware margins into Contrastive loss and propose a simple yet effective BC Loss ,where the margin tailors quantitatively to the bias degree of each user-item interaction.
We investigate the geometric interpretation of BC loss, then further visualizeand theoretically prove that it simultaneously learns better head and tail representations by encouraging the compactness of similar users/items and enlarging thedispersion of dissimilar users/items.
Over six benchmark datasets
, we use BCloss to optimize two high-performing CF models.
On various evaluation settings( i.e., imbalanced/balanced, temporal split, fully-observed unbiased, tail/head testevaluations), BC loss outperforms the state-of-the-art debiasing and non-debiasingmethods with remarkable improvements.
Considering the theoretical guaranteeand empirical success of BC loss, we advocate using it not just as a debiasingstrategy, but also as a standard loss in recommender models.
Codes are available athttps://anonymous.4open.science/r/BC-Loss-8764/model.py .
Introduction
At the core of leading collaborative filtering (CF) models is the learning of high-quality representationsof users and items from historical interactions.
However, most CF models easily suffer from thepopularity bias issue in the interaction data [1, 2, 3, 4].
Specifically, the training data distribution istypically long-tailed, e.g., a few head items occupy most of the interactions, whereas the majorityof tail items are unpopular and receive little attention.
The CF models built upon the imbalanceddata are prone to learn the popularity bias and even amplify it by over-recommending head items and under-recommending tail items.
As a result, the popularity bias causes the biased representationswith poor generalization ability, making recommendations deviate from users’ actual preferences.
Motivated by concerns of popularity bias, studies on debiasing have been conducted to lift the tail performance.
Unfortunately, most prevalent debiasing strategies focus on the trade-off betweenhead and tail evaluations (see Table 3), including post-processing re-ranking [5, 6, 7, 8, 9], balancedtraining loss [10, 11, 12, 9], sample re-weighting [13, 14, 15, 16, 17, 18], and head bias removalby causal inference [19, 20, 21, 22].
Worse still, many of them hold some assumptions that areinfeasible in practice, such as the balanced test distribution is known in advance to guide the hyperparameters’ adjustment [23, 22], or a small unbiased data is present to train the unbiased model
Consequently, they pursue improvements on tail items but exacerbate the performance
[Table]
o f i t
e m s mean angle (pos):0.387 mean angle (neg):1.861 posneg (f) Softmax loss (tail) tail user pos item neg item 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Angle 0.0 0.
o f i t
e m s mean angle (pos):0.226 mean angle (neg):1.604 posneg (g) IPS-CN [15] (tail) tail user pos item neg item 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Angle 0.0 0.
o f i t
e m s mean angle (pos):0.077 mean angle (neg):1.831 posneg (h) BC loss (tail)
In this paper, we conjecture that an ideal debiasing strategy should learn high-quality head and tailrepresentations with powerful discrimination and generalization abilities, rather than playing a tradeoff game between the head and tail performance.
Here we follow the prior studies [15, 23, 13, 14]to focus on one key ingredient in representation learning: the loss function.
Figure 1 depicts theitem representations, which is optimized via two non-debiasing losses (BPR [26] and Softmax [27])and one debiasing loss (IPS-CN [15]).
Wherein, representation discrimination is reflected in howwell the positive items of a user are apart from the negatives.
Our insights are: (1) For a user, thenon-debiasing losses are inadequate to discriminate his/her positive and negative items well, sincetheir representations are largely overlapped as Figures 4a and 4b show; (2) Although IPS-CN achievesbetter discrimination power in the tail group than BPR ( cf.
positive items get smaller angles to theego user in Figure 4g, as compared to Figure 4e), it gets worse discrimination ability in the head ( cf.positive items hold larger angles to the ego user in Figure 4c, as compared to Figure 4a).
Towards this end, we incorporate Bias-aware margins into Contrastive Loss and devise a simple yeteffective BC Loss to guide the head and tail representation learning of CF models.
Specifically, wefirst employ a bias degree extractor to quantify the influence of interaction-wise popularity — thatis, how well an interaction is predicted, when only popularity information of the target user anditem is used.
Interactions involving inactive users and unpopular items often align with lower biasdegrees, indicating that popularity fails to reflect user preference faithfully.
In contrast, interactionswith active users and popular items are spurred by the popularity information, thus easily inclining tohigh bias degrees.
We then move on to train the CF model by converting the bias degrees into theangular margins between user and item representations.
If the bias degree is low, we impose a largermargin to strongly squeeze the tightness of representations.
In contrast, if the bias degree is large, weexert a small or vanishing margin to reduce the influences of biased representations.
Through this way, for each ego user’s representation, BC quantitatively controls its bias-aware margins with itemrepresentations — adaptively intensifying the representation similarity among positive items, whilediluting that among negative items.
Benefiting from stringent and discriminative representations, BCloss significantly improves both head and tail performance.
Furthermore, BC loss has three desirable advantages.
First, it has a clear geometric interpretation,as illustrated in Figure 2.
Second, it brings forth a simple but effective mechanism of hard examplemining (See Appendix A.2).
Third, we theoretically reveal that BC loss tends to learn a low-entropycluster for positive pairs ( e.g., compactness of matched users and items) and a high-entropy spacefor negative pairs ( e.g., dispersion of unmatched users and items)
(See Theorem 1).
Considering thetheoretical guarantee and empirical effectiveness, we argue that BC loss is not only promising toalleviate popularity bias, but also suitable as a standard learning strategy in CF.
Preliminary of Collaborative Filtering (CF)
Task Formulation.
Personalized recommendation is retrieving a subset of items from a large catalogto match user preference.
Here we consider a typical scenario, collaborative filtering (CF) withimplicit feedback [28], which can be framed as a top- N recommendation problem.
Let O + ={ ( u, i ) |
y ui
= 1 } be the historical interactions between users U and items I , where y ui = 1 indicatesthat user u ∈ U has adopted item
i ∈ I before.
Our goal is to optimize a CF model ˆ y : U × I → Rthat latches on user preference towards items.
Modeling Scheme.
Scrutinizing leading CF models [26, 25, 29, 30], we systematize the commonparadigm as a combination of three modules: user encoder ψ ( · ) , item encoder ϕ ( · ) , and similarityfunction s ( · ) .
Formally, we depict one CF model as ˆ y ( u, i ) =
s ( ψ ( u ) , ϕ ( i )) , where ψ : U → R dand ϕ : I → R d encode the identity (ID) information of user u and item i into d -dimensional
representations, respectively
s :
R d × R d → R measures the similarity between user and itemrepresentations.
In literature, there are various choices of encoders and similarity functions:
• Common encoders roughly fall into three groups: ID-based ( e.g., MF [26, 29], NMF [31], CMN[32]), history-based ( e.g., SVD++ [29], FISM [33], MultVAE [30]), and graph-based ( e.g., GCMC[34], PinSage [35], LightGCN [25]) fashions.
Here we select two high-performing encoders, MFand LightGCN, as the backbone models being optimized.
• The widely-used similarity functions include dot product [26], cosine similarity [36], and neuralnetworks [31].
As suggested in the recent study [36], cosine similarity is a simple yet effectiveand efficient similarity function in CF models, having achieved strong performance.
For betterinterpretation, we take a geometric view and denote it by:
[Equation]
in which ˆ θ ui is the angle between the user representation ψ ( u ) and
item representation ϕ
( i ) .Learning
Strategy.
To optimize the model parameters
CF models mostly frame the top-
N rec-
ommendation problem into a supervised learning task
and resort to one of three classical learningstrategies: pointwise loss ( e.g., binary cross-entropy [
, mean square error [29]
, pairwise loss
( e.g., BPR [
, WARP
, and
softmax
loss
Among them
pointwise and
pairwise losses
are
long-standing and widely-adopted objective functions in
CF.
However
, extensive studies [9, 1, 39]have analytically and empirically confirmed
that using pointwise or
pairwise
loss is prune to propagate
more
information
towards the head user-item pairs, which amplifies popularity bias.
Softmax loss is much less explored in CF than its application in other domains like CV [40, 41].
Recent studies [36, 42, 43, 44, 45] find that it inherently conducts hard example mining over multiplenegatives and aligns well with the ranking metric, thus attracting a surge of interest in recommendation.
Hence, we cast the minimization of softmax loss [27] as the representative learning strategy:
[Equation]
where ( u, i ) ∈ O
+ is one observed interaction of user u , while N u
= { j | y uj = 0 } is the set ofsampled unobserved items that u did not interact with before; τ is the hyper-parameter known as the temperature in softmax [46].
Nonetheless, modifying softmax loss to enhance the discriminativepower of representations and alleviate the popularity bias remains largely unexplored.
Therefore,our work aims to devise a more generic and broadly-applicable variant of softmax loss for CF tasks,which can improve the long-tail performance fundamentally.
Methodology of BC Loss
On the basis of softmax loss, we devise our BC loss and present its desirable characteristics.
3.1 Popularity Bias Extractor
Before mitigating popularity bias, we need to quantify the influence of popularity bias on a singleuser-item pair.
One straightforward solution is to compare the performance difference betweenthe biased and unbiased evaluations.
However, this is not feasible as the unbiased data is usuallyunavailable in practice.
Statistical metrics of popularity could be a reasonable proxy of the biasedinformation, such as user popularity statistics p u ∈ P ( i.e., the number of historical items that useru has interacted with before) and item popularity statistics p i ∈ P ( i.e., the number of observedinteractions that item i is involved in).
If the impact of the interaction between u and i can becaptured well based solely on such statistics, the model is susceptible to exploiting popularity bias forprediction.
Hence, we argue that the popularity-only prediction will delineate the influence of bias.
Towards this end, we first train an additional module, termed popularity bias extractor, which onlytakes the popularity statistics as input to make prediction.
Similar to the modeling of CF ( cf. Section2), the bias extractor is formulated as a function ˆ y b : P × P → R :
[Equation]
(3) where the user popularity encoder ψ b : P → R d and the item popularity encoder ϕ b : P → R dmap the popularity statistics of user u and item i into d -dimensional popularity embeddings ψ b ( p u )and ϕ b ( p i ) , respectively; s : R d × R d → R is the cosine similarity function between popularityembeddings ( cf.
Equation (1)).
ˆ ξ ui is the angle between ψ b ( p u ) and ϕ b ( p i ) .
We then minimize the following softmax loss to optimize the popularity bias extractor:
[Equation]
This optimization enforces the extractor to reconstruct the historical interactions using only biasedinformation ( i.e., popularity statistics) and makes the reconstruction reflect the interaction-wise biasdegree.
As shown in Appendix B.5, interactions with active users and popular items are incliningto learn well via Equation (4). Furthermore, we can distinguish hard interactions based on the biasdegree, i.e., the interactions that can be hardly predicted by popularity statistics ought to be moreinformative for representation learning in the target CF model.
In a nutshell, the popularity biasextractor underscores the bias degree of each user-item interaction, which substantively reflects howhard it is to be predicted.
3.2 BC Loss
We move on to devise a new BC loss for the target CF model.
Our BC loss stems from softmax loss butconverts the interaction-bias degrees into the bias-aware angular margins among the representationsto enhance the discriminative power of representations.
Our BC loss is:
[Equation]
where M ui is the bias-aware angular margin for the interaction ( u, i ) defined as:
[Equation]
where ˆ ξ ui is derived from the popularity bias extractor ( cf.
Equation (3)), and π − ˆ θ ui is the upperbound to restrict cos( · + M ui ) to be a monotonically decreasing function.
Intuitively, if a user-item 𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊𝑴 𝒖 𝒊𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊
[Figure]
𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊𝑴 𝒖 𝒊𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊(b) BC Loss (2D) 𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊𝑴 𝒖 𝒊𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊
[Figure]
𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊𝑴 𝒖 𝒊𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊
[Figure]
pair ( u, i ) is the hard interaction that can hardly be reconstructed by its popularity statistics, it holds ahigh value of ˆ ξ ui and leads to a high value of M ui ; henceforward, BC loss imposes the large angularmargin M ui between the negative item j and positive item i and optimizes the representations of useru and item i to lower ˆ ξ ui .
See more details and analyses in Section 4.
It is noted that BC loss is extremely easy to implement in recommendation tasks, which onlyneeds to revise several lines of code.
Moreover, compared with softmax loss, BC loss only addsnegligible computational complexity during training ( cf. Table 5) but achieves more discriminativerepresentations.
Hence, we recommend to use BC loss not only as a debiasing strategy to alleviatethe popularity bias, but also as a standard loss in recommender models to enhance the discriminativepower.
Note that the modeling of M ui is worth exploring, such as the more complex version M ui = min
{ λ · ˆ ξ ui , π − ˆ θ ui } where λ controls the strength of the bias-margin.
Meanwhile, carefullydesigning a monotonically decreasing function helps to get rid of the upper bound restriction.
Wewill leave the exploration of bias-margin in future work.
Analyses of BC Loss
We analyze desirable characteristics of BC loss.
Specifically, we start by presenting its geometric interpretation, and then show its theoretical properties w.r.t.
compactness and dispersion of representations.
The hard mining mechanism of BC loss is discussed in Appendix A.2.
4.1 Geometric Interpretation
Here we probe into the ranking criteria of softmax loss and BC loss, from the geometric perspective.
To simplify the geometric interpretation, we analyze one user u with one observed item i andonly two unobserved items j and k .
Then the posterior probabilities obtained by softmax loss are:
[Equation]
[Equation] +exp(cos(ˆ θ uk ) /τ ) .
During training, softmax loss encourages the rankingcriteria ˆ θ ui < ˆ θ uj and ˆ θ ui < ˆ θ uk to model the basic assumption that the observed interaction ( u, i )indicates more positive cues of user preference than the unobserved interactions ( u, j ) and ( u, k ) .
Intuitively, to make the ranking criteria more stringent, we can impose an angular margin M ui on itand establish a new criteria ˆ θ ui + M ui < ˆ θ uj
and ˆ θ ui + M ui < ˆ θ uk .
Directly formulating this ideaarrives at the posterior probabilities of BC loss:
[Equation]
Obviously, BC loss is more rigorous about the ranking assumption compared with softmax loss.
See
Appendix A.2 for more detailed explanations.
We then depict the geometric interpretation and comparison of softmax loss and BC loss in Figure 2.
Assume the learned representations of i , j , and k are given, and softmax and BC losses are optimizedto the same value.
In softmax loss, the constraint boundaries for correctly ranking user u ’s preferenceare ˆ θ ui
=
ˆ θ uj and ˆ θ ui
= ˆ θ uk ; whereas, in BC loss, the constraint boundaries are ˆ θ ui + M ui
= ˆ θ
ujand
ˆ θ ui + M ui
= ˆ θ uk .
Geometrically, from softmax loss ( cf.
Figure 2c) to BC loss ( cf.
Figure 2d), itis a more stringent circle-like region on the unit sphere in the 3D case.
Further enlarging the margin
M ui will lead to a smaller hypercycle-like region, which is an explicit discriminative constraint on amanifold.
As a result, limited constraint regions squeeze the tightness of similar items and encouragesthe separation of dissimilar items.
Moreover, with the increase of representation dimension, BC loss has more restricted learning requirements, exponentially decreasing the area of constraint regions forcorrect ranking, and becomes progressively powerful to learn discriminative representations.
4.2 Theoretical Properties
BC loss improves head and tail representation learning by enforcing the compactness of matchedusers and items, while imposing the dispersion of unmatched users and items.
See detailed proof in
Appendix A.3.
Theorem 1.
Let v u .
=
ψ ( u ) , v i .
=
ϕ ( i ) , and c u = 1 |P u | (cid:80) i ∈P u v i , c i =
|P i | (cid:80) u ∈P i v u , where
P u
= { i | y ui
= 1 } and N u
= { i | y ui = 0 } are the sets of user u ’s positive and negative items,respectively; P i = { u |
y ui
= 1 } is the set of item i ’s positive users.
Assuming the representations ofusers and items are normalized, the minimization of BC loss is equivalent to minimizing a compactnesspart and a dispersion part simultaneously:
[Equation]
Discussion.
c u is the averaged representations of all items that u has interacted with, which describesu ’s interest; similarly, c i profiles its user group.
For the compactness part, BC loss forces the user’spositive items to be user-centric and vice versa.
From the entropy perspective, compactness part tendsto learn a low-entropy cluster for positive interactions, i.e., high compactness for similar users anditems.
For the dispersion part, for users and items from unobserved interactions, BC loss maximizesthe pairwise euclidean distance between their representations and encourages them to be distant fromeach other; Hence, from the entropy viewpoint, dispersion part levers the spread of representations tolearn a high-entropy representation space, i.e., large separation degree for dissimilar users and items.
Experiments
We aim to answer the following research questions:
• RQ1: How does BC Loss perform compared with debiasing strategies in various evaluations?• RQ2: Does BC loss cause the trade-off between head and tail performance?• RQ3: What are the impacts of the components ( e.g., temperature, margin) on BC Loss?
Baselines & Datasets.
SOTA debiasing strategies in various research lines are compared: samplere-weighting (IPS-CN [15]), bias removal by causal inference (MACR [22], CausE [19]), andregularization-based framework (sam+reg [12]).
Extensive experiments are conducted on eight realworld benchmark datasets: Tencent [47], Amazon-Book [48], Alibaba-iFashion [49], Yelp2018 [25],
Douban Movie [50], Yahoo!R3 [51], Coat [13] and KuaiRec [52].
For comprehensive comparisons,almost all standard test distributions in CF are covered in the experiments: balanced test set [22,23, 24], randomly selected imbalanced test set [10, 53], temporal split test set [20, 21, 12], andunbiased test set [13, 52, 51].
See more experiments on KuaiRec, Yahoo!R3, and Coat for unbiasedtest evaluation in Appendix B.3 and more comparison results between BC loss and other standardlosses (most widely used BPR [26], newest proposed CCL [54] and SSM [55]) in Appendix B.4.
5.1 Performance Comparison (RQ1)
5.1.1 Evaluations on Imbalanced and Balanced Test Sets.
Motivation.
Many prevalent debiasing methods assume that test distribution is known in advance[22, 23, 10], i.e., the validation set has similar distribution with the test set.
Moreover, only animbalanced or balanced test set is evaluated.
However, in real-world applications, the test distributionsare usually unavailable and can even reverse the prior in the training distribution.
We conjecture thata good debiasing recommender is required to perform well on both imbalanced and balanced testdistributions.
In our settings, no information about the balanced test is provided in advance.
Data Splits.
The models are identical across both imbalanced and balanced evaluations.
The testdistribution in the balanced evaluation is uniform, i.e., randomly sample 15% of interactions with [Table] equal probability w.r.t.
items.
Besides, the test splits for the imbalanced test are similarly longtailed like the train and validation sets, i.e., randomly split the remaining interactions into training,validation, and imbalanced test sets ( 60% : 10% : 15% ).
Results.
Table 1 reports the comparison of performance in imbalanced and balanced test evaluations.
The best performing methods are bold and starred, while the strongest baselines are underlined;
Imp.% measures the relative improvements of BC loss over the strongest baselines.
We observe that:
• BC loss significantly outperforms the state-of-the-art baselines in both balanced and imbal-
anced evaluations across all datasets.
In particular, it achieves consistent improvements over the best debiasing baselines and original CF models by 12 . 1% ∼ 58 .
This clearly demonstrates that BC loss not only effectively alleviates the amplification of popularity bias but also improves the discriminative power of representations.
Moreover, Table 5 shows the computational costs of all methods.
Compared to the backbone models, BC loss only adds negligible time complexity.
• Debiasing baselines sacrifice the imbalanced performance and perform inconsistently acrossdatasets.
Debiasing strategies generally achieve higher balanced performance at the expense of a
large imbalanced performance drop.
Specifically, the strongest baselines over all imbalanced testsets are the original CF models.
Worse still, as the degree of data sparsity increases, some debiasing methods fail to quantify the popularity bias and limit their bias removal ability.
For example, in thesparest Alibaba-iFashion dataset, the results of MF+IPS-CN, MF+CausE, LightGCN+MACR and
LightGCN+CausE on the balanced evaluation are lower than original CF models (MF or LightGCN).
In contrast, benefiting from popularity bias-aware margin, BC loss can learn discriminativerepresentations that accomplish more profound user and item understanding, leading to higher headand tail recommendation quality.
5.1.2 Evaluations on Temporal Split Test
Set
Motivation.
[Table]
In real applications, popularity biasdynamically changes over time.
Herewe consider temporal split test evaluation on Douban Movie where thehistorical interactions are sliced intothe training, validation, and test sets(70%:10%:20%) according to thetimestamps.
Results.
As Table 2 shows, BC loss is steadily superior to all baselines w.r.t.
all metrics on Douban
Movie.
For instance, it achieves significant improvements over the MF and LightGCN backbonesw.r.t.
Recall@20 by 10.2% and 10.4%, respectively.
This validates that BC loss endows the backbonemodels with better robustness against the popularity distribution shift and alleviates the negative influence of popularity bias.
Surprisingly, none of the debiasing baselines could maintain a comparableperformance to the backbones.
We ascribe the failure to their preconceived idea of tail items, whichpossibly change over time.
[Table] [Figure] [Figure]
B a l a n c e d N D
C G @
[Figure]
[Figure]
I m b a l a n c e d
N D C G @(b) MF+BC loss
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure]
[Figure] [Figure] [Figure]
B a l a n c e d N D
C G @
[Figure] [Figure]
I m b a l a n c e d
N D C G @(
[Figure]
5.2 Head, Mid, & Tail Performance (RQ2)
Motivation.
To further evaluate whether BC loss lifts the tail performance by inevitably sacrificingthe head performance, we divide the test set of Tencent into three subgroups, according to theinteraction number of each item: head (popular items that are in the top third), mid (normal itemsin the middle), and tail (unpopular items in the bottom third).
Most previous studies focus onaverage NDCG@20 for evaluation, especially balanced test evaluations [22, 23].
However, averagemetrics could be insufficient to reflect the performance of each subgroup.
A trivial solution toachieve high performance is promoting the rankings of low-popularity items in the recommendations.
In this case, only the average metrics are not reliable on the balanced test.
Therefore, we reportthe performance of individual subgroups on both balanced and imbalanced test sets for a morecomprehensive comparison.
Results.
Table 3 shows the evaluations of the head, mid, and tail subgroups.
The red and bluenumbers in percentage separately refer to the improvement and decline of each method relative to theoriginal CF model (MF or LightGCN).
We find that:
• BC loss is the only method that consistently yields remarkable improvements in every subgroup.
With a closer look at the head evaluation, BC loss shows its ability to learn more discriminative representations for popular items across imbalanced and balanced settings.
In particular, itachieves significant improvements over MF and LightGCN w.r.t.
head NDCG by 40% and 17% inthe imbalanced test evaluation, respectively.
We
attribute improvements to the usage of bias-awaremargin, which boosts the recommendation quality for the tail and head items.
• As the performance comparison among subgraphs in the imbalanced scenario shows, thebaselines enhance the tail performance but sacrifice the head performance.
Specifically,these baselines hardly maintain the head performance and show a clear trade-off trend betweenthe head and tail performance.
Taking MACR as an example, although the great improvement( +1662% ) over MF is achieved in the tail subgraph, it brings in the dramatic drop ( − 62% ) in thehead subgraph, which lowers the overall performance by a big drop ( − 60% ).
Here we ascribe thetrade-off to blindly promoting the rankings of tail items for matched and unmatched, rather thanimproving the discriminative power of representations.
5.3 Study on BC Loss (RQ3)
Effect of Bias-aware Margin.
Figure 3a displays the performance on balanced and imbalanced testsets on Tencent among softmax loss, BC loss with constant margin M [40], and BC loss with adaptivebias-aware margin.
BC loss achieves the best performance, illustrating that bias-aware margin indeedis effective at reducing popularity bias and learning high-quality representations.
Effect of Temperature τ .
BC loss has one hyperparameter to tune — temperature τ in Equation(5).
In Figure 3b and 3c, both balanced and imbalanced evaluations exhibit the concave unimodalfunctions of τ , where the curves reach the peak almost synchronously in a small range of τ .
Forexample, MF+BC loss gets the best performance when τ = 0 .
05 and τ = 0 .
07 in balanced andimbalanced settings, respectively; We observe similar trends on other datasets and skip them due tothe space limit.
This justifies that BC loss does not suffer from the trade-off between the balanced andimbalanced evaluations and improves the generalization without sacrificing the head performance.
Related Work
Prevalent popularity debiasing strategies in CF roughly fall into four research lines.
Post-processing re-ranking methods [5, 6, 7, 8, 9] are applied to the output of the recommendersystem without changing the representations of users and items.
The purposes of modifying theranking of models can be various: Calibration [5] ensures that the past interests proportions ofusers are expected to maintain at the same level; RankALS [6] aims to increase the diversificationof recommendation; FPC [8] investigates the popularity bias in the dynamic recommendation byrescaling the predicted scores.
Regularization-based frameworks [10, 11, 12, 9] explore the use of regularization that provides atunable mechanism for controlling the trade-off between recommendation accuracy and coverage.
The difference among these methods is the design of penalty terms: ALS+Reg [11] defines intra-listdistance as the penalty to achieve the fair recommendation; ESAM [10] introduces the attributecorrelation alignment, center-clustering, and self-training regularization to learn good feature representations; sam-reg [12] regularizes the biased correlation between user-item relevance and itempopularity; Reg [9] decouples the item popularity with the model preference predictions.
Sample re-weighting methods [13, 14, 15, 16, 17, 18], also known as Inverse Propensity Score (IPS)view the item popularity in the training set as the propensity score and exploit its inverse to re-weightloss of each instance.
To address the high variance of re-weighted loss, many of them [15, 14] furtheremploy normalization or smoothing penalty to attain a more stable output.
However, the unreliabilityof methods is due to their measurement of the propensity score, leveraging the item frequency butfailing to consider interaction-wise popularity bias.
Bias removal by causal inference methods [19, 24, 23, 20, 21, 22], getting inspiration from therecent success of counterfactual inference, specify the role of popularity bias in assumed causalgraphs and mitigate the bias effect on the prediction.
However, the causal structure is heuristicallyassumed based on the author’s understanding, without any theoretical guarantee.
BC loss opens up a possibility of conventional debiasing methods in CF that mitigate the popularitybias by enhancing the discriminative power.
Recent studies, boosting the discriminative featurespaces by modified softmax loss are mainly discussed in face recognition, where a constant margin isadded [40] to better classify.
We transfer it in CF and compare it with BC loss in Figure 3a.
Conclusion
Despite the great success in collaborative filtering, today’s popularity debiasing methods are stillfar from being able to improve the recommendation quality.
In this work, we proposed a simple yeteffective BC loss, utilizing popularity bias-aware margin to eliminate the popularity bias.
Groundedby theoretical proof, clear geometric interpretation and real-world visualization study, BC loss booststhe head and tail performance by learning a more discriminative representation space.
Extensiveexperiments verify that the remarkable improvement in head and tail evaluations on various test setsindeed comes from the better representation rather than simply catering to the tail.
The limitations of BC loss are in three respects, which will be addressed in future work: 1) the modeling of bias-aware margin is worth exploring, which could significantly influence the performance of
BC loss, 2) multiple important biases, such as exposure and selection bias, are not considered, and 3)more experiments comparing BC loss to standard CF losses ( e.g., cross-entropy, WARP) are neededto further demonstrate the power of BC loss in regular recommendation tasks (See comparison to BPR, CCL and SSM in Appendix B.4).
We believe that this work provides a potential researchdirection to diagnose the debiasing of long-tail ranking and will inspire more works.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[No] This paper
proposes a novel debiasing algorithm for recommendation system, which does not have any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We include
code by URL in abstract.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix B.1.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A In-depth Analysis of BC loss
A.1 Visualization of A Toy Experiment
Here we visualize a toy example on the Yelp2018 [25] dataset to showcase the effect of BC loss.
Specifically, we train a two-layer LightGCN whose embedding size is three, and illustrate the 3dimensional normalized representations on a 3D unit sphere in Figure 1 (See the magnified viewin Figure 4).
We train the identical LightGCN backbone with different loss functions: BPR loss,softmax loss, BC loss, and IPS-CN [15].
For the same head/tail user ( i.e., green stars), we plot 500items in the unit sphere covering all positive items ( i.e., red dots) and randomly-selected negativeitems ( i.e., blue dots) from both the training and testing sets.
Moreover, the angle distribution (thesecond row of each subfigure) of positive and negative items for a certain user quantitatively showsthe discriminative power of each loss.
We observe that:
• BC loss learns more discriminative representations in both head and tail user cases.
Moreover,
BC loss learns a more reasonable representation distribution that is locally clustered and
globally separated.
As Figures 4d and 4h show, for head and tail users, BC loss encourages around
40% and 55% of positive items to fall into the group closest to user representations, respectively.
In other words, these item representations are almost clustered to a small region.
BC loss also
achieves the smallest distance w.r
t. mean positive angle.
This verifies that BC loss tends to learn
a
high similar item/user compactness.
Moreover, Figure 4h presents a clear margin between positive
and negative items, reflecting a highly-discriminative power.
Compared to softmax loss in Figures
4b and 4f, the compactness and dispersion properties of BC loss come from the incorporation of
interaction-wise bias-aware margin.
• The representations learned by standard CF losses - BPR loss and softmax loss - are not
discriminative enough.
Under the supervision of BPR and softmax losses, item representations
are separably allocated in a wide range of the unit sphere, where blue and red points occupy
almost the same space area, as Figures 4a and 4b demonstrate.
Furthermore, Figure 4e shows
only a negligible overlap between positive and negative items’ angle distributions.
However, as
the negative items are much more than the positive items for the tail user, a small overlap will
make many irrelevant items rank higher than the relevant items, thus significantly hindering the
recommendation accuracy.
Hence, directly optimizing BPR or softmax loss might be suboptimal
for the personalized recommendation tasks.
• IPS-CN, a well-known popularity debiasing method in CF, is prone to lift the tail performance
by sacrificing the representation learning for the head.
Compared with BPR loss in Figure 4e,
IPS-CN learns better item representations for the tail user, which achieves smaller mean positive
angle as illustrated in Figure 4g.
However, for the head user in Figure 4c, the positive and negative
item representations are mixed and cannot be easily distinguished.
Worse still, the representations
learned by IPS-CN has a larger mean positive angle for head user compared to BPR loss.
This
results in a dramatic performance drop for head evaluations.
A.2 Hard Example Mining Mechanism - one desirable property of BC loss:
We argue that the mechanism of adaptively mining hard interactions is inherent in BC loss, whichimproves the efficiency and effectiveness of training.
Distinct from softmax loss that relies on thepredictive scores to mine hard negative samples [56] and leaves the popularity bias untouched, our
BC loss considers the interaction-wise biases and adaptively locates hard informative interactions.
Specifically, the popularity bias extractor ˆ y b in Equation (3) can be viewed as a hard sample detector.
Considering an interaction ( u, i ) with a high bias degree cos (ˆ ξ ui ) , we can only use its popularityinformation to predict user preference and attain a vanishing bias-aware angular margin M ui .
Hence,interaction ( u, i ) will plausibly serve as the biased and easy sample, if it involves the active usersand popular items.
Its close-to-zero margin makes ( u, i ) ’s BC loss approach softmax loss, thusdowngrading the ranking criteria to match the basic assumption of softmax loss.
In contrast, if the popularity statistics are deficient in recovering user preference via the popularitybias extractor, the interaction ( u, i ) garners the low bias degree cos (ˆ ξ ui ) and exerts the significantmargin M ui on its BC loss.
Hence, it could work as the hard sample, which typically covers the tailusers and items, and yields a more stringent assumption that user u prefers the tail item over the other popular items by a large margin.
Such a significant margin makes the losses more challenging tolearn.
In a nutshell, BC loss adaptively prioritizes the interaction samples based on their bias degreeand leads the CF model to shift its attention to hard samples, thus improving both head and tailperformance, compared with softmax loss ( cf.
Section 5.3).
A.3 Proof of Theorem 1
Theorem 1.
Let v u .
=
ψ ( u ) , v i .
=
{+ϕ ( i ) , and c u = 1 |P u | (cid:80) i ∈P u v i , c i = 
+}
|P i | (cid:80) u ∈P i v u , where
P u
= { i | y ui
= 1 } and N u
= { i | y ui = 0 } are the sets of user u ’s positive and negative items,respectively; P i = { u |
y ui
= 1 } is the set of item i ’s positive users.
Assuming the representations ofusers and items are normalized, the minimization of BC loss is equivalent to minimizing a compactnesspart and a dispersion part simultaneously:
[Equation]
Proof.
Let the upper-case letter V ∈ V be the random vector of representation and V
⊆ R d be therepresentation space.
We use the normalization assumption of representations to connect cosine and
Euclidean distances, i.e., if ∥ v u ∥ = 1 and ∥ v i ∥ = 1 , v Tu v i = 1 − 12 ∥ v u − v i ∥ 2 , ∀ u, i .
Let P u
= { i | y ui
= 1 } be the set of user u ’s positive items , P i
= { u |
y ui
= 1 } to be the set of itemi ’s positive users, and N u
= { i | y ui = 0 } be the set of user u ’s negative items.
Clearly, there exists anupper bound m , s.t. − 1 < cos(ˆ θ ui + M ui ) ≤ v Tu v i − m < 1 .
Therefore, we can analyze BC loss,which has the following relationships:
[Equation]
We now probe into the first term in Equation (8):
[Equation]
{+where the symbol c = indicates equality up to a multiplicative and/or additive constant; c u =
+}
{+|P u | (cid:80) i ∈P u v i is the averaged representation of all items that u has interacted with, which describes u ’s interest; c i = 
+}
|P i | (cid:80) u ∈P i v u is the averaged representation of all users who have adopteditem i , which profiles its user group.
We further analyze Equation (9) from the entropy view byconflating the first two terms:
[Equation]
where v ∈ { v u | u ∈ U} ∪ { v i | i ∈ I} summarizes the representations of users and items, with themean of c v .
Following [57], we further interpret this term as a conditional cross-entropy between
V and another random variable ¯ V whose conditional distribution given Y is a standard Gaussian
[Equation]
∼ N
[Equation]
c V
[Equation]
I ) :
[Equation]
[Equation]
where H ( · ) denotes the cross-entropy, and D KL ( · ) denotes the KL -divergence.
As a consequence,the first term in Equation (8) is positive proportional to H ( V | Y ) .
This concludes the proof for thefirst compactness part of BC loss.
We then inspect the second term in Equation (8) to demonstrate its dispersion property:
[Equation]
where we drop the redundant terms aligned with the compactness objective in the second line, andadopt Jensen’s inequality in the third line.
As shown in prior studies [58], minimizing this term isequivalent to maximizing entropy H ( V ) :
[Equation]
As a result, the second term in Equation (8) works as the dispersion part in BC loss.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
PS
[Table]
C
N [] ( t a il ) t
a il u s e r p o s i t
e m n e g
i t
e m 0 .0 .1 .1 .2 .2 .3 .
A n g l e 0 . 0 0 .
4 0 .% of items m e a n a n g l e ( p o s ) : 0 .m
e a n a n g l e ( n e g ) : 1 .p o
s n e g
( h )
BC l o ss ( t a il )
F i gu r e 4 :
M a gn i fi e d V
i
e w o f
F i gu r e 1 .
B ExperimentsB.1 Experimental Settings
Datasets.
We conduct experiments on five real-world benchmark datasets: Tencent [47], AmazonBook [48], Alibaba-iFashion [49], Douban Movie [50], and KuaiRec [52].
All datasets are publicand vary in terms of size, domain, and sparsity.
Table 4 summarizes the dataset statistics, where thelong-tail degree is monitored by KL-divergence between the item popularity distribution and theuniform distribution, i.e., D KL ( ˆ P data || Uniform ) .
A larger KL-divergence value indicates that theheavier portion of interactions concentrates on the head of distribution.
In the stage of pre-processingdata, we follow the standard 10-core setting [20, 59] to filter out the items and users with less thanten interactions.
Data Splits.
For comprehensive comparisons, almost all standard test distributions in CF arecovered in the experiments: balanced test set [22, 23, 24], randomly selected imbalanced test set[10, 53], temporal split test set [20, 21, 12], and unbiased test set [13, 52, 51].
Three datasets ( i.e.,
Tencent, Amazon-Book, Alibaba-iFashion) are partitioned into both balanced and randomly selectedimbalanced evaluations.
As an intervention test, the balanced evaluation ( i.e., uniform distribution)is frequently employed in recent debiasing CF approaches [22, 23, 24, 60].
Douban is split basedon the temporal splitting strategy [61].
KuaiRec is an unbiased, fully-observed dataset in which thefeedback of the test set’s interaction is explicitly collected.
Evaluation Metrics.
We adopt the all-ranking strategy [62], i.e., for each user, all items are rankedby the recommender model, except the positive ones in the training set.
To evaluate the quality ofrecommendation, three widely-used metrics are used: Hit Ratio (HR@ K ), Recall@ K , Normalized
Discounted Cumulative Gain (NDCG@ K ), where K is set as 20 by default.
Baselines.
We validate our BC loss on two widely-used CF models, MF [26] and LightGCN [25],which are representatives of the conventional and state-of-the-art CF models.
We compare BC losswith the popular debiasing strategies in various research lines: sample re-weighting (IPS-CN
[15]),bias removal by causal inference (MACR [22], CausE [19]), and regularization-based framework(sam+reg [12]).
We also compare BC loss to other standard losses used in collaborative filteringincluding most commonly used loss (BPR loss [26]) and newest proposed softmax losses (CCL [54]and SSM [55]).
Parameter Settings.
We conduct experiments using a Nivida-V100 GPU (32 GB memory) on aserver with a 40-core Intel CPU (Intel(R) Xeon(R) CPU E5-2698 v4).
We implement our BC loss in
PyTorch.
Our codes, datasets, and hyperparameter settings are available at https://anonymous.
all are optimized by Adam [63] optimizer with the batch size
64,learning rate as 1e-3, and the coefficient of regularization
1e- all experiments
Following thedefault embedding
LightGCN earlystop strategy training
Recall@20 on the validation set not for 10 successiveepochs grid search is conducted to tune the critical hyperparameters of each strategy to choose thebest models w.r.t.
Recall@20 on the validation set
For softmax, SSM, and BC loss, we search τ in[0.06, 0.14] with a step size of 0.02.
For CausE, 10% of training data with balanced distribution areused as intervened set
[Table]
and cf _ pen is tuned in [0 . 01 , 0 . 1] with a step size of 0.02.
For MACR, we
[Table]
[Table]
[Table]
follow the original settings to set weights for user branch α = 1 e − 3 and item branch β
= 1 e − 3 ,respectively.
We further tune hyperparameter c = [0 , 50] with a step size of 5.
For CCL loss, wesearch w in { 1 , 2 , 5 , 10 , 50 , 100 , 200 } , m in the range [0 . 2 , 1] with a step size of 0.2.
When it cometo the number of negative samples, the softmax, SSM, CCL, and BC loss set 128 for MF backboneand in-batch negative sampling for LightGCN models.
B.2 Training Cost
In terms of time complexity, as shown in Table 5, we report the time cost per epoch and in total ofeach baselines on Tencent.
Compared with backbone methods ( i.e., MF and LightGCN), BC lossadds very little computing complexity to the training process.
B.3 Evaluations on Unbiased Test Set
Motivation.
Because of the missing-not-at-random condition in a real recommender system, offlineevaluation on collaborative filtering and recommender system is commonly acknowledged as achallenge.
To close the gap, Yahoo!R3 [51] and Coat [13] are widely used, which offer unbiased testsets that are collected using the missing-complete-at-random (MCAR) concept.
Additionaly, newlyproposed KuaiRec [52] also provides a fully-observed unbiased test set with 1,411 users over 3,327videos.
We conduct experiments on all these datasets for comprehensive comparison, and KuaiRecis also included as one of our unbiased evaluations for two key reasons: 1) It is significantly largerthan existing MCAR datasets ( e.g., Yahoo! and Coat); 2) It overcomes the missing values problem,making it as effective as an online A/B test.
Results.
Table 6 and 8 illustrate the unbiased evaluations on KuaiRec, Yahoo!R3, and Coat datasetusing LightGCN and MF as backbone models, respectively.
The best performing methods are boldand starred, while the strongest baselines are underlined; Imp.% measures the relative improvementsof BC loss over the strongest baselines.
BC loss is consistently superior to all baselines w.r.t.
allmetrics.
It indicates that BC loss truly improves the generalization ability of recommender.
B.4 Performance Comparison with Standard Loss Functions in CF
Motivation.
To verify the effectiveness of BC loss as a standard learning strategy in collaborativefiltering, we further conduct the experiments over various datasets between BC loss, BPR loss, CCLloss [54], and SSM loss [55].
We choose these three losses as baselines for two main reasons: 1)
BPR loss is the most commonly applied in recommender system; 2) SSM and CCL are most recentproposed losses, where SSM is also a softmax loss and CCL employs a global margin.
Results.
Table 7 reports the performance on both balanced and imbalanced test sets on variousdatasets among different losses.
We have two main observations: (1) Clearly, our BC loss consistentlyoutperforms CCL and SSM; (2) CCL and SSM achieve comparable performance to BC loss in theimbalanced evaluation settings, while performing much worse than BC loss in the balanced evaluation
[Table]
[Table]
settings.
This indicates the superiority of BC loss in alleviating the popularity bias, and furtherjustifies the effectiveness of the bias-aware margins.
Table 8 shows the unbiased evaluations on Yahoo!R3 and Coat dataset.
With regard to all criteria, BCloss constantly outperforms all other losses.
It verifies the effectiveness of instance-wise bias marginof BC loss.
B.5 Study on BC Loss (RQ3) - Effect of Popularity Bias Extractor
Motivation.
To check the effectiveness of popularity bias extractor, we need to answer two mainquestions: 1) what kinds of interactions will be learned well by the bias extractor?
What does thelearned bias-angle distribution look like? 2) can BC loss benefit from the bias margin extractedaccording to the popularity bias extractor in various groups of interactions?
We devise two distinctexperiments to tackle these two problems.
Experiment Setting A. Users can be divided into three parts: head, mid, and tail, based on theirpopularity scores.
Analogously, items can be partitioned into the head, mid, and tail parts.
As such,we can categorize all user-item interactions into nine subgroups.
We have visualized the learnedangles for various types of interactions in the following table 5.
Results A.
The table 5 shows the learned angles over all subgroups.
We find that interactions betweenhead users and head items tend to hold small angles.
Moreover, as evidenced by the high standarddeviation, the interactions stemming from the same subgroup types are prone to receive a wide rangeof angular values.
This demonstrates the variability and validity of instance-wise angular margins.
Experiment Setting B. We partition interactions into disjoint subgroups on Tencent dataset, based onthe bias degree estimated by our popularity bias extractor.
These subgraphs are composed of (1) hardinteractions with low bias degrees (40% of total interactions, Popularity rank > 1000), and (2) easyinteractions with high bias degrees (15% of total interactions, Popularity rank < 100), respectively.
Then, we use four losses (BPR, CCL, SSM, and BC losses) to train the same MF backbone.
Results B. Figure 4 depicts the performance of each loss in the hard-interaction subgroup, easyinteraction subgraph, and all-interaction group, during the training phase.
With the increase of trainingepochs, the performance of BPR drops dramatically in the hard-interaction subgroup, while increasingin the all-interaction group.
Possible reasons are BPR is easily influenced by the interaction-wise bias:
BPR focuses largely on the majority subgroups of easy interactions, while sacrificing the performanceof the minority subgroups of hard interactions.
In contrast, the performance of BC loss consistentlyincreases over every subgroup during training.
This indicates that BC loss is able to mitigate thenegative influences of popularity bias, thus further justifying that the popularity bias extract capturesthe interaction-wise bias well.
[Figure]