Incorporating Bias-aware Margins into Contrastive Loss for Collaborative Filtering
Anonymous Author(s)
Abstract
Collaborative ﬁltering (CF) models easily suffer from popularity bias, which makesrecommendation deviate from users’ actual preferences.
However, most currentdebiasing strategies are prone to playing a trade-off game between head and tailperformance, thus inevitably degrading the overall recommendation accuracy.
Toreduce the negative impact of popularity bias on CF models, we incorporate Biasaware margins into Contrastive loss and propose a simple yet effective BC Loss ,where the margin tailors quantitatively to the bias degree of each user-item interaction.
We investigate the geometric interpretation of BC loss, then further visualizeand theoretically prove that it simultaneously learns better head and tail representations by encouraging the compactness of similar users/items and enlarging thedispersion of dissimilar users/items.
Over six benchmark datasets
, we use BCloss to optimize two high-performing CF models.
On various evaluation settings( i.e., imbalanced/balanced, temporal split, fully-observed unbiased, tail/head testevaluations), BC loss outperforms the state-of-the-art debiasing and non-debiasingmethods with remarkable improvements.
Considering the theoretical guaranteeand empirical success of BC loss, we advocate using it not just as a debiasingstrategy, but also as a standard loss in recommender models.
Codes are available athttps://anonymous.4open.science/r/BC-Loss-9DF2 .
Introduction
At the core of leading collaborative ﬁltering (CF) models is the learning of high-quality representationsof users and items from historical interactions.
However, most CF models easily suffer from thepopularity bias issue in the interaction data [1, 2, 3, 4].
Speciﬁcally, the training data distribution istypically long-tailed, e.g., a few head items occupy most of the interactions, whereas the majorityof tail items are unpopular and receive little attention.
The CF models built upon the imbalanceddata are prone to learn the popularity bias and even amplify it by over-recommending head items andunder-recommending tail items.
As a result, the popularity bias causes the biased representations with poor generalization ability, making recommendations deviate from users’ actual preferences.
Motivated by concerns of popularity bias, studies on debiasing have been conducted to lift the tail performance.
Unfortunately, most prevalent debiasing strategies focus on the trade-off between head and tail evaluations (see Table 3), including post-processing re-ranking [5, 6, 7, 8, 9], balanced training loss [10, 11, 12, 9], sample re-weighting [13, 14, 15, 16, 17, 18], and head bias removal by causal inference [19, 20, 21, 22].
Worse still, many of them hold some assumptions that areinfeasible in practice, such as the balanced test distribution is known in advance to guide thehyperparameters’ adjustment [23, 22], or a small unbiased data is present to train the unbiased model
Consequently, they pursue improvements on tail items but exacerbate the performance
[Table]
o f i t
e m s mean angle (pos):0.387 mean angle (neg):1.861 posneg (f) Softmax loss (tail) tail user pos item neg item 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Angle 0.0 0.
o f i t
e m s mean angle (pos):0.226 mean angle (neg):1.604 posneg (g) IPS-CN [15] (tail) tail user pos item neg item 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Angle 0.0 0.
o f i t
e m s mean angle (pos):0.077 mean angle (neg):1.831 posneg (h) BC loss (tail) sacriﬁce of head items, leading to a severe overall performance drop.
The trade-off between the headand tail evaluations results in suboptimal representations, which derails the generalization ability.
In this paper, we conjecture that an ideal debiasing strategy should learn high-quality head and tailrepresentations with powerful discrimination and generalization abilities, rather than playing a tradeoff game between the head and tail performance.
Here we follow the prior studies [15, 23, 13, 14]to focus on one key ingredient in representation learning: the loss function.
Figure 1 depicts theitem representations, which is optimized via two non-debiasing losses (BPR [26] and Softmax [27])and one debiasing loss (IPS-CN [15]).
Wherein, representation discrimination is reﬂected in howwell the positive items of a user are apart from the negatives.
Our insights are: (1) For a user, thenon-debiasing losses are inadequate to discriminate his/her positive and negative items well, sincetheir representations are largely overlapped as Figures 1a and 1b show; (2) Although IPS-CN achievesbetter discrimination power in the tail group than BPR ( cf.
positive items get smaller angles to theego user in Figure 1g, as compared to Figure 1e), it gets worse discrimination ability in the head ( cf.positive items hold larger angles to the ego user in Figure 1c, as compared to Figure 1a).
Towards this end, we incorporate Bias-aware margins into Contrastive Loss and devise a simple yeteffective BC Loss to guide the head and tail representation learning of CF models.
Speciﬁcally, weﬁrst employ a bias degree extractor to quantify the inﬂuence of interaction-wise popularity — thatis, how well an interaction is predicted, when only popularity information of the target user anditem is used.
Interactions involving inactive users and unpopular items often align with lower biasdegrees, indicating that popularity fails to reﬂect user preference faithfully.
In contrast, interactionswith active users and popular items are spurred by the popularity information, thus easily inclining tohigh bias degrees.
We then move on to train the CF model by converting the bias degrees into theangular margins between user and item representations.
If the bias degree is low, we impose a largermargin to strongly squeeze the tightness of representations.
In contrast, if the bias degree is large, weexert a small or vanishing margin to reduce the inﬂuences of biased representations.
Through this way, for each ego user’s representation, BC quantitatively controls its bias-aware margins with itemrepresentations — adaptively intensifying the representation similarity among positive items, whilediluting that among negative items.
Beneﬁting from stringent and discriminative representations, BCloss signiﬁcantly improves both head and tail performance.
Furthermore, BC loss has three desirable advantages.
First, it has a clear geometric interpretation,as illustrated in Figure 2.
Second, it brings forth a simple but effective mechanism of hard examplemining (See Appendix A.2).
Third, we theoretically reveal that BC loss tends to learn a low-entropycluster for positive pairs ( e.g., compactness of matched users and items) and a high-entropy spacefor negative pairs ( e.g., dispersion of unmatched users and items)
(See Theorem 1).
Considering thetheoretical guarantee and empirical effectiveness, we argue that BC loss is not only promising toalleviate popularity bias, but also suitable as a standard learning strategy in CF.
Preliminary of Collaborative Filtering (CF)
Task Formulation.
Personalized recommendation is retrieving a subset of items from a large catalogto match user preference.
Here we consider a typical scenario, collaborative ﬁltering (CF) withimplicit feedback [28], which can be framed as a top- N recommendation problem.
Let O + ={ ( u, i ) |
y ui
= 1 } be the historical interactions between users U and items I , where y ui = 1 indicatesthat user u ∈ U has adopted item
i ∈ I before.
Our goal is to optimize a CF model ˆ y : U × I → Rthat latches on user preference towards items.
Modeling Scheme.
Scrutinizing leading CF models [26, 25, 29, 30], we systematize the commonparadigm as a combination of three modules: user encoder ψ ( · ) , item encoder φ ( · ) , and similarityfunction s ( · ) .
Formally, we depict one CF model as ˆ y ( u, i ) =
s ( ψ ( u ) , φ ( i )) , where ψ : U → R dand φ : I → R d encode the identity (ID) information of user u and item i into d -dimensional
representations, respectively; s : R d × R d → R measures the similarity between user and itemrepresentations.
In literature, there are various choices of encoders and similarity functions:
• Common encoders roughly fall into three groups: ID-based ( e.g., MF [26, 29], NMF [31], CMN[32]), history-based ( e.g., SVD++ [29], FISM [33], MultVAE [30]), and graph-based ( e.g., GCMC[34], PinSage [35], LightGCN [25]) fashions.
Here we select two high-performing encoders, MFand LightGCN, as the backbone models being optimized.
• The widely-used similarity functions include dot product [26], cosine similarity [36], and neuralnetworks [31].
As suggested in the recent study [36], cosine similarity is a simple yet effectiveand efﬁcient similarity function in CF models, having achieved strong performance.
For betterinterpretation, we take a geometric view and denote it by:
[Equation]
in which ˆ θ ui is the angle between the user representation ψ ( u ) and item representation φ ( i ) .Learning
Strategy.
To optimize the model parameters, CF models mostly frame the top- N rec- ommendation problem into a supervised learning task, and resort to one of three classical learning strategies: pointwise loss ( e g., binary cross-entropy [37], mean square error [29]), pairwise loss(
e.g.,
BPR [26], WARP [38]), and softmax loss [28].
Among them, pointwise and pairwise losses are
long- standing and widely-adopted objective functions in CF.
However, extensive studies [9, 1, 39]have analytically and empirically conﬁrmed that using pointwise or pairwise loss is prune to propagatemore information towards the head user-item pairs, which ampliﬁes popularity bias.
Softmax loss is much less explored in CF than its application in other domains like CV [40, 41].
Recent studies [36, 42, 43, 44, 45] ﬁnd that it inherently conducts hard example mining over multiplenegatives and aligns well with the ranking metric, thus attracting a surge of interest in recommendation.
Hence, we cast the minimization of softmax loss [27] as the representative learning strategy:
[Equation]
where ( u, i ) ∈ O
+ is one observed interaction of user u , while N u
= { j | y uj = 0 } is the set ofsampled unobserved items that u did not interact with before; τ is the hyper-parameter known as the temperature in softmax [46].
Nonetheless, modifying softmax loss to enhance the discriminativepower of representations and alleviate the popularity bias remains largely unexplored.
Therefore,our work aims to devise a more generic and broadly-applicable variant of softmax loss for CF tasks,which can improve the long-tail performance fundamentally.
Methodology of BC Loss
On the basis of softmax loss, we devise our BC loss and present its desirable characteristics.
3.1 Popularity Bias Extractor
Before mitigating popularity bias, we need to quantify the inﬂuence of popularity bias on a singleuser-item pair.
One straightforward solution is to compare the performance difference betweenthe biased and unbiased evaluations.
However, this is not feasible as the unbiased data is usuallyunavailable in practice.
Statistical metrics of popularity could be a reasonable proxy of the biasedinformation, such as user popularity statistics p u ∈ P ( i.e., the number of historical items that useru has interacted with before) and item popularity statistics p i ∈ P ( i.e., the number of observedinteractions that item i is involved in).
If the impact of the interaction between u and i can becaptured well based solely on such statistics, the model is susceptible to exploiting popularity bias forprediction.
Hence, we argue that the popularity-only prediction will delineate the inﬂuence of bias.
Towards this end, we ﬁrst train an additional module, termed popularity bias extractor, which onlytakes the popularity statistics as input to make prediction.
Similar to the modeling of CF ( cf. Section2), the bias extractor is formulated as a function ˆ y b : P × P → R :
[Equation]
(3) where the user popularity encoder ψ b : P → R d and the item popularity encoder φ b : P → R dmap the popularity statistics of user u and item i into d -dimensional popularity embeddings ψ b ( p u )and φ b ( p i ) , respectively; s : R d × R d → R is the cosine similarity function between popularityembeddings ( cf.
Equation (1)).
ˆ ξ ui is the angle between ψ b ( p u ) and φ b ( p i ) .
We then minimize the following softmax loss to optimize the popularity bias extractor:
[Equation]
This optimization enforces the extractor to reconstruct the historical interactions and makes thereconstruction reﬂect the interaction-wise bias degree.
Furthermore, we can distinguish hard interactions based on the bias degree, i.e., the interactions that can be hardly predicted by popularitystatistics ought to be more informative for representation learning in the target CF model.
In anutshell, the popularity bias extractor underscores the bias degree of each user-item interaction, whichsubstantively reﬂects how hard it is to be predicted.
3.2 BC Loss
We move on to devise a new BC loss for the target CF model.
Our BC loss stems from softmax loss butconverts the interaction-bias degrees into the bias-aware angular margins among the representationsto enhance the discriminative power of representations.
Our BC loss is:
[Equation]
where M ui is the bias-aware angular margin for the interaction ( u, i ) deﬁned as:
[Equation]
) where ˆ ξ ui is derived from the popularity bias extractor ( cf.
Equation (3)), and π − ˆ θ ui is the upperbound to restrict cos( · + M ui ) to be a monotonically decreasing function.
Intuitively, if a user-itempair ( u, i ) is the hard interaction that can hardly be reconstructed by its popularity statistics, it holds a 𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊𝑴 𝒖 𝒊𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊
[Figure]
𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊𝑴 𝒖 𝒊𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊(b) BC Loss (2D) 𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊𝑴 𝒖 𝒊𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊
[Figure]
𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊𝑴 𝒖 𝒊𝒌 𝒋 𝑜 𝒊 𝒌 𝒋 𝑜 𝒊 𝑴 𝒖𝒊
[Figure]
high value of ˆ ξ ui and leads to a high value of M ui ; henceforward, BC loss imposes the large angularmargin M ui between the negative item j and positive item i and optimizes the representations of useru and item i to lower ˆ ξ ui .
See more details and analyses in Section 4.
It is noted that BC loss is extremely easy to implement in recommendation tasks, which only needs torevise several lines of code.
Moreover, compared with softmax loss, BC loss only adds negligiblecomputational complexity during training but achieves more discriminative representations.
Hence,we recommend to use BC loss not only as a debiasing strategy to alleviate the popularity bias, butalso as a standard loss in recommender models to enhance the discriminative power.
Note that themodeling of M ui is worth exploring, such as the more complex version M ui = min
{ λ · ˆ ξ ui , π − ˆ θ ui }where λ controls the strength of the bias-margin.
Meanwhile, carefully designing a monotonicallydecreasing function helps to get rid of the upper bound restriction.
We will leave the exploration ofbias-margin in future work.
Analyses of BC Loss
We analyze desirable characteristics of BC loss.
Speciﬁcally, we start by presenting its geometric interpretation, and then show its theoretical properties w.r.t.
compactness and dispersion of representations.
The hard mining mechanism of BC loss is discussed in Appendix A.2.
4.1 Geometric Interpretation
Here we probe into the ranking criteria of softmax loss and BC loss, from the geometric perspective.
To simplify the geometric interpretation, we analyze one user u with one observed item i andonly two unobserved items j and k .
Then the posterior probabilities obtained by softmax loss are:exp(cos(ˆ θ ui ) /τ )
exp(cos(ˆ θ ui ) /τ ) +exp(cos(ˆ θ uj ) /τ )
+exp(cos(ˆ θ uk ) /τ ) .
During training, softmax loss encourages the rankingcriteria ˆ θ ui < ˆ θ uj and ˆ θ ui < ˆ θ uk to model the basic assumption that the observed interaction ( u, i )indicates more positive cues of user preference than the unobserved interactions ( u, j ) and ( u, k ) .
Intuitively, to make the ranking criteria more stringent, we can impose an angular margin M ui on itand establish a new criteria ˆ θ ui + M ui < ˆ θ uj
and ˆ θ ui + M ui < ˆ θ uk .
Directly formulating this ideaarrives at the posterior probabilities of BC loss:
[Equation]
Obviously, BC loss is more rigorous about the ranking assumption compared with softmax loss.
See
Appendix A.2 for more detailed explanations.
We then depict the geometric interpretation and comparison of softmax loss and BC loss in Figure 2.
Assume the learned representations of i , j , and k are given, and softmax and BC losses are optimizedto the same value.
In softmax loss, the constraint boundaries for correctly ranking user u ’s preferenceare ˆ θ ui
=
ˆ θ uj and ˆ θ ui
= ˆ θ uk ; whereas, in BC loss, the constraint boundaries are ˆ θ ui + M ui
= ˆ θ
ujand
ˆ θ ui + M ui
= ˆ θ uk .
Geometrically, from softmax loss ( cf.
Figure 2c) to BC loss ( cf.
Figure 2d), itis a more stringent circle-like region on the unit sphere in the 3D case.
Further enlarging the margin
M ui will lead to a smaller hypercycle-like region, which is an explicit discriminative constraint on amanifold.
As a result, limited constraint regions squeeze the tightness of similar items and encouragesthe separation of dissimilar items.
Moreover, with the increase of representation dimension, BC losshas more restricted learning requirements, exponentially decreasing the area of constraint regions forcorrect ranking, and becomes progressively powerful to learn discriminative representations. [Table]
4.2 Theoretical Properties
[Table] We show that BC loss improves head and tail representation learning by enforcing the compactnessof matched users and items, while imposing the dispersion of unmatched users and items.
Theorem 1.
Let v u .
=
ψ ( u ) , v i .
=
φ ( i ) , and c u = 1 |P u | (cid:80) i ∈P u v i , c i =
|P i | (cid:80) u ∈P i v u , where
P u
= { i | y ui
= 1 } and N u
= { i | y ui = 0 } are the sets of user u ’s positive and negative items,respectively; P i = { u |
y ui
= 1 } is the set of item i ’s positive users.
Assuming the representations ofusers and items are normalized, the minimization of BC loss is equivalent to minimizing a compactnesspart and a dispersion part simultaneously:
[Equation]
Discussion.
c u is the averaged representations of all items that u has interacted with, which describesu ’s interest; similarly, c i proﬁles its user group.
For the compactness part, BC loss forces the user’spositive items to be user-centric and vice versa.
From the entropy perspective, compactness part tendsto learn a low-entropy cluster for positive interactions, i.e., high compactness for similar users anditems.
For the dispersion part, for users and items from unobserved interactions, BC loss maximizesthe pairwise euclidean distance between their representations and encourages them to be distant fromeach other; Hence, from the entropy viewpoint, dispersion part levers the spread of representations tolearn a high-entropy representation space, i.e., large separation degree for dissimilar users and items.
Experiments
We aim to answer the following research questions:
• RQ1: How does BC Loss perform compared with debiasing strategies in various evaluations? • RQ2: Does BC loss cause the trade-off between head and tail performance? • RQ3: What are the impacts of the components ( e.g., temperature, margin) on BC Loss?
Baselines & Datasets.
SOTA debiasing strategies in various research lines are compared: samplere-weighting (IPS-CN [15]),
bias removal by causal inference (MACR [22], CausE [19]), andregularization-based framework (sam+reg [12]).
Extensive experiments are conducted on six realworld benchmark datasets: Tencent [47], Amazon-Book [48], Alibaba-iFashion [49], Yelp2018 [25],
Douban Movie [50], and KuaiRec [51].
For comprehensive comparisons, almost all standard testdistributions in CF are covered in the experiments: balanced test set [22, 23, 24], randomly selectedimbalanced test set [10, 52], temporal split test set [20, 21, 12], and unbiased test set [13, 51, 53].
See more experiments on KuaiRec [51] for unbiased test evaluation in Appendix B.3.
5.1.1 Evaluations on Imbalanced and Balanced Test Sets.
Motivation.
Many prevalent debiasing methods assume that test distribution is known in advance[22, 23, 10], i.e., the validation set has similar distribution with the test set.
Moreover, only animbalanced or balanced test set is evaluated.
However, in real-world applications, the test distributionsare usually unavailable and can even reverse the prior in the training distribution.
We conjecture thata good debiasing recommender is required to perform well on both imbalanced and balanced testdistributions.
In our settings, no information about the balanced test is provided in advance.
Data Splits.
The models are identical across both imbalanced and balanced evaluations.
The testdistribution in the balanced evaluation is uniform, i.e., randomly sample 15% of interactions withequal probability w.r.t.
items.
Besides, the test splits for the imbalanced test are similarly longtailed like the train and validation sets, i.e., randomly split the remaining interactions into training,validation, and imbalanced test sets ( 60% : 10% : 15% ).
Results.
Table 1 reports the comparison of performance in imbalanced and balanced test evaluations.
The best performing methods are bold and starred, while the strongest baselines are underlined;
Imp.% measures the relative improvements of BC loss over the strongest baselines.
We observe that:
• BC loss signiﬁcantly outperforms the state-of-the-art baselines in both balanced and imbal
anced evaluations across all datasets.
In particular, it achieves consistent improvements
over the
best debiasing baselines and original CF models by 12 . 1% ∼ 58 .
This clearly demonstratesthat BC loss not only effectively alleviates the ampliﬁcation of popularity bias but also improvesthe discriminative power of representations.
Moreover, Table 5 shows the computational costs ofall methods.
Compared to the backbone models, BC loss only adds negligible time complexity.
• Debiasing baselines sacriﬁce the imbalanced performance and perform inconsistently acrossdatasets.
Debiasing strategies generally achieve higher balanced performance at the expense of alarge imbalanced performance drop.
Speciﬁcally, the strongest baselines over all imbalanced testsets are the original CF models.
Worse still, as the degree of data sparsity increases, some debiasingmethods fail to quantify the popularity bias and limit their bias removal ability.
For example, in thesparest Alibaba-iFashion dataset, the results of MF+IPS-CN, MF+CausE, LightGCN+MACR, and
LightGCN+CausE on the balanced evaluation are lower than original CF models (MF or LightGCN).
In contrast, beneﬁting from popularity bias-aware margin, BC loss can learn discriminativerepresentations that accomplish more profound user and item understanding, leading to higher headand tail recommendation quality.
5.1.2 Evaluations on Temporal Split Test Set
Motivation.
[Table]
In real applications, popularity biasdynamically changes over time.
Herewe consider temporal split test evaluation on Douban Movie where thehistorical interactions are sliced intothe training, validation, and test sets(70%:10%:20%) according to thetimestamps.
Results.
As Table 2 shows, BC loss is steadily superior to all baselines w.r.t.
all metrics on Douban
Movie.
For instance, it achieves signiﬁcant improvements over the MF and LightGCN backbonesw.r.t.
Recall@20 by 10.2% and 10.4%, respectively.
This validates that BC loss endows the backbonemodels with better robustness against the popularity distribution shift and alleviates the negative inﬂuence of popularity bias.
Surprisingly, none of the debiasing baselines could maintain a comparableperformance to the backbones.
We ascribe the failure to their preconceived idea of tail items, whichpossibly change over time.
[Table]
[Figure] [Figure] [Figure]
[Figure]
B a l a n c e d N D
C G @ 0.
[Figure]
I m b a l a n c e d
N D C G @(b) MF+BC loss
[Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure]
B a l a n c e d N D
C G @ 0.
[Figure]
I m b a l a n c e d
N D C G @(
[Figure]
loss
5.2 Head, Mid, & Tail Performance (RQ2)
Motivation.
To further evaluate whether BC loss lifts the tail performance by inevitably sacriﬁcingthe head performance, we divide the test set of Tencent into three subgroups, according to theinteraction number of each item: head (popular items that are in the top third), mid (normal itemsin the middle), and tail (unpopular items in the bottom third).
Most previous studies focus onaverage NDCG@20 for evaluation, especially balanced test evaluations [22, 23].
However, averagemetrics could be insufﬁcient to reﬂect the performance of each subgroup.
A trivial solution toachieve high performance is promoting the rankings of low-popularity items in the recommendations.
In this case, only the average metrics are not reliable on the balanced test.
Therefore, we reportthe performance of individual subgroups on both balanced and imbalanced test sets for a morecomprehensive comparison.
Results.
Table 3 shows the evaluations of the head, mid, and tail subgroups.
The red and bluenumbers in percentage separately refer to the improvement and decline of each method relative to theoriginal CF model (MF or LightGCN).
We ﬁnd that:
• BC loss is the only method that consistently yields remarkable improvements in every subgroup.
With a closer look at the head evaluation, BC loss shows its ability to learn more discriminative representations for popular items across imbalanced and balanced settings.
In particular, itachieves signiﬁcant improvements over MF and LightGCN w.r.t.
head NDCG by 40% and 17% inthe imbalanced test evaluation, respectively.
We attribute improvements to the usage of bias-awaremargin, which boosts the recommendation quality for the tail and head items.
• As the performance comparison among subgraphs in the imbalanced scenario shows, thebaselines enhance the tail performance but sacriﬁce the head performance.
Speciﬁcally,these baselines hardly maintain the head performance and show a clear trade-off trend betweenthe head and tail performance.
Taking MACR as an example, although the great improvement( +1662% ) over MF is achieved in the tail subgraph, it brings in the dramatic drop ( − 62% ) in thehead subgraph, which lowers the overall performance by a big drop ( − 60% ).
Here we ascribe thetrade-off to blindly promoting the rankings of tail items for matched and unmatched, rather thanimproving the discriminative power of representations.
5.3 Study on BC Loss (RQ3)
Effect of Bias-aware Margin.
Figure 3a displays the performance on balanced and imbalanced testsets on Tencent among softmax loss, BC loss with constant margin M [40], and BC loss with adaptivebias-aware margin.
BC loss achieves the best performance, illustrating that bias-aware margin indeedis effective at reducing popularity bias and learning high-quality representations.
Effect of Temperature τ .
BC loss has one hyperparameter to tune — temperature τ in Equation(5).
In Figure 3b and 3c, both balanced and imbalanced evaluations exhibit the concave unimodalfunctions of τ , where the curves reach the peak almost synchronously in a small range of τ .
Forexample, MF+BC loss gets the best performance when τ = 0 .
05 and τ = 0 .
07 in balanced andimbalanced settings, respectively; We observe similar trends on other datasets and skip them due tothe space limit.
This justiﬁes that BC loss does not suffer from the trade-off between the balanced andimbalanced evaluations and improves the generalization without sacriﬁcing the head performance.
Related Work
Prevalent popularity debiasing strategies in CF roughly fall into four research lines.
Post-processing re-ranking methods [5, 6, 7, 8, 9] are applied to the output of the recommendersystem without changing the representations of users and items.
The purposes of modifying theranking of models can be various: Calibration [5] ensures that the past interests proportions ofusers are expected to maintain at the same level; RankALS [6] aims to increase the diversiﬁcationof recommendation; FPC [8] investigates the popularity bias in the dynamic recommendation byrescaling the predicted scores.
Regularization-based frameworks [10, 11, 12, 9] explore the use of regularization that provides atunable mechanism for controlling the trade-off between recommendation accuracy and coverage.
The difference among these methods is the design of penalty terms: ALS+Reg [11] deﬁnes intra-listdistance as the penalty to achieve the fair recommendation; ESAM [10] introduces the attributecorrelation alignment, center-clustering, and self-training regularization to learn good feature representations; sam-reg [12] regularizes the biased correlation between user-item relevance and itempopularity; Reg [9] decouples the item popularity with the model preference predictions.
Sample re-weighting methods [13, 14, 15, 16, 17, 18], also known as Inverse Propensity Score (IPS)view the item popularity in the training set as the propensity score and exploit its inverse to re-weightloss of each instance.
To address the high variance of re-weighted loss, many of them [15, 14] furtheremploy normalization or smoothing penalty to attain a more stable output.
However, the unreliabilityof methods is due to their measurement of the propensity score, leveraging the item frequency butfailing to consider interaction-wise popularity bias.
Bias removal by causal inference methods [19, 24, 23, 20, 21, 22], getting inspiration from therecent success of counterfactual inference, specify the role of popularity bias in assumed causalgraphs and mitigate the bias effect on the prediction.
However, the causal structure is heuristicallyassumed based on the author’s understanding, without any theoretical guarantee.
BC loss opens up a possibility of conventional debiasing methods in CF that mitigate the popularitybias by enhancing the discriminative power.
Recent studies, boosting the discriminative featurespaces by modiﬁed softmax loss are mainly discussed in face recognition, where a constant margin isadded [40] to better classify.
We transfer it in CF and compare it with BC loss in Figure 3a.
Conclusion
Despite the great success in collaborative ﬁltering, today’s popularity debiasing methods are stillfar from being able to improve the recommendation quality.
In this work, we proposed a simple yeteffective BC loss, utilizing popularity bias-aware margin to eliminate the popularity bias.
Groundedby theoretical proof, clear geometric interpretation and real-world visualization study, BC loss booststhe head and tail performance by learning a more discriminative representation space.
Extensiveexperiments verify that the remarkable improvement in head and tail evaluations on various test setsindeed comes from the better representation rather than simply catering to the tail.
The limitations of BC loss are in three respects, which will be addressed in future work: 1) the modeling of bias-aware margin is worth exploring, which could signiﬁcantly inﬂuence the performance of
BC loss, 2) multiple important biases, such as exposure and selection bias, are not considered, and 3)more experiments comparing BC loss to standard CF losses ( e.g., cross-entropy, WARP) are neededto further demonstrate the power of BC loss in regular recommendation tasks.
We believe that thiswork provides a potential research direction to diagnose the debiasing of long-tail ranking and willinspire more works.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[No] This paper
proposes a novel debiasing algorithm for recommendation system, which does not have
any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We include
code by URL in abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix B.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]