F ROM I NTERVENTION TO D OMAIN T RANSPORTATION : A N OVEL P ERSPECTIVE TO O PTIMIZE R ECOMMENDATION
A BSTRACT
The interventional nature of recommendation has attracted increasing attention in recent years.
It particularly motivates researchers to formulate learning and evaluating recommendation as causal inference and data missing-not-at-random problems.
However, few take seriously the consequence of violating the critical assumption of overlapping, which we prove can significantly threaten the validity and interpretation of the outcome.
We find a critical piece missing in the current understanding of information retrieval (IR) systems: as interventions, recommendation not only affects the already observed data, but it also interferes with the target domain (distribution) of interest.
We then rephrase optimizing recommendation as finding an intervention that best transports the patterns it learns from the observed domain to its intervention domain.
Towards this end, we use domain transportation to characterize the learning-intervention mechanism of recommendation.
We design a principled transportation-constraint risk minimization objective and convert it to a two-player minimax game.
We prove the consistency, generalization, and excessive risk bounds for the proposed objective, and elaborate how they compare to the current results.
Finally, we carry out extensive real-data and semi-synthetic experiments to demonstrate the advantage of our approach, and launch online testing with a real-world IR system.
I NTRODUCTION
For information retrieval (IR) systems, the users’ willingness to interact is often intervened by the content we show.
Recommendations not only impact the potential response of the users, but also change the nature of the data collected for training machine learning models (Bottou et al., 2013).
The interventional nature of recommendation, as well as the fact that many questions in IR are counterfactual, e.g., ”what the response would have been if we recommended something else”, makes it natural to rephrase recommendation in the context of causal inference or missing data problems (Rubin, 2005; Little et al., 2002).
Among the two disciplines, technical tools of importance weighting (IW) and domain adaptation (DA) have been commonly applied.
Recently, there has been widespread interest in adapting those tools for learning the best recommendation (Schn- abel et al., 2016; Bonner and Vasile, 2018), but we find a critical discussion missing from most existing literature in this direction.
In many scientific disciplines, the effectiveness of IW and DA relies heavily on overlapping between source and target domain (Austin and Stuart, 2015; David et al., 2010).
The underlying reason also happens to be the motivation of exploration-exploitation techniques including bandits and reinforcement learning (Auer et al., 2002; Sutton and Barto, 1998): in the less-explored regions where few observations are collected, the confidence of selecting any hypothesis is discounted.
Unfortunately, modern IR systems are often cautious about exposing users to less relevant content for the sake of immediate satisfaction and revenue, even with the help of bandits or RL (Bonner and
Vasile, 2018).
In particular, many real-world recommenders are deterministic, e.g., an item is shown with probability either zero or one, which further limits the exploration and thus the coverage of collected data.
In Section 3, we rigorously prove for both IW- and DA-based learning the hardness or impossibility results caused by insufficient overlap.
Intuitively, when a less-explored instance is passed to a candidate hypothesis, even if DA or IW can improve the accuracy of the prediction, they inevitably increase the uncertainty of that prediction as a consequence of the bias-variance tradeoff.
To a certain point, the increased uncertainty from having too many instances in the weakly overlapped regions will fundamentally limit what any learning procedure can achieve.
Further, if the instance lies in the non-overlapped region, we can barely exceed regular extrapolation 1 (Xu et al., 2021b).
We resolve the fundamental limitation of insufficient overlapping in IR with the key insight that the interventional impact of recommendation is not exclusive to observed data, but also interferes with the target domain of interest — the instances that will be affected by the recommendations we are about to make.
For instance, the placement of a recommendation on the webpage proposed to the user directly influences the occurrence of clicks.
Therefore, we alternatively consider optimizing recommendation as searching for an intervention that best transports the patterns it learns from the source domain to its intervention domain.
The procedure is best described by transport instead of transfer because it will interfere with how the learnt patterns are carried to the target domain.
This change of view describes a complex but realistic working mechanism that involves learning, intervention, and transportation, as depicted in Figure 1.
On the one hand, we wish to learn patterns that can be transported to the target domain of interest.
On the other hand, the patterns we identify lead to the intervention (in the form of recommendation) that generates the target domain.
Following this working mechanism, we impose constraints on the domain transportation between the (reweighted) source domain and the future intervention domain.
We point out that reweighting is essential for learning from interventional feedback data (Rosenbaum and Rubin, 1983), especially for the working mechanism we identify here because the relationship between the source and target domain is more involved.
Towards this end, we propose a novel solution via transportation-constraint risk minimization.
We rigorously investigate for our objective:
• the consistency, generalization, and excessive risk bounds that provide theoretical insights and guarantees; • the equivalence to a two-player minimax game that can be efficiently solved by gradient descentascent algorithm; • convergence to the standard IW-based solution for the ideal fully-overlapped bandit feedback;
We demonstrate the empirical performance of our method via comprehensive experiments via simulation, real data analysis as well as online experimentation and testing in a real-world IR system
R ELATED W ORK
Bottou et al.
(2013) first reveals the importance of characterizing the working mechanism of IR systems.
However, they study specifically the advertising systems where the underlying causal structures are assumed given.
Our work significantly extends their discussion by moving beyond causal mechanisms.
Issues caused by ignoring the interventional nature of recommendation have been pointed out for numerous scenarios (Chen et al., 2020).
Steck (2010) and Liang et al.
(2016) address optimizing recommendation in the context of missing data and causal inference, and the majority of recent follow-up work investigate particularly inverse propensity weighting (IPW) and learning domain invariant representation methods (Chen et al., 2020; Saito, 2020; Yang et al., 2018; Joachims et al., 2017; Bonner and Vasile, 2018; Agarwal et al., 2019; Xu et al., 2020).
As mentioned in Section 1, they largely ignore the overlapping issue even when the recommendation is deterministic.
Our theoretical results on the interplay between overlapping and learning performance are novel and add to the current venue of revealing the tradeoff for IW- and DA-based learning (Byrd and Lipton, 2019; Johansson et al., 2019; Ben-David and Urner, 2012).
See Appendix D for further discussions on related work.
Existing solutions for insufficient overlapping focus primarily on reducing or calibrating the variance, e.g., as discussed in Swaminathan and Joachims (2015b); Wang et al.
Johansson et al.
They do not account for the unique working mechanism of IR where the domain discrepancy can be actively controlled.
We devise the domain-transportation constraint using the Wasserstein distance originated from the optimal transport theory (Courty et al., 2016; Redko et al., 2017).
The idea of converting a Wasserstein-constraint objective to two-player minimax game is similar to that of the generative adversarial network (Gulrajani et al., 2017), however, we do not make any generative assumption on the observed data.
Finally, learning from bandit feedback and the associated counterfactual risk minimization can be viewed as a special case of our framework (Swaminathan and
Joachims, 2015a).
We will use it as a test bed to examine the guarantees of the proposed approach.
P RELIMINARIES
We use upper-case letters to denote random variables and measures, bold-font letters for vectors and matrices.
We denote by ∥·∥ 2 , ∥·∥ ∞ , ∥·∥ L the ℓ 2 , ℓ ∞ norm and the Lipschitz constant of a function.
For clarity, we borrow the item recommendation setting to illustrate the IR system we discuss next.
The set of all users and items are given by U and I .
To be concise, we denote the features (or pre-trained embeddings) for users and items by x u for u ∈ U and x i for i ∈
I .
We study primarily the practical implicit-feedback setting with Y ui ∈ { 0 , 1 } , e.g., users express preferences implicitly via clicks.
Let D be the population of all possible ( x u , x i ) or ( u, i ) depending on the context.
The embeddings can also be free parameters under the representation mapping f from X or U ∪I to R d .
We let P be any base probability measure supported on the subsets of D , and let P n be the empirical versions according to the collected feedback of D n where n is the sample size.
We assume the weight of a user-item is positive and bounded by a constant M .
Equipped with either individual weights w ui ∈ [0 , M ] or weighting function w : U × I ( or X × X ) →
[0 , M ] , the reweighted measures are given by such as: P w ( u, i ) ∝ w ( u, i ) · P ( u, i ) .
The candidate recommendation algorithm f ∈ F outputs a score or probability for each user-item pair.
Top- k recommender often ranks the output and then renders the top- k items to the user.
We denote this procedure by the function: reco ( u, i ; f ) ∈ { 0 , 1 } .
Define the new measure on D induced by deploying f on D as: P f ( u, i ) ∝ reco ( u, i ; f ) · P ( u, i ) .
If we choose the base measure P as uniform, it is clear that P f is a uniform measure on the ( u, i ) pairs that will be exposed : { ( u, i ) ∈ D : reco ( u, i ; f ) = 1 } .
Finally, given a loss function ℓ for f , we define the corresponding risk under P : E P R ( f ) =
E ( u,i ) ∼ P ℓ (cid:0) f ( u, i ) , y ui (cid:1) , the weighted risk under P w : E P w R ( f ) =
E ( u,i ) ∼ P ℓ (cid:0) f ( u, i ) , y ui (cid:1) · w ( u, i ) .
Our setting covers a wide range of recommendation problems from classical collaborative filtering to sequential recommendation (when user embedding is a function of the item sequence), including those using bandits for exploration where f ( u, i ) becomes the exploration probability, and the base measure is given by the inverse propensities.
We defer the detail of that setting to Appendix A.1.
Consequence of Insufficient Overlapping .
We now rigorously justify our previous intuition that insufficient overlapping causes uncertainty that fundamentally limits what IW- and DA-based learning can achieve.
While the results we show for IW are universal, i.e., it applies to any hypothesis class, we discuss particularly for DA the skipgram negative sampling (SGNS)
algorithm (Mikolov et al., 2013).
The reason is that SGNS is a widespread technique for learning user or item embedding in IR systems (Xu et al., 2021a), and its more straightforward formulation will allow us to present concise theoretical results to the readers.
As a gentle introduction, IW-based learning uses the ratio Q ( u, i ) /P
( u, i ) – where P and Q represent the source and target domain – to reweigh the loss of each instance in order to correct the distribution shift.
On the other hand, DA-based learning assumes there exists some hypothesis that makes the optimal joint risk of inf f ∈F [ E Q R ( f ) + E P R ( f )] small.
For IW, we show the hardness of learning, i.e., the best target risk inf f E Q R ( f ) can be uncontrolled for any F due to insufficient overlapping.
For DA, we show the impossibility of learning, i.e., there does not exist a solution of SGNS that can make the optimal joint risk small when P and Q are not aligned.
Proposition 1.
Let P and Q be the training and target distribution supported on S P , S Q ⊆ D , and w ( u, i ) =
Q ( u,i ) P ( u,i ) , for ( u, i
) ∈ S P ∩ S Q .
D n consists of training instances sampled i.i.d from P .
Hardness of learning with IW : if S P = S Q , for any f ∈ F , with probability at least 1 − δ , it holds:
[Equation]
log N ∞ ( 1 n , F (cid:1) n , and P (cid:16)(cid:12)(cid:12) E Q R ( f ) − E P n,w R ( f ) (cid:12)(cid:12) ≳
(cid:112) ( d 1 ( P ∥ Q ) − 1) /n
(cid:17) > 0 under mild condition, where M := max { w ( u, i ) } , N ∞ ( 1 n , F (cid:1) :
= N ( 1 n , F , ℓ 2 n ∞ ) is the 1 n -covering number for F in ∥ · ∥ ∞ based on 2 n i.i.d samples from P , and d 1 ( P ∥ Q ) =
(cid:82) S Q ( dP/dQ ) dP is a divergence measure.
Further, if
S P ̸ = S Q , for any prior and posterior distributions π and τ on F , for risks associated with the 0loss, it holds with probability at least 1 − δ :
[Equation]
where d KL is the Kullback-Leibler divergence, d 1 ( P ∥ Q ) is the same as above but defined only on the overlapped part, and D n ( τ ) consists of disagreement terms detailed in Appendix A.2.
Impossibility of learning for DA with SGNS : let R ( f ) be the associated risk for the SGNS algorithm, and C u,i ( P ) , C u,i ( Q ) be the co-occurrence statistics under P , Q , as mentioned in Xu et al.
(2021a).
Even if S P = S Q , it holds that:
[Equation]
where c takes some positive value and equals to inf f E P R ( f ) + inf f E Q R ( f ) when P = Q a.s.
We defer all the proofs to Appendix A.2.
For IW, we reveal the generalization bounds for both the frequentist setting (the first set of bounds) and the PAC-Bayesian setting (the second bound).
While they emphasize conceptually on different learning guarantees (Germain et al., 2009), and while the latter can handle the existence of non-overlap ( S P ̸ = S Q ), we find them both showing insufficient overlapping (characterized by the discrepancy terms d 1 and d KL ) can cause arbitrarily large gaps leading to poor generalizations to the target domain.
For DA with SGNS, our lower bound reveals the impact of insufficient overlapping indirectly via a notion of co-occurrence statistics defined in
Xu et al.
(2021a).
In particular, if the source and target domains are well-aligned in terms of the co-occurrence statistics, the increased divergence term can make it impossible to find a hypothesis that simultaneously performs well in both domains.
Although this result holds only for SGNS, it nevertheless suggests the limitations of DA caused by insufficient overlapping in IR.
M ODEL AND T HEORY
As much as we would like to continue investigating the consequence of insufficient overlapping, for the purpose of this paper, we have shown the drawbacks of directly applying such as IW and DA to optimizing recommendation without accounting for the working mechanism of IR systems.
In what follows, we describe a principled solution motivated by our domain transportation view, which requires actively identifying the suitable training domain via reweighting.
Notably, we no longer perceive reweighting as a change-of-measure tool commonly implied in the causal inference and missing data solutions.
Instead, we use reweighing as a mechanism to help identify a particular training domain to ensure certain transportation properties (which we explain next) to the deployment domain 2 , which may concern only a subpopulation of all the potential user-item instances.
4. T RANSPORTATION - CONSTRAINED RISK MINIMIZATION
Throughout our discussion, we emphasize the transportability of patterns across domains, which can be difficult to define precisely.
However, if we make a a mild assumption that all patterns can be recovered by Lipschitz-bound functions, it immediately becomes clear that the Wasserstein distance can characterize our notion of transportability.
To show this point, we first describe an imbalanced binary classification problem with two domains P and Q .
Let ℓ 1 ( u ) =
up 1 and ℓ − 1 ( u ) =
[Equation]
be the positive and negative loss under prediction score u , where p 1 is the prior probability that a sample comes from P .
Interestingly, the L -Wasserstein distance between P and Q admits 3 :
[Equation]
which is exactly the optimal risk of the binary classification problem described in the first place!
Therefore, Wasserstein distance can be perceived as how well the classification patterns on P transports to Q by any means necessary, as long as they are Lipschitz-bounded.
Since the constant of L is simply a scaling factor, we simply continue with the 1 -Wasserstein distance.
Using d W to device the domain transportation contraint between P f and the particular training domain P w we seek to construct with our weighting mechanism, the learning objective is directly given by:
[Equation]
where D ( w, f ) is a shorthand for d W ( P w , P f ) , and ρ can be treated as a hyper-parameter.
As a sanity check, when ρ = 0 , the feasible set is given by: f ( u, i ) =
w ( u, i ) for all ( u, i ) , so we are directly optimizing the associated risk of the deployment domain.
However, this is a suboptimal practice for deterministic recommenders because in that case, only a small proportion of the feedback data will be considered and the result can be heavily biased toward the historical recommendation.
Therefore, a suitable ρ will give us the opportunity to better explore the whole feedback data.
Nevertheless, if ρ becomes too large, we are likely to overfit the feedback data due to the high capacity of f and w combined while the transportation constraint is too loose.
The proposed objective is tailored specifically to the working mechanism of IR systems, and it is natural to question the validity and effectiveness.
In what follows, we rigorously prove a series of guarantees associated with our novel learning objective, and discuss how it connect and compare to the existing results.
To begin with, we show that jointly optimizing w leads to the sample-consistency guarantee.
For clarity, we start with the feature-based learning where f ( u, i ) :
= f
( x u , x i ) , and defer the extension with the representation mapping to the next section.
Let Z :
= D × { 0 , 1 } , and we denote by ℓ f : Z → R + the composition with the loss function such that for z ui =
( x u , x i , y ui ) , we have ℓ f ( u, i ) :
= ℓ (cid:0) f ( u, i ) , y ui (cid:1) .
We use w ui as a shorthand for the individual weight.
Theorem 1.
Denote by D n ( w, f ) =
d W ( P n,w , P n,f ) .
Suppose that ℓ f is L-Lipschitz w.r.t ∥ · ∥ 2 .
Define [Equation] ( u,i ) ∈D n g ( u, i ) and M G = sup ∥
g ∥
L ≤ 1 , ( u,i ) ∈D g ( u, i ) .
Assume Z ∈
R d Z and w ∈ H .
Given any f ∈ F and ρ > 0 , it holds with probability at least 1 − δ that:
[Equation]
Here, ˜ w = arg min D n [Equation] w,
[Equation]
≤
[Equation]
E
[Equation]
R
[Equation]
f
[Equation]
[Equation]
arg [Equation] and R
[Equation] Also, [Equation] is given by
[Equation]
We relegate the proof of this theorem to Appendix A.4.
Crucially, other than showing that the sample-consistency result can still be achieved with the jointly-optimized weights, we point out that this bound is completely independent of distribution discrepancy terms that troubled IW and DA.
Notice that the bound depends on the weights only through R ¯ w ( f ) 2 and R ˜ w ( f ) 2 presented by the
C n term.
Given any f ∈ F , our result shows that both ¯ w and ˜ w are the minimizers of a weighted risk term, i.e. E P w R ( f ) ∝ (cid:80) D w ui ℓ f ( u, i ) , under some constraint involving ρ .
Therefore, thanks to the constraint of D ( w, f ) ≤ ρ and the joint optimization of w , R ¯ w ( f ) 2 and R ˜ w ( f ) 2 will have more controlled behaviors compared with the domain discrepancy terms in Proposition 1.
We now investigate the learning-theoretical guarantees of the proposed objective, particularly the generalization and excessive risk bounds.
Suppose the underlying labeling function of y ui with respect to ( x u , x i ) is deterministic.
With a little abuse of notation, we use z ∼ P n,f to denote sampling ( x u , x i , y ui ) via ( x u , x i ) ∼ P n,f and the deterministic labelling function.
Theorem 2.
Suppose inf ˜ z ∈Z { ℓ f (˜ z ) + λ ∥ z − ˜ z
∥ 2 } ≤ M F for all z ∈ Z and λ ≥ 0 , f ∈ F .
Let
R n ( F ) be the empirical Rademacher complexity: R n ( F [Equation]
= [Equation] [Equation] and σ takes {− 1 , 1 } with equal probability.
We consider individual weights w ui ∈ (0 , 1) for brevity.
Generalization Error : for any f ∈ F , it holds with probability at least 1 − δ that:
[Equation]
Excessive risk : define ˆ f = arg min f,D n ( w,f ) ≤ ρ E P n,w R ( f ) as the empirical optimal and f ∗ = arg min f,D ( w,f ) ≤ ρ E P w R ( f ) as the theoretical optimal.
With probability at least 1 − δ :
[Equation]
We defer both proofs to Appendix A.5.
Theorem 2 establishes two critical learning guarantees for the proposed transportation-constraint risk minimization.
In particular, the generalization error reveals how we resolve the previous drawback mentioned for using DA in IR system.
Recall from Section 3 that the joint optimal risk might be uncontrolled for DA since it can depend on how different the training and target domains are.
In this regard, with D ( w, f ) ≤ ρ explicitly constrains the discrepancy between the training and deployment domain, we have the additional − λρ term in the generalization error bound where the second term on the RHS can be think of as the empirical ”transported” risk.
In other words, the constraint ensures that the deployment risk will not deviate far from the training risk, and thus creates the possibility of learning transportable recommendation.
Further, the excessive risk bound suggests that the transportation constraint does not cost the convergence rate comparing with the standard empirical risk minimization guarantee (Mohri et al., 2018).
4. T OWARDS TRANSPORTATION - REGULARIZED RISK MINIMIZATION
To render the proposed objective computationally feasible, we apply a strong duality argument to show that there exists a regularized counterpart for (2), where we defer the proof to Appendix A.6.
Claim 1.
Suppose that ℓ ◦ F is convex.
It holds for any f ∈ F that:
[Equation]
The above duality result suggests we alternatively consider the regularized learning objective:
[Equation]
where the relaxation on max λ ≥ 0 has been made.
Here, λ is the tuning parameter that regularizes the domain transportation between the reweighted training domain and the deployment domain.
To give some insight for the transportation-regularized objective, we use learning from bandit feedback data as a test bed and examine how the optimal solution of (3) compares to the solution of the standard counterfactual risk minimization with complete overlapping (Swaminathan and Joachims, 2015a).
Let the logging policy by given by π .
It holds in the bandit off-policy learning setting that: P f ( u, i ) ∝ f ( u,i ) π ( u,i ) , where f ( u, i ) ∈ (0 , 1) now serves as the policy we try to optimize.
The counterfactual risk is given by: E P f R ( f ) :
= (cid:80) ( u,i ) ∈D f
( u,i ) π ( u,i ) ℓ ( f ( u, i ) , y ui ) , and we prove that the solution of (3) is consistent with that of the counterfactual risk minimization.
Theorem 3.
Let f ∗ = arg min f ∈F E P
f
R
( f ) be the solution of the counterfactual risk minimization, and ˆ f, ˆ w be the solution of the proposed objective in (3).
For suitably large λ s.t. λ ≥ O ( L ) ≥ 1 , it holds with probability at least 1 − δ that:
[Equation]
where M G , R w ( f ) 2 are defined as in Theorem 1, and π is the logging policy.
Intuitively, since both f and π are probability distributions with the same support in the idea bandit feedback setting, a suitably large λ will drive P w ultimately toward P f and lead to the consistency result.
We defer the proof of Theorem 3 to Appendix A.7.
This example reveals the effectiveness of the proposed transportation-regularized risk minimization, which we show is a strict extension of the counterfactual risk minimization.
For IR systems, the transportation-regularized risk minimization applies to a much broader setting including deterministic recommendation.
Nonetheless, directly solving (3) is still challenging due to the complications in computing the Wasserstein distance.
Inspired by how Wasserstein GAN transforms its objective to a two-player adversarial game (Gulrajani et al., 2017), we can also convert our objective to a minimax optimization problem discussed below.
4. M INIMAX O PTIMIZATION
Input: ˜ D ( t ) : a batch of training sample of size m at step t .
η : the learning rate for ascent step.
γ : shrinking parameter for the learning rate of descent
λ the regularization parameter. for each step t = 0 , 1 , . . .
do
Compute reco ( u, i ; f θ 1 ) for ( u, i ) ∈ ˜ D ( t ) (see Section 5 for the computation detail); θ ( [Equation] +1)← [Equation]
; θ ( [Equation] + [Equation]
)← [Equation] ( t ) 3 + η ∇ θ 3 L λ (cid:0) [ θ ( t ) 1 , θ ( t ) 2 ] , θ ( t ) 3 (cid:1) .
end
The key to our implementation is again perceiving Wasserstein distance as a binary classification problem with Lipschitz-bounded classifier as we discussed in Section 4.1.
Towards this end, we denote by g ∈ G any suitable classifier for computing d W according to (1).
Without loss of generality, we let w , f and g be parameterized by θ 1 , θ
2 and θ 3 .
It is straightforward to verify that the regularized objective in (3) transforms exactly into: min [ θ 1 , θ 2 ] max θ 3 L λ ([ θ 1 , θ 2 ] , θ 3 ) , where L λ given by:
[Equation]
In view of (4) as a two-player game, the first player optimizes f and w to minimize both the training risk and domain transportation, while the second player tries to increase the gap between the two domains.
The global equilibrium is reached when neither player can change the objective without knowing each other’s strategy, which means f and w cannot simultaneously improve the empirical risk and the domain transportation between P w and P f .
It suggests we have found the recommendation hypothesis that best transports the learnt patterns to its intervention domain.
We propose using the two-time-scale gradient descent-ascent algorithm (GDA) to solve the minimax optimization.
GDA is very efficient for solving objectives like (4) with both empirical and theoretical evidence (Lin et al., 2020), and it has been widely applied to optimize generative adversarial networks.
Since our solution will be deployed to real-world systems, we pay more attention to the convergence and stability of GDA.
In appendix A.8, we rigorously prove that for the bounded Lipschitz functions we consider, GDA will converge to the global Nash equilibrium with a proper learning-rate schedule.
Further, it is unlikely to bouncing around near the optimum.
Now that we have the performance guarantee, we apply the batch-wise GDA described in Algorithm 1 to solve our objective.
E XPERIMENT AND R ESULT
We present in this section the experiment settings, implementations, empirical results, as well as the analysis.
The complete experimental results and the real-world online testing performance are deferred to Appendix C. All the implementation codes are provided in the supplement material.
Computing the reco ( u, i ; f ) function .
For the deterministic top-K recommendation, reco ( u, i ; f ) requires computing and sorting the scores of { f ( u, i 1 ) , . .
f ( u, i |I| ) } for the user u .
This is expensive during training, so we refer to the widespread practice of downsampling.
In particular, for each ( u ′ , i ′ ) , we randomly sample m < |I| irrelevant items and see if i ′ is among the top-K (Krichene and Rendle, 2020).
While the sampled outcome may have certain bias, it significantly speeds up the training process while maintaining a decent accuracy under moderate m , e.g. m = 100 .
Benchmark datasets.
We refer to the MovieLens-1M , LastFM , and GoodReads datasets that are extensively employed by the IR literature.
We treat them as implicit feedback data, and the detailed description and processing are deferred to Appendix C.1.
Simulation studies. .
For implicit feedback, the response (click) is generated from the exposure and relevance status via: P ( Y ui = 1) =
P ( O ui = 1) · P ( R ui = 1) , where O ui indicates the exposure and
R ui is the relevance.
The relevance score is critical for unbiased offline evaluation, but it is unknown for real-world datasets.
We thus conduct simulation so we know what the ground truth relevance is.
To make sure the distribution of the simulated data conforms to the real-world feedback data, we first extract the exposure and relevance signals from public datasets (e.g. the three datasets described above) via models such as matrix factorization, and then generate the new feedback data by adding noise to the extracted relevance signal.
The detailed mechanisms are provided in Appendix C.1.1.
[Table]
Models and baselines methods.
The functions of f , w and g in the minimax objective (4) can be any recommendation algorithm that suits the context, such as matrix-factorization collaborative filtering ( MCF ), neural collaborative filtering ( NCF )
(He et al., 2017), and sequential recommendation model with the prevalent attention mechanism ( Attn ) as in Kang and McAuley (2018).
We index our domain domain transportation approach by such as DT-X , with X stands for the models of f , w and g .
We will experiment with a broad range of combinations for f , w and g as we present in Appendix C.3.
As for the baseline methods, other than the standard popularity-based recommendation ( Pop ) and MCF, we select particularly the approaches that adapts causal inference or missing data techniques, including the IPW-debaised MF method ( IPW-MF ) from Saito et al.
(2020), the user-exposure aware MF ( ExpoMF ) from Liang et al.
(2016), and the adversarial-counterfactual MF method ( AC-MF ) from Xu et al.
The model configurations and training details are deferred to Appendix C.2.
Training & evaluation .
We use the two-timescale GDA as described in Algorithm 1 with a preselected shrinkage parameter, and we will conduct its sensitivity analysis in Appendix C.4.
The configurations for both our model and the baselines are described in Appendix C.2.
We adopt the convention where the second-to-last interaction is used for validation, and the last interaction is used for testing.
For evaluation, in addition to the regular Hit@10 and NDCG@10 as in Rendle et al.
(2020), we further consider the relevance score of the top-10 recommendation, i.e. (cid:80) top-10 R ui , since it also reveals the quality of the recommendation and more closely resembles the deployment performance of E P f R ( f ) .
We have access to R ui during simulation, and on real-data analysis, we settle to the Y ui in testing data as an approximation.
The evaluation results are provided in Table 1.
Result analysis .
From Table 1, we first observe that the proposed methods achieves the best performances in both the benchmark and simulated datasets on all metrics.
In particular, the DT-X method significantly improves the performance of the original model X .
Also, our solution outperforms all the baselines, including those adapted from causal inference or missing data methods.
The rest simulation studies are provided in Appendix C.3, and their results admit similar patterns.
Finally, we conduct ablation studies on the transportation regularizer and the sensitivity analysis on λ (see
Figure 2c).
It is clear that without the transportation regularization, the performances degenerate significantly in both the real data and simulation studies.
Also, we find that the performance variation of the proposed approach is relatively steady under different λ .
Analysis on w .
We specifically analyze the learnt weights since they play a critical role in our approach.
On both MovieLens and GoodReads (Figure 2a), we find that: 1).
the learnt weights are larger on the positive training instances; 2).
for the ( u, i ) that are among P f (the recommended), the weights w ui are higher and relates positively with f ( u, i ) , while there is no such pattern for the non-recommended.
They both suggest the weights are indeed learnt to balance the domains of the positive feedback data and the instances recommended under f .
In Figure 2b, we observe from the Movielens simulation study that the learnt weights correlate positively to the exposure on the negative training data, and are distributed more evenly on the positive data.
It means that w ui learns to emphasize the instances that were more likely to be exposed but not clicked, which usually have strong negative signals, and the positive instances.
It is valuable for training under implicit feedback because the algorithm learns to locate the strong negative instances.
The remaining weight analysis, ablation studies, and sensitivty analysis are provided in Appendix C.3.
Real-world IR system.
We deploy our solution to a real-world IR system held by a major ecommerce platform in the U.S.
We deploy our transportation-regularized solution, and the online A/B/C testing against the standard and IPW-based solutions for an item recommendation task.
Due to the space limitation, we defer the details and analysis to Appendix C.6.
C ONCLUSION AND D ISCUSSION
In this paper, we study from the critical intervention perspective of recommendation and summarize a domain transportation view supported by the realistic working mechanism of IR systems.
We propose a principled transportation-constraint risk minimization for optimizing recommendation, and illustrate by both theories and experiments how we find it more favorable than directly importing the causal inference and missing data solutions.
However, we point out that our work does not aim to replace those solutions since we focus exclusively on optimizing recommendation.
For instance, we do not readily incorporate the known causal mechanisms (which might be available for certain applications) to answer questions that are relevant to the decision making in IR.
It will be interesting to explore how the presumed or discovered causal structures can interact with domain transportation and improve our solution.
Towards this end, our transportation-constraint risk minimization can be viewed as a modular toolkit that allows researchers to apply further insights to solve more complex IR problems.
Finally, we hope our work raises the awareness of developing IR-oriented solutions with a proper understanding of the system’s mechanism.
The novel perspectives, solutions, and theoretical results in our paper may also open the door to future research.
A A PPENDIX ON B ACKGROUND AND M AIN T HEORETICAL R ESULTS
We provide the supplementary background introduction, proofs for the claim, proposition and theorem, as well as the detailed experiment settings and complete numerical results in this part of the paper.
A. R ECOMMENDER WITH EXPLORATION POLICY , AND LEARNING FROM BANDIT FEEDBACK DATA
Using multi-armed bandits for exploration has attracted increasing attention from the recommendation literature in recent years.
The major advantage is that the interactive process between users and recommender is explicitly characterized in an online fashion.
Exploration also helps alleviate the exposure bias, popularity bias, position bias, etc, via more efficient exploitation of the feedback data.
A critical difference between the bandit recommender and the deterministic recommender we referred to in our paper is that an item is exposed to the user with randomness, and the exploration policy controls the randomness in the exposure.
Mathematically, we continue with X as the context (feature) space, I = { 1 , . . .
, k } be the action (item) space.
The exploration-exploitation strategy can be viewed as a sequential game between the user and recommender.
For notation simplicity, we introduce the stationary bandit setting where the exploration policy does not depend on the history.
In each round, an item is sampled according to the exploration policy π ( u, i ) :
=
π ( i | x u ) , which is a mapping between the item space and the probability space, and expose it to the user.
Then the user, whose feature is x u , reveal the regret (or reward) action i ∈ I , which corresponds to the Y u,i in our setting.
In the context of delivering recommendation, clearly we have: P ( reco ( u, i ; f ) = 1) =
π ( u, i ) during exploration, and in the context of learning from the logged feedback data, we have P ( O ui = 1) =
π ( u, i ) since it characterizes the probability of exposure.
Similar to the deterministic recommender, the challenge in bandit offline learning is largely due to its partial-observation nature as we do not observe the complete reward for all items.
However, unlike the deterministic recommender, the random exposure mechanism allows efficient estimation of the counterfactual regret, i.e. what the regret would have become if an alternative exploration policy was used.
The IPW estimator fits particularly well to this purpose, since it provides the unbiased estimation of the counterfactual regret:
[Equation]
Swaminathan and Joachims (2015a) refers to the IPW-reweighted risk, which essentially replace the regret Y ui by the corresponding risk under f , as the counterfactual risk for learning from the logged bandit feedback data.
It exactly corresponds to setting the base measure P with P IPW , so it holds:
[Equation]
A. P ROOF AND B ACKGROUND FOR THE IW AND DA BOUNDS IN P ROPOSITION
We first prove the upper bound for importance weighting under weak overlap .
Proof.
Our proof leverages the classical ”double sampling” technique from Anthony and Bartlett (2009).
We use ⃗ z =
[ z 1 , . . .
, z n ] to denote the observed samples, and ⃗ z ′ =
[ z ′ 1 , . . .
, z ′ n ] to denote an i.i.d copy of ⃗ z .
We first define by:
[Equation]
and
[Equation]
Given [Equation] let [Equation]
it holds:
[Equation]
where the last line follows from Lemma A.2.
Next, we define C ( ϵ, ℓ ◦ F , ℓ 1 ( P n,w )) be the ϵ -cover of ℓ ◦ F with the empirical ℓ 1 norm under P n,w such that for any f ∈ ℓ ◦ F , there exists ˜ f in
C
[Equation] [Equation] z
[Equation]
≤ ϵ , for ( z 1 , . . .
, z n ) sampled i.i.d from P .
It then holds:
[Equation]
t ) ≥ E Q R ( f )] ≤ 2 E ⃗ z ,⃗ z ′ sup f ∈F I [ UB 1 ( f,⃗ z ′ , t ) ≥ UB 2 ( f,⃗ z , t )
+ ϵ ] ( b ) ≤ 2 P σ ( ⃗ z ,⃗ z ′ ) (cid:0) ∃ ˜ f ∈ C ( ϵ, ℓ ◦ F , ℓ 1 ( P n,w )) : UB 1 ( f, σ ( ⃗ z ,⃗ z ′ ) , t ) ≥ UB 2 ( f, σ ( ⃗ z ,⃗ z ′ ) , t ) (cid:1) ≤ 8 N ( ϵ, ℓ ◦ F , ℓ 1 ( P n,w ))
· e − t , where (a) follows from the fact that E ⃗ z ′
I [ UB 1 ( f,⃗ z ′ , t ) ≥ E Q R
( f )] ≥ 12 as suggested by Lemma
A.2, and in step (b) we let σ ( ⃗ z ,⃗ z ′ )
i takes the value of z i , z ′ i with equal probability, and the inequality follows from the definition of the ϵ cover.
Notice that N ( ϵ, ℓ ◦ F , ℓ 1 ( P n,w )) ≤ N ( ϵ/M, ℓ ◦ F , ℓ n ∞ ) .
We take ϵ = 1 n , which solves for t =
c log 1 δ +log N ( ϵ/M, ℓ ◦F , ℓ n ∞ ) for some constant c and leads to the desired result after arranging terms.
The corresponding lower bound follows directly from Lemma A.4, where we make a mild assumption that there exists a hypothesis f ∈ F such that E P n R ( f )
= 1 , so it holds that E P n,w R ( f ) =
d ( P ∥ Q ) − 1 suggested by Lemma 1 of Cortes et al.
We then show the PAC-Bayesian result that stated for the partial-overlap case.
Our proof relies on the existing results in the PAC-Bayesian literature as in Germain et al. (2015).
Proof.
We use ℓ to denote the 0 − 1 loss for brevity.
The PAC-Bayesian approach concerns with the
Gibbs risk of E f ∼ τ E ( u,i ) ∼ Q ℓ ( f ( u, i ) , y ui ) , where τ is some posterior distribution on the hypothesis space F .
Clearly, when there are mismatch in the supports of S P and S Q , the Gibbs risk for the mismatched part is given by E f ∼ τ E ( u,i ) ∈S Q / S P R ( f ) .
We now focus on the overlapped part.
Using the classical decomposition result from Germain et al.
(2015), which states that:
[Equation]
where d D ( τ ) =
E f,f ′ ∼ τ E Q ℓ ( f ( u, i ) , f ′ ( u, i )) is a discrepancy term, and e Q ( τ ) = 2 E f,f ′ ∼ τ E Q ℓ ( f ( u, i ) , y ui ) ℓ ( f ′ ( u, i ) , y ui ) is the Gibbs risk for the joint hypothesis.
Therefore, we have:
[Equation]
where D ( τ ) essentially characterizes the disagreement of Q and P w in terms of the posterior τ on F .
Therefore, for any posterior distribution τ , we have: E f ∼ τ E Q R ( f ) ≤ E f ∼ τ E P w R ( f ) + D ( τ ) , so the next step is to bound the Gibbs risk of E f ∼ τ E P w R ( f ) .
Our strategy follows that of Germain et al.
, that we employ the convex function ∆ : [0 , M ] × [0 , M ] → R .
It then holds that:
[Equation]
where the last line comes from the Markov inequality, and w ( z k ) is a shorthand for w ( u, i ) for
[Equation]
We then focus on bounding:
[Equation]
Since ℓ f is the 0 − 1 loss, the term inside ∆ (setting aside w ( z i ) ) are essentially binomial variable of n trials with a success rate of E P R ( f ) .
Since ∆ is convex in both arguments and w ( u, i ) is given by Q ( u, i ) /P
( u, i ) , it is easy to verify that:
[Equation]
where d ( P, Q ) =
(cid:82) S Q ( dP/dQ ) dP as we mentioned in the statement.
It is then obvious that:
[Equation]
where we use r ∈ [0 , 1] to replace the role of E P R ( f ) , since the prior π is arbitrary, and Bin ( k ; n ; r ) is probability of having k success in n rounds of Bernoulli trail where the success rate is r .
It is shown by Germain et al.
(2009) that the RHS is O (1) by choose a particular ∆ , so we obtain the desired result by putting together the above arguments.
Now we briefly introduce the skip-gram negative sampling algorithm and prove its associated rersults in Proposition 1.
It have been shown by the recent work of Xu et al.
(2021a) that the user/item embeddings optimized via the skip-gram negative sampling (SGNS) algorithm are the sufficient dimension reduction of a specific co-occurrence statistics.
In particular, the SGNS objective function is given by:
[Equation]
where Φ( u ) , Φ( i ) are the embeddings for the user u and item i , k is the number of negative samples, and Neg ( I ) is the uniform negative-sampling distribution on the item set.
The risk objective is thus give by: E P R (Φ) =
(cid:80) ( u,i ) ∈D ℓ u,i .
The SGNS algorithm has been widely applied to train user and item embeddings for various recommendation tasks.
The sufficient dimension reduction (SDR), on the other hand, is inspired by the sufficient statistics of exponential family.
Broadly speaking, when predicting Y with X , if all the information in X about Y can be compressed into a dimension reduction f
( X ) , i.e. Y ⊥ X | f
( X ) or p ( Y | X ) and p (cid:0) Y | f ( X ) (cid:1) are the same, then f ( X ) is a sufficient dimension reduction (SDR).
According to Globerson and Tishby (2003), finding the SDR f ( X ) is equivalent to solving a optimization problem:
[Equation]
where d KL is the Kullback-Leibler divergence.
Proof.
Let R ui :=
N ui N u
N i be the relatedness measure defined in Xu et al.
(2021a), where N ui is the co-count of ( u, i ) , and N u ,
N i are the individual counts of u and i .
The co-occurrence statistics is given by C ui :
=
O ui | R ui , which is exposure O ui condition on the relatedness measure.
It is shown in the Claim 1 of Xu et al.
(2021a)
that:
[Equation]
so Φ gives the SDR of C ui in R d , according to the definition of (A.2).
Note that the terms in the
RHS of (A.3) are all binary KL divergence, and it easy to verify that binary KL divergence satisfies: d KL ( a ∥ c ) + d KL ( c ∥ b ) ≥ d KL ( a ∥ b )
+ ( c − a ) log c (1 − b ) (1 − c ) b .
Recall that C ui ( P ) and C ui
( Q ) are the co-occurrence statistics under P and Q , respectively.
Therefore, if we let Φ ( P ) and Φ ( Q ) be the solution to (A.3) under P and Q , by assuming S P = S Q , it holds that:
[Equation]
E P R (Φ) + E Q R (Φ) (cid:1) ≥ inf Φ E P R (Φ) + inf Φ E P R (Φ) ≥ |S P | (cid:88) ( u,i ) ∈S P d KL (cid:0) p
( O ui | Φ ( P ) ( u ) , Φ ( P ) ( i ))
(cid:13)(cid:13) C ui ( P )) (cid:1) + d KL (cid:0) p
( O ui | Φ ( Q ) ( u ) , Φ ( Q ) ( i ))
(cid:13)(cid:13) C ui ( Q )) =
|S P | (cid:88) ( u,i ) ∈S P d KL ( C ui ( P ) ∥
C ui ( Q ))
+ C, and C = inf Φ (cid:0) E P R (Φ) + E Q R (Φ) (cid:1) when P = Q a.s.
A. B ACKGROUND ON THE W ASSERSTEIN D ISTANCE
Perhaps the two most common discrepancy measures on probabilities are ϕ -divergence and integral probability metric (IPM).
Obviously, ϕ -divergence does not satisfy our need of characterizing domain transportation because it requires S P ⊆ S Q (weak overlap) to hold in practice.
IPM, on the other hand, provides the flexibility and is defined as
[Equation]
where G is a class of real-valued bounded measurable function.
In this direction, we find the Wasserstein distance from the optimal transportation (OT) theory Rachev (1985) a good fit to our goal.
Although the maximum mean discrepancy distance is also valid, its computation and related theoretical results rely on the specification of a reproducing kernel Hilbert space, whose usage is less common in the IR literature.
The origin of the Wasserstein distance is to give the optimal cost of transporting mass from one marginal distribution to another:
[Equation]
where Π( P, Q ) is the set of all joint probability measure whose marginals are P and Q , and c ( · , · ) is the cost function.
Clearly, d w,c ( P, Q ) relies on the geometry of P and Q if the cost function is wellchosen.
For recommender system, z is often given by ( x u , x i ) or (Φ( u ) , Φ( i )) , so it is natural to use the ∥ · ∥ 2 distance: c ( z , z ′ ) =
∥
z − z ′
∥ 2 .
We denote by d W for this particular cost function, which is also referred to as the 1 -Wasserstein distance.
The classical Kantorovich-Rubinstein theorem
Dudley (2018) connects the d W to IPM by proving the duality result of:
[Equation]
In the above formulation, the constant of 1 under sup is less critical since we can always rescale g .
The class of Lipschitz-bounded function covers a wide selection of models including neural networks as discussed in Fazlyab et al.
A. P ROOF FOR T HEOREM
We first state a proposition for the following proofs.
Proposition A.1.
Suppose ℓ f is semi-continuous for all f ∈ F , P is any distribution supported on
Z , and c ( z , z ′ ) =
∥
z − z ′
∥ 2 .
It holds:
[Equation]
Proof.
It is shown by Gao and Kleywegt (2016) that:
[Equation]
Following the above result, we immediately have:
[Equation]
Now we prove the result in Theorem
[Equation]
Proof.
We first define H 1 = { w ∈ H : d W ( P w , P f ) ≤ ρ }
[Equation]
and H 1 = { w ∈ H : d W ( P w , P f ) ≤ ρ +
[Equation]
} , where c w is the constant stated in Lemma A.5.
We then omit the w ∈ H in the following proofs for brevity.
By Lemma A.5, for any δ > 0 , it holds with probability at least 1 − δ that H 1 ⊆ H 2 , which suggests that for any given f ∈ F :
[Equation]
with probability at least 1 − δ , where B =
[Equation]
Now we define ˜ w = arg min w : D n ( w,f ) ≤ δ + B E P w R ( f ) .
Then using the result in Lemma A.1, for any given f ∈ F , it holds with probability at least 1 − δ 4 that:
[Equation]
where the last line follows from the definition of ˜ w .
Using Proposition A.1, we have:
[Equation]
where the second line uses the result in Lemma A.6 which bounds the λ ∗ that achieves the supremum with the Lipscitize constant L .
Combining the above results, it holds with probability at least 1 − δthat:
[Equation]
The upper bound can be shown in the same manner, by using [Equation] to replace ˜ w .
Combining the upper and lower bounds, we obtain the result stated in Theorem 1.
A. P ROOF FOR T HEOREM
Suppose ℓ is some bounded loss function, then M F stated in Theorem 2 must be finite for λ ∈ [0 , Lρ ] , otherwise it can be shown from Proposition A.1 and Lemma A.6 that: [Equation] · c [Equation]
z
[Equation]
˜ z )]
[Equation]
is unbounded for finite ρ , which contradicts that ℓ is bounded.
We first prove the generalization error bound.
Proof.
We use the shorthand ϕ f,λ ( z ) :
=
E z ∼ P f [inf ˜ z ℓ f (˜ z ) + λc (˜ z , z )] .
Since we do not make any parametric assumption on the weighting function w here (as the consequence of using w ∈ H has been stated in Theorem 1), we have:
[Equation]
Using the standard symmetrization argument, we can upper-bound E ψ λ =n [Equation] z
[Equation]
z i ) [Equation] with the empirical Rademarcher complexity:
[Equation]
where the expectation on LHS is taken with respect to the random variables of z 1 , . .
. , z n .
Clearly, ψ ( λ ) is bounded by M F that we illustrated in the beginning, so from McDiarmid’s inequality we have:
[Equation]
Also, note that (cid:113) 1 n ϵ
i
ϕ
f,λ ( z i ) is zero-mean and 1-subgaussian with respect to the ∥ · ∥ ∞ norm on F , so for any f 1 , f 2 ∈ F , it holds that:
[Equation]
Using the entropy integral bound on the empirical Rademarcher complexity term, we have:
[Equation]
since ℓ is Lipscitz (and we absorb the Lipscitz constant into c ).
Combining the above results, for any given λ ≥ 0 , it holds with probability at least 1 − δ that:
[Equation]
Then we apply the same union-bound argument from Theorem
Koltchinskii et al.
(2002) to extend the above result to any λ > 0 .
In particular, we pick the sequence of λ k = k and t k = t + √ log k , k = 1 , 2 , . . . .
By simple calculations and the sum of infinite series arguments, we reach the conclusino that for any f ∈ F , it holds with probability at least 1 − δ that:
[Equation]
Now we prove the excessive risk bound
[Equation]
Proof.
We start by defining:
[Equation]
E P n
[Equation]
w R
[Equation]
f ) .
Using the definition of ˆ f
[Equation]
we have:
[Equation]
Let ˆ λ = arg
max λ ≥
[Equation]
−
[Equation]
+ E z ∼ P ˆ f [inf ˜ z ∈Z ℓ ˆ f
(˜ z ) + λ · c ( z , ˜ z )] (cid:9) , and from Lemma A.6 we know that ˆ λ ≤ L .
It holds that:
[Equation]
Again applying the standard symmetrization argument and McDiarmid’s inequality to the last line (similar to what we have shown in proving the generalization bound), the following bound holds with probability at least 1 − δ/ 2 :
[Equation]
Similarly
[Equation]
we let
[Equation]
[Equation]
, and using the same arguments, we also have the following bound that holds with probability at least 1 − δ/ 2 :
[Equation]
Combining the above two bounds [Equation] we obtain the desired result.
A. P ROOF FOR C LAIM
Proof.
Given f ∈ F
[Equation]
the mapping of P w to d W ( P w [Equation] P f ) is convex in the space of probability measures.
Since d W ( P w [Equation] P f ) = 0 can be achieved by taking P w = P f a [Equation]
e
[Equation]
using the generalized Lagrange duality result in Lemma A.
[Equation]
it holds that:
[Equation]
Notice that for any π ∈ Π( P w , P f ) , where Π( P w , P f ) is the set of all joint distributions whose marginals are given by P w and P f , we have: (cid:82) Z ℓ f ( z ) dP w = (cid:82) Z (cid:82) Z ℓ f ( z 1 ) dπ ( z 1 , z 2 ) .
Therefore, we have:
[Equation]
using the definition of the Wasserstein distance introduced in Appendix A.3.
As a consequence [Equation] it holds that:
[Equation]
We first point out that:
[Equation]
and in the following proof we show that the other direction also holds, so the above inequality becomes equality.
Also, if we consider M as the space of all the regular conditional probabilities from z 2 to z 1 , it is evident that:
[Equation]
Furthermore
[Equation]
using Lemma A. [Equation] if we consider X as the space of all measurable mappings from Z to Z [Equation]
then it holds:
[Equation]
Consequently
[Equation]
if we let ˜ x ∈
X be the mapping that achieves ϵ -close to the above infimum (which requires certain continuity condition for ℓ f ) and M ( ·| z 2 ) be defined by ˜ x
( z 2 ) :
[Equation]
Since ϵ > 0 is arbitrary, combining (A.7), (A.8) and (A.9), we have:
[Equation]
z
[Equation]
) , and using the result in Proposition A.1, we obtain the final that for any f ∈ F :
[Equation]
A. P ROOF FOR T HEOREM
Proof.
We first let f ∗ =
arg min f ∈F E
P f
R
( f ) , and define w ∗ = f ∗ (suppose F ⊂ H ) such that
P n,w ∗ ( u, i ) =
P n,f ∗ ( u, i ) and P w ∗ ( u, i ) =
P f ∗ ( u, i ) .
Since F is the class of valid exploration policies in the bandit setting, we have f ( u, i ) ∈ (0 , 1) for all f ∈ F .
Note that by the definition of f ∗ and w ∗ :
[Equation]
Combining the results in Lemma A.1 and Lemma A.5, it holds with probability at least 1 − δ/ 2 for any f ∈ F and w ∈ H that:
[Equation]
Hence, the following inequality holds with probability at least 1 − δ :
[Equation]
On the other hand, with λ ≳
L ≥ 1 where L is the Lipschitz constant for ℓ ◦F , for any given f ∈ F and w ∈ H , it holds that:
[Equation]
where in the first line we use the background of learning from bandit feedback data where π is the logging policy (e.g. π ( z )
=
π ( u, i ) for z = ( x u , x i , y ui ) ); and the last line we use the definition of
Wasserstein distance under the Kantorovich-Rubinstein theorem (Appendix A.3).
Again, by combing Lemma A.1 and Lemma A.5, we can further bound the last line with probability at least 1 − δ/with:
[Equation]
Now we let ˜ f, ˜ w = arg min f ∈F ,w ∈H E P n,w R ( f ) + λ ·
d W ( P n,w , P n,f ) for λ ≳
L ≥ 1 .
Then it holds that:
[Equation]
where the slack terms are stated above.
Combining (A.10) and (A.11), we obtain the desired result.
A. A NALYSIS ON THE GDA ALGORITHM .
By using bounded Lipschitz functions for w θ 1 , f θ 2 , we will show below that the gradient of L λ is L − Lipschitz.
It is indicated by Daskalakis and Panageas (2018) that GDA with η ≤ 1 /L avoids unstable critical points for almost all initialization, which means that (cid:0) θ ( t 1 , θ ( t ) 2 , θ ( t ) 3 (cid:1) is unlikely to cycle as t grows.
Furthermore, any stationary point yielded by GDA with large γ is basically a local minimax point except degenerated case (Jin et al., 2020), and thus the convergence guarantee of Algorithm 1.
Here, a solution ( θ ∗ 1 , θ ∗ 2 , θ ∗ 3 ) is local minimax if there exits δ 0 > 0 and a function h satisfying lim δ
→ 0 h ( δ ) → 0 such that ∀ δ ∈ (0 , δ 0 ] , and any ( θ 1 , θ 2 , θ 3 ) satisfying (cid:13)(cid:13)
[ θ 1 , θ 2 ] − [ θ ∗ 1 , θ ∗ 2 ] (cid:13)(cid:13) 2 ≤ δ and ∥ θ 3 − θ ∗ 3 ∥ 2 ≤ h ( δ ) , it holds that: [Equation]
[Equation]
In other words, by using the two-time-scale GDA algorithm with a small η and a large γ , our resulting solution has a similar role as a local minimum/maximum in a minimization/maximization problem — the global Nash equilibrium, if any, must be one of the solutions yielded by the GDA in Algorithm 1.
Proof.
Here, we show that if w θ 1 , f θ 2 are bounded Lipschitz functions and g θ 3 is a Lipschitz function, the gradient of the minimax objective is also Lipschitz.
Recall the objective function in (6), i [Equation] e.,
[Equation]
It is straightforward to show that:
[Equation]
We impose a few assumptions on f θ 1 , w ui ( θ 2 ) and g ui ( θ 3 ) .
First, we assume they are bounded and their gradients are Lipschitz functions, which are standard assumptions for neural or factorization models.
Then, with a careful selection of the loss function ℓ (e.g., logistic loss, exp-loss, probit loss) and the reco function (e.g., top-K), we can ensure ℓ ui ( f θ 1 ) and reco ( u, i ; f θ 1 ) to have Lipschitz gradients.
Finally, by employing clipping tricks on the gradients to ensure they will not explode, which is also a common practice, we have universally bounded and Lipschitz functions that include: f
[Equation]
Therefore, the products and summations of these function is a L -Lipschitz function for some positive constant L .
B L EMMAS AND P ROOFS
In this part of the appendix, we state the important lemmas that we used in our proofs.
Lemma A.1 (Empirical Bernstein bound) .
Suppose that ℓ f : Z → [0 , 1] without loss of generality.
With w ( u, i ) =
Q ( u,i ) P ( u,i ) and z 1 , . .
z n sampled i.i.d from P , for any δ > 0 , it holds with probability at least 1 − δ that:
[Equation]
where R w ( f ) 2 = 1 n (cid:80) w 2 i ℓ f ( z i ) 2 .
The result is modified from Corollary 5 of Maurer and Pontil (2009).
Lemma A.2.
Let P and Q be the training and target distribution supported on S P , S Q ⊆ D , and w ( u, i ) =
Q ( u,i ) P ( u,i ) , for ( u, i
) ∈ S P ∩ S Q .
D n consists of training instances sampled i.i.d from P .
Given a single hypothesis f ∈ F , suppose w ui ∈ (0 , M ) , for any δ > 0 , it holds with probability at least 1 − δ that:
[Equation]
where d ( P ∥ Q ) =
(cid:82) S Q ( dP/dQ ) dP .
The above result for importance weighting of a single hypothesis is stated in the Theorem 1 of Cortes et al.
Lemma A.3.
Let f be a real-valued convex functional defined on a convex subset Ω of a vector space
X , and let G be a convex mapping of X into a normed space Z .
Suppose there exists x 1 such that
G ( x 1 ) < ρ and that inf { f ( x ) : G ( x ) ≤ ρ, x ∈ Ω } is finite, then it holds that:
[Equation]
The above Lagrange duality result is stated in Theorem 8.1 of Luenberger (1997).
Lemma A.4 (Lower bound based on maximum variance) .
Suppose f ∈ F takes values in [0 , 1] without loss of generality, and the variance of f is denoted by σ 2 ( f ) .
Let σ ( F ) =
sup f ∈F σ ( f ) , and assume 1 n ≤ σ 2 ( g ) <
+ ∞ .
It holds that:
[Equation]
The result is stated in Theorem 9 of Cortes et al.
Lemma A.5 (Finite sample bound for 1-Wasserstein distance) .
Suppose P and Q are supported on (perhaps non-overlapping subsets of) Z where Z ∈ R d and is bounded.
Let M G :=
sup g : ∥
g ∥
L ≤ 1 sup z ∈Z g ( z ) .
For all δ > 0 , it holds with probability at least 1 − δ that:
[Equation]
Proof.
The first part of the proof follows from the result on integral probability metric stated in
Theorem 11 of Sriperumbudur et al.
(2009), that for G = { g : ∥ g ∥
L ≤ 1 } , we have:
[Equation]
with probability at least 1 − δ , and R n ( G ) is the empirical Rademarcher complexity of G .
Using the entropy integral bound on the empirical Rademarcher complexity, we have:
[Equation]
Note that Z ∈ R d implies: N ( u, Z , ∥ · ∥ ∞ )
=
O ( u − d ) .
Using the fact that log( x + 1) ≤ x + 1 , we have: log N [Equation] u, G , ℓ 2 ( P n )) ≤ log N
[Equation]
u, G , ∥ · ∥ ∞
[Equation] ≤ c 1 u − [Equation] d +1) + c 2 u − d .
Since Z is bounded, the entropy integral bound becomes:
[Equation]
which directly leads to the desired bound where c w is the corresponding constant.
Lemma A.6.
Let c ( z , ˜ z ) :
=
∥
z − ˜ z
∥ 2 .
Define:
[Equation]
If all f ∈ F is L -Lipschitz
[Equation]
i
[Equation]
e [Equation]
sup z [Equation] z ′ | f [Equation]
z
[Equation]
|
[Equation] z [Equation] ≤ L , then it holds: λ ∗ ≤ L .
Proof.
Using the Lipschitz property
[Equation]
we have:
[Equation]
≥ − Lρ ( by choosing λ = L ) , which gives λ ∗ ≤ L .
Lemma A.7 (Change of measure inequality) .
For any P and Q on X , and for any measurable function f : X → R , it holds that:
[Equation]
This is a standard result that has been included in many textbooks for probability.
Lemma A.8 (Interchange of integration and minimization) .
Let X be a space of measurable functions from Z to R n with respect to u , a σ -finite measure on the underlying space.
Let f : Z × R n → R be a normal integrand, then it holds:
[Equation]
The result is simplied from Theorem 14.60 of Rockafellar and Wets (2009).
C C OMPLETE E XPERIMENT S ETTINGS , N UMERICAL R ESULTS , AND R EAL - WORLD D EPLOYMENT AND T ESTING A NALYSIS
We provide the complete experiment settings (real-data and simulation experiments), including the data description and preprocessing, experiment setups, model configurations and training details.
The complete numerical results and analysis are also presented in this part of the appendix.
Finally, we discuss the real-world online experiments by deploying our solution to a e-commerce platform.
The implementation code for the real-data and simulation experiments are provided in the supplement material.
All the reported results are averaged over ten independent runs.
C. D ATASET DESCRIPTION , PREPROCESSING AND EXPERIMENT SETUP
The detailed descriptions for the three real-world benchmark datasets are given as below.
• Movielens-1M 4 .
The dataset provides the users’ ratings for movies that they watched, which consists of around 1 million ratings collected from 60,40 users on 3,952 movies.
The movie ratings are ranged from 1 to 5, where a higher rating indicates more positive feedback.
Hence, the original Movielens-1M dataset possesses explicit feedback.
The traditional way of converting the dataset to implicit feedback is to set the rating cut-off at 3, as suggested by He et al.
We use the explicit feedback for semi-synthetic experiments, and use the converted explicit feedback for the real-data experiments.
• LastFM 5 .
The context of the LastFM dataset is music recommendation: each of the 1, listeners (recorded in the dataset) tags the artists they may find fond of overtime.
Since the tag is a binary indicator, the LastFM is an implicit feedback dataset.
There is a total of 186,479 tagging events, where 12,523 artists have been tagged.
• GoodReads 6 .
The book recommendation dataset is obtained from the users’ public shelves on Goodread.com .
We use the user review data on the history and biography sections for their richness.
There are in total 238,450 users, 302,346 unique books, and 2,066, ratings in these sections.
The rating range is also from 1 to 5, and a higher rating indicates more positive feedback.
We concert the dataset to implicit feedback in the same way as the Movielens-1M dataset for the real-data experiments.
Data preprocessing .
As for preprocessing, we point out that the Movielens-1M dataset has been filtered by the publisher, where each user has rated at least 20 movies.
For the LastFM and Goodread datasets, we first filter the infrequent items and users who have less than 20 total interactions as well.
C.1. S IMULATION SETTING
We generate the implicit feedback (clicks) according to the click model of: P ( Y ui = 1) =
P ( R ui = 1) · P ( O ui = 1) , where R ui and O ui indicates the relevance and exposure.
We refer to P ( R ui = 1) as the relevance score.
To carry over the inductive bias of the benchmark dataset to the generated data, we first learn P ( R ui = 1) and P ( O ui = 1) from the explicit feedback data of Movielens-1M and GoodReads.
Our simulation approach does not apply to the LastFM data because it lacks the explicit feedback.
Since the ratings are explicit indicators for relevance, we directly conduct the matrix factorization collaborative filtering (MCF) using the ratings as the response.
We then estimate P sim
( R ui = 1) via σ ( ˆ R ui − u ) p , where σ ( · ) is the sigmoid function, and
u and p are the two hyperparameters that help us control the overall relevance distribution between the original and generated data.
For both
Movielens-1M and GoodReads, we find u = 3 and p = 2 gives a decent approximation.
In this way, we inherit the inductive bias for the relevance to the simulated data.
As for the exposure, we first assume all the exposed movies/books are rated by the user, so we have O ui = 1 if and only if Y ui > 0 .
We then conduct another MCF using the converted O ui as response, and obtain the user and item embedding: Φ( u ) , Φ( i ) .
We further design the exposure model by P sim
( O ui = 1) =
σ ( MLP ([Φ( u ) , Φ( i )])) , where MLP ( · ) is a three-layer feed-forward neural network with ReLU activation and no bias terms, whose parameters are given by the random normal initialization.
In the sequel, the simulated click probability is given by: P sim ( Y ui = 1) =
P sim
( R ui = 1) · P sim ( O ui = 1) , from which we generated the click data by picking the cutoff probability that ensures the click v.s. non-click proportion of the generated data matches that of the original data.
C.1. T RAIN - VALIDATION - TEST SPLIT
We adopt the widely-acknowledged approach for splitting the dataset to training, validation and testing as in He et al.
(2017) and Rendle et al.
In particular, we leverage the ordering information and use the last interaction of each user for testing, the second-to-last interaction for validation, and the previous interactions for training.
All three datasets are provided with the ordering information, so we use the same splitting mechanism.
For the semi-synthetic dataset, since the simulation does not account for the ordering information, we treat the ordering as random and conduct the splitting in the same fashion.
We point out here that using sequential recommendation models is not reasonable in this case, so we do not experiment with the attention models ( Attn ) in the simulation experiments.
C.1. E VALUATION
As we discussed in Section 6, the three metrics that we use for both the real-data and semi-synthetic experiments are Rel@K , NDCG @K and Recall
@K.
We also adopt the widely-acknowledged formulation of NDCG@K and Recall@K as from He et al.
(2017) and Rendle et al.
(2020): for each positive instance (clicked data) of ( u ′ , i ′ , Y u
′ i ′ =
1) , we randomly sample 100 negative items { i 1 , . .
, i 100 } and compute NDCG@K and Recall@K according to where i ′ is ranked among the total of 101 items.
We point out that the recent work by Krichene and Rendle (2020) points out the potential bias issue of using the sampled metrics; however, it is still the mainstream approach that is used in the majority of existing literature.
We apply them here so our results can be directly compared with the results in the relevant literature.
To make up for the bias issue of using sampled metric, we further consider the Rel@K metric, which ranks all the items { i 1 , . .
, i |I| } and see where i ′ is placed for each u ′ .
In this way, we eliminate the bias of using sampled metric.
Essentially, Rel@K equals Recall@K with no sampling when each user interacts with only one item in the testing data (which happens in our experiments by the way we conduct the train-validation-test split).
The only difference in computing the three metrics for the real-data and semi-synthetic experiments is that we replace Y u ′ i ′ by the underlying relevance score of P sim
( R u ′ i ′ = 1) .
In this way, we obtain a more accurate characterization of the actual testing performance.
Finally, for the MovieLens and
Goodreads datasets, we adopt the tradition that the previously-interacted items are not considered during the evaluation for each user.
This is because in practice, the same movie or book should not be exposed to the user repeatedly.
C. M ODEL CONFIGURATION AND TRAINING
We provide the model configurations, training and computation details in this part of the appendix.
C.
M ODEL C ONFIGURATIONS
To achieve a fair comparison, we fix all the user and item embedding dimensions to d = 32 for the
CF models and sequential recommendation model.
For all the baseline models, we select the initial learning rate from { 0.001, 0.005, 0.01, 0.05, 0.1 } , and the ℓ 2 regularization parameter from { 0, 0.01, 0.05, 0.1, 0.2, 0.3 } .
The tuning parameters are selected separately to avoid excessive computations.
The other hyperparameters are identified according to the original papers as for the model-specific configurations, other than the embedding dimension, learning rate, and ℓ 2 regularization.
For instance, NCF considers the hidden dimensions of the feed-forward neural network as the hyperparameters, IPW-CF considers the IPW threshold, Attn considers the hidden dimension and number of attention blocks, AC-MF considers the adversarial strength of the IPW component.
We use the same range as provided in their published implementations, which we discuss later in Appendix C.2.2.
When implemented with our transportation-regularized risk minimization, we focus primarily on using the same ”base model” for f , w and g , e.g. with MCF, NCF or Attn.
In fact, we can pick any combinations for f , w and g , but we find the performances comparable to using the same ”base model”, so we randomly pick some of the configurations to report, instead of exhausting all the possibilities.
Finally, the hyperparameters of the ”base models” persist to our setting, in addition to the penalization parameter λ (in eq.(4)), which we select from { 0.005, 0.1, 0.3, 0.5 } .
C.2. T RAINING DETAILS
For both the proposed approach and the baseline methods, we adopt the widely-applied negativesampling-based training schema according to He et al.
(2017) and Rendle et al.
(2020), for both the real-data and simulation experiments.
In particular, for each positive instance ( Y ui = 1 ), we randomly sample three pairs of user-item tuple and treat them as the negative instances.
We point out that in many existing literature, the number of negative samples per positive instance is treated as a tuning parameter.
Here, we treat it as a fixed quantity and focus on tuning the model-specific configurations.
For all the experiments, we use the binary cross-entropy loss function for classification, and apply early stopping during training when the Rel@K metric does not improve for more than five epochs.
We record the model parameters after each epoch, and select the one that achieves the best validation performance for testing.
Transportation-regularized risk minimization .
When training with the proposed approach, the discount of learning rate (in Algorithm 1) is critical for the two-time-scale GDA.
In a sense it resembles the number of gradient updates under the same learning rate, and we focus on their relative ratios.
In practice, we allow the model of w in the minimization step to have a learning rate different from that of f .
We denote the ratio for f , w and g as step f , step w and step g .
In practice, we select the ratio among { 1 : 1 : 10 , 1 : 5 : 10 , 1 : 10 : 10 , 1 : 10 : 5 } for our experiments, and the sensitivity analysis is provided in Appendix C.3.1.
Baseline models .
The popularity ( Pop ) baseline is implemented with off-the-shelf code, the MCF and NCF models are implemented according to their original implementation published on Github 7 , the IPW-debaised MF ( IPW-MF ) and user-exposure aware MF ( ExpoMF ) models are implemented using their published code 8 with necessary modifications.
The adversarial-counterfactual MF ( ACMF ) method is also implemented using the published code 9 , which we apply directly to our datasets.
The sequential recommendation model with attention mechanism is also implemented according to the original code 10 .
We point out that most of the implementation provided by the authors include the set of hyper parameters they found optimal in their experiments.
We re-select those hyper parameters in each of our settings.
As for the doubly robust joint learning (DR-MF) approach (Wang et al., 2019), the method was originally proposed for the bandit feedback setting or when the feedback data is explicit and the propensity scores can be effectively estimated.
Since we focus on the implicit feedback setting, we use the clicks to estimate the propensities, and plug it to the DR approach.
The causal embedding approach (CE-MF), on the other hand, assumes the access to a uniform exposure dataset.
The authors propose a workaround to construct such data from non-uniform-exposure feedback data (Bonner and Vasile, 2018), however, they take advantage of the explicit feedback (rating).
Since we only have implicit feedback, we random subsample the click data and treat them as if they are from the uniform exposure.
Computation .
All the models are implemented with the auto-differentiation framework of PyTorch 11 .
The computations are conducted on a Linux cluster with 2 Nvidia V100 GPU machines (each with 32 Gb memory) and 32 CPU with a total memory of 100 Gb.
We use the sparse Adamoptimizer to update the hidden factors (user and item embeddings), and the usual Adam optimizer to update the remaining parameters.
This is because both the user and item embeddings are relatively sparse in the datasets.
Therefore, the Adam algorithm, which leverages the momentum of the gradients from the previous batch, may not be ideal for updating the item and user embedding in the current batch.
The sparse Adam optimizer address the above issue
[Table]
C. C OMPLETE N UMERICAL R ESULTS
[Table]
We present the complete numerical results in this part of the appendix.
We start with the testing performance with respect to the methods for MovieLens-1M that are not shown in the main paper due to the space limitation. [Table]
[Table]
Additional results for MovieLens-1M and its synthetic data .
In Table A.1, we show the additional results for the MovieLens experiments, including using the proposed domain-transportation approach with different configurations of f , w and g (denoted by DT-(f/w/g) ).
Note that Attn does not apply to the synthetic data, because the ordering in the synthetic data is randomly generated as we mentioned previously.
For notation, we use the short hand DT-(M/N/N) , where in the parenthesis are the assignments for f , w and g , and M is short for MCF , N is short for NCF .
Also, we experiment with the doubly-robust joint learning approach on MCF (denoted by DR-MF) as proposed by Wang et al.
(2019), and the causal embedding approach on MCF (denoted by CE-MF) proposed in Bonner and Vasile (2018).
The two methods were proposed for the explicit feedback data, and we observe that while they are still able to improve the performance of MCF in our implict feedback setting, their improvements are much less significant compared with the proposed approach.
Also, it appears that the causal embedding approach suffers from high variances, which is due to the workaround when constructing the ”uniform exposure” dataset from implicit feedback data.
Complete results for the LastFM data .
In Table A.3, we show the complete results for the LastFM experiments.
We do not conduct the synthetic experiments on the LastFM data, because unlike the other two datasets, the original dataset consists only of implicit feedback.
Therefore, neither the relevance nor the exposure can be efficiently estimated from the data.
The settings, interpretations and notations for our approach are the same as above.
Complete results for the Goodreads simulation experiment .
In Table A.2 and Table A.3, we show the complete results for the Goodreads experiments.
Similarly, Attn does not apply to the synthetic data for the same reason discussed above.
We then present the complete results in correspondence to Figure 2.
In Figure A.1, we the analysis of the learnt weights on the Goodreads data and its simulation.
In Figure A.2, we show the same set of analysis for the LastFM dataset.
Finally, in Figure A.3, we provide the ablation study and sensitivity analysis on λ , for both Goodreads and LastFM dataset.
C.3. S ENSITIVITY A NALYSIS ON THE OPTIMIZATION SETTING AND OVERLAPPING
To ensure the convergence of the minimax optimization without being trapped in a cycle, it is a standard practice to use the two-time-scale GDA algorithm.
Our situation is even more involved since we not only have a minimax game, but there are two minimization players.
Intuitively, f is moved only when P w has approximated P
f
sufficiently well.
It means that w has converged in the sense that d W ( P f , P w ) is sufficiently small.
Since we use the Monge and Kantorovich formulation to represent the 1-Wasserstein distance d W , i.e., d W ( P, Q ) = sup g :
∥ g ∥
L ≤ 1 (cid:82) D gdP − (cid:82) D gdQ , we need to ensure g to converge every time w moves so that the integral difference is approximately equal to d W ( P f , P w ) .
In other words, this is a three-layer loops: when f moves one step, we need w to move step w steps until convergence; when w moves one step, we need g to move step g steps until convergence.
Figure A.4a shows that as the step w increases when step g is fixed, the overall performance of our method is boosted, which aligns with the above claim that w should converge before f moves.
We observe similar phenomenon when step g increases as step w is fixed, which corroborates the claim that g should converge before w moves.
These results suggest that we might want to use a large step w and a large step g .
However, it can be computationally expensive.
From our experience, we can use a relatively large step g and a slightly smaller step w to achieve a good performance.
In addition, appropriate choices of learning rates lr f , lr w and lr g can expedite the convergence of w and g so that we can use less steps.
C. A NALYSIS FOR THE COMPLETE RESULTS
We first observe from Table A.1 that the sequential recommendation methods achieve overall better performances than the others.
It is expected because the sequential signal in the MovieLens-1M dataset is usually substantial, as point out by Kang and McAuley (2018).
Similar to the previous experiments, DT-Attn improves the performance of the regular Attn model.
As for the variations of our methods, they all improve the performance of the corresponding f model, and even outperforms 1M the DT methods where f , w and g use the same ”base model”.
This is a phenomenon that worth future investigation, since the results might indicate the DT-(f/w/g) methods combine the strengths of the base models.
The results for the Goodreads and its simulation experiments in Table A.2 and Table A.3 show the similar patterns.
Firstly, using the proposed transportation-regularized approach improves the performances of the baseline methods, and the best results are achieved by our DT-X methods.
Secondly, while the other enhanced MF/CF methods also improve upon the baseline MCF model, but their improvements are much less significant than the DT approach.
Thirdly, the DT-(f/w/g) methods also achieve more significant improvements compared with the DT-f methods.
We point out that the sequential recommendation method ( Attn ) does not particularly suit the dataset, since it is outperformed by the non-sequential methods.
Still, DT-Attn achieves better performances than Attn .
Finally, the additional results for the LastFM dataset in Table A.3 have the similar patterns.
As for the analysis on the learnt weights, we observe from Figure A.1 and Figure A.2 that the patterns we discussed in Section 6 for the MovieLens dataset and its simulation holds almost the same for the Goodreads and LastFM datasets.
Firstly, the learnt weights for the instances among the feedback data can be distinguished by whether the instance is positive or negative.
We observe that the positive instance tends to have larger weights than the negative instances, which means the weighting model is learnt to emphasise the ”higher-quality” positive implicit feedback.
Secondly, the learnt weights also get larger for the instances that will be recommended under f , and the larger the score of f , the higher the weights.
It means the weighting function works harder on balancing P f with P w , which conforms to our design.
Lastly, for the simulation, the learnt weights tend to be larger for the negative instances with higher exposure probability, which can be considered the strong negative instances.
This is because they are more likely to be exposed but are not clicked.
C. E FFICIENCY ANALYSIS
In terms of the space complexity, it is evident that our approach need to store the parameters and gradients of the f, w, and g models (if no parameter sharing is considered).
For the time complexity, however, the proposed approach is highly efficient due to the following reasons.
As it is shown in Figure A.5, although the DT-MF approach has a larger per-epoch training time, it makes training progress much faster than the original MCF thanks to the rewieghting and DT regularization.
For the Movielens dataset, DT-X catches up and outperforms X in a couple of training epochs even before the X model converges.
The update of f, w, and g can be efficiently parallelized.
Our approach does not change the inference time since only the f component will be used for evaluation and testing, just like when using f alone.
C. D EPLOYMENT TO REAL - WORLD IR SYSTEM AND ONLINE TESTING
An important motivation of our work is to solve the practical challenges of industrial IR systems, and the working mechanism we described in Section 1 applies to most settings.
In particular, many real-world recommenders are deterministic, and we can only expose a tiny proportion of the catalog to the customers due to the limited slots and business requirements.
For many industrial tasks, the revenue is directly related to the performance of recommender since unqualified recommendations may hurt customer satisfaction severely.
Therefore, the feedback data often covers extremely limited support of the product space.
It causes a realistic dilemma:
• if the candidate recommender f recommends products very differently from the previous exposure, then the feedback data may not provide enough evidence to assert the performance of f ; • if the candidate recommender f recommends products highly similar to the previous exposure, then f will inevitably inherit the various data bias.
[-+0.
-]
[-DT-D&W (variation2) +0.
 +0.09% +0.07%-] From this realistic perspective, our approach aims to find the right balance between the two extremes: the weighting model helps to reshape the feedback data domain (to better fit the objective risk), and the regularizer enforces f to be sufficiently close to the reshaped feedback data domain.
To test the competence of our approach for industrial tasks, we apply the proposed domain transportation approach to an industrial recommendation task, and conduct online A/B/C testing to examine the real-world performance.
The context of our task is to recommend items that are similar to the current webpage’s product for a major e-commerce platform in the U.S.
The platform hosts more than a hundred million products [Table] ranging from high-end electronics to daily grocery.
Each item has its contextual features, including rating, price, taxonomy etc.
The pre-trained embeddings are also available for the items obtained from a modified version of the SGNS algorithm.
When applying the item embedding to the model, we use the pre-trained embeddings as initialization and update the parameters with a lower learning rate than the other model parameters.
We employ an architecture that is similar to the classical Deepand-wide model, initially proposed by Cheng et al.
(2016) for recommending YouTube videos, with the difference that we do not use the real-time features here.
We trained three versions of the deep-and-wide model offline:
• control : directly training f using the implicit feedback data; • variation1 : first training an exposure model using the past exposure log, apply its normalized version (for controlling the variance) as the weighting function w , and then train f in a way similar to the baseline IPW-MF mentioned in Section 5; • variation2 : train f using the proposed transportation-regularized risk minimization, where w and f are neural CF models that take the item embeddings as input.
All three versions are trained under the same setting.
We deploy online A/B/C testing to examine the real-world performances, and the monitored metrics and testing results are provided in Figure A.6.
We see that our transportation-regularized risk minimization (variation2) consistently outperforms the original model (control) and the IW-adjusted model (variation1) in term of the gross merchandise value (GMV), which is the most critical metric for industrial recommender system.
Also, the proposed approach improves the other monitored metrics (lower panel of Figure A.6) more signif- icantly than the IW-adjusted model.
Our deployment results further suggest the capability of the proposed approach for improving the performance of industrial IR systems.
D E XTENDED L ITERATURE R EVIEW
When viewing our solution from the causal inference perspective, our objective resembles the counterfactual loss proposed by Shalit et al.
(2017) focusing on individual treatment effect, which was also discussed in the following work (Johansson et al., 2019; 2020).
Further, Johansson et al.
(2019) studies from the domain adaptation perspective the issues of insufficient overlapping.
Unlike these works where the target domain of interest is fixed in advance, IR systems get to choose the counterfactual world in which they want to act.
It leads to the main difference between our work and that line of research: we can control the domain in which the counterfactual risk is computed.
Recently, there has been fruitful research progress in off-policy learning and data missing-not-atrandom.
Again, most existing methods treat the uniform-exposed domain as the target, while we aim at the deployment domain formed by the IR system.
Further, most of them require either a sufficiently randomized logging policy or explicit ratings in the feedback, while our approach is primarily designed for implicit feedback under a deterministic policy.
For instance, Sachdeva et al.
(2020) studied the overlapping issue for learning from bandit feedback.
Similarly, the causal embedding approach (Bonner and Vasile, 2018) requires access to a subset of randomly exposed feedback, and many others rely on the explicit ratings to estimate the historical propensities (Wang et al., 2019; Saito, 2020; Liu et al., 2020).
On the other hand, the interventional nature of IR systems is also discussed in Xiao and Wang (2021), but their solution is devised from the reinforcement learning setting.
The Wasserstein distance has also been employed by distribution-robust optimization (DRO) (Kuhn et al., 2019).
For instance, Si et al.
(2020) uses Wasserstein distance to assert certain robustness to the contextual feature distribution during bandit off-policy learning, and Faury et al.
(2020) proposes a robust counterfactual learning framework to handle the various uncertainties in bandit feedback data.
Our work differs fundamentally from that research venue as we use Wasserstein distance to regularize learnt policy (via constraining its deployment domain) rather than assert its robustness.
Further, DRO methods often have a min-max formulation, while our initial objective in (2) and (3) tackle a min-min problem.